Package: ankh
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingAnkhis the first general-purpose protein language model trained on Google'sTPU-V4surpassing the state-of-the-art performance with dramatically less parameters, promoting accessibility to research innovation via attainable resources.This repository will be updated regulary withnew pre-trained models for proteinsin part of supporting thebiotechcommunity in revolutinizing protein engineering using AI.Table of ContentsInstallationModels AvailabilityDataset AvailabilityUsageOriginal downstream PredictionsFollowup use-casesComparisons to other toolsCommunity and ContributionsHave a question?Found a bug?RequirementsSponsorsTeamLicenseCitationInstallationpython-mpipinstallankhModels AvailabilityModelankhHugging FaceAnkh Largeankh.load_large_model()Ankh LargeAnkh Baseankh.load_base_model()Ankh BaseDatasets AvailabilityDatasetHugging FaceRemote Homologyload_dataset("proteinea/remote_homology")CASP12load_dataset("proteinea/secondary_structure_prediction", data_files={'test': ['CASP12.csv']})CASP14load_dataset("proteinea/secondary_structure_prediction", data_files={'test': ['CASP14.csv']})CB513load_dataset("proteinea/secondary_structure_prediction", data_files={'test': ['CB513.csv']})TS115load_dataset("proteinea/secondary_structure_prediction", data_files={'test': ['TS115.csv']})DeepLocload_dataset("proteinea/deeploc")Fluorescenceload_dataset("proteinea/fluorescence")Solubilityload_dataset("proteinea/solubility")Nearest Neighbor Searchload_dataset("proteinea/nearest_neighbor_search")UsageLoading pre-trained models:importankh# To load large model:model,tokenizer=ankh.load_large_model()model.eval()# To load base model.model,tokenizer=ankh.load_base_model()model.eval()Feature extraction using ankh large example:model,tokenizer=ankh.load_large_model()model.eval()protein_sequences=['MKALCLLLLPVLGLLVSSKTLCSMEEAINERIQEVAGSLIFRAISSIGLECQSVTSRGDLATCPRGFAVTGCTCGSACGSWDVRAETTCHCQCAGMDWTGARCCRVQPLEHHHHHH','GSHMSLFDFFKNKGSAATATDRLKLILAKERTLNLPYMEEMRKEIIAVIQKYTKSSDIHFKTLDSNQSVETIEVEIILPR']protein_sequences=[list(seq)forseqinprotein_sequences]outputs=tokenizer.batch_encode_plus(protein_sequences,add_special_tokens=True,padding=True,is_split_into_words=True,return_tensors="pt")withtorch.no_grad():embeddings=model(input_ids=outputs['input_ids'],attention_mask=outputs['attention_mask'])Loading downstream models example:# To use downstream model for binary classification:binary_classification_model=ankh.ConvBertForBinaryClassification(input_dim=768,nhead=4,hidden_dim=384,num_hidden_layers=1,num_layers=1,kernel_size=7,dropout=0.2,pooling='max')# To use downstream model for multiclass classification:multiclass_classification_model=ankh.ConvBertForMultiClassClassification(num_tokens=2,input_dim=768,nhead=4,hidden_dim=384,num_hidden_layers=1,num_layers=1,kernel_size=7,dropout=0.2)# To use downstream model for regression:# training_labels_mean is optional parameter and it's used to fill the output layer's bias with it,# it's useful for faster convergence.regression_model=ankh.ConvBertForRegression(input_dim=768,nhead=4,hidden_dim=384,num_hidden_layers=1,num_layers=1,kernel_size=7,dropout=0,pooling='max',training_labels_mean=0.38145)Original downstream PredictionsSecondary Structure Prediction (Q3):ModelCASP12CASP14 (HARD)TS115CB513Ankh Large83.59%77.48%88.22%88.48%Ankh Base80.81%76.67%86.92%86.94%ProtT5-XL-UniRef5083.34%75.09%86.82%86.64%ESM2-15B83.16%76.56%87.50%87.35%ESM2-3B83.14%76.75%87.50%87.44%ESM2-650M82.43%76.97%87.22%87.18%ESM-1b79.45%75.39%85.02%84.31%Secondary Structure Prediction (Q8):ModelCASP12CASP14 (HARD)TS115CB513Ankh Large71.69%63.17%79.10%78.45%Ankh Base68.85%62.33%77.08%75.83%ProtT5-XL-UniRef5070.47%59.71%76.91%74.81%ESM2-15B71.17%61.81%77.67%75.88%ESM2-3B71.69%61.52%77.62%75.95%ESM2-650M70.50%62.10%77.68%75.89%ESM-1b66.02%60.34%73.82%71.55%Contact Prediction Long Precision Using Embeddings:ModelProteinNet (L/1)ProteinNet (L/5)CASP14 (L/1)CASP14 (L/5)Ankh Large48.93%73.49%16.01%29.91%Ankh Base43.21%66.63%13.50%28.65%ProtT5-XL-UniRef5044.74%68.95%11.95%24.45%ESM2-15B31.62%52.97%14.44%26.61%ESM2-3B30.24%51.34%12.20%21.91%ESM2-650M29.36%50.74%13.71%22.25%ESM-1b29.25%50.69%10.18%18.08%Contact Prediction Long Precision Using attention scores:ModelProteinNet (L/1)ProteinNet (L/5)CASP14 (L/1)CASP14 (L/5)Ankh Large31.44%55.58%11.05%20.74%Ankh Base25.93%46.28%9.32%19.51%ProtT5-XL-UniRef5030.85%51.90%8.60%16.09%ESM2-15B33.32%57.44%12.25%24.60%ESM2-3B33.92%56.63%12.17%21.36%ESM2-650M31.87%54.63%10.66%21.01%ESM-1b25.30%42.03%7.77%15.77%Localization (Q10):ModelDeepLoc DatasetAnkh Large83.01%Ankh Base81.38%ProtT5-XL-UniRef5082.95%ESM2-15B81.22%ESM2-3B81.22%ESM2-650M82.08%ESM-1b80.51%Remote Homology:ModelSCOPe (Fold)Ankh Large61.01%Ankh Base61.14%ProtT5-XL-UniRef5059.38%ESM2-15B54.48%ESM2-3B59.24%ESM2-650M51.36%ESM-1b56.93%Solubility:ModelSolubilityAnkh Large76.41%Ankh Base76.36%ProtT5-XL-UniRef5076.26%ESM2-15B60.52%ESM2-3B74.91%ESM2-650M74.56%ESM-1b74.91%Fluorescence (Spearman Correlation):ModelFluorescenceAnkh Large0.62Ankh Base0.62ProtT5-XL-UniRef500.61ESM2-15B0.56ESM-1b0.48ESM2-650M0.48ESM2-3B0.46Nearest Neighbor Search using Global Pooling:ModelLookup69K (C)Lookup69K (A)Lookup69K (T)Lookup69K (H)Ankh Large0.830.720.600.70Ankh Base0.850.770.630.72ProtT5-XL-UniRef500.830.690.570.73ESM2-15B0.780.630.520.67ESM2-3B0.790.650.530.64ESM2-650M0.720.560.400.53ESM-1b0.780.650.510.63TeamTechnical University of Munich:Ahmed ElnaggarBurkhard RostProteinea:Hazem EssamWafaa AshrafWalid MoustafaMohamed ElkerdawyUniversity of Columbia:Charlotte RochereauSponsorsGoogle CloudLicenseAnkh pretrained models are released under the under terms of theCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International.Community and ContributionsThe Ankh project is aopen source projectsupported by various partner companies and research institutions. We are committed toshare all our pre-trained models and knowledge. We are more than happy if you could help us on sharing new ptrained models, fixing bugs, proposing new feature, improving our documentation, spreading the word, or support our project.Have a question?We are happy to hear your question in our issues pageAnkh! Obviously if you have a private question or want to cooperate with us, you can alwaysreach out to us directlyviaHello.Found a bug?Feel free tofile a new issuewith a respective title and description on the theAnkhrepository. If you already found a solution to your problem,we would love to review your pull request!.✏️  CitationIf you use this code or our pretrained models for your publication, please cite the original paper:@article{elnaggar2023ankh,
  title={Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling},
  author={Elnaggar, Ahmed and Essam, Hazem and Salah-Eldin, Wafaa and Moustafa, Walid and Elkerdawy, Mohamed and Rochereau, Charlotte and Rost, Burkhard},
  journal={arXiv preprint arXiv:2301.06568},
  year={2023}
}