Package: progres
Progres - Protein Graph Embedding SearchThis repository contains the method from the pre-print:Greener JG and Jamali K. Fast protein structure searching using structure graph embeddings. bioRxiv (2022) -linkIt provides theprogresPython package that lets you search structures against pre-embedded structural databases and pre-embed datasets for searching against. Searching typically takes 1-2 s and is faster for multiple queries. CurrentlySCOPe,CATH,ECODand theAlphaFold structures for 21 model organismsare provided for searching against. We may add others such as the wholeAlphaFold databaseand theESM Metagenomic Atlasin future.This is work in progress software - the implementation, API and results may change.
Appropriate version numbers will distinguish versions.
Since the pre-print the model has been updated and the AlphaFold structures have been made available to search against.
Please open issues orcontact mewith any feedback.
Training scripts and datasets will be made available on publication.InstallationPython 3.6 or later is required. The software is OS-independent.InstallPyTorch1.11 or later,PyTorch ScatterandPyTorch Geometricas appropriate for your system. A GPU is not required and will only provide speedup in certain situations since multiple workers can be used on CPU. Example commands:condainstallpytorch==1.12.0-cpytorch
condainstallpytorch-scatterpyg-cpygRunpip install progres, which will also installBiopython,mmtf-pythonandeinopsif they are not already present.The first time you run the software the trained model and pre-embedded databases (~340 MB) will be downloaded to the package directory fromZenodo, which requires an internet connection. This can take a few minutes.UsageOn Unix systems the executableprogreswill be added to the path during installation.
On Windows you can call thebin/progresscript with python if you can't access the executable.Runprogres -hto see the help text andprogres {mode} -hto see the help text for each mode.
The modes are described below but there are other options outlined in the help text.
For example the-dflag sets the device to run on; this iscpuby default since this is usually fastest for searching, butcudamay be slightly faster when embedding a dataset.Searching a structure against a databaseTo search a PDB filequery.pdbagainst domains in the SCOPe database and print output:progressearch-qquery.pdb-tscope95# QUERY_NUM: 1
# QUERY: query.pdb
# QUERY_SIZE: 150 residues
# DATABASE: scope95
# PARAMETERS: minsimilarity 0.8, maxhits 100, progres v0.2.0
# HIT_N  DOMAIN   HIT_NRES  SIMILARITY  NOTES
      1  d1a6ja_       150      1.0000  d.112.1.1 - Nitrogen regulatory bacterial protein IIa-ntr {Escherichia coli [TaxId: 562]}
      2  d2a0ja_       146      0.9988  d.112.1.0 - automated matches {Neisseria meningitidis [TaxId: 122586]}
      3  d3urra1       151      0.9983  d.112.1.0 - automated matches {Burkholderia thailandensis [TaxId: 271848]}
      4  d3lf6a_       154      0.9971  d.112.1.1 - automated matches {Artificial gene [TaxId: 32630]}
      5  d3oxpa1       147      0.9968  d.112.1.0 - automated matches {Yersinia pestis [TaxId: 214092]}
...-qis the path to the query structure file. Alternatively,-lis a text file with one query file path per line and each result will be printed in turn. This is considerably faster for multiple queries since setup only occurs once and multiple workers can be used.-tis the pre-embedded database to search against. Currently this must be either one of the databases listed below or the file path to a pre-embedded dataset generated withprogres embed.-fdetermines the file format of the query structure (guess,pdb,mmcif,mmtforcoords). By default this is guessed from the file extension, withpdbchosen if a guess can't be made.coordsrefers to a text file with the coordinates of a Cα atom separated by white space on each line.-sis the minimum similarity threshold above which to return hits, default 0.8.-mis the maximum number of hits to return, default 100.Query structures should be a single protein domain, though it can be discontinuous (chain IDs are ignored).
You can slice out domains manually using software such as thepdb_selrescommand frompdb-tools.Interpreting the hit descriptions depends on the database being searched.
The domain name often includes a reference to the corresponding PDB file, for example d1a6ja_ refers to PDB ID 1A6J chain A, and this can be opened in theRCSB PDB structure viewto get a quick look.The available pre-embedded databases are:NameDescriptionNumber of domainsSearch time (1 query)Search time (100 queries)scope95ASTRAL set ofSCOPe2.08 domains clustered at 95% seq ID35,3711.43 s2.47 sscope40ASTRAL set ofSCOPe2.08 domains clustered at 40% seq ID15,1271.36 s2.25 scath40S40 non-redundant domains fromCATH23/11/2231,8841.44 s2.42 secod70F70 representative domains fromECODdevelop28771,6351.48 s3.36 saf21orgAlphaFoldstructures for 21 model organisms split into domains byCATH-Assign338,2582.15 s7.88 sSearch time is for a 150 residue protein (d1a6ja_ in PDB format) on an Intel i9-10980XE CPU with 256 GB RAM and PyTorch 1.11.
Times are given for 1 or 100 queries.Pre-embed a dataset to search againstTo embed a dataset of structures, allowing it to be searched against:progresembed-lfilepaths.txt-osearchdb.pt-lis a text file with information on one structure per line, each of which will be one entry in the output. White space should separate the file path to the structure and the domain name, with optionally any additional text being treated as a note for the notes column of the results.-ois the output file path for the PyTorch file containing a dictionary with the embeddings and associated data. It can be read in withtorch.load.-fdetermines the file format of each structure as above (guess,pdb,mmcif,mmtforcoords).Again, the structures should correspond to single protein domains.Python libraryprogrescan also be used in Python, allowing it to be integrated into other methods:importprogresaspg# Search as above, returns a list where each entry is a dictionary for a query# A generator is also available as pg.progres_search_generatorresults=pg.progres_search(querystructure="query.pdb",targetdb="scope95")results[0].keys()# dict_keys(['query_num', 'query', 'query_size', 'database', 'minsimilarity',#            'maxhits', 'domains', 'hits_nres', 'similarities', 'notes'])# Pre-embed as above, saves a dictionarypg.progres_embed(structurelist="filepaths.txt",outputfile="searchdb.pt")importtorchtorch.load("searchdb.pt").keys()# dict_keys(['ids', 'embeddings', 'nres', 'notes'])# Read a structure file into a PyTorch Geometric graphgraph=pg.read_graph("query.pdb")graph# Data(x=[150, 67], edge_index=[2, 2758], coords=[150, 3])# Embed a single structureembedding=pg.embed_structure("query.pdb")embedding.shape# torch.Size([128])# Embed Cα coordinates and search with the embedding# This is useful for using progres in existing pipelines that give out Cα coordinates# queryembeddings should have shape (128) or (n, 128)#   and should be normalised across the 128 dimensioncoords=pg.read_coords("query.pdb")embedding=pg.embed_coords(coords)# Can take a list of coords or a tensor of shape (nres, 3)results=pg.progres_search(queryembeddings=embedding,targetdb="scope95")# Get the similarity score (0 to 1) between two embeddings# The distance (1 - similarity) is also available as pg.embedding_distancescore=pg.embedding_similarity(embedding,embedding)score# tensor(1.) in this case since they are the same embedding# Get all-v-all similarity scores between 1000 embeddingsembs=torch.nn.functional.normalize(torch.rand(1000,128),dim=1)scores=pg.embedding_similarity(embs.unsqueeze(0),embs.unsqueeze(1))scores.shape# torch.Size([1000, 1000])NotesThe implementation of the E(n)-equivariant GNN usesEGNN PyTorch.