OpenProteinSet: Training data for structural biology at scale
Authors: Gustaf Ahdritz
Nazim Bouatta
Sachin Kadyan
Lukas Jarosch
Dan Berenberg
Ian Fisk
Andrew Martin Watkins
Stephen Ra
Richard Bonneau
Mohammed AlQuraishi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: openproteinset
multiple sequence alignment
structural template
protein folding
alphafold 2
openfold
TL;DR: The largest database of multiple sequence alignment data, for use as training data in applications in bioinformatics.

Abstract: Multiple sequence alignments (MSAs) of proteins encode rich biological information and have been workhorses in bioinformatic methods for tasks like protein design and protein structure prediction for decades. Recent breakthroughs like AlphaFold2 that use transformers to attend directly over large quantities of raw MSAs have reaffirmed their importance. Generation of MSAs is highly computationally intensive, however, and no datasets comparable to those used to train AlphaFold2 have been made available to the research community, hindering progress in machine learning for proteins. To remedy this problem, we introduce OpenProteinSet, an open-source corpus of more than 16 million MSAs, associated structural homologs from the Protein Data Bank, and AlphaFold2 protein structure predictions. We have previously demonstrated the utility of OpenProteinSet by successfully retraining AlphaFold2 on it. We expect OpenProteinSet to be broadly useful as training and validation data for 1) diverse tasks focused on protein structure, function, and design and 2) large-scale multimodal machine learning research.

OpenProteinSet: Training data for structural biology at scale
Gustaf Ahdritz Harvard University gahdritzeg. harvard.
Nazim Bouatta
Laboratory of Systems Pharmacology, Harvard Medical School bazin_bouatta@hm hww.aad.a.................aa.... edu
Sachin Kadyan Columbia University
Lukas Jarosch
Columbia University
Daniel Berenberg
Prescient Design, Genentech & Department of Computer Science, New York University
Ian Fisk Flatiron Institute
Andrew M. Watkins
Stephen Ra
Prescient Design, Genentech
Prescient Design, Genentech
Richard Bonneau Prescient Design, Genentech
Mohammed AlQuraishi
Department of Systems Biology, Columbia University m. alquraisht @columbi edu
Abstract
Multiple sequence alignments (MSAs) of proteins encode rich biological infor- mation and have been workhorses in bioinformatic methods for tasks like protein design and protein structure prediction for decades. Recent breakthroughs like Al- phaFold2 that use transformers to attend directly over large quantities of raw MSAs have reaffirmed their importance. Generation of MSAs is highly computationally intensive, however, and datasets comparable to those used to train AlphaFold2 have been made available to the research community, hindering progress in machine learning for proteins. To remedy this problem, introduce OpenProteinSet, open-source corpus of more than 16 million MSAs, associated structural homologs from the Protein Data Bank, and AlphaFold2 protein structure predictions. We have previously demonstrated the utility of OpenProteinSet by successfully retraining AlphaFold2 it. We expect OpenProteinSet be broadly useful as training and validation data for 1) diverse tasks focused on protein structure, function, and design and 2) large-scale multimodal machine learning research.
Introduction
Multiple sequence alignments (MSAs) comprise sets of related protein sequences with their amino acid residues in correspondence ("aligned""). MSAs encode rich information about the functional and structural features of a protein family by summarizing the (co-Jevolutionary trajectory of its sequence.
MSAs are used in a wide variety of bioinformatic applications, including protein function prediction 3 4 5), protein language models (6]
7
8 9 10), discase variant prediction [IT] 12), phylogeny
protein design [5 16
17 protein classification [8], and, most notably, protein structure
prediction
19
26
29
20
24 Early work the latter, culminating
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

1
48
MR WWL.ELVWLCCACSSSFFPEEEEVVVIVVVVV I WENTIISSPAIIN IPALLAG
FIAVALVATILAGAAGGHHI PE VPV SALVG -LVGILLVAALAGAASKKPP PEPTNEE -IVVVVILLIGLGGECCC PAA.CCASSPPPPPPPPOO
Figure 1: MSA primer. Five rows of the OpenProteinSet MSA for PDB protein 3ZBI, chain C 22 Each row of MSA is protein sequence. Proteins are one-dimensional strings composed with a vocabulary of 20 amino acids-or "residues" each represented by a letter. The target or "query" protein is given in the first row of the MSA. Subsequent rows are evolutionarily related ("homologous") proteins retrieved from sequence database on the basis of similarity to the query sequence. To improve alignments and accommodate sequences whose length has changed over time, MSA alignment software can insert "gaps" (represented here by dashes) in or delete residues from homologous sequences. Highlights indicate conserved residues. The number of homologous sequences in an MSA ("depth") and their diversity both contribute to the MSA's usefulness.
in the original AlphaFold, achieved notable success by training models on summary statistics derived from MSAs T9
25
26
27
28
30
32
20 2T More recently, large transformer-lik neural
networks 33 that predict protein structure by directly attending over raw MSAs came to prominence (7). Among them, AlphaFold2 reached pear-experimental accuracy for most proteins the 14th biannual Critical Assessment of Structure Prediction (CASP) by attending over raw MSAs alongside structural templates of homologous proteins [23]. Follow-up work, including RoseTTAFole and the state-of-the-arr protein complex structure prediction model AlphaFold -uuliiee [24, 34), build on the same techniques. The dependence of these methods on sufficiently deep, diverse MSAS and close structural homologs is evidenced by the fact that they perform worst on proteins that lack them [23]
Despite the central importance of MSAs, the quantity of precomputed MSAs accessible the research community has not kept pace with the demands of modern machine learning methods. Large models like AlphaFold2 or MSA Transformer 7 were trained on internal datasets of millions of MSAs.
and the computation of various official databases of AlphaFold2 predictions [35] 36, 37 would have required hundreds of millions more. None of this data has yet been released the public, however, and existing public MSA databases [3
39
40) are comparatively small and outdated. Raw sequence
and structure data are available in large quantities under open licenses (23) 41 42 43) and there also exist several mature, open-source software suites for computing MSAs at varying levels of sensitivity 44 45 46). Together, these resources are sufficient to generate MSAs at scale; indeed, they were used to create the aforementioned unreleased datasets. Nevertheless, doing so is computationally expensive. Depending on target sequence length and the size of the sequence database being searched, generating single MSA with high sensitivity can take several hours. This effectively renders research at the forefront of protein machine learning and bioinformatics inaccessible to all but few large rescarch groups.
Here, we present OpenProteinnet,, a large corpus of precomputed MSAs suitable for training bioin- formatic models at the scale of AlphaFold2 and beyond. OpenProteinSet contains an updated reproduction of AlphaFold2's s unreleased training set, including MSAs and structural template hits for all unique Protein Data Bank (PDB) chains. It also incorporates more than sixteen million MSAs, computed for each cluster in Uniclust30 [47). From these, we identify maximally diverse and deep subset of MSAs that are well-suited for AlphaFold2-style training runs and provide associated AlphaFold2 structure predictions.
We have demonstrated the utility of OpenProteinSet by using it to train OpenFold, a trainable, open- source reproduction of AlphaFold2 [48]. achieving accuracy parity with that of DeepMind's original model. Model parameters resulting from these experiments have been made publicly available.
Not counting these validation experiments postprocessing, OpenProteinSet represents millions of compute-hours.
we provide an overview of the composition of OpenProteinSet in Section 3 Section describes our retraining experiments. We conclude with
After a brief review of related work in Section 2
discussion in 6

---

Sequence origin
Count (approx.)
MSA
Template hits
Structure
PDB (all unique chains)
140,000
Experimentally determined Predicted by AlphaFold2
Uniclust30 (filtered)
270,000
Uniclust30 (unfiltered)
16 million
X
X
Table 1: OpenProteinSet at glance.
2 Related work
MSAs in structural bioinformatics: Techniques based on identifying residue-residue correlations in MSAs ("co-variation analysis") are ubiquitous in structural bioinformatics. They have existed in various forms for more than two decades 49 50), but were initially constrained by the unavailability of sufficient protein sequence data to generate deep MSAs (i.e., comprising many highly diverse se- quences). With the onset of next- next-generation sequencing technology, exponential growth in sequenced genomes and metagenomes led to an explosion in the availability of protein sequence data.
This explosion enabled some of the first successful applications of MSA-based structure prediction To date, modern machine learning-base based approaches rely almost exclusively on MSAs. The first successful models applied residual and convolutional architectures to preprocessed MSA summary statistics [27 28 20
methods proteins [19 25 26
21 32). The MSA Transformer was the first to successfully apply transformers to a large corpus (26 million) of unprocessed MSAs in an unsupervised fashion [7], extending prior work on protein language models (PLMs) [TO 51).
30
31
Contemporaneously. AlphaFold2 was developed to take MSAs as input to predict protein structures and is additionally trained with an unsupervised BERT-style masked MSA prediction objective [52]. The resulting model, along with its successor AlphaFold-Multinee has been widely recognized as a revolution in protein structure prediction. Since then, protein structure prediction models that replace MSAs with embeddings from giant PLMs have emerged (22) 9 53). They show promise as an emerging technology, but they have so far failed to match the performance of MSA-basee methods across the board, significantly underperforming AlphaFold2-baseee entrants difficult targets the most recent installment of CASP [54).
While protein structure prediction is perhaps the most celebrated use case for MSAs, they are broadly used other areas of bioinformatics. Analogously to natural language processing, unsupervised language modeling of raw MSAs produces rich representations with broad applicability, including protein design [16), semantic similarity inference (55), and few-shot protein function prediction, where MSA-based models outperformed comparable models trained on individual sequences alone (5). Long before transformers, summary statisties manually derived from MSAs were already indispensable inputs for diverse tasks ranging from protein classification prediction [2).
18 to disease variant
There exists a large ecosystem of software for computing MSAs by querying large sequence databases. The commonly used programs HHMer [45) and HHblits 44) are highly sensitive, identifying evolutionarily related proteins with high recall and precision. These tools are slow and memory-intensivve however, they may run for several hours or even days to compute a single MSA. As alternative, the efficient MMSeqs2 method trades off sensitivity for an order-of-magniuee improvement in runtime and is commonly used for fast inference with AlphaFold2, most notably in ColabFold [56]. Like the MSAs on which AlphaFold2 was trained, OpenProteinSet MSAs are computed with HHMer and HHbbits for maximal sensitivity.
MSA software:
Responding to the high demand for precomputed MSAs, the community has produced a handful of public MSA repositories. ProteinNet, a repository standardized protein data for the purposes of machine learning, includes MSAs for approximately 100,000 Protein Data Bank (PDB) protein structures released before May 2016 [40). Earlier databases are much smaller and less diverse 38 39). After the initial release of OpenProteinSet in June 2022, handful of other open MSA repositories have begun to appear, including PSP, a repository of approximately million MS computed with MMSeqs2 [57), and similar reproduction of about 500,000 MSAs generated according to the procedures outlined in the AlphaFold2 paper (58). OpenProteinSet more accurate and larger than any other MSA database.
MSA databases:
3

---

Count
CDF
1.0e+02
0.0
1908
410 Protoin long th
Protein length
1.0000
UniRef90
1.0++00
Count
2030
5000
1000
2000
3153
M Gniffy
1.0e+04
0.6
1.00033
2000
2020 3000
4022
5000
1000
2000
3003
4000 5000
MSA depth
MSA depth
Figure 2: PDB MSA statistics. (First row) Number of proteins by sequence length in the PBD portion of OpenProteinSet (left) and the corresponding cumulative density function (CDF) (right). The mean length is 265; the median is 218. (Bottom rows) Depths of MSAs in the PDB portion of OpenProteinSet (left) and the corresponding cumulative density function (CDF) (right). Note that three MSAs are computed for each PDB chain in OpenProteinSet: one using BFD and Uniclust30 (top), one using UniRef90 (middle), and one using MGnify (bottom).
Methodology
OpenProteinSet consists of more than 16 million unique MSAs generated according to the procedures outlined the AlphaFold2 paper (23). This count includes MSAs for all 140,000 unique chains available in the PDB as of April 2022, immediately before the beginning of CASP15, and 16 million MSAs computed for each sequence cluster in Uniclust30 against the same database. From this latter set, we identify 270,000 maximally diverse representative clusters suitable to e.g. serve as the self-distillation set in the AlphaFold2 training procedure. Structural template hits and structure files also available for this set and all PDB chains.
For each PDB chain, we compute three MSAs using different alignment tools and sequence databases. JackHMMer 45 was used to separately search MGnify [2) and UniRef90 [59]] HHblits-v3 was used search the Big Fantastic Database (BFD) (23) and Uniclust30 [47]. BFD a large sequence database prepared for AlphaFold2 that draws its approximately 2.2 billion entries from reference databases, metagenomes, and metatranscriptomes. MgNify (as of 2019) is another environmental database of approximately 300 million sequences. UniRef90 and Uniclust30 are clusterings of UniprotKB [43) proteins at 90% and 30% pairwise sequence identity, respectively, using different clustering algorithms.

---

10
08
06-
1 20
1.00+00
1908
Protein length
Protoin long th
300
00
1. 90
1.6e225
MSA depth
MSA depth
Figure 3: Uniclust30 MSA statistics. (Top) Number of proteins sequence length in the Uniclust30 portion of OpenProteinSet (left) and the corresponding cumulative density function (CDF) (right). The mean length is 255; the median 153. (Bottom) Depths of MSAs the Uniclust30 portion of OpenProteinSet (left) and the corresponding cumulative density function (CDF) (right). The average MSA depth is 940; the median is 262.
Structural templates were identified by searching PDB70 (60] using the UniRef90 MSA using HHSearch (44). Corresponding structures can be retrieved from publicly available PDB mmCIF files using scripts OpenFold (48).
As in the procedure used to generate AlphaFold2's training set, we changed some of the default options of MSA generation tools. For list of specific command-line options changed, please consult the supplementary material. One important change is that HHBlits was run for three iterations.
To generate the Uniclust30 MSAs, performed all-against-all search on Uniclust30 using HHblits-v3 with the same parameter settings as before. This yielded approximately 16 million MSAs, one for each cluster.
To create filtered subset of diverse and deep MSAs, we then iteratively removed MSAs whose representative chain appeared in the greatest number of other MSAs. This was repeated until each representative chain appeared only its own MSA. For parity with corresponding (unreleased) AlphaFold2 set, we further removed clusters whose representative sequences were longer than 1 1,024 residucs or shorter than 200. Finally, we removed clusters whose corresponding MSAs contained fewer than 200 sequences, leaving just 270,262 MSAs. Template hits HHsearch against PDB70. For each representative chain in this subset, we generated structure predictions using OpenFold run with AlphaFold2 weights. Note that, unlike the hundreds of millions of AlphaFold2 predictions made available by DeepMind and EMBL-EBI [3] 36 37]. these are paired with high-quality, diverse MSAs, making it possible to them training data for structure prediction models. All of the above the 16 million unfiltered Uniclust30 MSAs and filtered-chain template hits and structure predictions-arr included in OpenProteinSet.
again computed using
Overall, the MSAs in OpenProteinSet represent more than four million hours computation. Its contents are summarized in Table [

---

All MSAs are in A3M format Template hits are provided in HHSearch's 's HHR format, while structure predictions are PDB format. All data is made available under the CC BY 4.0 license.
For all MSAs currently in OpenProteinSet, we used copies of UniRef90 downloaded on December 19, 2021, BFD downloaded on December 20, 2021, Uniclust30 downloaded December 28, 2021, and MGnify downloaded on January 14, 2022. To compute templates, we used PDB70 downloaded on December 19, 2021. In all cases, we used the most recent versions of each database available at the time. As we update OpenProteinSet with new sequences, we will continually upgrade them.
We used HH-suite version 3.3.0 (commit hash dc74ac) and jackhmmer from HMMER3.1.
4 Experiments
To demonstrate the utility of OpenProteinSet, we used it as training data for a replication of Al- phaFold2, groundbreaking but previously unreplicated protein structure prediction network trained on raw MSAs. Our AlphaFold2 training code is implemented in OpenFold, our open-source repro- duction of the AlphaFold2 training code (48).
First, we simulated the full AlphaFold2 training procedure outlined in Table 4 of the supplement to the AlphaFold2 paper. We used the PDB component of OpenProteinSet as the initial training set and our set of 270,000 filtered Uniclust30 proteins as the self-distillation set. We used a PDB cutoff of December 2021. Training was the full model from scratch, original AlphaFold2 weights were used as the pre-distillation model to generate Uniclust30 structure predictions.
cluster of 44 A100s. Given the prohibitive costs of training
To evaluate the resulting OpenFold weights against AlphaFold2, we computed model_1 1 predictions for each currently available "all groups" CASP15 domains (n = 90) and evaluated them using the GDT-TS score [61]. OpenFold reached a mean score of 73.8 (95% confidence interval = 68.6 78.8) while AlphaFold2 reached 74.6 (95% confidence interval = 69.7 79.22). Confidence intervals of each mean are estimated from 10,000 bootstrap samples. OpenFold did at least as well as AlphaFold2 on exactly 50% of targets. Superimposed predictions are shown in Figure 4
Weights from this experiment are available under permissive license in the OpenFold GitHub repository
Next, estimate variance from different weight initializations and other sources of randomness in training, we trained 15 models on the PDB tranche with different seeds for 10,000 initial training steps (compared more than 75,000 in the full training run), taking advantage of the fact that OpenFold/AlphaFoold) achieves much of its final accuracy relatively quickly (as much as 90% of its final accuracy in less than 3% of the total training time). We observe very little run-to-run variability. For assessment we use IDDT-Ca [62]. commonly used accuracy measure for protein structure predictions. We found that on validation set of 180 unseen CAMEO (63) proteins drawn over a three-month period IDDT-Ca was 0.866; the maximum value was 0.881 and the minimum was 0.848,
while the median was 0.868. Our final model trained for the full duration on both the PDB and filtered Uniclust30 datasets scores 0.907 on the same validation set.
For more details on both sets of experiments, including specific hyperparameter settings, consult the
OpenFold paper [48
5 Limitations
Many centralized sequence databases are rarely updated, and while we used the most recent versions of each wherever possible, most of the MSAs currently in OpenProteinSet were computed in early 2022. Given that the number and diversity of known sequences is continually increasing, this means that OpenProteinSet--lke any repository of precomputed MSAs mma age over time and need to be updated for optimal downstream performance. OpenProteinSet entries that currently have shallow MSAs or few structural homologs are particularly "vulnerable" in this regard. While we
A3M is plaintext format consisting of aligned sequences, per line (as in Figure and comment lines beginning with '>', Gaps are represented by dashes (+-") and insertions are represented with lowercase residue letters. PRL: https://iiiia..oooooollloooooopp

---

Figure 4: OpenFold trained with OpenProteinSet reproduces AlphaFold2. Superimposed Open- Fold (orange) and AlphaFold2 (blue) predictions on three CASPI5 domains: from left to right, T1109 (RMSD: 0.306), T1153 (RMSD: 0.263), and T1 (RMSD: 0.235).
may periodically expand OpenProteinSet with MSAs, do currently plan to update MSAs already in the dataset as new sequences become available.
We note too that we only evaluate OpenProteinSet popular applications. Nevertheless, the utility of large quantities of MSAs has been firmly established in diverse settings, and we have no reason to believe that OpenProteinSet MSAs in particular will be less useful.
monomeric structure prediction and not other
Discussion
With OpenProteinSet, we have greatly increased the quantity and quality of precomputed MSAs available to the molecular machine learning communities. The dataset has immediate applications to diverse tasks in structural biology. Below, for illustrative purposes, highlight handful of additional tasks and settings where we strongly expect high-quality multiple sequence alignments like those in OpenProteinSet to be immediately useful.
Unsupervised protein language models workhorses in the bioinformatic community, as, analogously to natural language models, they encode useful biological knowledge that allows them to reason about numerous protein protein-related related tasks. Most trained on individual protein sequences, but MSA Transformer, model trained on millions of (unreleased) Uniclust30 MSAs, was able to outperform conventional protein language models on downstream evaluations like protein design, and with fewer parameters T6). With OpenProteinSer,, a dataset of millions of comparable Uniclust30 MSAs, is now possible for the open-source community experiment with similar MSA language models, perhaps even in combination with widely available single-sequence data.
Protein language modeling:
have become
Orphan proteins:
One function of OpenProteinSet is to identify large number of proteins with few or no known homologs the time of its creation. "Orphan" proteins like these are often failure cases of models trained on protein data. In protein structure prediction, for example, MSA- models like AlphaFold2 and TTAFol are known to perform less well on proteins with shallow Protein language models are slightly less sensitive to MSA depth in some cases [22] but the gap persists there as well. We expect that large quantity of additional data on such proteins will be useful to validate and improve bioinformatic methods. Because OpenProteinSet
MSAs 23 24

---

effectively clusters sequence space, it also enables important validation experiments not possible with unclustered sequences alone, like training on one region of protein space and testing on another.
Multimodal deep learning: Beyond bioinformatics, a popular line of deep learning research studies the effects of training extremely large neural networks data from diverse modalities. While the most
commonly studied modality pairing is language and image data [66 67 68
69
70
711. unsupervised
73 and, indeed, raw protein sequence data-has been shown to enrich the knowledge and capabilities of models.
co-training on additional modclities-including audio (72), robotics tasks 71
Multimodal language models jointly trained on English text and biological sequence data have already been used to identify antein-protein interactions (74), classify adverse reactions to drugs [75], and caption molecules (76). The multimodal scientific language model Galactica was also trained on protein sequences [77]. More indirectly. protein data often appears component in benchmarks for multimodal training methods. It has recently been added to DABS, a multimodal benchmark for unsupervised learning techniques (78 77),, and has been used to study multimodal scaling laws in generative models [80] and test the capabilities of pretrained language models across modalities [8]]. As models become increasingly data-hungry. we believe databases like OpenProteinSet will be valuable on both of these fronts, as reservoirs of biological knowledge for generalist multimodal language models and also as tools for the empirical study of multimodal training per se.
Overall, we hope that OpenProteinSet will further democratize research in bioinformatics, machine learning on proteins, and beyond.

---

Acknowledgments and Disclosure of Funding
We would like thank the Flatiron Institute for providing computing resourees and Amazon Web Services for hosting OpenProteinSet. Individually, we would like to thank Milot Mirdita and Martin Steinegger for their valuable support and expertise.
G.A. is supported by Simons Investigator Fellowship. NSF grant DMS-2134157, DARPA grant W91 1011N2001002 0021, and DOE grant DE-SC0022199, N.B. is supported by DARPA PANACEA program grant 1-00011-1-220022 and NCT grant U54-CA225088. M.A. is a member of the Scientific Advisory Boards of Cyrus Biotechnology. Deep Forest Sciences, Nabla Bio, Oracle Therapeutics, and FL2021-002, Foresite Labs company.
References
[1] A. Rivera-Calzada et al. Structure of bacterial type IV secretion core complex at subnanometre resolution. The EMBO Journal (2013), 1195-1204. DOT: 10. 10/8/ eemo../ 2013.
[2] F. Madeira, M. Pearce, A. R. N. Tivey, Basutkar, J. Lee, O. Edbali, N. Madhusoodanan,
A. Kolesnikov, and R. Lopez. Search and sequence analysis tools services from EMBL-EBI in
2022. Nucleic Acids Research 50. W1 (2022), W276 W279. DOI: 10. 1093/ hat///arrgkac240
[3] S. de Oliveira and C. Deane. Co-evolution techniques are reshaping the way we do structural bioinformatics. FI F1000Research 6 (2017), 1224. DO1: 10. 12w88/6682ff http://f000reeeercccco 11543.
[4] A. Riesselman, J. B. Ingraham, and D. S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods 15 (10 2018), 816-822. DOI:
10. 91-1338//15392-1133 018-11592-188-00 -18-0088--------11---1----------
[5] J. Meier, R. Rao, R. Verkuil, Liu, T. Sercu, and A. Rives. Language models enable zero-shot prediction of the effects of mutations protein function. In: Advances in Neural Information Processing Systems. Ed. by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan. Vol. 34. 2021, 29287- 293033. https / //ppcceedngcee/seeeeeeoe neurips oc/paper Paper. pdf [6] R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives. Transformer protein language models are unsupervised structure learners. BioRxiv (2020). DOE: 10. 1101/2000 12. 15. 422761
Files/paper/200 http://pprpp0/0...l..fff/33333/9 /tF.///11..888811022022200000030
[7] M. Rao, Liu, Verkuil, Meier, J. Canny, Abbeel, T. Sercu, and A. Rives. MSA Transformer. In: Proceedings of the 38th International Conference on Machine Learning. Ed. by M. Meila and Zhang. Vol. 139. Proceedings of Machine Learning Research. July mlr httss//v39/rraa// a. html
2021, 8844-8856. htttps://proceedinn
[8] A. Rives al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences 118.15
(2021), e201623911 18. DOT: 10. 1073/ /117//////a///// pnas 2016239118
[9] Z. Lin et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379.6637 (2023), 123 130. DO1: 10. w2t///ceeece/ ade2574! [10] E. C. Alley, G. Khimulya, S. Biswas, M. AlQuraishi, and G. M. Church. Unified rational protein engineering with sequence- based deep representation learning. Nature Methods 16 (12 2019), 1315-1322. DOI: 10. 100.10///444992-- 019 019-0-----100--1-1111
[11] Riesselman, J. B. Ingraham, and D. S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods 15 (10 Nov. 2018), 816-822, DOI:
10. 10-11088/4199--115
[12] J. Frazer, Notin, M. Dias, A. Gomez, K. Min, K. Brock, Y. Gal, and D. Marks, Disease variant prediction with deep generative models of evolutionary data. Nature 599 (7883 Nov.
2021), 91-95. DO1: 10. 1000///115500 0211///4161002----100 021-04043--88
[13] K. Nguyen, X. Guo, and Y. Pan. Phylogeny in Multiple Sequence Alignments. In: Multiple Biological Sequence Alignment. 2016. Chap. 6, 103-112. DOI: https: / org/ 10
[14] H. Ashkenazy, I. Sela, F. Levy Karin, G. Landan, and T. Pupko. Multiple Sequence Alignment Averaging Improves Phylogeny Reconstruction. Systematic Biology 68.1 (June 2018), 117-
130. DOI: 10. 1093/ 1193/sysbbo/syy083

---

[15] A. Hawkins- Hooker, F. Depardieu, S. Baur, G. Couairon, A. Chen, and D. Bikard. Generating functional protein variants with variational autoencoders. PLOS Computational Biology 17.2 (Feb. 2021). 1-23. DOT: 10. 1371/journal pcbi 1008736
[16] D. Sgarbossa, U. Lupo, and A.-F. Bitbol. Generative power of protein language model trained on multiple sequence alignments. eLife 12 (Feb. 2023), e79854. DOI: 10. 7554/eLife eLife. 79854
[17] V. Frappier and A. E. Keating, Data-driven computational protein design. Current Opinion in Structural Biology 69 (2021), 63-69, DO1: 009
1016/j sbi 2021. 03
[18) N. Halabi, O. Rivoire, S. Leibler, and R. Ranganathan. Protein Sectors: Evolutionary Units of Frree-Dimensionl Structure. Cell 138 (4 2009). P774-786. DO1: 10. ce11.200991999 07.038
[19] M. Weigt, R. A. White, H. Szurmant, A. Hoch, and T. Hwa. Identification of direct residue contacts in protein-protein interaction by message passing. Proceedings of the National Academy of Sciences 106.1 (2009), 67-72. DOI: 100.1073/pna 0805923106
[20] Y. Liu, P. Palmedo, Q. Ye, B. Berger, and J. Peng. Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks. Cell Systems 6 (1 2018), 65-74. DOI: 10. 1016/]
cels. 2017. 11.
[21] J. Xu, M. McPartlon, and J. Li. Improved protein structure prediction by deep learning irrespective of co-evolution information. Nature Machine Intelligence 3 (7 2021), 601-609. 10. 10388/ #155///222252-222 02--22255--00100-------
[22] R. Chowdhury et al. Single-sequence protein structure prediction using a language model and deep learning. Nature Biotechnology (2022). DO1: 10. 1038/ s41587 022 01432-w
[23] J. Jumper et al. Highly accurate protein structure prediction with AlphaFold. Nature 577 (7792
2021), 583-589. DO1: 10. 1038/ 021 021-0-19------11--22--
[24] M. Baek al. Accurate prediction of protein structures and interactions using three-track neural network. Science 373.6557 (2021), 87 1-876. D01: 10. 1126/science. abj8744
[25] D. T. Jones, D. W. A. Buchan, D. Cozzetto, and M. Pontil. PSICOV: precise structural contact prediction using sparse inverse covariance estimation on large multiple sequence alignments. Bioinformatics 28.2 (Nov. 2011), 184-190. DOI: 10. 1093/bioiinit ormatics/ hammaaiittttr33e
[26] D.S. Marks, I.. Colwell, R. Sheridan, T. A. Hopf, A. Pagnani, R. Zecchina, and C. Sander. Protein 3D Structure Computed from Evolutionary Sequence Variation. PLOS ONE 6.12
(2011), 1-20. DOI: 10. 1371/journal. pone. 0028766
[27] D. T. Jones. Singh, T. Kosciolek, and S. Tetchner. MetaPSICOV: combining coevolution methods for accurate prediction of contacts and long range hydrogen bonding in proteins. Bioinformatics 31 (7 2015), 999-1006. DOT: [0. wtt.:///o...ooomttc///iiiiiiiiie
[28] V. Golkov, M. J. Skwark, A. Golkov, A. Dosovitskiy, T. Brox, J. Meiler, and D. Cre- mers. Protein contact prediction from amino acid co-evolution using convolutional net- works for crapu-valued images. In: Advances in Neural Information Processing Sys- tems. Ed. by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Vol. 29. 2016. https: : proceedings neurips cc/ paper, cc/paper files paper / 2016 file/ 2cad8fa47bber b
paff
[29] S. Wang, Sun, Li, Zhang, and J. Xu. Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model. PLOS Computational Biology 13.1 (2017), 1-34. DOI:
10. 10.177//journal .cbi. 1005324
[30] J. Ingraham, A. Riesselman, C. Sander, and D. Marks. Learning Protein Structure with a Differentiable Simulator. In: International Conference Learning Representations, 2019. https://openrew/ wttp://oooooodddddyr/re/eee///e
net
[31] M. AlQuraishi. End-to-End Differentiable Learning of Protein Structure. Cell Systems 8.4
(2019), 292-301.c3, DO1: 10. 1016/ cews. 2019. 03.
[32] A. W. Senior et al. Improved protein structure prediction using potentials from deep learning.
Nature 577 (7792 2020). 706-710. DOI: 10. /455//41166666 019 1923-7
[33] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. 2017. DO1: 10. 48550/555XIV 03762
[34] R. Evans et al. Protein complex prediction with AlphaFold Multimer... bioRxiv (2022). DOI:
10. 1111111/// 2021. 10. 04. 463034
10

---

[35] K. Tunyasuvunakool et al. Highly accurate protein structure prediction for the human proteome. Nature 596 (7873 2021). 590-596. DO1: 10. 1010000//880660 021 021-0888--111111-1111
[36] M. Varadi et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research 50.D1 DO1: 10. 1wt10099/naaa nar /1t.109//naa/ggaa110055 gkab1061
(2021),
[37] E. Callaway. 'The entire protein universe': Al predicts shape of nearly every known protein. Nature 608 (4 2022), 15-16. DO1: 10. 1008///11188--02222088------
[38] R. P. Joosten, T. A. te Beek, E. Krieger, M. L. Hekkelman, R. W. W. Hooft, R. Schneider,
C. Sander, and G. Vriend. A series of PDB related databases for everyday needs. Nucleic Acids Research 39 (Database issue 201 1), D41 1-D419. DOI: [0. 10910093//a 2009//arr/gqql0/
[39] S. Ovchinnikov, H. Kamisetty, and D. Baker. Robust and accurate prediction of residue-residuu interactions across protein interfaces using evolutionary information. eLife 3 (2014). Ed. by
B. Roux, e02030. DOI: 10. 7554/ 7554//5554...... el..f.0003000
[40] M. AlQuraishi. ProteinNet: a standardized data set for machine learning of protein structure.
BMC Bioinformatics 20.311 (1 2019). DOI: 10. 1086/1166660 s sa2ss---99-9 019-01999911 -9332-0
[41] wwPDB Consortium. Protein Data Bank: the single global archive for 3D macromolecular
structure data. Nucleic Acids Research 47.D1 (2018),
DO1: 10. 1093 /093/nar/ /nar/ /
gky949
[42] A. L. Mitchell et al. MGnify; the microbiome analysis resource in 2020. Nucleic Acids Research 48.D1 (2020), 3570-D578 DOI: 10. 1003/////////2210335
[43] UniProt Consortium. UniProt: the universal protein knowledgebase in 2021. Nucleic Acids
Research 49.D1 (2021). 1480-D489. DO1: 10. 1003//arr/gaaa1100/00
[44] M. Remmert, A. Biegert, A. Hauser, and J. Söding. HHbbits: lightning- fast iterative protein sequence searching by HMM-HMM alignment. Nature Methods 9 (2 2012), 173-175. DOI:
10. N0.1038//mmeth) 1818
[45] L. Johnson, R. Eddy, and E. Portugaly. Hidden Markov model speed heuristic and iterative HMM search procedure. BMC Bioinformatics 11 (1 2010), 431. DOI: 10. 1111118861444 1tt///441-220555
11-431
[46]
M. Steinegger and J. Söding. Clustering huge protein sequence sets in linear time. Nature Communications 9 2018), 2542. DOI: 10. 011////1177--1-0-9-8888
[47] M. Mirdita, von den Driesch, C. Galiez, M. J. Martin, J. Söding, and M. Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic Acids Research 45.D1 (2017), D170- DO1: 10. ha//3///a//gw1008
[48] G. Ahdritz et al. OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization. BioRxiv (2022). DOI: 10. 11w11111122.... 11. .0
517210
[49] A. S. Lapedes, N. Santa Fe Inst., B. G. Giraud, L. C. Liu, and G. D. Stormo. Correlated mutations in protein sequences: Phylogenetic and structural effects. Lecture Notes Monogr. DOI: [10. (etpi....222222 21722296863
Ser. 33 (1999), 236-256.
[50] D. de Juan, F. Pazos, and A. Valencia. Emerging methods in protein co-evolution. Nature Reviews Geneties 14 (4 2013), 249-261. DOI: 10. 1008/ nrg3414 [51] M. Heinzinger, A. Elnaggar, Y. Wang, C. Dallago, D. Nechaev, F. Matthes, and B. Rost. Modeling aspects of the language of life through transfer-leaming protein sequences. BMC Bioinformatics 20.723 (2019). DO1: 10. 1186/ s 128ss- 0000--09932222-1
[52] J. Devlin, M.-W. Chang. K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2019.
[53] R. Wu et al. High-resolution de novo structure prediction from primary sequence. bioRxiv
(2022). DO1: 10. 11.11202222.2.. 07 21. 500999
[54] A. Elofsson. Progress at protein structure prediction, as seen in CASPI5. Current Opinion
in Structural Biology 80 (2023), 102594. DO1: httpss://oo....../ http:///ooooooo////////oo org/ /10 1ww11011..1.1 sbi 2023
102594
[55] S. Unsal, Atas, M. Albayrak, K. Turhan, A. C. Acar, and T. Dogan, Learning functional properties of proteins with language models. Nature Machine Intelligence 4 (3 2022), 227-245. DO1: 10. s42256 022 00457-9

---

[56] M. Mirdita, K. Schütze, Y. Moriwaki, L. Heo, S. Ovchinnikov, and M. Steinegger. ColabFold: making protein folding accessible all. Nature Methods 19 (6 2022). 679-682. DO1: 10 1038/s4 011///4112222222212
[57] S. Liu al. PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction. 2022.
[58] Z. Li, X. Liu, W. Chen, F. Shen, H. Bi, G. Ke, and L. Zhang. Uni-Fold: An Open-Source Platform for Developing Protein Folding Models beyond AlphaFold. bioRxiv (2022). DOI:
10. 1101/1022. 08. 04 502811
[59] B. E. Suzek, Y. Wang. H. Huang, P. B. McGarvey, C. H. Wu, and Uniprot Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics 31 (6 2013). 926-99262222 9229932. DO1: 10. 1ww1/099/bboo htt//bioinfiiio wwn.oormtisssbeeeeeeoo
[60] M. Steinegger, M. Meier, M. Mirdita, H. Vöhringer. S. J. Haunsberger, and J. Söding. HH- suite3 for fast remote homology detection and deep protein annotation. BMC Bioinformatics 20 2019), 473. DOI: 10. 1186/ 28912-89--99-0011111 3019-7
[61]
Zemla. LGA: a method for finding 3D similarities protein structures. Nucleic Acids Research 31 (13 2003), 3370-3374. DOI: 10.
[62] V. Mariani, M. Biasini, A. Barbato, and T. Schwede. IDDT: a local superposition -frettoooo-eeeeeeee score for comparing protein structures and models using distance difference tests. Bioinformatics 29 (21 2013), 2728.. DOT: 10. wtg.///oinffrmattccc/bbbboaaaaaa
[63] J. Haas, A. Barbato, Behringer, Studer, S. Roth, M. Bertoni, K. Mostaguir, Gumienny, and T. Schwede. Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12. Proteins: Structure, Function, and Bioinformatics 86 (Suppl 201 8), 387-398. DOT: 1et::11.100////1111 1002000000roo 20002//r..22431 25431
[64] Z. Lin et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv (2022).
[65] A. Madani et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology (2023), 1546-1696. DO1: 10. N0..133//44188-0000 -11-2-----1222--
[66] Radford al. Learning Transferable Visual Models From Natural Language Supervision. 2021.
[67] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-Shot Text-to- Image Generation. 2021.
[68]
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical Text- Coxt-Conditioal Image Generation with CLIP Latents. 2022.
[69]
J.-B. Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. 2022.
[70]
P. Wang, A. Yang. R. Men, J. Lin, S. Bai, Z. J. J. Ma, C. Zhou, J. Zhou, and H. Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Lequence-to-Sequence Learn ing Framework. In: Proceedings of the 39th International Conference on Machine Learning. Ed. by K. Chaudhuri, S. Jegelka, L. Song, Szepesvari, G. Niu, and S. Sabato. Vol. 162, Proceedings of Machine Learning Research. 2022, 123340,33,,, https:// mlr. www.ppeesseeee /atess//112//// 20t0///an22al..mmm 2tml
[71]
D. Driess et PaL.ME:: An Embodied Multimodal Language Model. 2023.
[72]
R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. MERLOT Reserve: Neural Script Knowledge Through Vision and Language and Sound. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2022, 16375-16387.
[73] S. Reed et al. Generalist Agent. Transactions on Machine Learning Research (2022). Featured Certification, Outstanding Certification. https:// wwtp://w.eeeeneeen//e/
jij
[74] P. Dutta and S. Saha. Amalgamation of protein sequence, structure and textual information for improving protein-protein interaction identification. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. July 2020, 6396-6407. DOI: 10
18653/v1/200 acl mal1mmmin.550
[75] A. Sakhovskiy and E. Tutubalina. Multimodal model with text and drug embeddings for adverse drug reaction classificattonn. Journal of Biomedical Informatics 135 (2022), 104182. DOI: htwphhtsss//o...or////ee
jbi. 2022. 104182
12

---

[76] C. Edwards, T. Lai, K. Ros, G. Honke, K. Cho, and H. Ji. Translation between Molecules and Natural Language. 2022.
[77] R. Taylor, M. Kardas, G. Cucurull, Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A Large Language Model for Science. 2022.
[78] A. Tamkin, V. Liu, R. Lu, D. Fein, C. Schultz, and N. Goodman. DABS: Domain-Agnostic Benchmark for Self-Supervised Learning. In: Thirty-fifth Conference on Neural Information
Processing Systems Datasets and Benchmarks Track (Round 1). 2021. https://openrreiee
net Set/foruumiid=Uu2myyme
[79] A. Tamkin, G. Banerjee, M. Owda, V. Liu, S. Rammoorthy, and N. Goodman. DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision. In: Thitt Conference Neural Information Processing Systems Datasets and Benchmarks Track. 2022, https
//
/ppenreview wwtp/nnennn//i.. /tt.:/ornnnd----11111111
[80] A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling Laws for Generative Mixed-Modal Language Models. 2023.
[81] K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Pretrained Transformers as Universal Computa- tion Engines, 2021.
13


Reviews:
[{"review_id": "BOgRCASUew", "reviewer": "Reviewer_UMX4", "summary": "", "questions": "", "limitations": "The match to the venue is not completely clear to me in the sense that this may only be relevant to those in the protein folding community and not ML at large. The authors do not discuss how the ML community can make use of this data apart from training protein structure prediction models. In its current form the paper reads more like it would be most relevant to the bioinformatics community. Perhaps the authors can discuss in more depth how the MSAs can be used in other ML tasks and how to incorporate them into learning models.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* It is quite apparent that this will be a very useful resource for the protein folding community and the bioinformatics community at large.\n* Obtaining this resource at such a scale is computationally inaccessible to many researchers, adding to its value.\n* The paper is well-written and technically sound.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Dyer1mv1Ap", "reviewer": "Reviewer_GyWH", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper introduces a large set of protein MSAs, which could have a considerable impact on the community. \n- The generation of MSAs is computationally costly and the authors used a sophisticated protocol similar to the AlphaFold2 procedure.\n- The structure information linked to each MSA is a very valuable addition.\n- Generally, the paper describes high-quality research.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "u9qtqJTCGT", "reviewer": "Reviewer_S8Br", "summary": "", "questions": "", "limitations": "The authors point out that MSAs can become outdated in the sense that with growing sequence DBs MSAs can become deeper or individual sequences might change. Is there any plan for OpenProteinSet to be a resource that gets updates, or is it a one-off static dataset? Both are valid, just would be good to know.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Very large scale\n- Convincing use case", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dtbLBVdqyQ", "reviewer": "Reviewer_yCQz", "summary": "", "questions": "", "limitations": "The authors discuss how multiple sequence alignment datasets may soon become out-of-date due to the rapidly-growing nature of protein sequence repositories. They also discuss how they only evaluate the effectiveness of their proposed multiple sequence alignment dataset through monomeric protein structure prediction. However, it remains to be shown how chain-paired multiple sequence alignments can specifically benefit multimeric protein structure prediction and related tasks. As such, it is currently unclear whether the authors' proposed multiple sequence alignments will prove useful outside of the task of monomeric structure prediction.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The collection of multiple sequence alignments that authors have curated appears to be soundly constructed, and their computational experiments in retraining OpenFold seem to confirm this point.\n2. Seemless integration of these alignments with OpenFold is a useful feature for future users of this collection of alignments.\n3. The authors' analysis of their collected protein sequence data, with further discussion in the manuscript, may be useful to users of this alignment dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents OpenProteinSet a database of pre-computed MSAs (aligned proteins) that is critical for numerous protein ML tasks. 
The paper does a great job at explaining the value of pre-computed MSA databases and introduces the largest public MSA protein dataset, which was already successfully used to train OpenFold. The authors also did a great job at addressing the reviewer concerns.
Decision: Accept (Poster)