MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions
Authors: Le Zhang
Jiayang Chen
Tao Shen
Yu Li
Siqi Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Protein Language Model
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We introduce MSA-Generator, a self-supervised model that generates virtual MSAs, enhancing protein structure predictions in key benchmarks.

Abstract: Deep learning models like AlphaFold2 have revolutionized protein structure prediction, achieving unprecedented accuracy. However, the dependence on robust multiple sequence alignments (MSAs) continues to pose a challenge, especially for proteins that lack a wealth of homologous sequences. To overcome this limitation, we introduce MSA-Generator, a self-supervised generative protein language model. Trained on a sequence-to-sequence task using an automatically constructed dataset, MSA-Generator employs protein-specific attention mechanisms to harness large-scale protein databases, generating virtual MSAs that enrich existing ones and boost prediction accuracy. Our experiments on CASP14 and CASP15 benchmarks reveal significant improvements in LDDT scores, particularly for complex and challenging sequences, enhancing the performance of both AlphaFold2 and RoseTTAFold. The code is released at \url{https://github.com/lezhang7/MSAGen}.

MSA Generation with Segs2Seqs Pretraining: Advancing Protein Structure Predictions
Le Zhang 1.3 Jiayang Chen Tao Shen Yu List, Siqi Sun
Fudan University Shanghai Artificial Intelligence Laboratory Mila, Université de Montréal
1 The Chinese University of Hong Kong Zelixir Biotech le. zhang@mi quebbe,, liyu@cse cuhk.edu. edu. Siqisun@fudan.edu..
Abstract
Deep learning models like AlphaFold2 (Jumper et al. 2021) have revolutionized protein structure prediction, achieving unprecedented accuracy. However, the dependence on robust multiple sequence alignments (MSAs) continues to pose challenge, especially for proteins that lack wealth of homologous sequences. To overcome this limitation, we introduce MSA-Gennrator, a self- self-supervised gen- erative protein language model. Trained on a peuueecceto-sqquene task using automatically constructed dataset, MSA-Generator employs protein-specific attention mechanisms to harness large-scale protein databases, generating vir- tual MSAs that enrich existing ones and boost prediction accuracy. Our experi- ments on CASP14 and CASP15 benchmarks reveal significant improvements LDDT scores, particularly for complex and challenging sequences, enhancing the performance of both AlphaFold2 and Rose Toseo..Fold. The code is released at https:///githu wtttshuu.cmmmmeeeaeeeeeee wwmmIeezangg///AAGG.
Introduction
The challenge of protein structure prediction (PSP), a central issue in structural biology, has under- gone remarkable transformation thanks to advances in deep learning. Among these developments, AlphaFold2 (AF2) stands out for its exceptional performance, primarily attributed to its effective use of multiple sequence alignments (MSAs) (Jumper et al. 2021). MSAs constructed by querying a protein sequence against extensive databases using sophisticated search algorithms, resulting in collections of homologous sequences that encapsulate evolutionary information. This information serves as the cornerstone for many PSP models. However, not all protein sequences have a wealth of homologous counterparts. In such cases, the most advanced search algorithms may struggle construct high-quality MSAs, thereby limiting the performance of MSA-dependent models like AF2 (Jumper et al. 2021 Wang et al. 2022a), illustrated in fig.
Inspired by the generative capabilities language models (Rafeel el al. 2020 Touvron et al. 2023 Chung al. 2022, Chen al. 2021), recognize their potential extend beyond textual data. Viewing protein sequences through this lens, we propose an innovative approach to generating virtual yet constructive MSAs, akin generating text. These novel alignments provide supplemental evolutionary information, thereby enhancing the efficacy of protein structure predictions. Within the realm of PSP. tertiary structure prediction remains critical challenge, occupying central role in molecular biology by revealing intricate protein functions and interactions. While secondary structure predictions (Rost& & Sander 1993) offer valuable insights, is the tertiary predictions that provide comprehensive understanding of protein's 's complex conformation.
WWrkss done during internship at Shanghai Artificial Intelligence Laboratory, co-first authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Search-base MSA
Generative- MSA
Highqpaaity Viriaa MSA
Query Proten Sequerce ETYEL L D 9
KTYDL D...
Low-quaiity MSA
MSA Generetor
LeaClase
D
KSYHLAD KTYDLSD.
BTD
CTV
low MMA
Inferie Strustara Fradeteer
Prediction Algorithm
Ground Truth
AlphaFold2 Prediction Conanched MMSe]]
Improved Prediction Results WoaldWorrd Challinge
AlphaFold2 Prediction (Senerated MSA)
(L001 41.3
1007
Figure 1: (Top Left) Certain protein sequences lack rich homologs, leading to poor MSA quality with conven- tional search algorithms. (Top Right) We propose generative model MSA-GGenrator to produce informative MSA for these queries, offer a potential solution to such challenge.
For bio-tasks such as protein structure prediction, gene sequence alignment, RNA secondary structure determination, microbial community analysis, and evolutionary construction, where enhancing downstream performance necessitates multiple sequences, we introduce the eqquences-to-sequeneee (seqs2seqs) generation task. Unlike the conventional soguence-to-sqquence (seq2seq) tasks aasks-e.g.. machine translation, which requires a strict one-to-one correspondence between source sequence x and a target sequence y g-seqs2seqs is designed for flexibility. The task aims to generate multiple coherent sequences from given sequences. Each generated sequence preserves patterns from the input, but there's no strict correspondence between the input and output. Instead, we prioritize maintaining interconnected patterns across them. This design endows the task with self-supervised nature due to its intrinsic adaptability.
In the context of protein, such flexibility enables easy extraction of a portion of the MSA as the source, with the remainder acting the target. Harnessing search algorithms, our framework adeptly extracts source and target data from comprehensive protein databases, paving the way for self-supervised pre-training. To our knowledge, this marks initial step in tapping into self-supervised generative pretraining to bolster protein structure prediction accuracy.
We introduce MSA-Generator, protein language model pre-trained using the seqs2seqs its pretext task. Specialized simultaneously generating multiple sequences, effectively captures global structural information from input MSAs. This approach facilitates rapid, de novo sequence creation, improves inferior MSAs (see fig. [ and boasts adaptability across various protein domains, adeptly navigating computational challenges.
To summarize the main contribution of this article:
Innovative unsupervised Seqs2Seqs Task Proposition: We propose the unsupervised sequences-to-sequeene (seqs2seqs) task, promising approach for generating informative protein sequences, with potential applications extending to other areas like RNA. Integrated with search algorithms, this task streamlines generative pre-training by automating data retrieval from expansive protein databases.
Launch MSA-Generator Model: MSA-Generator, our state-of-the-e generative protein model, is uniquely devised to employ the seqs2seqs task on self-curated dataset. It is optimized for multi-sequence generation, skillfully extracting global MSA insights, and has demonstrated adaptability across diverse protein domains.
Robust Empirical Validation: We validate our approach's 's potency, showcasing significant improvements on AlphaFold2 and RoseTTAF using CASP14 and CASP15 benchmarks. This signifies a practical leap forward in tackling the intricate challenge of protein folding. It also showcases the promise of seqs2seqs pretraining in the field of bioinformatics.

---

2
Related Work
Protein Structure Prediction Proteins, despite their immense diversity, are composed of just 20 unique amino acids. Their physical structures, which dictate their functions and characteristies, are fundamental to understanding the essence of life. While the field has witnessed significant advancements like AF2 (Jumper et al. (021) and RoseTd (Back et al. (021), challenges remain. The success of AF2 can be ascribed to its adept utilization of MSAs, constructed by search algorithms such as DSepMSA (Zhang et al. 2019), JMcRH (Johnson et 2010) and MMseqs2 (Steinegger & Söding 2017) across vast databases including Unikef (Suzek et al. (007) and BFD based on protein query sequence. Conversely, single-sequent sequence prediction methodologies (Chowdhury et al.) 2021 Lin et al. [022]] Chowdhury al., 2022 Wu et al., 2022b) often underperform in comparison. However, for protein queries devoid of extensive family representations, obtaining quality MSAs is challenging. Consequently, the proficiency of MSA-driven techniques diminishes. In this context, our proposed method leverages generative techniques to combat the paucity of homologs in protein sequences. presenting a solution when traditional techniques falter.
Protein Language Models Language models, initially designed for language processing. have found a expanding role in bioinformatics, primarily for protein sequence representation. This improvement in performance largely stems from the adaptation of the masked language modeling (MI.) strategy, a concept inspired by BERT (Devlin et al. 2019). The ProtTrans series. which includes models such as ProtBert, PI, ProtXLNet, and ProtT's (FInaggar et al., 2021 along with ProteinBERT (Brandes et al., 2022) and ESM models like ESM-1b (Rives et al. 20211) and ESM-2 (Lin et al., 2022), exemplifies the transformative role of Transformer architectures in this field. Further reinforcing the connection between language models and protein structure prediction, research has demonstrated associations between protein representations learned by these models and contact maps, revealing evolutionary patterns that are crucial to the success of AlphaFold2 (AF2) (Vig iet al. 2020] Rao et al. 2020). These findings have shifted research focus back towards leveraging multiple sequence alignments (MSAs) rather than relying solely single sequences. A notable example is the MSA Transformer (Rao et al., [202T), which applics the MLM strategy specifically to MSAs, thereby capturing rich evolutionary information and advancing the field of protein structure prediction.
Protein Sequence Generation Beyond masked language modeling (MLM), a variety of generative techniques exist, each with its distinct objectives and methodologies. For instance, Potts models (Figliuzzi et al. 2018 Russ et al. (2200) are crafted specifically for individual MSA sets from which they're re derived (Zhang et al. 2022). However, their shortcomings in adapting to different MSA sets (Sgarbossa et al. | 2022) have spurred the development of generative language models. An exemplar, ESMPair (Chen et al. 2022), constructs MSAs Interologs by classifying sequences based on their taxonomic lineage. In contrast, both ProGen (Madani et al. (2200) and ProGen2 (Nijkamp et al. b focus on single-sequence generation, sidestepping the integral component of MSA. Another research direction involves VAE-based models (Riesselman et al. 2018 McGee et al. 2021| Sinai ct al. 2017). originally developed for mutation evaluation. The challenge of efficiently sampling from distributions create diverse and long sequences limits their application to downstream tasks. A study by Sgarbossa et al. (2022) utilized the MSA Transformer in a repetitive mask-and-reveall methodology, which unfortunately led compromise in sequence diversity. Of notable mention is EvoGen (Zhang et al., 2022), aiming parallelly at producing virtual MSAs for Protein Structure Prediction. However, EvoGen uniquely operates as a meta-generative model, requiring guidance from Alphafold2 to hone its MSA generation prowess.
Despite the lengthy context associated with MSA Generation, work connects self-supervised learning with MSA generation. We highlight generative MSA pretraining, introducing an unsuper- vised sesuuences-to-sqqueeee task specifically tailored for efficient MSA generation. To the best of our understanding, this marks a significant stride in the realm of protein sequence generation.
3 Sequences-o-Sequeeees Generative Pretraining
We present the saauennce-tt-Sequenee generation task and the methodology for automatic dataset construction in section 3.1 Details of the proposed MSA-Generator provided in section 3.2
3

---

In section 3.3 we delve into the ensemble approach of MSA- Generator for optimizing the Protein
Structure Prediction (PSP) task.
3.1 Sequencee-to-Seqquenes Generation For Protein Sequences
Addressing the challenge sparse homologous matches in Multiple Sequence Alignments (MSA) within real-world databases, we introduce the Segueencestt-Sequeeeee for creating virtual MSAs. This approach differs from the conventional sequence-to- to-suueece frameworks used in machine translation, which typically enforce a strict one-to-one correspondence between source sequence r and target sequence y. Instead, seqs2seqs allows for more flexibility, focusing identifying shared intrinsic patterns between source sequences X and target sequences Y. For protein MSAs, this involves integrating extensive evolutionary data, both across and within the sequences, to reveal co-evolutionary relationships.
(seqs2seqs) Task, designed
The inherent adaptability of the seqs2seqs model permits the self-supervised nature of the task and seamless gathering of substantial quanti- ties of source and target sequences from pro- tein sequence databases. This is achieved by deploying sequence searching algorithms like JacKHMMER (Johnson al., 2010) and MM- seqs2 (Steinegger & Söding 2017). Our pro- cess began with selecting sequences from the UniRef90 database (Suzek et al. 2007) initial queries. Subsequently, the JackHMMER algo- rithm (Johnson al. 2010) was employed itera- tively to identify homologous sequences within the database, based the query sequences fol- lowing MSA dataset construction pipeline of AlphaFold2. This process iterated until additional sequences emerged, searching parameters are detailed in appendix C sequences retrieved, random selection made, designating query with some the source X and the remainder as the target Y, as illustrated in fig-2] Notably, the assurance of co-evolutionary relationships is intrinsically facilitated by the search algorithm's 's mechanism.
Sequence-to-Sequence
Seoree
stret Sea 29-ona
Sereet Benjoerr COMMOLL
cormagorcerce
Helle, how are you?"
Sequences-to-Sequencce
SYULND-6
malitinary Information
Taget
L
Figure Difference between seq2seq seqs2seqs and Automated Data Collection Process.
For every batch of
3.2 Sequences-oo-Sequeness Architecture
MSA generator
Encoder Hidden States
Non
Nom
Nom
Oroen Coberr Masita
Moon
Lio
Orginal MSA (Low-quality
Aet
Augmented
Depthwwise Average
Encoder
Decoder
Comprehensive States (L*H)
Attention
Cross-Rav
Quary
Hidden
Key & Value
Compreheneive States
States
(L*H)
(D*L*H)
Figure 3: MSA--Gnnratoo Overview (Top) Overview of the architecture, processing pipeline, and module attention operations. (Bottom) Illustration of the attention mechanism. A red star represents single query position, and the red boxes indicate keys and values utilized in attention processing and calculations.

---

We pretrain a transformer based model (Vaswani et al. 2017). denoted as ASS-Gener,,or, via the unsupervised Sequences-tor to-nnncesttt-Seeeeessss task. The Generatrr framework incorporates an encoder- decoder structure. The encoder contextualizes the input MSA data, while auto-aggressive decoder produces sequences derived from this context (refer section 3). capture expansive evolutionary information from the input MSA both horizontally and vertically, the encoder integrates the tied-row and column attention mechanism (Rao et al. 2021]. As the decoder concurrently generates multiple sequences- interacctn with each other and the input MSA-it is enhanced with two additional modules beyond the conventional transformer. The Cruss-Row Attention designed to efficiently acquire global representation by amalgamating comprehensive states. Meanwhile, to emphasize the vital conservative trait of amino acids (Dayhoff et al. 1978 Henikoff & Henikoff 1992 Jones et al., 1992), introduce the Pooss-Colunn Attention, which, during the generation of the token at time step 1, directs its attention to the 1-th token of all input sequences.
Tied-Row Attention Building upon the foundation laid by the MSA Transformer (Rao al. 2021] we incorporate a shared-attention mechanism. This is achieved by aggregating the attention map weights across each sequence from MSA RDxL prior to applying the softmax function. Notably, each sequence utilizes the same attention weight matrix. For the d-th row, the associated query. key, and value matrices are denoted as Qd. Kd, and Va e Rixh, respectively. These matrices are derived via three distinct learned projection matrices. The computation of the shared attention weight matrix formulated as:
D Qakk Σ x(Dh))
WTR softmax
RRAX
In this context, A(D,4) vDh serves as the square-root normalization. This normalization is crucial in mitigating potential linear scaling of attention weights with the sequences. The resultant representation for the d-th row is obtained through WTRVa.
It's important to highlight that, in our decoder, we deliberately bypass the tied-row attention. This decision aids in maintaining diversity in the generated sequences. Instead, we lean towards a conventional self-attention mechanism.
Cross-Row Attention Contrary to tasks like machine translation, where the target attends only to a single input during decoding, the essence of seqs2seqs lies in discerning intrisic patterns common to both source and target sequences. This necessitates a holistic comprehension of the input, implying that when generating a sequence, the decoder should attend to the entirety of the input. A naive concatenation would yield a representation with dimensions DDLL rendering it computationally expensive and thus impractical.
To address this, we introduce an efficient strategy that calculates the depth-wist wise average pooling of encoder hidden states Hene, represented as H. 1o2 Σ Hd eng E R This serves as global representation of the input and is crucial for cross-attention during decoding. Here, Ke H and V. = H.W W, signify the key and value matrices, while Q = Xaee W, stands for query matrix projected from decoder hidden states X The Cross-Row attention is:
QKT Vo.
CR-AutnQ Ke, V.) softmax
Each sequence generation process can access comprehensive information, attending to the same keys and values, mirroring the co-evolutionary patterns of the input MSA simultaneously thus permitting fast parallel generation of multiple sequences (middle at bottom in fig. 3).
Self/Cross -Column Attention In MSA, each column represents residues or nucleotides specific position across sequences, revealing conserved regions essential for understanding biological functions, structural stability, or evolutionary significance (Dayhoff et al. 1978 Jones et al.. 1992.
Henikoff & Henikoff 1992). Drawing inspiration from the -irretilni attention proposed in Ho et al. (2019), we introduce a self-column attention encoder for a comprehensive representation akin to Rao et al. (2021) and cross-column attention in decoder capture conservation characteristics.
5

---

To facilitate both attention mechanisms, the representation matrix X E I Dxxxx transposed prior the execution of self and cross attention:
needs to be
(qaa K cot
ColAtt(Qcot Keot Veot) -(sottmax
Veol
(3)
vh
For the self-column attention, projections from X yield Qeot: Kools Voot E BLxxxx In contrast, for the cross-column attention, Qeol is determined from decoder hidden states as X dec W whereas Koot and Veot are projected from encoder hidden states as H enc (see fig. 3 3ootto right).
Pre-training objective We employ the seqs2seqs task to pretrain MSA-Generator. For given source MSA X E R the loss is computed with respect to the target MSA Y E D'x as follows:
D' L
Leeqe2oegs
log P(vi X)
(4)
D' x I
1=0
It's crucial to note that each sequence E Y is generated referencing the entire source matrix X, and this generation occurs parallel owing the thoughtful design of the architecture.
Pretrained MSA-Generator adopts 12 transformer encoders/decooders with 260M parameters, 768 embedding size, and 12 heads. It's pretrained with ADAM-W at 5e rate, 0.01 linear warm-up, and square root decay for 200k steps on A100 GPUs, batch size of 64, using a dataset containing 2M MSAs constructed as described in section
3.3 Generation and Ensemble Strategy
During inference, for every query sequence x, we initially use a search algorithm to assemble a MSA denoted as X. This is subsequently inputted into MSA-Gennrator to produce MSA Y. The concatenated MSA X + Y serves as input for the subsequent task. Nucleus sampling (Holtzman al. 2019)) set with top-p=50 and top-k= 10, is implemented foster unique sequences and curtail redundancy.
For the purpose of optimizing the PSP task and yielding informative sequences, adopt PLDDT as our selection criterion, leveraging our ability for swift MSA generation. PLDDT (Kryshtafovych let al. 2019 Jumper et al. 202I) measures the accuracy of predicted inter-residue distances for each residue in protein, serves as a confidence indicator, with elevated scores hinting at potentially more accurate predictions. Utilizing pLDDT, we enhance each MSA through multiple runs, computing corresponding pLDDT scores for each. The MSA with the premier PLDDT score subsequently selected the optimal ensemble result and employed to determine the prediction accuracy relative to the ground truth.
4 Empirical Validation
4.1 Setup
The tertiary structure of protein is pivotal, directly determining its functionality. In structural biology, the tertiary structure not only reveals the overall conformation a protein but also inherently includes insights from secondary structures (Rao et al. 2021 Jones 1999), such the arrangement and orientation of a-helices and fi-shetts, and from contact predictions (Wang et al. (017). denoting the spatial interactions between amino acid pairs. In essence, the tertiary structure offers holistic view, allowing direct inference of localized structural features and interactions between amino acids. Leveraging tools like AlphaFold2 lets directly obtain this comprehensive structural data, thereby bypassing intermediary steps like secondary structure and contact prediction. Given the centrality of tertiary structure prediction in protein functional studies, we have prioritized this task and adopted Local Distance Difference Tests 2 (L.DT) (Mariani et al. 2013) and PLDDT as our metric for evaluating prediction accuracy. We also adopt Template modeling score (TM-Score) and Global distance test (GDT-TS) to measure global structure prediction accuracy. Specifically, we assess
OOeenStructure is used for LDDT calculation

---

MSA- -Ss--Genrrtoor by comparing preotein tertiary structures predicted from PSP algorithm, namely AlphaFold2 and RoseTTAFold, with various input MSAs. Demonstrating the usefulness of generated MSAs and the efficacy of MSA-Gennrator.
Benchmark & Dataset We employ CASP14/15 as our test set, a prestigious dataset that encom- passes proteins from a broad spectrum of biological families. The creation of a vast protein structure prediction dataset is prohibitively expensive, and given that AF2 has already trained on all previously available structures, this dataset emerges as the best evaluation benchmark. It's important to highlight that sequences from CASP14/15 aren't part of our pretraining dataset, and our evaluations precede the AF2 version updated with CASP14/15 information.
Our primary interest lies in challenging protein sequences devoid of homologues, rendering traditional search algorithms ineffective. For every query in our test dataset, we use JackHMMER to search within UniRef90, which contains 70 million sequences, in order to gather related homologues. We define two scenarios: (1) artificial challenging MSAs where we purposefully pick the top 15 homologues for each test set query as an artificial gold standard. From these, homologues are further sampled as the artificial baseline, offering a synthetic challenge. (2) real-world challenging MSAs, which includes 20 sequences from test set, each with homologues less than 20, significantly challenge PSP algorithms. All assessments are executed in zero-shot setting.
4.2 Are Virtual MSAs as good as Real MSAs ?
Depth:
15
15
b
Baseline
Virtual
Distribution of LDDT improvements
Mean:
68.5
53.5
68.3
100
12.3
100
LDDT (Baselins or Virtual)
80
22.30
30.9
(9 (.24)
LDDT
60
1100
001
40-
cae Gisaliza
33.4
20
20
40
60
80
100
Baseline Virtual
LDDT (Real)
Figure 4: Artificial Challenging Cases Results from AlphaFold2 (a) Violin plots of LDDT distribution. (b) x-axis represents LDDT of artificial gold standard, and the y-axis represents LDDT of arrificial baseline and artificial augmentation. Dashed-ling represents 95% confidence intervals (c) Pic chart of LDDT improvements in intervals. The inner circle represents comparison with the baseline, while the outer circle represents comparison with the real.
We employ Artificial challenging MSAs to thoroughly compare our generated virtual MSAs with conventional searched real MSAs to demonstrate that virtual MSAs can closely approximate real ones in downstream tasks. Specifically, for every Baseline MSA, we deploy
to produce Virtual MSA that poses the same depth of the Real MSA. For each MSA, we employ an ensemble of three runs using the strategy outlined in section 3.3 Refer to fig. 4 for the results.
The fig. (a) shows that the LDDT distribution of the baseline suffers sharp decline when reduced from 15 to 5 sequences. This underscores the importance of MSA quality for cutting-edge PSP algorithms. Yet, when supplemented by MSA-Generatorf generative virtual MSA, the gap to the Real narrows considerably, reflecting LDDT enhancement of 12.8. This underscores the effectiveness of our generated MSAS.
A more granular observation in fig. (b) illustrates that the majority of baseline data points are below the diagonal, while most Virtual points sit above some even considerably outpacing the Real. fig, (c) illustrates the statistics on improvements in intervals. It's evident that our generative virtual MSAs effectively improve results for 72.8% of protein sequences (over Baseline). Remarkably, nearly half of the generated virtual MSAs even outperform the real searched MSAs. This emphasizes the importance of generative MSAs and the potential of the seqs2seqs task in uncovering co-evolutionary patterns within bio-sequences. Among them, there are even 6 generative virtual MSA that surpass real MSA by more than 30 LDDT, including most notable T1032-DI (+46.02 L.DTT), T1054-D1 (+46.8 LDDT), and T1070-D2 (+61.22 LDDT), suggesting that without generative virtual MSA.
7

---

current PSA algorithm may fail on these queries. Comprehensive results for individual MSAs can be found in the Appendix.
4.3
Real- Real-World World MSA Challenges
CASPI4 Det. =
CASPIS (avg. Det. =.4)
PSP Algorithm
PLDDT1
LDDTI
Score
GDETS 1 PLDDII
LDDTT
TM-Se
GDT-TS
single-eeneccce-bbsee
ESMFold
43.3
41.9
0.56
0.47
46.0
53.4
0.47
0.41
OmegafFold
43.1
0.51
0.44
49.6
0.44
0.39
MSA baved
RoseTAFFold
63.5
51.3
0.56
0.51
62.6
52.1
0.53
0.46
Ros:TTAFood-Pots Generation
63.2
48.9
0.50
94,4
51.0
0.52
Rose Ros:TTAFold- -erood- Unmasking
63.9
52.2
0.57
0.53
65.3
55.3
0,55
0.48
RoseTTAR TTAFood-MMA Generator
68.9....
56.3,50
0.61 5%
69.0.
58.4.68
195.
435
Fllfa
65.1
53.2
0.59
0.54
65.1
55.6
0.55
0.49
AlptsdFold2-Pots Generation
63.8
0.60
0.55
0,56
ApphaFol2--Iteate Unmasking
65.2
54.6
0.61
0.57
69.5
57.3
0.57
0.51
AlphaFold2+MSA Generator
71.6 10.4
57.5
0.65
0.00
73.7
63.7
0.61 4456
0.53,.
Table I: Real-World MSA Challenges average pLDDT, LDDT and RMSD enhancement scores, averaged over
3 runs; avg. Det. represents average depth of MSA.
Our ultimate goal is produce high-quality MSAs for protein sequences with few homologues. Current search algorithms often fail to construct quality MSAs for these. making PSP algorithms similarly struggle with accurate predictions. The Real-world challenging MSAs evaluation is devised to test the efficacy of the seqs2seqs generative pretraining approach in addressing this challenge. For this, we curate sequences with fewer than 20 homologues using the search method detailed in section 4.1 For every identified MSA, we employ depth across three independent runs. Subsequently, we measure the ensemble LDDT by inputting them to the PSP algorithm following section 3.3 For comparison, our benchmarks include the strong single-sequence folding technique, ESMFold (Lin al. 2022) and OmegaFold (Wu et al. 2022a): we apply the same evaluation set up for the iterative unmasking strategy highlighted in (Sgarbossa ctt al. 2022); and generation with Potts models (Figliuzzi et al. 2018).
to generate an MSA of identical
Table I presents the PLDDT, LDDT, TM-Score and GDT-TS improvements achieved through various MSA generation techniques across different models. The wongll-sequnnce-eee
models lags behind MSA-based strategies. AF2 consistently outperforms Rose' RoseTTAFon TTAFold in both metrics. Potts models, intriguingly, don't demonstrate a pronounced ability to produce effective MSAs. This is evidenced by their marginally reduced average performance in both metrics, echoing findings from (Sgarbossa) al. 2022; Rao et al., 2020). While iterative unmasking with MSA Transformer (Sgarbossa et al., 2022) can generate usable MSAs in certain scenarios, the MSAs it produces are often less diverse due to the inherent unmasking process, thus limiting its enhancement potential.
In contrast, the method we propose demonstrates sigmificant enhancements across all metrics in both models. This suggests that the MSAs produced by our approach are predominantly informative. In particular, 75% of the MSAs experienced substantial improvement, leading to an average increase LDDT scores of 4.3 for CASP14 and 8.1 for CASP15. Our observations reveal a consistent 6% improvement in both TM-Score and GDT-TS for CASP14, alongside a 6% increase in TM-Score and a 4% rise in GDT-TS for CASP15. These results underscore our method's efficacy in enhancing both local and global structure predictions.
Remarkable LDDT enhancements were observed in T1093-D1 (with homologous), moving from 45.5 to
Orphan25
PLDDT
LDDTT
TM-SScore
GDT-TS
AlphaFold2
77.2
61.6
0.61
0.62
70.77, and in T1113 (with 10 homolo- gous), rising from 32.6 to 80.6. How- ever, in specific MSAs, like T1094- D2 (7 homologous. PLDDT +2.8, LDDT -0.1), T1099-D1 (8 homolo- gous, OLDDT+1.11 LDDT -0.5), and T1178 (15 homologous, pLDDT +2.2,
AlphaFold2+Pots Generation
68.9
49.3
0.49
0.43
AlphaFold22 Unmasking
78.9
62.5
0.64
0.63
AphhaPod2+MMS--oone
81.8
66.4
0.69
0.67
Table 2: Comparison of AlphaFold2 variants the Orphan25 dataset.
LDDT -12.3), an elevation in PLDDDT was offset by a decline in LDDT. This suggests that pL.DDT might not always be consistent indicator for selection strategy outlined in section 3.3 Detailed
8

---

results for each individual MSA are available in the appendix A more challenging aspect of our study involves enhancing the prediction of structures for orphan protein sequences. To evaluate our method in this context, we included results from an orphan protein family, Orphan25 Wang et al. (20226). We employed MMseqs2 to search against the UniRef30 and ColabFoldDB databases Mirdita et al. (022), with ColabFoldDB being extension of BFD/MGmify, enriched with metagenomic sequences from diverse environments. By selecting sequences without homologues, we identified test set comprising 10 proteins (6WKY, 6WL0, 6XA1, 6XN9, 6XYI, 7A5P, 7AL0, 7JJV). The outcomes, summarized in table 2 indicate that our approach demonstrates substantial benefits for orphan protein sequences. This suggests that our method is particularly advantageous for challenging inputs.
4.4
Re-Evaluating PLDDT Selection Metric
T100S OIICEP
50
Selection strategy
Searched Roal
Best-LDDT Best-pLDDT
T1110 (CASP 15)
T1122 (CASP18)
LDDT Improvement
30
Tross-D1
20
(CABP14)
(CABP14)
Generative Vimual
Loo JADEHeS
10
0-
5
10
15
20
-10
Depth
T1175 (CASP15)
-20
Greun Toah
Predccor with Searthed Tae MSA Aot Predico Genaraiiv ISA
Figure 5: (Left) LDDT improvement selected by different criteria; (Right) Protein structure visualization with PLDDT and LDDT selected by Best-pl DDDT.PLDDT.
We sought to examine the effecttiveness of PLDDT as a criterion. As previously highlighted, for specific proteins, improvements PLDDT do not necessarily correlate with increases in LDDT. To delve deeper, we conducted an experiment where LDDT was directly calculated for each enhanced MSA in section 4.3 We then selected the highest LDDT as the output, bypassing the of PLDDT as an intermediary metric.
Erowstc Bara
Augemanta
Papior
The disparity between pLDDT-based and LDDT- based predictions shown fig. (g.55)sm suggests that PLDDT may not always be the best criterion. noticeable gap exists between the two crite- ria (evident in the blue and red regions). For
(a) (Left) Results of Ensemble Runs (Right) Results of Augmentation Factor
example, proteins T1064-D1 (depth=9) and T1122 (depth=1) exhibit significant gaps be- tween scores chosen by LDDT versus pLDDT,
Avorage Shannor Entreay scross MSAs
Distibutioon Corretation Coetfic ents
highlighted in red boxes. Interestingly, for T1178 (depth=15), the highest PLDDT selec- tion scores -12.3 against the baseline, while LDDT selection results in a +2.7. This implies that some generated MSAs, even with lower PLDDT scores, can enhance the LDDT. This
3.00
Average Sharnon Ennopy
Corrlation Coefiicient
indicates that our approach has untapped po- tential that could benefit from more nuanced selection criteria. The ideal situation would be for both predictions to produce the same signif- icant improvement, as emphasized by the green boxes and visualized in fig. (Right) Notably, MSA--Generato shows notable improvements for protein sequences with few homologs, em- phasizing its utility real-world protein folding challenges.
Ours
Iterative Unmask
Inerative_Unmaskk
Comiltiin Confiicints
(b) (Left) Box Plot of Averaged Shannon Entropy (Right) Violin Plot of Pearson correlation coefficient between Searched Real PSSM and Generated Virtual PSSM
Figure 6: Ablation and MSA feature.

---

4.5 MSA Diversity & Conservation
We directly evaluated the generated MSAs based on two key characteristics: diversity and con- servation. All experiments setup are consist with section 4.3 To measure diversity, we analyzed the average Shannon Entropy across MSA columns. Compared
the Iterative Unmask method (Sgarbossa et al., 2022), our technique, as illustrated in fig. 6bb (left), consistently yields higher Entropy, suggesting increased diversity. For conservation, we examined the Position- Speecff-
Scoring Matrix (PSSM) (Altschul et al., 1997) of both original and MSA-Cienerato- MesA-eeenerattorgeeeraaaeeddd MSAs. The PSSM gauges conservation of specific amino acids. The Pearson correlation coefficient between the searched real and generated virtual PSSMs, shown in fig. (right, highlights the retention of amino acid conservation in generative virtual MSAs (see fig. for visualization of conversation). The outcomes highlight MSA-Generator s effectiveness as an MSA generator and demonstrate the broader capabilities of the seqs2seqs framework.
4.6 Ablation Study
Following section setup, we investigated the impact of Ensemble Runs and Sequence Augmenta- tion Factor on PSP outcomes, shown in fig. [6a While additional ensemble runs lead to improved PLDDT scores, gains in LDDT plateau after three runs, even with higher computational expenses. Hence, we chose runs optimize performance and efficiency. Furthermore, higher augmentation factor does not always yield better results; the identical input MSA might lack new insights, and extra sequences risk introducing noise.
Conclusion
We present an unsupervised seqs2seqs task, accompanied by automated dataset construction pipeline, designed to pre-train MSA-Cenerator for simultaneous multi-sequence generation. Rigorous experimentation underscore the effectiveness, diversity, and conservation feature of generated virtual MSAs, amplifying the prowess stalwarts like AlphaFold2 in scenarios where conventional methods come up short. Furthermore, protein sequence families a zero-shot fashion. Our findings highlight the immense promise of the unsupervised seqs2seqs task, pointing towards its prospective utility in broader spectrum of bio-sequences, thereby amplifying its benefits.
approach demonstrates generalization across wide array of
Limitation
While MSA-Generae demonstrates promising improvements in protein structure prediction, there are several limitations to consider.
First, did not conduct scale-up experiments to explore the effects of increasing model size and expanding the pre-training dataset. is possible that larger models and more comprehensive training data could further enhance the generative capabilities for sequence prediction, potentially improving structure prediction accuracy. Second, our current approach relies on selecting the maximum PLDDT score from multiple runs, which may not be the most optimal method and introduces inefficiency due to the need for repeated predictions. Developing more refined metric to assess MSA quality or better predict the reliability the final structure could significantly improve efficiency and consistency in future work. Lastly, although our model effectively handles proteins with limited homologous sequences, the generated MSAs typically do not achieve substantial depth. Increasing the depth of these generated MSAs would require higher computational resources, as the complexity scales with MSA depth. Addressing this challenge will be crucial for extending the applicability of our approach to broader range of protein sequences.
Acknowledgments
The technical support extended by the Mila IDT team managing the computational infrastructure is greatly appreciated. This project was partially supported by Shanghai Artificial Intelligence Laboratory. S.S. is supported by funds from Netmind Al and Protagolabs Inc.
10

---

References
Stephen Altschul, Thomas I. Madden, Alejandro Schäffer, Jinghui Zhang, Zheng Zhang, Webb Miller, and David J Lipman. Gapped blast and psi-blast: new generation of protein database search programs. Nucleic acids research, 25(17):3389 3402, 1997.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N. Kinch, R. Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R. Glassman, Andy DeGiovanni, Jose H. Pereira, Andria V. Rodrigues, Alberdina A. van Dijk, Ana C. Ebrecht, Diederik J. Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K. Rathinaswamy, Udit Dalwadi, Calvin K. Yip, John Burke, K. Christopher Garcia, Nick V. Grishin, Paul D. Adams, Randy J. Read, and David Baker. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557):771-87, 2021. doi: 10. 1 1226/ccience.ab88554. URL https wwww.sciinee www.ssccennn...e.eeeooo http///i//a////e/ 1126/ science. abj8754
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2110 2022.
Bo Chen, Ziwei Xie, Jiezhong Qiu, Zhaofeng Ye, Jinbo Xu, and Jie Tang. Improved the protein complex prediction with protein language models. bioRxiv, pp. 2022-00220000 09, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Blizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam. Vedant Misra, Evan Morikawa. Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Charlotte Rochereau, George M Church, Peter Karl Sorger, and Mohammed N AlQuraishi Single-sequence protein structure prediction using language models from deep learning. bioRxiv, 2021.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M Church, et al. Single-sequence protein structure prediction using a language model and deep learning. Nature Biotechnology, 40 (11):1617-1623, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetueed language models, 2022.
M Dayhoff, R Schwartz, and B Orcutt. 22 model of evolutionary change in proteins. Atlas of protein sequence and structure, 5:345-352, 1978.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Ming-Wei
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of life's 's code through self-supervised deep learning and high performance computing. 2021.
Matteo Figliuzzi, Pierre Barrat-Charlaie and Martin Weigt. How pairwise coevolutionary models capture the collective residue variability in proteins? Molecular biology and evolution, 35)4): 1018-1027, 2018
11

---

Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences, 89(22):00155100919, 1992.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint arXiv: 12100, 2019.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2019.
I. Steven Johnson, Sean R Eddy, and Elon Portugaly. Hidden markov model speed heuristic and iterative hmm search procedure. BMC bioinformatics, 11(1):1-8, 2010.
David T Jones. Protein secondary structure prediction based on position-specific scoring matrices. Journal of molecular biology, 292(2):195-20) 1999.
David T Jones, William R Taylor, and Janet M Thornton. The rapid generation of mutation data matrices from protein sequences. Bioinformaticx, 833):275-22,, 1992.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool. Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)-roum) xiii. Proteins: Structure, Function, and Bioinformatics, 87(12):1011-102) 2019.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Ali Madani. Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arxiv:2004. 03497, 2020.
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddr: a local superposition- free score for comparing protein structures and models using distance difference tests. Bioinfor- matics, 29(21):2722-2777 2728, 2013.
Francisco McGee, Sandro Hauri, Quentin Novinger, Slobodan Vucetic, Ronald M Levy, Vincenzo Carnevale, and Allan Haldane. The generative capacity of probabilistic protein sequence models. Nature communications, 12(1):6302, 2021.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible all. Nature methods, 19(6))679-66) 682, 2022.
Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: Exploring the boundaries of protein language models." arxiv, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21 21(140):1-67, 2020.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure leamners. Biorxiv, 2020.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny. Picter Abbeel. Tom Sercu, and Alexander Rives. Msa transformer. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8844 8856. PMLR, 18-24 Jul 2021. W htt.intsss/prrceedingg. mlr press/v139/rac22 html
Adam J Riesselman, John Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816822) 2018.
12

---

Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu. Demi Guo, Myle Ott, Lawrence Zitmick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15))e211233111, 2021.
Burkhard Rost and Chris Sander. Prediction of protein secondary structure at better than 70% accuracy. Journal of molecular biology, 232(():584-59,, 1993.
William P Russ, Matteo Figliuzzi, Christian Stocker, Pierre Barrat-Charlai, Michael Socolich, Peter Kast, Donald Hilvert, Remi Monasson, Simona Cocco, Martin Weigt, al. An evolution-based model for designing chorismate mutase enzymes. Science, 369(6502):440-4455 2020.
Damiano Sgarbossa, Umberto Lupo, and Anne-Florence Bitbol. Generative power of protein language model trained on multiple sequence alignments. bioRxiv, 2022.
Sam Sinai, Eric Kelsic, George Church, and Martin Nowak. Variational auto-encoding protein sequences. arXiv preprint arXiv:1712.03346, 2017.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(11):1026-1028, 2017.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 23(10):1282-1288, 2007.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anme Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: interpreting attention in protein language models. arXiv preprint arXiv:2006. 15222,, 2020.
Qin Wang, Jiayang Chen, Yuzhe Zhou, Yu Li, Liangzhen Zheng, Sheng Wang. Zhen Li, and Shuguang Cui. Contact-distil: Boosting low homologous protein contact map prediction by self-supervised distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4620-4627, 2022a.
Sheng Wang, Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLOS Computational Biology, 13())ee1003324, jan 2017. doi: pcbi. 1005324
pob.
1005324. BtRphttss///di..oon
137 1371%/3f
Wenkai Wang, Zhenling Peng, and Jianyi Yang. Single-sequence protein structure prediction using supervised transformer protein language models. Nature Computational Science, 2(12):804-88 814, 2022b.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-077 07, 2022a.
Ruidong Wu, Fan Ding, Rui Wang. Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, 2022b.
Chengxin Zhang, Wei Zheng, S. M. Mortuza, Yang Li, Yang Li, and Yang Zhang. Deepmsa: constructing deep multiple sequence alignment improve contact prediction and fold-recognition for distant-homology proteins. Bioinformatics, 2019. URI,httpt //api. http:::://pppaaaaaeeeeeeee semanticscholaar
Org:/oorpusID:2081.. 208144255
13

---

Jun Zhang, Sirui Liu, Mengyun Chen, Haotian Chu, Min Wang, Zidong Wang, Jialiang Yu, Ningxi Ni, Fan Yu, Diqing Chen, et al. Few-shot learning of accurate folding landscape for protein structure prediction. arXiv preprint arXi::2208. 09652, 2022.
14

---

A Pretraining Details
We have compiled pretraining dataset containing 2M MSAs from four databases: Uniref90 v.2022.00 _04 using pipeline discussed in section 3.1 Specifically, for each MSA we randomly se- lect 10-30 sequences and query as source X and another 10-30 sequences as target Y.
B MSA Visualization
Our goal is to explore the variations MSA sequences using MSA-Generator, Accordingly, we depict the MSA's colored distribution in Fig using Jalviex Observing the columnar distribution, it's evident that the produced MSA bears resemblances to the original sequences but introduces unique variations that encapsulate the external insights derived from MSA-Gennrator, This show the conservative feature reserved by the MSA and enhanced diversity as well.
T1093-D1
T1096-D1
T1094-D2
11
Figure colored-distribution MSA. different colors represent different amino acids in protein sequence, from top to bottom is T1060s2-D1, T1093-D1, T1096-DI. since MSA-Generator augment one times more sequences, the top half each diagram represents original MSA, and bottom half represent generated MSA
C Search Parameter
We use JackHMMER build pretraining MSA dataset. We adopt default parameters, including: -E
= =0.001, -N=5, -Z=1000, -incE=0.01.
D Relation between MSA depth, quality, and downstream performance
The relationship between Multiple Sequence Alignment (MSA) depth, its quality, and the consequent impact on downstream performance critical factor our analysis. For an in-depth discussion on how MSA depth influences quality and downstream tasks, we refer the reader to Figure 5 (a) in the AlphaFold2 paper by Jumper al. JJumper et al.) (202T), where the authors illustrate that a reduction in MSA depth can lead to decreased quality and hinder performance in downstream applications.
In support of this, our Figure which presents violin plot, shows that baseline MSA depth of 5 tends to underperform when compared to real and virtual depths of 15. This observation is particularly evident from the distribution of scores, where the baseline depth of 5, represented blue, is more concentrated towards lower scores, unlike the distributions for real and virtual depths of 15, depicted in yellow and red respectively.
Additionally, an intuitive approach would be introduction random protein sequences to the MSA, implementation of 'Iterative Unmasking' baseline already employed similar concept which
'This figure is intended solely for the visualization of the generated Multiple Sequence Alignments, and does contribute the analytical evaluations of our study.
15

---

Method
CSAPI4-plddt
CSAPI4-Iddt
CSAPII--lddt
CSAPI5-Iddt
AlphaFold2
65.1
53.2
65.1
55.6
AlphaFold225% random
64.3
50.6
62.3
50.9
100% random
60.5
49.5
59.6
43.2
1lphaFold2+15% random
55.6
44.3
53.8
39.6
AlphaFold2+Ours
71.6
57.5
73.7
63.7
Table 3: Comparison of different methods on CSAP14 and CSAP15 datasets.
randomly unmasked tokens using the MSA Transformer. Despite this strategy, as shown in our Table I the improvement in results was not significant. To address further concerns, explored the introduction of random sequences by substituting 5%, 10%, and 15% of the tokens within the MSA, thereby artificially increasing its depth documented in Table This modification, however, led to a decline in downstream performance, thereby affirming the effectiveness of our original method.
E
Detailed Results
Real-Word Difficult and Challenging Results
Detailed results for Section and are pre-
sented in Table 4
pLDDT
LDDT
Depth
Org
Runl
Run2
Run3
Org
Runl
Run2
Run3
T1037-D1
36.0M
40.68
36.04
36.04
24.09
26.08
24.09
24.09
T1042-D1
2
41.99
42.25
45.45
47.86
32.42
32.19
32.43
32.84
T1064-D1
60.34
63.1
65
31.25
37.48
31.34
51.36
T1074-D1
85.96
85.96
85.96
81.14
81.14
81.14
81.14
T1082-D1
10
92.31
92.79
92.31
92.31
87.37
88.49
87.37
87.37
T1093-D1
62.1
81.26
62.1
62.1
45.5
70.77
45.5
45.5
T1094-D2
90.54
90.54
93.34
91.47
77.12
77.12
77.02
76.47
T1096-D1
70.28
86.25
83.28
61.92
62.83
71.19
69.23
T1096-D2
2
45.55
50.01
54.55
50.24
34.07
36.56
36.98
34.66
T1099-D1
8
89.27
89.27
90.38
75.12
74.27
75.12
74.62
T100-112
42.16
47.27
42.16
42.16
34.93
35.87
34.93
34.93
T1113
10
43.5
82.4
76.4
77.0
32.6
80.6
78.9
78.0
T1119
2
93.4
94.5
92.4
93.6
91.1
90.8
90.4
90.0
1122
1
65.9
80.2
78.6
79.8
43.3
73.4
70.2
83.2
1125
15
49.6
54.4
54.8
53.2
38.2
45.0
45.0
43.2
1130
50.6
60.2
60.0
58.9
39.4
46.0
46.9
45.8
1131
1
40.6
46.0
45.4
46.2
32.6
48.6
38.2
38.6
1178
15
79.2
80.6
81.4
80.3
58.2
60.9
45.9
58.9
1194
14
93.4
94.1
93.4
93.4
90.2
91.7
90.5
91.7
Table 4: pLDDT (left) and LDDT (right) improvement over difficuut MSA (depth=220) of runs.
Artificial Extreme Challenging Results
Results for section 4.2
shown in table 5 and table
16

---

MSA-ID
Gold
Original
Augl
Aug2
Aug3
ENs
over original
over gold
TI024-D1
87.10
81.56
77.25
81.62
81.62
81.62
0.06
-5.48
T1024DD2
84.44
86.66
79.67
82.84
82.84
82.84
-3.82
-1.60
T1025-D1
83.81
82.04
78.92
76.61
76.61
78.92
-3.12
4.89
T1026-D1
82.27
48.72
22.59
79.82
79.82
79.82
31.10
-2.45
T1027-D1
48.81
39.15
37.31
46.38
46.38
46.38
7.23
-2.43
T1028-D1
75.57
32.35
52.62
52.09
52.09
52.62
20.27
-22.55
T1029-D1
47.62
47.43
47.76
47.44
47.44
47.76
0.33
0.14
T1030-D1
86.85
64.92
86.68
86.56
86.56
86.68
21.76
-0.17
T1030-D2
80.99
81.76
82.06
82.06
82.06
-0.19
T1031-D1
65.65
43.58
30.75
37.99
37.99
37.99
-5.99
27.66
T1032-D1
22.48
19.90
68.50
62.11
62.11
68.50
48.60
46.02
T1034-D1
86.75
83.15
83,65
84,05
84,05
84.05
0.90
-270
T1035-D1
32.52
33.78
34.56
36.32
36.32
36.32
2.54
3.80
T1036s I-DI
67.02
26.11
79.99
79.13
79.13
79.99
53.88
12.97
T1038-D1
26.03
38.04
27.61
27.61
38.04
-17.76
T1038-D2
80.60
76.37
61.02
58.57
$3.99
61.02
15.35
-19.58
T1041-D1
62.13
45.60
32.14
0.00
34.74
34.74
10.86
-27.99
T1045sI-D1
56.74
33.67
89.22
83.99
88.47
89.22
55.55
32.48
T1045&2-D1
27.65
24.44
22.81
24.89
21.61
24.89
0.45
-2766
T1046s I-DI
87.67
72.65
87.33
0.00
87.98
87.98
15.33
0.31
T1046s2-D1
51.58
$2.44
36.05
52.44
-11.44
T1047s1-D1
60.01
59.75
61.65
61.23
61.65
2.89
T1047s2-D1
40.65
39.
69.07
70.64
71.69
71.69
32.57
31.04
T1047s2-D2
58.59
59.58
64.60
64,30
67.72
67.72
8.14
9.13
T104762-D3
58.12
51.61
57.63
58.18
57.53
58.18
6.57
0.06
T1048-D1
70.60
72.34
71.81
71.75
71.82
71.82
-0.52
1.22
T1049-D1
80.28
81.03
72.12
8.18
T1050-D1
89.66
88.46
73.27
72.51
72.97
73.27
-15.19
-16.39
TI050-D2
81.05
76.84
76.81
78.93
0.00
78.93
2.09
-2.12
TI050-D3
71.11
70.25
48.28
51.18
49.74
51.18
-1.007
-9,933
T1052-D1
77.26
68.84
82.18
83.82
83.08
83.82
14.98
6.56
TI052-D2
48.22
34.95
57.37
59.27
30.91
59.27
24.32
11.05
T1052-D3
89.62
85.22
-1.19
T1053-D1
72.86
30.58
73.75
74.09
65.32
74.09
43.51
1.23
T1053-D2
75.75
75.61
77.31
76.25
0.00
77.31
1.70
1.56
TI054-D1
34.64
31.68
70.68
81.44
0.00
81.44
49.76
46.80
T1055-D1
67.90
55.88
19.03
25. 25.46
21.62
25.46
-30.42
-42.44
Table 5: artificial challenging MSAs results (1/2).
17

---

MSA-ID
Gold
Original
Aug1
Aug2
Aug3
ENs
original
TI056-DI
75.61
76.11
64.62
65.39
62.93
65.39
-10.72
-10.2
T1057-D1
86.18
52.96
81.38
80.30
81.04
81.38
28.42
-4.0
TI0S8-D1
45.72
38.
37.47
41.31
39.76
41.31
3.21
44.1
T1058-D2
51.18
29.59
61.13
60.41
60.73
61.13
31.54
9.95
T1060s2-D1
75.25
72.84
74.01
0.00
74.01
35.57
-1.24
T1060s3-DI
80.69
78.72
78.31
78.89
78.89
-1.80
T1061-DO
60.24
57.51
54.74
0.00
58.36
$8.36
0.85
0.85
T1061-D1
37.50
24.28
51.45
46.94
50.60
51.45
27.17
13,95
T106I-D2
53.34
31.82
58.05
50.32
$5.92
58.05
26.23
4.71
TI061-D3
81.92
42.92
77.44
77.34
76.29
77.44
34.52
.4.8
T1062-D1
59.10
72.66
72.61
59.58
0.00
72.61
-0.05
13.51
T1065s1-D
44.32
90.31
91.00
0.00
2.41
T1065x2-D1
70.46
63.90
80.77
87.09
83.49
87.09
23.19
16.63
T1067-D1
78.55
27.33
79.05
75.29
77.91
79.05
51.72
0.50
T1068-D1
85.05
35.20
89.55
88.93
89.92
89.92
54.72
4.87
T1070-D1
53.44
49.68
$5.69
52.55
52.73
55.69
6.01
2.25
T1070-D2
29.16
35.04
87.50
86.84
90.38
90.38
55.34
61.22
T1070-D3
73.32
73.56
73.50
0.00
73.56
0.24
-0.56
T1070-DA
73.83
30.62
74.39
84.71
85.07
85.07
$4.45
11.24
T1073-D1
76.17
76.19
76.79
0.00
76.11
76.79
0.60
0.62
T1076-D1
83.10
65.52
71.31
68.61
68.35
71.31
5.79
-11.79
TI078-D1
77.22
54,56
0.00
64.61
72.25
72.25
17.69
4.97
T1079-D1
84.51
79.44
69.34
68.08
79.44
5.00
T1080-D1
59.76
67.91
67.67
68.08
68.08
4,40
T1083-D1
78.49
78.26
78.02
77.48
77.72
78.02
-0.24
-0.47
T1084-D1
86.64
86.47
49.09
86.03
86.25
86.25
-0.22
-0,39
T1087-D1
80.76
77.32
37.14
39.73
69.27
69.27
-8.05
-11.49
T1088-D1
26.48
24.00
33,92
34.10
54.40
54.40
30.40
27.92
T1089-D1
64.70
64.30
67.15
-10.40
-16.44
T1090-D1
76.73
55.34
37.65
0.00
56.06
57.65
-19,08
T1091-D1
43.89
43.26
70.35
71.45
76.76
76.76
33.50
32.87
T1091-D2
81.76
40.56
50.57
0.00
47.30
50.57
10.01
-31.19
T1091-D3
71.58
52.00
75.93
0.00
75.98
75.98
23.98
4.40
T1091-D4
79.76
56.64
83,20
0.00
78.01
83.20
26.56
3.44
T1092-D1
56.56
33.97
38.95
40.48
40.48
-23.99
T1092-D2
70.61
53.20
43.76
26.54
43.80
43.80
-26.81
T1093-D2
29.91
34.69
28,20
28.12
31.08
31.08
-3.61
1.17
T1094-D1
28.74
26.73
21.89
21.63
20.72
21.89
-4.84
-6.85
T1095-D1
60.06
55.57
35.95
36.50
32.78
36,30
-19.07
-23.56
T1098-D1
57.45
29.82
49.36
45.94
36.76
49.36
19.54
-8.09
T1098-D2
37.88
31.58
45.70
45.89
45.36
45.89
14.31
8.01
T1100-D1
57.78
65.70
60.39
60.01
61.10
61.10
-4.60
3,32
T1101-D1
88.14
87.57
87.95
87.64
86.52
87.95
0.38
-0.19
T1101-D2
78.74
77.81
43.27
42.09
74.91
74.91
-2.00
-3.83
Average
33.45
61.77
57.14
56.43
66.32
12.77
40.11
Table artificial challenging MSAs results (2/2).
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided a proper justification given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract is summary of the content of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the last paragraph of Conclusion section.
19

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details of training and inference in section 3 and 4.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We released the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (lttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
yes, provided in section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We didn't provide statical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We perform the task with one gpu, and we provide training details for pretrain- ing.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow protocol.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspppppssttteeeeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide codebase.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: Not relevant.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "FmZGH6obvK", "reviewer": "Reviewer_3nEJ", "summary": "The paper proposes a method to generate MSA sequences to provide more alignments of the MSA. MSA-Generator can increase the depth of the MSA input, and thus incorporate more information. MSA-Generator demonstrates its capacity to synthesize higher-quality MSA via experiments on the CASP dataset.", "questions": "- See above.", "limitations": "- Further experiments on CAMEO can be helpful.\n\n- MSA_Generator's capacity on Individual sequence. Experiments are not necessary, maybe some case study or analysis.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- MSA-Generator can alleviate the shortage burden of MSA data in proteomics research. Protein sequences are in large amounts, which MSA data are limited or require heavy computation costs to obtain. MSA-Generator tries to resolve the challenge by leveraging the generative language models to generate more/argument alignments to increase the depth of MSA.\n\n- The MSA data generated by the MSA-Generator helped the existing MSA-based structure prediction models to achieve better performance (AlphaFold2 and RoseTTAFold) on the CASP14 and CASP 15 datasets.", "weaknesses": "- There are some other datasets available for experiments. Have you considered conducting experiments on datasets like CAMEO?\n\n- Figure 4 (c) shows the distribution of LDDT improvement. Can you provide a more precise clarification on Virtual MSAs' pros and cons compared with real MSAs?\n\n- Will the MSA-Generator be helpful for individual protein sequences as the input? There are some works on single-sequence protein language models worth discussing in your related work section.\n\n[1] ProteinBERT: a universal deep-learning model of protein sequence and function.\n[2] Modeling aspects of the language of life through transfer-learning protein sequences.\n[3] Modeling Protein Using Large-scale Pretrain Language Model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s feedback and would like to clarify the following points:\n\n1. **CAMEO Results**\n    \n    Thank you for suggesting the inclusion of CAMEO as an additional benchmark. We have taken this into consideration and conducted further experiments. For the CAMEO benchmark, we searched its MSA on the UniCluster30 database using HHblits and identified protein sequences with fewer than 20 homologs as the Real-World Challenge set from CAMEO following [1]. The results of our method are presented below:\n    \n    | CAMEO(avg Depth=8.5) | pLDDT | LDDT | TM-Score | GDT-TS |\n    | --- | --- | --- | --- | --- |\n    | ESMFold | 49.3 | 46.8 | 0.65 | 0.51 |\n    | OmegaFold | - | 47.9 | 0.59 | 0.47 |\n    | RoseTTAFold | 69.8 | 57.0 | 0.62 | 0.55 |\n    | RoseTTAFold+Potts Generation | 69.6 | 56.7 | 0.59 | 0.50 |\n    | RoseTTAFold+Iterative Unmasking | 70.2 | 60.1 | 0.62 | 0.57 |\n    | RoseTTAFold+MSA-Generator | 75.6 | 62.9 | 0.69 | 0.62 |\n    | AlphaFold2 | 72.6 | 59.4 | 0.69 | 0.61 |\n    | AlphaFold2+Potts Generation | 72.3 | 59.0 | 0.64 | 0.57 |\n    | AlphaFold2+Iterative Unmasking | 74.2 | 60.6 | 0.70 | 0.63 |\n    | AlphaFold2+MSA-Generator | 77.2 | 64.2 | 0.73 | 0.67 |\n    \n    These results are consistent with our findings on the CASP14/15 dataset.\n    \n2. **Virtual MSAs' pros and cons compared with real MSAs**\n    \n    Thank you for raising this valid question. In section 4.2, we compare the performance of generated virtual MSAs with real MSAs under simulated conditions where we downsample 5 sequences as the baseline MSA and 15 sequences as the real MSA. Our findings in this controlled simulation indicate that virtual MSAs are generally as effective as real MSAs, and in some cases, virtual MSAs even outperform real MSAs.\n    \n    However, we want to emphasize that our study primarily focuses on situations where no real MSA can be constructed using search algorithms, as discussed in section 4.3. We do not suggest that virtual MSAs are superior to real MSAs overall. Instead, in cases where real MSAs cannot be found, virtual MSAs provide a valuable alternative for enhancing protein structure prediction, and this is where our method makes a significant contribution.\n    \n3. **Individual Protein Sequence**  \n    \n    Yes, MSA-Generator also benefit individual protein sequences as the input. We have included results for proteins with only single sequences and presented these findings in our global response. Please refer to it for detailed discussion.\n    \n4. **Relevant Work** \n    \n    Thank you for mentioning these relevant works ([1][2][3] in the comments). We will include them in our revision. However, the works referenced focus on encoder models that aim to learn good representations for protein sequences, whereas our method is an encoder-decoder model focused on generating high-quality MSAs.\n    \n    We have adopted single-sequence models, including ESMFold [2] and OmegaFold[3], as our baselines in section 4. We will incorporate a more detailed discussion in section 2 in our revision.\n    \n[1] Chen B, Bei Z, Cheng X, et al. MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training[J]. arXiv preprint arXiv:2406.05347, 2024.\n\n[2] Lin Z, Akin H, Rao R, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction[J]. BioRxiv, 2022, 2022: 500902.\n\n[3] Wu R, Ding F, Wang R, et al. High-resolution de novo structure prediction from primary sequence[J]. BioRxiv, 2022: 2022.07. 21.500999.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2a3IHUfAaC", "reviewer": "Reviewer_6doy", "summary": "This paper introduces a method for generating multiple sequence alignments (MSAs) using a self-supervised seq2seq task. By leveraging large-scale protein databases, this approach produces virtual, informative MSAs that enhance the performance of protein structure prediction models such as AlphaFold2 and RoseTTAFold. The improvements are especially noticeable for proteins without extensive homologous families, demonstrating that data augmentation techniques can also be effectively applied in the protein domain.", "questions": "See above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of a seqs2seqs task for MSA generation is innovative, leveraging the power of self-supervised learning to improve protein structure predictions.\n2. The method demonstrates marked improvements in lDDT and GDT-TS scores on challenging protein sequences using CASP14 and CASP15 benchmarks.\n3. The enhanced MSA has the potential to serve as an auxiliary dataset for training on protein-related tasks. It would be beneficial if the authors could make their code and the dataset publicly available, allowing the community to take full advantage of these resources.", "weaknesses": "1. As the main contribution of this work is generating more homologous sequences based on the original low-quality MSA, it would be valuable to investigate how the MSA generator improves structure prediction for orphan proteins.\n2. What criteria are used to determine the quality of homologous sequences? Are they based on a small number of sequences, or do they consider homologous sequences with only sequence similarity but not structural similarity? Additionally, are the pre-trained datasets composed solely of high-quality homologous sequences?\n3. Using MSA as input demands significant memory resources. What thresholds are set for the input source MSA and target MSA depth during training? Is there a relationship between the depth of the source MSA and the depth of the target MSA? For instance, is the depth of the source MSA greater than that of the target MSA?\n4. I noticed that the tied row attention mechanism is employed in the encoder. To reduce memory usage, have you considered introducing this shared attention mechanism in the cross-row attention within the decoder?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s feedback and would like to clarify the following points:\n\n1. **Orphan proteins**\n    \n    We have included results for proteins with only single sequences and presented these findings in our global response. Please refer to it for detailed discussion.\n    \n2. **Criteria are used to determine the quality of homologous sequences**\n    \n    To measure the quality of generated homologous sequences, we use AlphaFold2 and evaluate the predicted protein structure accuracy against the reference structure as the metric. The sequences are selected based solely on the depth of the MSA, without considering structural similarity.\n    \n    For pre-training data construction, we use the protein sequence search tool JackHMMer, following AlphaFold2, and set the e-value to 0.001 to ensure high-quality homologous sequences for pre-training. The search criterion is based on sequence similarity measured by profile hidden Markov model. For more details, please refer to [1].\n    \n3. **MSA depth**\n    \n    Thank you for raising this valid question. Generating MSAs can indeed be computationally expensive when the input is very deep. However, our goal is to enhance MSAs in situations where rich MSAs are not available, which means the depth of the MSA is typically shallow, making our framework accessible.\n    \n    During training, we randomly select 10-30 sequences as input and randomly sample another 10-30 sequences as the target from the JackHammer search results, as detailed in Appendix A. There is no requirement for the input to be deeper than the target to mimic real-world applications.\n    \n4. **Tied-row attention** \n    \n    We appreciate you raising this valuable question. Tie-row attention is designed to compress global information from all input MSAs, which is valid for the encoder as it aims to learn a global representation for the input MSA. However, we did not include this shared MSA mechanism in the decoder to avoid assigning the same weight to each sequence during decoding, thereby ensuring the generation of **diverse** output sequences.\n\n[1] Potter S C, Luciani A, Eddy S R, et al. HMMER web server: 2018 update[J]. Nucleic acids research, 2018, 46(W1): W200-W204.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzDIhKOjIp", "reviewer": "Reviewer_xVQk", "summary": "This work introduces MSA-Generator to generate virtual, informative MSAs. The generated MSAs can advance protein structure prediction.", "questions": "See weaknesses", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The MSA generation and protein structure prediction problems studied in this work are important.\n- The writing is clear and the method is easy to follow.", "weaknesses": "- The technology novelty is somewhat limited. The model architecture mainly follows the MSA transformer, and the training framework is similar to Seq2Seq.\n- I notice that there are several related work that also studies the problem of generating MSA to advance protein structure prediction [1]. The differences and advantages should be discussed.\n- The ablation studies are weak. More design choices should be verified.\n\n[1] MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Novelty**\n    \n    The innovation of our work lies in our pioneering approach to **self-supervised pretraining for Multiple Sequence Alignment (MSA) generation**. While tied-row attention and self-column attention are similar to the mechanisms in the MSA Transformer, it's important to note that the MSA Transformer is an encoder-only model, primarily designed for learning MSA representations. In contrast, our model is an encoder-decoder model, which is specifically tailored for generating MSAs that are more effective for structural prediction. This distinction leads to significant differences in both architecture design and training approach.\n    \n    Additionally, our **seqs2seqs** framework extends the vanilla seq2seq by focusing on generating multiple sequences that share co-evolutionary information in parallel, owing  to the cross-row/column attention we introduced. Our contribution is valuable as we demonstrate the efficacy of large-scale self-supervised MSA generation.\n\n2. **Related Work**\n    \n    We have already incorporated most of the prior protein/MSA generation methods in the related work section. While we appreciate your reference to MSAGPT [1], this work was actually released (8 Jun) after the NeurIPS submission deadline (22 May). Therefore, it was not possible to include a discussion and comparison in our draft. However, we are happy to include it in our revision.\n\n3. **Ablation Study**\n    \n    Thank you for your feedback regarding the ablation studies. We acknowledge the importance of verifying design choices; however, conducting extensive ablation studies is computationally expensive for our end-to-end pre-training model. \n    \n    Additionally, each component of our model is indispensable for its proper functioning. We appreciate your understanding and are open to discussing alternative approaches for future work.\n\nWe hope the discussion addresses your concerns, and welcome further discussion.\n\n[1] Chen B, Bei Z, Cheng X, et al. MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training[J]. arXiv preprint arXiv:2406.05347, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwoF0S8vlO", "reviewer": "Reviewer_Gq4n", "summary": "The paper introduces MSA-Generator, a novel self-supervised generative protein language model designed to address the limitations in protein structure prediction due to shallow multiple sequence alignments (MSAs). This model, pre-trained on a sequences-to-sequences task using an automatically constructed dataset, incorporates protein-specific attention mechanisms. These features enable MSA-Generator to produce virtual, enriched MSAs from large-scale protein databases, improving the quality of MSAs particularly for proteins without extensive homologous families.", "questions": "See weakness.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Results on CASP14 and CASP 15 seem promising. The method enhances more than one pretrained model, showing the improvement on MSA quality is general.\n- Low-quality MSA and lack of MSA have been a challenging problem in protein predictions. This work address an important bottleneck in the field.\n- Interesting insights are raised in anlaysis. The generated MSA improves quality in terms of diversity and conservation areas.", "weaknesses": "Generally the paper is well-written and performs adequate analysis on important benchmarks. There a few weaknesses that could be addressed to improve the work:\n\n- Generated MSA improve performance on large-scale pretrained frameworks which are already pre-trained on MSA. Specifically, Alphafold2 has been found to greatly benefit from MSA sequence numbers as well as diversity. However, for traditional models like Potts/ co-evolution based statistic models, could generated MSA improve performance? This could showcase whether generated MSA could replace MSA as a representative input feature.\n\n- JackHMMER is not a commonly used MSA tool and may show declined performances. Do authors consider using more advanced tools like hhblits and mmseqs2? Also what is the e-value of the generated MSA compared to MSA ?\n\n- Can this method be used to generate a MSA from scratch rather than augment MSA? MSA computation bottleneck is a even more severe problem and generation could improve it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your feedback and advice on improving the work, here we provide more discussion:\n\n1. **Generated MSA for Traditional Models**\n    \n    This is indeed an interesting and valuable question. We adopted CCMpred [1], one of the leading graphical models for protein contact map prediction, to evaluate how our augmented MSA benefits these traditional models. We measured the Average Top L/5 Precision on the real-world challenge set from CASP14, which includes targets with fewer than 10 homologs (T1064-D1, T1093-D1, T1100-D2, T1096-D2, T1099-D1, T1096-D1). The results are as follows:\n    \n    |  | Original | Augmented |\n    | --- | --- | --- |\n    | Top L/5 | 0.176 | 0.205 |\n    \n    These results suggest that the generated MSA can also improve the performance of traditional methods, suggesting the widely usage of the proposed model.\n\n2. **MSA Search Algorithm** \n    \n    We follow the **MSA dataset construction pipeline of AlphaFold2** [2], which employs JackHMMER. We appreciate your suggestion to consider advanced tools like HHblits and MMseqs2 and plan to integrate these tools to construct larger datasets in our future work.\n    \n    Regarding the E-value, to our best knowledge, it measures the likelihood of an **individual sequence alignment** occurring by chance, not multiple sequence alignments (MSAs). Therefore, we believe comparing the E-value of one MSA to another is inappropriate. When using JackHMMER with UniRef90 in our setup, the E-value threshold is set to 0.001.\n\n3. **Generate MSA from scratch**\n    \n    We have included results on orphan proteins in our **global response**, which involves generating MSAs from scratch.\n    \n    However, generating MSAs from scratch is inherently challenging. Without proper contextual information from an input MSA, synthesizing multiple sequences that share co-evolutionary knowledge becomes difficult. One potential solution is to scale up the training size, with the hope that larger models, having seen more sequences, can achieve zero-shot MSA generation.\n    \n    Regarding the computation required for MSA generation, given the heavy generation cost of larger models, we do not anticipate that it would be much more efficient than search algorithms if there are many homogeneous sequences. Instead, we see the value of applying generation methods to proteins that lack homologs.\n    \n[1] Seemayer S, Gruber M, Söding J. CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations[J]. Bioinformatics, 2014, 30(21): 3128-3130.\n    \n[2] Jumper J, Evans R, Pritzel A, et al. Highly accurate protein structure prediction with AlphaFold[J]. nature, 2021, 596(7873): 583-589.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' efforts and feedback. We noticed a common interest in whether the proposed method could benefit single protein sequences, also referred to as orphan protein sequences.

To address this, we conducted experiments using the entire CASP14/15 dataset (the dataset used in Section 4.3) with only single protein sequences as input. Additionally, for a more comprehensive understanding, we included results from a orphan protein family, Orphan25 [1]. We used MMseqs2 to search against UniRef30 and  ColabFoldDB [2], which was built by expanding BFD/MGnify with metagenomic sequences from various environments, selecting sequences with no homologues, and obtained 10 proteins as a test set (6WKY, 6WL0, 6XA1, 6XN9, 6XYI, 7A5P, 7AL0, 7JJV). We conducted the experiment using the same setup as in Section 4.3. The results are detailed below:

| CASP14&15 | pLDDT | LDDT | TM-Score | GDT-TS |
| --- | --- | --- | --- | --- |
| AlphaFold2 | 43.8 | 26.9 | 0.30 | 0.28 |
| AlphaFold2+Potts Generation | 37.7 | 22.2 | 0.21 | 0.23 |
| AlphaFold2+Iterative Unmasking | 48.2 | 30.8 | 0.32 | 0.33 |
| AlphaFold2+MSA-Generator | 57.2 | 36.2 | 0.39 | 0.37 |

| Orphan25 | pLDDT | LDDT | TM-Score | GDT-TS |
| --- | --- | --- | --- | --- |
| AlphaFold2 | 77.2 | 61.6 | 0.61 | 0.62 |
| AlphaFold2+Potts Generation | 68.9 | 49.3 | 0.49 | 0.43 |
| AlphaFold2+Iterative Unmasking | 78.9 | 62.5 | 0.64 | 0.63 |
| AlphaFold2+MSA-Generator | 81.8 | 66.4 | 0.69 | 0.67 |

The results with orphan protein sequences on the two datasets suggest that our method can also benefit orphan proteins and be particularly helpful for challenging inputs (CASP), further strengthening the effectiveness of our approach.

[1] Wang W, Peng Z, Yang J. Single-sequence protein structure prediction using supervised transformer protein language models[J]. Nature Computational Science, 2022, 2(12): 804-814.

[2] Mirdita M, Schütze K, Moriwaki Y, et al. ColabFold: making protein folding accessible to all[J]. Nature methods, 2022, 19(6): 679-682.

Meta-review: We recommend acceptance as a poster:
* MSA-Generator can alleviate the shortage burden of MSA data in proteomics research
* Results on CASP14 and CASP 15 are promising
* Interesting insights are raised in the analysis
* Authors improved the paper during the rebuttal, with news results CAMEO and Orphan25

To authors: please make sure you include all reviewer comments in the final revision.
Decision: Accept (poster)