MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:13:05

Abstract: Deep learning models like AlphaFold2 have revolutionized protein structure prediction, achieving unprecedented accuracy. However, the dependence on robust multiple sequence alignments (MSAs) continues to pose a challenge, especially for proteins that lack a wealth of homologous sequences. To overcome this limitation, we introduce MSA-Generator, a self-supervised generative protein language model. Trained on a sequence-to-sequence task using an automatically constructed dataset, MSA-Generator employs protein-specific attention mechanisms to harness large-scale protein databases, generating virtual MSAs that enrich existing ones and boost prediction accuracy. Our experiments on CASP14 and CASP15 benchmarks reveal significant improvements in LDDT scores, particularly for complex and challenging sequences, enhancing the performance of both AlphaFold2 and RoseTTAFold. The code is released at \url{https://github.com/lezhang7/MSAGen}.

Main paper: 
 introduction: 
 The challenge of protein structure prediction (PSP), a pivotal issue in structural biology, has has experienced transformative progress due to the deep learning revolution. Among the advancements, AlphaFold2 (AF2) (Jumper et al., 2021) shines particularly bright, credited chiefly to its adept use of multiple sequence alignments (MSAs). MSAs, derived from querying a protein sequence against vast databases using search algorithms, represent aggregations of homologous sequences that capture evolutionary information, acting as the foundation of many PSP models. However, not all protein sequences possess a rich set of homologous counterparts. This scarcity often means that even advanced search algorithms struggle to construct high-quality MSAs, leading to a compromised efficacy for MSA-reliant models like AF2 (Jumper et al., 2021; Wang et al., 2022) as illustrated in fig. 1 . Inspired by the generative prowess of language models (Raffel et al., 2020; Touvron et al., 2023; Chung et al., 2022; Chen et al., 2021) , we see their potential beyond textual data. Through this lens, we liken protein sequences to text, aiming to innovatively generate virtual but constructive MSAs. These newly created alignments endow predictions with supplemental evolutionary information, amplifying the potential of protein structure predictions. In the domain of PSP, tertiary structure prediction emerges as a crutial challenge. It holds a central position in molecular biology, shedding light on intricate protein functions and interactions. And while secondary structure predictions (Rost & Sander, 1993) bring forth invaluable nuances, it's the tertiary predictions that offer a comprehensive view of a protein's intricate conformation.QueryFor bio-tasks such as protein structure prediction, gene sequence alignment, RNA secondary structure determination, microbial community analysis, and evolutionary tree construction, where enhancing downstream performance necessitates multiple sequences, we introduce the sequences-to-sequences (seqs2seqs) generation task. Unlike the conventional sequence-to-sequence (seq2seq) tasks-e.g., machine translation, which requires a strict one-to-one correspondence between a source sequence x and a target sequence y-seqs2seqs is designed for flexibility. The task aims to generate multiple coherent sequences from a given sequences. Each generated sequence preserves patterns from the input, but there's no strict correspondence between the input and output. Instead, we prioritize maintaining interconnected patterns across them. This design endows the task with a self-supervised nature due to its intrinsic adaptability.In the context of protein, such flexibility enables easy extraction of a portion of the MSA as the source, with the remainder acting as the target. Harnessing search algorithms, our framework adeptly extracts source and target data from comprehensive protein databases, paving the way for self-supervised pre-training. To our knowledge, this marks an initial step in tapping into self-supervised generative pretraining to bolster protein structure prediction accuracy.We introduce MSA-Generator, a protein language model pre-trained using the seqs2seqs as its pretext task. Specialized in simultaneously generating multiple sequences, it effectively captures global structural information from input MSAs. This approach facilitates rapid, de novo sequence creation, improves inferior MSAs (see fig. 1 ), and boasts adaptability across various protein domains, adeptly navigating computational challenges.To summarize the main contribution of this article:• Innovative unsupervised Seqs2Seqs Task Proposition: We propose the unsupervised sequences-to-sequences (seqs2seqs) task, a promising approach for generating informative protein sequences, with potential applications extending to other areas like RNA. Integrated with search algorithms, this task streamlines generative pre-training by automating data retrieval from expansive protein databases. • Launch of MSA-Generator Model: MSA-Generator, our state-of-the-art generative protein model, is uniquely devised to employ the seqs2seqs task on a self-curated dataset. It is optimized for multi-sequence generation, skillfully extracting global MSA insights, and has demonstrated adaptability across diverse protein domains. • Robust Empirical Validation: We validate our approach's potency, showcasing significant improvements on AlphaFold2 and RoseTTAFold using CASP14 and CASP15 benchmarks. This signifies a practical leap forward in tackling the intricate challenge of protein folding. It also showcases the promise of seqs2seqs pretraining in the field of bioinformatics. 
related work: 
 Protein Structure Prediction Proteins, while diverse, are built from a mere 20 unique amino acids. Their physical structure, which determines their function and attributes, is pivotal to grasping the essence of life. While the field has witnessed significant advancements like AF2 (Jumper et al., 2021) and RoseTTAFold (Baek et al., 2021) , challenges remain. The success of AF2 can be ascribed to its adept utilization of MSAs, constructed by search algorithms such as DeepMSA (Zhang et al., 2019 ), JackHMMER (Johnson et al., 2010) and MMseqs2 (Steinegger & Söding, 2017) across vast databases including UniRef (Suzek et al., 2007) and BFD based on a protein query sequence. Conversely, single-sequence prediction methodologies (Chowdhury et al., 2021; Lin et al., 2022; Chowdhury et al., 2022; Wu et al., 2022b) often underperform in comparison. However, for protein queries devoid of extensive family representations, obtaining quality MSAs is challenging. Consequently, the proficiency of MSA-driven techniques diminishes. In this context, our proposed method leverages generative techniques to combat the paucity of homologs in protein sequences, presenting a solution when traditional techniques falter.Protein Language Models Language models, initially designed for language processing, have found a expanding role in bioinformatics, primarily for protein sequence representation. This uplift in performance largely stems from the adaptation of the masked language modeling (MLM) strategy, a concept inspired by BERT (Devlin et al., 2019) . ProtTrans series (including ProtBert, ProtTXL, ProtXLNet, and ProtT5 (Elnaggar et al., 2021) ) and the ESM models like ESM-1b (Rives et al., 2021) and ESM-2 (Lin et al., 2022) epitomize the influence of Transformer architectures in this domain.Further solidifying this connection, research (Vig et al., 2020; Rao et al., 2020) has unveiled ties between protein representations and contact maps, highlighting evolutionary patterns vital to AF2's triumph. Such revelations have realigned research interests towards MSAs from single sequence.Here, the MSA Transformer (Rao et al., 2021) stands out, harnessing MLM specifically on MSAs.Protein Sequence Generation Beyond masked language modeling (MLM), a variety of generative techniques exist, each with its distinct objectives and methodologies. For instance, Potts models (Figliuzzi et al., 2018; Russ et al., 2020) are crafted specifically for individual MSA sets from which they're derived (Zhang et al., 2022) . However, their shortcomings in adapting to different MSA sets (Sgarbossa et al., 2022) have spurred the development of generative language models. An exemplar, ESMPair (Chen et al., 2022) , constructs MSAs of Interologs by classifying sequences based on their taxonomic lineage. In contrast, both ProGen (Madani et al., 2020) and ProGen2 (Nijkamp et al., 2022) focus on single-sequence generation, sidestepping the integral component of MSA. Another research direction involves VAE-based models (Riesselman et al., 2018; McGee et al., 2021; Sinai et al., 2017) , originally developed for mutation evaluation. The challenge of efficiently sampling from distributions to create diverse and long sequences limits their application to downstream tasks.A study by Sgarbossa et al. (2022) utilized the MSA Transformer in a repetitive mask-and-reveal methodology, which unfortunately led to a compromise in sequence diversity. Of notable mention is EvoGen (Zhang et al., 2022) , aiming parallelly at producing virtual MSAs for Protein Structure Prediction. However, EvoGen uniquely operates as a meta-generative model, requiring guidance from Alphafold2 to hone its MSA generation prowess.Despite the lengthy context associated with MSA Generation, our work connects self-supervised learning with MSA generation. We highlight generative MSA pretraining, introducing an unsupervised sequences-to-sequences task specifically tailored for efficient MSA generation. To the best of our understanding, this marks a significant stride in the realm of protein sequence generation. 
sequences-to-sequences generative pretraining: 
 We present the Sequences-to-Sequences generation task and the methodology for automatic dataset construction in section 3.1. Details of the proposed MSA-Generator are provided in section 3.2. In section 3.3, we delve into the ensemble approach of MSA-Generator for optimizing the Protein Structure Prediction (PSP) task.Query Protein SequenceUniCluster BFDK T Y N L N D … G N -S Y N L N D … G N K T Y D L N D … -- K S G N L S D … G N K S Y N L N D … G N K T Y D L S D … -- K S Y N L N D … -N K T Y D L S D … -- MSA Search & Pairing Algorithm K T Y N L N D … G N -S Y N L N D … G N K T Y D L N D … -- Source K S G N L S D … G N K S Y N L N D … G N K T Y D L S D … -- K S Y N L N D … -N K T Y D L S D … -- Target Shared co- evolutionary information Sequences-to-Sequences K T Y N L N D … G N Sequence-to-Sequence Source TargetHello, how are you?" Bonjour, comment ça va?strict one-to-one correspondence Recognizing the pivotal role of Multiple Sequence Alignments (MSA) and the challenge posed by sequences with sparse homologous matches in real-world databases, we present the Sequences-to-Sequences (seqs2seqs) Task tailored for synthesizing virtual MSAs. Unlike traditional sequence-tosequence framework often seen in machine translation task, which dictate a rigid one-to-one mapping between source sequence x and target sequence y, seqs2seqs adopts a more flexible approach. Rather than necessitating a direct match, the framework emphasizes that source sequences X and target sequences Y should exhibit shared intrinsic patterns. In the context of protein MSAs, this pattern alludes to co-evolutionary insights, necessitating the capturing and amalgamation of extensive evolutionary data across and within the input sequences, both horizontally and vertically.The inherent adaptability of the seqs2seqs model permits the self-supervised nature of the task and seamless gathering of substantial quantities of source and target sequences from protein sequence databases. This is achieved by deploying sequence searching algorithms like JackHMMER (Johnson et al., 2010) and MMseqs2 (Steinegger & Söding, 2017) . Our process began with selecting sequences from the UniRef90 database (Suzek et al., 2007) as initial queries. Subsequently, the JackHMMER algorithm (Johnson et al., 2010) was employed iteratively to identify homologous sequences within the database, based on the query sequences. This process was iterated until no additional sequences emerged, searching parameters are detailed in appendix C. For every batch of sequences retrieved, a random selection was made, designating query with some as the source X and the remainder as the target Y , as illustrated in fig. 2 . Notably, the assurance of co-evolutionary relationships is intrinsically facilitated by the search algorithm's mechanism. We pretrain a transformer-based model (Vaswani et al., 2017) , denoted as MSA-Generator, via the unsupervised Sequences-to-Sequences task. The MSA-Generator framework incorporates an encoderdecoder structure. The encoder contextualizes the input MSA data, while an auto-aggressive decoder produces sequences derived from this context (refer to section 3). To capture expansive evolutionary information from the input MSA both horizontally and vertically, the encoder integrates the tied-row and column attention mechanism (Rao et al., 2021) . As the decoder concurrently generates multiple sequences-interacting with each other and the input MSA-it is enhanced with two additional modules beyond the conventional transformer. The Cross-Row Attention is designed to efficiently acquire a global representation by amalgamating comprehensive states. Meanwhile, to emphasize the vital conservative trait of amino acids (Dayhoff et al., 1978; Henikoff & Henikoff, 1992; Jones et al., 1992) , we introduce the Cross-Column Attention, which, during the generation of the token at time step t, directs its attention to the t-th token of all input sequences.Tied-Row Attention Building upon the foundation laid by the MSA Transformer (Rao et al., 2021) , we incorporate a shared-attention mechanism. This is achieved by aggregating the attention map weights across each sequence from MSA ∈ R D×L prior to applying the softmax function. Notably, each sequence utilizes the same attention weight matrix. For the d-th row, the associated query, key, and value matrices are denoted as Q d , K d , and V d ∈ R L×h , respectively. These matrices are derived via three distinct learned projection matrices. The computation of the shared attention weight matrix is formulated as:W T R = softmax D d=1 Q d K T d λ(D, h) ∈ R L×L (1)In this context, λ(D, h) = √ Dh serves as the square-root normalization. This normalization is crucial in mitigating potential linear scaling of attention weights with the sequences. The resultant representation for the d-th row is obtained throughW T R V d .It's important to highlight that, in our decoder, we deliberately bypass the tied-row attention. This decision aids in maintaining diversity in the generated sequences. Instead, we lean towards a conventional self-attention mechanism.Cross-Row Attention Contrary to tasks like machine translation, where the target attends only to a single input during decoding, the essence of seqs2seqs lies in discerning intrisic patterns common to both source and target sequences. This necessitates a holistic comprehension of the input, implying that when generating a sequence, the decoder should attend to the entirety of the input. A naive concatenation would yield a representation with dimensions R D•L×h , rendering it computationally expensive and thus impractical.To address this, we introduce an efficient strategy that calculates the depth-wise average pooling of encoder hidden states H enc , represented asH c = 1 D D d=1 H d enc ∈ R L×h .This serves as a global representation of the input and is crucial for cross-attention during decoding. Here, K c = H c W k and V c = H c W v signify the key and value matrices, while Q = X dec W q stands for query matrix projected from decoder hidden states X dec . The Cross-Row attention is:CR-Attention(Q, K c , V c ) = softmax( QK T c √ h )V c (2)Each sequence generation process can access comprehensive information, attending to the same keys and values, mirroring the co-evolutionary patterns of the input MSA simultaneously thus permitting fast parallel generation of multiple sequences (middle at bottom in fig. 3 ).Self/Cross -Column Attention In MSA, each column represents residues or nucleotides at a specific position across sequences, revealing conserved regions essential for understanding biological functions, structural stability, or evolutionary significance (Dayhoff et al., 1978; Jones et al., 1992; Henikoff & Henikoff, 1992) . Drawing inspiration from the vertical-direction attention proposed in Ho et al. (2019) , we introduce a self-column attention in encoder for a comprehensive representation akin to Rao et al. (2021) , and a cross-column attention in decoder to capture conservation characteristics.To facilitate both attention mechanisms, the representation matrix X ∈ R D×L×h needs to be transposed prior to the execution of self and cross attention:EQUATIONFor the self-column attention, projections from X T yield Q col , K col , V col ∈ R L×D×h . In contrast, for the cross-column attention, Q col is determined from decoder hidden states as X T dec W q , whereas K col and V col are projected from encoder hidden states as H T enc (see fig. 3 bottom right).We employ the seqs2seqs task to pretrain MSA-Generator. For a given source MSA X ∈ R D×L , the loss is computed with respect to the target MSA Y ∈ R D ′ ×L as follows:EQUATIONIt's crucial to note that each sequence y ∈ Y is generated referencing the entire source matrix X, and this generation occurs in parallel owing to the thoughtful design of the architecture.Pretrained MSA-Generator adopts 12 transformer encoders/decoders with 260M parameters, 768 embedding size, and 12 heads. It's pretrained with ADAM-W at a 5e -5 rate, 0.01 linear warm-up, and square root decay for 200k steps on 8 A100 GPUs, batch size of 64, using a dataset containing 2M MSAs constructed as described in section 3.1.During inference, for every query sequence x, we initially use a search algorithm to assemble a MSA denoted as X. This is subsequently inputted into MSA-Generator to produce MSA Y . The concatenated MSA X ⊕ Y serves as input for the subsequent task. Nucleus sampling (Holtzman et al., 2019) , set with top-p=50 and top-k=10, is implemented to foster unique sequences and curtail redundancy.For the purpose of optimizing the PSP task and yielding informative sequences, we adopt pLDDT as our selection criterion, leveraging our ability for swift MSA generation. pLDDT (Kryshtafovych et al., 2019; Jumper et al., 2021) measures the accuracy of predicted inter-residue distances for each residue in a protein, serves as a confidence indicator, with elevated scores hinting at potentially more accurate predictions. Utilizing pLDDT, we enhance each MSA through multiple runs, computing corresponding pLDDT scores for each. The MSA with the premier pLDDT score is subsequently selected as the optimal ensemble result and employed to determine the prediction accuracy relative to the ground truth. 
setup: 
 The tertiary structure of a protein is pivotal, directly determining its functionality. In structural biology, the tertiary structure not only reveals the overall conformation of a protein but also inherently includes insights from secondary structures (Rao et al., 2021; Jones, 1999) , such as the arrangement and orientation of α-helices and β-sheets, and from contact predictions (Wang et al., 2017) , denoting the spatial interactions between amino acid pairs. In essence, the tertiary structure offers a holistic view, allowing direct inference of localized structural features and interactions between amino acids. Leveraging tools like AlphaFold2 lets us directly obtain this comprehensive structural data, thereby bypassing intermediary steps like secondary structure and contact prediction. Given the centrality of tertiary structure prediction in protein functional studies, we have prioritized this task and adopted Local Distance Difference Tests1 (LDDT) (Mariani et al., 2013) and pLDDT as our metric for evaluating prediction accuracy.Specifically, we assess MSA-Generator by comparing preotein tertiary structures predicted from PSP algorithm, namely AlphaFold2 and RoseTTAFold, with various input MSAs. Demonstrating the usefulness of generated MSAs and the efficacy of MSA-Generator.Benchmark & Dataset We employ CASP14/15 as our test set, a prestigious dataset that encompasses proteins from a broad spectrum of biological families. The creation of a vast protein structure prediction dataset is prohibitively expensive, and given that AF2 has already trained on all previously available structures, this dataset emerges as the best evaluation benchmark. It's important to highlight that sequences from CASP14/15 aren't part of our pretraining dataset (refer to appendix A for detailed explanation), and our evaluations precede the AF2 version updated with CASP14/15 information.Our primary interest lies in challenging protein sequences devoid of homologues, rendering traditional search algorithms ineffective. For every query in our test dataset, we use JackHMMER to search within UniRef90, which contains 70 million sequences, in order to gather related homologues. We define two scenarios: (1) artificial challenging MSAs , where we purposefully pick the top 15 homologues for each test set query as an artificial gold standard. From these, 5 homologues are further sampled as the artificial baseline, offering a synthetic challenge. (2) real-world challenging MSAs , which includes 20 sequences from test set, each with homologues less than 20, significantly challenge PSP algorithms. All assessments are executed in a zero-shot setting. We employ Artificial challenging MSAs to thoroughly compare our generated virtual MSAs with conventional searched real MSAs to demonstrate that virtual MSAs can closely approximate real ones in downstream tasks. Specifically, for every Baseline MSA, we deploy MSA-Generator to produce a Virtual MSA that poses the same depth of the Real MSA. For each MSA, we employ an ensemble of three runs using the strategy outlined in section 3.3. Refer to fig. 4 for the results.The fig. 4 (a) portrays that the LDDT distribution of the baseline suffers a sharp decline when reduced from 15 to 5 sequences. This underscores the importance of MSA quality for cutting-edge PSP algorithms. Yet, when supplemented by MSA-Generator's generative virtual MSA, the gap to the Real narrows considerably, reflecting a LDDT enhancement of 12.8. This underscores the effectiveness of our generated MSAs.A more granular observation in fig. 4 (b) illustrates that the majority of baseline data points are below the diagonal, while most Virtual points sit above it, some even considerably outpacing the Real. fig. 4 (c) illustrates the statistics on improvements in intervals. It's evident that our generative virtual MSAs effectively improve results for 72.8% of protein sequences (over Baseline). Remarkably, nearly half of the generated virtual MSAs even outperform the real searched MSAs. This emphasizes the importance of generative MSAs and the potential of the seqs2seqs task in uncovering co-evolutionary patterns within bio-sequences. Among them, there are even 6 generative virtual MSA that surpass real MSA by more than 30 LDDT, including most notable T1032-D1 (+46.02 LDDT), T1054-D1 (+46.8 LDDT), and T1070-D2 (+61.22 LDDT), suggesting that without generative virtual MSA, current PSA algorithm may fail on these queries. Comprehensive results for individual MSAs can be found in the table 5.Our ultimate goal is to produce high-quality MSAs for protein sequences with few homologues. Current search algorithms often fail to construct quality MSAs for these, making PSP algorithms similarly struggle with accurate predictions. The Real-world challenging MSAs evaluation is devised to test the efficacy of the seqs2seqs generative pretraining approach in addressing this challenge.For this, we curate sequences with fewer than 20 homologues using the search method detailed in section 4.1. For every identified MSA, we employ MSA-Generator to generate an MSA of identical depth across three independent runs. Subsequently, we measure the ensemble LDDT by inputting them to the PSP algorithm following section 3.3. For comparison, our benchmarks include the strong single-sequence folding technique, ESMFold (Lin et al., 2022) and OmegaFold (Wu et al., 2022a) ; we apply the same evaluation set up for the iterative unmasking strategy highlighted in (Sgarbossa et al., 2022) ; and generation with Potts models (Figliuzzi et al., 2018) . strategies. AF2 consistently outperforms RoseTTAFold in both metrics. Potts models, intriguingly, don't demonstrate a pronounced ability to produce effective MSAs. This is evidenced by their marginally reduced average performance in both metrics, echoing findings from (Sgarbossa et al., 2022; Rao et al., 2020) . While iterative unmasking with MSA Transformer (Sgarbossa et al., 2022) can generate usable MSAs in certain scenarios, the MSAs it produces are often less diverse due to the inherent unmasking process, thus limiting its enhancement potential.In contrast, our proposed method manifests marked improvements in both metrics across both models. This indicates that the MSAs generated by our technique are typically informative. Specifically, 75% of the MSAs were effectively enhanced, resulting in an average LDDT boost of 4.3 on CASP14 and 8.1 on CASP15. Remarkable LDDT enhancements were observed in T1093-D1 (with 3 homologous), moving from 45.5 to 70.77, and in T1113 (with 10 homologous), rising from 32.6 to 80.6. However, in specific MSAs, like T1094-D2 (7 homologous, pLDDT +2.8, LDDT -0.1), T1099-D1 (8 homologous, pLDDT +1.11, LDDT -0.5), and T1178 (15 homologous, pLDDT +2.2, LDDT -12.3), an elevation in pLDDT was offset by a decline in LDDT. This suggests that pLDDT might not always be a consistent indicator for selection strategy outlined in section 3.3. Detailed results for each individual MSA are available in the appendix DWe sought to examine the effectiveness of pLDDT as a criterion. As previously highlighted, for specific proteins, improvements in pLDDT do not necessarily correlate with increases in LDDT. To delve deeper, we conducted an experiment where LDDT was directly calculated for each enhanced MSA in section 4.3. We then selected the highest LDDT as the output, bypassing the use of pLDDT as an intermediary metric. The detailed results are displayed in appendix D.The disparity between pLDDT-based and LDDT-based predictions shown in fig. 5 suggests that pLDDT may not always be the best criterion. A noticeable gap exists between the two criteria (evident in the blue and red regions). For example, proteins T1064-D1 (depth=9) and T1122 (depth=1) exhibit significant gaps between scores chosen by LDDT versus pLDDT, highlighted in red boxes. Interestingly, for T1178 (depth=15), the highest pLDDT selection scores -12.3 against the baseline, while LDDT selection results in a +2.7. This implies that some generated MSAs, even with lower pLDDT scores, can enhance the LDDT. This indicates that our approach has untapped potential that could benefit from more nuanced selection criteria. The ideal situation would be for both predictions to produce the same significant improvement, as emphasized by the green boxes and visualized in fig. 5 (Right). Notably, MSA-Generator shows notable improvements for protein sequences with few homologs, emphasizing its utility in real-world protein folding challenges. 3 . While additional ensemble runs lead to improved pLDDT scores, gains in LDDT plateau after three runs, even with higher computational expenses. Hence, we chose three runs to optimize performance and efficiency. Furthermore, a higher augmentation factor does not always yield better results; the identical input MSA might lack new insights, and extra sequences risk introducing noise. 
conclusion: 
 We present an unsupervised seqs2seqs task, accompanied by an automated dataset construction pipeline, designed to pre-train MSA-Generator for simultaneous multi-sequence generation. Rigorous experimentation underscore the effectiveness, diversity, and conservation feature of generated virtual MSAs, amplifying the prowess of stalwarts like AlphaFold2 in scenarios where conventional methods come up short. Furthermore, our approach demonstrates generalization across a wide array of protein sequence families in a zero-shot fashion. Our findings highlight the immense promise of the unsupervised seqs2seqs task, pointing towards its prospective utility in a broader spectrum of bio-sequences, thereby amplifying its benefits. 
 
 Appendix: We have compiled a pretraining dataset containing 2M MSAs from four databases: Uniref90 v.2022 04 using pipeline discussed in section 3.1. Specifically, for each MSA we randomly select 10-30 sequences and query as source X and another 10-30 sequences as target Y .Our goal is to explore the variations in MSA sequences using MSA-Generator. Accordingly, we depict the MSA's colored distribution in We use JackHMMER to build pretraining MSA dataset. We adopt default parameters, including: -E = 10, -N=5, -Z=1000, -incE=0.01. OpenStructure is used for LDDT calculation https://openstructure.org/ 


Decision: We recommend acceptance as a poster:
* MSA-Generator can alleviate the shortage burden of MSA data in proteomics research
* Results on CASP14 and CASP 15 are promising
* Interesting insights are raised in the analysis
* Authors improved the paper during the rebuttal, with news results CAMEO and Orphan25

To authors: please make sure you include all reviewer comments in the final revision.

Reviews: review: paper_summary: paper_summary: The paper proposes a method to generate MSA sequences to provide more alignments of the MSA. MSA-Generator can increase the depth of the MSA input, and thus incorporate more information. MSA-Generator demonstrates its capacity to synthesize higher-quality MSA via experiments on the CASP dataset.
strength_weakness: strength_weakness: - There are some other datasets available for experiments. Have you considered conducting experiments on datasets like CAMEO?

- Figure 4 (c) shows the distribution of LDDT improvement. Can you provide a more precise clarification on Virtual MSAs' pros and cons compared with real MSAs?

- Will the MSA-Generator be helpful for individual protein sequences as the input? There are some works on single-sequence protein language models worth discussing in your related work section.

[1] ProteinBERT: a universal deep-learning model of protein sequence and function.
[2] Modeling aspects of the language of life through transfer-learning protein sequences.
[3] Modeling Protein Using Large-scale Pretrain Language Model.
questions: questions: - See above.
limitations: limitations: - Further experiments on CAMEO can be helpful.

- MSA_Generator's capacity on Individual sequence. Experiments are not necessary, maybe some case study or analysis.
paper_summary: paper_summary: This paper introduces a method for generating multiple sequence alignments (MSAs) using a self-supervised seq2seq task. By leveraging large-scale protein databases, this approach produces virtual, informative MSAs that enhance the performance of protein structure prediction models such as AlphaFold2 and RoseTTAFold. The improvements are especially noticeable for proteins without extensive homologous families, demonstrating that data augmentation techniques can also be effectively applied in the protein domain.
strength_weakness: strength_weakness: 1. As the main contribution of this work is generating more homologous sequences based on the original low-quality MSA, it would be valuable to investigate how the MSA generator improves structure prediction for orphan proteins.
2. What criteria are used to determine the quality of homologous sequences? Are they based on a small number of sequences, or do they consider homologous sequences with only sequence similarity but not structural similarity? Additionally, are the pre-trained datasets composed solely of high-quality homologous sequences?
3. Using MSA as input demands significant memory resources. What thresholds are set for the input source MSA and target MSA depth during training? Is there a relationship between the depth of the source MSA and the depth of the target MSA? For instance, is the depth of the source MSA greater than that of the target MSA?
4. I noticed that the tied row attention mechanism is employed in the encoder. To reduce memory usage, have you considered introducing this shared attention mechanism in the cross-row attention within the decoder?
questions: questions: See above.
limitations: limitations: Yes
paper_summary: paper_summary: This work introduces MSA-Generator to generate virtual, informative MSAs. The generated MSAs can advance protein structure prediction.
strength_weakness: strength_weakness: - The technology novelty is somewhat limited. The model architecture mainly follows the MSA transformer, and the training framework is similar to Seq2Seq.
- I notice that there are several related work that also studies the problem of generating MSA to advance protein structure prediction [1]. The differences and advantages should be discussed.
- The ablation studies are weak. More design choices should be verified.

[1] MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training
questions: questions: See weaknesses
limitations: limitations: None
paper_summary: paper_summary: The paper introduces MSA-Generator, a novel self-supervised generative protein language model designed to address the limitations in protein structure prediction due to shallow multiple sequence alignments (MSAs). This model, pre-trained on a sequences-to-sequences task using an automatically constructed dataset, incorporates protein-specific attention mechanisms. These features enable MSA-Generator to produce virtual, enriched MSAs from large-scale protein databases, improving the quality of MSAs particularly for proteins without extensive homologous families.
strength_weakness: strength_weakness: Generally the paper is well-written and performs adequate analysis on important benchmarks. There a few weaknesses that could be addressed to improve the work:

- Generated MSA improve performance on large-scale pretrained frameworks which are already pre-trained on MSA. Specifically, Alphafold2 has been found to greatly benefit from MSA sequence numbers as well as diversity. However, for traditional models like Potts/ co-evolution based statistic models, could generated MSA improve performance? This could showcase whether generated MSA could replace MSA as a representative input feature.

- JackHMMER is not a commonly used MSA tool and may show declined performances. Do authors consider using more advanced tools like hhblits and mmseqs2? Also what is the e-value of the generated MSA compared to MSA ?

- Can this method be used to generate a MSA from scratch rather than augment MSA? MSA computation bottleneck is a even more severe problem and generation could improve it.
questions: questions: See weakness.
limitations: limitations: Yes.

Comments: title: Thanks for the response
comment: Thanks for the rebuttal.

I don't argue that using Transformers lacks novelty. I think the contribution from the encoder to the encoder-decoder is not as significant as the authors claim, because both the encoder and decoder are based on the MSA-Transformer mechanism. Additionally, no ablation experiments were conducted to explore whether this is the optimal solution for MSA generation.

For the references I provided, I would like to express that a similar approach can be taken to first conduct small-scale experiments to ablate design choices before moving on to large-scale pre-training. This will ensure that the proposed model is solid.

After reading responses to other reviewers, I will maintain my score.
Thank you for the detailed responses. After reviewing them, I believe my original score remains appropriate for this work, and I will maintain it as is. Thank you again for your efforts!
Dear Reviewer,

Thank you once again for taking the time to review our paper. Could you please review our rebuttal to see if it has addressed your concerns at your earliest convenience? The discussion period ends in approximately 24 hours. If our response resolves your concerns, we kindly ask you to consider adjusting the scores.

Best regards,
The Authors
Dear Reviewer,

Thank you once again for taking the time to review our paper. Could you please review our rebuttal to see if it has addressed your concerns at your earliest convenience? The discussion period ends in approximately 24 hours. If our response resolves your concerns, we kindly ask you to consider adjusting the scores.

Best regards,
The Authors
1. **Novelty**

We respectfully disagree with the argument regarding the novelty of our work. There is a significant distinction between the encoder-only paradigm and the encoder-decoder paradigm. Our model's innovation lies in its specialized approach to generating Multiple Sequence Alignments (MSAs) for structural prediction—an approach that differs markedly from the objectives of the MSA Transformer. Additionally, we disagree with the notion that utilizing a transformer-based architecture lacks novelty. By that logic, seminal models like BERT, GPT-1, 2, 3, Vision Transformer, and their successors would also lack novelty due to their reliance on attention mechanisms. However, many successful works, such as the MSA Transformer and the ProGEN family, also leverage methodologies from the Transformer architecture. This has not diminished their impact or novelty .

2. **Comparison with MSAGPT**

We would first like to emphasize that, according to the review guidelines, content point 6, papers are not required to compare against **contemporaneous works appearing less than two months before the submission deadline**, let alone works published after the deadline. Nevertheless, we would like to address your concerns regarding MSAGPT.

MSAGPT is a strong follow-up to our work, with some minor differences. First, MSAGPT incorporates the latest techniques from the NLP community, such as RLHF and RoPE, using rejection sampling from AlphaFold2’s feedback and adopting DPO based on AlphaFold2 to further fine-tune the models. Second, MSAGPT is a larger model, with 3B parameters trained on 16M data, while our model is much smaller, with 260M parameters trained on 2M data. One significant difference lies in the architectural design, which we have highlighted as a key contribution of our work. MSAGPT directly adopts the decoder from Transformer to **generate sequences in a 1-dimensional manner**. This generation paradigm is highly inefficient for deep MSAs, resulting in overlong output sequences with a complexity of $O(M^2 \times L^2)$, where M is the depth and L is the length. In contrast, our decoder supports the generation of MSA in a **parallel manner**, significantly reducing the computational cost to $O(M \times L^2)$. Furthermore, MSAGPT heavily relies on AlphaFold2 during training, which could introduce bias and unintended data leakage. In contrast, our model operates independently of other models, resulting in a more straightforward training process and yielding more reliable results.

We would also like to underscore that **our work serves as the foundational basis for MSAGPT**, as indicated by **our model's role as an important baseline in their comparisons**. Therefore, we believe it is inappropriate to compare follow-up works while assessing our contributions.

3. **Training Details**

Our model was trained for 200k steps on 8 A100 GPUs, as detailed in line 202, with pretraining taking approximately 100 hours. While we appreciate the reference provided, it is important to note that the referenced work uses 250M sequences and is an encoder-only model, making it neither small-scale nor directly comparable to our work.
Thank you to the authors for the rebuttal. Unfortunately, I did not find your response very convincing. This makes me hesitant to increase my score. 

1. I do not believe that the proposed encoder-decoder model has significant novelty compared to the encoder-only MSA Transformer, as I did not notice substantial improvements in the decoder and pre-training loss.

2. I realize that MSAGPT was released after the deadline, but I think that comparing it during the rebuttal period would help me better understand the advantages of MSA-Generator.

3. It is essential to report the time spent on pre-training for the community. Besides, small-scale pre-training experiments also help in proposing a more solid model [1].

[1] Rives, Alexander, et al. "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences." Proceedings of the National Academy of Sciences 118.15 (2021): e2016239118.
We sincerely appreciate your feedback and thoughtful review. While we understand the suggestion to include experiments on different protein architectures, we respectfully request more detailed guidance on the specific additional experiments you would recommend.

Our work primarily aims to generate MSA for **protein structure prediction**, as highlighted in the paper’s title. While we recognize the value of exploring broader models requiring MSA, our focus has been on state-of-the-art methods, particularly those built upon deep learning techniques in this domain. We believe the current results sufficiently support our claims within the scope of protein structure prediction.
I thank the authors for their endeavor in preparing the rebuttal. The rebuttal largely solved my question 1 & 3. I choose to keep my score after reading the responses. I hope the authors can include more experiments on different protein architectures (pre-trained, trained from scratch model, and traditional potts models) that involve MSA for comprehensive analysis.