MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training
Authors: Bo Chen
Zhilei Bei
Xingyi Cheng
Pan Li
Jie Tang
Le Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Computational Biology
Protein Language Model
Protein Structure Prediction
MSA Generative Pre-Training
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a novel MSA generative pre-training framework to yield faithful and informative MSA to promote structure prediction accuracy in a low-MSA regime. Studies of transfer learning also show its great potential to benefit other protein tasks.

Abstract: Multiple Sequence Alignment (MSA) plays a pivotal role in unveiling the evolutionary trajectories of protein families. The accuracy of protein structure predictions is often compromised for protein sequences that lack sufficient homologous information to construct high-quality MSA. Although various methods have been proposed to generate high-quality MSA under these conditions, they fall short in comprehensively capturing the intricate co-evolutionary patterns within MSA or require guidance from external oracle models. Here we introduce MSAGPT, a novel approach to prompt protein structure predictions via MSA generative pre-training in a low-MSA regime. MSAGPT employs a simple yet effective 2D evolutionary positional encoding scheme to model the complex evolutionary patterns. Endowed by this, the flexible 1D MSA decoding framework facilitates zero- or few-shot learning. Moreover, we demonstrate leveraging the feedback from AlphaFold2 (AF2) can further enhance the model’s capacity via Rejective Fine-tuning (RFT) and Reinforcement Learning from AF2 Feedback (RLAF). Extensive experiments confirm the efficacy of MSAGPT in generating faithful and informative MSA (up to +8.5% TM-Score on few-shot scenarios). The transfer learning also demonstrates its great potential for the wide range of tasks resorting to the quality of MSA.

MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre- Pre-Training
Bo Chen -f, Zhilei Bei Xingyi Cheng", Pan Li .ie Tang', Le Song?:3
Tsinghua University 'BioMap Research MBZUAA
cb210mails tsinghua. edu.
Abstract
Multiple Sequence Alignment (MSA) plays a pivotal role in unveiling the evolution- ary trajectories of protein families. The accuracy of protein structure predictions is often compromised for protein sequences that lack sufficient homologous informa- tion to construct high-quality MSA. Although various methods have been proposed to generate virtual MSA under these conditions, they fall short in comprehensively capturing the intricate co-evolutionary patterns within MSA require guidance from external oracle models. Here we introduce MSAGPT, novel approach to prompt protein structure predictions via MSA generative pre-training the low- MSA regime. MSAGPT employs a simple yet effective 2D evolutionary positional encoding scheme to model the complex evolutionary patterns. Endowed by this, its flexible ID MSA decoding framework facilitates zero- or few-shot learning. More- over, we demonstrate that leveraging the feedback from AlphaFold2 can further enhance the model's capacity via Rejective Fine-tuning (RFT) and Reinforcement Learning from AF2 Feedback (RLAF). Extensive experiments confirm the efficacy of MSAGPT in generating faithful virtual MSA to enhance the structure prediction accuracy (up to +8.5% TM-Scorre on few-shot scenarios). The transfer learning capabilities also highlight its great potential for facilitating other protein tasks.
Introduction
The advent of deep learning has significantly propelled progress across various scientific domains, exemplified by breakthroughs such as AlphaFold series [1, 2] for accurate biomolecular interaction predictions, AlphaGeometry [3] for intricate geometry and mathematical reasoning to name few. Among these, AlphaFold2 (AF2) represents landmark within structural biology. achieving an in silico precision of approximately 90% atomic accuracy that rivals wet lab experiments on protein structure predictions (PSP). The remarkable success of AF2 can be attributed to its innovative end-to-end use of co-evolutionary information supported by Multiple Sequence Alignment (MSA). MSA aggregates homologous sequences from vast databases, providing a comprehensive overview of evolutionary trajectories, which is critical for accurately predicting protein structures [1, 2, 4]. An illustrative example in Figure ((a) showcases that the correlations analysis among amino acids (AAs) sites could reveal contacts or conservative regions in the folding structure. Unfortunately, not all proteins possess rich set of homologous counterparts. Statistical investigations reveal that approximately 20% metagenomic proteins [5] and around 1 11% proteins from eukaryotic and viral origins [6] are classified as "orphan" proteins. This presents significant challenge for MSA- search algorithms in constructing high-quality MSA, consequently impeding the performance of PSP models [2].
BC and ZB contributed equally. Work done while interned BioMap.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Conservation
Zero Shot
FowSSun
V
TM Score
59
RFT&DPO
F
Y
P
W H
L
VI
MSAGPT
YI G II VI N
Folding
FF
W
Contact
F
A
K
Y
K
W
II W
49-
AF2 MSA
MSA-Aug.
EvoGen
EvoDif
Co-evolmation
44
(a) A toy example of MSA.
(b) Overall performance comparisons.
Figure 1: (a) The illustration of MSA and (b) performance comparisons between MSAGPT and advanced baselines on three natural MSA-scarce benchmark.
Drawing on the impressive capabilities of large language models endowed either by the au- toencoding [7] the autoregressive language modeling regime [8. 9], protein language models (PLMs) have been developed to unveil the evolutionary patterns and sequence characteristics in- trinsic to protein structures. Specifically, generative PLMs [10, 11, 12], trained on vast protein databases [13, 14, 15, 16] have achieved unparalleled success in generating novel proteins with desired structural properties. These achievements underscore the efficacy of language models in identifying evolutionary patterns within individual protein sequences. Inspired by this, subsequent works [17, 18] attempt to further integrate MSA as the input or by directly generating virtual yet informative MSA [19, 20, 21] to provide additional evolutionary insights. These approaches usually adopt customized attentions that merely allow attention aggregated among specific directions, such as axial attention [22], for separately analyzing the row- and column- wiie co-evolutionary patterns in MSA. However, these attention mechanisms usually have low efficiency in capturing the evolutionary information in MSA, or even fail to adequately capture intricate co-evolutionary dynamics. Taking Figure ((a) as an example, it is imperative to concurrently analyze the pairwise or high-order rela- tionships of amino acid sites across all homologs to deduce the structural constraints influencing the folding structures, which may not achieved by customized attention. The limited capacity may result in compromised performance on the task that highly resorts to co-evolutionary information.
Built upon the insights mentioned above, we introduce MSAGPT, a simple yet effective framework that prompts protein structure prediction via MSA generative pre-training. This method facilitates de novo MSA generation, aiding in protein structure prediction scenarios with limited MSA available. MSAGPT is characterized by its unique features:
2D Evolutionary Positional Encoding. We employ innovative dual-axis positional encoding scheme that captures column- and row-wise co-evolutionary information concurrently, This method provides a comprehensive understanding of complex evolutionary relationships with high efficacy. enhancing the model's generative capabilities.
ID Zero-/Few-Shot MSA Decoding. With 2D positional encoding. MSAGPT re-formalizes MSA generation as a one-dimensional sequence generation task, optimized by the simple next-token- prediction objective. This enables MSAGPT to conduct zero- or few-shot MSA generation under flexible in-context learning framework.
Learning from AlphaFold2 Feedback. MSAGPT further utilizes feedback from AlphaFold2 to reduce hallucinations during MSA generation. This approach ensures the generation of reliable and informative MSA, thus enhancing protein structure prediction.
Extensive experiments conducted on three benchmarks, CAMEO [23], CASP, and PDB [14], demon- strate the superior capacity of MSAGPT in generating high-quality MSA. Notably, MSAGPT outper- forms existing MSA generation models on both zero- and few-shot scenarios. Impressively, AF2 with virtual MSA generated by MSAGPT significantly improves the structure prediction accuracy than that with natural MSA on cases with limited homologous information. Moreover, the subsequent Rejective Fine-tuning (RFT) and learning from AF2 feedback (RLAF) stage further enhance the model's 's ability to generate informative and faithful MSA, outperforming the original MSAGPT by a large margin, as shown in Figure 1(b). Additionally, we demonstrate that virtual MSA can also benefit other tasks.

---

Retrieved Lov-quality MSA
Poniuue /
MSA Generation MISAPT
FAPYEWIKES
Auguented Generated
LONNT
Query preten
No USA
Resomtaiie
Clenstema Dereder
SISGPT
LAGYGH
Zero Sbot
LAGY
FAPYPW
Few-sbtt
LINITO T
Igsel
C. [4
FARYEWIN
Protcin Databese
Podiiio
Paddiin
0 200000
212345
01134561
1111111
LK-Y-n n
LAGYCH
Indeiur MSA
3D Evolrionary Positional Encoding
ID Zero-Fewesbo MSA Decoding
Ground Truth
MSA-hhssed Protein Stucture Prediction Algonihm
Ground Structure
VS AlphaFold2 Predictions
Edppkeooht2, RescFT.toid, mic
VS
ALDDV-12.
6L.0
AA AlphaFoldZ Production (Generated MSAI
(Retrieved MSA)
Enhamced by
7M--Soor--21.5
FM Scorem
The overall framework of prompting protein structure predictions via MSA genera- tion. Left: The challenge faced by conventional search algorithms on protein with scarce homologous sequences, resulting in suboptimal alignments. Boddle-to-Right MSAGPT generates informative and high-quality MSA for such challenging queries, presenting promising approach to overcoming these limitations. [M] denotes the sequence separator. [S], [E] are the special tokens to represent the start or end of MSA generation.
Figure 2:
We expect MSAGPT to become integral in supplementing protein-related tasks requiring critical evo- lutionary information from MSA. The model is available at https//jttaa.cmm/HUUUMMMS.C.
2
Related work
Protein Structure Prediction. Proteins are fundamental to the various biological processes that sustain, grow, and protect living organisms. Groundbreaking deep learning approaches [1, 2, 4] have been developed to predict the folding structures based on their sequences. These methods have achieved structure prediction accuracy conventional wet-lab experiments. The success largely relies on the utilization of MSA, which are retrieved through search algorithms [24, 25, 26, 27] against vast databases [13, 14, 15, 16]. However, challenges arise with "orphan" protein sequences, which lack sufficient homologous sequences for accurate structure prediction. Single-sequence PSP methods [1 28, 29, 30] are designed to infer folding structures directly from the query protein sequences. Despite these advancements, the accuracy of predictions from single-sequence methodologies generally falls short of those derived from MSA-based algorithms.
Protein Language Models. Protein Language Models (PLMs), such as ESM [28, 31], ProGen [10, 32], etc [12, 33, 34] have emerged groundbreaking development in computational biology. PLMs are trained on single sequences, towards understanding protein structural features or enabling the generation of diverse and realistic protein sequences. MSA Transformer
[17] further incorporates MSA as the input, achieving better performance than these single sequence models, underscoring the importance of utilizing the evolutionary information from MSA [35, 36, 37]. To enhance MSA generation, MSA-Augmentor [20]. PoET [19] employ the seqs2seqs pre-training, which adopts the sequential axial attention mechanism to capture the evolutionary data across and within the input sequences, both horizontally and vertically. EvoGen [21], serving the meta generative model, aims producing virtual MSA for enhancing protein structure predictions. However, highly resorts to external structural prediction models to optimize its performance.
Aligning with Human Preferences. Aligning language models with human preferences has been shown to be effective in improving generation quality [8, 38, 39, 40]. Existing methods typically employ supervised fine-tuning using human-annotated datasets or reinforcement learning from human feedback pipelines [38, 39]. Inspired by these, we utilize the feedback from AlphaFold2 to further enhance the generation capability of the pre-trained model, which helps mitigate hallucinations and enables the model to generate accurate and reliable MSA.
3 Preliminary
Definition Multiple Sequence Alignment (MSA). Given the query protein sequence Q E A 1 where & denotes the set of alphabetic symbols used to represent the 20 basic amino acids and L
3

---

ie., Poli = (0, 1..... .L). and the second positional P.b. (0, 1, 1.. .N.. The two positional ids projected into two vectors added the input represents the number of amino acids per sequence, the MSA M E A NN of Q is comprised of N homologous protein sequences, which can be obtained either by searching over protein databases or generating with MSA generation methods.
Problem 1 Prompting Protein Structure Prediction by MSA Generation. Given Q with initial MSA Mint A AnXL as the prompt, where nt indicates the zero-shot MSA generation and signifies the few-shot MSA generation, we target at learning a function f to generate virtual MSA Mgen Amxt based on Q and Miin such that the structure prediction accuracy based on the augmented MSA Maug E A mx significantly surpasses that based on the initial MSA M
Maug f(2, Mmals
Nlaco (Q, MMuu) Lace Minis)
where the Nace is prediction accuracy comparing the prediction result of AF2 and the ground truth.
In this paper, we mainly focus on improving the structure prediction accuracy in the low-MSA regime, ie., the cases that lack sufficient number of homologous sequences.
4 Methodology
Given a query sequence and its retrieved natural MSA, we aim to comprehensively understand the co-evolutionary patterns MSA, such that we can generate informative virtual MSA for prompting protein structure prediction in the low-MSA regime. Conceptually, the co-evolutionary information is analogous to the covariance matrix mathematics, depicting the correlations among amino acids by comparing pairwise or high-order correlations among amino acid sites in MSA, as depicted in Figure ((a). To achieve this goal, MSAGPT contains two key adoptions, distinguishing it from existing MSA-based PLMs that rely customized attentions [2. 17, 20, 19]: 2D Evolutionary Positional Encoding. Introduces an adaptive dual-axis positional encoding scheme that captures column- and row-wise co-evolutionary information concurrently. And 1D Zero-/Few-Shot MSA Decoding. Re-formalizes MSA generation as a one-dimensional sequence generation task based on the proposed 2D positional encoding scheme, which enables MSAGPT to conduct zero- few-shot context learning MSA generation framework. The overall framework is illustrated in Figure 2.
4.1 2D Evolutionary Positional Encoding
Vanilla transformers typically use ID positional embeddings to incorporate absolute and relative positional information tokens. However, when dealing with MSA, which represents stacked homologs, the structure is different. Each row of MSA corresponds to distinct protein sequence, while each column represents the evolutionary trajectories of a specific amino acids (AAs) site. To effectively capture the evolutionary patterns, recent approaches [2. 17, 20] have employed decoupled axial attentions, which are designed to capture explicit co-evolutionary information along the rows (protein sequences) and columns (AAs sites). However, these methods often suffer from low efficiency in capturing the information dynamics or fail to capture the evolutionary information adequately.
In light of this, introduce novel two-dimensional evolutionary positional encoding scheme, illustrated in Figure 2. Given an MSA M E N'L, we define a 2D positional id matrix P E N2xNNL XL, where the first positional id Po e NIxNXL indicates the column position, P1 indicates the row position, i.e.,
token embeddings. We utilize the Rotary Positional Encoding (RoPE) [41] technique, specifically adapting its wwo-dimensional variant" to suit 2D positional encoding requirements.
Comparison with Axial Attentions. Considering the 2D positional id (Po, Pt), the self-attention among AAs (a, 3) meets the following unit patterns, illustrated in Figure 3:
pa p & PT + pf Indicates and 3 reside in the same site across dif- ferent protein sequences, such as the AA pair (A, K) and (P. G), enabling column- wise self-attention that highlights evolutionary patterns conserved across sequences. 'https//kkeueen htt.://wwwu..mmmrrciv////////

---

P * Po & P Suggests a and 3 are positioned in the same protein sequence but at different sites, such the AA pair (A, P) and (K, G), facilitat- ing row-wise self-attention that captures sequence-specific features.
Acial Attention Francwork
MSAGPT Evalutionay Entanced
Romasaisc
Cosurto
FAPYPW LKAYGH
PAPYPW LKAYCH
F FAPYPW LKAYGH 0012348 012345 R000000 IIIII
P
GKPA
Po * Po & Pr + P.P Denotes and B lack explicit correlation, such as the AA pair (A, G) and (P, K); may be serving as anchor nodes for complex co-evolutionary information diffusion.
Information Flow
Figure 3: Comparisons among the axial attention (ex- emplified by [17]) and the one in MSAGPT in single layer. Here we focus on the information aggregated to the AA "G". The 2D evolutionary position enhanced attention shows higher efficiency than the decoupled axial attentions with one-step aggregation attain sufficient information.
Conceptually, the 2D positional encod- ing encapsulates the explicit row- and column- wise self-attention patterns with high efficacy. Moreover, it allows unrestricted information diffusion, that is, enabling any two amino acids to attend to one another. Such framework facilitates unveiling complex co-evolutionary patterns, such as high-order correlations among AAs, that customized self-attentions might overlook.
4.2 1D Zero-/Few-Shot MSA Decoding
Leveraging the 2D evolutionary positional encoding, we further release the stacked MSA decoding task into the scalable ID sequence generation framework, without compromising the integrity of co-evolutionary information. Specifically, we convert the MSA M E A N.I. into the flatted ID sequence M E A IXNNT. along the row axis to ensure that during inference. Similarly, the 2D positional matrix P E N2x is reshaped into flattened format, p NIX2xNL, This allows the model conduct simple auto-regressive generation process, as illustrated in Figure 2.
generate the MSA sequentially
Discussions. Admittedly, introducing 2D positional encoding introduces higher time complexity in comparison to conventional customized attention mechanisms (from O(N + O(NT?) to ((22 However, it is worth noting that the original stacked nature of MSA poses challenges for integrating it with acceleration techniques used in large language models, such as Flash Attention [42, 43]. The ID decoding framework, conversely, can be easily scaled accommodate in-context learning frameworks, such as retrieval augmented generation, to further enhance the model's generation capability and expand its application range. From a practical standpoint, the high parallelism of the ID decoding framework demonstrates superior inference speed, benefiting from techniques like Flash Attention and KV-cache, while incurring negligible memory overhead compared to customized attention mechanisms. For further details, please refer to Appendix Section A.4.
5 The Training Pipeline of MSAGPT
The training pipeline involves three successive stages: Stage 1: MSA Generative Pre- Fre-Training to obtain the base MSA generation model; Stage 2: Rejective Fine-tuning (RFT) to instruct the base model with high-quality MSAs via AF2 annotations, which reduce generation hallucinations Stage 3: Reinforcement Learning from AlphaFold2 Feedback (RLAF) to further enhance RFT model's 's capabilities based on the feedback of AF2. (See Appendix Section A for training details.)
5.1 Stage 1: MSA Generative Pre-Training
Pre-Training Dataset. We utilize the Uniclust30 MSA dataset from OpenProteinSet [44], which is processed through an all-against-all search on Uniclust30 [45] using HHblits [46]. This results in approximately 16 million MSAs (See Appendix A.1 for Details).
Pre-training Objective. We adapt the language modeling objective [47] to the MSA generation task. The cross-entropy loss for modeling the intrinsic distribution of MSA M E A IXNL defined as:
I.
Lee = EMS
Σ
log (MM M
5

---

where M/ E 1 is ID flatted version of the input MSA and is the learned parameter.
5.2 Stage 2: Rejective Fine-tuning (RFT)
Noted that the pre-trained dataset inevitably contains noisy co-evolutionary patterns, such as large portions of deletions and insertions, which may mislead the base model to yield hallucinated cases, i... the linguistically reasonable but intrinsically unfaithful MSA. Thus we select further fine-ture tune-tune the base model via a rejective sampling procedure based on the AF2-annotation.
MSAs to
RFT Dataset. We collect 120,780 protein sequences with structures from Protein Data Bank (PDB) [14]. For the sequence Q, we search its MSA M E A Nx from UniClust30 [45] with .... m;} with replacement, where the sampled data, we filter out the MSA with depth N fewer than [n i/2]. Subsequently, we employ AF2 to score the sampled subset using the structure prediction accuracy T Tace( (Q, mz). Then the RFT dataset DRET defined as:
HHblits [46]. Then we sample several MSA subsets m {m1. m.,....
m, A xx and n < N. To assure the information density
DRFT ((Q, m. (T) m²)) > 0, (Taee m²) Maaco (Q; -)) 02]
where laae (Q. (Q.
indicates the prediction accuracy without using MSAs. We set the sampling number i 10, the depth of sampled MSA n 16, 0.9, and 02 0.2, which results in approximately 60k samples. The base model is fine-tuned on DRFT with the same pre-training objective.
5.3 Stage 3: Reinforcement Learning from AlphaFold2 Feedback (RLAF)
We further employ AF2 as the reward model to perform the Reinforcement Learning with AF2 Feedback (RLAF) using Direct Preference Optimization [38] (DPO) to further guide the RFT model decode meaningful structure-related MSA patterns that align with the preference of AF2.
RLAF Preference Dataset. For each query Q from the PDB, we use the RFT model to generate its MSA M E A NxL in zero-shot manner. Then, we also sample several MSA subsets m
{Qlt), mm m J k=1
(k)
K
{m1, m2, ... mi} and obtain the preference dataset DDPO
as follows,
DpPO {(Q, mw: mi)l (Tace (Q, mmw) Taco (Q, m2) 0g)
where we set the 02- = 0.3, rendering the number preference data Doro = 11k.
RLAF Training Objective. The adapted DPO loss is defined as:
( (Q) log (m(il11) Teer (m Q)
LIPO Dij -log log
log Trer(m (Q)
where Te and Nee are initialized by the RFT model and Jet is frozen while Ty is optimized. During the RLAF training phase, we found that merely using the DPO loss led to training instability. Thus adopt the pre-training loss Loe for the chosen answer my as a regularization term with the coefficient factor X in the total loss mitigate this issue. The total loss == T.PO + XLcr. 0.1. Another critical coefficient B, which measures the penalty intensity for incorrect answers set to B = 0.1.
Experiments
6.1 Setup
Benchmarked Dataset. We employ the datasets from CAMEO [23], CASP14&15, and PDB [14], which are esteemed benchmarks protein structure analysis spanning a diverse array of biological protein families. For each protein sequence, we search its MSA on UniClust30 database [45] using HHHbltt [46]. Given our focus on addressing the challenge presented by cases with limited MSA information, we establish two specific benchmarks to represent the MSA-scarce conditions:

---

Table 1:
The performance of structure prediction on three natural MSA-scarce benchmarks. avg. Depth represents the average depth of searched MSA across all query sequences. Compared with the base model, the RFT and DPO models achieve higher TM-Score while with lower pLDDT values. (See Appendix Table 5 for more results.)
CAMEO (avg. Depth 8.5)
CASP
PDB
(avg. Depth = 4.6)
(avg. Depth 2.6)
Model
Zero-Shot
Few-Shot
Zero-Shot
Few-Shot
Zero-Shot
Few-Shot
PLDDT TM
pLDDT TM
pLDDT TM
pLDDT TM
pLDDT TM
pLDDT TM
AF2 MSA
63.8
55.4
77.4
71.4
44.0
32.6
54.2
44.1
55.2
45.6
61.0
52.3
EvoDiff
68.0
59.1
70.1
60.1
45.0
29.5
50.6
38.1
54.8
45.2
59.1
47.2
MSA-Aug.
67.7
59.2
77.4
72.1
56.8
36.6
63.4
46.3
61.9
49.8
66,0
55.3
EvoGen
66.1
60.3
78.6
75.3
48.2
38.4
55.1
48.5
57.6
49.5
62.8
55.4
MSAGPT
70.8
61.4
80.8
75.2
59.0
39.8
65.4
51.0
68.6
53.4
71.3
59.6
+ RFT
68.0
60.5
79.8
76.4
56.8
40.2
64.0
53.6
66.8
53.4
70.3
60.1
68.9
62.7
80.2
76.7
54.2
43.7
62.7
57.0
64.5
53,6
68.0
59.7
+ DPO
(43.1) (42.4)
(+2.2)
(+1.4)
(+2.2) (+5.3)
(+2.0)
(+8.5)
(+6.7)
(+3.8)
(45.3) (+4.7)
Table Zero-shot evaluation on artificial MSA- scarce benchmark (GDT stands for GDT-TS).
Table
Evaluation of selection methods.
Model
CAMEO
CASP
PDB
Model
рTM
pLDDT TM
GDT
LDDT
MSAGPT-DPO
76.7
57.0
59.7
PLDDT Selection
77.5
57.6
60.5
AF2 MSA
28.9
69.6
42.0
62,4
67.0
EvoDiff
34.7
70.8
44.0
64.1
69.9
Table 4: Evaluation on transfer learning.
MSA-Aug.
33.2
74.3
45.8
65.9
71.7
EvoGen
31.7
71.6
44.6
64.6
70.5
Model
|
CtP
SsP
LocP
MIB
MSAGPT
37.9
80.4
50.7
70.7
76.1
w/o Virtual MSA
11.6
66.5
58.3
57.5
+RFT DPO
37.9
79.7
51.6
71.4
76.9
w/ Virtual MSA
13.1
69.0
56.4
60.3
Natural MSA-scarce Benchmark. We identify 200 protein sequences with the number of searched MSA fewer than 20 (8 from cameo, 13 from CASP14& 15, 179 from PDB).
Artificial MSA-scarce Benchmark. We collect approximately 8k protein sequences based on the PDB released before 2024-01-22 without searching MSA.
All MSA from the test set are removed from the pre-train dataset.
Baselines. To assess the performance of MSAGPT, adopt AF2 the benchmark MSA-based PSP algorithm. For MSA generation baselines, we compare MSAGPT, its RFT-version and its DPO-version with two advanced MSA generation algorithms: EvoDiff [34], which utilizes the diffusion framework for controllable protein sequence generation. Specifically, we use the EvoDiff- MSA for the MSA generation. MSA-Augmentor [20], which utilizes pre-training architecture incorporating an encoder and a decoder based on the axial attention [17] and EvoGen [21], which employs a meta generative model framework with customized attention, leveraging guidance from AF2 to refine its MSA generation. As PoET [19] is designed for mutational scanning tasks, we don't take as the baseline. Additionally, we include the reference model AF2 MSA, which utilizes all the searched natural MSA for prediction.
sesuuecce-to-sequenee
MSA Generation Pipeline. Given that MSAGPT can perform flexible zero- or few-shot MSA generation to accommodate different levels of available evolutionary information, we define two generation settings to evaluate models' performances under varying conditions:
Zero-Shot Generation. MSA generation conducted using only the query sequence as input, emphasizing the model's ability to infer necessary evolutionary patterns without additional contexts.
Few-Shot Generation. All the searched natural MSA are viewed as the prompt to inform the few-shot MSA generation process. Then the generated MSA, combined with the initial prompts, serves as augmented data for structure predictions.
7

---

60
TV-Scare
55-
5.5
5TA DV
The ScoSoore
50-
50
45
Zem-Shal Fon-Stul
45
pLaoT TN
16
20
24
1D 2nd 1D 1st
Model Variams
(a) Effects of MSA Depths
(b) Effects of Selection Methods
Figure 5: Ablation study with positional embedding variants.
Figure 4: The effect of different MSA depths and selection methods. The X-axis indicates the different MSA depths. The Y-axis represents the TM-Score. The dashed line denotes the non-selection baseline.
Evaluation Metric. We use several widely-used metrics assess structural similarity between predicted structures and ground truth: TM-Score, GDT-TS, and LDDT. Additionally, we include PTM and PLDDT, the corresponding predicted metrics estimated by AF2. All metrics are normalized from 0 to 100 for comparison, with higher scores indicating higher confidence and usually a more accurate prediction (see Appendix Section B for details).
6.2 MSAGPT's Virtual MSAs Reflect the Co-evolutionary Information
Table and 2 showcase the comparative results in two benchmarks across different baselines. Notably, AF2 MSA, which relies solely on the limited searched MSA without incorporating virtual MSA, exhibits the worst performance. Predictions enhanced with MSA generated by MSA-Augmentor or EvoGen surpass the performance of AF2 MSA. This underscores the critical role of high-quality MSA in enhancing the accuracy of cutting-edge PSP algorithms. Overall, MSAGPT surpasses other advanced baselines by a large margin on both benchmarks, specifically achieving +1 1.4% improvement on CAMEO, +8.5% CASP, and +4.7% PDB, measured by TM-Score, on the natural MSA- scarce benchmark. This significant improvement demonstrates not only the superior accuracy and effectiveness of MSAGPT but also its robustness handling cases with noisy or low-quality MSA.
Moreover, compared with the base model, the RFT and DPO models achieve higher golden metric scores, that is, GDT, LDDT, and TM--ccrre, but with lower predictive score, that is, the value of PTM and PLDDT. This discrepancy might arise from the presence highly confident (according to PTM and pLDDT) but lower-scored decoys (according to TM-Scorre), observed in [21], indicating that aligning with the preference dataset, which is filtered based on TM-Score, makes the model more inclined to generate truly informative MSA rather than hallucinated ones.
Statistically, MSAGPT effectively improves the prediction accuracy for 91.0% and 88.9% of protein sequences with limited MSA when compared to AF2 MSA Zero-Shur and Few-shot scenarios, respectively. This significant finding highlights the potential our MSAGPT framework to uncover and leverage co-evolutionary patterns within biosequences. Notably, also discuss the scenario with abundant natural MSA in the Appendix Section B.2.
6.3 Rethinking the MSA Selection Strategy
We further study the effect of different depths of virtual MSA, as shown Figure 4(a). We observe trend where the relative improvement in structure prediction accuracy decreases as the depth of virtual MSA increases. The accuracy based on MSA with 64 MSA sequences even underperforms those based on only 16 or 32 sequences. We hypothesize that increasing the number of virtual MSA beyond certain threshold may introduce dilution effect, where the density of valuable co-evolutionary signals is compromised by the inclusion of the hallucinated generation noise. To alleviate this, we explore MSA selection strategies for filtering out low-quality, noise-inducing sequences while retaining those that contribute positively to the accuracy of structure predictions, illustrated in Figure 4(b) (See Appendix Section for details).
ID Sequence Similarity or Diversity Measure. We first arrange MSA by their similarity to the query sequence in descending order. The results reveal that prioritizing MSA based on their high similarity to the query, termed static similarity (STA-SIM), does not improve prediction

---

accuracy compared to the non-selection approach (N/A). On the contrary, the static diversity (STA- DIV) strategy, which favors MSA with lower similarity rankings, slightly outperforms the baseline, highlighting the importance of sequence diversity in enhancing MSA quality. Moreover, we employ the dynamic approach, initially selecting the most (or least) similar MSA to the query sequence and progressively incorporating additional MSA based on their average similarity to the cumulatively selected set, termed as dynamic similarity (DYN-SIM) and dynamic diversity (DYN-DIV).
The results further confirm the advantage of fostering diversity within MSA rather than selecting only the sequences with high similarities to the query sequence. We also inspect the effectiveness of the widely-adopted MSA trimming (TRMM) strategy [21], which yields a similar TM-Score to the non-selection baseline, undermining its efficacy in selecting MSA with high quality.
3D Structure Affinity Measure, We assume that the generated sequence with high quality should exhibit structural congruity with the query sequence, thereby emitting strong co-evolutionary signals, To validate this, we rank sequences within MSA by their predicted tertiary structures according to the pTM,, a predicted TM score [2], pI.DIT, and TM-Score, from highest to lowest. These approaches, especially when guided by the PLDDT score, consistently select high-quality MSA, evidenced by the enhanced TM-Score. We compare the non-selection methods (N/A) and PLDDT selection methods on the three benchmarked datasets on few-shot generation scenarios in Table 3. This confirms our hypothesis that structural similarity plays a crucial role in effective MSA selections.
6.4 Transfer Learning of MSAGPT
Since protein structures largely dictate their functions, the virtual MSA, enhancing structure prediction, should similarly benefit other protein tasks. To validate this, we focus on two protein structural tasks: Contact Prediction (CtP) and Secondary Structural Prediction (SsP) and two protein functional tasks: Localization Prediction (LocP) and Metal Ion Binding (MIB) [11]. We sample 1,000 sequences from each benchmark and conduct 5-fold cross- cross-validatte validation (See Appendix Section B.3 for details).
Results. Table demonstrates that incorporating MSA from MSAGPT consistently surpasses merely using the single sequence on most tasks. Yet, it achieves inferior performance on the LocP task, which agrees with the observation [48] that protein language models may not present scaling behavior on several protein functional or property prediction tasks. Nevertheless, the results show the great potential of MSAGPT to contribute to wide range of tasks with generated MSA. We are motivated to explore additional transfer tasks to assess MSAGPT's utility across various domains further.
6.5 Ablation Study
To understand the effect of various positional encoding strategies on capturing co-evolutionary patterns, we design four model variants: 1D 1D_gpt: Adopts the standard GPT positional encoding; 1D_2nd: Utilizes only the second-dimensionall of the 2D evolutionary positional encoding mechanism; ID 1st: Utilizes the dist-dimensional positional encoding: 2D full: Implements the 2D evolutionary positional encoding mechanism (See Appendix Section B for details).
Results. Figure 5 showcases the TM-score distribution across different model variants. The ID_ gpt exhibits the lowest performance, attributed to its simplistic approach of treating the MSA as concatenation of homologous sequences, thereby failing to discern any co-evolutionary patterns. Both the ID_Ist and ID_ __2nnd demonstrate significant improvement over ID gpt, by explicitly encoding column- or row-wise relationships within the MSA, respectively. Notably, the performance of 1D_ 1st is better than that of ID> 22_2nd, suggesting that column-wise covariance patterns play a more crucial role in structural predictions than row-wise patterns. This aligns with the understanding that the permutation of sequence order does not alter the covariance information among residue sites [17]. Remarkably, the 2D_ fullll variant, which incorporates the proposed 2D evolutionary positional encoding, outperforms all other models, which underscores its effectiveness in capturing the intricate evolutionary information present in MSA.
7 Limitations
In this section, we discuss some limitations that should be resolved in future work.

---

Scaling behavior of MSAGPT. While we have showcased the effectiveness of MSAGPT in gen- erating informative virtual MSA, is important note that our pre-training was conducted with a model containing 2.8 billion parameters. The performance and behavior of MSAGPT, when scaled concerning dataset size, model size, and total compute resources, remain unknown.
Transfer Learning a wide range of tasks. While we have demonstrated the transferability of MSAGPT on several tasks, including protein structure prediction and protein function prediction, its performance on a broader range of tasks remains an open question. The ability of model to transfer its learned knowledge and adapt to new tasks is a critical aspect of transfer learning. While MSAGPT has shown promising results the tasks it was evaluated on, it is important to assess its performance on a more diverse set of tasks spanning various domains and problem types.
Border Impact
The aim of this paper is to improve the accuracy of protein structure prediction in cases with limited homologous sequences. The generated MSA also shows great potential to transfer to other protein- related tasks. By leveraging the information encoded in the generated MSAs, it is possible to enhance the performance of various protein-related tasks beyond structure prediction. However, the generative MSA may be misused to contaminate the high-quality nature MSA databases. Thus, it is necessary to train a classifier to distinguish the real from MSSAGPT-generatee MSA.
Conclusion
This paper introduces MSAGPT, a novel approach that prompts protein structure prediction via MSA generative pre-training, to enable accurate protein structure predictions situations where co-evolutionary information is scarce. To meticulously characterize the co-evolutionary patterns within MSA, MSAGPT designs two innovative techniques: the 2D Evolutionary Positional Encoding scheme and the ID Zero-FFew-Shot MSA Decoding mechanisms. The post-alignment learning from AlphaFold2 feedback further enhances the quality of MSA generation. Empirical experiments conducted on variety of benchmarks have demonstrated MSAGPT's robustness and effectiveness. In the future, we plan to apply MSAGPT to broader areas, particularly for tasks that heavily rely on co-evolutionary information, and investigate the aforementioned limitations.
Acknowledgments. This work has been supported by the NSFC for Distinguished Young Scholar 62425601, New Cornerstone Science Foundation through the XPLORER PRIZE and Tsinghua University Initiative Scientific Research Program.
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 2021.
[3] Tricu H Trinh, Yuhuai Wu, Quoe Le. He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 62567995):476-482 2024.
[4] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871-778 2021.
[5] William R Pearson.
An introduction to sequence similarity ("homology") searching. Current protocols in bioinformatics, 42(1):3-1, 2013.
[6] Nelson Perdigão, Julian Heinrich, Christian Stolte, Kenneth Sabír, Michael Buckley, Bruce Tabor, Beth Signal, Brian S Gloss, Christopher J Hammang, Burkhard Rost, et al. Unexpected
10

---

features of the dark proteome. Proceedings of the National Academy of Sciences, (12(52)::5888 15898 15903, 2015.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180004805 2018.
Pre-training of
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303./2712: 2023.
(9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini (.XXi::3322.19977 2023.
[10] Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell Systems, 14(1 14(11):968-97) 2023.
[11] Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, al. xtrimopgim: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401.06/999 2024.
[12] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 a deep unsupervised language model for protein design. Nature communications, 13(1 (51444438, 2022.
[13] Baris E Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 23(1)):128- 1288, 2007.
[14] Helen M Berman, John Westbrook, Zukang Feng. Gary Gilliland, Talapady Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1):235-242, 2000.
[15] Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nature methods, 16(7):603-606, 2019.
[16] Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 91)):5542, 2018
[17] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pages 1844-8856. PMLR, 2021.
[18] He Zhang, Fusong Ju, Jianwei Zhu, Liang He, Bin Shao, Nanning Zheng, and Tie- Liu. Co-evolution transformer for protein contact prediction. Advances in Neural Information Processing Systems, 34:14222-14266 2021.
[19] Timothy Truong Jr and Tristan Bepler. Poet: generative model of protein families as sequences-of-seqqeeness Advances in Neural Information Processing Systems, 36, 2024.
[20] Le Zhang, Jiayang Chen, Tao Shen, Yu Li, and Siqi Sun. Enhancing the protein tertiary structure prediction by multiple sequence alignment generation. arXiv preprint N.Xi::2306.08224, 2023.
[21] Jun Zhang, Sirui Liu, Mengyun Chen, Haotian Chu, Min Wang. Zidong Wang, Jialiang Yu, Ningxi Ni, Fan Yu, Dechin Chen, et al. Unsupervisedly prompting alphafold2 for accurate few- shot protein structure prediction. Journal Chemical Theory and Computation, 19622):8460- 8471, 2023.
[22] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 1011:119...21180, 2019.

---

[23] Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous automated model evaluation (cameo) complementing the critical assessment of structure prediction in Proteins: Structure, Function, and Bioinformatics, 86:387-398, 2018
casp12.
[24] Chengxin Zhang, Wei Zheng. SM Mortuza, Yang Li, and Yang Zhang. Deepmsa: constructing deep multiple sequence alignment to improve contact prediction and fold-recognition for distant- homology proteins. Bioinformatics, 36(7):2105-2) 12, 2020.
[25] Wei Zheng, Qiqige Wuyun, Yang Li, Chengxin Zhang, P Lydia Freddolino, and Yang Zhang. Improving deep learning protein monomer and complex structure prediction using deepmsa2 with huge metagenomics data. Nature Methods, pages 1-11, 2024.
[26] L Steven Johnson, Sean R Eddy, and Elon Portugaly. Hidden markov model speed heuristic and iterative hmm search procedure. BMC bioinformatics, 11:1-8, 2010.
[27] Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(1 12266:1006-1028, 2017.
[28] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637: 2023.
[29] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang. Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pages 2022-07, 07, 2022.
[30] Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M Church, al. Single- sequence protein structure prediction using a language model and deep learning. Nature Biotechnology, 40(11):1617-162) 2022.
[31] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (10115):2006233113 2021.
[32] Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. pages 1-8, 2023.
[33] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self sulf-super learning. IEEE transactions on pattemn analysis and machine intelligence, 44(10):7112-113 2021.
[34] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X Lu, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. BioRxiv, pages 2023-09, 2023.
[35] Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883);11-5, 2021.
[36] Bo Chen, Ziwei Xie, Jiezhong Qiu, Zhaofeng Ye, Jinbo Xu, and Jie Tang. Improved the het- erodimer protein complex prediction with protein language models. Briefings in Bioinformatics, 24(4)bbbad221, 2023.
[37] Pascal Sturmfels, Jesse Vig, Ali Madani, and Nazneen Fatema Rajani. Profile predic- tion: An alignment- based pre-training task for protein sequence models. arXiv preprint arXiv:2012.00195. 2020.
12

---

[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D MMnninn, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprini arXiv:2305. 18290, 2023.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06347 2017.
[40] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 18005, 2023.
[41] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiig, 568:127063, 2024.
[42] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
[43] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08661, 2023.
[44] Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Lukas Jarosch, Dan Berenberg, Ian Fisk, Andrew Watkins, Stephen Ra, Richard Bonneau, and Mohammed AlQuraishi. Openproteinset: Training data for structural biology scale. Advances in Neural Information Processing Systems, 36, 2024.
[45] Milot Mirdita, Lars Von Den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic acids research, (5(DI):D170DDII 2017.
[46] Martin Steinegger, Markus Meier, Milot Mirdita, Harald Vöhringer, Stephan J Haunsberger, and Johannes Söding. Hh-suite3 for fast remote homology detection and deep protein annotation. BMC bioinformatics, 20(1):1-15, 2019.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[48] Sanncesca-Zhouaat Li, Ava P Amini, Yisong Yue, Kevin K Yang, and Alex X Lu. Feature reuse and scaling: Understanding transfer learning with protein language models. bioRxiv, pages 2024-02, 2024.
[49] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:187771991,, 2020.
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
13

---

Appendix
Table of Contents
Training Settings and Hyper-parameter Studies.
A.1
Pre-Training
A.2
RFT
15
A.3
RLAF
16
A.4
Inference Efficiency
17
B Experimental Settings
18
B.I
Evaluation Details
18
B.2 Experiments MSA-abundant Scenario
................................
18
B.3 Setup of Transferability of MSAGPT Other Tasks
18
B.4 Setup of Ablation Study
18
C Selection Strategy Details and PLDDT Evaluation
19
Protein Structure Prediction Analysis compared with natural MSA
E Protein Structure Prediction Improvement after DPO
20
A Training Settings and Hyper-parameter Studies.
The overall training pipeline is illustrated in Figure 6.
Preference Data Construction
Preference Data Construction
MSAGPT (BASE)
MSAGPT (SPT)
MSAGPT (DPP))
Preference Datu Construction
Ground Truth Struciure
Query Protcia
LAGY...H
Sab sample
0.9
LKWY AH
Hemolegs
UKW .AH
LAGY...8H
Sampling
Sub
0.3
Filtering Score
Outpet
PAPY..PN
SubsampleN
AF2 0.6
LXTV...TII
Hemologs
Sampling
Figure 6: The overall training pipeline and the illustration of preference dataset construction process for SFT and DPO learning stages.
A.1 Pre-Training
To obtain high-quality MSA, we first screen out clusters with sequence lengths from 25 to 2000, and only retain sequences with the minimum identity of 30% and the largest proportion of gap tokens no more than 10%. The clusters with more than 10 sequences are left. As the MSA obtained by MSA search tools, inevitably contain noisy co-evolutionary patterns, such large portions of deletions.
14

---

Table 5: benchmarks.
The performance (pTM, GDT, LDDT) of structure prediction on three natural MSArscarce
CAMEO
CASP (avg. Depth = 4.6)
PDB
(avg. Depth = 8.5)
(avg. Depth = 2.6)
Model
Zero-Shot
Few-Shot
Zero-Shot
Pew-Shot
Zero-Shot
Few-Shot
1PTM
GDT
LDD[ pTM
GDT
LDD[
pTM
GDT
LDDD
pTM
GDT
LDDI
pTM
GDT
LDD
pTM
GDT
LDDT
AF2 MSA
14.3
60.8
66.7
58.2
75.5
80.3
31.2
29.5
40.5
40.8
41.6
51.9
40.3
43.7
56.6
46.2
50.4
62.8
EvoDitt
54.2
62.4
68.1
48.1
63.0
67.7
40.2
27.3
38.4
44.1
35.6
46.7
47.4
43.3
55,9
45.5
45.0
57.1
MSA-Aug.
47.2
62.3
68.3
58.4
75.1
80.6
39.1
33.2
42.7
44.3
43.2
51.9
44.8
47.6
59.8
48.8
52.8
65.2
EvoGen
45.0
63.5
68.8
61.8
78.8
82.8
36.0
34.8
44.9
42.8
45.1
54.6
44.3
47.0
59.4
49.1
53.0
65.4
MSAGPT
51.4
69.4
63.6
78.8
83.2
40.0
36.3
45.5
47.7
47.8
56.0
48.6
50.6
53.3
53.7
57.0
69.2
RFT
48.1
63.5
68.1
62.4
79.0
83.1
41.2
36.1
45,5
49.8
49.8
56.0
48.6
50.5
62.4
53.6
57.2
69.0
DPO
49.4
64.4
68.7
63.0
79.4
83.5
40.9
39,5
48.1
49.8
53.1
60.0
47.6
50.9
63.2
52.7
57.8
69.5
insertions, and gaps. Many previous works aim to filter high-quality MSAs by clustering or ranking them based on predefined rules. One primary rule is to find MSA sequences most similar to the main sequence with fewer gaps, as these are more likely to represent informative co-evolutionary patterns. Following this idea, we sample MSA sequences using similarity- basee weighted sampling, where sequences more similar to the query protein are more likely to be selected and ranked higher. Moreover, we also randomly shuffle the sequences in the selected MSA by a certain proportion to avoid injecting the order bias.
Regarding the backbone of MSAGPT, we employ the standard transformer decoder framework [47, 49] and train the model with 2.8 billion parameters owning 36 layers, 2560 embedding size, and 40 attention heads. We employ batches of 48 MSAs with each MSA containing 12,288 residues. We follow BF16 mixed-precision pre-training strategy. We use Adam [50] as our optimizer with 31 = 0.9, 32 0.95, eps = 10-8 and a learning rate of 1.2 x 10-4, We use a cosine learning rate schedule, with a warmup of the first 2.5% steps, and decay the final learning rate down 10% of the peak learning rate. We use weight decay of 0. and gradient clipping of 1.0 without dropout. For the tokenization of the protein data, we use the residue-level tokenizer which is adopted in several PLMs [28, 11, 10]. To save the GPU memory and accelerate the pre-training process, we substitute the standard self self-attention module with the Flash Attention-vl 1 [42] in each layer. All models are trained on 24 A800 GPUs for 254k updates, consuming about 150 billion tokens. This process consumes approximately 150 billion tokens, requiring around 2.7 x 10 floating point operations (FLOPs).
Pre-trained Dataset Figure illustrates the length and depth distribution of the pre-training dataset. Moreover, we implement& thorough filtering process to eliminate any potential data leakage. Specifically, we remove all MSAS of sequences in the test sets (CAMEO, CASP, and PDB) from the pre-training dataset. Furthermore, we ensure that any sequence in the pre-training with a similarity greater than 0.9 to a sequence in the test set is excluded. To validate this filtering process, we used the HHblits [46] tool to retrieve sequences from the test set and calculate their maximal similarity distribution with sequences in the pre-training dataset. The results, illustrated in Figure 9b) show that the maximum similarity is 0.89, confirming that there is no data leakage in the pre-training dataset.
A.2 RFT
We fine-tune the base model using the pre- cross-entropy loss on DRET with training only one epoch. Specifically, we adopt the same experimental settings as that used in the pre-training stage, except for the learning rate of 1.0 x 10-5 by default. Following the pre-training phase, the model undergoes a rejective ane-tuning process, which is more energy-efficien This stage is executed on x A800 GPUs for single epoch for about two days.
RFT Dataset Filtering Threshold When curating the RFT dataset, we first sample multiple MSA subsets for each protein structure, and select high-quality MSA subsets based on the following criteria: (1) the absolute structure prediction accuracy using the MSA subset, as measured by TM-score, should be larger than 0,, and (2) the relative improvement of the prediction accuracy after using the MSA subset, as compared to single sequence prediction, should be larger than 0z. We set 02 = 0.9, and experiment with different 0g values, as shown in table 7. The RFT model trained with dataset
15

---

200000
700000
175000
600000
150000
125000
400000
Count
100000
300000
200000
100000
250
500
750
1000
1250
1500
1750 2000
250
500
750
1000
1250
1500
1750 2000
Num. Seq
Len Seq
Figure 7: The length and depth distribution of the pre-training dataset.
Generation Time vs. Total Token Length
256
128
Generation Time (min)
64
16
8-
4
2
10
20
30
40
50
Total Token Length (k)
Figure 8:
The correlation between total token length (the protein sequence length multiplied by the number of generated MSAs) and the inference time (minutes). In most cases (total token length 20K), the generation time of MSAGPT is lower than the AF2 search pipeline requiring more than 30 minutes. The result shows AGPT can generate substantial sequence lengths within practical time, thus affirming its scalability and efficiency.
filtered by 02 0.2 yields the best result, indicating that the relative information gain provided by MSA is valuable indicator for curating high quality datasets for RFT. Moreover, 0₂ = 0.5 results in 20% decrease in dataset size, leading inferior RFT model performance, highlighting the necessity of an intricate balance between data quality and data volumn.
A.3 RLAF
We fine-tune the RFT model using the DPO algorithm DDPO with training only epoch. Specifically, we adopt the batch size of 1 with each MSA subset containing a maximum of 16,384 residues. We also AdamW [50] with the learning rate of 1.0 10-6 by default. We linearly warmup the learning rate from 0 to 1.0 x 10-6 over the first 0.1 0.1% steps. This stage is also executed on 8 A800 GPUs for a single epoch for about one day
Table 6:
Performance comparison between different data source and filtering threshold values.
Data Source
nature(0.2)
nature(0.3)
generated(0.33
nature((.)--eenrattell..)
MSAGPT+ RLAFF 62.6
64.5
63.5
62.7
16

---

Table 7:
Performance comparison between different relative improvement threshold 82 values.
Threshold 0z
0
0.2
0.5
MSAGPT+ RFT
61.2
62.5
61.3
0.3
0.9-
Mav Sd Similarity
MAX: 0.896
Frequency
0.7
0.6
0.5
0.1
0.4
MEAN
0.3
0.2
MEADIAN:0.11
0.0
4
16
20
MSAs
(a) Depth Distribution.
(b) Similarity Distribution.
Figure 9: (a) The distribution of MSA depth of benchmarked datasets and (b) the similarity distribution of sequences in the test set, as retrieved from the pre-training set using HHblits.
RLAF Dataset. We conducted experiments with different data sources and filtering thresholds defined the minimum relative improvement of the good over the bad case in DPO data pairs- for the RLAF dataset, as detailed in Table 6. Utilizing only natural MSA subsets sampled from PDB, found that higher 03 values lead improved model performance, suggesting a correlation between the disparity within data pairs and DPO effectiveness. Interestingly, the quality of MSA subsets generated by the RFT model surpasses that of natural MSA subsets 0g of 0.2. However, the performance declines when natural MSAs are mixed with generated MSAs, compared to using single data source during training. This indicates that maintaining distribution homogeneity is crucial for effective DPO alignment.
A.4 Inference Efficiency
Generally, it's vital to consider not just the immediate resource consumption during pre-training and post-alignment, but also the long-term utilization of these models. Once pre- preitraned, MSAGPT demonstrates significant efficiency, capable of generating protein sequences with up 100,000 amino acids in under 8 hours. This efficiency underscores the model's 's value, especially when amortized over application lifespan and subsequent fine-tunings for specific tasks.
Regarding the scalability the MSAGPT. We present the inference time with different total lengths (measured by protein sequence length multiply the number of generated sequences.), shown in Figure 8.
The result showcases MSAGPT's ability to generate substantial sequence lengths within practical time frames, thus affirming its scalability and efficiency.
Table 8:
The paired Student's t-test between MSAGPT and other baselines on three benchmarks based on the TM-Score, where the p-value less than 0.05 indicates the result is said be statistically significant.
CAMEO (avg. Depth 8.5)
CASP
PDB (avg. Depth 2.6)
(avg. Depth = 4.6)
Model
Zhro-Shot
Few-Shot
Zero-Shot Few-Shot
Zero-Shot
Few-Shof
AF2 MSA
0.014
0.023
0.008
0.007
5e-7
8e-9
EvoDiff
0.016
le-5
Se-4
7e-5
0.012
le-8
MSA-Aug.
0.023
0.014
0.044
0.015
6e-6
le-7
EvoGen
0.038
0.027
0.067
0.016
le-8
le-9
MSAGPT
17

---

Table 9: The results of 5-fold cross-validation performance between with or without virtual MSA generated by MSAGPT on four protein-relate tasks.
2
3
5 ACC
AVG
Model
Top (1.55)
ACC
ACC
ACC
w/o Virtual MSA (CIP)
11.4
14.3
10.7
9.9
11.8
11.6
w/ Virtual MSA (CtP)
14.1
13.7
13.4
11.8
12.3
13.1
w/o Virtual MSA (SsP)
67.7
65.8
64.0
68.9
66.2
66.5
w/ Virtual MSA (SsP)
70.5
67.8
67.5
69.0
69.0
w/o Virtual MSA (LocP)
56.0
64.5
48.0
59.0
57.0
58.3
w/ Virtual MSA (LocP)
47.0
58.5
53.5
64.0
59.0
56.4
Virtual MSA (MIB)
58.0
53.5
49.5
67.5
57.5
w/ Virtual MSA (MIB)
61.5
57.0
63.0
53.0
67.0
60.3
B
Experimental Settings
B.1 Evaluation Details
We employ three golden metrics: TM-Score, widely-used metric for assessing the structural similarity between predicted structures and ground truth, LDDT, the local distance difference test score measures how well local interactions in reference structure are conserved the protein model being assessed, GDT-TS, the global distance test to represent "total score", a measure of similarity between two protein structures with known amino acid correspondences (e.g. identical amino acid sequences) but different tertiary structures, and two predicted metrics: pLDDT, the predicted local distance difference test measuring the local confidence of per-residue and pTM,, integrated measure of how well AlphaFold2 has predicted the overall structure. All metrics are normalized from 0 to 100 for comparison, with higher scores indicating higher confidence and usually more accurate prediction. where indicates a perfect match between two structures. Each run across independent runs. For each run, adopt the different temperatures (T {0.8, 1.0)) along with different nucleus sampling factors (P {0.8, 1.0)), experimenting with varying the number of generated MSAs in 8, 16, 32, and 64. The final performance is determined by first identifying the predicted structure with the highest accuracy across these different depths, and then averaging the results across the test set.
B.2 Experiments on MSA-abundant Scenario
We compare the results of query sequences with abundant natural MSAs those with abundant natural MSAs augmented by MSAGPT's generated MSAs on CAMEO set. For this comparison, we sample 128, 256, and 512 sequences from both the natural MSAs and the generated MSAs, as shown in Table 10. These results indicate that the inclusion generated MSAs has no significant effect on the performance in MSA-abundant conditions, which is consistent with previous findings that when more than 64 MSAS as input, AF2 predicts a "converged" structure.
B.3 Setup of Transferability of MSAGPT to Other Tasks
We utilized the MSA Transformer [17] as the backbone model with the task-specific heads. We on experiments. All the task benchmarks are obtained following the pipeline in [11]. For each task, sample 1000 protein sequences with the corresponding labels. Then we use MSAGPT-DPO to generate 32 virtual MSAs with the generation strategy T=0.8 and P=0.8. Both setups are trained briefly (for one epoch) for 5-fold cross-validation as shown in Table 9, and we report the average performance.
finetune MSA transformer with the head with
5 and batchsize
B.4 Setup Ablation Study
Experiments are conducted based on models with 150 million parameter size encompassing 30 layers, 20 attention heads, 640 embedding dimensions. These models are trained across approximately 30 billion tokens, amounting around 40k training steps, maintaining consistency in paraer-prrmmeter
18

---

Table 10: datasets.
Performance comparison in MSA-abundant scenarios across all 194 cases in the CAMEO
# Natural MSA
Virtual MSA
TM
GDT-TS
0
39.1
28.5
128
0
85.1
82.0
128
128
85.3
82.2
0
86.2
83.8
256
256
86.0
83.5
512
0
86.5
84.2
512
512
86.4
84.0
settings with MSAGPT, except for variations in the positional encoding mechanism. The efficacy of each variant is assessed through zero-shot MSA generation on the CASP test set.
C Selection Strategy Details and PLDDT Evaluation
To evaluate the effectiveness of different selection strategies, we extracted 4, 8, 12, 16, 24, and 32 sequences from 48 zero-shot generated MSA for each method and computed the median TM-scores (Figure 4(b)) and pLDDT scores (Figure 10) across 33 test cases. The strategies are detailed below.
Static Similarity Static Diversity Strategy: We select the top-k generated MSA with the highest lowest sequence identity the query sequence. Sequence identity is deter- mined by the percentage of identical residues between the two sequences.
DOT
DV% S73
DY%.01 DM
T31M
pTh
PLOO7
Dynamic Similarity/ Static Diversity Strategy: Starting with the MSA most least similar the query sequence, we sequentially incorporate MSA into the selected set based on the highest or lowest average sequence identity with all sequences already included, until reaching total k k MSA.
TM
Selected MSA Deeth
Figure 10: different selection methods. Dashed red line represents using all generated se- quences a given depth. Solid lines rep- resent selecting subset of given depth from 48 generated sequences with a spe- cific strategy. The curves are smoothed using the Exponential Moving Average with alpha=0.3.
The pLDDT curves across
Trimming Strategy: Suggested by EvoGen, this method filters out MSA with less than 50% coverage or sequence identity to the query sequence above 90% or below 20%. Subsequently, it iteratively selects MSA with the clos- est sequence identity the query and average sequence identity below 90% with all the chosen MSA.
pTM / PLDDT / TM Score Strategy: For each generated MSA, we remove the gaps and predict its structure using AF2. The structures are then ranked based on the pTM score (as reported by AF2), the pLDDT score (as reported by AF2), the TM score compared to the query sequence's ground truth structure (calculated by US Align), and the MSA corresponding to the top-k structures for each metric selected accordingly.
D Protein Structure Prediction Analysis compared with natural MSA
We present detailed visual comparison of protein structures predicted by AlphaFold2 (AF2) utilizing MSA augmented by MSAGPT, against those predicted with natural MSA. This comparison, as depicted in Figure 12, highlights the remarkable capability of MSAGPTin enhancing the accuracy of structure predictions to levels that closely rival, and in some cases surpass, those based on naturally derived MSA.
We delve into visualized analysis of protein structures predicted using AlphaFold2 (AF2) with MSA augmented by our proposed model (MSAGPT), alongside those augmented by EvoGen and
19

---

7mny B
TM-Score = 70.5
TM-Score 17.1
TM-Score 17.6
7ywe B
TM-Score 97.2
TM-Score 32.3
TM-Score =45.6
7tdy B
TM-Score 95.6
TM-Score =29,4
TM-Score =22.5
Figure 11:
Visualization of improved structure prediction compared with baseline models. Yellow: Ground truth; Pink: Predictions based MSA generated by MSAGPT, Blue: Predictions from MSA generated by EvoGen; Green: Predictions utilizing MSA generated by ASA-Augmenter.
MSA-Augmenter. This comparison, visualized in Figure 11, encompasses a spectrum of proteins, ranging from short sequences with relatively simple structures, like 7mnv B, long sequences with complex configurations, such as 7tdv B. It includes proteins characterized by multiple beta sheets, exemplified by 7ywg. as well as those rich alpha helices, such as 7tdv B. Across these diverse cases, MSAGPT significantly surpasses both EvoGen and MSA- MSA-Augmenter, improving the TM score maximum of 0.9.
By detailed examination, we observe that while the MSA augmented by the baseline models assist AF2 in accurately predicting local structures and folds, they fall short in aligning the global composition and orientation with the ground truth structure, which is effectively addressed by MSA generated by MSAGPT. The local structures, which generally more discernible from the spatial arrangements of adjacent amino acids, contrast with the global structures whose prediction relies heavily on comprehensively understanding the co-evolutionary information within MSA. These co-evolutionary patterns, indicating proximity in three-dimensional space through simultancous mutations at multiple positions, crucial for accurate global structure prediction. These findings underscore MSAGPT's impressive capability to comprehensively capture and utilize co co-evolutionary information, thereby significantly enhancing the accuracy of protein structure predictions. More visualization cases about the predictions based on MSA generated by MSAGPT and the predictions based on the natural MSA are illustrated in Appendix D.
E Protein Structure Prediction Improvement after DPO
Figure 13 represents the comparison before and after the DPO training, depicting notable enhance- ments in structure prediction accuracy. Figure and 155 provide an in-depth analysis the generated MSA for each case. Specifically, residues 43, 53, 71-79, 105-111, 122, 132 and 157-166 in the
20

---

7vrb A
TM-Score 84.7
TM-Score = 55.8
T1033
TM-Score = 61.73
TM-Score = 38.5
7vu7 A
TM-Score = 70.1
TM-Score 49.5
Figure 12: Visualization of improved structure prediction compared with nature MSA Yellow: Ground truth; Pink: Predictions based on MSA generated by MSAGPT; Blue: Predictions from MSA generated by natural MSA.
21

---

7wme A
TM-Score = 599.6
TM-Score 96.1
7sxb A
TM-Score 46.6
TM-Score 78.9
Figure 13: Visualization of improved structure prediction after DPO. Yellow: Ground truth; Blue: Predictions based on MSA generated by MSAGPT; Pink: Predictions based on MSA generated by MSAGPT-DPO.;
MSA of 7wme_A along with residues 22-27, 53, and 73 in the MSA of 7sxb_A, display distinct characteristics pre- and PPSt-DPO training.
22

---

generated by MSAGPT
(b) generated by MSAGPT-DPO
Figure 14: Residue Distribution of Generated MSA for 7wme_A. The red box indicates natural MSA used as prompts during generation. The blue box indicates generated MSA. Residues are colored using the clustal scheme by Jalview.
23

---

(a) generated by MSAGPT
(b) generated by MSAGPT-DPO
Figure 15: Residue Distribution of Generated MSA for 7sx A. The red box indicates natural MSA used as prompts during generation. The blue box indicates generated MSA. Residues colored using the clustal scheme by Jalview.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly made statements to claim the contributions in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed several limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

This work is primarily focused on empirical experiments and extensive val- idation rather than providing theoretical results. Therefore, the paper does not include set of assumptions or proofs for theoretical results since it is not the main emphasis of the research.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in the Appendix and the released model weight at
to ensure the reproduction of all the
experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation details in the Appendix to ensure the production of all the experimental results. Moreover, we also released model weight at http://iihub.co///HCCD//MM/
to the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
(tailic/giides//deeuuniiii cot.uuiiiiiiooioccocioooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. //nips.cc//ppppp wttpccccpppcpccpeeeeeeecccc htppuuiiiiuuddssssseeeppppeeseel CoodeSummis sssisssss ssssssoliccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in the main content the Appendix to ensure the reproduction of all the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have presented the results of the paired Student's 's t-test between our model and other baselines on three benchmarks (Table in the Appendix Table 8.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the implementation details in the Appendix and the released model weight at https//gtttbb..omm//UUUMMMMAAAA to ensure the reproduction of all the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tpurii////uiips/s
Answer: [Yes]
Justification: We have reviewed and obeyed the NeurIPS Code of Ethics https: cc/ htc/public/Ethiccetto caGuddell lines.....
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have already discussed the broader impact in Section 8.
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have already discussed the broader impact with potential risk in Section 8.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have mentioned all the sources of used data, code, and models, and given the credited to the corresponding authors or organizations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We
release the inference script
and
model weight at
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't include human subjects crowdsourcing in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't include human subjects or crowdsourcing this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "t0c1B0lWUM", "reviewer": "Reviewer_wE2x", "summary": "This paper proposes a method to generate multiple sequence alignments for a given protein sequence. To model the co-evolutionary information, the paper proposes 2d evolutionary positional encoding. After pretraining on the alignment sequences, the models are fine-tuned with AlphaFold2 annotations to avoid hallucinations.", "questions": "See above", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The studied problem, generating multiple sequence alignment is interesting and novel. \n* The proposed method is technically sound.\n* The paper is well-written and well-structured.", "weaknesses": "* Missing important experimental details. The paper omits crucial experimental details, particularly those related to pretraining. This omission significantly impacts the study's credibility and reproducibility. Especially, it's important to explain the process of hyperparameter selection is not explained in detail.\n* Evaluation is not rigorous. There is no clear explanation of steps taken to prevent data leakage or the inclusion of data similar to the evaluation set in the training data. The absence of structural or temporal splits is particularly problematic, as these are crucial for assessing a model's performance in truly novel scientific scenarios. \n* Evaluation is limited. It would be interesting to design studies to directly investigate the evolutionary patterns learned in the multiple sequence alignment algorithm. Also the paper fails to include many important protein function tasks in its evaluation.\n* The results presented in the paper lack error bars, which is particularly problematic for results that are close, such as Table 2 and 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and careful assessment of our work. We address your concerns below,\n\n**About Weakness-1: the missing important experimental details.** The training details and experimental settings, including the processes for Pre-training, Rejective Finetuning, and DPO, as well as the hyperparameter selection, are thoroughly discussed in Appendix Section C. \n\n**About Weakness-2: the data leakage prevention and test data split.** \n\n*Data Leakage Prevention*. As outlined in Section 6.1 of our paper, we implemented a thorough filtering process to eliminate any potential data leakage. Specifically, we removed all MSAs of sequences in the test sets (CAMEO, CASP, and PDB) from the pre-training dataset. Furthermore, we ensured that any sequence in the pre-training set with a similarity greater than 0.9 to a sequence in the test set was excluded. To validate this filtering process, we used the HHblits tool to retrieve sequences from the test set and calculate their maximal similarity distribution with sequences in the pre-training dataset. The results, are illustrated in Figure. 1 in the attached PDF, shows that the maximum similarity is 0.89, confirming that there is no data leakage in the pre-training dataset.\n\n*Temporal and structural splits*. For the pre-training dataset, we used the OpenProteinSet, containing protein sequences collected before December 28, 2021. For structural predictions, we followed AlphaFold2's evaluation settings, using PDB datasets from before January 22, 2024, CASP14 from May to August 2020, CASP15 from May to August 2022, and CAMEO after August 20, 2020. Our primary goal is to improve structural prediction accuracy in low-MSA regimes using generated virtual MSAs. Therefore, our methods need to generalize across different protein families and timelines. We ensured that sequences in the test set were not included in the pre-training set, as AlphaFold2 does, and conducted experiments on three well-benchmarked datasets to confirm the robustness and generalizability of our approach.\n\n**About Weakness-3: the clarification of evaluation.** Our primary goal is to enhance low-resource structure prediction using generated virtual MSA. The main experiments focus on protein structure predictions in zero-shot and few-shot scenarios on a Natural MSA-scarce benchmark. We also present results on artificially MSA-scarce and MSA-abundant scenarios (see Tables 4 and 6 in the attached PDF). Additionally, we demonstrate our model's transferability across four protein tasks, highlighting MSAGPT’s potential to impact a broad range of protein-related tasks with generated MSA.\n\n**About Weakness-4: the statistical significance test of results.** We have addressed statistical significance in several ways. For results in Tables 1 and 2, t-test results demonstrating significance against baseline methods are in Appendix Table 6. For Table 3, we conducted 5-fold cross-validation and reported average performance to mitigate random effects, detailed in Appendix Table 7.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttoCbE6U50", "reviewer": "Reviewer_zdxm", "summary": "Protein structure prediction tools such as alpha fold take a query protein sequence, expand it to a multiple sequence alignment (MSA) of related natural sequences, and then feed this alignment into the model. The first expansion step isn't possible, however, for proteins that don't have many natural relatives, and there are many such 'orphans'. This paper demonstrates that the MSA can be replaced with a set of virtual sequences sampled from a generative model. One of the key contributions of the paper is showing how to find tune those generative model explicitly for the sake of improving the downstream protein structure prediction performance using LM preference optimization techniques such as DPO.", "questions": "**Tiny evaluation sets\nThe evaluation metrics are reported on very tiny evaluation sets (L247; 8 from cameo, 13 from CASP14&15, 179 from PDB). This makes me worry that the differences in performance are not statistically significant, or don't reflect performance on the broad distribution of orphan proteins that users may want to perform prediction for. \n\nI see two ways forward: (1) demonstrate that the proteins in these tiny eval sets are somewhat representative and that the differences in metrics are statistically significant or (2) change the eval setup to use a bigger eval set.\n\nI think (2) is much easier. To do this, can't you turn any example in the full eval set (e.g., the PDB) into an orphan? You could do the zero-shot eval using just the original sequence, with an MSA containing only virtual sequences. With this, can you report performance on the full eval sets?\n\n**Fine tuning decreases pLDDT\n\nI understand the argument that pLDDT decreased and TM score increased because the fine tuning targeted improvements to TM score. However, it's unfortunate that pLDDT increased. If you fine tune for pLDDT, would that increase at the expense of TM? Could you extend the fine tuning to improve both (such as using a composite reward function based on both)? \n\n**Minor points\nL24 is inaccurate: \"The remarkable success of AF2 can be attributed to its innovative use of co-evolutionary information supported by the Multiple Sequence Alignment (MSA).\" Coevolution had been used for structure prediction for many years before AF. AF got better performance because of general model scale and processing the MSA end-to-end instead of using pre-computed features of the MSA.\n\nL113: what does 'homogenous' mean in this context?\n\nL149: As far as I can tell, there are no semantics to the ordering of the rows. The row positional encoding is basically a unique id for which row it belongs to. Perhaps it would be better to use a different representation for the row index that doesn't reflect linear ordering so much.\n\nL180: What order do you use for flattening the axes? Have you tried both? \n\nI'm curious if there is any noticeable difference in the sequences generated after DPO vs. the base model. Are there any sort of degeneracies due to reward hacking, etc?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Produces impressive performance improvements for protein structure prediction in the important regime where the query protein is far from other natural proteins. \n\nDemonstrates that good performance can be achieved without modeling techniques specific to multiple sequence alignments (such as axial attention). Using a vanilla setup is nice because it enables using off-the-shelf LM systems that have improvements such as flash attention, etc.\n\nUses modern tricks of the trade for improving generative models using DPO, etc.\n\nThe evaluation compares to multiple recent papers for generating virtual MSAs.\n\nDoes a systematic investigation into how to select generated virtual sequences to include in an MSA (L296).", "weaknesses": "The evaluation sets are very very small (see below). This is why I gave 'soundness' a score of 2. \n\nThe advanced fine tuning techniques don't uniformly improve eval metrics (see below).\n\n==Update after author's response==\nI have raised my score to 'accept' and raised my soundness score, since the response adequately addressed my concerns about evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and constructive suggestions for our work. We addressed your questions as follows:\n\n**About Question-1: add more evaluations.** We have adopted both suggestions to confirm the superiority of MSAGPT:\n+ Statistical Significance of Metrics: We conducted a paired Student's t-test between MSAGPT and other baselines. The results, shown in Appendix Table 6 in the paper, indicate that the virtual MSA generated by MSAGPT significantly improves structure prediction accuracy in cases with limited MSA compared to other baselines.\n+ Evaluation on a Larger Set and more metrics: We created the Artificial MSA-Scarce benchmark based on the PDB dataset released before 2024-01-22. We collect approximately 8k protein sequences after filtering and perform zero-shot evaluations using these sequences with MSAs containing only virtual sequences. The results are detailed in Table. 4 in the attached PDF. These results demonstrate that the generated MSA significantly improves structure prediction accuracy in cases with limited MSA than other baselines. \n\n**About Question-2: the explanation of decreased pLDDT**. The predictive metrics estimated by AlphaFold2, such as pLDDT and pTM, measure the confidence level of AlphaFold2's predictions, rather than reflecting the true structural prediction accuracy. pLDDT can be easily influenced by adding more MSAs. We conducted an ideal experiment to compare the performance among original sequences, augmented by randomly generated MSAs, and augmented by MSAGPT MSA as shown in Table. 5 in the attached PDF. We can see even with randomly generated MSAs, the pLDDT and pTM values are higher than with the original sequences, while the TM score decreases due to the introduction of noisy MSAs. Therefore, we adopt the TM score as the primary metric to select high-quality MSAs for supporting the subsequent RFT and DPO processes. Additionally, we provide other oracle metrics including GDT_TS, LDDT in Table 1,2,3 in the attached PDF. The results show that with TM score as the reward signal, other oracle metrics, i.e., GDT_TS and LDDT also improve, while the predicted metric pTM shows the opposite trend. Thus, the reasonable explanation for the decreased pLDDT and pTM metric is that the post-alignment process reduces hallucination scenarios.\n\n**About Question-3: correct the clarification on the utilization of MSA in AF2 .** Thanks for your constructive feedback. We will incorporate your suggestions to clarify this in the next version of our paper.\n\n**About Question-4: the typo of \"homogenous\".** Thank you for pointing out the typo. The correct term should be \"homologous,\" which, in the context of biology, refers to organs or sequences that are similar in position, structure, and evolutionary origin, but not necessarily in function.\n\n**About Question-5: the semantics of MSA rows.** The MSA obtained by MSA search tools, such as HMM search, inevitably contain noisy co-evolutionary patterns, such as large portions of deletions, insertions, and gaps. Many previous works aim to filter high-quality MSAs by clustering or ranking them based on predefined rules. One primary rule is to find MSA sequences most similar to the main sequence with fewer gaps, as these are more likely to represent informative co-evolutionary patterns. Following this idea, we sample MSA sequences using similarity-based weighted sampling, where sequences more similar to the query protein with less gaps are more likely to be ranked higher and selected. Our ablation study results, shown in Figure 5, confirm that compared to 2D positional encoding, the 1D positional encoding (which only retains the column site position while abandoning the row ordering information) performs worse. This indicates that incorporating row-order semantics through similarity-based sampling improves performance.\n\n**About Question-6: the flattening rules.** As demonstrated in the overall framework in Figure 2 in the paper, we flatten the MSA along the row axis to ensure we can generate the MSA sequentially during inference. Flattening along the column axis is also theoretically reasonable, as the ordering information is bounded by the 2D evolutionary position embedding. However, when generating the MSA sequentially during inference, we would need to reverse the positional IDs along the row side. This differs from the training pattern and could introduce unforeseen errors, which require further investigation.\n\n**About Question-7: the degenerated cases after DPO.** The case studies showing the differences in generated sequences before and after DPO are already presented in Appendix Section G and Figures 11 to 15. Generally, we do not observe significant differences in the generated sequences, except for a few degenerate cases, including generated MSAs that do not match the length of the query sequences or MSAs that contain only gaps or a single type of residue. The proportion of these degenerate MSAs is approximately 6% for DPO-generated MSAs compared to 1% for the base model. To address these degenerate cases, online RLHF like PPO, which directly involves AF2 in the training procedure, may be more effective. However, this approach faces efficiency issues due to the lower inference efficiency of AF2, which needs further investigation.\n\nWe guarantee that we will include all experimental results and discussions in the next version of our paper. Your feedback has been invaluable in refining our research. If we've addressed your concerns, we hope you might consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVqIWo2jL7", "reviewer": "Reviewer_Reh4", "summary": "The paper proposes MSAGPT, a novel method for generating MSAs. Utilizing 2D evolutionary positional encoding, MSAGPT reformalizes MSA generation as a one-dimensional sequence generation task optimized with a simple GPT objective. The model incorporates feedback from AlphaFold2 to reduce hallucinations during MSA generation via DPO fine-tuning. Experimental results on curated datasets demonstrate that MSAGPT enhances protein structure prediction in low-MSA scenarios, achieving improved structural reconstruction scores.", "questions": "1. I feel like most innovations are credited to 2D RoPE positional embedding, limiting the scope of novelty. Also, could you provide detailed explanation of this method in English? The reference is in Chinese.\n\n2. As mentioned in weakness, please provide a detailed comparison of the efficiency of MSAGPT with other MSA generative models.\n    \n3. How does the model perform in MSA-abundant conditions? This should also be evaluated.\n    \n4. For the “prediction accuracy” on lines 218 and 121, could you specify the metrics used (e.g., TM-score, RMSD, lDDT, pLDDT)?\n    \n5. minor: How does pLDDT selection help in finding structurally similar sequences, as mentioned on line 329?\n    \n6. minor: On line 230, do you mean the DPO dataset contains 11k samples?", "limitations": "Please refer to the Weaknesses and Questions sections. I hope the authors can address concerns regarding efficiency, provide more comparative analysis with baseline models, and offer further explanation on the use of 2D RoPE positional embedding.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, easy to follow, and the proposed framework is simple yet effective, and is straightforward to implement.\n\n- The use of 2D positional encoding to re-formalize MSA generation as a one-dimensional sequence generation task is innovative and allows for zero- or few-shot MSA generation under a flexible in-context learning framework. It points out a possible direction of playing 2D sequences with novel positional encoding.", "weaknesses": "- **Efficiency Concerns:** Flattening 2D sequences into 1D for interaction with self-attention increases time complexity, even with FlashAttention. While Figure 8 shows MSAGPT's generation time is lower than the AF2 search pipeline, a comparison with other MSA generative models' efficiency is necessary.\n\n- **More Comparative Analysis:** The paper should include a comparison with diffusion-based models for generating protein sequences, such as EvoDiff MSA. Also, could you also report the RMSD (GTD_TS) score.\n\n- **Limited Use Case:** The practical use of generating virtual MSAs is limited to models that utilize MSAs, such as MSA Transformer or AlphaFold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About Question-1: the explanation of 2D RoPE and the novelty clarification.** \n\n*2D RoPE Explanations*. Rotary Positional Embeddings (RoPE) encode position information of tokens with a rotation matrix that naturally incorporates explicit relative position dependency. First, consider the 1D Rotary Positional Embedding. Given any two-dimensional feature vector $x_i \\in \\mathcal{R}^2$ at position $m$. The position embedding can be expressed as: \n<p>\n$f_{\\{q,k\\}}(x_m, m) = \\mathbf{R}_{\\theta,m}^2 W_{\\{q,k\\}} x_m$. \nSuch that $$q_m^Tk_n = (\\mathbf{R}_{\\theta,m}^2 W_q x_m) ^ T(\\mathbf{R}_{\\theta,n}^2 W_k x_n)=x_m^TW_q\\mathbf{R}_{\\theta,n-m}^2W_kx_n\n$$\nHere: $\\mathbf{R}_{\\theta,\\{m,n\\}}^2$  is the rotation matrix that depends on the position. $W_{\\{q,k\\}}$ is a learnable weight matrix. The key to RoPE embeddings is to determine the rotation matrix:\n$$\n\\mathbf{R}_{\\theta, m} = \\begin{pmatrix}\n\\cos m\\theta & -\\sin m\\theta  \\\\\n\\sin m\\theta & \\cos m\\theta \n\\end{pmatrix}\n$$\nAfter some derivations, given a 2D position $(m,n)$, a solution for the 2D RoPE is obtained as:\n$$\n\\mathbf{R}_{\\theta,(m,n)} = \\begin{pmatrix}\n\\cos m\\theta & -\\sin m\\theta & 0 & 0 \\\\\n\\sin m\\theta & \\cos m\\theta & 0 & 0 \\\\\n0 & 0 & \\cos n\\theta & -\\sin n\\theta \\\\\n0 & 0 & \\sin n\\theta & \\cos n\\theta \n\\end{pmatrix}\n$$\n</p>\nThis solution is easy to understand. It is a block matrix composed of two 1D RoPEs, essentially dividing the input vector into two halves, applying the 1D RoPE to $m$ for one half and the 1D RoPE to $n$ for the other half. From this form, we can also easily generalize to RoPE for 3D, 4D, and other dimensions.\n\n*The novelty clarification*. The principle of multi-dimensional positional encoding has been explored across various domains to address specific challenges inherent to those fields with different intrinsic design proposes. In the MSA generation scenario, incorporating a dual-axis positional encoding scheme is driven by the unique requirements of modeling the complex dynamics of evolutionary patterns in protein homologous sequences, which involves identifying simultaneous mutations across multiple amino acid sites (columns) in different homologs (rows), other high-level interactions. Therefore, a multi-dimensional encoding approach, as compared to a decoupled single-dimensional approach, is both distinct and critical. In light of this, we adapt the RoPE-2D relative positional encoding extended from 1D RoPE to capture these patterns.\n\n\n**About Question-2:  the efficiency comparison.** We compared the generation speed between MSAGPT and several baseline generative models, including the newly-added EvoDiff. These models were run on a single A100 80G GPU under the direct sequential generation regime, and we report the average tokens per second (toks/s) for generating 2k tokens, averaged over three runs.\n\n| Model      | Toks/s |\n|------------|--------|\n| MSA-Aug.   | 0.35     |\n| EvoGen     | 0.94     |\n| EvoDiff    |    1.16   |\n| MSAGPT     | 0.92    |\n\nFrom this comparison, we can conclude that MSA-Aug. achieves the lowest inference efficiency due to its encoder-decoder framework. EvoGen and our proposed MSAGPT have similar inference speeds, with the diffusion framework (EvoDiff) showing better inference efficiency. However, EvoDiff shows worse MSA generation quality in the structural prediction tasks (The performance comparison refers to the Table. 1, 2, 3,4 in the attached PDF). \n\n\n**About Question-3: the performance on MSA-abundant conditions.** We compare the results of query sequences with abundant natural MSAs to those with abundant natural MSAs augmented by MSAGPT's generated MSAs on CAMEO set. For this comparison, we sample 128, 256, and 512 sequences from both the natural MSAs and the generated MSAs. The results are shown in Table. 6 in the attached PDF. These results indicate that the inclusion of generated MSAs has no significant effect on the performance in MSA-abundant conditions, which is consistent with previous findings that when more than 64 MSAs as input, AF2 predicts a \"converged\" structure.\n\n**About Question-4: the metrics measuring the \"prediction accuracy\".** The prediction accuracy refers to the TM-score, which serves as the golden metric for evaluating the accuracy of predicted protein structures compared with the ground truth structures. Additionally, we provide other metrics such as pTM, GDT_TS, and LDDT in Table 1,2,3,4 in the attached PDF. The results indicate that enhancements in TM-score are consistently accompanied by improvements in these golden metrics, i.e., GDT_TS  and LDDT, confirming the robustness and reliability of our predictive method across different evaluation criteria.\n\n**About Question-5: the pLDDT selection strategy.** pLDDT selection helps identify structurally similar sequences by providing a confidence measure in predicted protein structures without needing ground truth. Higher pLDDT scores highlight regions predicted with greater accuracy, indicating that the corresponding virtual MSA is informative and structurally similar. This confidence-based filtering focuses on the most reliable parts of the predicted structures for more accurate identification. For detailed selection processes, please refer to Appendix E.\n\n**About Question-6: the number of cases used in DPO.** We construct RLAF preference dataset\n$\\\\{Q^{(k)}, m_{w}^{(k)}, m_{l}^{(k)}\\\\}_{n=1}^{N}$ for the DPO training, where $N=11k$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response on Newly-Added Comprehensive Evaluations and Claims

Dear Reviewers,

Thank you for your insightful feedback and constructive suggestions. We have incorporated additional experimental results, **detailed in the attached PDF**, and provided thoughtful discussions to address your concerns point by point, demonstrating the strengths of our work. Specifically, we have:

- Clarified the issue of data leakage in the pre-training data.
- Added statistical significance tests showing that our approach significantly outperforms baselines.
- Introduced a large artificially MSA-scarce zero-shot evaluation benchmark with approximately 8k newly-released PDB datasets.
- Evaluated our model on MSA-abundant scenarios using the 194 CAMEO dataset.
- Included more comprehensive evaluations with additional metrics such as pTM, LDDT, and GDT_TS, advanced baselines like EvoDiff,a diffusion-based sequence generation model, and efficiency comparisons.

We guarantee that all experimental results and critical discussions will be included in the next version of our paper. We appreciate the time and effort you have dedicated to reviewing our work. Your valuable comments have helped us refine our research. If our rebuttals have addressed your critical concerns, we hope you can consider raising your score.

Best regards,  
Authors of MSAGPT

Meta-review: The paper introduces MSAGPT, a method that generates multiple sequence alignments (MSAs) by considering it as a 1D sequence generation task, using a 2D evolutionary positional encoding. The paper demonstrates that such generated alignments can substantially improve protein structure prediction accuracy in cases with low-depth MSAs. 

The reviewers found the paper well-written and easy to follow, and the method novel and technically sound. They highlighted impressive performance gains achieved in the shallow-MSA regime. The primary concerns raised were regarding the experiments, requesting more details and clarity regarding potential data-leakage. The authors addressed these issues in their rebuttal, and post-rebuttal, all reviewers clearly recommended the paper be accepted.
Decision: Accept (poster)