Co-evolution Transformer for Protein Contact Prediction
Authors: He Zhang
Fusong Ju
Jianwei Zhu
Liang He
Bin Shao
Nanning Zheng
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: protein contact prediction
transformer
co-evolution principle

Abstract: Proteins are the main machinery of life and protein functions are largely determined by their 3D structures. The measurement of the pairwise proximity between amino acids of a protein, known as inter-residue contact map, well characterizes the structural information of a protein. Protein contact prediction (PCP) is an essential building block of many protein structure related applications.  The prevalent approach to contact prediction is based on estimating the inter-residue contacts using hand-crafted coevolutionary features derived from multiple sequence alignments (MSAs). To mitigate the information loss caused by hand-crafted features, some recently proposed methods try to learn residue co-evolutions directly from MSAs. These methods generally derive coevolutionary features by aggregating the learned residue representations from individual sequences with equal weights, which is inconsistent with the premise that residue co-evolutions are a reflection of collective covariation patterns of numerous homologous proteins.  Moreover, non-homologous residues and gaps commonly exist in MSAs. By aggregating features from all homologs equally, the non-homologous information may cause misestimation of the residue co-evolutions.  To overcome these issues, we propose an attention-based architecture, Co-evolution Transformer (CoT), for PCP. CoT jointly considers the information from all homologous sequences in the MSA to better capture global coevolutionary patterns. To mitigate the influence of the non-homologous information, CoT selectively aggregates the features from different homologs by assigning smaller weights to non-homologous sequences or residue pairs.  Extensive experiments on two rigorous benchmark datasets demonstrate the effectiveness of CoT. In particular, CoT achieves a $51.6\%$ top-L long-range precision score for the Free Modeling (FM) domains on the CASP14 benchmark, which outperforms the winner group of CASP14 contact prediction challenge by $9.8\%$.

Co-evolution Transformer for Protein Contact Prediction
He Zhang"!
Fusong Ju
Xi'an an Jiaotong University
Microsoft Research Asia fusongj u@mi www.oft. com
mao736487788sstu.22 tu. edu, en
Jianwei Zhu
Liang He Microsoft Research Asia
Bin Shao
Microsoft Research Asia
Microsoft Research Asia
j jianzzh@microsoof..com
1 Lihe@microsoft.coom com
binshao@mi www.aaammm.orooc...ommmmmmmmm com
Nanning Zheng!
Tie-Yan Liu
Xi'an Jiaotong University unzheng@mail il, xut.. edu. en
Microsoft Research Asia tyliu@mi tyliu@microsoot. com
Abstract
Proteins are the main machinery of life and protein functions are largely deter- mined by their 3D structures. The measurement of the pairwise proximity between amino acids of a protein, known as inter-residue contact map, well characterizes the structural information of protein. Protein contact prediction (PCP) an es- sential building block of many protein structure related applications. The prevalent approach to contact prediction is based on estimating using hand-crafted coevolutionary features derived from multiple sequence align- ments (MSAs). To mitigate the information loss caused by hand-crafted features, some recently proposed methods to learn residue co-evolutions directly from MSAs. These methods generally derive coevolutionary features by aggregating the learned residue representations from individual sequences with equal weights, which is inconsistent with the premise that residue co-evolutions are a reflection of collective covariation patterns of numerous homologous proteins. Morcover, non-homologous residues and gaps commonly exist in MSAs. By aggregating features from all homologs equally, the non-homologous information may cause misestimation of the residue co-evolutions. To overcome these issues, we propose attention-based architecture, Co-evolution Transformer (CoT), for PCP. CoT jointly considers the information from all homologous sequences in the MSA to better capture global coevolutionary patterns. To mitigate the influence of the non- homologous information, CoT selectively aggregates the features from different homologs by assigning smaller weights to hon-hooologous sequences or residue pairs. Extensive experiments on two rigorous benchmark datasets demonstrate the effectiveness of COT. In particular. CoT achieves 51.6% top-L long-ram ranng precision score for the Free Modeling (FM) domains on the CASP14 benchmark, which outperforms the winner group of CASP14 contact prediction challenge by 9.8%.
inter-residue contacts
"Work during an internship at Microsoft Research. EEuul contribution.
1 *Corrsponniigg authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
Most in vivo biological processes are carried out by proteins, whose functions are mainly determined by their 3D structures [1]. Structural information is crucial for understanding the functions of a protein. The inter-residue contact map which measures the pairwise proximity between all amino acid pairs well characterizes the structural information of a protein. Protein contact prediction (PCP) an important building block of many protein structure related applications, such as protein structure prediction [2, 3], protein complex assembly [4], and protein design [5]. It is so important that it is one of the challenges Critical Assessment of protein Structure Prediction (CASP), which is the "world championship" of the computational structural biology field.
The prevalent PCP approaches were built atop the co-evolution principle that the spatially proximate residues tend to co-evolve to maintain the functions of a protein [6-8]. The existing PCP methods generally infer pairwise coevolutionary patterns from MSAs. Among them, direct coupling analy- sis (DCA) [9] techniques are widely used to obtain cocvolutionary features by fitting Potts models or calculating precision matrix [10, 11]. Subsequently, variety of deep-learning based methods were proposed to leverage the DCA-based features to estimate inter-residue contacts [3, 12-15]. However, DCA techniques only consider single-residue and pairwise statistics of the sequences, ignoring high-order interactions among the residues within sequence.
To mitigate the information loss caused by hand-crafted features (e.g., DCA-based features), some recently proposed methods try to learn residue co-evolutions directly from MSAs [2, 16, 17]. For example, the SOTA of them, CopulaNet [2] learns residue representations from individual sequences and then aggregates these features with equal weights to derive residue co-evolutions. This causes two issues: 1) Inferring residue representations from individual sequences independently is inconsistent with the premise that residue co-evolutions are reflection of collective covariation patterns of numerous homologs [18]: 2) Assigning equal weights to the features coming from different homologs ignores the fact that, the quality of homologs varies a lot because of the existence of non-homologous residues and gaps 20].
In this paper, we propose an attention-based architecture, Co-evolution Transformer (CoT), to address mitigate the issues discussed above. The core component of CoT is the co-evolution attention (CoA) module. Different from the previous methods that focus on extracting information from individual sequences, CoA is capable of incorporating the residue co-evolution patterns derived from all homologous sequences into an attention function to learn residue representations. Moreover, the CoA learns to automatically weight the residue representations learned from different homologs, and then selectively aggregates the features to construct the co-evolution attention map. This design mitigates the influence of hon-hoool information.
CoT dramatically outperforms the baseline methods on two rigorous benchmarks CASPI4 [21] and CAMEO (Continuous Automated Model EvaluatiOn) [22]. In particular, CoT achieves 51.6% top-L. long-range precision score for the Free Modeling (FM) domains on the CASP14 benchmark, which outperforms the winner group of CASPI4 contact prediction challenge by 9.8%. Our code will be released at /tttt:://ggiii wwt./otooo/eeeeeeeeeeeeeeeeeeeee http://ccoooott//ooteeoooddddadd http://wnnnnnntoo/eeeeeeeeeeeeee //re/maiii main/ coevolution_to tr sf ormer.
2 Related Work
Protein contact prediction is a binary classification task for amino acid residue pairs. A residue pair is called a contact if their distance is less than or equal to a distance threshold, typically SÅ, i.c., 8 x 10 10 m.
As widely acknowledged, co-evolution information is closely correlated to the contacts. In order to extract the co-evolution patterns for residue pairs, multiple sequence alignments (MSAs) are generated from raw protein sequences. For a target protein sequence, a generated alignment consists of multiple sequences with each corresponding position being an aligned residue or a gap (annotated by a dash), and these aligned sequences as a whole are called multiple sequence alignment. Many protein databases as well as various search schemes have been proposed to generate MSAs efficiently [19, 20]. There are roughly three categories of methods for predicting contacts from MSAs, namely, unsupervised methods, supervised methods, and pre- pre-training based methods.

---

Unsupervised Methods To quantify the strength of the direct relation between the residue pairs of a protein sequence while excluding effects from other residues, many statistical modeling methods based on direct coupling analysis (DCA) have been proposed to fit Potts models [10] or precision matrix [11] to MSAs, e.g., mean-field DCA [9], sparse inverse covariance [11] and pseudo-likelihood maximization [10, 23, 24], to name few. These methods further exploit dedicated scores based on DCA for contact prediction. However, they only consider single-residue and pairwise statistics of the sequences, thus fail to capture high-order interactions among the multiple residues within a sequence.
Supervised Methods By taking DCA-derived scores as features, deep neural networks based supervised methods significantly outperform the unsupervised methods [3, 12-15]. However, the information lost by the DCA-based features from the sequences is still not recoverable. To mitigate this issue, several models are proposed to learn residue co-evolution information directly from the sequences in the MSAs [2. 16, 17, 25J. Among them, CopulaNet [2], the SOTA of the CASPI3 benchmark, derives coevolutionary features differentially by aggregating the learned residue rep- resentations from the sequences. However, although they are capable of modeling the high-order interactions among the multiple residues within single sequence, the global information carried by the MSAs is ignored because they still model the protein sequences independently. AlphaFold2 [25] is claimed to be modeling the full MSAs, achieving an amazing performance on the CASP14 14 structure prediction task. Although its performance not participate in the CASP14 14 PCP task meanwhile no further details of their methods are publicly available.
the structure prediction task is remarkable, they did
Pre-training Based Methods Following the pre-train fine-tune paradigm, pre-trained language models are adapted to representation learning for single protein sequences from the unlabeled data [26-31]. Many of them take contact prediction an important downstream task to validate their performance. While these methods show another solution to this task, they are still at an early stage thus cannot achieve comparable performance to the SOTA approaches currently. To further improve the performance, a pre-trained language model named MSA Transformer is proposed to learn better MSA representation directly [29]. MSA Transformer did solid work on learning evolution information from unlabeled MSAs. However, non-homologous subsequences are inevitably introduced during the learning process.
To exploit the co-evolution information from the full MSAs effectively, our proposed CoT model is built upon co-evolution attention, novel attention mechanism dedicated to incorporate the co- evolution information directly from MSAs in a supervised way.
3 Co-evolution Transformer
Co-evolution Transformer (CoT) is constructed by stacking several repeated CoT layers, Each CoT layer is composed of two attention modules, i.e., co-evolution attention (CoA) module and self-attention module, as shown in Figure 1.
Given a prepared MSA, the stacked CoT layers are used to learn the residue representations. Residue co-evolutions are derived from the representations of the final layer and further employed to estimate the inter-residue contacts.
In this section, we first start with brief introduction to the vanilla Transformer Encoder, followed by the detailed descriptions of each CoT component. To better illustrate the co-evolution attention mechanism, the proposed co-evolution attention mechanism and the self-attention mechanism discussed in the end.
3.1 Vanilla Transformer Encoder
Transformer is a network architecture built on the attention mechanism for machine translation. The Transformer encoder widely adopted for machine learning tasks due to its excellent feature extraction capability when modeling long-distance interactions in sequences [32-34]. Each Trans- former encoder layer consists of two modules, i.e., multi-head self-attention (MHSA) module and position-wise fully connected feed-forward (FFN) module. To connect these two modules, residual
3

---

Residue Co-evolutons
Co-evolution Transformer Layer
Contact Map
Input MSA G
R
Q
Q
G G
W
R Q P
G
K K
L
E E E.
W
Co-evolution Attention
R Q
G
G
L
Self-attention
K Q
K
S
C R
N
Compatibility Function of co-evolution Attention
Residue
Embedding
Co-evolutions
Selective Pooling
Conv
Outer Product
Figure An Overview Co-evolution Transformer. Given an input MSA, several CoT layers stacked to learn residue representations, which are then used to derive residue co-evolutions to estimate inter-residue contacts. Each CoT layer consists of a co-evolution Attention (CoA) module and self-attention module.
connections [35] and layer normalization (LAYERNORM) [36] are applied as below:
LAYERNORM(Z + Mz)A(z)),
(I)
I LAYERNORM(;: + FFN())),
(2)
The multi-head self-attention module is also adopted by the CoT layer as component to learn the residue representations of different sequences in the MSA, however, iterating through the columns instead of rows/scquences.
3.2 Co-evolution Attention Module
For a target protein sequence and its MSA, the goal of the Co-evolution Attention Module to leverage the whole MSA to derive pairwise inter-residue interaction features, namely residue co-evolutions. Intuitively, the residue co-evolutions are analogous to covariance matrix in mathematies, depicting the correlations between any two residue positions (two columns in the MSA). Then, these features are used as attention maps to guide representation learning each sequence in the MSA. To achieve this goal, CoA module employs two consequent submodules, i.c. co-evolution aggregation and co-evolution enhancement. The co-evolution aggregation submodule is designed to generate the coevolutionary features by aggregating pairwise interactions from all homologs in the MSA, while the co-evolution enhancement submodule further enhances the coevolutionary features and derives the final co-evolution attention. The overview of the CoA module is illustrated in Figure 2.
Given the target protein sequence (r.,)2... where L is the length the sequence, the corre- sponding representation of its MSA denoted as X k dxdmudel where K is the number of homologous sequences and dmodet is the hidden dimension of the residue representation. A detailed representation schema of the MSA is described in Appendix.
For the k-th sequence in the MSA, the overall procedure of the CoA module in each layer can be summarized follows:
X = LAYERNORM + COATTN(X)),
(3)
x LAYERNORM + FFN(X*))
(4)

---

Key Residue
Painspecific Sequence Weights (Sh)
Linear Outer product
SoftMaa ax
Query Residue
S elecctvve Pooling
Conv
x. L
Linear Outer Product
72
MSA Embedding
Aggregated Ca-evolution Map
Enhanced Co-evolution Map
LXL
Co-evolution Aggregation
Co-evolution Enhancement
Figure Schema of Co-evolution Attention. Given residue pair (ra, rz). we first learn K covariance features and set of pair-specific sequence weights Sty. k 1... K. The weights used to selectively aggregate the K features. Finally, the aggregated feature is fed to convolutional module obtain the co-evolution attention map. For better visualization, omit the feature channels and the details about how co-evolution attention attend to residue embeddings.
where & E [1... K] and COATTN is defined as: COATTN(X) = CoNcAT(bead, ..... head head, ATTNN A) X W..
where h [l... (f]i is the head index, A Lx the co-evolution feature map generated with deo as its feature dimension, ATTN:(X, X A) RLXL is the h-th co-evolution attention head, X E RLxd, is the /-th segment of X* and W,, E RRdoxd are learnable weights (d, = dmodet/ H). Note that the co-evolution attention ATTN, (X, A) is shared by all the K homologous sequences within the same head.
Submodule 1: Co-evolution Aggregation To depict inter-residue interactions, the co-evolution aggregation submodule exploits outer product on the residue representations. A selective pooling operation is then applied the aggregated outer products from all homologous sequences, the disturbance of non-homologous information can be greatly reduced by this weighting mechanism.
Two tensors P. Q E R K Ixddmode generated from X by separate linear projections. For the residue pair (r., in the k-th sequence of the MSA, the pair co-evolution features Aiy are calculated by:
Σ st o (P. & p
Aij PROJ
and
S - exp(Q) Ot),
where @ is the outer product operator, C is the element- wise multiplication operator, PROJ is a flatten operation followed by a linear projection, converting the aggregated pair co-evolution features to A Lxxxxx. and Z stands for the normalization factor. The weight for the pair representation is denoted as St. where Stz E Redmodet (elmodelddmadd Xdmadet whose elements fall in the range of [0, 1]. The weights normalized over the sequences, thus we have Σx St 1.
Submodule 2: Co-evolution Enhancement To model the high-order interactions among multiple residues, the co-evolution map A is concatenated with A' from the previous CoT layer, the values
5

---

fed into a convolutional module (CONV) to generate an enhanced co-evolution map by:
A = CoNv([A'; A)).
(9)
In practice, we adopt ResNet [35] as the convolution module. Note that we simply assign A' an all-zero tensor for the first layer.
The h-th co-evolution attention head is projected from A by
(X, A) = SOFTMAX M,).
(10)
where Ma E RRd are learnable weights.
3.3 Why Co-evolution Attention
straightforward way model the co-evolution is by applying self-attention individual sequences in the MSA. In this case, for a residue pair (ri, rÿ) in the k-th homolog, self-attention measures their compatibility (attention weight, AWr by:
A x (TX
(11)
where U, V are learnable weights.
In comparison, the attention weight of CoA, calculated by the co-evolution aggregation module (AGGRegATE) and the co-evolution enhancement module (ENHANCE), can be summarized as:
AÚ = AGGREGATE & Xj), Ai ENHANCE(AAj NEIGHBOR(i, f)),
(12)
(13)
Ar x SOFTMAX(A;].
(14)
Compare with self-attention, co-evolution attention more expressive at three aspects: 1) CoA leverages the global information from all the homologs instead of single homolog to derive the attention, which fits better to the residue co-evolution insight; CoA handles the non-homologous information naturally by the selective pooling operation during aggregation, providing solution to widely existed but inevitable dilemma for MSAs; 3) CoA enhances the co-evolution signal by propagating information from the neighbors, making it easier to capture the high-order interactions among multiple residues.
4
Experimental Evaluation
We have conducted extensive experiments and analyses to evaluate the effectiveness of CoT.
4.1 Experiment Setup
Benchmark Two standard benchmarks are used for the model evaluation in the conducted ex- periments, i.e., CASP14 and CAMEO. 1) CASP14 is the latest and most important benchmark protein contact prediction [21]. This benchmark includes three kinds of protein domains, i.e., FM (22 domains), FM/TBM (14 domains), and TBM (50 domains), where domain is protein sequence prepared by the CASP organizers. The first protein among them was released on May. 18, 2020. 2) CAMEO is another benchmark to evaluate weekly-updated protein structure submissions continuously [22]. The proteins are classified into three categories: easy, medium, and hard. Among them, 176 hard targets, released in the last year (from 2020-04-1 to 2021 1021-00 1021--0-1), selected for model evaluation.
Dataset All models are trained on 96, 167 protein structures (chains) collected from PDB [37] (before Apr. 1, 2020), which are split into the train and validation sets (95, 667 and 500 proteins, respectively). For each protein sequence, we generate its MSA by searching UniRef30 (version 2020-02). UniRef90 (version 2020-02), BFD30 (version 2019-03), MGnify90 (version 2019-05) with HHHblts (version 3.3.0) and HMMER (version 3.3.2). The details of the MSA generation procedure described in Appendix.

---

Table 1: Comparison on CASP14 and CAMEO (Precision@L))
CASP14
CAMEO
Methods
FM (22)
FM/TBM (14)
TBM (50)
Hard (176)
RaptorX [14]
33.9
58.1
63.1
53.2
trRosetta [15]
31.3
57.6
61.1
50.1
CopulaNet [2]
38.5
62.2
65.5
56.5
COT-SA (ours)
41.8
59.2
67.9
59.8
CoT (ours)
48.2
66.7
75.6
66.6
Table 2: Comparison on CASPI4, Gr. 368, Gr. 488, and Gr. 010 are the results the top-3 groups in the CASP14 challenge. CoT refers to the results of CoT with MSA selection.
FM (22)
FM/TBM (14)
TBM (50)
Method
L
L/2
L/5
L
L/2
L/5
L
L/2
L/5
Gr. 368
41.8
55.7
66.6
64.5
78.6
87.4
73.1
87.1
94.5
Gr. 488
40.4
52.9
65.0
63.6
78.8
88.5
72.0
86.9
93.7
Gr. 010
39.6
53.4
63.8
61.5
77.0
86.8
66. 1
80.9
89.5
CoT (ours)
51.6
68.2
79.9
66.8
82.2
90.5
77.9
91.0
96.1
Following the procedure of trRosetta [15], the contact prediction task converted into multi-class classification task. The inter-residue distance range is divided into 37 bins, i.e., (OA, 2.5 Å}. (2.5 AI. (20.0 Å, +oo), while the models are trained with the bin labels. For contact, the summed probability value of the bins with distance less than 8 Å are used the final prediction.
Evaluation
Metrics For the evaluation criterion, the prevalent metrics employed, which Precision L, Precision@L/9 and Precision@L/S of long-range residue contacts, where Precision@n stands for the precision score for the top-n pairs of the highest probability in the predicted contact map. Here, refers to the length of protein sequence and long-range means there are at least 23 other residues between these two residues the sequence.
Implementations Given an MSA, 256 sequences are randomly sampled and cropped by length 200. The CoT model is equipped with CoT layers with hidden size 128 and the attention head number as 8. All models are trained with Adam optimizer [38] via a cross-entropy loss for 100k iterations. The learning rate, the weight decay, and the batch size are set to 10-1 0.01, and respectively. The hyperparameters of the CoT model is selected according to the performance the validation set, and a detailed comparison of different hyperparameter settings are summarized in Appendix. The total training cost of the CoT model is about 30 hours on Tesla V100 GPU cards.
4.2 Evaluation Results
For the sake of fairness, CoT is compared with three SOTA methods for contact prediction, including RaptorX [14], trRosetta [15] and CopulaNet [2], the identical MSAs by searching BFD30 using HHblits with default parameters. As shown in Table 1. CoT outperforms the baselines. while surpassing CopulaNet, the best of the SOTAs, by 9.7%, 4.5%. 10.1% and 10.1% for @recision L scores on four kinds of targets, respectively. MSA Transformer [29] not included in the comparison due to the different task settings (pretrained vs. supervised) and the unavailability of its supervised fine- tuning codebase. Nevertheless, we rerun the CoT model on the CASP13-FF dataset to compare with it, obtaining 65.0% Precision L score, which is better than 57.1% reported in MSA Transformer. detailed discussion is described in Appendix.
Methods used by the groups participated in the CASP14 challenge are considered to the best for the benchmark due to their deep optimization towards MSA generation and model ensemble strategy. To further evaluate the performance of the proposed method, the top-3 winner groups/methods
7

---

30
R
40
60
40
8
50
100
100
100
100
100
6
40
60
#b
20
60 60
10c
(a) Self-attention
(b) CoA wi Enhan.
(c) CoA
(d) Ground-truth Contacts
Figure 3: Comparison of the extracted attention maps for 4q2z_H. H. (a) w/o co-evolution enhancement; (c) CoT model: (d) Ground-truth contact map. The red circle covers typical long-range contacts.
CoT-SA model; (b) CoT model
Table 3: Ablations for CoA on CASP14 (Precision@ L). AGGRE., ENHAN. and SA refer to the co-evolution aggregation submodule, the co-evolution enhancement submodule, and the self-attention module, respectively.
CASP14
CAMEO
AGGRE.
ENHAN.
SA
FM (22)
FM/TBM (14)
TBM (50)
Hard (176)
Selective Pooling
48.2
66.7
75.6
66.6
Average Pooling
V
42.7
62.2
73.6
64.2
Selective Pooling
41.6
61.2
70.3
61.8
Selective Pooling
46.4
66.9
74.8
66.3
CASP14 are compared with COT. Different groups use different data sources to build their MSAs. To climinate the variance different data sources, the most confident prediction (the prediction with the highest probability score) of CoT on 12 MSAs from different databases with different search settings, denoted as CoT', is selected as the final prediction.
As shown in Table 2, CoT' outperforms the best method. On the hardest domains (CASP14-FM), CoT even increases 9.8%, 12.5%. 13.3% scores for all the three metrics, compared with Gr. 368, the best group on this benchmark.
4.3 Comparison with Self-attention
To compare co-evolution attention with self-attention, a CoT model variant where the CoA module replaced by the self-attention module is implemented (denoted CoT-SA). Both CoT and CoT-SA are evaluated on the CASP14 and CAMEO benchmarks. As shown in Table 1, the CoT outperforms CoT-SA by 6.4%, 7.5%. 7.7%, and 6.8% Precision scores on four kinds of targets, indicating that CoA is much better than self-attention for this task. To further understand the co-evolution attention, the two attention matrices for both models are visualized. As shown in Figure (a) and (c), the co-evolution attention patterns are much closer to the ground-trute contact map than that of self-attention, illustrating that CoA more effective in extracting contact patterns.
4.4 Ablation Study
To evaluate the contribution of each model component, we set up ablative configurations for the co-evolution aggregation submodule, the enhancement submodule, and the self-attention module.
Experiments with various ablative configurations are conducted listed in Table 3. For the co- evolution aggregation submodule, an average pooling is implemented as an alternative of selective pooling, where the features of different sequences are aggregated equally. The result shows that selective pooling obviously plays critical role in the model, as the Precision@I score of the models with selective pooling on FM domains increased from 42.7% to 48.2% 2% compared with that with average pooling.

---

744
T55
7105
1127
Tagar
ooqqueeee weights
Log. d'squenne weights
4.17
4.5
1.01
3.56
15
29
15
29
Sequence (b)
Sequenie
(c)
(a) Part of MSA for T1061-D1 (residue 40-130), where the purple region represents aligned residues and '-' stands for gap. (b) The sequence weight distribution for the pair (144, T55). (c) The sequence weight distribution for the pair (T1o5, T127).
Figure 4: Learned sequence weights for T1061-D1.
Figure (a) is an MSA for the CASP14 target T1061-D1. Since the model is not sensitive to MSA sequence order, we rearrange the MSA for better visualization. Then we visualize the sequence weights CoT learned from the MSA for two selected residue pairs (r44, rss) and (ries, '127). Figure (b) and (c) show that selective pooling assigns larger weights to more homologous sequences as expected. Meanwhile, the difference between two sequence weight distributions of the two pairs demonstrates that the model able learn pair-specific weights.
The co-evolution enhancement is a convolutional module to learn correlations among multiple residues, and the overall performance reduces significantly FM domains from 48.2% to 41.6% when removing this module, as compared in Table 3. These results illustrate that the high-order residue interaction is very important for contact prediction, as the interaction might reflect some structure patterns (c.g, structure motifs) [14, 391. Moreover, as illustrated in Figure 3, by applying co-evolution enhancement, long-range residue interactions can better extracted.
By ablating the self-attention module, the overall model performance drops slightly, indicating that the CoA module is the main contributor to achieve the performance.
4.5 Model Analysis
The Effeet of MSA Quality The quality of predicted contacts is highly related to Mery [40], which number the non-redundant sequence homologs in the MSA (sequences more than 70% identity is considered redundant). For the CASP14 14 FM domains, the correlation coefficients between the logarithm of Mefs and the Precision@ L scores is 0.63, demonstrating that the performance of contact prediction is strongly correlated to MSA quality. CoT cannot predict the contact maps of some proteins very well (c.g.. the Precision@11 L score T1093-D1 is only 20.6%). For most of these proteins, the Mefs is small less than 20, indicating that low-quality MSAs still bottleneck for contact prediction.
The Effect of Training Data Most methods generate the training data from the same data sources, ie.. public protein sequence databases (UniProt, Metagenome database) and structure databases

---

(PDB). However, different methods customize the data processing pipelines (e.g.. MSA generation strategies), causing slightly different training data for the models. To study the effect of training data, we train another model Co on a smaller training dataset but of comparable size to that used by CopulaNet [2], i.e., PDB30. The performance of CoT* is slightly lower than CoT, e.g., 46.3% vs. 48.2% for Precision@L. on the CASP14 FM domains and 66. 1% vs. 66.6% on the CAMEO hard targets. But it still outperforms CopulaNet a lot, for example, 46.3% vs. 38.2% 2% for Precision on CASP14 FM domains. These results demonstrate that the performance gain of CoT is not mainly achieved by the larger size of the training data.
Conclusion
We propose an attention-based architecture (CoT) to learn residue coevolutions from multiple sequence alignments (MSAs). As the core component CoT, co-evolution attention (CoA) leverages the full information of an MSA to learn residue representations by treating coevolutionary patterns as attention. Moreover, it employs a selective pooling operation to mitigate the influence of non- homologous information. The experimental results two rigorous benchmarks demonstrate the effectiveness of COT.
On the other hand, the experimental results also reveal failure case of CoT, i.e., CoT cannot accurately predict the contacts of proteins with low-depth MSAs. This is an issue shared by other existing approaches as well. How to address the issue caused by low-quality MSAs remains an open problem. We believe pretrained protein models may be a potential solution to it [26-30].
Acknowledgements
We thank all the anonymous reviewers for their valuable comments. The work was supported in part with National Key R&D Program of China Grant 2017YFA0700800.
References
[1] David Whitford. Proteins: structure and function. John Wiley & Sons, 2013.
[2] Fusong Ju, Jianwei Zhu, Bin Shao, Lupeng Kong, Tie-Yan Liu, Wei-Mou Zheng, and Dongbo Bu. CopulaNet: Learning residue co-evolution directly from multiple sequence alignment for protein structure prediction. Nature Communicationss 12(1):1-9, 2021.
[3] Andrew W Senior, Richard Evans, John Jumper. James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(779))706-710, 2020.
[4] Yumeng Yan and Sheng-You Huang. Accurate prediction of inter-protein residue-residue contacts for homo-oligomeric protein complexes. Briefings in Bioinformatics, 2021.
[5] Ivan Anishchenko, Tamuka Martin Chidyausiku, Sergey Ovchinnikov, Samuel J Pellock, and David Baker. De novo protein design by deep network hallucination. bioRxiv, 2020.
[6] Thomas A Hopf, Lucy Colwell, Robert Sheridan, Burkhard Rost, Chris Sander, and Debora S Marks. diree-dimensional structures of membrane proteins from genomic sequencing. Cell, 14917)11607-162,, 2012.
[7] Debora S Marks, Thomas A Hopf, and Chris Sander. Protein structure prediction from sequence variation. Nature biotechnology, 30(1 1):1022-1000 2012.
[8] Qi Wu, Zhenling Peng, Ivan Anishchenko, Qian Cong, David Baker, and Jianyi Yang. Protein contact prediction using metagenome sequence data and residual neural networks. Bioinformat- ics, 36(1):41 2020.
[9] Faruck Moreos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora Marks, Chris Sander, Riccardo Zecchina, José N Onuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, E1893)) 2011.
10

---

[10] Magnus Ekeberg, Cecilia Lövkvist, Yueheng Lan, Martin Weigt, and Erik Aurell. Improved contact prediction proteins: using pseudolikelihoods to infer potts models. Physical Review E, 87(1):017777 2013.
[11] David T Jones, Daniel WA Buchan, Domenico Cozzetto, and Massimiliano Pontil. PSICOV: precise structural contact prediction using sparse inverse covariance estimation on large multiple sequence alignments. Bioinformatics, 28(2): 184-190, 2012.
[12] David T Jones and Shaun M Kandathil. High precision in protein contact prediction using fully convolutional neural networks and minimal sequence features. Bioinformatics, 34(19):3308- 3315, 2018.
[13] Sheng Wang, Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLoS computational biology, 13())=e1005324 2017.
[14) Jinbo Xu. Distance-base based protein folding powered by deep learning. Proceedings of the National Academy of Sciences, 685634)118566-6665, 2019.
[15] Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, (113)))446-15000 2020.
[16] Shaun M Kandathil, Joe Greener, Andy M Lau, and David T Jones. Deep learning- based prediction of protein structure using learned representations of multiple sequence alignments. bioRviv, 2020.
[17] Claudio Mirabello and Björn Wallner. rawMSA: End-to-end deep learning using raw multiple sequence alignments. PloS one, 022200222 2019.
[18] Adam Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816 822, 2018
[19] Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. HHblits: lightning- fast iterative protein sequence searching by mmm-hmm alignment. Nature methods, 9(2):733- 175, 2012.
[20] Julie Thompson, Benjamin Linard, Odile Lecompte, and Olivier Poch. comprehensive benchmark study of multiple sequence alignment methods: current challenges and future perspectives. PloS one, 6(3)e18093, 2011.
[21] Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (CASP)- Round XIII. Proteins: Structure, Function, and Bioinformatics, 17(12):0011-1020 2019.
[22] Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in casp12. Proteins: Structure, Function, and Bioinformatics, 86:387-398, 2018.
[23] Sivaraman Balakrishnan, Hetunandan Kamisetty, Jaime G Carbonell, Su-In Lee, and Christo- pher James Langmead. Learning generative models for protein fold families. Proteins: Structure, Function, and Bioinformatics, 79(4): (144):061--1758 2011.
[24] Stefan Seemayer, Markus Gruber, and Johannes Söding. CCMpred foapred-fastta and precise prediction of protein residue-residue contacts from correlated mutations. Bioinformatics, 30(21):3128- 3130, 2014.
[25] John Jumper, Evans, A Pritzel, T Green, M Figurnov, Tunyasuvunakool, Ronneberger, R Bates, A Zidek, A Bridgland, et al. High accuracy protein structure prediction using deep learning. Fourleenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book), 22:24, 2020.

---

[26] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer. Debsindhu Bhowmik, et al. ProfTrans: Towards cracking the language of life's s code through self supervied deep learning and high performance computing. arXiv preprint 200iv:2007.06225, 2020.
[27] Roshan Rao, Nicholas Bhattaachara, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. Advances in Neural Information Processing Systems, 32:9689, 2019.
[28] Roshan Rao, Sergey Ovchinnikov, Joshua Meier, Alexander Rives, and Tom Sercu. Transformer protein language models are unsupervised structure learners. bioRxiv, 2020.
[29] Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. bioRxiv, 2021.
[30] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 11((15), 2021.
[31] Pascal Sturmfels, Jesse Vig. Ali Madani, and Nazneen Fatema Rajani. Profile predic- tion: An alignment-based pre-training task for protein sequence models. arXiv preprint arXiv:2012.00195, 2020.
[32] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. arXiv preprint arXiv:1506.00503, 2015.
[33] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Putlern Recognition, pages 244-253, 2019.
[34] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey visual transformer. arXiv preprint arXiv:2012. 12556, 2020.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[36] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607. 064500 2016.
[37] Stephen Burley, Helen M Berman, Charmi Bhikadiya, Chunxiao Bi, Li Chen, Luigi Di Costanzo, Cole Christie, Ken Dalenberg, Jose M Duarte, Shuchismita Dutta, et al. RCSB Protein Data Bank: biological macromolecular structures enabling research and education in fundamental biology, biomedicine, biotechnology and energy. Nucleic acids research, 47(D1):4644-4474, 2019.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[39] Qi Zhang, Jianwei Zhu, Fusong Ju, Lupeng Kong, Shiwei Sun, Wei-Mou Zheng, and Dongbo Bu. ISSEC: inferring contacts among protein secondary structure elements using deep object detection. BMC bioinformatics, 21(1):1-13, 2020.
[40] Zhiyong Wang and Jinbo Xu. Predicting protein contact map using evolutionary and physical constraints by integer programming. Bioinformatics, 2911)::266-i2773 2013.
12


Reviews:
[{"review_id": "z0OyyFI5wTY", "reviewer": "Reviewer_WfQv", "summary": "This manuscript describes a novel deep learning architecture for predicting protein contact maps. This is a well-studied problem for which deep models already provide the state of the art.  The innovation here is to use a particular type of Transformer architecture that models more dependencies than previous models.  The approach is well described, seems sensible, and gives good performance on a challenging benchmark. On the other hand, the empirical results are unsatisfying in some respects (outlined below), and the benchmarking suffers from a problem (different methods trained on different training sets).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key idea of this work is that, in predicting co-evolutionary information for contact prediction, it is important not only to weight sequences non-uniformly and to capture dependencies between residues within each sequence, as has been done by previous model, but also to capture dependencies between sequences in the multiple sequence alignment (the \"co-evolution enhancement\" model).  The model architecture is well described in Figures 1-2 and Section 3, though I wish the architecture of CopulaNet had been explained in a bit more detail, so as to highlight more clearly which aspects of the proposed model are novel.\n\nThe text makes three primary claims (lines 173-179) about what makes this model novel: \"1) COA leverages the global information from all the homologs instead of single homolog to derive the attention, which fits better to the residue co-evolution insight; 2) COA handles the non-homologous information naturally by the selective pooling operation during aggregation, providing a solution\nto a widely existed but inevitable dilemma for MSAs; 3) COA enhances the co-evolution signal by\npropagating information from the neighbors, making it easier to capture the high-order interactions\namong multiple residues.\"   I would have liked to see each of these claims directly supported by empirical evidence in Section 4.  The results in Sections 4.3 and 4.4 go partway in this direction but do not directly test the above hypotheses.  For instance, 4.3 shows that the proposed co-evolution layers give better results overall than a standard self-attention layer but does not delve into the source of this improvement. \n\nThe results in Table 1 are problematic because the different methods are not trained on the same data.  I actually assumed that they were, at first, and only figured out for sure that they were not when I got to Section 4.5 (line 273).  The text should say this explicitly up front, since it makes the performance results reported in Table 1 less trustworthy.  We don't know whether the differences in performance are due to differences in the training set or due to modeling choices.  This is, in my opinion, a major problem with the reported results, though apparently it's a problem in the field as a whole (\"... most state-of-the-art models are trained on their private training data, which are usually unavailable.\").  The results on lines 275-279 go partway toward mitigating this problem.\n\nline 220: \"The improvement on FM/TBM domains (0:8%) is not that significant as the others (> 9:0%).\" In fact, no statistical significance was assessed for any of these results. This should be relatively straightforward to do by, e.g., computing the performance measures separately per protein and then doing a statistical test (e.g., paired t-test) on those measures.\n\nI could not make sense of these sentences: \"Different groups use different data sources to build their MSAs. To eliminate the variance of different data sources, the most confident prediction of CoT on 12 MSAs from different databases with different search settings, denoted as CoTy, is selected as the final prediction.\"  It sounds like we are comparing models trained on different training sets.  How does the proposed strategy address this problem?\n\nWe are told that \"Figure 4 (b) and (c) show that selective pooling assigns larger weights to more relevant sequences as expected,\" but I am not sure what \"more relevant\" means here.  The plot seem to show only that the weights are not uniform.\n\nFigure 3 provides anecdotal evidence that including the co-evolution enhancement module improves prediction of long-range contacts. It would have been better to systematically measure this in a quantitative way.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "UWqYlM2Aw0e", "reviewer": "Reviewer_qdTM", "summary": "The authors present a model that predicts contact maps from multiple sequence alignments. The paper proposes a weighted sum of k, LxL attention maps (k=#sequences, L=sequence length), where a per-residue-pair weight is learned across all sequences in the MSA. Afterwards a convolutional layer is applied to refine the prediction. The authors show that the co-evolution transformer outperforms existing supervised contact prediction methods on CASP14.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of computing per-residue-pair weight across all sequences in the MSA is the main contribution of the paper. The general concept of attention maps for contacts and convolutional layers for refinement are well-known techniques in the field. The authors have clearly demonstrated the uniqueness and contribution of their idea through proper ablations. Related work is adequately cited. \n\nThe explanation of the co-evolution transformer is sound. One detail the authors should clarify in Line 161 (submodule 2) is when “the co-evolution map A is concatenated with A' from the previous CoT layer”, how is this handled in the first CoT layer when A’ does not yet exist? As a side question, why did the authors not additionally try a ResNet for the co-evolution enhancement? The expectation is that this would perform at least as well as the conv-linear-softmax block proposed in the paper.\n\nThere seems to be little comparison of Co-evolution Transformer performance on the different MSA generation methods (different databases). This is an interesting topic, and it is not clear why the authors did not mention it in the main paper.\n\nMost claims are well supported, but two are suspect.\nLine 97: [Pretraining-based methods] are still at an early stage thus cannot achieve comparable performance to the SOTA approaches currently.\nLine 104: [MSA Transformer] it still cannot well extract the co-evolution information due to the non-homologous subsequences inevitably introduced by the generated MSAs.\n\nThese claims don't seem to be correct. Re the claim on line 97, MSA Transformer (Rao, et al. 2021) shows SOTA contact prediction on CASP13 free modeling targets relative to trRosetta. As such, it’s not clear why this model wasn’t included in the comparisons. Re the claim on line 104, Rao, et al, 2021 show the model extracts co-evolution information from MSAs through experiments with unsupervised contact prediction. These claims should be revised, and the paper would be strengthened by a comparison.\n\n50.4 Precision @ L on CASP14 free modeling targets compared to 41.8 is compelling. When comparing to the known methods (RaptorX, TrRosetta, CopulaNet), the authors should clarify whether the precision scores from the baseline methods are based on predicted contact maps OR the contact maps of predicted 3D structures. \n\nAs the current field stands, there is no single accepted way on how deep learning models should process MSAs. This paper presents a unique approach that merits consideration from the broader community. \n\nAdditional notes: Please define how you determine “the most confident prediction” [line 226]\n\n---------------------------------\nEDIT: See discussion around \"Training data and comparisons\". Authors have agreed to address the temporal split in the revision, which will strengthen the comparison to other methods.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the main limitation of their work, namely that the co-evolution transformer does not perform well on low-depth MSAs. Solving this problem is beyond the scope of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "UN94mPIEMZI", "reviewer": "Reviewer_iKoC", "summary": "The authors address an important problem in structural biology and obtain what looks like state-of-the-art performance.  The proposed method predicts inter-residue contacts of a protein based on its sequence presented as a multiple sequence alignment (MSA).  The method is innovative in the way it represents an MSA using attention on individual sequences within the MSA, avoiding the step of summarizing the MSA as a \"profile\" that summarizes amino acid composition in each position, a common way of representing MSAs.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n- The idea of using attention to selectively focus on individual sequences within the MSA is innovative and interesting.  In their case study the authors demonstrate how this allows the method to focus on sequences for which no gap is present in a given position.\n\n- State-of-the-art performance on a highly important problem in structural biology.\n\nWeaknesses:\n\n- I think the Co-evolution aggregation submodule (equations 7 and 8) and the co-evolution enhancement submodule could have been explained a bit more clearly.\n\n\nComments:\n\n- If there is a gap in the original sequence in the MSA, that position  cannot be a part of any interactions.  It is not clear how this is addressed in the method.\n\n- It is not clear how the MSA is provided as input.  Given my understanding of the model, I assume the input is a one hot encoding of each sequence in the MSA.  That step is also missing in Figure 1.\n\n- Section 3.3 compares the expressivity of the proposed CoT layer with a simpler attention model over individual sequences.  However, in my opinion, a more natural comparison is with a model that summarizes the MSA by a sequence profile.\n\n- It's not clear how the selected model parameters were chosen.  No model selection strategy was described.\n\nMinor comments:\n\nPlease discuss the following method, as it also uses attention:\nChen, C., Wu, T., Guo, Z., & Cheng, J. (2021). Combination of deep neural network with attention mechanism enhances the explainability of protein contact prediction. Proteins: Structure, Function, and Bioinformatics, 89(6), 697-707.\nI suspect it does not perform as well as the proposed method since its performance is similar to RaptorX.\n\n- The following statement is incorrect:  \"Transformer is a network architecture built on the self-attention mechanism\" - transformers are constructed on the basis of attention rather than self-attention.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper wishes to address the issue of protein contact prediction problem, State of the art approach uses hand-crafted features through Multiple Sequence Alignments(MSAs). This paper proposes a Transformer architecture to automatically identify features suitable for Protein Contact Prediction problem. All the referees were unanimous that while the ideas are novel but their comparison with the state of the art 
is problematic. We hope that the authors can address these concerns in the final manuscript.




Decision: Accept (Poster)