Transformer protein language models are unsupervised structure learners
Authors: Roshan Rao
Joshua Meier
Tom Sercu
Sergey Ovchinnikov
Alexander Rives
Venue: ICLR 2021 Poster (2021)
Keywords: proteins
language modeling
structure prediction
unsupervised learning
explainable

Abstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model.

Published conference paper at ICLR 2021
TRANSFORMER PROTEIN LANGUAGE MODELS ARE UNSUPERVISED STRUCTURE LEARNERS
Roshan Rao"
Joshua Meier
Tom Sercu
UC Berkeley zmrad@beekkleey.edt
Facebook Al Research jmeierêfb. com
Facebook Al Research
tserculfb com
Sergey Ovchinnikov
Alexander Rives
Harvard University solg. ha rvard.
Facebook Al Research & New York University ari arivesêcs.co nyu.edu
ABSTRACT
Unsupervised contact prediction is central uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from set of related sequences. In the past year, protein language models have emerged as po- tential alternative, but performance has fallen short of state-of-the-ae approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state- of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with single forward pass of an end-to-end model
INTRODUCTION
Unsupervised modeling protein contacts has an important role in computational protein de- sign (Russ et al., 2020; Tian et al., 2018; Blazejewski et al.] (2019) and is a central element of all current of-te-of-the-arr structure prediction methods (Wang al. 2017 Senior al., 2020 Yang ataal|2019) The standard bioinformatics pipeline for unsupervised contact prediction includes mul- tiple components with specialized tools and databases that have been developed and optimized over decades. In this work propose replacing the current multi-stage pipeline with single forward pass of a pre-trained end-to-end protein language model.
In the last year, protein language modeling with an unsupervised training objective has been inves- tigated by multiple groups (Rives et al., 2019; Alley et al. 2019; Heinzinger al. 2019 Rao et al., 2019 Madani et al. 2020). The longstanding practice in bioinformatics has been linear models on focused sets of evolutionarily related and aligned sequences; by contrast, protein language model- ing trains nonlinear deep neural networks large databases of evolutionarily diverse and unaligned sequences. High capacity protein language models have been shown learn underlying intrinsic properties proteins such as structure and function from sequence data (Rives et al., (019).
A line of work in this emerging field proposes the Transformer for protein language modeling RRvvss al. 2019 Rao al. 2 Originally developed in the NLP community to represent long range context, the main innovation of the Transformer model its use self-attention (Vaswani et al., 2017). Self-attention has particular relevance for the modeling of protein sequences. Unlike con- volutional or recurrent models, the Transformer constructs pairwise interaction map between all positions in the sequence. principle this mechanism has an ideal form to model protein contacts.
In theory, and-to-end learning with a language model has advantages over the bioinformatics pipeline: (i) it replaces the expensive query, alignment, and training steps with a single forward
"Work performed during an internship Facebook.
Weights for all ESM-1 and ESM-Ib models, as well regressions trained these models be found httptttp////... //gi thub. com/ reseaookkeeerccheee

---

Published conference paper at ICLR 2021
pass, greatly accelerating feature extraction; and (ii) it shares parameters for all protein families, enabling generalization by capturing commonality across millions of evolutionarily diverse and un- related sequences.
We demonstrate that Transformer protein language models learn contacts in the self-attention maps with state-of-the-art performance. We compare ESM- (Rives et al. 2020), a large-scale (650M parameters) Transformer model trained on UniRef50 (Suzek ct al. (007] to the Gremlin (Kamisetty al., 2013) pipeline which implements log linear model trained with pseudolikelihood (Balakr-
ishnan al., 2011] Ekeberg al. 2013). Contacts can extracted from the attention maps
the Transformer model by sparse linear combination of attention heads identified by logistic re- gression. ESM-1b model contacts have higher precision than Gremlin contacts. When ESM and Gremlin are compared with access to the same set sequences the precision gain from protein language model is significant; the advantage holds on average even when Gremlin is given access to optimized set of multiple sequence alignments incorporating metagenomies data.
We find a linear relationship between language modeling perplexity and contact precision. We also find evidence for the value parameter sharing: the ESM-1b model significantly outperforms Gremlin on proteins with low-depth MSAs. Finally we explore the Transformer language model's ability to generate sequences and show that generated sequences preserve contact information.
2 BACKGROUND
Multiple Sequence Alignments (MSAs) A multiple sequence alignment consists of set of evo- lutionarily related protein sequences. Since real protein sequences are likely to have insertions, dele- tions, and substitutions, the sequences are aligned by minimizing a Levenshtein distance-like metric over all the sequences. In practice heuristic alignment schemes are used. Tools like Jackhmmer and HHblits can increase the number and diversity of sequences returned by iteratively performing the search and alignment steps (Johnson al. 2010, Remmert al. 2012).
Metries For protein of length L,, we evaluate the precision of the top L., L/2,, and L/5 contacts for short range (i - jl E [6, 12)), medium range (/i -1l jl E [12, 24)), and long range (i = jl E [24, co)) contacts. We also separately evaluate local contacts (li (li-jl E [3, 6)) for secondary structure prediction in Apppendix A.9 general, all contacts provide information about protein structure and important interactions, with shorter-range contacts being useful for secondary and local structure, while longer range contacts are useful for determining global structure (Taylor et al. 2014).
RELATED WORK
There is a long history of protein contact prediction (Adhikari & Cheng 2016) both from MSAs, and more recently, with protein language models.
Supervised contact prediction Recently, supervised methods using deep learning have resulted breakthrough results in supervised contact prediction (Wang et al. 2017]] Jones & Kandathil] 2018; Yang et al., 2019 Senior et al. 2020) Adhikari & Elofsson (2020). State-of-the art methods use deep residual networks trained with supervision from many protein structures. Inputs are typically covariance statistics (Jones & Kandathil (2018) Adhikari & Elofsson, 2020). or inferred coevolu-
tionary parameters (Wang al. [017]] Liu al. 2018 Senior et al., [020 Yang et al.) 2019). Other recent work with deep learning uses sequences evolutionary features as inputs (AIQuraishi) 2018 Ingraham et al., 2019). Xu et al. (2020) demonstrates the incorporation of coevolutionary features critical to performance of current state-of-the-art methods.
Unsupervised contact prediction In contrast to supervised methods, unsupervised contact pre- diction models are trained on sequences without information from protein structures. In principle this allows them to take advantage of large sequence databases that include information from many sequences where no structural knowledge is available. The main approach has been to learn evolu- tionary constraints among a set of similar sequences by fitting a Markov Random Field (Potts model) to the underlying MSA. technique known as Direct Coupling Analysis (DCA). This was proposed by Lapedes et al. (1999) and reintroduced by Thomas et al. 2008) and Weigt al. 2009)

---

Published conference paper at ICLR 2021
Various methods have been developed fit the underlying Markov Random Field, including mean- field DCA (mfDCA) (Morcos et al. 2011), sparse inverse covariance (PSICOV) (Jones et al., 2 and pseudolikelihood maximization (Balakrishnan et al. 201 Ekkeberg et al. 2013;
et al., 2014). Pseudolikelihood maximization is generally considered sfate-of-the- art for unsupervised con- tact prediction and the Gremlin (Balakrishnan et al. 2011) implementation is used as the baseline throughout. We also provide mfDCA and PSICOV baselines, Recently deep learning methods have also been applied fitting MSAs, and Riesselman al. (2018) found evidence that factors learned by a VAE model may correlate with protein structure.
Structure prediction from contacts While we do not perform structure prediction in this work, many methods have been proposed to extend contact prediction to structure prediction. For example, EVFold (Marks al. 2011) and DCAFold (Sulkowska et 2012) predict co-evolving couplings using a Potts Model and then generate 3D conformations by directly folding an initial conformation with simulated annealing, using the predicted residue-residuc contacts constraints. Similarly, FragFold (Kosciolek Jones 2014) and Rosetta (Ovchinnikov et al. (2016) incorporate constraints from a Pouls Model into a fragment assembly based pipeline. |Senior et al. (2019), use features from Potts model fit with pseudolikelihood maximization to predict pairwise distances with deep residual network and optimize the final structure using Rosetta. All of these works build directly upon the unsupervised contact prediction pipeline.
Contact prediction from protein language models Since the introduction of large scale language
models for natural language processing (Vaswami et al., 2017 Devlin al., 2019), there has been considerable interest in developing similar models for proteins (Alley et al. 2019; Rives et
2019
Heinzinger et al. 2019 Rao et al. 2019 Elnaggar et al. 2020 Lu et al. 2020) Madani et al.
2020 Shen et al. (202T). Rives et al. (2019) were the first to study protein Transformer language
models, demonstrating that information about residue-residue contacts could recovered from the learned representations by linear projections supervised with protein structures. Recently Vig et all] (202)) performed extensive analysis of Transformer attention, identifying correspondences biologically relevant features, and also found that different layers of the model are responsible for learning different features. In particular Vigg et al. (2020) discovered a correlation between self- attention maps and contact patterns, suggesting they could be used for contact prediction.
Prior work benchmarking contact prediction with protein language models has focused on the pervised problem. [Bepler & Berger (2019) were the first to fine-tune an LSTM pretrained on protein sequences to fit contacts. |ao et al. (2019) and Rives et al. (2020) perform benchmarking of mul- tiple protein language models using deep residual network fit with supervised learning on top of pretrained language modeling features.
In contrast to previous work on protein language models, we find that a state-of-the-ar unsupervised contact predictor can be directly extracted from the Transformer self-attention maps. We perform a thorough analysis of the contact predictor. showing relationships between performance and MSA depth as well as language modeling perplexity. We also provide methods for improving performance using sequences from MSA and for sampling sequences in manner that preserves contacts.
MODELS
We compare Transformer models trained on large sequence databases to Potts Models trained on individual MSAs. While Transformers and Potts Models emerged in separate research communities, the two models share core similarities (Wang & Cho 2019) which we exploit here. Our main result is that just as Gremlin directly represents contacts via its pairwise component (the weights), the Transformer also directly represents contacts via its pairwise component (the self-attention).
4.1 OBJECTIVES
For a set of training sequences, X, Gremlin optimizes the following pseudolikelihood loss, where a single position is masked and predicted from its context. Inputs are aligned, so all have length L:
(X;
E x~X
Σ 1
log (egp(l//Tssii (1
(1)

---

Published conference paper at ICLR 2021
Symmetrive APC
Regression
Stop Gratiien
Finetune: Maximize Probabilty
M
Mark
JIN
Sequence
Stractue
Tarahase
Danabase
Pretoening: Predict Mooked Towara
Figure 1: Contact prediction pipeline. The Transformer is first pretrained on sequences from a large database (Uniref50) via Masked Language Modeling. Once finished training. the attention maps extracted, passed through symmetrization and average product correction, then into regression. The regression is trained on a small number (n s 20) proteins to determine which attention heads are informative. At test time, contact prediction from an input sequence can be done entirely on GPU in single forward pass.
The masked language modeling (MLM) loss used by the Transformer models can be seen as a generalization of the Potts Model objective when written as follows:
CMLM(X; = EE R Σ
logp(sa/22gemmm
(2)
<Emmsk
In contrast to Gremlin, the MLM objective applied by protein language modeling is trained on unaligned sequences. The key distinction of MLM is mask and predict multiple positions con- currently, instead masking and predicting one at a time. This enables the model to scale beyond individual MSAs to massive sequence datasets. In practice, the expectation under the masking pat- tern is computed stochastically using single sample each epoch.
4.2 GREMLIN
The log probability optimized by Gremlin is described in section A.3 Contacts are extracted from the pairwise Gremlin parameters by taking the Frobenius norm along the amino acid dimensions, resulting in an I. x L. coupling matrix. Average product correction (APC) is applied to this coupling matrix to determine the final predictions (Appendix A.2).
Gremlin takes MSA input. The quality of the output predictions highly dependent the construction of the MSA. We compare to Gremlin under two conditions. the first condition, we present Gremlin with all MSAs from the trRosetta training set (Yang et al.] (2019). These MS generated from all of Uni and also supplemented with metagenomic sequences when the depth from Uniref100 too low. The trRosetta MSAs are a key ingredient in the state-of-the- art protein folding pipeline. See Yang et al.) (2019) for a discussion on the significant impact of metagenomic sequences the final result. In the second setting, allow Gremlin access only the same information as the ESM Transformers by generating MSAs via Jackhmmer on the ESM training set (a subset of Uniref50). See Appendix A.5 Jackhmmer parameters.
4.3 TRANSFORMERS
evaluate several pre-trained Transformer models, including ESM-1 (Rives et al. 2019). ProtBert-
BFD (Elnaggar et al.) 2020) and the TAPE Transformer (Rao et al. 2019). The key differences between these models the datasets, model sizes, and hyperparameters (major architecture dif- ferences described in Table 3). Liu et al. (2019) previously showed that these changes can have a significant impact on final model performance. In addition to ESM-1, we also evaluate an updated version, FESM-Ib, lb, which is the result of hyperparameter sweep. The differences are described in

---

Published conference paper at ICLR 2021
Table 1: Average precision on 14842 test structures for Transformer models trained on 20 structures.
6
sep
12
12 S sep < 24
24 S sep
Model
L
L/2
L/5
L
L/2
L/5
L
L/2
L/5
Gremlin (ESM Data)
15.2
23.0
37.8
18.1
27.9
44.3
31.3
43.1
55.5
mfDCA (trRosetta Data)
16.3
23.7
35.8
19.7
29.8
45.5
33.0
43.5
54.2
PSICOV (rRosttta trrCoverrRosetta Data)
13.4
23.6
39.2
18.3
28.4
45.7
32.6
45.2
58.1
Gremlin (trRosetta Data)
17.2
26.7
44.4
21.1
33.3
52.3
39.3
52.2
62.8
TАРE
9.9
12.3
16.4
10.0
12.6
16.6
11.2
14.0
17.9
ProtBERT-BFD
20.4
30.7
48.4
24.3
35.5
52.0
34.1
45.0
57.4
ESM-1 layer)
11.0
13.2
15.9
11.5
14.6
19.0
13.2
16.7
21.5
ESM-I (12 layer)
15.2
21.1
30.5
18.1
24.7
34.0
23.7
30.5
39.3
ESM-1 (34 layer)
20.3
30.2
46.0
23.8
34.3
49.2
34.7
44.6
56.0
ESM-1b
21.6
33.2
52.7
26.2
38.6
56.4
41.1
53.3
The Transformer processes inputs through series blocks alternating multi-head self- self-attention and fera-forward layers. In each head of a self-attention layer, the Transformer views the encoded representation as set of query- verry- triples. The output of the head is the result of scaled dot-product attention:
Section A.4
Attention
softmax softmax(QK//vn)
Rather than only computing the attention once, the multi-head approach rums scaled dot-produce attention multiple times in parallel and concatenates the output. Since self-attention explicitly con- structs pairwise interactions (QK") between all positions the sequence, the model can directly represent
interactions. In this work, we demonstrate that the QK" pairwise "self attention maps" indeed capture accurate contacts.
4.4 LOGISTIC REGRESSION
To extract contacts from a Transformer, we first pass the input sequence through the model to obtain the attention maps (one map for each head in each layer). We then symmetrize and apply APC each attention map independently. The resulting maps are passed through an Lı-regularized logistic regression, which is applied independently at each amino acid pair (i, j). At training time, only do not backpropagate through the entire model. At
train the weights of the logistic regression; test time, the entire prediction pipeline can run in a single forward pass, providing a single end-to- end pipeline for protein contact prediction that does not require any retrieval steps from sequence database. See Appendix A.7 full description of the logistic regression setup.
5 RESULTS
evaluate models with the 15051 proteins in the trRosetta training dataset (Yang et al. 2019). removing 43 proteins with sequence length greater than 1024, since ESM-Ib was trained with a context size of 1024. Of these sequences, Jackhmmer fails on 126 when attempt construct MSAS using the ESM training set (see [Appendix A.5). This leaves with 14882 total sequences. We reserve 20 sequences for training, 20 sequences for validation, and 14842 sequences for testing.
Table 1 shows evaluations of Gremlin, ESM-1, ESM-1b well as the TAPE and ProtBERT-BEE BFD models. Confidence intervals within 0.5 percentage points for all statisties in Tables I and In Table all Transformer model contact predictors are trained with logistic regression on 20 protemns We find that with only 20 training proteins ESM-1b has higher precision than Gremlin for short, medium, and long range contacts.
*PSICVV fails converge on 24 sequences using default parameters. Following the suggestion in http:/asppedddeee
thu.... com /ps pred/ psi covl increase p 0.005, 0.01, and thereafter by increments of 0.01, maximum O.I. PSICOV fails converge altogether on 6 14842 sequences. We assign 0 for these sequences.

---

Published conference paper at ICLR 2021
Table 2: ESM-1b Ablations with limited supervision and with MSA information. n is the number of logistic regression training proteins. is the number of sequences ensembled over.
S sep <
12
I sep <
S sep
Model
Variant
L
L/2
L/5
L
L/2
L/5
L
L/2
L/5
ESM Data
15.2
23.0
37.8
18.1
27.9
44.3
31.3
43.1
53.5
Gremlin
trooeetta Data
17.2
26.7
44.4
21.1
33.3
52.3
39.3
52.2
62.8
top-1 heads
16.8
23.4
34.8
19.8
27.6
40.2
29.3
38.1
50.0
top-5 heads
19.2
28.5
44.5
23.3
33.8
49.0
35.0
45.2
57.3
top-10 10 heads
20.0
30.1
47.4
24.7
36.0
52.2
38.5
49.4
61.1
ESM-Ib (Ablations)
n=1, s=1 s=1
19.4
29.7
47.1
25.1
37.1
54.0
39.2
50.6
63.0
n=10, s=1
21.4
32.9
52.3
26.1
38.5
56.4
40.8
52.9
65.7
n=20, s=1
21.6
33.2
52.7
26,2
38.6
56.4
41.1
53.3
66.1
MSA, s=1
18.4
28.1
45.5
23.9
36.1
53.7
39.9
51.3
63.0
n=20, s=16
21.9
33.8
53.6
26.7
39.4
57.5
41.9
54.3
67.3
ESM-1b (s seqs)
n=20, s=32
22.0
34.1
54.0
26.9
39.8
58.1
42.3
54.8
67.8
n=20, s=64
22.1
34.3
54.3
27.1
40.1
58.5
42.6
55.1
68.2
In addition to this set, we also evaluate performance on 15 CASPI3 FM Domains in Appendixx A.6 On average ESM-1B 1b has higher short, medium, and long range precision than Gremlin on all metrics, sequences. and in particular can significantly outperform on MSAs with low effective number We also provide a comparison to the bilinear model proposed by Rives et al. (2020). The logistic regression model achieves billnearr model achieves a long range precision of 20.1, an increase of only 1.5 points despite being trained on 700x more structures.
rangeange contact precision at L of 18.6. while the fully supervised
5.1 ABLATIONS: LIMITING SUPERVISION
While the language modeling objective is fully unsupervised, the logistic regression is trained with a small number supervised examples. In this section, we study the dependence the results on this supervision, providing evidence that the contacts are indeed learned in the unsupervised phase, and the logistic regression is only necessary extract the contacts.
Top Heads Here we use the logistic regression only to determine the most important heads. Once they are selected, we discard the weights from the logistic regression and simply average the atten- tion heads corresponding to the top-k weight values. By taking the single best head from ESM- we come close to Gremlin performance given the same data, and averaging the top-5 heads allows us to outperform Gremlin. Averaging the top-10 10 heads outperforms full logistic regression on all other Transformer models and comes close Gremlin given optimized MSAs.
Low-N The second variation we consider is to limit the number of supervised examples provided the logistic regression. We find that with only a single training example, the model achieves a long range top-L precision 39.2, which statistically indistinguishable from Gremlin (p 0.05). Using only 10 training examples, the model outperforms Gremlin on all the metrics. Since these results depend on the sampled training proteins, also show a bootstrapped performance distribution using 100 different logistic regression models in [Appendix A.10 We find that with protein, performance can vary significantly, with long range precision mean of 35.6, median of 38.4, and standard deviation 8.9. This variation greatly decreases when training on 20 proteins, with long range top-L precision mean of 40.1, median of 41.1, and standard deviation of 0.3. See Fig. 12 for the full distribution on all statistics.
MSA Only Finally, we consider supervising the logistic regression only with MSAs instead of real structures. This is the same training data used by the Gremlin baseline. To do this, we first train Gremlin on each MSA. We take the output couplings from Gremlin and mark the top L couplings with sequence separation in each protein as true contacts, and everything else false contacts,

---

Published conference paper at ICLR 2021
1,00
ESM (6)
Lang Pell
0.3
ESM (12)
0.75
15
FSM (34)
ESM-1b
0.2
ESM lb
0.50
0.25
0.1
0.00
10
0.0
0.5
1.0
0.0
0.5
1.0
Validation Perplexity
Gremlia (ESM Data)
Gremlin (rRosetta Data)
Left: Language modeling validation perplexity on holdout of Unire150 vs. contact pre- cision over the course pre- vre-trannig. ESM-1b was trained with different masking so perplexities between the versions are not comparable. Right: Long range P@L performance distribution ESM-Ib vs. Gremlin. Each point is colored by the log of the number of sequences in the MSA used train Gremlin.
Figure 2:
creating binary decision problem. When trained on MSAs, we find this model achieves a long range P@L of 39.9, and generally achieves similar long range performance to Gremlin, while still having superior short and medium range contact precision.
5.2 ENSEMBLING OVER MSA
Transformer models fundamentally single-sequence models, but further boost perfor- mance by ensembling predictions from multiple sequences in the alignment. To do so, we unalign each sequence in the alignment (removing any gaps), pass the resulting sequence through the Trans- former and regression, and realign the resulting contact maps the original aligned indices. For these experiments, we use the logistic regression weights trained on single-sequence inputs, rather than re-training the logistic regression on multi-sequence inputs. We also simply take the first s sequences in the MSA. Table shows performance improvements from averaging 16, 32, and 64 sequences.
To better understand this result, we return to the single- sequence setting and study the change prediction when switching between sequences in the alignment. We find that contact precision can vary significantly depending on the exact sequence imput the model, and that the initial query sequence the MSA does not necessarily generate the highest contact precision Fig. 9).
Lastly, Alley et al. (2019) presented a method of fine-tuning where a pretrained language model is further trained on the MSA of the sequence of interest ('evotuning'), ). Previously this has only been investigated for function prediction and for relatively low-capacity models. We fine-tune the full ESM-Ib model (which has 50x more parameters than UniRep) on 380 protein sequence families. We find that after 30 epochs of fine-tuning, long range P@ L increases only slightly, with average 1.6 percentage points FFig. 16).
5.3 PERFORMANCE DISTRIBUTION
Although our model is, on average, better than Gremlin at detecting contacts, the performance dis- tribution over all sequences in the dataset is still mixed. ESM-Ib is consistently hetter at extracting short and medium range contacts (Fig. 7). but only slightly outperforms Gremlin long range contacts when Gremlin access to Uniref100 and metagenomic sequences. [Fig. 2 shows dis- tribution of long range P@L for ESM-1b vs. Gremlin. Overall, ESM- lb has higher long range P@ on 55% of sequences in the test
In addition, we examine the relationship between MSA depth and precision for short, medium, and long range contacts (Fig. 3). Although our contact prediction pipeline does not make explicit use of MSAs, there is still some correlation between MSA depth and performance, since MSA depth measure of how many related sequences are present in the ESM-Ib training set. We again see that ESM-1b 1b consistently outperforms Gremlin at all MSA depths for short and medium range sequences. We also confirm that BSM-1b outperforms Gremlin for long range contact extraction for sequences with small MSAs (depth < 1000). Ib also outperforms Gremlin on sequences with

---

Published conference paper at ICLR 2021
6
12 6:24
24
01.3
ESM-Ib
Gremlin
0.2
0.2
0.1
0.1
10.0
12.5
logytcepth)
loge(depch))
logy(depth)
Figure 3: Gremlin (trRosetta) performance binned by MSA depth. For comparison, ESM-Ib perfor- mance is also shown for the sequences in each bin.
the very largest MSAs (depth > 16000), which is consistent with prior work showing that Gremlin performance plateaus for very large MSAs and suggests that SbM-1b does not suffer from the same issues (Anishchenko et al. 2017).
5.4 LOGISTIC REGRESSION WEIGHTS
In Section 5.1 we show that selecting only sparse subset of the attention heads can yield good results for contact prediction. Overall, the Ly-regularized logistic regression identifies 102 660 heads as being predictive of contacts (Fig. 6b). Additionally, we train separate logistic regressions identify contacts different ranges These regressions identify an overlapping. but non-identical set of useful attention heads. Two attention heads have the top-10 10 highest weights for detecting contacts at all ranges. One attention head is highly positively correlated with local contacts, but highly negatively correlated with long range contacts. Lastly, identify total of 104 attention heads that are correlated (positively or negatively) with contacts at only one of the four ranges, suggesting that particular attention heads specialize in detecting certain types of contacts.
5.5 PERPLEXITY VS. CONTACT PRECISION
Fig. 2 explores the relationship between performance on the masked language modeling task (vali- dataion perplexity) and contact prediction (Long Range P@ L.. A linear relationship exists between validation perplexity and contact precision for each model. Furthermore, for the same perplexity, the 12-layer ESM-1 model achieves the same long range P@L L as the 34 layer ESM-1 model, suggesting that perplexity is good proxy task for contact prediction. ESM-1 and ESM-Ib models are trained with different masking patterns, so their perplexities cannot be directly compared, although linear relationship is clearly visible in both. ESM-1 and ESM-1b have a similar number of parameters: the key difference is in their hyperparameters and architecture. The models shown have converged pre-training, with minimal decrease in perplexity (or increase in contact precision) in the later epochs. This provides clear evidence that both model scale and hyperparameters play a significant role in model's ability to learn contacts.
5.6 CALIBRATION, FALSE POSITIVES, AND ROBUSTNESS
One concern with neural networks is that, while they may he accurate on average, they can also pro- duce spurious results with high confidence. We investigate this possibility from several perspectives. First, we find that logistic regression probabilities are close true contact probability amean-square стror = 0.014) and can be used directly as a measure of the model's 's confidence (Fig. IT).
Second, we analyze the false positives that the model does predict. We find that these are very likely to be within Manhattan distance of 1-4 of true contact (Fig. 13ap. This suggests that false positives may arise due to the way a contact is defined (Cb-Cb distance within angstroms), and could be marked as true contacts under a different definition (Zheng & Grigoryan 2017). Further, when we explore an example where the model's predictions are near true contact, we that the example in question is a homodimer, and that the model is picking up on inter-chain interactions (Fig. 14a). While these do not determine the structure of the monomer, they are important for its function (Anihhceenko et al. 2017).

---

Published conference paper at ICLR 2021
Local
Short
Medium
Long
30
27
Layer
6
-10
12
Head
Head
Head
Head
Figure 4: Logistic regression weights trained only on contacts in specific ranges: local [3, 6), short range [6, 12), medium range [12, 24), long range [24, oo).
Third, we test the robustness of the model to insertions by inserting consecutive alanines at the beginning. middle, end of 1000 randomly chosen sequences. We find that ESM-1b tolerate up to 256 insertions at the beginning or end of the sequence and up to 64 insertions the middle the sequence before performance starts to sigmificantly degrade. This suggests that ESM-1b 1 learns a robust implicit alignment of the protein sequence. See AAppeddxx A.12 for more details.
5.7 MSA GENERATION
Wang & Cho 2019) note that Transformers trained with the MLM objective can be used gener- atively. Here, we consider whether generations from ESM-Ib preserve contact information. The ability to generate sequences that preserve this information a necessary condition for generation of biologically active proteins (Hawkins Hooker et al. (021). We perform this evaluation by taking input protein, masking out several positions, and re-predicting them. This process repeated 10000 times to generate a pseudo-MSA for the input sequence (Algorithm [T]. We feed the resulting MSA into Gremlin to predict contacts, Over all sequences from our test set, this procedure results long range contact P@L of 14.5. Fig. 17 shows one example where the procedure works well, with Gremlin on the pseudo-Me -SSA having long range P@L 52.2.
6 DISCUSSION
Transformer protein language models trained with an unsupervised objective learn the tertiary struc- ture of protein sequence in their attention maps. Residue-residue contacts can be extracted from the attention by sparse logistic regression. Attention heads are found that specialize in different types of contacts. ablation analysis confirms that the contacts are learned without supervision, and that the logistic regression is only necessary to extract the part of the model that represents contacts.
These results have implications for protein structure determination and design. The initial studies proposing Transformers for protein language modeling showed that representation learning could be used to derive state-of-the-art features across variety of tasks, but were not able to show a henefit the fully end-to-end setting (Rives ct al., 2019) Rao ct al., (0199] [Elnagaar et al. (200)). For the first time, we show that protein language models can outperform of-te-oftthearr art unsupervised structure learning methods that have been intensively researched and optimized over decades.
Finally, we establish link between language modeling perplexity and unsupervised structure learn ing. A similar scaling law has heen observed previously for supervised secondary structure pre- diction (Rives al.] 2019), and parallels observations in the NLP community (Kaplan al. 2022)] Brown et al. 2020). Evidence of scaling laws for protein language modeling support future promise as models and data continue to grow.
ACKNOWLEDGMENTS
We thank Justas Dauparas for valuable input and initial analysis. Sergey Ovehinnikov was supported by NIH Grant DP50D026389.

---

Published conference paper at ICLR 2021
REFERENCES
Badri Adhikari and Jianlin Cheng. Protein residue contacts and prediction methods. In Methods in Molecular Biology, volume 1415, pp. 463-476. Humana Press Inc., aug 2016. doi: 10.1007/ 978- 095-9999-377---- 24. URL/bttss:/pubnee ncbi. nim. nih htw.hl.ooo2221155133
Badri Adhikari and Ame Elofsson. DEEPCON: Protein contact prediction using dilated con- volutional neural networks with dropout. Bioinformaties, 36(2)470 jan 2020. ISSN
14602059. doi: 10. 00.1090hhionffommatis>br....... URL
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-only deep representation learning. Nature Methods, 12:1315- 3 2019. ISSN 15487105. doi: 10.1101/589333, htRthttps:// //www.
biorxiv org/ ctt.://ootetttteeco htw.//ent//001.111/5333333
Mohammed AlQuraishi. End-to-end differentiable learning protein structure. bioRxiv, pp. 265231, 8 2018. doi: 10.1101/265231. URL http:::t///// http::://www.biii http://wwwwwwwiiccco////t//otttt wwwpoowwwww.o.o.ooteee........e.
ear war_:y/218/0//2//////
Ivan Anishchenko, Sergey Ovchinnikov, Hetunandan Kamisetty, and David Baker. Ori- gins of coevolution between residues distant in protein 3D structures. Proceed- ings of the National Academy of Sciences of the United States of America, 114 (34):9122-9127. 8 2017. ISSN 10916490. doi: 10. 1073/p0as. 1702664114. URL
https:/ www. pww.ppassa........ erg/ http:///wwwwpp.oaooococc/oooo/aa /ttp///00//111166111111111 absttract.
//wwwpprs htt.:///conneeetaaaa////////////
Sivaraman Balakrishnan, Hetunandan Kamisetty, Jaime G. Carbonell, Su-In Lee, and Christo- pher James Langmead. Learning generative models for protein fold families. Proteins: Structure, Function, and Bioinformatics, 79(4):0011 1078, 4 2011. ISSN 08873585. doi:
10.1002/prot.2294.. URL uttp:/tp://ds: webiiiiwllee.. www. wtt../o/..........e.ooo 22934
seuuecce embeddings using information from Tristan Bepler and Bonnie Berger. Learning protein sequence
structure, 2 2019.
abs/
Tomasz Blazejewski, Hsing-I Ho, and Harris H Wang. Synthetic sequence entanglement augments stability and containment of genetic information in cells. Science, 365(6453):595-599 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2031: 2020.
James A. Cuff and Geoffrey J. Barton. Evaluation and improvement of multiple sequence meth- ods for protein secondary structure prediction. Proteins: Structure, Function and Genetics, 34 (4):508-519 1999. ISSN 08873585.
doi: /tt./////02211000000004119990110 /
AID-PROTI103.0.C002244 URL /twpbeedd ncbi nlm.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North (A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Min- nesota, 6 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19. 1423, URL http: /ttp://axxiii owg/ abs /1w10.1110.. 04805
S. D. Dunn, L. M. Wahl, and G. B. Gloor. Mutual information without the influence of phylogeny entropy dramatically improves residue contact prediction. Bioinformatics, 24(3):333-340, 2008. ISSN 13674803. doi: 10. (09./hooiffrmmaiicc/hm6644
Magnus Ekeberg, Cecilia Lövkvist, Yueheng Lan, Martin Weigt, and Erik Aurell. Improved contact prediction in proteins: Using pseudolikclihoods to infer potts models. Phys. Rew E, 87:012707, Jan 2013, doi: 10.1103/hyykkev.....07770 URL https: /tt::///inn aps. org do1/10.
1103/PhysRevE. 87.012707
10

---

Published conference paper at ICLR 2021
Ahmed Elnaggar, Michael Heinzinger. Christian Dallago, Ghalia Rihawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards Cracking the Language of Life's Code Through Self- Supervised Deep Leamning and High Performance Computing, 7 2020. URL http://arxiv
org/ aba/ 2007. 06225
Rocío Espada, R. Gonzalo Parra, Thierry Mora, Aleksandra M. Walczak, and Diego U. Ferreiro. Capturing coevolutionary signals inrepeat proteins. BMC Bioinformaticx, 16(1):1, dec 2015. ISSN 14712105. doi: 10.1 18590001---0----8----88- URL http::/t////. http::///mmmoioofoooooooooo
biomedcentral.co com wttp:///..oo.aacaeeaeaeaaaaeeeee
Robert D. Finn, Alex Bateman, Jody Clements, Penelope Coggill, Ruth Y. Eberhardt, Sean R. Eddy, Andreas Heger, Kirstie Hetherington, Liisa Holm, Jaina Mistry, Erik L.L. Sonnhammer, John Tate, and Marco Punta. Pfam: The protein families database, 1 2014. ISSN 03051048. URL
httpp:: hwwp:://wwww.. ncbi nim. nih.
Alex Hawkins-Hooker. Florence Depardieu, Sebastien Baur, Guillaume Couairon, Arthur Chen, and David Bikard. Generating functional protein variants with variational autoencoders. PLOS Computational Biology, 17(2)11233 02 2021. doi: 1jj377/joumll.pcbi. 100873. https:::
7/doi. wwg.aooggg/eo 10. /371/ Journal pcbl. 1008736
Michael Heinzinger, Ahmed Elnaggar, Yu Wang. Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost. Modeling the language of life Deep Learning Protein Sequences. bioRxiv, pp. 614313, 2019. doi: 1011/11133333. URL https:: http://wwwwwbooxx.ccooo
content/100 2111//0011111111111100
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with differentiable simulator. 7th International Conference Learning Representations, ICLR 2019, 2019.
L. Steven Johnson, Sean R. Eddy, and Elon Portugaly. Hidden Markov model speed heuristic and iterative HMM search procedure. BMC Bioinformatics, 11(1):431, 8 2010. ISSN 14712105, doi: 10/118611440 1178666471-21------ 1-431.
com htmm/articless/m 10. 1186/ 1111:111111111-11---11111 11-431
David T. Jones and Shaun M. Kandathil. High precision in protein contact prediction using fully convolutional neural networks and minimal sequence features. Bioinformatics, 34(19):
3308-3315, oct 2018. ISSN 14602059. doi: 10. 10.1099/iionnnmmaiccchh....111 URL lttpps:
// //academic. oup http////....foommaaaaaaaaatttaae
49977145
David T. Jones, Daniel W. A. Buchan, Domenico Cozzetto, and Massimiliano Pontil. PSICOV: precise structural contact prediction using sparse inverse covariance estimation large multiple sequence alignments. Bioinformatics, 2812):88-199. 2011. ISSN 1367-4803, doi: 10.1093/
bioinformatiis/bur68..
Hetunandan Kamisetty, Sergey Ovchinnikov, and David Baker. Assessing the utility of coevolution- based
contact predictions sequence- and structure-ric rich era. Proceedings of the National Academy of Sciences, (5439):15674115550 2013. ISSN 0027-8424. doi: 10. 1073/ pnas. 131 131404511400 URL www. pnas. org/ content (ttpattennn/1////1//
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08364 2020.
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjærgaard Jensen, Vanessa Isabell Jurtz. Casper Kaae Sønderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, and Paolo Marcatili. NetSurfPP2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 87(6):200-522 6 2019. ISSN 0887-3585. doi: 10. 25774 URL
https://on nel Sbrrrr.rrr.rr wi ley. www.o///ooaaaa 100.///.....0....... 25674

---

Published conference paper at ICLR 2021
Tomasz Kosciolek and David Jones. De novo structure prediction of globular proteins aided by sequence variation-derived contacts. PLOS ONE, 9(3):1-15, 03 2014. doi: 10. 10. / 00921197. htR.ptt::: : //o.i www.or........ 10. 1371/ /ournal pone. 0092197
Alam S. Lapedes, Bertrand G. Giraud, LonChang Liu, and Gary D. Stormoo. Correlated mutations in models of protein sequences: Phylogenetic and structural effects. Lecture Notes-Monogruph Series, 33:236-256, 1999. ISSN 07492170. URL http:/ /www. 1stor. org stable/ 4356049
Yang Liu, Perry Palmedo, Qing Ye, Bonnie Berger, and Jian Peng. Enhancing Evolutionary Cou- plings with Deep Convolutional Neural Networks. Cell Systems, 6(1))655744.3, jan 2018. ISSN 24054720. doi: 20.1016fj.elss gov T0V/292751737
URL https::/ppubeed..oo/ nnm.mmmm.... nih.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. 7 2019. URL hBt::p//aaxLL..erg///////111
abs httpg///bb///.....2/2
11977.16922
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization. bioRxiv, pp. doi: 228.022222..0900.288282322
0920009.04.283990 2020.
URL httpp:://dl..oo
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi. Po-Ssu Huang, and Richard Socher. ProGen: Language Modeling for Protein Gener-
ation. 3 2020. htttt:///aaax:
org ab///////// / 22204.04449
Dehora S. Marks, Lucy J. Colwell, Robert Sheridan, Thomas A. Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander. Protein 3d structure computed from evolutionary sequence variation. PLOS ONE, 6(12)11-22)) 200 12 2011. doi: 10. 10.137 /jjurraalllllonoo wwt.:lpooee....... 0028766. URL Itttp:://do.
pone. 0028766
Faruck Morcos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora Marks, Chris Sander, Riccardo Zecchina, José N. Omuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue cocvolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, 108(49):E1293- B1301, 2011. ISSN 0027-8424. doi: 10.1073/ pnas.1111471108. URL www. wwws org/ content /10/ /9/.////////////3///////////9//3
Sergey Ovchinnikov, David E. Kim, Ray Yu-Ruei Wang, Yuan Liu, Frank DiMaio, and David Baker. Improved de novo structure prediction in caspll by incorporating coevolution infor- mation into rosetta. Proteins: Structure, Function, and Bioinformatics, 84(S1):67-75, 2016. doi: 10.1002//po0.2499744 URL htttttt:////oiieeeeee
www.y..l........................ com. doi abss
wi
2002/p20t.24974
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duches- nay. Scikit-leamm: Machine Learning in Python. Journal of Machine Learning Research, 12(85): 2825-2830, 2011. ISSN 1533-7928. htthhttp://cct
learn. source net
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE. In Neural Information Processing Systems. Cold Spring Harbor Laboratory, 6 2019, doi: 10.1601667622.. URL 08230
https:/ /doi. orq/10. 1101111110001 http///6622htttpp//aaaaaa http://tttpp...a....o.ooaaaaass
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Süding. HHblits: lightming-fast iterative protein sequence searching by HMM-HMM alignment. Nature Methods, 9 2 2012. ISSN 1548-7091.
doi: 10. 1038/mehhht 1888. URL hhttp://wwer http::://wwww.attreee. www.:rattrrrcoomm
art wlw.l////e.o nweth 1818
Adam Riesselman, John B. Ingraham, and Debora Marks. Deep generative models of genetic variation capture the effects of mutations, Nature Methods, 15(10):816 -22, 10 2018. ISSN 15487 105. doi: 10. 1038/44 192--01-01188-4
12

---

Published conference paper at ICLR 2021
Alexander Rives. Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised leannin to 250 million protein sequences. bioRviv, 2019. doi: 10. 101//2022.. URL https: /www.
Alexander Rives, Joshua Mcier, Tom Sercu, Siddharth Goyal, Zeming Lin, Demi Guo, Myle Ott, C. Lawrence Zitmick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, 2020. doi: 10.1 101/622803. URL wwww biorxiv wttp://oo..eet///att///0///0/0/0
622803
William P Russ, Matteo Figliuzzi, Christian Stocker, Pierre Barrat-Charlaix, Michael Secolich, Peter Kast, Donald Hilvert, Remi Monasson, Simona Cocco, Martin Weigt, al. An evolution-ber -bssed model for designing chorismate mutase enzymes. Science, 369(6502):440 2020.
Stefan Seemayer, Markus Gruber, and Johannes Söding. CCMpred- fast and precise pre- diction of protein residue-residue contacts from correlated mutations. Bioinformatics, 30(21)3118-31300 11 2014. ISSN 1460-2059. doi: 10. 09////ooonffrrrmaiiih/uuS0
URL
http: ncbi. http://www.........oooooooeee/// (5tmmmeeee22244456
bio.nformat.cc//tu5000
Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Ko- ray Kavukcuoglu, and Demis Hassabis. Protein structure prediction using multiple deep neu- ral networks in the 13th Critical Assessment of Protein Structure Prediction (CASP13). Pro- teins: Structure, Function, and Bioinformatics, 87(12): 141-1 148, 12 2019. ISSN 0887-3585. URL htttss://oonnn in brary. wi ley. com/ doi / abs/10
doi: 10.4002//r0..25844.
1 1002/pzot.2583 25834
Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin židek, Alexander W.R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Ko- ray Kavukeuoglu, and Demis Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792).706- 701-7000 1 2020. ISSN 14764687. doi: 1038/ orq/ 1 119-8660099923---1
$41586-019- 11586-019-192377
Hongyu Shen, Layne C. Price, Mohammad Taha Bahadori, and Franziska Seeger. Improving generalizability of protein sequence models via data augmentations, 2021. URL https:
//openreview.
doi: 10. 1orwmmidd-Kkwwssse... 2Sd]
Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nature Merhods, 16(7):603-606, 2019. ISSN 1548-7105.
URL httpp:://dbi.org//00
841592-0192 0437-4
J. I. Sulkowska, F. Morcos, M. Weigt, T. Hwa, and J. N. Onuchic. Genomics-aided structure pre- diction. Proceedings of the National Academy of Sciences, 10.1073/PNAS. 1207864109. URL /ww. phas httpaa/cc///g/////0/00111133//// 1207864103
2012. doi:
Baris E. Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H. Wu. UniRef: Comprehensive and non-redundant UniProt reference clusters. Bioinformatics, 23(10):1282 1288, 5 2007. ISSN 13674803. doi: 10. wt...///bbiiiifomattt///mtmdddd UniRef50 database licensed under (CC BY 4.0).
URL http::/tp:// /ww.
uniprot. org.
Todd J. Taylor, Hongjun Bai, Chin Hsien Tai, and Byungkook Lee. Assessment of CASP10 contact- assisted predictions. Proteins: Structure, Function and Bioinformatics, S2(SUPL..)) 2 2014. ISSN 08873585. doi: 00.1002/prot.243677
URL https: www ncbi n1m. nih.
gov/pmc/articcm httpaaatttti.eeeeeeeee1111
13

---

Published conference paper at ICLR 2021
John Thomas, Naren Ramakrishnan, and Chris Bailey-Kellogg Graphical models of residue cou- pling in protein families, apr 2008. ISSN 15455963. URL http:///pmmdddd...m ncbbi.d.ncbi..... ncbi. nlm.
nih. gov/ 1845144 51428/
Pengfei Tian, John M Louis, James I. Baber, Annie Aniana, and Robert B Best. Co-evolutionary fitness landscapes for sequence design. Angewandte Chemie International Edition, 56724) 5678. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances Neural In- formation Processing Systems, pp. 5998-6008, 2017. URL https://papen paapppaer.ipps......... cc/ pdf
Jesse Vig. Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. BERTology Meets Biology: Interpreting Attention in Protein Language Models. bioRxiv, pp. 2020 06 26. 174417. 2020. doi: 10. 10.10002022........7707
URL http:://axxiy
abs 2006. 15222
Alex Wang and Kyunghyun Cho. BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. 2 2019. /tl.pttpp///axi..... wwtp:///gg////aaaoooe/oo/ss//s/s 902 04094
Sheng Wang, Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLOS Computational Biology, 13(1):1334, 01 2017. doi: 10. /jooaa/laaal...ii.10000. 1005.24244.. URL pcbi 1055324
Martin Weigt, Robert A. White, Hendrik Szurmant, James A. Hoch, and Terence Hwa. Identification of direct residue contacts in protein- protein-protein protein interaction by message passing. Proceedings of the National Academy of Sciences of the United States America, 106(1)x67-74) jan 2009. ISSN
00278424, doi: /www phas org/ content /106 absttt//aa.aaatcctttca 106/1/ http:////hhhtpp///
0805923106. URL. ponttps://wwaet pnas org/ wttpo//geggg//////////o///////tt content/
Jinbo Xu, Matthew Mcpartion, and Jin Li. Improved protein structure prediction by deep learning
irrespective of co-evolution information. bioRxiv, pp. 2020.10.12.336859, oct 2020. doi: 10.1101/2020 12.33689. URL https: http:://www...loxxx..ooooo/ooott
1101/
2020. 10. 12. http://sw....aaaaaooo//oo//oaoaa
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng. Sergey Ovchinnikov, David Baker, and John Harvard. Improved protein structure prediction using predicted inter-residue orienta- tions. bioRxiv, pp. 846279, 2019. doi: 10.1101/846279. URL httph:: //tp::://www. wwww www.wwwww.bb....../.o..
Chengxin Zhang, Wei Zheng, M Mortuza, Yang Li, and Yang Zhang. DeepMSA: construct- ing deep multiple sequence alignment to improve contact prediction and fold-recognition for distant-homology proteins. Bioinformatics, 36(7):2105-2122 apr 2020. ISSN 1367-4803, doi: 10. 1..00993oonnfommacc//r2223.. URL httphtttss///d.org//// http://g/.......//ooooooooraaaaa
bt.z 633
Fan Zheng and Gevorg Grigoryan. Sequence statisties tertiary structural motifs reflect protein stability. PLoS ONE, 12(5)201182272, 5 2017. ISSN 19326203. doi: 10.137 1/377//jorall.po. 0178272. URL https: /oi
pone 0178272
14

---

Published conference paper at ICLR 2021
Table 3: Major Architecture Differences Protein Transformer Language Models
Name
Layers
Hidden Size
Attn Heads
Parameters
Dataset
TAPE ProIBERT-BED
768
92M
Pfam
30
1024
16
420M
BFDI00
ESM-1 (6 layer)
768
12
43M
Uniref50
ESM-1 (12 layer)
768
85M
Uniref50
ESM-1 (34 layer)
34
1280
20
670M
Uniref50
ESM-18 lb
33
1280
20
650M
Uniref50
A APPENDIX
A.1 NOTATION
In the figures, report contact precision in the range of 0.0 1.0. In the text and in the tables, we report contact precision in terms of percentages, the range of to 100.
A.2 AVERAGE PRODUCT CORRECTION (APC)
In protein contact prediction, APC is commonly used correct for background effects of entropy and phylogeny (Dunn al. 2008). Given an coupling matrix APC defined as
F F
F Fiy
(3)
Where Fi, Fy. and are the We apply APC independently to the symmetrized attention maps of each head the Transformer. These corrected attention maps are passed in as input to logistic regression.
the i-th j-th column, and the full matrix respectively.
A.3 GREMLIN IMPLEMENTATION DETAILS
Gremlin trained by optimizing the pseudolikelihood of V. which correspond pairwise and individual amino acid propensities. The pseudolikelihood approximation models the conditional distributions of the original joint distribution and can be written:
exp (Via + Σj E 10 Wejab)
log p( jest W. Vi) loog
(4)
Σen exp (Vic + Lj=1f Ee (0rg b)
subject to the constraint that W;. 0 for all i, and that Wajab is symmetric in both sequence (i,j) and amino acid (a, b). Additionally, Gremlin regularization parameter that is adjusted based on the depth of the MSA.
A.4 ESM-1 IMPLEMENTATION DETAILS
The original ESM-1 models were described in (Rives et 2019). ESM-1 trained on Uniref50 contrast to the TAPE model, which is trained on Přam (Finn et al. 2014) and the ProIBERT-BED model, which is trained on Uniref100 and BFD100 (Steinegger et al. 2019). ESM- lb is new model, which the result of an extensive hyperparameter sweep that was performed on smaller layer models. ESM-Ib is the result of scaling up that model to 33 layers.
Compared to ESM-1, the main changes in ESM-1b are: higher learning rate; dropout after word embedding; learned positional embeddings; final layer input/output word embeddings. Weights for all ESM-I and ESM-Ib models can be found https://github. com hatm///fccbooeeeercc/eeeeee
before the output; and tied
A.5 JACKHMMER DETAILS
We use Jackhmmer version 3.3.1 with a bitscore threshold of 27 and iterations to construct MSAs from the ESM training set. The failures on 126 sequences noted in [Section 4.4 result from a seg- mentation fault in hmmbuild after several iterations (the number of successful iterations before the
15

---

Published conference paper at ICLR 2021
Table 4: Average metrics on 15 CASP13 FM Targets. All baselines use MSAs generated via the trRosetta MSA generation approach.
6 S sep <
12 S sep <
24
24 S sep
Model
Variant
L/2
L/5
L
L/2
L/5
L
L/2
L/5
mfDCA
11.0
13.6
19.7
12.8
17.9
26.2
14.4
19.4
26.6
Baselines
PSICOV
10.6
14.0
18.3
12.2
17.1
25.9
14.1
19.8
27.9
Gremlin
12.1
16.1
23.6
14.5
20.8
32.5
16.8
23.4
28.5
top-1 heads
11.8
15.8
23.8
17.0
20.8
29.6
13.6
17.9
22.7
top-5 hcads
15.3
20.9
29.6
18.7
27.0
33.0
14,6
20.6
26.8
top-10 heads
16.6
22.7
32.1
21.8
29.5
39.8
17.9
23.2
30.4
ESM-1b 1b (attention)
n=1, s=1
16.4
23.5
34.7
23.0
30.8
41.6
18.1
23.3
29.9
n=10, s=1
18.6
25.3
39.3
24.1
31.9
41.4
18.7
25.2
33.2
n=20,
19.3
37.0
24.0
31.5
40.2
18.6
25.0
33.8
MSA, s=1
14.2
20.3
30.5
21.0
29.1
42.3
18.4
23.7
31.5
ESM-Ib (bilinear)
n=20
14.1
17.7
19.8
17.7
20.9
27.9
11.2
13.9
17.0
n=14257
20.8
31.1
45.9
25.7
33.7
43.3
20.1
26.2
34.2
segmentation fault varies depending the input sequence). Since we see this failure for less than 1% the dataset we choose ignore these sequences during evaluation.
Additionally, we evaluated alternate MSAs by running Jackhmmer until NefT 128 was achieved (with maximum of iterations), a procedure described by /Zhang et al. (2020). This resulted very similar, but slightly results (average long range P@L 29.3, versus 31.3 when always using the output maximum.
eighth iteration). We therefore chose to report results using the 8 iteration
A.6 RESULTS ON CASP13
In Table 4 report results on the 15 CASP13 Free Modeling targets for which PDBs were pub- licly released. specific domains evaluated are: T0950-D1, T0957s2-D1, T0960-D2, T0963-D2, T0968s1-D1, T0968s2-D1, T0969-D1, T0980s I-DI, T0986s2-DI, T0990-D1, T0990-D3, T1000- D2, I3-DI, T1021 s3-D2, TI022s1-DI. ESM-Ib is able to outperform Gremlin, and simply averaging the t0 heads of ESM-1b 1b has comparable performance to Gremlin.
addition, we compare our logistic regression model to the bilinear contact prediction model pro- posed by Rives et al. (2020). This model trains two separate linear projections of the final represen- tation layer and computes contact probabilities via the outer product of the two projections plus bias term, which generates the following unnormalized log probability:
log p(contact) (sW2) +
(5)
Here x sequence-length vector of features in RZxd Each W; matrix in Rdxi where k hyperparameter controlling the projection size.
We train this model in both the limited supervision (n = 20) and full supervision (n = 14257) set- the same 20 proteins used to train the sparse logistic
ting. For the limited supervision setting, regression model. For the full supervision setting we generate a 95/5% random training/validation split of the 15008 trRosetta proteins with sequence length < 1024.
We performed independent grid searches over learning rate, weight decay, and hidden size for the settings. For the 20 setting, found learning rate of 0.001, weight decay of 10.0, and projection size 512 had best performance on the validation set. For the n 14257 setting we found a learning rate of 0.001, weight decay of 0.01, and projection size of 512 had best performance
PPSICVV fails converge nttps://githus www/pao////o.oooo ppre/ppp htt///sscoooo/oo cov increase rho 0.005 for those domains.
targets with default parameters. We follow the procedure suggested

---

Published conference paper at ICLR 2021
Long Range P@I
Long Range P@1.5
neff
10
100
1000
0.6
ESM-1b
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gremlin
Gremlin
Figure 5: Results on 15 CASP13 FM Domains colored by Neff.
0.25
30
0.01
5
27
0.05
24
0.08
0.20
0.1
21
0.12
Layer
18
0.15
0.15
15
-5
12
0.2
0.10
9
0.5
1.0
0.05
3
6
8
1011
12
15
16
1920
0 9
12 18
Training Examples
Head
(a) Gridsearch
(b) Weights
Figure (a) Gridsearch on logistic regression over number of training examples and number reg- ularization penalty. Values shown are long range P@L over a validation set of 20 proteins. (b) Per-head and layer weights of the logistic regression on the best FSM-Ib model.
on the validation set. All models were trained to convergence to maximize validation long range P@L with patience of 10. The n = 20 models were trained with batch size of 20 (i.e. 1 batch epoch) and the 14257 models were trained with batch size of 128.
The bilinear model performs very poorly in the limited supervision setting, worse than simply taking the top-1 attention head. With full supervision, it moderately outperforms the logistic regression for increase in long range P@L L of 1.5 while using 700x more data.
Fig. 5 we display results on the 15 FM targets colored by effective number of sequences. ESM-Ib shows higher precision at and L/5 on average, and is sometimes significantly higher for sequences with low Neff. Since ESM-16 1b training data was generated prior to CASP13, this suggests ESM-1b is able generalize well new sequences.
17

---

Published conference paper at ICLR 2021
A.7 LOGISTIC REGRESSION DETAILS
Given a model with layers, H heads, and an imput sequence x of length L, let Amh be the L x contact map from the h-th head in the m-th layer. We first symmetrize this map and apply APC and let ambiy be the coupling weight between sequence position and in the resulting map. Then define the probability a contact between positions and according to a logistic regression with parameters 3:
pl p)
(6)
1+ exp (-8 --E=
Σ
Burh mhis
To fit (3 let D a set of training proteins, be a minimum sequence separation, and À a regularization weight. The objective can then be defined as follows:
ba-k Ld
L(D,3)
II dED
II
II
i-1
j-itk
M H
8 max L(D: 3
Σ
Σ
1mh
(8)
We fit the parameters 3 via scikit-lcarn (Pedregosa et al. 2011) and do not backpropugate the gradi- ents through the attention weights. In total, our model learns MH parameters, many of which thanks the /, regularization.
There are three hyperparameters our training setup: the number of proteins our training set D. the regularization parameter X, and the minimum sequence separation of training contacts k. We find that performance improves significantly when increasing the D from protein to 10 proteins, but that performance gains drop off when 2 increases from 10 to (Fig. T). Through hyperparameter sweep, we determined that the optimal À is 0.15. We find that ignoring local contacts (ji - jl < 6) also helpful. Therefore, unless otherwise specified, all logistic regressions 20, 0.15, kt 6. See,Fig. far a gridsearch over the number of training proteins regression penalty. We used 20 training proteins and 20 validation proteins for this gridsearch. Fig. 6b shows the weights of the final logistic regression used for ESM-1b.
trained with [D]
A.8 PERFORMANCE DISTRIBUTION
Fig. 7 shows the full distribution of performance of ESM-1b compared with Gremlin. When we provide Gremlin access Uniref100, along with metagenomic sequences, ESM-1b still consistenly outperforms Gremlin when extracting short and medium range contacts. For long range contacts, Gremlin is much more comparable, and has higher contact precision on 47% of sequences. With access to the same of sequences, ESM-Ib consistently outperforms Gremlin in detecting short, medium, and long range contacts. This suggests that ESM- can much better extract information from the same set of sequences and suggests that further scaling of training data may improve ESM- 1b even further.
This analysis is further borne out Fig. Given the same set of sequences, ESM-1b outperforms Gremlin on average for short, medium, and long-range contacts regardless the depth of the MSA generated from the ESM- lb training set.
Additionally, we find that ESM-1b MSA (Fig. 9p. This is not possible for Gremlin, which a family-level model. leverage this a fairly simple way to provide a modest boost to the contact precision of ESM-Ib (Section 5.2).
provide varying contact maps for different sequences in the
A.9 SECONDARY STRUCTURE
Section 5.4 we show that some heads that detect local contacts (which often correspond to see- ondary structure) are actually negatively correlated with long range contacts. We test ESM-1b's ability detect secondary structure via attention by training separate logistic regression the

---

Published conference paper at ICLR 2021
Short
Medium
Long
1.00
0.75
ESM-1b
0.50
0.25
15.0
0.00
12.5
0.0
0.5
1.0
0.0
0.5
1.0
Gremlin (ESM Data)
Gremlin (ESM Data)
Gremlin (ESM Datal
1.00
7.5
0.75
5.0
ESM-1b
0.50
0.25
0.00
0.5
0.0
0.5
1.0
0.0
0.5
1.0
Gremlin (trResetta Data)
Gremlin (tRoseeta Data)
Gremlin (trRosetta Datal
Figure 7: Short, medium, and long range P@L performance distribution of ESM- vs. Gremlin. Each point is colored by the log, of the number of sequences in the MSA.
12
12 S x 24
24 S
0.3
0.2
0.4
POL.
0.2
0.2
0.1
5
10
15
10
15
10
15
(depth))
logy(depth)
logg(depth)
s
12 < < 24
24 <
0.3
ESM-Ib Gremlin
P0L
0.2
0.1
0.1
10.0
logo(dopth)
logy(depth)
log:(dcpth)
Figure 8: Gremlin performance binned MSA depth using both ESM (top) and trRosetta (bottom) MSAs. For comparison, Eb performance also shown for the sequences in each bin.
Netsurf dataset (Klausen et al. 2019). As with the logistic regression on contacts, we compute at- tentions and perform APC + symmetrization. To predict the secondary structure of amino acid i, we feed as input the couplings anahi for each layer m, for each head h, and for E | 5, i 5). for total of 7260 input features. Using just 100 of the 8678 training proteins, we achieve 79.3% accuracy on 3-class secondary structure prediction on the CB513 test set (Cuff & Barton, 1999). Figure 10 shows the importance of each layer predicting the three secondary structure classes.
19

---

Published conference paper at ICLR 2021
5jnp
2qzb
3h8h
5c5r
Long Range P@t.
0.5
20 1: 10
1000
0
100
0
1000
1000
5e3e
3k60
3pjv
3a98
Long Range
0.5
0.0
1000
0
500
1000
1000
3h8d
70
3r72
1o82
Long Range
0.5
500
0
100
0
2000
100
2ymz
5nxk
2cye
5e3e
Long Range P
0.5
0.0
2000
500
2000
1000
Sort Index
Sort Index
Sort Index
Sort Index
Figure 9: Distribution of contact perplexity when evaluating diffcrent sequences from the same MSA. The x-axis shows the index of each sequence, sorted in ascending order by hamming distance from the query sequence (query sequence always index 0). y-axis shows long range P@L. The black line indicates Gremlin performance on that MSA.
There are spikes in different layers for all three classes, indicating that particular heads within those layers specializing in detecting specific classes of secondary structure.
Fig. shows importance each Transformer layer to predicting each of the three secondary struc- ture classes. We see that, as with contact prediction, the most important layers are in the middle layers (14-20) and the final layers (29-33), Some layers spike more heavily on particular contact classes (e.g. layer 33 important for all classes, but particularly important for 3-strand predic- tion). This suggests that particular heads within these layers activate specifically for certain types secondary structure.
A.10 BOOTSTRAPPED Low-N CONFIDENCE INTERVAL
Section 5.] shows results from Low-N supervision on 1, 10, and 20 proteins. Since performance in
this case depends the particular proteins sampled we use bootstrapping to determine confidence interval for each of these estimates. Using the full training, validation, and test set of 14882 proteins, we train 100 logistic regression models using a random sample of N proteins, for N = 1. 10, and 20. Each model is then evaluated the remaining 14882 N proteins. The full distribution of samples
20

---

Published conference paper at ICLR 2021
5
Helix
1.0
Perfect
Strand
ESM-1 (6)
0.8
Regression Norm
Other
ESM-1 (12)
ESM-1 (34)
3-
Actual
0.6
ESM-Ib
2-
0.4
0.2
0
0.0
30
1.00
Layer
Prodicted
Figure 10: L2 norm of weights for 3-class secondary structure prediction by Transformer layer.
Figure 11: Calibrated probability real con- tact given predicted probability of contact over all test proteins.
# Train
10
Procision
0.4
0.2
0.0
L/2
L/5
L/2
L/5
L/2
L/5
Short
Medium
Long
Figure 12: Distribution of precision for all reported statistics using 100 different logistic regression
models. Each regression model is trained on random sample of N 1, 10, 20 proteins.
can be seen Fig. 12
The confidence interval estimates for long range precision with 1, 10, and 20 training proteins are: 35.6 + 1.8, 40.6 + 0.1, and 41.0 + 0.1 respectively.
A.11 MODEL CALIBRATION AND FALSE POSITIVES
Vig et al. 2020) suggested that the attention probability from the TAPE Transformer was well- calibrated estimator for the probability of contact. In Ini. 11 we examine the same with the logistic regression trained on the ESM-1 and ESM-1b models. We note that ESM- 1b, in addition to being more accurate overall than Gremlin, also provides actual probabilities.
wTwe
We find that as with model accuracy, model calibration increases with larger scale and better hyper- parameters. The 12, 34 layer ESM-1 models have mean-squared error of 0.074, 0.028, and 0.020 between predicted and actual contact probabilities, respectively. ESM-Ib has mean squared error of 0.014. Mean squared error is computed between contact probabilites split into 20 bins ac- cording to the scikit-learn calibration curve function. It is therefore reasonable to use the logistic regression probability as measure of the model's confidence.
In the case of false positive contacts we attempt to measure the Manhattan distance between the coordinates predicted contacts and the nearest true contact (Fig. 13a). We observe that the Man- hattan distance between the coordinates
false positive contacts are often very close (Manhattan distance between 1-4) to real contacts, and that very few false positives have Manhattan distance 10 from true contact. With threshold contact probahility of 0.5, 83.8% of proteins have least
21

---

Published conference paper at ICLR 2021
0.8
p(contact) 2
0.1
10c
0.6
Likelihood
0.5
0.7
Count
10
0.4
0.9
10a
0.2
Manhattan Distance
Manhattan Distance
Manhattan distance nearest contact.
(b) Count of predictions Manhattan distance.
Figure (a) Distribution of Manhattan distance between the coordinates of predicted contacts and the nearest true contact at various thresholds of minimum p(contact). distance of zero cor- responds to a true contact. (b) Actual counts of predictions by Manhattan distance across the full datasct (note y-axis log scale).
Intra-Chain
All Contacts
ESM-Ib
Gremlin
400
200
400
0
200
400
200
400
0
200
400
(a) Intra-chain, inter-chain, predicted contacts Smlt, which is homodimer.
Contacts
ESM-Ib
Gremlin
50
100
150
100
100
(b) Real and predicted contacts for the CTCF protein (pdbid: 5yel).
Figure 14: Illustraiion of two modes for ESM-Ib where significant numbers of spurious contacts are present
predicted. (a) Predicted contacts which do oceur in the full homodimer complex, but as intra-chain contacts. (b) CTCF protein contacts. small band of contacts near the 30-residue off- diagonal is predicted by 1b. This band, along with additional similar bands are also predicted by Gremlin.
predict contact with Manhattan distance >> 4 the nearest contact. This drops 71.7% with threshold probability of 0.7, and 52.5% with threshold probability 0.9.
Fig. 14 highlights two modes for ESM-1b lb where signficant numbers of spurious contacts are pre- dicted. Fig. 14a shows one example where the model does appear to hallucinate contacts around residues 215 and 415, which do not appear in the contact map for this protein. However, this protein

---

Published conference paper at ICLR 2021
Average
1a27
0.7
0.35
0.6
0.30
moxel
Long Range P@L
ТАРЕ
0.5
0.25
ESM-Ib
LongRango
location
0.4
0.20
beginning
midpoint
0.3
0.2
0.10
0.1
0.05
16
128
256
16 32
128 256
Insertion Length
Insertion Length
Figure 15: Robustness of ESM-1b and TAPE models to insertions of Alanine at the beginning, middle, and end of sequence
a homodimer and these contacts are present in the inter-chain contact map. This suggests that some 'highly incorrect' false positives may instead be picking
inter-chain contacts. Fig. 14b shows example of repeat protein, for which evolutionary coupling methods known pick up on additional 'bands' of contacts (Espada et al., 2015 [ Anishchenko al.) 2017). Multiple bands are visible in the Gremlin contact map, while only the first band, closest to the diagonal, is visible in the ESM-1b contact map, More analysis would be necessary to determine the frequency of these modes, along with additional potential modes.
A.12 ALIGNMENT
One hypothesis as to the benefit of large language models as opposed to simpler Potts models that they may be able to learn an implicit alignment due to their learned positional embedding. For a Potts Model, an alignment enables model to relate positions in the sequence given evolutionary context despite the presence of insertions or deletions. We test robustness of the model to insertions by inserting consecutive alanines at the beginning, middle, or end of 1000 randomly chosen sequences with initial sequence length 512 (we limit initial sequence length in order to avoid out-of- memory issues after insertion). We find that ESM- lb can tolerate up to 256 insertions at the beginning or end of the sequence and up to 64 insertions in the middle of the sequence before performance starts to significantly degrade. This suggests that ESM-1b learns robust implicit alignment of the protein sequence.
On the other hand, we find that the TAPE Transformer is less robust to insertions. On one sequence (pdbid: la27), we find the TAPE Transformer drops in precision by 12 percentage points after adding just 8 alanines the beginning of the sequence, while ESM-Ib sees minimal degradation until 256 alanines are inserted. We hypothesize that, because TAPE was trained on protein domains, it did not leam to deal with mis-alignments in the input sequence.
A.13 EVOLUTIONARY FINETUNING DETAILS
We finetuned each model using learning rate of le-4, 16k warmup updates, inverse square root learning schedule, and maximum of 30 epochs. This resulted in varying number of total updates depending on the size of the MSA, with larger MSAs being allowed to train for more updates. This should ideally help prevent the model from overfitting too quickly on very small MSAs. We use a variable batch size based on the length of the input proteins, fixing maximum of 16384 tokens per batch (so for length 300 protein this would correspond to a batch size of 54), We use MSAs from trRosetta for finetuning all proteins with the exception of GvGFP, where we use the set of sequences from Alley et al. 2019).
23

---

Published conference paper at ICLR 2021
Ia3a
No Evotuning
Evotuned
0.020
A Long Runge P@L
100
0.015
0.010
100
av GFP Contacts
No Evolunnig
Evotuned
0.005
0
0.000
100
20
epoch
200
200
200
200
Figure 16: Left: Average change in contact precision vs, number of finetuning epochs over 380 proteins. Right: Real and predicted contacts before and after evolutionary finetuning for 1a3a and avGFP. For la3a, long range P@L improves from 54.5 61.4. For avGFP, long range P@ im- proves from 7.9 to 11.4.
3qhp Contacts
ESM- lb
Generated MSA
50
100
150
100
100
100
Figure 17: Contacts for 3qhp from Gremlin trained on pSeudo-MSA generated by ESM-Ib, pared to real and ESM-Ib predicted contacts. The generated MSA achieves a long-range P@L 52.2 while the attention maps achieve precision of 76.7.
A.14
MSA GENERATION
Result: Generated MSA
input // protein sequence
curr input/ // optionally, the input can be repeated for batching for 0 - 10000 do
masked = mask 20% of positions in curr, pred = model(masked);
currfmasked positions] masked positions]:
MSA.append(curr);
if random() < 0.1 then curr inputt
end end
Algorithm 1: Quickly generate pseudo-MSA from an input sequence.
Algorithm I presents the algorithm used generate PSeudo-MSAs from ESM-Ib. Each pseudo- MSA is passed to GREMLIN in order to evaluate the preservation contact information (Fig. 17).


Reviews:
[{"review_id": "goICfmHPbVf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper performs a number of analyses centered around the ability of transformer-based language models trained on protein sequence data to learn representations useful for predicting protein secondary and tertiary structure (the latter as contact maps). Specifically, the paper studies several pre-trained transformer models by fitting an L1-penalized logistic regression to amino acid pair contacts. Several experiments are performed to showcase that (i) transformer-based representations can outperform state-of-the art methods based on MSA in terms of contact prediction precision; (ii) that the necessary information for contact predictions in these representations is learned in an unsupervised manner (and not by the logistic regression put on top of these representations); and (iii) that the contact prediction probabilities are reasonably well calibrated.\n\n**Score justification**\nIn its current form the paper presents interesting analyses, but has overall limited novelty. The ability of transformer models to learn representations predictive of secondary and tertiary structure has been demonstrated before (including in the papers proposing the models used by the authors). Furthermore, I have some questions regarding the methodology employed by the authors.\n\n\n**Major comments**\n* The main metric employed by the authors is the precision of the top L (protein length) contact prediction for a given range (P@L). I wonder why the authors do not also consider recall at L as an accompanying metric for reporting the results.\n* When comparing ESM to the baseline Gremlin method, the authors consider two scenarios: (i) Gremlin trained on the trRosetta data; and (ii) Gremlin trained on the same data as the ESM transformer model. Overall, Gremlin trained on the ESM data - which is arguably the correct baseline for the ESM model -  performs worse than Gremlin trained on the trRosetta data. Why is that the case? How does the procedure for preparing MSA for the ESM data compare to that of the trRosetta data? Can it be tuned to improve Gremlin's performance?\n* The paper compares several transformer models that differ primarily in the model size, dataset size and hyper-parameters. As can be seen from Table 1 of the manuscript, these differences are clearly important for the contact prediction task and thus should be summarized and discussed in more detail.\n* From what I understand the sequences from the testing set of the contact prediction problem (or sequences highly similar to them) could appear in the training sets of the considered transformer models. This creates some information leakage. It's unclear from the results presented in the paper whether it is an issue or not - how does contact prediction precision / recall change as sequence similarity to the ESM training set drops?\n* The authors present analysis on the usefulness of the representations learned by various attention heads for contact prediction; and on robustness of such predictions. I wonder how robust the results of these analyses are - they appear to have been performed using a single checkpoint of the ESM model, which is a result of stochastic training from random initialization.\n* In the Appendix the authors talk about the benefit of using predicted contact maps for inferring the all-atom protein 3D structure. However no results on this are presented. I would be very eager to see the comparison of 3D structure accuracy inferred with ESM-predicted and Gremlin-predicted contacts.\n\n**Minor comments**\n* Introduction talks about the ESM-1b model but (as far as I can tell) a reference isn't provided until a later section.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a2VIZand6bJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper shows that Transformers trained unsupervised on millions of protein sequences learn information about protein contacts by using attention maps for contact prediction. The paper is mostly clearly written and discusses server interesting ablation experiments. However, two recent papers that appeared on arXiv before the ICLR submission deadlines also use Transformers for protein contact prediction. These papers and other methods for contact prediction beyond Gremlin are not described. I therefore consider the contributions as insufficient for an ICLR submission.\n\n## Major comments\n\n1. Using Transformer attention maps for protein contact prediction is not new. See Rives et al, 2020, ‘Biological structures and functions emerge…’, section 5.2, and Vig et al, 2020, ‘Bertology’ section 4.2. Both publications appeared on arXiv at least one month before the ICLR submission deadline and are not clearly discussed in the paper.\n\n2. The introductions discusses existing work on Transformers for protein languages models. Existing methods for contact prediction (beyond Gremlin), however, are not described sufficiently.\n\n3. It is unclear which sequences were used for training the Transformer models and how similar they are to test sequences.\n\n4. The paper compares Transformers to Gremlin. However, it is unclear how well they perform to the CASP state-of-the art (see also Rives et al, 2020).\n\n5. Section 3.4  does not describe clearly enough how attention maps were used for predicting contact maps. How were attention maps symmetrized? Which layers and heads were used and how were they aggregated? What is the number of resulting features that were used to train the logistic regression model? APC is not described or cited.\n\n6. Section 4.5 discusses that Transformers can be also used for secondary structure prediction. This is not new (see Rives 2020 and Vig 2020) and does not fit well to the rest of the paper, which is about contact prediction. \n\n6. Section 4.8: Using transformers for generating proteins with natural properties is not new (see Madani et al, 2020, ‘ProGen’ or Rives et al, 2020). ‘Wang & Cho’ were not the first who used Transformers generativity (see Vaswani, 2017).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AsQVaEgEL_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors show that transformer protein language models can learn protein contacts from the unsupervised language modelling objectives. They also show that the residue-residue contacts can be extracted by sparse logistic regression to learn coefficients on the attention heads. One of the advantages of using transformers models is that they do not require an alignment step nor the use of specialized bioinformatics tools (which are computationally expensive). When compared to a method based on multiple sequence alignment, the transformers models can obtain a similar or higher precision.\n\nContributions of this paper are:\n- showing that the attention maps built in Transformer-based protein languages learn protein contacts, and when extracted, they perform competitively for protein contact prediction;\n- a method for extracting attention maps from Transformer models;\n- a comparison between a recent protein transformer protein language model (which does dot require sequence alignment), and a pseudo-likelihood-based optimization method that uses multiple sequence alignment;\n- an analysis of how much the supervised learning (logistic regression) contributes to the results.\n\nThe paper covers a relevant topic and it is easy to read. \n\nHowever, I have a number of concerns. The main contribution of the paper is that attention maps built in Transformer-based protein languages learn protein contacts and can be used for protein contact prediction. However, this was reported before in Rives et al.(2019) (doi: 10.1101/622803). Also, several methods have been developed for this problem, but are not included in the comparisons. Finally, the provided implementation details are not sufficient to reproduce the results of the paper. \nI detail some of these concerns below, together with questions/suggestions for improvements:\n\n1) I would recommend comparing transformers to other methods besides Gremlin, or justify why other methods were not included. This review can be helpful:\n\n(Adhikari B, Cheng J., 2016.. doi: 10.1007/978-1-4939-3572-7_24)\n\nAlso, more recent methods that were published after the review are:\n\n(Badri Adhikari, 2020. https://doi.org/10.1093/bioinformatics/btz593)\n\n(Luttrell  et al., 2019. https://doi.org/10.1186/s12859-019-2627-6)\n\n(Gao et al.,2019. https://doi.org/10.1038/s41598-019-40314-1)\n\n(Ji S et al., 2019. https://doi.org/10.1371/journal.pone.0205214)\n\n2) On page 7, the authors state that \"We find that the logistic regression probabilities are reasonably well calibrated estimators of true contact probability and can be used directly as a measure of the model's confidence (Figure 10a)\". However, from the plot in Figure 10a, it is not totally clear that the probabilities are well calibrated. Could the authors add more justifications of why they consider it well calibrated? Could they also show a comparison of the calibration of the other transformer models, perhaps using MSE as a calibration metric?\n\n3) To understand the occurence of false positives, the authors analyze the Manhattan distance between the predicted contact and the true contact, which is between 1 and 4 for most false positives. They also show an example of a homodimer, for which predictions were far from the true contacts, and explain that the model is picking up inter-chain interactions. Could the authors report how many predictions have a Manhattan distance larger than 4? Is this one example representative of the group of false positives far from the true contact? Maybe the authors could analyse whether this happens in most of the cases.\n\n4) While ESM-1 is open-source and publicly available, this is not the case for ESM-1b. In section A.5, the authors provide implementation details as differences between ESM-1 and ESM-1b, stating “Compared to ESM-1, the main changes in ESM-1b are: higher learning rate; dropout after word embedding; learned positional embeddings; final layer norm before the output; and tied input/output word embeddings. The weights of all ESM models throughout the training process were provided to us by the authors.”. In my opinion, this is not enough to reproduce the results in this paper. To make it reproducible, the authors need to provide a detailed enough description of the differences to make the reader able to implement ESM-1b, or provide the weights and hyperparameters required to reproduce their results.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "34HFsILFxM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, the authors present a method for predicting residue-residue contacts within protein structures using the attention layers learned by transformer language models. Using the largest transformer language models trained to data, the authors show good performance for contact prediction. The paper is clearly written and easy to follow.\n\nThe general concept of fine tuning protein language models for contact prediction has circulated for some time which lessens the core contribution, but the authors approach is surprisingly data efficient and accurate. Overall this is an interesting work, though there is quite a bit of background on contact prediction missing. This paper is also very application specific and may not present new machine learning methods of general interest to the ICLR community. The existence of previous language model-based contact prediction methods reduces the novelty of this work, especially given that the model used here is from Rives et al. 2019, who already look at contact prediction. Furthermore, no comparisons with state-of-the-art evolutionary coupling-based or language model-based contact prediction methods are performed. With this in mind, the manuscript may be better suited to submission at a biology specific venue.\n\nAdditional specific comments follow below.\n\nMajor comments:\n1.\tMissing related work: there are a number of highly relevant prior works that are not mentioned/discussed. In particular, “Deep generative models of genetic variation capture the effects of mutations” – Riesselman et al. 2018 was, as far as I know, the first paper to show that deep generative models capture structure information (see Figure 6). Following that, “Learning protein sequence embeddings using information from structure” – Bepler & Berger 2019 was, to my knowledge, the first paper to propose deep language models (alignment free) for learning protein sequence representations and used those unsupervised representations for contact prediction. Furthermore, there has been extensive work in improving contact prediction using sequence + co-evolutionary features. See, for example, “Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks” Liu et al. 2018 and “Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model” Wang et al. 2017. Other papers looking at protein structure prediction from sequence with deep learning, though they are less directly relevant, include “End-to-End Differentiable Learning of Protein Structure” AlQuraishi 2018 and “Learning Protein Structure with a Differentiable Simulator” Ingraham 2019.\n2.\tBefore this work, others have looked at fine tuning language models for contact prediction. How do those approaches compare with the approach presented here? Rives et al look at contact prediction in their manuscript describing the transformer model (which is the same model used here) on CASP 11-13 (see Table 5 in their manuscript). How does that approach compare with this one? Likewise for Bepler & Berger\n3.\tMany methods have surpassed GREMLIN for contact prediction using evolutionary couplings. How do those approaches compare with this one? It would be helpful to see how this approach compares with truly state-of-the-art contact prediction methods. Reporting results on the CASP data would help to make this comparison.\n\nMinor Comments:\n1.\tAlthough multiple sequence alignment methods have challenges especially as related to evolutionary coupling prediction, these methods have been heavily optimized for decades. The authors should provide citations for claimed failings such as “failure to find an optimal alignment” and “suboptimality of the substitution matrix and gap penalty.” Certainly, these may be sources of error in alignments, but I am not aware of any studies of the frequency or impacts of these errors on evolutionary coupling analysis. If these studies exist, I encourage the authors to cite them. If they do not exist, I suggest the authors focus on well known sources of error here (namely, alignment depth) and provide references.\n2.\tThe authors use the language model without fine tuning, but the model could be fine tuned for each protein using its MSA. It’s great that contacts can be predicted without fine tuning, but it would be interesting to investigate whether additional gains can be made.\n3.\tEight iterations of jackhmmer is a lot. In my personal experience, jackhmmer often diverges at 3+ iterations. By this I mean, the set of sequences and HMM learned by jackhmmer drift far away from the original sequence/family. Did the authors perform and quality checks of these alignments to ensure jackhmmer did not diverge?\n4.\tHow are sequence depths in Figure 3 calculated? Is this the raw number of sequences in each MSA or is it after applying some sort of neighborhood weighting to calculate an effective number of sequences? \n\nThings that would improve my rating:\n1.\tProvide a more comprehensive background review.\n2.\tCompare with state-of-the-art evolutionary coupling-based contact prediction methods.\n3.\tCompare with other language model-based contact prediction methods.\n4.\tWhat should interest the general machine learning community about this paper? What can we learn that might lead to better ML methods in the future? Convince me that this doesn’t belong in a bioinformatics venue!\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have done a very thorough job of responding to the comments from reviewers. The paper has a clear contribution, namely that attention maps predict contacts as well as existing unsupervised pipelines. This paper deserves to be published.

In the final version, the authors should discuss briefly "BERTology Meets Biology: Interpreting Attention in Protein Language Models"(https://openreview.net/forum?id=YWtLZvLmud7) and "Improving Generalizability of Protein Sequence Models via Data Augmentations" (https://openreview.net/forum?id=Kkw3shxszSd). However, the authors should also make sure that the final version respects the ICLR length limits.

I am recommending poster acceptance because the results are anticlimactic given the recent success of Deepmind at CASP 2020.


Decision: Accept (Poster)