BERTology Meets Biology: Interpreting Attention in Protein Language Models
Authors: Jesse Vig
Ali Madani
Lav R. Varshney
Caiming Xiong
richard socher
Nazneen Rajani
Venue: ICLR 2021 Poster (2021)
Keywords: interpretability
black box
computational biology
representation learning
attention
transformers
visualization
natural language processing

Abstract: Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. In this work, we demonstrate a set of methods for analyzing protein Transformer models through the lens of attention. We show that attention: (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We find this behavior to be consistent across three Transformer architectures (BERT, ALBERT, XLNet) and two distinct protein datasets. We also present a three-dimensional visualization of the interaction between attention and protein structure. Code for visualization and analysis is available at https://github.com/salesforce/provis.

Published as a conference paper at ICLR 2021
BERTOLOGY MEETS BIOLOGY: INTERPRETING ATTENTION IN PROTEIN LANGUAGE MODELS
Jesse Vig' Ali Madani' Lav R. Varshney!2 Caiming Xiong'
Richard Socher'
Nazneen Fatema Rajani'
'Salesforce Research, UUniversity of Illinois at Urbana- -haampaiin (ivig, amadaniiai.............maaa exi ong, rsocher, nazneen. f varshneyeillinois.
ABSTRACT
Transformer architectures have proven to learn useful representations for pro- tein classification and generation tasks. However, these representations present challenges in interpretability. In this work, we demonstrate set of methods for analyzing protein Transformer models through the lens of attention. We show that attention: (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increas- ing layer depth. We find this behavior to be consistent across three Transformer architectures (BERT, ALBERT, XI XLNet) and two distinct protein datasets. We also present three-dimensional visualization of the interaction between atten- tion and protein structure. Code for visualization and analysis is available at httpaa/acocccaaeefoo////oo/ooo/
INTRODUCTION
The study of proteins, the fundamental macromolecules governing biology and life itself, has led to remarkable advances in understanding human health and the development of disease therapies. The decreasing cost of sequencing technology has enabled vast databases of naturally occurring proteins (El-Gebali et al. 2019a), which are rich in information for developing powerful machine learning models of protein sequences. For example, sequence models leveraging principles of co-evolution, whether modeling pairwise or higher-order interactions, have enabled prediction of structure or function (Rollins al. 2019).
Proteins, as a sequence of amino acids, can be viewed precisely as a language and therefore modeled using neural architectures developed for natural language. In particular, the Transformer (Vaswani et al. 2017), which has revolutionized unsupervised learning for text, shows promise for similar impact protein sequence modeling. However, the strong performance of the Transformer comes at the cost of interpretability. and this lack of transparency can hide underlying problems such as model bias and spurious correlations (Niven & Kao, 2019 Tan & Celis 2019 Kurita et al. 2019)). In response, much NLP research now focuses on interpreting the Transtormer, e.g., the subspecialty of "BERTology" (Rogers et al., 2020), which specifically studies the BERT model (Devlin et al., 2019).
In this work, we adapt and extend this line of interpretability research to protein sequences. We analyze Transformer protein models through the lens of attention, and present set of interpretabilitt methods that capture the unique functional and structural characteristics of proteins. We also compare the knowledge encoded attention weights to that captured by hidden-stee representations. Finally, present visualization of attention contextualized within dimensiamenii protein structure.
Our analysis reveals that attention captures high-level structural properties of proteins, connecting amino acids that are spatially close in three-dimensional structure, but apart in the underlying sequence (Figure Taj. We also find that attention targets binding sites, a key functional component of proteins (Figure lb.. Further, show how attention is consistent with classic measure of similarity between amino acids -the substitution matrix. Finally, we demonstrate that attention captures progressively higher-level representations of structure and function with increasing layer depth.

---

Published as a conference paper at ICLR 2021
(a) Attention in head 12-4, which targets amino acid pairs that are close in physical space (see inset subsequence 117D-1571) but lie apart in the sequence. Example is de novo designed TIM- barrel (SBVL) with characteristic symmetry.
(b) Attention in head 7-1, which targets binding sites, a key functional component of proteins. Example is HIV-I protease (7IVP)). The primary location receiving attention is 27G, a binding site for protease inhibitor small maal--moeccule drugs.
Examples of how specialized attention heads in a Transformer recover protein structure and function, based solely on language model pre-training. Orange lines depict attention between amino acids (line width proportional to attention weight; values below 0.1 hidden). Heads were selected based on correlation with ground ground-truth annotations of contact maps and binding sites. Visualizations based on the NGL Viewer (Rose et al., 2018 Rose & Hildebrand 2015 Nguyen et al. 2017).
Figure I:
In contrast to NLP, which aims to automate capability that humans already have understanding natural language- prroeinn modeling also seeks to shed light on biological processes that are not fully understood. Thus we also discuss how interpretability can aid scientific discovery.
2 BACKGROUND: PROTEINS
In this section we provide background on the biological concepts discussed in later sections.
Amino acids. Just language is composed of words from shared lexicon, every protein sequence formed from a vocabulary of amino acids, of which 20 are commonly observed. Amino acids may be denoted by their full name (e.g., Proline), 3-letter abbreviation (Pro), single-letter code (P).
Substitution matrix. While word synonyms are encoded a thesaurus, proteins that are similar in structure or function are captured in a substitution matrix, which scores pairs of amino acids how readily they may be substituted for one another while maintaining protein viability. One common substitution matrix BLOSUM (Henikoff & Henikoff 1992). which is derived from co-occurrence statistics of amino acids in aligned protein sequences.
Protein structure. Though protein may be abstracted sequence of amino acids, represents a physical entity with well-defined three- diree-dimensiona structure (Figure T. Secondary structure describes the local segments of proteins; two commonly observed types are the alpha helix and beta sheet. Tertiary structure encompasses the large-scale formations that determine the overall shape and function of the protein. One way to characterize tertiary structure is by a contact map, which describes the pairs of amino acids that are in contact (within 8 angstroms of one another) in the folded protein structure but lie apart (by at least 6 positions) in the underlying sequence (Rao et al.) 2019).
Binding sites. Proteins may also be characterized by their functional properties. Binding sites protein regions that bind with other molecules (proteins, natural ligands, and -mall-molecule drugs) to carry out specific function. For example, the HIV-1 protease is enzyme responsible for critical process in replication of HIV (Brik& & Wong] 2003). It has a binding site, shown in Figure that is target for drug development to ensure inhibition.
nost-transaaional modifications. After a protein is translated from RNA, it may undergo additional modifications, e.g. phosphorylation, which play key role in protein structure and function.
2

---

Published as a conference paper at ICLR 2021
3 METHODOLOGY
Model. We demonstrate our interpretability methods on five Transformer models that were pretrained through language modeling of amino acid sequences. We primarily focus on the BERT-Base model from TAPE (Rao et al. 2019), which was pretrained Pfam, dataset of 31M protein sequences (EI- Gebali et al. 2019b). We refer to this model as TapeBert. We also analyze 4 pre-trained Transformer
models from Prot Trans (Elnaggar al. 2020): ProtBert and ProtBert-BFD, which are 30-layer,
16-head BERT models; ProtAlbert, a 12-layer, 64-head ALBERT (Lan et al., (020) model; and
ProtXI.Net, 30-layer, 16-head XLNNet ( (Yang et al.
2019) model. ProtBert-BFD was pretrained on
BFD (Steinegger & Söding] (018). a dataset of 2.1B protein sequences, while the other ProtTrans models were pretrained on 1ni (Suzek et al. 2014), which includes 216M protein sequences.
A summary of these 5 models is presented in Appendix A.T
Here we present an overview of BERT, with additional details on all models in Appendix A.2 BERT inputs a sequence of amino acids x (Z1,.... In.) and applies a series of encoders. Each encoder layer l outputs sequence of continuous embeddings (h mechanism. Each attention head in a layer produces a set of attention weights a for an input, where at.j > 0 the attention from token to token j. such that Σ, os.j 1. Intuitively, attention weights define the influence of every token on the next layer's 's representation for the current token. We denote a particular head by <layer>-<head index>, e.g. head 3-7 for the 3rd layer's 7th head.
hn using multi-headed attention
Afention analysis. We analyze how attention aligns with various protein properties. For properties of token pairs, e.g. contact maps, we define an indicator function fi,,j) that returns 1 if the property is present in token pair (i.j) (e.g.. if amino acids i and j are in contact), and 0 otherwise. We then compute the proportion of high-attention token pairs (ax. > 0) where the property is present, aggregated over dataset X:
X ΣΣΣ x
Pa(f) = -10=
f(i.))
Laig
La., >0
xEX
where 0 is a threshold to select for high-confidence attention weights. We also present an alternative, continuous version of this metric in Appendix [B.]]
For properties of individual tokens, e.g. binding sites, we define fii, to return 1 if the property is present in token j (e.g. if j is a binding site). In this case, (f) equals the proportion of attention that is directed to the property (e.g. the proportion of attention focused on binding sites).
When applying these metrics, we include two types of checks to ensure that the results are not due to chance. First, we test that the proportion of attention that aligns with particular properties is significantly higher than the background frequency of these properties, taking into account the Bonferroni correction for multiple hypotheses corresponding to multiple attention heads. Second, we compare the results to a null model, which is an instance of the model with randomly shuffled attention weights. We describe these methods in detail in Appendix [B.2]
Probing tasks. We also perform probing tasks on the model, which test the knowledge contained in model representations by using them inputs to classifier that predicts property of interest (Veldhoen et al.] 2016 Conneau et al. 2018 Adi et al., (016). The performance of the probing classifier serves as measure of the knowledge of the property that encoded in the representation. We run both embedding probes, which assess the knowledge encoded in the output embeddings of each layer, and attention probes (Reil al., 2019 Clark et al. 2019), which measure the knowledge contained in the attention weights for pairwise features. Details are provided in Appendix B.3
For our analyses of amino acids and contact maps, we use curated dataset from TAPE based on ProteinNet (AIQuraishi] 2019; Fox et al. 2013; Berman et al. 2000; Moult et al., 2018). which contains amino acid sequences annotated with spatial coordinates (used for the contact map analysis). For the analysis of secondary structure and binding sites we use the Secondary Structure dataset (Rao et al. 2019 Berman et al. [2000] Moult et al. 2018 Klausen et al. 2019) from TAPE. We employed a taxonomy of secondary structure with three categories: Helix, Strand, and Turn/Bend, with the last two belonging to the higher- beta sheet category (Sec. 2 We used this taxonomy to study how the model understood structurally distinct regions of beta sheets. We obtained leven-level binding site and protein modification labels from the Protein Data Bank (Berman et al.] 2000). For analyzing attention, used random subset of 5000 sequences from the training split of the
Datasets.
3

---

Published as a conference paper at ICLR 2021
Tv. Actentionn
Max
% Attention
49%
30%
158-
56%
(a) TapeBert
(b) ProtAlbert
% Attention
Max
% Attention
Max
% Attention
Max
30
30
28
28
28
26
45%
20
22
30%
20
22
Layer
30%
30%
16
15%
10%
50%.
40%
25%
Head
Head
Head
(c) ProtBert
(d) ProtBert-BBE
(e) ProtXLNet
Figure 2: Agreement between attention and contact maps across five pretrained Transformer models from TAPE (a) and ProtTrans (b-e). The heatmaps show the proportion of high-confidene
attention weights (a;,j > 0) from each head that connects pairs of amino acids that are in contact with one another. In TapeBert (a), for example, we can see that 45% of attention in head 12-4 (the 12th layer's 4th head) maps contacts. The bar plots show the maximum value from each layer. Note that the vertical striping in ProtAlbert (b) is likely due to cross-layer parameter sharing (see Appendix A.3
respective datasets (note that none of the aforementioned annotations were used in model training). For the diagnostic classifier, we used the respective training splits for training and the validation splits for evaluation. See Appendix B.4 for additional details.
Experimental details We exclude attention to the [SEP] delimiter token, as it has been shown to be a "no-op" attention token (Clark al., 2019), well as attention to the [CLS] token, which is not explicitly used in language modeling. We only include results for attention heads where at least 100 high-confidence attention to select for high-confidence attention while retaining sufficient data for analysis. We truncate all protein sequences length of 512 to reduce memory requirements
available for analysis. We set the attention threshold to 0.3
We note that all of the above analyses are purely associative and do not attempt to establish a causal link between attention and model behavior (Vig et al., 2020 Grimsley et al. 2020), nor explain model predictions (Jain & Wallace, 2019: Wiegreffe & Pinter 2019).
WHAT DOES ATTENTION UNDERSTAND ABOUT PROTEINS?
4.1 PROTEIN STRUCTURE
Here we explore the relationship between attention and tertiary structure, as characterized by contact maps (see Section 2). Secondary structure results are included in Appendix C.I
Attention aligns strongly with contact maps in the deepest layers, Figure 2 shows how attention aligns with contact maps across the heads of the five models evaluated based the metric defined in Equation [ The most aligned heads are found in the deepest layers and focus up to 44.7% (TapeBert), 55.7% (ProtAlbert), 58.5% (ProtBert), 63.2% (ProtBert-BFD), and 44.5% (ProtXI.Net) of attention on contacts, whereas the background frequency of contacts among all amino acid pairs in the dataset 1.3%. Figure La shows an example of the induced attention from the top head in TapeBert. We note that the model with the single most aligned head Protoert-BBil ---Berrr-BD--ss the largest model (same size as ProteinBert) 420M parameters (Appendix A.1 and it was also the only model pre- rreatriined on the
194% of sequences had length less than 512. Experiments performed on single 16GB Tesla V-100 GPU.
Heads with fewer than 100 high-confidence attention weights across the dataset grayed
4

---

Published as a conference paper at ICLR 2021
Tv. Actention
Max
% Attention
Max
45%
50%
15%
(a) TapeBert
(b) ProtAlbert
% Attention
Max
% Attention
Max
% Attention
Max
50%
30
30
28
40%
28
28
40%
12%
22
22
20
30%
16
20%
4%
10%
10%
50%
25%
NEC PONE
10%
Head
Head
Head
(c) ProtBert
(d) ProtBert-BFD
(e) ProtXLNet
Figure 3: Proportion of attention focused on binding sites across five pretrained models. The heatmaps show the proportion of high- confieenfee attention (as..) >> 0) from each head that is directed to binding sites. In TapeBert (a), for example, we can see that 49% of attention in head 11-6 (the I Ith layer's 6th head) is directed to binding sites. The bar plots show the maximum value from each layer.
largest dataset, BFD. It's possible that both factors helped the model learn more coucturally-alignnn attention patterns. Statistical significance tests and null models are reported in Appendix C.2
Considering the models were trained on language modeling tasks without any spatial information, the presence of these structurally-awaat attention heads is intriguing. One possible reason for this emergent behavior is that contacts are more likely to biochemically interact with one another, creating statistical dependencies between the amino acids in contact. By focusing attention the contacts of a masked position, the language models may acquire valuable context for token prediction.
While there seems be strong correlation between the attention head output and classically-der -eeffne contacts, there are also differences. The models may have learned differing contextualized or nuanced formulations that describe amino acid interactions. These learned interactions could then be used for further discovery and investigation or repurposed for prediction tasks similar to how principles of coevolution enabled a powerful representation for structure prediction.
4.2 BINDING SITES AND POST-TRANSLATIOON MODIFICATIONS
We also analyze how attention interacts with binding sites and post-translational modifications (PTMs), which both play key role in protein function.
Attention targets binding sites throughout most layers of the models. Figure 3 shows the propor- tion of attention focused on binding sites (Eq.[) across the heads of the 5 models studied. Attention binding sites most pronounced in the ProtAlbert model (Figure 3b), which has 22 heads that focus over 50% of attention on bindings sites, whereas the background frequency of binding sites in the dataset is 4.8%. The three BERT models (Figures 3a 3c and 3d) also attend strongly binding sites, with attention heads focusing up to 48.2%, %0.7%, and 45.6% of attention on binding sites, respectively. Figure Th) visualizes the attention in one strongly-aligne head from the TapeBert model. Statistical significance tests and comparison to null model are provided in Appendix C.3
ProtXLNet (Figure 3e) also targets binding sites, but not as strongly as the other models: the most aligned head focuses 15. % of attention on binding sites, and the average head directs just 6.2% of attention to binding sites, compared to 13.2%. 19.8%, 16.0%, and 15.1% for the first four models in Figure 3 It's unclear whether this disparity is due to differences in architectures pre- pre-training objectives; for example, ProtXLNet uses a bidirectional auto regresseeee pretraining method (see Appendix A.2), whereas the other models all masked language modeling objectives.
5

---

Published as a conference paper at ICLR 2021
Helk
Hellx
10%
0.6
0%
Turn/Bend
Turn/Ben
20%
0.2
0%
0.8
Strand
Strand
0.2
Contact Binding Site
0.16
Contact Binding
0.14
0.12
0%
0.10
Embediing probe
Attundon
0.15
1%
0.10
0.05
0%
12
Layer
Layer
Figure 4: Each plot shows the percentage of attention focused on the given property, av- eraged over all heads within each layer. The plots, sorted by center of gravity (red dashed line), show that heads in deeper layers focus relatively more attention on binding sites and contacts, whereas attention toward specific secondary structures is more even across lay- ers.
Figure 5: Performance of probing classifiers by layer, sorted by task order in Figure 4 The embedding probes (orange) quantify the knowledge of the given property that is en- coded in each layer's output embeddings. The attention probe (blue), show the amount of in- formation encoded in attention weights for the (pairwise) contact feature. Additional details are provided in Appendix B.3
Why does attention target binding sites? In contrast to contact maps, which reveal relationships within proteins, binding sites describe how protein interacts with other molecules. These external interactions ultimately define the high-level function of the protein, and thus binding sites remain conserved even when the sequence motifs in binding sites are mainly restricted to specific families or superfamilies of proteins (Kinjo & Nakamura 2009), and binding sites can reveal evolutionary relationships among proteins (Lee et 2017). Thus binding sites may provide the model with high-level characterization of the protein that is robust to individual sequence variation. By attending these regions, the model can leverage this higher-level context when predicting masked tokens throughout the sequence.
whole evolves (Kinjo & Nakamura 2009). Further, structural
Atfention targets PTMs in a small number of heads. small number of heads in each model con- centrate their attention very strongly on amino acids associated with post-translational modifications (PTMs). For example, Head 11-6 in TapeBert focused 64% of attention on PTM positions, though these occur at only 0.8% of sequence positions in the dataset Similar to discussion on binding sites, PTMs are critical to protein function (Rubin & Rosen 1975) and thereby are likely to exhibit behavior that is conserved across the sequence space. See Appendix C.4 for full results.
4.3 CROSS-LAYER ANALYSIS
We analyze how attention captures properties of varying complexity across different layers of TapeBert, and compare this to probing analysis of embeddings and attention weights (see Section 3).
Attention targets higher-level properties in deeper layers. As shown in Figure 4
deeper layers focus relatively more attention on binding sites and contacts (high-level concept), whereas secondary structure (low- mid-level concept) is targeted more evenly across layers. The probing analysis of attention (Figure [
blue) similarly shows that knowledge of contact maps (a pairwise feature)
Thiis head also targets hinding sites (Fig. 3a
percentage of 49%.

---

Published as a conference paper at ICLR 2021
% Attention
Max
% Attention
Max
12
12
80%
10
10
15%
Layas
Lave
40%
10%
20%
5%
12
100%
12
20%
Figure 6: Percentage of each head's attention focused on amino acids Pro (left) and Phe (right).
98
06
02
-22
-44
CIGHGII PLMMPORST
ACDEFGG W.N
Figure 7: Pairwise attention similarity (left) vs. substitution matrix (right) (codes in App. C.5
encoded in attention weights primarily in the last 1-2 layers. These results are consistent with prior work in NLP that suggests deeper layers in text-based Transformers attend more complex properties (Vig & Belinkov 2019) and encode higher-level representations (Raganato & Tiedemann
20199)
2018 Peters al. 2018 Tenney al. 2019 Jawahar et al. 2019).
The embedding probes (Figure 5 orange) also show that the model first builds representations of local secondary structure in lower layers before fully encoding binding sites and contact maps in deeper layers. However, this analysis also reveals stark differences in how knowledge of contact maps accrued in embeddings, which accumulate this knowledge gradually over many layers, compared to attention weights, which acquire this knowledge only in the final layers in this case. This example points out limitations of common layerwise probing approaches that only consider embeddings, which, intuitively, represent what the model knows but not necessarily how it operationalizes that knowledge.
4.4 AMINO ACIDS AND THE SUBSTITUTION MATRIX
In addition to high-level structural and functional properties, we also performed a fine- lne-grained analysis of the interaction between attention and particular amino acids.
Attention heads specialize in particular amino acids. We computed the proportion of TapeBert's attention each of the 20 standard amino acids, shown in Figure for two example amino acids. For 16 of the amino acids, there exists an attention head that focuses over 25% of attention on that amino acid, significantly greater than the background frequencies of the corresponding amino acids, which range from 1.3% to 9.4%. Similar behavior was observed for ProtBert, ProtBert-BFD. ProtAlbert, and ProtXLNet models, with 17, 15, 16, and 18 amino acids, respectively, receiving greater than 25% of the attention from at least one attention head. Detailed results for TepeBert including statistical significance tests and comparison
null model are presented in Appendix C.5
Attention is consistent with substitution relationships. A natural follow-up question from the above analysis whether each head has "memorized" specific amino acids target, or whether has actually learned meaningful properties that correlate with particular amino acids. To test the latter hypothesis, we analyze whether amino acids with similar structural and functional properties are attended to similarly across heads. Specifically, distribution attention across heads between all pairs distinct amino acids, as shown in Figure 7 (left) for TapeBert. For example, the entry for Pro (P) and Phe (F) is the correlation between the two heatmaps in Figure [ We compare these scores to the BLOSUM62 substitution scores (Sec. 2 in Figure 7 (right), and find Pearson correlation of 0.73, suggesting that attention is moderately
compute the Pearson correlation between the

---

Published as a conference paper at ICLR 2021
consistent with substitution relationships. Similar correlations observed for the ProtTrans models: 0.68 (ProtBert), 0.75 (ProtBert- -eee,, 0.60 (ProtAlbert), and 0.71 (ProtXLNet). As a baseline, the randomized versions of these models (Appendix B.2) yielded correlations of -0.02 (TapeBert), 0.02 (ProtBert), -0.03 (ProtBert-BFD), -0.05 (ProtAlhert), and 0.21 (ProtXI.Net).
5 RELATED WORK
5.1 PROTEIN LANGUAGE MODELS
Deep neural networks for protein language modeling have received broad interest. Early work applied the Skip-gram model (Mikolov et al.) 2013) to construct continuous embeddings from protein sequences (Asgari & Mofrad 2015). Sequence-only language models have since been trained through autoregressive or autoencoding self-supervision objectives for discriminative and generative tasks, for example, using LSTMs or Transformer-based architectures (Alley et al.) 2019 Bepler & Berger 2019; Rao et al. 2019 Rives et al.] 2019). TAPE created a benchmark of five tasks to assess protein sequence models, and ProtTrans also released several large-scale pretrained protein Transformer models (Elnaggar et al.
2020). [Riesselman et al. (2019); )) Madani et al. (2020) trained autoregressive
generative models to predict the functional effect of mutations and generate natural-like proteins.
From an interpretabilityy perspective, |Rives et al. (2019) showed that the output embeddings from a pretrained Transformer can recapitulate structural and functional properties of proteins through learned linear transformations. Various works have analyzed output embeddings of protein models through dimensionality reduction techniques such as PCA or t-SNE (EInaggar et al. 2020 Biswas et al. 2020). In our work, we take interpretabilityy firlltttyyyittttttt perspective to focus the internal model representations, specifically attention and intermediate hidden states, across multiple protein language models. We also explore novel biological properties including binding sites and post-translational modifications.
5.2 INTERPRETING MODELS IN NLP
The rise of deep neural networks in ML has also led to much work on interpreting these so-called black-box models. This section reviews the NI.P interpretability literature on the Transformer model, which is directly comparable to our work on interpreting Transformer models of protein sequences.
Interpreting Transformers. The Transformer is a neural architecture that uses attention ac- celerate learning (Vaswani et al.) 2017). In NLP, transformers are the backbone of state-of-the-arr pre-trained language models such as BERT (Devlin et al. 2019). BERTology focuses on interpreting what the BERT model learns about language using suite of probes and interventions (Rogers et al. 2020). So-called diagnostic classiffers are used to interpret the outputs from BERT's layers (Veldhoen al. 2016). At high level, mechanisms for interpreting BERT can be placed into three main categories: interpreting the learned embeddings (Ethayarajh) 2019 Wiedemann et al., 2019 Mickus
let al.
2020; Adi et al. 2016 Conneau 2016:Conneau et al. 2018), BERT's learned knowledge of syntax (Lin et al.
2019
Liu et al. 2019: Tenney et al.) 2019 Htut et al.) 2019: Hewitt & Manning 2019 Goldberg
2019
and BERT's learned knowledge of semantics (Tenney al. 2019 Ettinger 2020
Interpreting attention specifically. Interpreting attention on textual sequences is well-
established area of research (Wiegreffe & Pinter) 2019: Zhong et al. [2019 Brunner et al. 2020
Hewitt & Manning 2019). Past work has been shown that attention correlates with syntactic and
semantic relationships in natural language in some cases (Clark et al. 2019 Vig & Belinkov 2019
Htut et al. 2019). Depending on the task and model architecture, attention may have less or more
explanatory power for model predictions (Jain& & Wallace [019 Serrano & Smith 2019 Pruthi et al.
2020, Moradi et al. 2019. Vashishth et al. 2019). Visualization techniques have heen used to convey the structure and properties of attention in Transformers (Vaswani et al. 2017: Kovaleva et al. 2019
Hoover et al. 20200 Vig 2019). Recent work has begun to analyze attention in Transformer models
outside of the domain of natural language (Schwaller et al. 2020 Payne al. 2020
Our work extends these methods to protein sequence models by considering particular biophysical properties and relationships. We also present joint cross-layer probing analysis of attention weights and layer embeddings. While past work in NLP has analyzed attention and embeddings across layers, believe the first to do in any domain using a single, unified metric, which enables to
8

---

Published as a conference paper at ICLR 2021
directly compare the relative information content of the two representations. Finally, we present novel tool for visualizing attention embedded in
structure.
CONCLUSIONS AND FUTURE WORK
This paper builds on the synergy between NLP and computational biology by adapting and extending NLP interpretability methods to protein sequence modeling. We show how Transformer language model recovers structural and functional properties of proteins and integrates this knowledge directly into its attention mechanism. While this paper focuses on reconciling attention with known properties of proteins, one might also leverage attention to uncover novel relationships or more nuanced forms of existing measures such as contact maps, as discussed in Section 4.1 In this way, language models have the potential to serve as tools for scientific discovery. But in order for learned representations to accessible domain experts, they must be presented in an appropriate context to facilitate discovery. Visualizing attention in the context of protein structure (Figure I is one attempt to do so. We believe there the potential develop such contextual visualizations learned representations in range of scientific domains.
ACKNOWLEDGMENTS
We would like to thank Xi Victoria Lin, Stephan Zheng, Melvin Gruesbeck, and the anonymous reviewers for their valuable feedback.
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg, Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv: 04207 [cs.CL].. 2016.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature Methods, 16(12):1315-1333 2019.
Mohammed AlQuraishi. ProteinNet: standardized data set for machine learning of protein structure. BMC Bioinformatics, 20, 2019.
Ehsaneddin Asgari and Mohammad RK Mofrad. Continuous distributed representation of biological sequences for deep proteomics and genomics. PLOS One, 10(1 1), 2015.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic Acids Research, 28(1): 235-242, 2000.
Surojit Biswas, Grigory Khimulya, Ethan C. Alley, Kevin M. Esvelt, and George M. Church. Low-n protein engincering with data-cfficient deep learning. bioRxiv, 2020. doi: 10.1101/ 2020.01.23.91700 URL https:: . wwwpww.wwwwwww..wwwwwwwwwwwwwwww http:///oooteeeeeaaa//////////// 31/2020.
Ashraf Brik and Chi-Huey Wong. HIV-1 protease: Mechanism and drug discovery. Organic & Biomolecular Chemistry, 1(1):5-14, 2003.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Watten- hofer. On identifiability in Transformers. In International Conference on Learning Representations, 2020. URL
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT's attention. In BlacKBBaNNIP@AAT,220011

---

Published as a conference paper at ICLR 2021
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2126-2136, 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, 2019. Association for Computational Linguistics.
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean Eddy, Aurélien Luciani, Simon C. Potter, Matloob Qureshi, Lorna J. Richardson, Gustavo A. Salazar, Alfredo Smart, Erik L.. L.. Sonnhammer, Layla Hirsh, Lisanna Paladin, Damiano Piovesan, Silvio C. E. Tosatto, and Robert D. Finn. The Pfam protein families database in 2019. Nucleic Acids Research, 47(DI):D42--D333 January 2019a. doi: 10.1093/nar/gky995.
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean R Eddy, Aurélien Luciani, Simon C Potter, Matloob Qureshi, Lorna J Richardson, Gustavo A Salazar, Alfredo Smart, Erik L L Sonnhammer, Layla Hirsh, Lisanna Paladin, Damiano Piovesan, Silvio C E Tosatto, and Robert D Finn. The Pfam protein families database in 2019. Nucleic Acids Research, 47(D1):D427 D432, 2019b. ISSN 0305-1048. doi: 10. 10.10933/ar/gky9955
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing. arXiv preprint arXiv:2007.06223 2020.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, Hong Kong, China, 2019. Association for Computational Linguisties.
Allyson Eutinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
Naomi Fox, Steven E Brenner, and John-Mare Chandonia. SCOPe: Structural classification of proteins extended, integrating scop and astral data and classification of new structures. Nucleic Acids Research, 42(D1))D334 D309, 2013.
Yoav Goldberg. Assessing BERT's syntactic abilities. arXiv preprint 1901v 2019.
Christopher Grimsley, Elijah Mayfield, and Julia R.S. Bursten. Why attention is not explanation: Surgical intervention and causal reasoning about neural models. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 1780-1790, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL http::////w/
aclweb. org/ http///nntoooooyy/2222a/ 1rec-1.220
S Henikoff and G Henikoff. Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences, 89(22) 1992. ISSN 0027-8424 doi: 10.1073/ pnas.89.22 09........222. 10915. URL htt.hhttss:///t http://wwwpppp......am.../..a..t ppas. /ttp://onnntnntnn////n////
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
pp. 4129 4138, 2019.
Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187-196. Association for Computational Linguistics, 2020.
10

---

Published as a conference paper at ICLR 2021
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R Bowman, Do attention heads in BERT track syntactic dependencies? arXiv preprint arXiv:1911.12246. 2019.
Sarthak Jain and Byron C. Wallace. Attention not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 3543-3556, June 2019.
Ganesh Jawahar, Benoft Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In ACL. 2019 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. URL
Akira Kinjo and Haruki Nakamura. Comprehensive structural classification of ligand-binding motifs in proteins. Structure, 17(2), 2009.
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Jurtz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, et al. NetSurfP-2.0t Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 2019.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky, Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp. 4365 4374, Hong Kong, China, 2019. Association for Computational Linguistics.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pp. 166-172, Florence, Italy, 2019. Association for Computational Linguisties.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
Juyong Lee, Janez Konc, Dusanka Janezic, and Bernard Brooks. Global organization of a binding site network gives insight into evolution and stracture-function relationships of proteins. Sci Rep, (11652), 2017.
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACI. Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP,, pp. 241-253, Florence, Italy, 2019. Association for Computational Linguisties.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 1073-1094, Association for Computational Linguistics, 2019.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497, 2020.
Timothee Mickus, Mathieu Constant, Denis Paperno, and Kees Van Deemter. What do you mean, BERT? Assessing BERT as a Distributional Semantics Model. Proceedings of the Society for Computation in Linguistics, 3, 2020.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems Curran Associates, Inc., 2013.
26. pp. 3111-3119.

---

Published as a conference paper at ICLR 2021
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. Interrogating the explanatory power of attention in neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 221-230, Hong Kong, November 2019. Association for Computational Linguistics. doi: 10. 10.18665/1//D11956222 URL http::tts:///tttt //wp::::/wwww..// ac wew.leeee www.wrgg//eeo..............o..we
org/ hogg//ndi anthology/
D1 D19-5624
John Moult, Krzysztof Fidelis, Andriy Kryshtafovych, Torsten Schwede, and Anna Tramontano. Critical assessment of methods of protein structure prediction (CASP)-Round XII. Proteins: Structure, Function, and Bioinformaties, 86:7-15, 2018. ISSN 08873585. doi: 10. 20.1002/rot.25415.
URL httphhttp:/deee www.o..wwwly... www.lley.coomlle. wttp:/cccmo......
/0. 10000/0..0000000
25415
Hai Nguyen, David A Case, and Alexander $ Rose. V.G..vieeiinteraattin molecular graphics for
Jupyter notebooks. Bioinformatics, 34(7):1241-1242, 12 2017. ISSN 1367-4803, doi: 10.1093/ bioinformaticssbux788, URL
Timothy Niven and Hung-Yu Yu Kao. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4658-4664, Florence, Italy, 2019, Association for Computational Linguistics.
Josh Payne, Mario Srouji. Dian Ang Yap. and Vineet Kosaraju. Bert learns (and teaches) chemistry. arXiv preprint arXiv:2007.16012 2020.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499-1509, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v//DI8-111 URL
https:// www ac www.clwwbb.coo www.:/wwwww..........o..ooee htt.e//..rrrrattoooog/o//----11-
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. Learning to deceive with attention-based explanations. In Annual Conference of the Association for Computa- tional Linguistics (ACL), July 2020. URL
Alessandro Raganato and Jörg Tiedemann. An analysis of encoder representations in Transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyyiin and Interpreting Neural Networks for NLP, pp. 287-297, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.11653/v1/W11-555 5431. URL https://www
acl hntg//athhllggo /w1g///18-500 5431
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with TAPE. In Advances in Neural Information Processing Systems, 2019.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of BERT. H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. pp. 8594-8603. Curran Associates, Inc., 2019.
Adam J Riesselman, Jung-Eun Shin, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora Marks. Accelerating protein design using autoregressive generative models. bioRxiv, pp. 757252, 2019.
Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv. pp. 622803, 2019.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866. 2020.
Nathan J Rollins, Kelly P Brock, Frank J Poelwijk, Michael A Stiffler, Nicholas P Gauthier, Chris Sander, and Debora S Marks. Inferring protein 3D structure from deep mutation scans. Nature Genetics, (1(7):1 170, 2019.
12

---

Published as a conference paper at ICLR 2021
Alexander S. Rose and Peter W. Hildebrand. NGL Viewer: web application for molecular vi- sualization. Nucleic Acids Research, 43(W1):W57 W579, 04 2015. ISSN 0305-1048. doi: 10.1095hnar/gk4402. URLhttps::// /tt::///oi..oo wttpoo.ogg.......... 10930.0...11//... gkv402
Alexander S Rose, Anthony R Bradley, Yana Valasatava, Jose M Duarte, Andreas Prlić, and Peter W Rose. NGL viewer: web-based molecular graphics for large complexes. Bioinformatics, 34(21): 3755-3758, 05 2018. ISSN 1367-4803,
doi: 10..09/hooonffrmmaiicc/yy/1l URL https
//doi.org/1i http://ggw.....oooooooooooooaaa/
Charles Rubin and Ora Rosen. Protein phosspooylooooitioonnnn. Anmaa Annual Review of Biochemistry, 44:831-887,
1975. Utthttss:://dd
org http:/::///01../11111111140
44.4 004151
Philippe Schwaller, Benjamin Hoover, Jean-Louis Reymond, Hendrik Strobelt, and Teodoro Laino. Unsupervised attention-guided atom-mapping. ChemRxiv, 5 2020. doi: 10.26434/
chemmxiv.12298555wv1 URL /attps://chemxxzz http://hhwnnnt.r....raraar/eeee/ Attention: Guided_ Atom Mappottg/22225555500000
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931-2951, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19-12811881 URL https
//www. www.wa.wlwbb.... org http://oorr..atttoooog..........
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature Communications, 9(2542), 2018. doi: 10. 10.1038/<41467-018 11-00964-55.
Baris E. Suzek, Yuqi Wang, Hongzhan Huang, Peter B. McGarvey, Cathy H. Wu, and the UniProt Con- sortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, (166))2 932, 11 2014. ISSN 1367-4803. doi: 10.1/93/ bioinformaticc/buu733. Utt.ttts:://////tct . wrg. 10. 10w3/0099/iiiio ormt htttmmaataa/ttt///
Yi Chern Tan and Elisa Celis. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems 32, pp. 13230-1322
Curran Associates, Inc., 2019.
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, 2019, Association for Computational Linguistics.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. Attention inter- pretability across NLP tasks. arXiv preprint arXiv: /2/8, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Sara Veldhoen, Dieuwke Hupkes, and Willem H. Zuidema. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@NIPS, 2016.
Jesse Vig. A multiscale visualization of attention in the Transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37-42, Florence, Italy, 2019, Association for Computational Linguistics.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention Transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63-76, Florence, Italy, 2019. Association for Computational Linguistics.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pp. 1 12401, 2020.
Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv: 10430, 2019.
13

---

Published as a conference paper at ICLR 2021
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, November 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, dAlché-Bol Buc, E. Fox, and R. Gamett (eds.), Advances Neural Information Processing Systems, volume 32, pp. 5753-5763. Curran Associates, Inc., 2019.
Ruiqi Zhong, Steven Shao, and Kathleen McKeown. Fine-grained sentiment analysis with faithful attention, XrXiv preprint arXiv:
2019.
14

---

Published as a conference paper at ICLR 2021
A MODEL OVERVIEW
A.1 PRE-TRAINED MODELS
Table D provides overview of the five pre-trained Transformer models studied in this work. The models originate from the TAPE and ProtTrans repositories, spanning three model architectures: BERT, ALBERT, and XLNet.
Table I: Summary of pre-trained models analyzed, including the source of the model, the type of
Transformer used, the number of layers and heads, the total number of model parameters, the source of the pre-training dataset, and the number protein sequences the pre- tre-training dataset.
Source
Name
Type
Layers
Heads
Params
Train Dataset
# Seq
TAPE
TapeBert
BERT
12
12
94M
Pfam
31M
ProtTrans
ProtBert
BERT
30
16
420M
Uniref100
216M
ProtTrans
ProtBert-BFD
BERT
30
16
420M
BFD
2.1B
ProfTrans
ProtAlbert
ALBERT
12
64
224M
0niref100
216M
Pro[Trans
ProtXLNet
XLNet
30
16
409M
Uniref100
216M
A.2 BERT TRANSFORMER ARCHITECTREE
Stacked Encoder: BERT uses a stacked-encoder architecture, which imputs sequence of tokens = (Z)...., and applies position and token embeddings followed by series of encoder layers. Each layer applies multi-head self-attention (see below) combination with a feedforward network, layer normalization, and residual connections. The output of each layer is sequence of contextualized embeddings (h (h("...., h!²
......
Self-Attentio:: Given an input æ = (x1, In). the self-attention mechanism assigns to each token pair i, attention weight ox, > 0 where E, Ci.j 1. Attention in BERT is bidirectional. In the multi-layer, multi-head setting, is specific to layer and head. The BERT-Base model has 12 layers and 12 heads. Each attention head learns a distinct set of weights, resulting in 12 x 12 = 144 distinct attention mechanisms in this case.
The attention weights oijj are computed from the scaled dot-product of the query vector of i and the key vector of j. followed by softmax operation. The attention weights then used produce weighted sum of value vectors:
QQT Vdz
Attention(Q. K) softmax
using query matrix Q. key matrix K, and value matrix V. where de is the dimension of K. In multi-head setting, the queries, keys, and values are linearly projected h times, and the attention operation performed in parallel for each representation, with the results concatenated.
A.3 OTHER TRANSFORMER VARIANTS
The architecture of ALBERT differs from BERT in two ways: (1) It shares parameters across layers, unlike BERT which learns distinct parameters for every layer and (2) It uses factorized embeddings, which allows the input token embeddings to be of different (smaller) size than the hidden states. The original version of ALBERT designed for text also employed sentence-order prediction pretraining task, but this was used the models studied in this paper.
ALBERT:
XLNet:
Instead of the masked- -anguugge modeling pretraining objective use for BERT, XLNet uses bidirectional rego-segressive pretraining method that considers all possible orderings of the input factorization. The architecture also adds a segment recurrence mechanism to process long sequences, well relative rather than absolute encoding scheme.
15

---

Published as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENTAL DETAILS
B.I ALTERNATIVE ATTENTION AGREEMENT METRIC
Here we present an alternative formulation to Eq. | based on an attention weeihtte-- average. We define an indicator function fi. j) for property f that returns if the property is present in token pair (i,j) (i.e. if amino acids and j are in contact), and zero otherwise. We then compute the proportion of attention that matches with over dataset X as follows:
[2 |2|
12|
Pa(s) = -E
ΣΣΣ aEX i=1==
(3)
=EXi
where (1)(x) denotes the attention from to j for input sequence s.
B.2 STATISTICAL SIGNIFICANCE TESTING AND NULL MODELS
We perform statistical significance tests to determine whether any results based on the metric defined in Equation are due to chance. Given property f. as defined in Section 3 we perform a two- proportion z-test comparing (I) the proportion of high-confidence attention ares (ai. 0) for which f(i,j) = 1, and (2) the proportion of all possible pairs i, for which f(i,j) = 1. Note that the first proportion is exactly the metric Pa (f) defined in Equation | (e.g. the proportion of attention aligned with contact maps). The second proportion is simply the background frequency of the property (e.g. the background frequency of contacts). Since extract the maximum scores over all of the heads in the model, we treat this as case of multiple hypothesis testing and apply the Bonferroni correction, with the number of hypotheses m equal the number of attention heads.
As an additional check that the results did not occur by chance, we also report results on baseline (null) models. We initially considered using two forms of null models: (1) a model with randomly initialized weights. and (2) a model trained on randomly shuffled sequences. However, in both cases, none of the sequences in the dataset yielded attention weights greater than the attention threshold 6. This suggests that the mere existence of the high- high-confidence attention weights used in the analysis could not have occurred by chance, but it does not shed light on the particular analyses performed. Therefore, we implemented an alternative randomization scheme in which we randomly shuffle attention weights from the original models as a post-procee post-processing step. Specifically, we permute the sequence of attention weights from each token for every attention head. To illustrate, let's say that the original model produced attention weights of (0.3, 0.2, 0.1, 0.4, 0.0) from position in protein sequence x from head h, where |æ|| = 5. In the null model, the attention weights from position in sequence x in head h would be a random permutation of those weights, e.g., (0.2, 0.0, 0.4, 0.3, 0.1). Note that these are still valid attention weights as they would sum to I (since the original weights would sum to by definition). We report results using this form of baseline model.
B.3 PROBING METHODOLOGY
Embedding probe. We probe the embedding vectors output from each layer using a linear probing classifier. For token-level probing tasks (binding sites, secondary structure) we feed each token's s output vector directly to the classifier. For loken-pair probing tasks (contact map) we construct a pairwise feature vector by concatenating the elementwise differences and products of the two tokens' output vectors, following the TAPF implementation.
We task task-specific evaluation metrics for the probing classifier: for secondary structure prediction, we measure F1 score; for contact prediction, we measure precision @ L/5, where L is the length of the protein sequence, following standard practice (Moult al. 2018); for binding site prediction, we measure precision @ L 20, since approximately one in twenty amino acids in each sequence is a binding site (4.8% in the dataset).
Attention probe. Just as the attention weight os.j is defined for a pair of amino acids (i,j), so is the contact property fii,j), which returns true if amino acids and j are in contact. Treating the attention weight feature of a token-pair (i,j), we can train a probing classifier that predicts the
https: http::///ggthuu..ccoooo//g/ggg// howg///oogaabba /tape
16

---

Published as a conference paper at ICLR 2021
contact property based on this feature, thereby quantifying the attention mechanism's 's knowledge of that property. our multi-head setting, we treat the attention weights across all heads a given layer feature vector, and use probing classifier to assess the knowledge of given property in the attention weights across the entire layer. As with the embedding probe, we measure performance of the probing classifier using precision @ I/55, where I. is the length of the protein sequence, following standard practice for contact prediction.
B.4 DATASETS
We used two protein sequence datasets from the TAPE repository for the analysis: the ProteinNet dataset (AIQuraishi 2019 Fox et al. 2013 Berman et al. 22000 Moult et al. 2018)) and the Sccondary Structure dataset (Rao et al. [019 Berman et al. 2000 Moult et al.) 2018; Klausen et al. 2019). The former used for analysis of amino acids and contact maps, and the latter used for analysis of secondary structure. We additionally created a third dataset for binding site and post-translational modification (PTM) analysis from the Secondary Structure dataset, which binding site and PTM annotations obtained from the Protein Data Bank's 's Web API We excluded any sequences for which annotations were not available. The resulting dataset sizes shown in Table 2 For the analysis of attention, a random subset of 5000 sequences from the training split of each dataset was used, as the analysis was purely evaluative. For training and evaluating the diagnostic classifier, the full training and validation splits were used.
augmented with
Table 2: Datasets used in analysis
Dataset ProteinNet
Train size
Validation size
25299
224
Secondary Structure
8678
2170
Binding Sites / PTM
5734
1418
C ADDITIONAL RESULTS OF ATTENTION ANALYSIS
C.1 SECONDARY STRUCTURE
Artontion
M5x
40%
375
50%
0%
20
30%
Medd
(a) TapeBert
(b) ProtAlbert
Attention
Max
Attention
Attention
80%
26
26
60%
60%
20
20
Layer
18
40%
14
20%
20%
0%
suh
0%
50%
Head
Head
Head
(c) ProtBert
(d) ProtBent-BDD
(e) ProtXINet
Figure 8: Percentage each head's attention that is focused on Helix secondary structure.
17

---

Published as a conference paper at ICLR 2021
% Attiontion
Max
% Attention
75%
.0%
(a) TapeBert
(b) ProtAlbert
%6 Altention
Max
% Attention
Max
% Altention
Max
28
28
60%
22
22
16
40%
20%
0%
50%
50%
0%
50%
Head
Head
Head
(c) ProtBert
(d) ProtBert-BED
(e) ProIXLNet
Figure 9: Percentage of each head's s attention that is focused on Strand secondary structure.
% Actennton
Max
% Attention
58%
(a) TapeBert
(b) ProtAlbert
% Attention
Max
% Attention
Max
% Attention
Max
28
40%
28
15%
22
30%
22
40%
30%
15%
10%
0%
60%
NIC CONTO
50%
0%
NeC SONES
28%
Head
Head
Head
(c) ProtBert
(d) ProtBentBBDD
(e) ProtXLNet
Figure 10: Percentage of each head's attention that focused on Turn/Bend secondary structure.
18

---

Published as a conference paper at ICLR 2021
C.2
CONTACT MAPS: STATISTICAL SIGNIFICANCE TESTS AND NULL MODELS
Attention %
Attention %
40
20
12422
12-1
125
9-22
8.1
TapeBert
(b) ProtAlbert
Attemtion %
80
Attention
20
20
3014
18-10
20.7
2042
34
300
104
(c) ProtBert
(d) ProtBert- BFD
Attemttonn
20
26-
27:13
6334
25300
25-2
(e) Prt
Figure 11: Top 10 heads (denoted by <layer>-<head>> for each model based on the proportion of attention aligned with contact maps [95% conf. intervals]. The differences between the attention proportions and the background frequency of contacts (orange dashed line) are statistically significant (p 0.00001). Bonferroni correction applied for both confidence intervals and tests (see App. B.2
15
Attention %
1166
20
(a) TapeBert-Rand Random
(b) BouAAleet-Randooe
10
Attention %
Attention
43
7.25
11.7
25:23
6:40
12.12
10.2
4.30
(c) ProtBert-Random
(d) ProtBert-BBP BFF---DDDDRRo
Attention
(e) MeeXLLeee-Rnnnnn
Figure 12: Top- 10 contact-aligned heads for null models. See Appendix B.2 for details.
19

---

Published as a conference paper at ICLR 2021
C.3
BINDING SITES: STATISTICAL SIGNIRICANCE TESTS AND NULL MODEL
Attention N
Attentio %
11422
74
322
347
TapeBert
(b) ProtAlbert
40
Attention %
Attention
20
24-
22-22
22-3
433
5.20
264
231
(c) ProtBert
(d) ProtBert-Bii BFD
Attemtton
12446
99
22:22
13.10
140
(e) Pet
Figure 13: Top 10 heads (denoted by <layer><<head>) for each model based on the proportion of attention focused on binding sites [95% conf. intervals]. Differences between attention proportions and the background frequency of binding sites (orange dashed line) are all statistically significant (p 0.00001). Bonferroni correction applied for both confidence intervals and tests (see App. B.2.
60
Attention %
10
40
12.
114
1242
2:59
12:40
35
60
12
10
(a) SopeBert-Randowe
(b) MaitAllee-Randooe
10
40
Attention
22.15
6.3
0.9
36-22
25-15
13:25
(c) PooBert-RRndom
(d) ProtBert-BFDD BFFD-Rnndom
22.0
22.3
17:10
19:20
10:13
14-13
16.9
F-t
23:18
11:34
(e) ProtXLNet Random
ProtXLNet-Random
Figure 14: Top- 10 heads most focused on binding sites for null models. See Appendix B.2 for details.
20

---

Published as a conference paper at ICLR 2021
C.4 POST-TRANSLATIONAL MODIFICATIONS (PTMS)
Actentionn
Max
Attention
69
15%
25%
TapeBert
(b) ProtAlbert
Attention
Max
Attention
Max
Altention
Max
40%
78
30
28
30
12%
6%.
24
24
30%
22
22
24%
38
18
4%
18
20%
10
16%
3%
25%
6 25%
Shs
Head
(c) ProtBert
(d) ProtBert-BFD
(e) ProtXLNet
Figure 15: Percentage of each head's attention that is focused on post-translational modifications.
21

---

Published as a conference paper at ICLR 2021
Attertion
40
Attention
40
126
97
11422
20.56
3.3
3.29
10:17
TapeBert
(b) ProtAlbert
40
Attention %
Attention %
40
26
25
260
23.3
21:22
25
200
261
204
26-13
(c) ProtBert
(d) ProtBert-Be BFD
7.5
Attention
2.5
0.
11:6
127
1236
1812
166
2014
15-4
12.5
(e) Net
Figure 16: Top heads (denoted by <layer>-<head>) for each model based on the proportion of attention focused PTM positions [95% conf. intervals]. The differences between the attention proportions and the background frequency of PTMs (orange dashed line) are statistically significant (p 0.00001). Bonferroni correction applied for both confidence intervals and tests (see App. B.2).
Attention %
20
116
10-7
9-11
1:57
7:55
TapeBert Random Tandooo
(b) RotAlleet-Random
Attention %
Attention
20
27.15
26-12
26:2
20.9
12.4
12
39
(c) PootBet-RRando
(d) ProtBert- -FF-----mD
Attentio
28:15
1377
M-22
22
24-24
1044
18:3
19.3
15:10
(e)
ProtXLNet Random Renddoomm
Figure 17: Top-10 10 heads most focused on PTMs for null models. See Appendix B.2 for details.
22

---

Published as a conference paper at ICLR 2021
C.5
AMINO ACIDS
607-
40%
15
10%
45%
LavE
128
20%
(a) ALA
(b) ARG
(c) ASN
% Attention
Max
Atteetion
Max
Attention
Max
10%
Layer
<85
10%
365
20%
5/%
50%
0%
5/
Fredd
Head
(d)
ASP
CYS
(f) GLN
% Attention
Max
% Attention
Max
% Attention
Max
10%
12%
477-
Laye
60%
byy
Laye'
10%
08.
504
Head
Huad
(g)
GLU
(h) GLY
(i) HIS
% Attention
Max
Attention
Max
Attention
Max
208
10
20%
Layer
12%
0%
10%
0%
09
H20d
(i) ILE
(k) LEU
(I) LYS
Attention
Attention
6076
20%
609
30
30%
15%
20%
50%
25%
100
Head
Head
(m) MET
(n) PHE
(o) PRO
Figure 18: Percentage of each head's attention that is focused the given amino acid, averaged over a dataset (TapeBert).
23

---

Published as a conference paper at ICLR 2021
Attention
Max
Attention
Max
Attention
Mix
12%
10%
10
17%
(a) SER
(b) THR
(c) TRP
Attention
Attention
32%
15%
10
20%
109
Layd
25%
Heed
(d)
(c) VAL
Figure 19: Percentage of each head's attention that is focused the given amino acid, averaged over a dataset (cont.)
Table 3: Amino acids and the corresponding maximally attentive heads in the standard and randomized versions of TapeBert. The differences between the attention percentages for TapeBert and the background frequencies of each amino acid are all statistically significant (p < 0.00001) taking into account the Bonferroni correction. See Appendix [B2 for details. The bolded numbers represent the higher of the two values between the standard and random models. all cases except for Glutamine, which the amino acid with the lowest top attention proportion in the standard model (7.1), the standard TapeBert model has higher values than the randomized version.
TapeBert
TapeBert
TapeBert-Random Random
Abbrev
Code
Name
Background %
Top Head
Attn %
Top Head
Attn %
Ala
A
Alanine
7.9
12-11
25.5
11-12
12.1
Arg
R
Arginine
5.2
12-8
63.2
12-7
8.4
Asn
N
Asparagine
4.3
8-2
44.8
8-2
6.7
Asp
D
Aspartic acid
5.8
12-6
79.9
10.7
Cys
C
Cysteine
1.3
11-6
83.2
11-6
9.3
Gln
Q
Glutamine
3.8
11-7
7.1
12-1
9.2
Glu
E
Glutamic acid
6.9
11-7
16.2
11-4
11.8
Gly
G
Glycine
7.1
2-11
98.1
11-8
14.6
His
H
Histidine
2.7
9-10
56.7
11-6
5.4
lle
Isoleucine
5.6
11-10
27.0
9-5
10.6
Leu
Leucine
9.4
2-12
44.1
12-11
13.9
Lys
K
Lysine
6.0
12-8
29.4
6-11
12,9
Met
M
Methionine
2.3
3-10
73.5
9-3
6.2
Phe
F
Phenylalanine
3.9
12-3
22.7
12-1
6.7
Pro
Proline
4.6
1-11
98.3
10-6
7.6
Ser
S
Serine
6.4
12-7
36.1
11-12
11.0
Thr
T
Threonine
5.4
12-7
19.0
10-4
9.0
Trp
W
Tryptophan
1.3
11-4
68.1
9-2
3,0
Tyr
Y
Tyrosine
3,4
12-3
51.6
12-11
6.6
Val
V
Valine
6.8
12-11
34.0
8-2
15.0
24


Reviews:
[{"review_id": "h-GnHWxXjC1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General comments:\n\n1) authors claim\n\n\"We note that all of the above analyses are purely associative and do not attempt to establish a causal link between attention and model behavior (Vig et al., 2020; Grimsley et al., 2020), nor to explain model predictions”. \n\nand in the related work section\n\n\"In our work, we take an interpretability-first perspective to focus on the internal model representations, specifically attention and intermediate hidden states, across multiple protein language models. We also explore novel biological properties including binding sites and post-translational modifications.”\n\nHowever the structure of evidence is very phenomenological in nature — x (attention at head H, layer L) is observed which coincides with phenomena Y (binding/PTM/). \n\n\n2) no cross-model analysis. Why are we observing different outcomes in different models?\n\nSpecific comments:\n\nsection 4.1 \"Considering the model was trained on a masked language modeling task without any spatial information, the presence of these structurally-aware attention heads is intriguing”\n\t- why strongest alignment in only few heads and in high layers?\n\t- \"While there seems to be a strong correlation between the attention head output and classically-defined contacts, there are also differences”. What are the differences?\n\n\t- Attention as a well-Calibrated predictor section. Authors offer no explanation why the ECE is higher for none TapeBert. \n\t\t\nsection 4.2 *why* is the causal reason for attention focusing on binding sites or PTMs?\n\nsection 4.4: what about the other 4 amino acids that dont have 25% allocated attention in TapeBert? what is it about those amino acids?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YsoptjIClH_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this manuscript conducted a comprehensive analysis of the interpretability of self-attention language models when learning from protein sequences. Specifically, five multi-head self-attention models from NLP were used to model protein sequences. The following tasks were conducted: (1) whether attention captures secondary and tertiary structural information, (2) how and why attention targets binding sites and post-translational modifications, (3) how attention captures properties of varying complexity across different encoding layers, and (4) relationship between pairwise attention amino acids and substitution matrix. Through measures and visualizations, the authors can discover interesting relations between attention and various structural properties. In general, I think it is a good analysis paper. Even though there is no contributions in modelling, the authors can use suitable tools to probe and show the relationships between attention and protein structural informations. I believe the findings of this work is informative and inspiring to researchers in bioinformatics.\n\nThis paper can be enhanced if consistency and difference between these five attentions models can be further analyzed in the four tasks. What would be the most suitable model for protein sequences?\n\nMoreover, I would suggest the authors to discuss how many layers and heads are needed for protein sequence modelling. What will happen if 50 or just 5 layers are used for the analysis in this paper? Will the same conclusions be drawn? \n\nMinors: \n1. transformer -> Transformer\n2. The list of references can be further formatted. For example, \"Advances in neural information processing systems\" -> \"Advances in Neural Information Processing Systems\".\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "73GNQxHQnF0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe authors analyzed how attention and embeddings of Transformers trained on protein sequence correlate with protein properties such as pairwise contacts, binding sites, and post-translational modifications. The paper extends existing papers such as Rives 2020 ‘Biological structure and function emerge…’ by showing that layers learn more and more complex protein features with increasing layer depth and by proposing new visualization techniques. The paper is mostly clearly written while the methodological contributions are incremental. The evaluation needs to be strengthened.\n\n# Major comments\n1. The paper is missing two important baselines: 1) a Transformer with randomly initialized weights and 2) a Transformer trained on randomly shuffled protein sequences (with the same amino acid frequencies. These baselines will show if Transformers actually learn protein features, or if the correlation between network features and protein features is an artifact of the Transformer architecture. It has been shown that randomly initialized convolution networks already encode edge detectors in their weights without any training. Something similar can hold true for proteins.\n\n2. To further support that Transformer learns protein features, I would like to see how features specialize during training on protein features. For example, how do figure 2 and figure 6 look at different numbers of training epochs?\n\n3. The attention analysis (equation 1) depends on a threshold \\theta. How was it chosen and how sensitive are results to this threshold?\n\n4. Instead of using equation 1 for analyzing ‘alignment’ of attention $\\alpha_{i, j}$ with the property $f(i, j)$, I suggest computing the Spearman correlation between  $\\alpha_{i, j}$ and  $f(i, j)$ over all samples in the training set. Using the Spearman correlation is a more clear measure for quantifying the relationship between a feature and a property and does not depend on threshold.\n\n5. Using Expected Calibration Error (ECE) for analyzing the calibration of attention is a straight-forward extension of using ECE for analyzing calibration of a ML model, which is widely performed. I also doubt the usefulness of analyzing calibration of attention since practitioners are unlikely to use a particular attention map directly for contact prediction, but train a ML model on top of attention maps to predict contacts (see Rives 2020). What matters is if the resulting ML model is calibrated.\n\n6. For the binding site and post-translational modification (PTM) analysis (section 4.2) it is unclear how $f(i, j)$ (equation 1) is defined. A binding site and (PTM) is a property of a single position $i$ not a pair of positions $(i, j)$.\n\n7. Section 3, datasets: How similar are test sequences to sequences that were used for training Transformer models? Sequences must not overlap and must have a maximum similarity of, e.g., 80%.\n\n8. It is unclear which Transformer architecture was used for Figure 5 and Figure 6. For all analysis, please mention the Transformer architecture and refer to additional results for other architectures. Also discuss in each paragraph if the observation holds true for all architectures or not.\n\n9. Section 4.2, Figure 4: I agree that TapeBert attends most strongly to the last layers but I cannot see a clear pattern for the other architectures (App C3). \n\n# Minor comments\n10. Section 3, Probing task’: Please also cite Biswas 2020 (‘Low-N protein engineering …’).\n\n11. Section 4.1; Figure 2: ‘alignment’ can be misunderstood as ‘sequence alignment’ in the context of protein. Please use another term, e.g. ‘association’ or ‘correlation’.\n\n12. Section 4.1: What are ‘implicit features’?", "extra_scores": {}, "extra_text": {}}, {"review_id": "5DSlC9bQLGr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper offers an in-depth analysis of attention in large-scale language models including (AL)BERT and XLNet. Key \"headline\" findings:\n\n- Attention maps highlight amino acid pairs that are far in sequence space but near in coordinate space.\n- Attention maps highlight binding sites within proteins.\n- Attention maps capture local secondary structure.\n- Attention heads have affinities to specific residue types.\n\nMore nuanced findings:\n\n- The evolution across layers of attention-encoded information differs from embedding-encoded information: latter accumulates steadily while former can be isolated to single (end) layer.\n- Evidence indicates that attention heads' affinities to specific residue types rests on physical properties rather than mere identity.\n\n### Merits\n\nThis is an interesting study that is modest in its characterization of its contributions but diligent in its empirics. Too many papers in this space try to train giant models and cherry-pick tasks for which those models \"beat\" non-neural baselines, while contributing too little (in the reviewer's humble opinion) to our understanding of what's going on. The authors have shed light on the attention mechanism, showing the intuitive but notable results that attention correlates with coordinate-space distance, binding sites, secondary structure, and residue identity.\n\nI particularly liked the authors' experimental design in 4.4 to shed light on whether identity or structural/functional properties are driving the attention.\n\nThe paper is generally well-written and will be accessible to those without prior exposure to protein modeling. The authors' efforts toward accessibility are clear both in the length of background exposition and in the terminology: although I use the term \"residue\" in this review, I appreciate the authors' choice of \"amino acid\".\n\n### Drawbacks\n\n**Calibration**: I had to read the \"Attention is a well-calibrated predictor of contact maps in some cases\" section about 10 times over before I had a grasp on what was going on. Even with my current understanding of the section, I'm extremely skeptical about this specific study. Why do I expect attention (which produces a point on the simplex) to be at all calibrated to a contact map (which is an arbitrary pairwise zero-one matrix)? Suppose we had a very tightly clustered protein and a high-enough radius such that all residues were considered to be in contact. No possible attention map would be well-calibrated to the ground-truth contacts.\n\nI would recommend removal of this section. It is a weak bordering on nonsensical result which seems to have been plucked from a single attention head. I suspect this result is not robust and in any case it does not contribute much conceptual understanding.\n\n**Embeddings vs. attention**: In related work, the paper cites prior work that analyze the output embeddings of large scale models for structural and functional properties. The paper itself focuses on properties captured through attention. As a reader, I'm left somewhat unsatiated. What's captured in embeddings that attention represents less well? What's captured in attention that might not necessarily emerge through a metric space defined in embedding land? The paper missed an opportunity in my opinion to do a systematic comparison of information encoding in attention vs. embeddings, aside from what's shown in Figure 6.\n\n### Recommendation\n\nThis paper does not make, nor claim to make, novel methodological contributions in the modeling of proteins. What it does is to study Transformer attention using previously-known techniques but applied to protein sequence models. It overall performs an admirable job of doing so. We at ICLR care not only about \"learning representations\", but about assessing what precisely those representations have learned, so the paper is certainly appropriate for ICLR.\n\nMy main hesitation with recommending acceptance is the whole attention calibration business. The metric is mathematically unsound (contact maps certainly do not sum either row-wise or column-wise to one) and the fact that this paper found a \"well-calibrated\" attention head isn't all that notable.\n\nWith that caveat, I cautiously recommend acceptance. I will raise my score upon either removal of \"attention calibration\" from the paper or an *extremely* convincing explanation of why I've misinterpreted the metric / why I should care about the calibration of an attention map with an arbitrary unnormalized zero-one matrix.\n\nUPDATE: The authors have removed the attention calibration study. My new score is a 7. This is a good paper and I heartily recommend its acceptance.\n\n### Comments\n\n- p1: \"substitution properties\" is a handwavy term. Please make the meaning more precise.\n- p3: \"We note that all of the above analyses are purely associative and do not attempt to establish a causal link between attention and model behavior (Vig et al., 2020; Grimsley et al., 2020), nor to explain model predictions (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).\" very good thank you!\n- p4: very nice ALBERT visualization and consistent with the model architecture!\n- p6: \"deeper layers focus relatively more attention on binding sites and contacts\". OK, contacts sure, but binding sites?", "extra_scores": {}, "extra_text": {}}, {"review_id": "-iMrH3KnAfm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is paper is an investigation of the potential biological interpretation of attention in five pretrained protein sequence Transformer models (one from TAPE and four from ProtTrans). It seems that the meatiest contribution in this paper involves the use of the attention analysis metric p_alpha(f) to measure the co-occurrence of attention values (above threshold alpha) and a contact matrix f. A related contribution is the definition of attention calibration, which is a slight variant of Expected Calibration Error applied to evaluating the calibration between attention values and a discretized contact matrix. \n\nMy biggest concern with this paper is that they analyze Transformer models by looking for a single attention head that maximizes p_alpha(f) without, as far as I can tell, really evaluating how often these multiple comparisons might lead to a high p_alpha(f) value by chance. Several compelling examples are shown for individual attention heads (e.g. repeated evaluation of TapeBert 12-4). These individual heads really could just be the locus in which a pretrained model encodes understanding of structural adjacency but...there has to be a null model when trying every node of a big neural network and taking the max value of an evaluation metric. The paper repeatedly plots the background frequency of contacts between residues in the protein data but that's not the same as determining how often your analysis would yield high values just by chance. \n\nSimilarly, what's the null model for the calibration analysis? The plot, like others, has an orange line showing the background frequency of contacts but that seems really unrelated to calibration. I don't know what the null model *should* be but this paper should endeavor to show that finding one calibrated attention head (among the many possibilities) is surprising. \n\nOther comments:\n\n* In the first background items, on \"Amino acids\", the description derails into discussion of substitution matrices. If understanding substitution matrices is important to the paper, give them their own bullet point. Otherwise it's a strangely specific technique to discuss while describing amino acids in general. \n\n* \"every protein sequence is formed from a vocabulary of 20 standard amino acids\" it's strange to say this and then talk about PTMs a few paragraphs down. If you're presenting PTMs to the networks you're evaluating then you're likely using a larger amino acid alphabet (I think TapeBert has 25 amino acids?)\n\n* This description of secondary structure seems odd: \"Secondary structure describes the local segments of proteins, and may be divided into three broad categories: Helix, Strand, and Turn/Fold, as well as an Other category for local structure that falls outside these categories.\" If you asked most biologists what the broad categories of secondary structure are would probably say \"[alpha] Helix\" \"[beta] sheet\", it didn't even occur to me to break sheets into strands and turns. I guess it makes sense that these categories might occur in some secondary structure prediction datasets or tasks, but it seems like the details of some dataset are being presented as representative of biology. \n\n* Slightly confusing terminology here: \"Binding sites are protein regions that bind with other molecules to carry out a specific function. For example, the HIV-1 protease is an enzyme responsible for a critical process in replication of HIV (Brik & Wong, 2003). It has a binding site, shown in Figure 1b, that is a target for drug development to ensure inhibition.\" I think it's important to distinguish the active sites of a protein where that protein binds some natural target or ligand and the binding site of a drug which inhibits the protein. These often overlap to some degree but the binding site of a small molecule inhibitor can be a much smaller pocket than a natural ligand. \n\n* \"Attention targets PTMs in a small number of heads\": I wish it was clearer what this analysis meant. Were the input sequences given as unmodified AAs and then you're evaluating which residues are often observed as modified in vivo? Or did you give the networks sequences with PTMs in them and then this attention pays attention to when rare AAs get used?\n\nOK, after this list of complaints, I want to come back to what I like about this paper. I think many people (myself included) suspect that attention in protein Transformer language models operates in biologically interpretable ways. It's really nice to actually show it and also show that attention is compatible with other simpler biological notions (like AA substitution matrices). But: it's not enough to just look for the best node in each network, you have to make some kind of effort to quantify how like that finding would be by chance in a large enough network. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers an in-depth analysis of attention in large-scale language models including (AL)BERT and XLNet in the context protein representation learning, and obtains many interesting findings. This is not a typical paper with novel technologies proposed, instead, it studies the existing technologies in a specific (biology) context and explains what the learned representations and attention map really mean. 
All the reviewers see the value in this paper and give positive feedback in general. At the same time, they also raised a few concerns, e.g., regarding the claim on “well-calibrated" attention head, on some missing details of the algorithm description and the experiments,  on phenomenon vs. causality of the finding, etc. The authors really did a very good job in their rebuttal and paper revision, and most of these concerns were (at least partially) addressed, and a few reviewers raised their scores. With this, we are quite confident that this paper is above the bar of ICLR. 

Decision: Accept (Poster)