Convolutions are competitive with transformers for protein sequence pretraining
Authors: Kevin K Yang
Alex Xijie Lu
Nicolo Fusi
Venue: Submitted to ICLR 2023 (2023)
Keywords: protein
pretrain
convolution

Abstract: Pretrained protein sequence language models largely rely on the transformer architecture. However, transformer run-time and memory requirements scale quadratically with sequence length. We investigate the potential of a convolution-based architecture for protein sequence masked language model pretraining and subsequent finetuning. CNNs are competitive on the pretraining task with transformers across several orders of magnitude in parameter size while scaling linearly with sequence length. More importantly, CNNs are competitive with and occasionally superior to transformers across an extensive set of downstream evaluations, including structure prediction, zero-shot mutation effect prediction, and out-of-domain generalization. We also demonstrate strong performance on sequences longer than the positional embeddings allowed in the current state-of-the-art transformer protein masked language models. Finally, we close with a call to disentangle the effects of pretraining task and model architecture when studying pretrained protein sequence models. 

Under review as a conference paper at ICLR 2023
CONVOLUTIONS ARE COMPETITIVE WITH TRANSFORM- ERS FOR PROTEIN SEQUENCE PRETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained protein sequence language models largely rely on the transformer archi- tecture. However, transformer run-time and memory requirements scale quadrat- ically with sequence length. We investigate the potential of CNN-based archi- tecture for protein sequence masked language model pretraining and subsequent finetuning. CNNs are competitive on the pretraining task with transformers across several orders of magnitude in parameter size while scaling linearly with sequence length. More importantly, CNNs are competitive with and occasionally superior to transformers across an extensive set of downstream evaluations, including structure prediction, zero-shot mutation effect prediction, and out-of-domain generalization. We also demonstrate strong performance embeddings allowed in the current state-of-the tftth-t transformer protein masked lan- guage models. Finally, close with call disentangle the effects of pretraining task and model architecture when studying pretrained protein sequence models.
sequences longer than the positional
INTRODUCTION
Large pretrained protein language models, largely relying on the attention based transformer (Vaswani al., 2017) architecture, have advanced the ability of machine-learning methods to predict pro- tein structure and function from sequence, especially when labeled training data is sparse. Most modern self-supervised protein sequence pretraining combines a transformer model with either an 202T,
autoregressive likelihood (Madani et al.] 2020 2021 Ferruz et al. 2022 Hesslow et al. 2022) or
with the masked language modeling (MI.)) task introduced for natural language by BERT (bidi- rectional encoder representations from transformers) (Devlin et al.) 2018). Pretrained transformer protein MLMs contain structural information (Rao et al.
2019 Rives et al. 2021 Chowdhury et al..
1021 are zero-shot predictors of mutation 2021), encode evolutionary trajectories (Hie et al. 2022a
fitness effects (Meier et al., 2021). improve of generalization on protein engineering datasets (Dallago et al. 2021) ) and suggest improved sequences for engineering (Hie et al., 2022b).
Protein MLMs are now incorporated into the latest machine-learning methods for detecting signal
peptides (Tefeel et al. 2021) and predicting intracellular localization Tataaatiiiinnnuuuurrrrr et al. 2022).
One drawback of transformers is that the compute and memory required by the attention layers scale quadratically with input sequence length. In addition, transformer attention is invariant to position, so transformer sequence models include a positional embedding. Depending on the formulation, these embeddings can be difficult to extend past the maximum length seen during training. As a result, some popular pretrained protein transformer models limit the input length during pretraining and inference; for example, ESM has maximum input length of 1022 residues. Of the 42 million cluster representatives in the March 2020 release of UniRef50 (Suzek et al., 2015), 1.1 million, or 2.6%, are longer than 1022 residues. This includes many proteins of interest, such as the SARS-Cov-2 spike glycoprotein and Streptococcus pyogenes CRISPR-associaal endonuclease Cas9.
Furthermore, there has been little investigation of how model architecture interacts with pretraining tasks protein sequences. Transformers can perform the masked language model task on protein sequences, and pretraining improves the performance of transformers on downstream protein structure and property prediction tasks. However, it is important to disentangle pretraining from architectural advances and consider them independently. We seek to do this by investigating the effectiveness of pretrained and naive convolutions for proteins.

---

Under review as a conference paper at ICLR 2023
We train protein sequence convolutional masked language models UniRef50, which refer to as CARP (Convolutional Autoencoding Representations of Proteins). Our CARP models are competitive with transformers the pretraining task, given comparable parameter sizes. The largest CARP, with approximately 640M learnable parameters (CARP-640M) is competitive with the current state-of-the-art transformer protein sequence masked language model, ESM (Rives al. 202T] Meier et al. 2021) on a variety of downstream prediction tasks, including structure prediction, zero- shot mutation effect prediction, and out-of-domain generalization on biologically-relevnn protein engineering datasets. Because CARP scales linearly in computation with the input sequence and does not rely on an imput positional embedding,
straightforward to apply to sequences longer than the longest sequences in training, which we demonstrate with zero-shot predictions of mutation effects in CRISPR-Cast These empirical results demonstrate need deepen our understanding of protein sequence pretraining by disentangling the effects of architecture and the pretraining task. Finally, while performance on structure prediction tasks improves as model size and pretraining performance improve, this is not the case for all fitness prediction tasks, demonstrating also need to deepen our understanding of how pretraining relates to downstream tasks.
output probabiltiess
softmax
p xxy
convolution
convolution
GeLU
layer
layer norm
dilated conv.
ByteNet blocks
GeLU
layer norm
A or ag)
convolution
up-embedding
GeLU
layer norm ()
down-embedding
inpuls
(a) CARP
(b) ByteNet block
Figure 1: The CARP architecture.
2 CONVOLUTIONAL PROTEIN SEQUENCE MASK LANGUAGE MODELS
We pretrain CARP using the masked language model (MLM) objective described in Rives et al. (202T). Each sequence is corrupted by changing some tokens special mask token another amino acid token, and the model is tasked with reconstructing the original sequence. Specifically, 15% of tokens from each sequence are randomly selected for supervision. For those 15% of tokens, 80% are replaced by the mask token, 10% are replaced by a randomly-chosen amino acid, and 10% remain unchanged. The model is trained minimize the cross-entropy loss between its predictions for the selected tokens and the true tokens at those locations. We train on the cluster representatives from the March 2020 release UniRef50, with approximately 83k sequences held for validation and another 210k sequences held out for testing, leaving 41.5 million sequences for training.
CARP combines the ByteNet encoder dilated CNN architecture from Kalchbrenner al. (2016) with simple input embedding and output decoding layers, shown in Figure 1a CARP begins with an embedding layer, which maps an input sequence of L tokens z E D+ to an 8-dimensional intermediate embedding, followed by linear mapping into the model dimension d: E 4xd This passes through a stack of n ByteNet dilated CNN blocks Figure with residual connections in between followed by final layer norm to produce the encoder representation en E R //xd and finally a linear decoder maps this to the x logits, where t is the number possible tokens. The 1 x convolution layer in every ByteNet block dilated and padded preserve sequence length. Dilation
2

---

Under review as a conference paper at ICLR 2023
increases the CNN perceptive field exponentially with the number of layers in order obtain global context for long input sequences. The CNN dilation rate doubles every layer up to a maximum rate r (for our experiments from a dilation rate of 1. While convolution kernels are homogenous across locations, the ByteNet architecture preserves the order of those locations. We found that adding positional embeddings does not improve pretraining performance.
128). The scheme is repeated multiple times in the network, always starting
We varied the number of parameters in CARP from approximately 3000 640 million by setting the model dimension d, setting the encoder hidden dimension he to either d or 2. and setting the number of layers. All models are trained with the Adam optimizer, a maximum learning rate of 0.001, a linear warmup for 16,000 steps, and dynamic batching to maximize GPU usage. The largest model, CARP-640M, was trained on 128 32GB Nvidia V100 GPUs for 620,000 updates, or approximately 56 days.
3 RELATED WORK
CNN language models CARP's architecture is based on ByteNet (Kalchbrenner et al. (0166), which introduced a dilated convolutional seq2seq framework for neural machine translation. Work in natural language processing (Kalchbrenner et al. 2016 Wu al. 2019 Tay al. (02T) hints that pretrained attention-fie free convolutional neural networks (CNNs) can be competitive with pretrained transformers while scaling linearly with sequence length, CARP directly applies this work to protein sequence pretraining.
Protein sequence pretraining ESM-1b (Rives et al. 2021) is a 650-millionn-parametr transformer protein masked language model trained on the March 2018 release of UniRef50 (Suzek et al., (007))
ESM-1v (Meier et al.] 202T) uses the same transformer architecture, but is optimized for mutation- effect prediction by training UniRef90 instead of UniRef50. TAPE (Rao et al. 2019) is smaller transformer protein masked language model trained on protein domains from Pfam (Mistry et al.) 2021) instead of the full protein sequences found in UniRef. ProtTraas (Elnaggar al. 2021) explores the use of different transformer architectures language modeling tasks and larger datasets. and ProtBERT-BFD which
The most comparable ProtTrans models are ProUBERT-nniRfTie transformers protein masked language models trained on UniRef100 and BFD Steiinegger and Söding 2018 Steiineger et al. 2019), respectively. ProteinBERT (Brandes et al.) 2021) introduces a global attention mechanism and an additional functional annotation prediction task during pretraining. |Rao al. (202T) extends the transformer masked language model scheme to multiple sequence alignments.
220-milloon-parmeee
Convolutional models protein sequences Shin et al. 2022 train autoregressive convolutional models on protein families, but do not attempt to train a single model over the breadth of known protein sequence diversity. [u et al. (2020) use a small convolutional encoder for a noise-contrastive pretraining task on proteins, but do not give it global context make the model autoencoding. Bileschi et al. (2022) use a similar convolutional architecture to our model to learn functional annotations for unaligned protein sequences. However, their task is not autoencoding, and they do not consider performance on downstream tasks.
By combining denoising autoencoding task with dilated CNN architecture, we begin to disentangle the effect of pretraining task from the effect of model architecture.
PRETRAINING PERFORMANCE
Our largest model, CARP-640M, has test loss of 2.02, comparable to ESM-1b, which has 650 million parameters and a loss of 1.96 on its test set. Note that ESM-1b was trained and tested on an earlier version of UniRef50 with different train/test splits than CARP our ESM models. (Throughout, ESM-1b refers specifically to the 650-million parameter transformer trained on the March 2018 UniRef50 release and deseribed in Rives al. (2021). while ESM refers to small transformer masked language models based off of the ESM- architecture. Likewise, CARP refers to any ByteNet masked language model, while CARP-X refers to the model with approximately X
3

---

Under review as a conference paper at ICLR 2023
parameters.) Hyperparameters for lefferennt-szedd versions of CARP and ESM are found in Tables A1 and A2 respectively.
For comparison, also trained transformer models with comparable numbers of parameters using the ES architecture described in Rives ct al. (202)) on our UniRef50 dataset. As shown in Figure 2a CARP's performance the pretraining task is comparable to ESM's across several orders of magnitude variation in the number of parameters when using the same pretraining dataset. Figure shows MLM loss by length for CARP-640M and ESM-1b on their respective test sets, smoothed with a window of 30 in the length dimension. For both models, the pretraining loss improves quickly until the sequence length reaches about 500, and then slowly thereafter. The maximum input length for ESM-1b is 1022. but we calculate losses for CARP-640M for sequences with to 4096 residues. These results show that convolutions can perform protein sequence masked language modeling comparably to tranformers without suffering from quadratic dependence between runtime and sequence length.
2.8
CARP-640M ESM-1b
2.6
2.4
2.4
2.2
architecture
2.0
2.2
CARP
ESM
1.8
2.0
ESM-1b
10
15
20
1000
2000 length
3000
4000
log parameters
MLM loss size
(b) loss length (smoothed)
Figure 2: Comparisons between CARP and the ESM-1b transformer.
DOWNSTREAM TASKS
One goal of protein ML.Ss is encode information in their output representation model weights that improves performance on downstream prediction tasks. Downstream evaluation can be zero-shot (without access labels for further training), the pretrained model can be frozen and a small neural network decoder can be trained to predict labels from the pretrained model's 's output representations (pt-fr), or the new decoder and pretrained model can be finetuned together (pt-ft). We use the output from the final layer norm in Figure Talas the output representation. Unless otherwise noted, the new decoder consists neural network with hidden size d. For tasks with labels, we evaluate both pt-fr and pt-ft and compare ESM- lb or ESM-Iv. We finetune models with a maximum learning rate of 0.0001, a linear warmup over 1000 steps, and early stopping based the validation set. Finetuning was performed 32 GB V V100; depending on the task, finetuning took between several minutes to 48 hours. Where relevant, we also compare the CARP architecture with randomly-initialin weights (na-fr and na-ft), linear ridge regression, and the small CNN described in Dallago et al. (2021) and [Shanehsazzadeh et al. 2020).
learned attention that converts the output from L d to d followed by a 2-layer
one
5.1 PROTEIN STRUCTURE
One of the most striking successes of protein MLMs is their ability to encode structural information without access to structural labels during pretraining. We evaluate CARP-640M's ability to encode structural information through 3 tasks:
1. Remote contact prediction asks model to predict whether the C3 atoms of two residues separated by at least 24 residues in the primary structure are within 8 Angstroms other in the three-dimensional structure. We train the trRosetta YYng al. 2020) training set
4

---

Under review as a conference paper at ICLR 2023
and evaluate the precision of the top L predictions the CAMEO hard (Haas al. 2018 and CASP13-FM (Shrestha et al. 2019) test sets. For contact prediction, we downsample CARP embeddings to 128 dimensions, perform outer product to produce 2-dimensional embeddings, and then pass that to a 24-layer dilated residual CNN based on the trRosetta architecture. This is the same as the procedure used by ESM-1b.
2. Remote homology detection asks model detect structural similarity across distantly- related sequences. We evaluate accuracy on the fold-level holdout set from TAPE.
3. 3-class secondary structure prediction asks model to predict whether each residue in a protein is part of a helix, strand, or other. We use the training and validation sets from TAPE and evaluate accuracy the CB513 test set. For this task, train neural network consisting of two CNN layers, an LSTM, and a linear head on top of the pretrained model, described in Rives al. (2021).
As shown in Table I pretraining improves performance for structure prediction tasks, and CARP- 640M is competitive with ESM-1b. These results show that pretrained convolutions learn structural information from single sequences, just pretrained transformers do.
Table Structure prediction tasks. Values for ESM-1b are taken from Rives et al. (0021) Uncertain- ties are standard deviations on 3 replicates with different weight initializations.
Task
Method
Model
CASP-13 FM
CAMEO
remote homology
secondary structure
pt-fr
CARP-640M
23.7
42.0
0.24:10.008
0.83
BSM-1b
28.2
44.4
0.82
pt-ft
CARP-640M
0.28=#0.008
0.83:10.001
ESM-1b
0.33
na-fr
CARP-640M
9.7
12.6
0.09
0.65#00022
na-ft
CARP-640M
0.09 + 0.02
0.71+0.0005
5.2 ZERO- SHOT MUTATION EFFECT PREDICTION
Large language models predict experimental measurements of protein function without further training on sequence- fetteence-ftness measurements or sets of evolutionarily-relate sequences Hie et al. (2022a): [Meier al. (202T). Following Meier et al. (2021), we CARP-640M on 41 deep mutational scanning datasets originally compiled by Riesselman et al. (2118). These datasets measure the effects of thousands of mutations combinations of mutations to parent sequence. Details
described in Section
Figure compares zero-shot performance for CARP-640M. ESM-Ib. Ib, ESM-Iv, position-specific scoring matrices (PSSM), and ProtBert-BFD, ESM-IV v results are for an ensemble of five transformers. Averaged across the 41 datasets, CARP-640M has Spearman correlation of 0.49, compared to 0.46 for ESM-1b, b,, 0.51 for ESM-1v, 0.46 for PSSM, and 0.43 for ProtBERT-BFD. CARP-6400 outperforms ESM- lb on 22 of 41 datasets, ESM-Iv 18 out of 41 datasets, PSSM on 26 out of 41 datasets, and ProdBERT-BFD on 25 out of 41 datasets.
Meier (202 found that using the full UniProt sequences instead of only the sequence of the mutated domain results in better zero-shot predictions. However, this is not always possible with ESM- 1x, as some UniProt sequences for these proteins are longer than 1022 residues. As further proof of concept, made zero-shot predictions for the effects of mutations in Cas9 from Streptococcus pyogenes (Spencer and Zhang 2017). which is 1368 residues long, and obtain a Spearman correlation of 0.26. These results show that pretrained convolutions make zero-shot predictions protein mutation effects on fitness, including on sequences longer than allowed by ESM-Ix.
5

---

Under review as a conference paper at ICLR 2023
5.3 OUT-OF-DOMAIN FITNESS PREDICTION
Another motivation for pretrained protein sequence models is that they may be able to generalize after fine-tuning in ways that helpful for protein engineering. For example, a protein engineer may want to train a model on single mutants and make predictions for sequences with multiple mutations, or train a model that is accurate for sequences with fitmess greater than what is seen the training set. Here, we evaluate CARP-640M on tasks from the following landscapes from FLIP (Dallago et al. 2021
1. AAV (Table 2):
Adeno-associated virus (AAV) capsid proteins are responsible for helping the virus integrate a DNA payload into target cell (Vandenberghe et al. 2009). and there is great interest engineering versions of these proteins for gene therapy (Büning et al. 2015) Barnes et al., 2019). Bryant et al. (2021) measure a rich mutational screening landscape of different VP-1 AAV proteins.
2. GBI 3 GB1 is the binding domain of protein G. an immunoglobulin binding protein found in Streptococcal bacteria. In their original study, Wu et al. (2016) measured the fitness of 149,361 out of 160, 000 possible combinations of mutations at 4 positions.
For each landscape, evaluate several tasks, including:
x-vs-many: Train sequences with up to mutations and test on the remainder of the landscape.
mut-des: Train on sequences sampled from mutagenesis libraries and test on sequences designed by machine-learning models (AAV only).
low-vs-high Train on sequences with fitnesses below the wild-type and test on sequences with fitnesses above the wild-type.
We compare results to ESM-1b and the same decoder neural network, linear ridge regression, and small CNN.
Table 2: Performance on the FLIP AAV tasks. Values for models other than CARP-640M are taken from Dallago al. (20TT). Uncertainties for ESM-lb and CNN are standard deviations over 10 random seeds. Uncertainties for CARP-640M are standard deviations over 3 random seeds. Dallago al. (2021) do not provide uncertainties for the mut-des task because of the computational cost.
Task
Method
Model
I-vs-many
2-vs-many
7-vs-many
mut-des
low-vs-high
pl-fr
CARP-640M
0.31
0.51 18
0.58
0.75+10.08
0,25
ESM-1b
0.03+0.11
0.6110.04
0.65+10.01
0.76
0.38
pt-ft
CARP-640M
0.73+0.05
0.81+0.03
0.77+00.03
0.85+0.003
0.19=00.08
na-fr
CARP-640M
0.48=0.07
0.50+00.05
0.60+00005
0.76+00002
0.21
ESM-1b
0.1810.01
0.20+:0.33
0.38
0.56
0.06+0001
na-fr
CARP-640M
0.04
0.50+0043
0.38+0.37
0.84+0.01
0.24
ridge
0.22
0.03
0.65
0.68
0.12
baseline
CNN
0.35=+0.11
0.58+
0.73=0.004
0.71
0.28+0002
In general, pretraining improves CARP-640M's performance on these tasks, and fine-tuning the entire model outperforms freezing the pretrained weights. Comparisons to the baselines show that pretraining most helpful when generalizing from single mutants to multiple. When not fine-tuning all the way through, there is little benefit from pretraining, and on some tasks pretraining hurts performance. CARP-640M outperforms ESM-Ib on generalizing from few mutations to more, but ESM- is better at generalizing from low-fitness training higher-fitness sequences. These results show that pretrained convolutions help generalization to types of sequence variation not seen during training. On 1B1, finetuning ESM-1b end-to-end instead of freezing the pretrained weights hurts its performance, while CARP-640M benefits from full finetuning. We not finetune ESM- -b end-to-end other tasks because of the computational cost. In addition, CARP-640M provides

---

Under review as a conference paper at ICLR 2023
better representations without pretraining than ESM-1b all the AAV tasks and of the 4 GB1 tasks, showing that the architecture alone also influences generalization.
Table 3: Performance (Spearman correlation) the FLIP GB1 tasks. Values for models other than CARP-640M and ESM-1b with full finetuning are taken from FLIP. Uncertainties for ESM-1b frozen and CNN are standard deviations over 10 random seeds. Uncertainties for CARP-640M and ESM- -b with full finetuning are standard deviations over random seeds.
Task
Method
Model
1-vs-many
2-v8 2-vs-many
3-vs-many
low-vs-high
pt-fr
CARP-640M
0.15+0.11
0.18+0023
0.62++0.06
0.12++0.03
ESM-1b
0,29++0.02
0.47
0.79++0.01
0,53++0.03
pt-ft
CARP-640M
0.1910.26
0.73
0.87:10.004
0.43
ESM-1b
0.11+0.11
0.67
0.66:10.18
0.42
na-fr
CARP-640M
0.03:10.03
0.07
0.71
0.35
ESM-Ib
0.12:10.11
0.21
0.52+10.01
0.32
na-ft
CARP-640M
0.11#0.07
0.38
0.68+0.33
0.23
ESM-Ib
0.05
0.14
0.10#0.13
-0.04
ridge
0.28
0.59
0.76
0.34
baseline
CNN
0.15++0.09
0.39++0.04
0.81++0.004
0.47++0.01
5.4
IN-DOMAIN PROPERTY PREDICTION
Finally, we consider fitness-pres preess-predittio tasks that do not require difficult biological generalization (Table 4h. We evaluate three sequence-fitness regression tasks:
Fluorescence requires the model to predict the effect of one or more mutations on the brightness of green fluorescent protein. The data was originally collected by |Sarrisyann et al. (2016). We use the data splits provided in TAPE.
2. Stability requires the model predict small protein's 's resistance protease degradation. The data was originally collected by Rocklin et al. (2017). We use the data splits provided in TAPE.
3. Meltome-mise requires the model to predict the melting temperature of variety of proteins from across the domains of life. The data was originally collected by Darzab et al. 2020). We the cluster representatives and data splits provided in FLIP.
in addition to two intriniialll-ddsordeeee region (IDR) function classification tasks taken from Zarin MMseqs2 (Steinegger and Söding 2017) cluster
et al. 1202I) For the IDR datasets, sequences to 50% identity and then randomly assign clusters to training, validation, or testing.
1. Cdc28 binding requires the model to predict whether IDR is target of Cdc28.
2. Mitochondria targeting requires the model to predict whether an IDR targets its protein for transport into the mitochondria.
In general, while pretraining improves HARP-640M:s performance on these tasks, neither of the large pretrained models consistently out-perform the baseline models on these tasks. Almost all the models perform very well on the IDR tasks, indicating that performance is saturating on these tasks. Nevertheless, CARP-640M generally comparable to 1b, showing that once again pretrained convolutions comparable pretrained attention.
5.5 EFFECT OF MODEL SIZE AND PRETRAINING PERFORMANCE ON DOWNSTREAM PERFORMANCE
To investigate the effects of model size and pretraining performance downstream performance, finetune pretraining checkpoints for CARP-600k, CARP-76M, and CARP-6 640M on secondary struc- ture, remote homology, and the FL.IP GB1, AAV, and meltome tasks. In each of these experiments,
7

---

Under review as a conference paper at ICLR 2023
Table Performance on in-domain tasks, For fluorescence, stability, and meltome, values reported are Spearman correlation. For the IDR tasks, values reported are area under the reciever operating curve. Values for ESM-1b on fluorescence and stability are taken from Rives al. (202T). Values for baselines fluorescence and stability are taken from FLIP. Uncertainties for ESM- and CNN are standard deviations over 10 random seeds. Uncertainties for CARP-640M are standard deviations over 3 random seeds. We do not calculate uncertainties on meltome due to the computational cost.
Task
Method
Model
fluorescence
stability
meltome
Cdc28
mito.
pt-fr
CARP-640M
0.58=0.02
0.62:10.03
0.54
0.84
0.86+10002
ESM-1b
0.67 +0.01
0.91++0.004
0.90++0.01
pt-ft
CARP-640M
0.68+00.002
0.72++0.01
0.53
0.88+0.02
0.89+-0.004
ESM-1b
0.68
0.71
0.89+0.01
0.88++0001
na-fr
CARP-640M
0.62++0.01
0.52
0.29
0.84
0.86++0.01
ESM-1b
0.45+10.03
0.88
0,84
na-ft
CARP-640M
0.58
0.65
0.30
0.79
0.87+00001
ESM-1b
0.83=0.02
0.85+0,02
ridge
0.68
0.48
0.17
0.52
0.53
CNN
0.67
0.51
0.34++0.01
0.84
0.87+00.22
we initialize the prediction head with the same weights across all model checkpoints of the same size. Figures 3a and AAaa show that structure prediction improves smoothly the model size increases and the model is pretrained longer. This confirms that, as for transformers, pretraining imparts structural information CNNs. However, Figures 3b A3b and A3c shows that this relationship does exist for the out-of-domant FLIP tasks. In many cases, a small amount of pretraining is sufficient to outperform the naive baseline, and further pretraining has an unpredictable and often negative effect on performance. Using the FL.IP meltome task as example of an in-domain class, Figure A3d shows that performance generally improves as CARP is pretrained, but the pretraining effect saturates, and CARP-76M outperforms CARP-640M.
Figure shows that the average zero-shot performance improves with both model size and pretraining performance. However, this is not the case for every individual dataset within DeepSequence. 4b) a case where zero-shot performance peaks and then declines as CARP pretrained. The Spearman correlation between the pretrain loss and zero-shot Spearman correlation range from (monotonic increase in zero-shot performance with more pretraining) and -0.9, as shown in Figure A2 The average the DeepSequence datasets is 0.40 for CARP-640M, 0.48 for CARP-76M. and 0.23 for CARP-600K. Although CARP-640M has better overall zero-shot performance than CARP-76M, CARP-76M more consistently improves with more pretraining than CARP-640M. The heterogeneity in the relationship between pretraining performance and zero-shot performance suggests that many but all zero-shot tasks in DeepSequence are strongly determined by structural stability.
0.80
0.78
0.76
parameters
0.74
640M
76M
two-vs-many high
0.72
600k
2.5
2.4
2.3
2.2
2.5
2.4
2.3
2.2
2.0
pretrain loss
pretrain loss
(a) Secondary structure
(b) GB1
Figure 3: Effect of model size and checkpoint pretrain loss downstream performance.
8

---

Under review as a conference paper at ICLR 2023
0.2
0.2
0.0
0.1
/
-0.2
2.5
2.4
2.3
2.2
2.1
2.0
2.5
2.4
2.3
2.2
2.1
2.0
pretrain loss
pretrain loss
(a) DeepSequence-ment
(b) DMS YAPI human
Figure 4: Effect of model size and checkpoint pretrain loss zero-shot performance.
CONCLUSIONS
We have shown that convolutions can be comparable to superior to transformers on both the MLM pretraining task and variety of downstream protein sequence modeling tasks, and that convolutions, like transformers, benefit from pretraining. Furthermore, without pretraining, convolutions and transformers perform differently on downstream tasks, showing the important of disentangling pretraining and architecture. Unlike transformers, convolutions scale linearly with input sequence length, which becomes important when modeling long protein sequences. Work in natural language processing has also shown that convolutions can require fewer FLOPs of compute than transformers, for short sequences (Tay et al. 202T). In addition, while use standard dilated convolutions, there are more efficient convolution variants designed for sequence modeling (Wu et al. 2019) that may further improve model speed.
Limitations However, convolutions may not be competitive with transformers on tasks where a cross- self-attention inductive bias is explicitly needed or desired for interpretability. For example, is
possible to extract structural contact maps from pretrained transformer self-attention matrices (Rao
al. 2020), and self-attention matrices contain information about binding sites (Vig et al.] (2220)
convolutions lack an obvious equivalent. In addition, it is more natural to extend attention based interaction sites. The transformer's quadratic dependence on
models to predict protein-proted
sequence length can also be ameliorated with approximate attention methods (Child et al. 2019
Beltagy et al., 2020 Kitaev et al. 2020 TTa et al., [2020a] Wang al., [2200] Zaheer et al. 2020,
Katharopoulos et al. 2020; Choromanski et al.) (020a), but the choice of approximation matters for performance and the best method is not always clear a priori (Tay et al., (020b). On proteins, Choromanski al. (2020a) and Choromanski et al. (2020b) show that Performer approximate attention can perform well for autoregressive and masked protein language models, respectively, while BroTeinBERT combines fast global attention mechanism with masked language and functional annotation prediction pretraining (Brandes et al., 2021).
Outlook Currently, pretrained protein language models tightly-coupp to the transformer ar- chitecture, and the effects of the pretraining task can be conflated with the effects of the pretrained architecture. Our pretrained convolutional models may provide complementary inductive biases those found in pretrained transformer models, making them useful alternatives for practitioners. Unfortunately, we also find that, while masked language model pretraining is very effective for imparting models with structural knowledge, the relationship between model size, pretrain loss, and downstream performance more fraught for out-on-domain protein engineering tasks, indicating the need for more effective pretraining tasks. We hope that this work is the first step in investigating the independent and interaction effects of pretraining and architecture for protein sequence modeling, While we evaluate the effects of masked language model pretraining, transformers have also been used for autoregressive language model pretraining (Madani et al., 2020) and pairwise masked language modeling (He et al.] 2021). and combining structural information Mannsorr et al.) 2021 Zhang et al.) 2022], McPartlon et al. 20.2 Hsu et al., 2022; Chen et al.) 2022 Wang et al. 2022) or functional annotations (Brandes al. 2021) offers further directions for protein pretraining tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang. and Richard Socher. ProGen: Language modeling for protein generation. arXiv, 2020.
Ali Madani, Ben Krause, Eric Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, 2021.
Noclia Ferruz, Steffen Schmidt, and Birte Höcker. A deep unsupervised language model for protein design. bioRxiv, 2022.
Daniel Hesslow, N. ed. Zanichelli, Pascal Notin, lacopo Poli, and Debora S. Marks. RITA: a study on scaling up generative protein sequence models. 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. arXiv, 2018.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with TAPE. In Advances in Neural Information Processing Systems, pages 9686-9698, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proc. Natl. Acad. Sci. U. S. A., 118(15), April 2021.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Charlotte Rochereau, George M. Church, Peter K. Sorger, and Mohammed AlQuraishi. Single-sequence protein structure prediction using language models from deep learning. bioRxiv, 2021.
Brian L. Hie, Kevin K. Yang, and Peter S. Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 2 2022a. ISSN 24054712. doi: wetp://..clss.222......
Brian Hie, Ellen D Zhong, Bonnic Berger, and Bryan Bryson. Learning the language of viral evolution and escape. Science, 371(6526) January 2021.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, K. Nguyen, P.S. Liang, J.W. Vaughan, and Y. Dauphin, editors, Advances in Neural Information Processing Systems 34, 2021.
Christian Dallago, Jody Mou, Kadina Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
Brian L Hie, Duo Xu, Varun R Shanker, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, and Peter S Kim. Efficient evolution of human antibodies from general protein language models and sequence information alone. bioRxiv, 2022b.
Felix Teufel, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Magnús Halldór Gíslason, Silas Irby Pihl, Konstantinos D) Tsirigos, Ole Winther, Søren Brunak, Gunnar Heijne, and Henrik Nielsen. SignalP 6,0 achieves signal peptide prediction across all types using protein language models. bioRxiv, 2021.
Vineet Thumuluri, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Henrik Nielsen, and Ole Winther. DeepLoe 2.0: multi-label subcellular localization prediction using protein language models. Nucleic Acids Research, 2022.
10

---

Under review as a conference paper at ICLR 2023
Baris E Suzek, Yuqi Wang. Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformaties, 31(6):926-955 2015.
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv preprini 1610. 2016.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with 2019.
lightweight and dynamic convolutions. arXiv preprint
Yi Tay, Mostafa Dehghami, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint arXiv:2105.03322, 2021.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. UniRef: comprehensive and non-redundant UniProt reference clusters. Bioinformatics, 23(10):1222-1223
2007.
Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik LL Sonnhammer, Silvio CE Tosatto, Lisanna Paladin, Shriya Raj. Lorna J Richardson, et al. Pfam: The protein familics database in 2021. Nucleic acids research, 49(D1):D422DDA
2021.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards cracking the language of life's code through Self-Supervised learning. May 2021.
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 9(1):1-8, 2018.
Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nature methods, 16(7):603-66) 606, 2019.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: A universal deep-learning model of protein sequence and function. bioRxiv, 2021.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. In Marina Meila and Tong Zhang, editors, Proceedingy of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8844-8856. PMLR, 18-24 Jul 2021. URL https: httss::///ccceedngggg mlr. www.ppressvvee. sttess/v139/res www.1///rraaaa.... html
Jung-Eun Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-Supervised contrastive learning of protein representations by mutual information maximization. November 2020.
Maxwell L Bileschi, David Belanger, Drew H Bryant, Theo Sanderson, Brandon Carter, D Sculley, Alex Bateman, Mark A DePristo, and Lucy J Colwell. Using deep learning to annotate the protein universe. Nature Biotechmology, pages 1 1-6, 2022.
Amir Shanehsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv preprint arXiv:201 1.0343, 2020.
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3):1496-1355 2020.
Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous Automated Model Evalua- tion (CAMEO) complementing the critical assessment of structure prediction CASP12. Proteins: Structure, Function, and Bioinformatics, 86:387-398, 2018.

---

Under review as a conference paper at ICLR 2023
Rojan Shrestha, Eduardo Fajardo, Nelson Gil. Krzysztof Fidelis, Andriy Kryshtafovych, Bohdan Monastyrskyy. and Andras Fiser. Assessing the accuracy of contact predictions in CASP13. Proteins: Structure, Function, and Bioinformatiis, 87(12):1058-1065 2019.
Adam J Riesselman, John Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, (5(10)881--82) 2018.
Jeffrey M Spencer and Xiaoliu Zhang. Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains. Scientific reports, 701):1114, 2017.
LH Vandenberghe, JM Wilson, and G Gao. Tailoring the AAV vector capsid for gene therapy. Gene therapy, 16(3):311-319, 2009.
Hildegard Büning, Anke Huber, Liang Zhang, Nadja Meumann, and UIrich Hacker. Engincering the AAV capsid optimize vector-hoostinneractions Current opinion in pharmacology, 24:94-104, 2015.
Christopher Barnes, Olivia Scheideler, and David Schaffer. Engincering the AAV capsid to evade immune responses. Current opinion in biotechnology, 60:99-103, 2019.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick F Riley, George M Church, Lucy Colwell, and Eric Kelsic. Deep diversification of AAV capsid protein by machine learning. Nature Biotechnology, 39(6):991-65) 2021.
Nicholas C Wu, Lei Dai, Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. Elife, 5:e1 2016. doi: 10.7554/el.ife. 16665.
Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry Ivankov, Nina Bozhanova, Mikhail S Baranov, Onuralp Soyle- mez, Natalya S Bogatyreva, Peter K Vlasov, Evgeny S Egorov, Maria D Logacheva, Alexey Kondrashov, Dmitry M Chudakov, Ekaterina V Putintseva, Ilgar Z Mamedov, Dan Tawfik, Kon- stantin A Lukyanov, and Fyodor A Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(7603)3397, 2016. doi: 10. 1038//a/ee 1995.
Gabriel Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, al. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, 357 (6347):168-11) 175, 2017. doi: 10. 10.1122 1.1126//ccinnce.aa00603.
Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stochr, et al. Meltome -thas- proteome stability across the tree of life. Nature methods, 17(5):495-503 2020.
Taranch Zarin, Bob Strome, Gang Peng, Iva Pritišanac, Julie D Forman-Ko -ay, and Alan M Moses. Identifying molecular features that are associated with biological function intrinsically disordered protein regions. Elife, 10:e60220, 2021.
Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(1 1):1066-1028, 2017.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations, 2020.
Jesse Vig, Ali Madani, Lav Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models. arXiv preprint arXiv:2006. 15222, 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904, 10509, 2019.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The Long-document transformer. arXiv preprint 15X 2020.
12

---

Under review as a conference paper at ICLR 2023
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint 2XXi::2001.04551,, 2020.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438-9447. PMLR, 2020a.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766 2020.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283 17297, 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers RNNs: Fast autoregressive transformers with linear attention. International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020b.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. Masked language model- ing for proteins via linearly scalable long-context transformers. arXiv preprint (xXi::0006.0355, 2020b.
Liang He, Shizhuo Zhang, Lijun Wu, Huanhuan Xia, Fusong Ju, He Zhang, Siyuan Liu, Yingce Xia, Jianwei Zhu, Pan Deng, et al. Pre-training co-evolutionary protein representation via pairwise masked language model. arXiv preprint arXiv:21 155527 2021.
Sanaa Mansoor, Minkyung Back, Umesh Madan, and Eric Horvitz. Toward more general embeddings for protein design: Harnessing joint representations of sequence and structure. bioRxiv, 2021.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein structure representation learning by geometric pretraining. 2022.
Matt McPartlon, Ben Lai, and Jinbo Xu. A deep SE equivvarai model for learning inverse protein folding, bioRxiv, 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning, ArXiv, 0bs/2204.042233, 2022.
Zichen Wang, Steven A. Combs, Ryan Brand, Miguel Calvo Rebollar, Panpan Xu, George Price, Nataliya Golovach, Emmanuel Oluwatobi Salawu, Colby Wise, Sri Priya Ponnapalli, and Peter M. Clark. LM-GVP: extensible sequence and structure informed deep learning framework for protein property prediction. Scientific Reports, 12, 2022.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 HYPERPARAMETERS FOR PRETRAINED MODELS OF DIFFERENT SIZES
All models are trained for 2 weeks on 1-8 32GB V100 GPUs with dynamic batching. Table summarizes the hyperparameters for CARP. Table A2 summarizes the hyperparameters for ESM.
Table A1: CARP model hyperparameters. Max tokens is the maximum number of tokens per GPU per batch during training.
Model
Parameters
Layers
d
dMLP
Max tokens
GPUs
CARP-640M
643M
56
1280
1280
11000
128 x 32GB V100
CARP-76M
75.7M
32
1024
512
60000
16 x 32 GB V100
CARP-38M
37.9M
16
1024
512
40000
8 x 32 GB V100
CARP-24M
23.9M
16
256
128
400000
2 32
GB V100
CARP-600K
608k
16
128
64
600000
1 X 32 GB V100 1 1 X 16
CARP-40K
415k
16
32
600000
GB V100
CARP-4k
3670
16
8
4
600000
GB V100
Table A2: ESM model hyperparameters.
Parameters
Layers
Heads
d
dMLP
GPUs
86.5M
12
12
768
3972
8
32
GB V100
44.0M
12
768
3072
8 x
32
GB V100
2.92M
192
768
2
GB V100
561k
96
384
1 x 32
GB V100
41.5k
24
96
1
GB V100
4890
4
4
16
I
x
32
GB V100
A.2 ZERO- SHOT FITNESS PREDICTION
We score sequences by masking every mutated position and computing the log odds ratio between the mutated and wild-type residues each mutated position, assuming an additive model when sequence contains multiple mutations:
Σ PEP
logu(z (x logp(x") wt p..pp)
(I)
where P indicates the mutated positions.
14

---

Under review as a conference paper at ICLR 2023
0.8
model
ESMI-1
0.7
PSSM
0.6
ProtBERT-BFO
[Speamman
0.4
0.3
0.1
0.0
BLAT ecoLx amgunaation2213
PAdl rEAST Peidzzl 1-doubles ARRE PIEAE
HEA2 KLOW DEGI Hikkemee' 2014
TMN SILSO
USC3 HUsGA, Rothzl1
SUMO) HUMAN, GAL4 TEAST Si Sendure2211
BLAY ECOL? I Cotermee 1a
BLAT CCOLA Tenalfone prnalforz01 a 4MB HAEAE CREAASTARI TATA TABILZZED,
L2
2018
Pot, HV CA Raungrzzel a
POLG Fa412 THETH
15
HEPPy TEAST golonz Balonzute
TPMI, RUMAR FowterZo
grt ECouI Katenyy
BRCA) HUMAA
HB2 YEAST STA59 Jmmerrta Ammertbe 65
ECOL pal polia0220''
use MOU SE Klevts 201 sintted
HD Full Bioomzzz
YARI JOMMN fredfio Slgges
BF520 are TIM THESA to
LIPSTST 2le
FTEN HURAN rowle 20 1e
BRCAL HUMAN BRCT
RASH Huuless Kunvan
It so) rEAST Balonzo15
RL-00) YEAST Fressr 2016
HUMEN Reth2017
CALRI RUGAA 2012
azviss LiesT Whishentt A800 HiMMA PA FLU Stor20
a01 VEAST Shisheariinss
freldio 201211
PADP
Figure Al: Zero-shot protein fitness prediction. Comparison across 41 deep mutational scanning datasets from DeepSequence. Points are Spearman correlation on each dataset. Horizontal lines show the average Spearman correlation across the datasets. Values for ESM- ESM-1v, 1v, PSSM, and ProtBERT-BFD are taken from Meier et al. (2021).
1.0
0.5
Spearman p
0.0
-0.5
CARP-640M
CARP-76M model
CARP-600k
Figure A2: Spearman correlation between pretrained model checkpoint loss and zero-shot Spearman correlation across 41 deep mutational scanning datasets from DeepSequence.
A.3 PRETRAIN PERFORMANCE VS DOWNSTREAM PERFORMANCE
15

---

Under review as a conference paper at ICLR 2023
0.8
parameters
640M
0.7
76M
accuracy
0.6
600k
task
0.5
secondary structure
0.4
remote homology
0.3
0.2
2.4
2.3
2.2
2.1
2.0
pretrain loss
(a) Structure
1.0
one-vs-many two-vs-many
three-vs-many
Spearman p
low-vs-high
0.4
sampled
0.2
-0.2
-0.4
2.5
2.4
2.3
2.2
2.1
2.0
pretrain loss
(b) GB1
one-vs-many
two-vs-many
seven-vs-many
Spearman
low-vs-high mut-des
0.4
0.2
0.0
2.5
2.3
2.1
2.0
pretrain
(c) AAV
0.60
0.55
Spearman
0.50
0.45
0.40
0.35
0.30
2.5
2.4
2.3
2.2
2.1
2.0
pretrain loss
(d) Meltome
Figure A3: Downstream performance pretrain loss for secondary structure, remote homology, and FLIP tasks.
16


Reviews:
[{"review_id": "yPh9weoTM-", "reviewer": "Reviewer_79qs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "the submission compared convolution neural networks of the same building block and the transformer models on the impact of masked pre-training on downstream tasks including secondary structure predictions and protein design tasks. Results show that there isn't a significant difference between convolutional neural networks and transformer models on downstream tasks. Based on the empirical evidence, the submission argues that the effect of pre-training should be decoupled from the architercture.", "summary_of_the_review": "I would vote for weak acceptance of this paper so that researchers in the field are aware of the decent performance of convolutional neural networks and their argument on decoupling.", "strength_and_weaknesses": "*Strength*\n\n1. The presentation is clear, and the decoupling claim is reasonable.\n\n2. Empirical results are comprehensive, and downstream tasks contain testing various aspects of a pre-trained model.\n\n\n*Weaknesses*\n\n1. there are many ways to handle longer sequences during inference for transformers, for example, from the perspective of positional embeddings, one can use a set of fixed embeddings or recently proposed rotary embeddings.\n\n2. it wasn't immediately clear to me whether positional embeddings were used for convolutional neural networks. Since convolution kernels are homogenous across locations, I imagine that positional embeddings would be helpful.\n\n3. it always puzzles me whether long-distance attention is learnt in transformers, and I understand that people have designed downstream tasks that require long-distance attention to perform well, but what if the realised tasks themselves could be solved with a decent accuracy using only short-distance attention? I was wondering if there exists work on checking the biases in the downstream tasks? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z3xOD7DXpZ", "reviewer": "Reviewer_i3wu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper reports a set of experimental results based on protein sequence masked language modeling using a convolutional architecture, which they name \"CARP\", comparing to ESM-1b, a previously published transformer based model. The authors show that\n  * CARP performs similarly to ESM-1b in terms of MLM loss, for different orders of magnitudes of parameter counts,\n  * CARP is sometimes better than ESM-1b on some downstream tasks, specifically when the evaluation set is out of domain, but it's worse on tasks that involve structure prediction of proteins,\n  * CARP might generalize better to sequence lengths longer than the ones seen in training.\n\nThe authors make some progress towards disentangling the effect of pretraining data and transformer architecture on protein sequence pretraining applications. They show that in some cases pretraining more actually hurts performance on downstream tasks.", "summary_of_the_review": "The paper experiments with a convolutional architecture for protein sequence pretraining and shows that in some downstream tasks it can outpeform a recently published transformer model. The improvements seem to mostly be on out-of-domain tasks (Table 2).\n\nIt's a bit unclear how significant these out-of-domain improvements are, since they are only compared to the ESM-1b baseline, which seems to do terribly on some of these tasks.\n\nStill the discussion is quite interesting to read and it could conceivably lead to a better architecture in future work, by combining the ESM and CARP architectures perhaps, or by finding a way to improve the transformer architecture to do better in out-of-domain evaluations.", "strength_and_weaknesses": "Strengths:\n* The paper gives an interesting account of current work on protein sequence language modeling. This might be useful for researchers unfamiliar with these applications.\n* The observation that the authors’ CNN (CARP) obtains nearly the same loss during pretraining as a transformer (ESM) is quite interesting and surprising. (Figure 2) \n* The paper makes the interesting observation that model size and pre-training don’t always improve downstream tasks.\n* The out-of-domain results seem to strongly favor CARP. This could be interesting, since it shows that for some tasks CNNs might be generalizing better in this application space.\n \nWeaknesses:\n* The paper says “The **primary** drawback of transformers is that the compute and memory required by the attention layers scale quadratically with input sequence length“. This is too strong a statement. Quadratic dependency on the length is, at best, **one** of the drawbacks, but probably not the primary one. There are many versions of transformers that don’t have quadratic dependence on the length of the sequence and can outperform convolution approaches, e.g. Big Bird (Zaheer et al, 2020, cited in this paper), Performer (Choromanski et al, 2020, also cited in this paper).\n* The paper claims that “[positional] encodings can be difficult to extend past the maximum length seen during training”. Where are the limits coming however on the input length during training? If the length during training is capped artificially because of memory requirements, then this limitation can be removed again by using an architecture like Big Bird, and by simply further training Big Bird on longer input sequences after the initial pre-training.\n* Figure 2b is potentially misleading, since it’s comparing MLM loss on different test sets (March 2018 UniRef50 for ESM-1b vs March 2020 UniRef50 for CARP). Why didn’t the authors evaluate ESM-1b on the March 2020 UniRef50 test set? The MLM loss for higher sequence lengths is also very noisy, indicating that there are perhaps very few sequences of high lengths?\nFor example the CNN results in Table 1 are markedly lower than the transformer based ESM-1b baseline. The authors show this table to show that CNNs are not at too large a disadvantage compared to transformers, but the gap still seems large. ESM-1b also wins on zero-shot evaluations, although CNNs seem very close in this case.\n* The paper draws a distinction between convolutional nets and transformers, but it should be conceptually simple to create a hybrid architecture, by adding an attention mechanism and positional embeddings in the convolution. It could be interesting to examine this in future work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QLyavW9Du7", "reviewer": "Reviewer_JHic", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Protein sequence language models are mostly based on the Transformer architecture; while the complexity of it increases exponentially. Thus, this paper studies the use of CNNs instead of Transformer for masked language model pre-training and finetuning. Authors proposed the CARP model for protein sequence pre-training. CNN exhibits the linear complexity according to the length of sequences. More importantly, in some downstream evaluations for estimating the structure prediction and zero-shot mutation effect prediction and for OOD generalization, CNN performs better than the Transformer.\n", "summary_of_the_review": "The motivation for this paper is good; while I do not think the current draft is effectively offering the evidence for better speed and accuracy. Thus, I recommend reject for this paper.\n", "strength_and_weaknesses": "Authors proposed the alternative approaches to the Transformer using the CNN architecture. Standard dilated convolutions are involved to further boost the speed. \n\nCompared to the ESM-1b paper, the results are not good;sometimes showing inferior results. In some fields, the proposed CNN-based model is showing better accuracy in some cases; at the same time it is showing inferior results for other cases. I am not clearly convinced with the authors’ arguments saying that CNNs are alternative to the Transformers. \nAlso, there is no result for longer sequences; while this was the initial question for this draft.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject