Improving Generalizability of Protein Sequence Models via Data Augmentations
Authors: Hongyu Shen
Layne C. Price
Mohammad Taha Bahadori
Franziska Seeger
Venue:  (2021)

Abstract: While protein sequence data is an emerging application domain for machine learning methods, small modifications to protein sequences can result in difficult-to-predict changes to the protein's function. Consequently, protein machine learning models typically do not use randomized data augmentation procedures analogous to those used in computer vision or natural language, e.g., cropping or synonym substitution. In this paper, we empirically explore a set of simple string manipulations, which we use to augment protein sequence data when fine-tuning semi-supervised protein models. We provide 276 different comparisons to the Tasks Assessing Protein Embeddings (TAPE) baseline models, with Transformer-based models and training datasets that vary from the baseline methods only in the data augmentations and representation learning procedure. For each TAPE validation task, we demonstrate improvements to the baseline scores when the learned protein representation is fixed between tasks.  We also show that contrastive learning fine-tuning methods typically outperform masked-token prediction in these models, with increasing amounts of data augmentation generally improving performance for contrastive learning protein methods.  We find the most consistent results across TAPE tasks when using domain-motivated transformations, such as amino acid replacement, as well as restricting the Transformer attention to randomly sampled sub-regions of the protein sequence.  In rarer cases, we even find that information-destroying augmentations, such as randomly shuffling entire protein sequences, can improve downstream performance. 

Under review as a conference paper at ICLR 2021
IMPROVING GENERALIZABILITY OF PROTEIN QUENCE MODELS WITH DATA AUGMENTATIONS
SE-
Anonymous authors Paper under double-blind review
ABSTRACT
While protein sequence data is an emerging application domain for machine learn- ing methods, small modifications to protein sequences can result in difficult-to- predict changes to the protein's function. Consequently, protein machine learning models typically do not use randomized data augmentation procedures analogous to those used in computer vision or natural language, e.g., cropping or synonym substitution. In this paper, we empirically explore set of simple string manip- ulations, which we use to augment protein sequence data when fine-tuning semi- supervised protein models. We provide 276 different comparisons to the Tasks Assessing Protein Embeddings (TAPE) baseline models, with Transformer-basee models and training datasets that vary from the baseline methods only the data augmentations and representation learning procedure. For each TAPE validation task, we demonstrate improvements to the baseline scores when the learned pro- tein representation is fixed between tasks. We also show that contrastive learn- ing fine-tuning methods typically outperform masked-token prediction in these models, with increasing amounts of data augmentation generally improving per- formance for contrastive learning protein methods. We find the most consistent results across TAPE tasks when using Nomain-motivated transformations, such amino acid replacement, as well as restricting the Transformer attention to ran- domly sampled sub-regions of the protein sequence. rarer cases, we even find that informatiionddssroiiin augmentations, such as randomly shuffling entire pro- tein sequences, can improve downstream performance.
INTRODUCTION
Semi-supervised learning has proven to be effective mechanism to promote generalizability for protein machine learning models, as task-specific labels are generally very sparse. However, with other common data types there are simple transformations that be applied to the data in order to improve a model's ability to generalize: for instance, vision models use cropping, rotations, or color distortion; natural language models can employ synonym substitution; and time series data models benefit from window restriction or noise injection. Scientific data, such as a corpus of protein sequences, have few obvious transformations that can be made to that unambiguously preserve the meaningful information in the data. Often, an easily understood transformation to protein sequence (e.g., replacing an amino acid with a chemically similar one) will unpredictably produce either very biologically similar or very biologically different mutant protein.
In this paper, we take the uncertainty arising from the unknown effect of simple data augmentations in protein sequence modeling as an empirical challenge that deserves robust assessment. To our knowledge, no study has been performed to find out whether simple data augmentation techniques improve suite of protein tasks. We focus on fine-tuning previously published self-supervised mod- els that are typically used for representation learning with protein sequences, viz. the transformer- based methods of Rao al. (2019) which have shown the best ability to generalize a set of biological tasks, which are referred to as Tasks Assessing Protein Embeddings (TAPE). We test one more of the following data augmentations: replacing an amino acid with pre-defined alternative; shuffling the input sequences either globally or locally; reversing the sequence; or subsampling the sequence focus only local region (see Fig.

---

Under review as a conference paper at ICLR 2021
P
P
(a) Replacement (Dictionary)
bb) Replacement (Alanine)
(c) Global Random Shuffling
(d) Local Sequence Shuffling
(e) Sequence Reversion
Subsampling
Figure I: Diagram of data augmentations. We study randomly replacing residues (with probability p) with (a) a chemically-motivtte dictionary replacement or (b) the single amino acid alanine. We also consider randomly shuffling either (c) the entire sequence or (d) local region only. Finally, we look at (e) reversing the whole sequence and (f) subsampling to a subset of the original.
We demonstrate that the protein sequence representations learned by fine-tuning the baseline models with data augmentations results in relative improvements between 1% (secondary structure aceu- racy) and 41% (fluorescence p), as assessed with linear evaluation for all TAPE tasks we studied. When fine-tuning the same representations during supervised learning on each TAPE task, we show significant improvement compared to baseline for 3 out of 4 TAPE tasks, with the fourth (fluo- rescence) within lo performance.
also study effect of increasingly aggressive data aug- mentations: when fine-tuning baseline models with contrastive learning (Hadsell et al. 2006 Chen ett al. 2020a) we see a local maximum in downstream performance as a function of the quantity of data augmentation, with "no augmentations" generally under-performing modest amounts of data augmentations. Conversely, performing the same experiments but using masked-token prediction in- stead of contrastive learning, we detect a minor trend of decreasing performance on the TAPE tasks as we more frequently use data augmentations during fine-tuning. We interpret this as evidence that contrastive learning techniques, which require the use of data augmentation, are important methods that can be used to improve generalizibility of protein models.
RELATED WORKS
Self-supervised and semi- semi-supervised methods have become the dominant paradigm in modeling pro- tein sequences for use in downstream tasks. |ao et al. (2019) have studied next-token and masked- token prediction, inspired by the BERT natural language model (Devlin et al. 2018). miesselman ett al. 2019) have extended this to autoregressive likelihoods; and Rives et al. (2019). Heinzinger et al. (2019) and Alley et al. (2019) have shown that unsupervised methods trained unlabeled sequences are competitive with mutation effect predictors using evolutionary features. Of impor- tance this work seff-supervised learning algorithms employed for other data types that or learn data augmentations. For example, Giiarrs et al. (2018) learn image features through ran- dom rotations; Dosovitskiy their correlations to the original samples. yan den Oord et al. (2018) uses contrastive methods to predicts future values of an input sequence. We consider sequence augmentations in natural lan- guage as the most relevant comparison for the data augmentations we study in this paper. Some commonly applied augmentations on strings include Lexical Substitution (Zhang et al. 2015). Back Translation (Xie et al. 2019a). Text Surface TransfornationPermalln
al. (2014) and Noroozi & Favaro (2016) study image patches and
(Coulombe) (018). Random Noise Injection (Xie et al. 201 Wei & Zou 2019), and Synonym Replacement, Random Swap, Random Deletion (RD) Wei & Zou 2019). However, sequence augmentations designed for natural languages often require the preservation of contextual meaning of the sentences, a factor that is less explicit for protein sequences.
Contrastive Learning is a set of approaches that learn representations data by distinguishing pos- itive data pairs from negative pairs (Hadsell et al. 2006). SimCLR (v1 & v2) (Chen et al.) 2020a b) describes the current ofate-of-thh--ar contrastive learning technique.; we use this approach liberally in this paper not only because performs well, but because it requires data transformations to
2

---

Under review as a conference paper at ICLR 2021
Pro-traim self- supervised model sequence
Augmenttdd training
Lioevr ovaliation
Finc-lure tune the best lineur evallation
Contrnative
Misket- Toien Loss
Loss
Tesk-specitte
Quz-llaeer
Masked Toben Hoad
Lese
Self
Denae Laver
Taslesgysiiie
Tank-specifi Heedd
Or
Self-
Scif- Sepervisdl Mönde:
Superviod
Model
Sapariieed Model
Salf- Supervised Midell
With Augmentations
c
S
Pfam validation set
Downstream datasets
Downstrcam datanets
Prepuration
tuning
Linear Evaluation
Model Freezer
Figure 2: Diagram of experimental approach (see Sect. 3.1 We use dashed boxes to indicate
different steps: semi-supervised pre-training. augmented learning, linear evaluation, and finally fine- tuning the best performing augmented models downstream tasks. In each box, we include the general model architectures, with major aub-modules in different colors. The model freezer indicates the semi-supervised model is not updated during linear evaluation:.
ist. Since we focus on protein sequence transformations, the contrastive learning part described in both (Chen et al. 2020a b) is focus. Following Chen et al. (2020a b), denote input data as a E D. with being our training set; we then define an embedding function, fu x H with RN, and mapping function ge + z, with z E RMM, where and 0 the learned model weights. For any æ, we form two copies # = 4 (x) and ø2 12 (x) given functions & l2 ~ T, where T denotes the distribution of the augmentation functions. Given D of size N, the contrastive loss is written as:
L
1 2N
Σ
[1
(1)) where I(u, = log 2 1(=(2) :
(2)
(2)
simi uz mi/r
E Enoguu (imuu,/)//
Here, Zi,x = go (fw (t: (xx) ))). sim (:) :) is cosine similarity, and (0, oo) is a scalar temperature; we choose 7 0.2. By minimizing the contrastive loss, we obtain the learned h as the encoded feature for other downstream tasks. Note, the contrastive loss takes z's as inputs, whereas the encoded feature is h, which is the variable after the function (-) and before go(-).
METHOD
3.1 EVALUATION PROCEDURE & APPROACH TO EXPERIMENT CONTROL
Our goal demonstrate that training self-supervised protein sequence models, with simple string manipulations as data augmentations, will lead to better performance on downstream tasks. To attempt to control external variables, we study the following restricted setting; we provide the pro- cedural diagram in Figure 2 and the corresponding explanations of the four major steps below (See Appendix A for training setups details.):
Baseline.- A self-supervised model Mo is trained on non-augmented sequence data Daeq to do representation learning. To have consistent baseline, we set M, to the Transformer-based model trained and published in Rao et al. (2019), without modification. This was trained with masked- token prediction on Pfam protein sequence data (EI-Gebali et al. 2019); has 12 self-attention layers, heads per layer, and 512 hidden dimensions, yielding 38M parameters in total.
Augmented training on validation set.- We fine-tune My on augmented subsets D C Daegs given set of pre-defined data transformations Taug. We define Maug as the final trained model derived from Tang (Dsgg) (Degg) with M, as the initial conditions for the model parameters. We explore two different methods of fine-tuning augmented data contrastive task (as in Eq. 1 and
3

---

Under review as a conference paper at ICLR 2021
masked-token task (exponentiated cross entropy loss) well as different combinations of data augmentations. We use reduced subsets |vval < [Dzeql to both reduce the computational cost of running bulk experiments, as well as to protect against overfitting. For consistency, we inherit the choice of Daal from the cross-validation split used to train My in |Rao et al. (2019). To adapt the same baseline model Mo to different self-supervised losses, we add loss-specific randomly- initialized layer to the My architecture: contrastive learning with a fully connected layer that outputs 256 dimensional vectors and masked-token uses fully connected layers with layer normalization to output one-hot vectors for each of the masked letters. We define our different choices of T Taug in the next section.
Linear evaluation on TAPE- To assess the representations learned by Maug. we evaluate perfor- mance on four TAPE downstream training tasks (Rao et al., 2019): stability, fluorescence, remote homology, and secondary structure. For consistency, we use the same training, validation, and test- ing sets. The first two tasks are evaluated by Spearman correlation (p) to the ground truth and the latter two by classification accuracy. However, we do not consider the fifth TAPE task, contact map prediction, as this relies only on the single CASP12 dataset, which has an incomplete test set due to data embargoes (AlQuraishi 2019). Secondary structure prediction is sequence-to weeeenceett--qquueee to-sequence task where each input amino acid is classified to a particular secondary structure type (helix, beta sheet, or loop), which is evaluated data from CASP12, 1S1 and CB513 (Berman al. 2000; Moult let al. [ (2188) 2018/K [Klausen et al. (019), with specifically "3-class" classification accuracy being the metric in this paper. The remote homology task classifies sequences into one of 1195 classes, representing different possible protein folds, which are further grouped hicrarchically into families, then super- families; the datasets are derived from from/Fox et al. (203)). The fluorescence task regresses protein sequence to real-valued log-fluored reg-fluorescence intensity measured in |Sarkiyyan et al. (2016). The sta- bility task regresses sequence to a real-valued measure of the protein maintaining its fold above a concentration threshold (Rocklin et al. 2017). We perform linear evaluation by training only single linear layer for each downstream task for each contrasiieelearningg model Mang. but not changing the parameters of Maug. and its corresponding learned encodings, across all tasks. To compare the contrastive learning techniques to further fine-tuning with masked-token prediction, we identify the best-performing data augmentations per-task, then replace the MCL with the masked-token model with the same augmentation Mag then also do linear evaluation on MMT.
Full fine-tuning on TAPE.- For the best- best-performing augmented models in the linear evaluation task (either MCL ang or aug ), we further study how the models improve when allowing the parameters of Mang to vary along with the linear model during the task-specific supervised model-tuning.
3.2 DATA AUGMENTATIONS
We focus on random augmentations to protein primary sequences, both chemically and non- chemically motivated (see Fig. [T. Each of these data augmentations has reasons why it both might help generalize the model and might destroy the information contained in the primary sequence.
Replacement (Dictionary/Alanine)) [RD &
We randomly replace, with probability p. the ;=1 with the most similar amino acid to it A4, ac-
4th amino acid in the primary sequence S = [ cording to a replacement rule; we do this independently for all i. We treat p hyperparameter and will assess how affects downstream TAPE predictions. For Replacement (Dictionary), following French & Robson (1983), we pair each naturally-oceuring amino acid with a partner that belongs to the same class (aliphatic, hydroxyl, cyclic, aromatic, basic, or acidic), but do not substitute anything for proline (only backbone cyclic), glycine (only an H side-chain), tryptophan (indole side chain), or histidine (basic side chain with no size or chemical reaction equivalent). We experimented with different pairings, finding little difference in the results; our best results were obtained with the final mappings: [[A.V], [S,T], [F, Y,, [K, [C.M], [D.E], [N. [V.]]]. We also study replacing residues with the single amino acid, alanine (A), as motivated by
mutagenesis (Cunning-) ham & Wells. 1989). Single mutations to A are used experimentally to probe the importance of an amino acid because Alanine resembles a reduction of any amino acid to its Cg, which eliminates functionality of other amino acids while maintaining certain back rigidity and is thus consid- ered to be minimally disruptive to the overall fold of protein, although many interesting exceptions can still occur by these mutations.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Best linear evaluation results. Bold refers cases that outperform the TAPE baselines; and red is the task-wise best-performing result. MT and CL refer to training with masked-token prediction and contrastive learning, respectively. Stability and fluorescence are scored by Spearman correlation and remote homology (fold, family, superfamily) and secondary structure (CASP12, TS115, CB513) by classification accuracy. Bootstrap errors are reported per task by taking the maximum error found for any of the models. (Also see Appendix C
Scenario
Stability
Fluor.
Remote Homology
2nd Structure
MT: TAPE Baseline
0.498
0.256
[0.200, 0.625, 0.231]
[0.699, 0.756, 0.727]
МТ: No Aug. (7 0)
0.534
0.275
[0.206, 0.636, 0.241]
[0.706, 0.771, 0.729]
MT: Best Aug.
0.516
0.301
[0.207, 0.637, 0.241]
[0.716, 0.771, 0.735)
CL: No Aug.
0,512
0.334
[0.146, 0.529, 0.163]
[0.667, 0.725, 0.678)
CL: Single Aug.
0.562
0.343
[0.183, 0.720, 0.243)
[0.700, 0.757, 0.727]
CL: Leave-one-out
0.337
0.323
[0.168, 0.686, 0.222]
[0.700, 0.756, 0.723]
CL: Pairwise
0.537
0.361
[0.219, 0.718, 0.255]
[0.702, 0.759, 0.726]
Bootstrap (<)
+0.011
++0.006
[0.015, 0.014, 0.012]
+00021,, 0.009, 0.015)
Global/Local Random Shuffling (GRS& & LRSJ.- reshufffe the protein sequence, both glob- ally and locally. For S [A.))=>> define index range [o, with B N, then replace amino acids A; in this range with a permutation chosen uniformly at random. We define Global Random Shuffling (GRS) with 1 and 3 N and Local Random Shuffling (LRS) with the starting point of intervals chosen randomly between E [ 2] and B min(N a + 50), ensuring least two amino acids get shuffled. While shuffling aggressively destroys protein infor- mation, models trained with shuffling can focus more permutation cermutatooniivaaiian invariant features, such as the overall amino acid counts and sequence length of the original protein.
Sequence Reversion & Subsampling [SR & SS].- For Sequence Reversion, we simply reverse the sequence: given S {A4)) we map i + - Note that protein sequence is oriented, proceeding sequentially from the N- to the C-terminii; reversing a protein sequence changes the entire structure and function the protein. However, including reversed sequences might encourage the model to use short-range features more efficiently, as it has for seq2seq LSTM models (Sutskever) ct al. 2014). For Subsampling, we let the sequence index range from E [a, 3]. then uniformly sample
N 2] and then preserve A, for many properties pertaining to the global fold of protein are due to long-range interactions between residues that are well- sellasepreteed in the primary sequence, properties such as proteosomal cleavage or docking depend more heavily on the local sequence regime, implying that training while prioritizing local features might still improve performance.
(a, a + ...., min(N. N, 50)). While
Combining Augmentations.- We consider applying augmentations to the fine-tuning of semi- supervised models both individually and together. For Single Augmentations we consider only one of the augmentations at time. With Leave- One--ne---- Out Augmentation we define lists of augmentations to compare together; for each list of augmentations, we iteratively remove one augmentation from the list and apply all other augmentations during training. Finally, in Pairwise Augmentation all pairs of augmentations are considered.
4 RESULTS
Assessing data- aagmentd representations with linear evaluation.- The core result of this paper uses linear evaluation methods to probe how well data augmentations are able to improve the learned representation of protein sequences. In Table we summarize the best results from various data augmentation procedures, highlighting all the cases that outperform the TAPE Baseline. We compare identical model architectures trained with the same data, but using various data augmen-
5

---

Under review as a conference paper at ICLR 2021
Stability
Fluorescence
R0(0.011
0.962
0.537
0:149
0.134
0.265
0.390
0.53
0:297
5.316
0.313
0.351
5:30
0.20
0.35
60(0.5)
0.352
0,327
5.121
0013
0.355
0,426
0.239
9,271
0.34]
0.341
6.355
0.28
0.43
0.33
R4/0.011
0218
0.844
0.203
0.309
0.33
0.31.
0,277
0.281
0.314
5.00
0.25
0.32
0.33
0.31
189
0.134
0.013
0.338
0.157
0.392
0.23
0.29
5R
2625
0.115
0.203
5.087
0:157
0.238
0:253
0.18
0.308
0.315
5.281
0.332
0.303
0.230
0.238
0.26
55
1.9
0.426
0.309
9.26
0.302
0.263
0.355
0014
0.266
0.269
9.256
0.283
0.276
6,296
0.243
-027
Secondary Structure (CASP12)
Remote Homology (F (Family)
0.683
0.665
0.654
0.664
0.701
R010.5
-0.09
0.68
-0.6
0.62
RA69.01 011
0.644
0.680
0.572
0.663
0.665
0.792
-0.67
0.696
0.324
2.401
0.442
0.340
2,444
0.728
-0.59
GRS
6.605
0.424
0.672
3.688
0.671
0.683
0.416
0.435
3.409
0.435
0.040
0.55
1R5
0.654
0.657
0.663
0.655
0.663
0.695
0.665
0.65
5.449
0.423
2.349
0.415
2.434
0.674
0.51
0.47
0 466
0.631
0.665
0.671
0.663
0.664
0.692
6:422
0.486
2.464
0.362
0.434
6.430
0.683
A21
0.750
0,702
0.663
0.855
0.822
0.725
0.63
6.730
0.000
0,730
0,480
0.674
5.50
0.720
0.39
6010.00
203
01
693
L85
0.62
8ote. su
RDi
M6.011
L85
+7
0.35
Figure 3: Contrastive learning performance with pairwise & single augmentations in linear evalua- tion for 4 different tasks. The axes refer to different augmentations, with diagonal being a single aug- mentation. The values in the heatmaps are correlation (stability and fluorescence) and classification accuracy (remote homology and secondary structure). We do not consider two pairs: RD(p 0.01) & RD(p = 0.5) and GRS LRS, due to redundancy. The per-task performance of the masked-token TAPE Baseline model is colored white in each subfigure; red is better performance, blue is worse (Also see Appendix B
tations, to two baselines: (1) the transformer-based self-supervised model from Rao et al. (2019) which we call the TAPE Baseline; and (2) a contrastive learning model trained in the SimC ap- proach, but using data augmentations, i.e., only use the negative sampling part of SimCLR; we call this the Contrastive Baseline. We also report standard deviation (o) for the major figures of this paper by bootstrapping the testing results 5,000 times, with convergence after 3, 000 samples.
We see broad improvement when using contrastive learning with data augmentations in comparison to both baselines for the stability, fluorescence, and remote homology tasks, and better-or-similat results for secondary structure prediction. For stability, we find +0,064 in Spearman correlation (p) (best augmentation: RD(p 0.01 /0.5)) in comparison to the TAPE Baseline and +0.050 compared to the Contrastive Baseline. For fluorescence and remote homology, we see improvement to both the TAPE and Contrastive Baselines, with the pairwise combination of RD(p 0.01) & LSS) and RA(p 0.01) & SS yielding the best results, respectively. Compared to the TAPE Baseline, we obtain +0. 105 in p for fluorescence and /+1.9% +9.3%, +2.4% J in classification accuracy for re- mote homology on the three test sets; compared to the Contrastive Baseline, we find +0.027 in p and [+7.3% +18.9 9%. +9. +9.2%] in classification accuracy. The masked-token prediction model trained with RA(p 0.01) & SS performs the best secondary structure, with /+1.7% 1.5%, +0.8%) in the classification accuracy on the test sets compared to the TAPE Baseline and (+4. 9%, +4.6%, +5.7% compared the Contrastive Baseline. The best-performing contrastive learning model for secondary structure provides (+3.5%. +3.4%, +4.8%) in classification accuracy compared to Con- trastive Baseline. Our interpretation is that augmentation and contrastive learning provide better encoded feature spaces that help improving the performance on protein's 's downstream tasks. See for the complete results of linear evaluation experiments.
Appendix B
Figure 3 demonstrates our linear evaluation results using contrastive learning based on the compo- sition of pairs of data augmentations. For stability, amino acid replacement (with either a dictionary (RD) or alanine alone (RA)) consistently improves performance compared to the TAPE baseline, as well as to other augmentation strategies, which typically underperform the baseline. Fluorescence sees improvements using all data augmentations, but random shuffling (LRS & GRS) well as the binomial replacement of both types result in the best individual performance. For remote homology, it is apparent that subsampling plays important role in model performance given the improve- ment it introduces on the three testing sets; the "family" homology level is included here and the other remote homology tasks are qualitatively similar. Similarly, we see that data augmentation

---

Under review as a conference paper at ICLR 2021
Contrastive, Stability Fleenesence
Contrastive: Remote Homoloy
Contrastive: Secordary Structrre
Stability Fluorescence
CA5P12
0.7
Family Supertamily
TS115
Clsifration Accrssy
CB513
0,39
0.697
0.660
0.450
0.0.1
0.75
0.7
0.297
0 280
0,206
Clossi
>
0.232
0,210
9.70
00.750
8.878
0.674
0.71
0.060
0.602
8.44
6.176
0.172
0 2.00
0.75x
0.0
0.0
0.50
Binoriall
Uinarnal
0.0
Moskedd-Tbeen Srability Clupresreare
3.0
Masked Thieen: Remote Homology
0.40
Masked-Token: Secondary Structure
Stability Fluorescence
Fold
CASP12
Family
TS115
Clasalicaaton Accursay
Superfamily
Clasiiiaain Accuracy
CB513
0.534
0.515
0.404
0 836
0.857
0.624
0.00
0.771
0.770
0.771
0.771
0.004
0.729
0.727
0.727
0.727
6.3
0.275
0.27
0.287
0.256
0.241
0.243
0.231
0.306
6.706
0.09
0.2
0.206
0.207
0.201
0.200
0.1
6.
Augmantation
0.3 53
Nugtmnnaatoon
0.3
Reueon n0
8.1
61
Figure 4: Top row: Effects of Binomial Replacement p for linear evaluation on constrastive learn- ing models for TAPE tasks. Bottom row: Effects of augmentation ratio y for linear evaluation on TAPE's self-supervised model with the best performing task-specific augmentations (see "Best Aug." row in Table 2 using masked token prediction. The subfigures include linear evaluation results with different augmentation ratios, y. ">=0.0" refers to fine-tuning with no data augmenta- tions, whereas "Baseline" refers to the TAPE pre-trained model with no further training. (Also see Appendix(C]
procedures that use subsampling tends to yield better performance than alternatives, with the best performing approach using subsampling alone. For complete heatmaps for the 6 remote homology and secondary structure testing sets, please refer Fig. in Appendix B
Effeet of increasing augmentation rates.- Fig. presents results on the effect of varying data augmentations in two cases: (1) increasing the amino acid replacement probability p for the Re- placement Dictionary [RD] strategy with contrastive learning (top row); and (2) by augmenting increasingly larger fractions of the input data according to the best augmentations (see "Best Aug." row in Table 2) found in linear evaluation for masked-token prediction (bottom row). We define the augmentation ratio y as the fraction of the samples in the validation dataset that are randomly augmented per epoch; for contrastive learning fine-tuning, data augmentations are required for every data element.
For masked-token prediction, we see little change in performance for any task as a function of ? using the for any of the best corresponding augmentation strategies. However, there is a small, but consistent, reduction in performance with increasing 7, implying that masked-token prediction is not always able to significantly improve its performance by using data augmentations. is clear that large augmentation ratios 7 ~ 1 hurt the model performance on Stability, Fluorescence and Remote Homology tasks. We also see that the TAPE Baseline model generally performs worse than further training with no data augmentation, indicating further training of the baseline model with the same data and procedure can improve the performance of Rao et al. (2019).
However, for contrastive learning, we see clear evidence that data augmentations can help general- ization. We see increasing Spearman correlation for stability and fluorescence tasks and increasing accuracy for remote homology and secondary structure with increasing p for p < 0.01. We see consistent decrease in all metrics, to the lowest seen amount for each task, for replacement prob- ability p 0.1 and then recovery to larger (sometimes the largest seen) for higher replacement p 0.5. However, no augmentation, p = O, consistently underperforms as compared to alternative values of p 0.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Model fine-tuning results, with associated training method and data augmentation proce- dure for each task. Testing sets for remote homology: (fold, family, superfamily), and for secondary structure: (CASP12, TS115, CB513).
Scenario
Stability
Fluor.
Remote Homology
2nd Structure
TAPE Best
0.730
0,680
[0.210, 0.880, 0.340]
[0.710, 0.770, 0.730J
[0.209
[0.711+0.015,
Our Best
0.748=1..00
0.677=0.004
0.921=
0.778=00008,
0.377=0.014]
0.739=00003]
Best Models Best Aug.
CL. RD(0.01 or 0.5)
CL. RD(0.01)) LRS
CL.
MT RA(0.01) & SS
RA(0.01) & SS
Effect of contrastive learning-- To assess the relative effects of contrastive learning and masked- token prediction, we compare results between the two approaches with or without data augmenta- tions. All information for this comparison is in Fig. and Table [ It is unsurprising that using the identity function as data transform in SimCLR (Eq. yields little increase in generalizabil- ity: indeed, we see that masked-token prediction has better performance than contrastive learning for all tasks with data augmentations (Fig. y 0 vs = 0). However, sults when comparing contrastive learning vs masked-toker prediction methods with the same data augmentation techniques. As seen in Table ("MR: Best Aug." " row highest/red numbers in "CI.: ** rows): contrastive learning significantly improves over masked-token prediction for stabil- ity (+0.046 in p), fluorescence (+0.061 in p), and remote homology ((+1.2%, +8. 18,,%, +1.4%] in classification accuracy); and masked-token prediction improves over contrastive learning for sec- ondary structure ((+1.4%, + +1.2%, +0.9% in classification accuracy). Overall, we cannot conclude from these pairs of linear evaluation studies that contrastive learning definitively performs better or worse than masked-token prediction on all downstream tasks: different tasks benefit from different training procedures and different combinations of data augmentations. However, we observe that the overall best results from Table | utilize the combination of contrastive learning with pairs of data augmentation for all tasks besides secondary structure prediction.
mixed
Exploring the best performance via full fine-tuning. We provide results of the best perform- ing fine-tuned models (on downstream tasks) and the comparison to the TAPE's original baselines in Table in order to verify whether the learned representations of the best models provide good initialization points for transfer learning. Here, have done full fine-tuning only the best per- forming, per-task models found during the linear evaluation study (see Table [.. Notice that the baseline comparison changes in this table than for the linear evaluation results above because we allow the optimization to also adjust the parameters of the self-supervised models for every task (the TAPE baselines in Table are from Rao et al. (2019)) The fine-tuned, futa-aug models out- perform the TAPE baseline on stability (+0.018 in Spearman correlation p), remote homology and secondary structure; and they perform within one a on fluorescence, although the large difference in performance between full fine-tuning and linear evaluation on the fluorescence tasks indicates that most of the model's predictive capacity is coming from the supervised learning task itself. The ran- dom amino acid replacement strategy is consistent approach that achieved our best performance for all tasks and subsampling performed well on tasks that depend on the structural properties of pro- teins (remote homology and secondary structure): [+0.0%, +4.1 1% +3.7% classification accuracy for remote homology and [+0.1% +0.8% +0.99 40. classification accuracy for secondary structure.
5 CONCLUSION
We experimentally verify that relatively naive string manipulations can used as data augmenta- tions to improve the performance of self-supervised protein sequence on the TAPE validation tasks. We demonstrate that, in general, augmentations will boost the model performance, in both linear evaluation and model fine-tuning cases. However, different downstream tasks benefit from different protein augmentations; no single augmentation that we studied was consistently the best. However, the approach that we have taken, where we fine-tune a pretrained model the validation set
8

---

Under review as a conference paper at ICLR 2021
quires significantly lower computational cost than training the full training set. Consequently, modeler interested in small number of downstream tasks would not be over-burdened to attempt fine-tuning of the semi-supervised model on a broad range of data augmentation transformations.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, et al. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):135--1322, 2019.
Mohammed AlQuraishi. Proteinnet: a standardized data for machine learning of pro-
tein structure. BMC bioinformatics, 20(1):1-10, 2019. URL htttps://github. com/
sgj.aborattory/rrteeinns
HM Berman, Westbrook, Z Feng, al. The protein data bank. Nucleic acids research, 28(1):
235-242, 2000.
T Chen, S Kornblith, M Norouzi, and G Hinton. A simple framework for contrastive learning of visual representations. In Thirty-seventh International Conference on Machine Learning. 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33, 2020b.
Coulombe. Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint 1812::181,. 2018.
BC Cunningham and JA Wells. High-resolution epitope mapping of hgh-receptor interactions by alanine-scanning mutagenesis. Science, 244(4908); 1081-1085, 1989.
J Devlin, MW Chang, K Lee, and K Toutanova. BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding. arXiv preprint arXiv: 1810.08855, 2018.
Dosovitskiy, Springenberg, M Riedmiller, and Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in neural information processing sys- tems, pp. 766-774, 2014.
El-Gebali, Mistry, Bateman, al. The pfam protein families database 2019. Nucleic acids research, 47(D1 (7(D1))D447-D443, 2019.
NK Fox, SE Brenner, and JM Chandonia. Scope: Structural classification of proteins-extendeed integrating scop and astral data and classification of new structures. Nucleic acids research, 42 (D1):D304-D33) D309, 2013.
S French and B Robson. What is conservative substitution? Journal of molecular Evolution, 19 (2):171-175, 1983.
S Gidaris, Singh, and Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Hadsell, $ Chopra, and Y LeCun. Dimensionality reduction by learning invariant map- ping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'O6 volume pp. 1735-1742. IEEE, 2006.
Michael Heinzinger, Ahmed Elnaggar, Yu Wang, al. Modeling aspects of the language of life through transfer-leariing protein sequences. BMC bioinformatics, 20(1):723, 2019.
MS Klausen, MC Jespersen, H Nielsen, et al. Netsurfp-2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 2019.
J Moult, K Fidelis, Kryshtafovych, al. Critical assessment of methods of protein structure pre- diction (CASP)-Round XII. Proteins: Structure, Function, and Bioinformatics, 86:7-15, 2018. ISSN 08873585.
doi: 10.1002/ppot.2415. URL http:://ooi..... http/ddowwwwwyy.ccc com 10000//
prot 25415
M Noroozi and P Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Rao, N Bhattacharya, N Thomas, et al. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems, pp. 9689-9701, 2019.
10

---

Under review as a conference paper at ICLR 2021
AJ Riesselman, JE Shin, AW Kollasch, et al. Accelerating protein design using autoregressive generative models. bioRxiv, pp. 757252, 2019.
A Rives, S Goyal, Meier, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. bioRxiv, pp. 622803, 2019.
GJ Rocklin, TM Chidyausiku, Goreshnik, et al. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, (5776347)1168-1777 2017.
KS Sarkisyan, DA Bolotin, MV Meer, et al. Local fitness landscape of the green fluorescent protein. Nature, 333 2016.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
van den Oord, Y Li, and O Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: htti:.880.037445 2018.
J Wei and K Zou. Eda: Easy data augmentation techniques for boosting performance on text clas- sification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 6383-6389) 2019.
Xie, Dai, E Hovy, et al. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904. 12848, 2019a.
Xie, SI Wang, Li, al. Data noising as smoothing in neural network language models. In 5th International Conference on Learning Representations, ICLR 2019b.
X Zhang, J Zhao, and Y LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.
11

---

Under review as a conference paper at ICLR 2021
Table 3: Hyperparmeter Setups for Models in Different Tasks
Levels
Hyperparmeters # GPUs
Batch Size
Learning Rate
Epochs
Gradient Acc
SimCLR / TAPE Baseline
512
le-4/le-5
8
30
8
Stability
256
le-4/1e-5
1
60
4
Fluorescence
256
le-4/le-5
1
60
16
Remote Homology
256
5c-4/5e-5
1
60
8
Secondary Structure
256
le-4/1e-5
40
8
APPENDIX
A TRAINING DETAILS
For the augmented training, we focused training the self-supervised part of the model. Namely, we apply the hyperparameters in 3ablee] Row I, on the self-supervised part in either the SimCLR (contrastive learning) or the masked-toke prediction model. Here train all the models for 30 epochs on the Pfam validation set in order to make a relatively fair comparison. As discussed in the main paper, after the augmented training is finished, we perform linear evaluations on the pre-trained model in the previous steps with the hyperparameters listed in Table Row 2-5 for the 4 downstream tasks. All of the linear evaluation results shown in the main paper and appendix are based on the best results we find after the augmented training and linear evaluation with the hyperparameters in Table 3 For model fine-tunin, since models trained with contrastive learning and with TAPE's semi-supervised learning models have different statistics (model parameters are different), we do not consider using the same set of hyperparameters. Instead, we report the best results in comparison to the TAPE's result. The corresponding hyperparameter setups of the best cases described above, including the augmented training and linear evaluation, can be found Table 4 addition, the optimizer being applied is AdamW, which is identical to the one in TAPE. Given the NVIDIA V100 GPUs we use have 16 GB memory, a memory limitation, we constrain the sequence length S 512 to enable the training. The batch size we report in appendix are the total batch that considers the number of GPUs. "Gradient Acc" in Table 3 is short for "Gradient Accumulation Steps", which describes how many steps per gradient update the model.
B LINEAR EVALUATION RESULTS
Here we provide comprehensive linear evaluation results of contrastive learning with single aug- mentation, leave-one-out and pairwise augmentation. To Simplify the tables, we use the same ab- breviations applied in the main paper to indicate the augmentations. We summarize the best results after the training and evaluation according to Section (A for both single augmentation and pairwise augmentations in Figure 5 We diverging palette with the center (gray/white) being the best linear evaluation baseline results with TAPE's pre-trained model and warmer colors refering to the better-than- -aaaelna results and cooler colors being worse-than than--slaa-b results. The diagonal values come from single augmentation setup and all other values come from pairwise augmentation setup. Table and Table 6 also include Figure 5 corresponding values. By checking the values of the figure and tables, we observe the following: (1) For stability, the binomial replacement works well with single augmentation and pairwise augmentation. There is no improvement from leave-one-out augmentation cases. (2) For fluorescence, the pairwise augmentation with binomial replacement and shuffling can improve the model performance. (3) For remote homology, we clearly see improve- ments coming from subsampling on all of the three testing sets. And this is independent of other augmentations in the pairwise case. (4) For secondary structure, we do not observe gains from either single or pairwise augmentation, which is consistent with the discussion in the main paper. The best case for secondary structure comes from the masked-token prediction model with augmentations. Besides the results above, we also provide leave-one-out results in Table with different leave- one-out cases that contain different augmentations. With leave-one-out augmentations, we observe only several cases where the leave-one-out cases outperform TAPE's baselines across 4 downstream
12

---

Under review as a conference paper at ICLR 2021
Table 4: Configurations of the best models
Configurations of Best Models in Augmented Training
Stability
Fluorescence SimCLR-basse
Remote Homology
Secondary Structure
Model Type
SimCLR-based
SimCLR-basee
TAPE-based
Batch Size
512
512
512
512
Gradient Acc
8
8
8
8
Learning Rate
le-5
le-5
le-5
le-5
# GPUs
8
8
8
8
# Epochs
30
30
30
30
Aug. Type
Single
Pairwise
Pairwise RA + SS
Pairwise RA + SS
Aug. Config
RD(p = 0.01 /0.50)
RD(p = 0.01) + LRS
Configurations of Best Models in Linear Evaluation
Stability
Fluorecsence "iae
Remote Homology Secondary Structure
Model'Typ
SimCLR-based
SimCLRR-assee
TAPE-based
Batch Size
256
256
256
64
Gradient Acc
4
16
8
Learning Rate
le-4
le-4
5e-4
le-4
# GPUs
1
1
1
# Epochs
60
60
40
Configurations of Best Models in Fine-tuning
Stability SimCLR-baad
Fluorecsence SimeLR-bbased
Remote Homology SimCLP CanCLR-based
Secondary Structure
Model Type
TAPE-based
Batch Size
256
32
32
64
Gradient Ace
16
16
Learning Rate
le-4
3e-5
3e-5
le-4
# GPUs
1
1
# Epochs
60
60
20
50
Table 5: Linear Evaluation Results of Single Augmentation
Scenario
Stability
Fluorescence
Remote Homology
Secondary Structure
Baseline
0.489
0.256
[0.200, 0.625, 0.231]
[0.699, 0.756. 0.727]
RD(p 0.01)
0.562
0.297
[0.173, 0.660, 0.210]
[0.674, 0.713, 0,679]
RD(p 0.5)
0.562
0.259
[0.153 ,0.529 ,0.166]
[0.683, 0.724, 0.684]
RA(P = 0.01)
0.528
0.277
[0. 164 0.491, 0.160]
[0.680, 0.706, 0.674]
GRS
0.289
0.336
[0.110, 0.439, 0.100]
[0.668, 0.683, 0.646]
LRS
0,338
0.343
[0.100, 0.475, 0.095]
[0.666, 0.679, 0,646]
SR
0.258
0.333
[0.104, 0.433, 0.200]
[0.664, 0.682, 0.646]
SS
0.355
0.281
[0. 10,183, 0.720, 0.243]
[0.705, 0.735, 0.707]
tasks. Nevertheless, the leave-one-out results provide insights that should not combine arbitrary augmentations given the decrease the performance we observe in leave-one-out cases.
13

---

Under review as a conference paper at ICLR 2021
Stability
Fluorescence
RD(0.01)
R
0.562
0.527
0.121
0.00
0.013
0:35
0.426
RDI0.5)
0.259
0.271
0,341
0.34
0.335
0.89
0.33
RAC0.011
0.537
0.527
0.528
0.28
0.284
0203
0.309
0.38
RA(0.01)
0.336
0.271
0.277
0.296
0.314
0.281
0.25
0.32
0.149
0.121
0.289
0.087
0.27
0.33
0.341
0.286
0.332
0.31
0.28
-0.30
LRS
LRS
0.343
0.23
-0.29
SR
0.265
0.155
0.203
0.087
0.157
0.288
0.263
0.19
SR
0.308
0.335
0.281
0.332
0.303
0.333
0.296
00.28
0.390
0.426
0.309
0.26
0.302
0.263
0.355
0.268
0.289
0.256
0.283
0.276
0.296
0.281
R1)
RD10.9
R4.0.01
GRS
LAS
Roc0.01)
RD10.9
RA(0.01)
GRS
LRS
Remote Homology (Fold)
Remote Homology (Family)
Remote Homelogy (Superfamily)
RO10.5)
0.353
0:159
0.02
0.695
0.392
5.383
5.20
0.529
3.594
0.035
6.5
0.06
0.146
0.185
070
5.069
0.095
232
0.23
R400.011
0.199
0.159
2.164
0.225
0.111
0:322
0.210
2.17
0.008
0.491
0.462
0:340
0.444
0.718
0.59
0.23
0.183
5.002
0.004
0.15
URC
0110
0.00
3.111
0.100
0.220
3.124
2.40
4402
3.340
0.475
0.434
0.674
0.009
0.053 0.082
5.005
0.090
0.232
0.23
0111
0.292
6.131
0.386
0:634
0.683
0.101
5,090
0.000
0.32
0.183
2.11
0.624
0.722
0.39
0.222
0.53
0.00
-..00
Rodt (01t:5)
80)
[015-5)
RDB 3)
Ractos
RESO
Secondary
(CASP12)
Secondary
(TS115)
Secondary Structure (C8513)
B674
8A65
0.701
0.737
0:675
0.75
0.579
0.583
5.03
0.600
8010.5)
0.631
8.00
0:724
0:522
0.752
0.582
2.500
0.002
-0.70
8400.011
0683
3.5M
5 3.680
8472
5.603
0.565
0.702
2.777
0.701
0.706
0.74
0.675
0.003
0.759
0.008
0.613
0.574
0.611
5.646
0.60
0.776
0.571
0.70
0.55
U6
5.65
6:01
0.675
0.752
0.00
0.545
5.046
0.617
-0
5R
0664
0.631
5:655
0:671
0.663
0.664 0.692
0.82
0.500
0.81
0.851
0.682
0.753
0.64
0.823
0.543
0.84
0.567
0.666
0.724
2.53
0.69
0.53
k060.0
82l0.5)
ORC
MIGSU 57
485
Figure: 5: Contrastive Learning Performance with Pairwise & Single Augmentation in Linear Eval- uation. The heatmaps include the performance of contrastive learning models with pairwise/single augmentations for 4 different downstream tasks considering all possible testing sets. Both x and y axes refer different augmentations. The diagonal of heatmaps refer the single augmentation cases. All other cells refer to cases with pairwise augmentations. The values in the heatmaps refer to evaluation results: "p" for Stability and Fluorescence, and "Classification Accuracy" for Secondary Structure and Remote Homology.
14

---

Under review as a conference paper at ICLR 2021
Table 6: Linear Evaluation Results of Pairwise Augmentation
Scenario
Stability
Fluorescence
Remote Homology
Secondary Structure
Baseline
0,489
0.256
[0.200, 0.625, 0.231]
[0.699, 0.756, 0.727]
RD(p 0.01) RA(p 0.01)
0.537
0.316
[0.189, 0.696. 0.223]
[0.683, 0.737, 0.688]
RD(p 0.01) & GRS
0.149
0.341
[0.101, 0.419, 0.095]
[0.665, 0.676, 0.645]
RD(p 0.01) SR
0.265
0.308
[0.111, 0.422, 0.091]
[0.664, 0.682, 0.646)
RD(P 0.01) & LRS
0.134
0.361
[0.097, 0,449, 0.089]
[0.654, 0.675, 0.633]
RD(p 0.01) SS
0.390
0.268
[0. 0.730, 0.262]
[0.701, 0.759, 0.725]
RD(p 0.5) & RA(p 0.01)
0.527
0.271
[0.159, 0.594, 0.185]
[0.644, 0.701, 0.663]
RD(p = 0.5) & GRS
0.121
0.341
[0.092, 0.405, 0.077]
[0.624, 0.649, 0.603]
RD(p 0.5) & SR
0.155
0.335
[0.092, 0.446, 0.095]
[0.631, 0.646, 0.603]
RD(p = -..5) & LRS
0.013
0.341
[0.095, 0,403, 0.083]
[0.637, 0.633, 0.605]
RD(p 0.5) & SS
0.426
0.289
[0.181, 0.688, 0.232]
[0.700, 0.753, 0.722]
RA(P 0.01) & GRS
0.218
0.286
[0.125, 0,449, 0.103]
[0.672, 0.679, 0,644]
RA(P 0.01) SR
0.203
0.281
[0.123, 0.444, 0.104]
[0.665, 0.680, 0.645]
RA(P 0.01) & LRS
0.284
0.314
[0.111, 0.349, 0.082]
[0.663, 0.679, 0.646
RA(p 0.01) SS
0.309
0,256
[0.219, 0.718. 0.255]
[0.702, 0.759, 0.726]
GRS & SR
0.087
0.332
[0.131, 0.362, 0.099]
[0.671, 0.681, 0.641]
GRS SS
0.276
0.283
10.115, 0.440, 0.108]
[0.663, 0.700, 0.668)
SR & LRS
0.157
0.303
[0.129, 0.434, 0.098]
[0.663, 0.681, 0.647]
SR & SS
0,263
0.296
[0.186, 0.683, 0.205]
[0.692, 0.753, 0.724]
LRS & SS
0.302
0.276
[0.174, 0.674, 0.212]
[0.695, 0.756, 0.724]
Table 7: Best linear evaluation results. Bold refers to cases that outperform the TAPE baselines; and red is the task-wise best-performing result. MT and CL refer to training with masked-token prediction and contrastive learning, respectively. Stability and fluorescence are scored by Spearman correlation (p) and remote homology and secondary structure by Cross Entropy. Bootstrap errors are reported per task by taking the maximum error found for any of the models.
Svenario
Stability
Fluor.
Remote Homology
2nd Structure
MT: TAPE Baseline
0.498
0.256
[4.433, 1,862, 4.378]
[0.661, 0.541, 0.616]
MT: No Aug. (y 0)
0.534
0.275
[4.445, 1.860, 4.381]
[0.643, 0.532, 0.608]
MT: Best Aug.
0.516
0.301
[4.437, 1.872, 4.404]
[0.649, 0.529, 0.602]
CL: No Aug.
0,512
0.334
[4.884, 2.283, 4.900]
[0.753, 0.625, 0.713]
CL: Single Aug.
0.562
0.343
[4.709, 1.285, 4.214)
[0.672, 0.561, 0.649]
CL: Leave-one-out
0.337
0.323
[5.001, 1.479, 4.460)
[0.705, 0.574, 0.674]
CL: Pairwise
0.537
0.361
[4.433, 1.279, 4.045)
[0.676, 0.560, 0.638)
Bootstrap (<)
10.011
#0.006
[000000, 0.005, 0.007]
=00033, 0.018, 0.009]
C SUPPLEMENTARY
In this section, we provide supplementary results. Specifically, we provide the complete comparison table the masked-token prediction model with different augmentation ratio in Table 9 We also provide corresponding plot in Figure and analylsis the main paper. Besides, Table provides similar information Table 2 in the main paper, except for values for remote homolody and secondary structure here being the cross entropy, rather than classification error.
15

---

Under review as a conference paper at ICLR 2021
Table 8: Linear Evaluation Results with Leave-one-out Augmentations
Meta Description
Cases
Leave-onc-out Augmentations
Case
[RD(p D. RA, GRS, SR, SS]
Case 2
[RD(p 0.50), RA, GRS, SR, SS]
Case 3
[RD(p 0.01), RA, LRS, SR, SS]
Case 4
[RD(p 0.50), RA, LRS, SR, SS]
Stability
Case
Case
Case
Case 4
All entries
0.337
0.161
0.212
0.248
No RD(0.01 0.5)
0.230
0.230
0.335
0.230
No RA
0.136
0.113
0.207
0.195
No GRS LRS
0.222
0.088
0.222
0.088
No SR
0.326
0.145
0.243
0.138
No SS
0.103
-0.082
0.054
-0.018
MT: TAPE Baseline
0.489
Fluorescence Case 2
Case
Case 3
Case 4
All entries
0.293
0.314
0.281
0.281
No RD(0.01 or 0.5)
0.250
0.250
0.284
0.250
No RA
0.229
0.222
0.280
0.283
No GRS LRS
0.288
0.323
0.288
0.323
No SR
0.260
0.268
0.286
0.308
No SS
0.288
0.263
0.308
0.251
MT: TAPE Baseline
0.256
Remote Homology
Case
Case 2
Case 3
Case 4
All entries
[0.104, 0.435, 0.102]
[0.091, 0.407, 0.097]
[0.132, 0.624, 0.179]
[0.128, 0.543, 0.153]
No RD(0.01 or 0.5)
[0.111, 0.441, 0.100]
[0.111, 0.441, 0.100]
[0.128, 0.614, 0.178)
[0.111, 0.441, 0.100)
No RA
[0.108, 0.433, 0.106]
[0.085, 0.411, 0.099]
[0.125, 0.615, 0.178]
[0.130, 0.537, 0.148)
No GRS or LRS
[0.166, 0.688, 0.214]
[0.152, 0.609, 0.180]
[0.166, 0.688, 0.214]
[0.152, 0.609, 0.180)
No SR
[0.105, 0.436. 0.100]
[0.084, 0.296, 0.086]
[0.168, 0.686, 0.222]
[0.135, 0.636, 0.192]
No SS
[0.104, 0.428, 0.08]
[0.105, 0.339, 0.086]
[0.105, 0.435, 0.089]
[0.095, 0.363, 0.084)
MT: TAPE Baseline
[0.200, 0.625, 0.231]
Secondary Structure Case
Case
Case 3
Case 4
All entries
[0.664, 0.680, 0.648]
[0.671, 0.702, 0.673]
[0.690, 0.750, 0.718)
[0.691, 0.744, 0.715)
No RD(0.01 or 0.5)
[0.659, 0.701, 0.666]
[0.659, 0.701, 0.666]
[0.688, 0.751, 0.720]
[0.659, 0.701, 0.666]
No RA
[0.663, 0.701, 0.667]
[0.581, 0.588, 0.567]
[0.125, 0.615, 0.178]
[0.685, 0.745, 0.715)
No GRS or LRS
[0.695, 0.754, 0.723]
[0.687, 0.746, 0.718)
[0.695, 0.754, 0.723]
[0.152, 0.609, 0.180]
No SR
[0.673, 0.703, 0.668]
[0.670, 0.705, 0.675]
[0.700, 0.756, 0.723]
[0.700. 0.747, 0.719]
No SS
[0.665, 0.684, 0.645]
[0.638, 0.658, 0.615]
[0.665, 0.682, 0.645)
[0.630, 0.658, 0.624)
MT: TAPE Baseline
[0.699, 0. 756, 0.727]
16

---

Under review as a conference paper at ICLR 2021
Table 9: Ratio Effects on the Masked- Token Model
Scenarios
Results
Baselines
Augmentations
0.1
0.016
Stability
0.5
0.503-0.007
0.48911.0077
RD(p 0.01/0.5)
1.0
0.488=0.000
0.1
0.301=0.005
Fluorescence
0.5
0.009
0.256:10.006
RD(p 0.01) & LRS
0.287 1.0
[0.207:10.015,
0.1
0.637-10.01,
0.241=0.012]
[0.201
[0.200=0.001,
Remote Homology
0.5
0.618++0001,
0.625
RA(p 0.01) & SS
0.223+00012)
0.231+0.012]
[0.181
1.0
0.61410.015.
0.218::0.01]
[0.706++00016
0.1
0.770-10.008.
0.727+00004)
10.012
[0.096
Secondary Structure
0.5
0.771-10.000
0.756:10.008.
RA(p 0.01) & SS
0.727+0.004]
0.727+0.004]
[0.716:10.015.
1.0
0.771 0.000.
0.735+
17


Reviews:
[{"review_id": "U-EVXO1i0-U", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper explores the impact of different types of data augmentations for protein sequence data, and does a thorough benchmark analysis on them. The authors used a pre-trained transformer model, fine tuned the model on augmented data using two approaches, namely, contrastive learning and masked token prediction. This finetuned model was evaluated with an added linear layer on a range of tasks.\n\nOverall, I vote for strong acceptance as I am genuinely excited about the paper. My reasons are below-\n\nPros:\n1. There are no known good augmentations for protein sequences. This is probably the first paper that tries a wide range of data augmentations for protein sequences, and tests these augmentations on a wide range of benchmarked tasks.\n2. The design of the data augmentations is creative. Specially, random dictionary and random alanine replacement augmentations seem to improve the downstream performances quite consistently. \n3. The paper has done comprehensive experiments showing most possible configurations of the augmentations and their combination with contrastive learning and mask token prediction. \n\nThe paper is so comprehensive in its experiments in the scope of a conference that I can’t really think about any strong cons. I would just ask the authors if there are any plans to bring these augmentations to a library so that the community can start trying them rather than implementing the augmentations themselves? Thanks for this wonderful paper.\n\n\nUPDATE on 12/7/2020:\nI read the other reviews and the authors’ replies.\nI hold onto my score and recommend the paper to be accepted. Here are some points I would like to highlight.\n\n1. Anonreviewer1 suggests the work is incremental, yet there’s no citation to previous work on protein data augmentation. This is in fact the first major work tackling this issue.\n2. Anonreviewer1 points out the authors did not use a SOTA model. This is true, but the model they are using is little more than 1 year old. Moreover, while I agree with Anonreviewer1 that different models need to be used to see how generalizable the data augmentations are, I don’t see this as a reason to reject a paper that is tackling an orthogonal issue of what modes of protein data augmentations work.\n3. Both anonreviewer1 and anonreviewer3 are pointing out that different augmentations seem to have worked best for different tasks, and hence the results are not strong or useful. But how would we know this if no one had even tried it? The authors found the results as such and that’s an important contribution to the community's knowledge.\n4. I am not seeing the problem of data augmentation done on the validation set that anonreviewer1 is suggesting. The test set results should not be affected in a corrupt way in this case.\n5. I disagree with anonreviewer3’s comment that the augmentations are with unreasonable intuitions. I don’t think that’s an objective reason to reject a paper.\n6. I agree with other reviewers that the contact prediction task should have been evaluated with whatever dataset that was available. Even then, the other 4 tasks in my opinion are enough as a first attempt at this area.\n\nOverall, investigating the protein data augmentation regime is something that is not fancy like building a new model or beating the latest SOTA yet a necessary task for which we should thank the authors. There are of course problems that will indeed need to be addressed in further work but there are also important insights in the paper.\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LYR3uG85w1g", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A suite of data augmentations is presented for improving protein language models.\n\nStrengths\n- Data augmentation and contrastive models are underexplored areas in protein language models. The authors are able to improve upon the TAPE baseline. It would be interesting to see if these data augmentations could be used on recent models (ESM-1 and ProTrans) to achieve a new SOTA.\n\nWeaknesses\n- All evaluation was performed on the TAPE model, which was released >1 year ago. Since then, there have multiple models released that have much better performance (ProTrans from Elnaggar, et al.; ESM-1 from Rives, et al.).\n- Related work is incomplete. For example, it says SimCLR is the current SOTA contrastive learning technique. There is already SimCLR-v2 that should be cited, among others. Even though the authors use the models from Rao, et al. 2019, the authors should also cite Alley, et al. 2019, Heinzinger et al. 2019, and Rives, et al. 2019 which were first to introduce semi-supervised learning for protein language models.\n- The data augmentation is done on the validation set. This means there is no longer a validation set! Their model performs better than the ones presented by Rao, et al. but that could be because additional data was incorporated here. The authors include a baseline where they continued training Rao, et al. on the validation set, but it would be better practice to augment data from the training set.\n- The authors do not evaluate on contact prediction because CASP12 \"has an incomplete test set due to data embargoes.\" However, CASP12 ended 4 years ago and the data is now available. In fact, the authors use CASP12 for secondary structure prediction, so clearly they have access to the data. Furthermore, the authors could consider additional structural sets, such as structural holdout at family level; temporal holdout to CASP12/CASP13; other at the very least a sequence based split.\n- The data shuffling will be predominantly shifted to the right because beta = min(N, alpha + 50).\n\nAdditional\n- In the abstract, the authors refer to fine-tuning semi-supervised protein models. This is not very specific, as the authors actually continue the pre-training procedure with the data augmentations. \n- For the replacement augmentation, why do you randomly replace with a single amino acid instead of rotating between, e.g. all hydroxyl or all cyclic amino acids?\n- Specify that secondary structure is \"3-class\" \n- On your table, you should clearly write the three test sets used for remote homology. \n- The authors may want to see \"Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization\" by Lu, et al. which came out around the time of the ICLR deadline.\n- It would be helpful to summarize on a table which data augmentations worked best for each method\n\nOverall, this is an underexplored area, but is incremental to previous work. If the authors had shown better results, I would have increased my rating. However, the lack of rigor, novelty, nor impressive results means this paper does not meet the high bar for an impactful paper at ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CI2-HMf-Ze", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\n- This paper experimentally investigates the effects of data augmentation for protein deep learning.\n- The authors first pretrained a transformer-based NN without data augmentation; then trained the model using data augmentation; finally, each TAPE task was fine-tuned.\n- For data augmentation strategies, they used the Random dictionary, Random Alanine, Global/Local Shuffling, Sequence Reversion, and Subsampling.\n- In the data augmentation phase, the Masked-token prediction (MT) and the Contrastive Learning (CL) have been studied.\n\n### Strong Points\n\n- I agree that investigating the data augmentation effect is important in protein deep learning because effective strategies are not very straightforward compared to other application domains, such as image processing.\n- Experiments are conducted against wide downstream tasks. The authors found that CL with random replacement would be an effective data augmentation.\n- The paper is well-written and easy to follow.\n\n### Weak Points\n\n- Randomness in the experiments: Are the reported values averaged over several attempts with different random seeds (i.e., initial weight of neural network, or randomness in data augmentation)? The authors have (seemingly) reported standard deviation only by the bootstrap over test data. I think this is important because, for example in Fig.4, the reported values seem to be affected by randomness (i.e., at p=0.1); the authors do not provide a reason for this (so I understood that it comes from randomness).\n- Possibility of the other data augmentation: This paper argues that the random replacement strategies (RD/RA) work well for downstream tasks. However, this direction is not investigated deeply. It would be better if the authors will compare it with other replacement strategies. For example:\n  - What happens if we use the other amino acid mappings described in Section 3.2? The authors reports only the best mapping, i.e., [[A,V], [S,T], [F,Y], [K,R], [C,M], [D,E], [N,Q], [V,I]]; how is the impact when using the other mapping?\n  - What happens if we replace an amino acid with the other one that is uniform-randomly chosen from the other 19 possibilities? (Although the authors do not investigate it due to the lack of biological reasons, the following paper reports that this strategy also improves the accuracy for the secondary structure prediction task; Neural Edit Operations for Biological Sequences, NeurIPS 2018)\n- The insights found and argued in this paper are not very strong. Actually, the authors say \"different downstream tasks benefit from different protein augmentations\" at the conclusion. This is a very well-known fact in the deep-learning community.\n\n\n### Minor Points\n\n- In \"Replacement (Dictionary/Alanine)\" paragraph in Section 3.2, the period \"(A).\" should be removed.\n\n\n### Evaluation\n\nOverall, this paper provides an interesting direction for protein learning, where effective data augmentation strategies are not very straightforward compared to other application domains (e.g., image processing). Although I agree that this is an important topic, I have several concerns as mentioned in the \"Weak Points\" above. Please clarify and address my concerns.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5wetgrdMFQJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors suggest that training self-supervised protein sequence models with string manipulations as data augmentations will lead to better downstream performance.\nThey study several different types of data augmentations, (1) dictionary replacement, (2) global/local random shuffling, (3) sequence reversion and subsampling, and (4) a combination thereof.\nThe authors show promising results, where contrastive training improves benchmark results on 3 of the four tasks proposed in TAPE.\nAdditionally, data augmentations show better results on all tasks, beating the TAPE baseline significantly.\n\nI find it exceptionally interesting that sequence reversal has such a negative impact on pretraining.\nI would never have expected the order of protein sequence has such a large effect on downstream tasks.\nAdditionally, this is a very good line of work to pursue, since contrastive methods are known to work on images, but have not shown to beat MLM on NLP tasks.\nIt is interesting to find out whether they do well on protein language modeling tasks.\n\nHowever, there are a few glaring weaknesses in the paper.\n\n(1)\nThe authors do not even attempt to test on contact prediction, citing a data embargo.\nIn fact, many of the CASP12 structures are available, and are tested on in various papers, including the TAPE baseline, trRosetta, etc.\nI believe contact prediction is a main problem to show gains on out of the 5 TAPE tasks, and without showing it in some form, this paper should not be accepted. (See CASP competitions)\n\n(2)\nThe results are reported to be best across all data augmentations taking the best per task.\nThis seems unfair, as the representations learned should not depend on the downstream tasks.\nSince each augmentation seem to have extremely different effects on each task, I would not say that on the whole data augmentation is actually useful in building a protein representation.\nFor example, checking the best model for stability, RD(0.01), I find that this augmentation only does better than the baseline on fluorescence and RH@family.\n\n(3)\nI have a few worries about the augmentation strategies used.\n- For dictionary replacement, a much more natural choice is to follow probabilities outlined in BLOSUM or PAM substitution matrices.\n  I believe this would be a stronger result than Alanine substitution, though the hand designed replacement scheme is quite interesting.\n- GRS and LRS do not make much sense to me.\n  It's quite unnatural to shuffle amino acids, and it's surprising to me that these representations would do as well as it does on secondary structure.\n  In fact, I don't think GRS should be able to do better than a linear model on just amino acid identities on secondary structure prediction, so either there is a bug here or this is a quirk of the dataset.\n  I also think LRS with a percentage of the sequence length is more interesting, and probably ends up being similar to the MLM loss if you use 15%.\n- In figure 4 - it's not clear why the contrastive models all have a drop in performance at p=0.1. I don't see any reason that this should happen across multiple tasks and augmentation strategies.\n\n\nI also offer some small suggestions for a more clear and better paper:\n- Sec 2, last sentence of paragraph 1 claims protein sequences do not need to preserve contextual meaning.\n  This is simply wrong. \n  They must preserve the ability to fold into useful biological structures, and augmentation strategies like random shuffling remove this property - please remove this sentence.\n- The 'MT' task is more commonly referred to in literature as MLM (Masked Language Modeling), I suggest to use this term instead.\n- Figure 3's color scheme is counter-intuitive and not consistent across tasks.\n  Why is gray the best on two tasks, where as light red is best on remote homology and dark red best on fluorescence?\n  Please use a more intuitive color scheme, where perhaps the bold colors show where the best results are.\n- Please permute the numbers for remote homology so it is ordered (fold, superfamily, family) - this is the natural ordering for this task.\n- Table 7: why are the linear evaluation results now reported by cross entropy rather than classification accuracy as in previous tables?\n\n\nAs a summary, my main criticisms of this paper is (1) not including contact prediction, (2) picking the data augmentation to use for each task, and (3) augmentations with unreasonable intuitions.\n\nEdit: Read responses, bumped score up to a 4, check full response below", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tests out some straightforward data augmentation strategies on the protein inputs to the transformer used in the TAPE paper.  Overall, there is insufficient intellectual merit to warrant publication at ICLR. As a side-note, the quality of the manuscript in terms of scholarliness of presentation was overall lacking.
Decision: Reject