Masked inverse folding with sequence transfer for protein representation learning
Authors: Kevin K Yang
Hugh Yeh
Niccolò Zanichelli
Venue: Submitted to ICLR 2023 (2023)

Abstract: Self-supervised pretraining on protein sequences has led to state-of-the art performance on protein function and fitness prediction. 
However, sequence-only methods ignore the rich information contained in experimental and predicted protein structures.
Meanwhile, inverse folding methods reconstruct a protein's amino-acid sequence given its structure, but do not take advantage of sequences that do not have known structures.
In this study, we train a masked inverse folding protein language model parameterized as a structured graph neural network. 
We then show that using the outputs from a pretrained sequence-only protein masked language model as input to the inverse folding model further improves pretraining perplexity. 
We evaluate both of these models on downstream protein engineering tasks and analyze the effect of using information from experimental or predicted structures on performance. 

Under review as a conference paper at ICLR 2023
MASKED INVERSE FOLDING WITH SEQUENCE TRANS- FER FOR PROTEIN REPRESENTATION LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Self--uper pretraining protein sequences has led to state-of-thh the art perfor- mance on protein function and fitness prediction. However, sequence-oni methods ignore the rich information contained in experimental and predicted protein struc- tures. Meanwhile, inverse folding methods reconstruct protein's 's amino-acid sequence given its structure, but do not take advantage of sequences that do not have known structures. In this study, we train a masked inverse folding protein language model parameterized as structured graph neural network. We then show that using the outputs from a pretrained sequence-onf protein masked language model as input to the inverse folding model further improves pretraining perplexity. We evaluate both of these models on downstream protein engineering tasks and analyze the effect of using information from experimental or predicted structures on performance.
INTRODUCTION
Large pretrained protein language models (MLMs) have advanced the ability of machine-learning methods to predict protein structure, function, and fitness from sequence, especially when labeled training data is sparse. The recent state-of-the-art, inspired by BERT ((bidirectional encoder repre- sentations from transformers) (Devlin al. 2018). uses increasingly-Large transformer (Vaswani eet al., 2017) models to reconstruct masked and mutated protein sequences taken from databases such as UniProt (UniProt Consortium (022). UniRef (Suzek et al., (015), and BFD (Steinegger al. 2019, Steinegger & Söding 2018). Pretrained protein MLMs contain structural information (Rao al. 2019 Rives et al. 2022 Chowdhuur al., (02T), encode evolutionary trajectories (Hie et al. 2022b (2221) are shoo-shot predictors of mutation fitness effects (Meier et al. [202)) improve
out-of-domain generalization on protein engincering datasets (Dallago et al.) 2021) 1). and suggest improved sequences for engineering (Hie et al. 2022a). Protein MLMs are now incorporated into the latest machine-learning methods for detecting signal peptides (Teufel et al. 2021 and predicting intracellular Decalization(Thumul rich information contained in experimental and predicted protein structures, especially as the number
et al.) 2022). However, only training sequences ignores the
of high-quality structures from AlphaFold (Jumper et al. 2021 Varadi et al. 2022 increases.
Meanwhile, inverse folding methods reconstruct a protein's 's amino-acid sequence given its structure. Deep (earning-based inverse folding usually parametrized as a graph neural network (GNN) (In- graham et al. 22019; Strokach et al. 2020; Jin et al., 2021: Jing et al. (2020) or SE(3)--quivarian transtormer (McPartion et al.] 2022) that either reconstructs or autoregressively decodes the amino- acid sequence conditioned on the desired backbone structure. The ability to generate amino acid sequences that fold into desired structure is useful for developing novel therapeutics (Chevalier et al. 2017), biosensors (Quijano-Rbbic et al., 2021), industrial enzymes (Siegel et al. 2010). and targeted small molecules (Lucas Kortemme| 2020). Furthermore, single-chain inverse folding approaches could be coupled with recent sequential assembly based multimer structure prediction techniques (Bryant et al. 2022) for -xeedbbakkbone multimer design.
However, we are primarily interested in using inverse folding pretraining task, with the intuition that incorporating structural information should improve performance on downstream tasks. Fur- thermore, current inverse folding methods must be trained sequences with known or predicted structures, and thus do not take maximal advantage of the large amount of sequences that do not have known structures of the menagerie of pretrained protein ML.ss For example, UniRef50

---

Under review as a conference paper at ICLR 2023
contains 42 million sequences, while the PDB (Rose et al. 2016) currently contains 190 thousand ayperimentally-measseee measured protein structures.
In this study, train Masked Inverse Folding (MIF) protein masked language model (MLM) pa- rameterized as a structured graph neural network (SGNN) (Ingraham et al. 2019p. To our knowledge, this is the first example of combining the MLM task with structure in pretraining task. We then show that using the outputs from a pretrained sequence-only protein MLM as input to MIF further improves pretraining perplexity by leveraging information from sequences without experimental structures. We will refer to this model as Masked Inverse Folding with Sequence Transfer (MIF-ST). Figure compares the previous sequence-only dilated convolutional protein MLM (CARP). MIF, and MIF-ST. This is a novel way of transferring information from unlabeled protein sequences into a model that requires structure. We evaluate MIF and MIF-ST on downstream protein engineering tasks and analyze the effect of experimental and predicted structures on performance. Finally, we comment on the state of pretrained models for protein fitness prediction.
2
MIF AND MIF-ST
CNN
GNN
GNN
distances dPedral angles
distances di d'hedral lng les
CNN
mask coruupt
UniRel50 (42M sequences)
CATH (19K structures and sequences)
(19k structures sequences)
(a) CARP
(b) MIF
(c) MIF-ST.
Figure 1:
Summary of models: (a) the Convolutional Autoencoding Representations of Proteins protein masked language model, (h) the Masked Inverse Folding model, and (c) the Masked Inverse Folding with Sequence Transfer model.
2.1 BACKGROUND
Proteins are chains of amino acids that fold into three-dimensional structures. In masked language modeling pretraining on protein sequences, a model learns reconstruct the original protein sequence from a corrupted version, and then the model likelihoods are used to make zero-shot predictions or the pretrained weights are used starting point for training on downstream task, such as structure or fitness prediction. For example, ESM (Rives et al.) 2021) and CARP (Yagg et al. 2022) use the corruption scheme first described in BERT (Devlin et al. 2018). With vocabulary acids, we start from an amino-acid sequence of length L of amino acids E T < i S L, 15% of positions M are selected uniformly at random. 80% these are changed to a special mask token, 10% are randomly mutated to another amino acid, and the remaining 10% are unchanged to generate Snolsed. The model Iearns to predict the original amino acids:
of amino
( Snoisea) Vi E MI
by minimizing the negative log likelihood at positions E ML.
2.2 MASKED INVERSE FOLDING
While MLM pretraining on protein sequences can encode structural and functional information, adding information about the protein's 's backbone structure improves sequence recovery. A protein's backbone structure consists of the coordinates for each amino-acid residue's 's C, C.,. Cg, and atoms, leaving information about the side chains (which would trivially reveal each residue's amino-acid
2

---

Under review as a conference paper at ICLR 2023
identity). We call the pretraining task of reconstructing corrupted protein sequence conditioned on its backbone structure Masked Inverse Folding, which illustrated in Figure lb] We use the BERT corruption scheme and train the model reconstruct the original amino acids conditioned on the corrupted sequence and the backhone structure:
(8; Suoised, structure) Vi E M
After pretraining, a MIF model can be used to perform any downstream task that a sequence-onjy PMLM can, with the caveat that structure must be provided. Intuitively, cructure-conditiona pretraining and having access to structures for the downstream task should both improve performance. We now discuss masked inverse folding detail.
2.2.1 EMBEDDING BACKBONE STRUCTURE AND SEQUENCE
cP
CR
(j)
4p
Figure 2: The backbone atoms amino-acid residues i and j with their dihedral and planar angles highlighted.
We represent protein backbone structures graphs G (V,E) where each node V is amino acid connected by edges & to its k-nearest amino-acid neighbors in the structure. We set k = 30 throughout. Each node's s structural input features consist of the sine and cosine transformations of its dihedral and planar angles to its nearest neighbors in the primary structure:
V {sin, cos} x {wi,+41, 8,t1, 01, Pe+1, (P.+1) E I0
Note that coordinates symmetric whereas 6 and coordinates asymmetric and depend on residue order, so we encode both the forward and backwards angles in the forward and backward direction, i.e. dh.+1.i, and 01,++1, respectively. iilusteisl the backbone atoms of two residues and shows their dihedral and planar angles. Dihedral angles, planar angles, and residue distances used are unconventional protein definitions and follow trRosetta (Yang al. 2020) conventions.
illustrates
The input edge features for the th residue consist of the dihedral and planar angles and the Euclidean distance ((,)) between the C3 atom of residue and the C3 atoms of its k-nearest neighbors, (i;
E.j {di,}} } + fsin, cos) {ui, Oz.j. B3 Pi.j, Poa) 1x11
where N (i, k) and j / i.
We embed each token in into a vector of size dy:
S embed (s) E 4xd,
(5)
and concatenate it to the structural input node features to arrive at:
V concat (V,S) (=00+d)
(6)
3

---

Under review as a conference paper at ICLR 2023
Finally, embed each edge and node into the model dimension d:
v W, V by RRdd
(7)
ETd W.E + be E BLxxxd
(8)
Throughout, set d = 256 and d2 30.
2.2.2 STRUCTURED GNN
We parametrize MIF bidirectional version of the structured GNN from Ingraham et al. 2019). The node and edge embeddings V and E are passed to standard message-pasee multilayer perceptron aggregation function. The m GNN layer takes input node representations V and edge representations E and outputs V and E Within each layer, we first gather relational information from every neighboring node
GNN with a
RTTh concat
[mm-1 Vm-1 JEN(AY) (EN()))) E kx
(9)
and then compute the incoming messages at each node:
hTTh = Agg (R(f Rd
(10)
We parameterize fmsg as three layer neural network with hidden dimension d and ReLU non- linearities and Agg as a mean over the neighbor dimension. We then compute new node embeddings with another ferd-forward neural network.
V fupdate (V: h!"")
(1I)
The sequence logits are computed as a linear mapping of the final node representations.
2.2.3 DATASETS AND MODEL TRAINING
We trained a 4-layer MIF on the CATH4.2 dataset Sillitoe et al. (2019) using the training, validation, and testing splits from Ingraham et al. (2019), in which there is overlap between CATH topology classifications between data splits. MIF was trained with dynamic batch sizes to maximize GPU utilization with maximum batch size of 6000 tokens or 100 sequences, the Adam optimizer, maximum learning rate of 0.001, and linear warmup over 1000 steps. Models were trained on one Nvidia V100 GPU for approximately one day, until validation perplexity stopped improving.
Previous work (Yang et al.] 2022) trained CARP-640M, dilated convolutional protein masked language model with approximately 640 million parameters trained on UniRef50 that achieves comparable results to the state-of-the-art transformer protein MLM, ESM-1b, which has similar number of parameters and is trained on an earlier release of UniRef50. Importantly, sequences with greater than 30% identity to the CATH test set were removed from TARP-640M's training set in order to obtain fair evaluation on the CATH test set. As shown Table conditioning on the backbone structure drastically improves perplexity and sequence recovery compared to CARP-640M, despite MIF having 20 times fewer parameters and being trained on only the 19 thousand examples in CATH compared the 42 million sequences in UniRef50. Increasing the GNN depth 8 layers does not improve pretraining performance, so we use MIF with layers for all following experiments. For comparison, we also train AM-arrameter GVP (iing et al.) (020)) on the same masked inverse folding task (GVPMIF). The GVP architecture improves pretraining perplexity and recovery, but we find that it does not improve performance on downstream tasks.
2.3 MASKED INVERSE FOLDING WITH SEQUENCE TRANSFER
While conditioning structure improves sequence recovery compared to sequence-only pretraining, we hypothesized that transferring information from sequences for which no structure available should further improve performance the pretraining task. Therefore, transfer sequence
4

---

Under review as a conference paper at ICLR 2023
Table Pretrained models. Parameters the number of parameters trained on CATH4.2. Perplexity and recovery are on the CATH4.2 test set from (Ingraham et al. 2019).
Regime
Model
Parameters
Perplexity
Recovery
sequence only
CARP-640M
640M
7.06
40.5%
sequence & structure
MIF-4
3,4M
4.95
49.9%
MIF-8
6.8M
5.00
46.7%
GVPMIF
3.5M
4.68
51.2%
+sequence transfer -UniRef50 pretraining
MIF-ST
3,4M
4.08
55.6%
MIF-ST
3.4M
5.70
45.4%
information from CARP-640M by directly replacing the sequence embedding in Equation with the outputs from CARP-640M pretrained on UniRef50, as shown in Figure IC] MIF-ST was trained with identical hyperparameters MIF. The pretrained CARP-640M weights were not finetuned during training on CATH4.2. As shown in Table T. sequence transfer improves perplexity and recovery on the CATH4.2 test set over both CARP-640M, and MIF. Using the CARP-640M, architecture with randomly-initialized weights as input to MIF did not improve performance, showing simply increasing model capacity insufficient and that sequence transfer is necessary for the improvement.
3 RELATED WORK
Sequence-only protein language models A large body of work has recently studied application of language models to sequence-only protein generation (Madani al. 2020 2021
Shin et al. 2021 Ferruz et al., 2022 Hesslow et al., 2022) and representation learning. See Wu et al. 2021 for more comprehensive review of de novo protein sequence design with deep generative models. Alley et al. (2019) demonstrate that the internal representation learned by an LSTM-based autoregressive protein language model trained a large protein sequence dataset can be leveraged for a wide variety of downstream tasks, including stability, fluorescence and secondary structure prediction. |Rives et al. (202T) Rao et al. (2019) 2020); EInaggar al. (202T); Brandes al. (202T) improve upon this by using the transformer architecture Vaswani et al. (2017). replacing the autoregressive pretraining task with a bidirectional denoising task, and scaling up the model and dataset sizes. |Rao al. (2021) further extend protein language models by allowing it to attend to multiple sequence alignments. (Yang et al. 2022) replace the transformer attention module with dilated convolutions. MIF and MIF-ST build on this body of work by combining the denoising task with structural conditioning.
Fixed-backbone protein design This is similar in spirit to work fixed-backbone protein design, which involves the design of proteins with given target backbone structure. Outside of deep-learning based methods, researchers use packing algorithms (Dahiyat & Mayo 1997 Street & Mayo 1999
DeGrado et al. 1991 Harbury al., 1998), physics-based energy functions (Alford et al. 2017). or mathh structural motifs to sequence motifs (Zhou et al.) 2020). More recent methods attempt to invert
deep-learning protein structure prediction models (Jendrusch et al. 2021 Moffat al. 2021 2022
Anishchenko et al.) 2021 Norn et al.) 2021 Wang et al. 2021). D. See Ovehinnikov & Huang 2021
for more comprehensive review of exed-backbone protein design approaches.
Our method is most similar to work that directly conditions sequence generation on an encoding
of the backbone structure. [Ingraham et al. (2019) conditions autoregressive sequence model on inter-residue distances and angles. Although Ingraham et al. (2019) focuses on a Structured Transformer, they note that replacing the transformer attention mechanism with simple multilayer perceptron improves performance, and we use this Structured GNN architecture in both MIF and MIF-ST. Jing al. (2020) improves on the results Ingraham al. (2019) by modeling all the backbone coordinates with new geometric vector perceptron (GVP) architecture. McPartion et al. (2022) replaces GVP with an SE(3)-equivariant transformer. [Dauparas et al. (0222) and Shi et al. (2022) further improve performance the inverse folding task by further optimizing the architecture and decoding schemes. MIF-ST uses sequence transfer via a sequence masked language model to improve MIF. Similarly, Strokach et al. (2020) augments experimental structures by assuming that sequence homologs will fold to the same structure, while Hsu et al. (2022) augments GVP with 12 million structures predicted by AlphaFold2. In contrast to these methods, focus is on learning
5

---

Under review as a conference paper at ICLR 2023
sequence from structure pretraining task instead of as the primary design task, and we therefore train a bidirectional denoising model instead of an autoregressive language model.
Representations of protein structure While we condition structure and reconstruct sequence, there are other methods for incorporating protein structural information, such as predicting structure similarity between protein sequences (Bepler & Berger 2019)) corrupting and reconstructing the
structure in addition to the sequence (Mansoor et al.] [021]] Chen et al. (022), encoding surface
features (Townshend et al. 2019), contrastive learning (Zhang al. 2022] Cao et al., (202I),
graph encoder without sequence decoding (Somnath et al. [22]] | Fuchs et al. 2020). LM-GVP uses the same architecture as MIF-ST consisting of a pretrained language model feeding into GNN that encodes backbone structure (Wang et al.) (022). However, in LM-GVP the structure- module is used as finetuned prediction head without any pretraining.
DOWNSTREAM TASKS
We evaluate MIF and MIF-ST downstream tasks relevant protein engineering, including out-of-domain generalization and zero-shot mutation effect prediction.
4.1 OUT-OF-DOMAIN GENER ALIZATION
It is desirable for pretrained protein models to be able to make the types of out-of-domain predictions that often occur in protein engineering campaigns. For example, a protein engineer may want to train a model on single mutants and make predictions for sequences with multiple mutations, or train a model that is accurate for sequences with fitness greater than what is seen in the training set.
We finetune and evaluate on two fitness landscapes:
1. Rma NOD: Wu al. (209)) explore how mutations seven positions of the Rhodothermus marinus (Rma) nitric oxide dioxygenase (NOD) enzyme influences enantioselectivity on the reaction of phenyldimethyl silane with ethyl 2-diazopropanoate. We train on 214 variants with mutations at 4 positions and randomly split the variants with mutations at all 7 positions between 40 validation variants and 312 test variants. This tests the model's ability to generalize to mutations at unseen positions based on a small training set. We use PDB 6WK3 as the structure. Measurements were retrieved from ProtaBank (Wang et al.) 2018).
2. GB1: Wu et al. (2016) performed 4-site combinatorial deep mutational scan protein G domain B1, an immunoglobulin- bindinn protein expressed in Streptococcal bacteria. We use splits from FLIP (Dallago al.) 202I over the same GB1 landscape. These splits test generalization from fewer to more mutations or from lower-fitness sequences to higher-fitness sequences. We use PDB 2GI9 (Franks et al., 2006) as the structure.
We compare MIF and MIF-ST to CARP-640M GVPMLM, and 1b. All large models finetuned end-to-end on a single Nvidia V100 GPU with a 2-layer perceptron as the predictive head until the validation performance stops improving. In addition, we the small CNN from (Yang et al. 2022) and ridge regression as baselines.
As shown in Table 2 no model or pretraining scheme outperforms all others on both MSE and Spearman for the Rma NOD task. For protein engineering tasks, rank ordering is generally more important than minimizing error, so we will primarily compare the Spearmans. However, ridge regression achieves the best Spearman at the cost of a very high MSE. The small CNN is strong baseline, with good performance by both metrics. MIF-ST outperforms CARP-640M and MIF with and without pretraining on Spearman. GVP generally does poorly on this task, while ESM-Ib achieves the best MSE but worse than MIF-ST and the small CNN on Spearman.
Table shows results on the GB1 tasks. All models except GVP consistently benefit from pretraining on the GBI tasks. Combining structure conditioning with sequence transfer seems to help slightly, with the biggest gains coming when the training set is limited to single- and double-mutants. However, for the most challenging I-vs-many split, ridge regression results in the best performance, and for low- vs-high, the small CNN results in the best performance. MIF-ST with random weights consistently converged to a degenerate solution where it predicts the same value for all sequences in the test set for all 3 random seeds for 3 of the 4 splits.

---

Under review as a conference paper at ICLR 2023
Table 2: Out-of-domain prediction of Rma NOD enantioselectivity. deviation over 3 random seeds.
Uncertainties are standard
MSE
Spearman
Model
pretrained
naive
pretrained
naive
CARP-640M
0.14 + 0.03
0.32 + 0.043
0.69 + 0.05
0.70 + 0.03
MIF
0.12 + 0.05
0.19 + 0.007
0.66 0.11
0.66 + 0.09
MIF-ST
0.15 + 0.04
0.19 + 0.004
0.77 + 0.03
0.73 + 0.05
GVPMIF
0.18 0.004
0.16 0.044
0.27 0.67
0.55 0.19
EBM-1b
0.08 = 0.005
0.18 = 0.001
0.74 + 0.01
0.69 + 0.05
small CNN ridge
0.12 0.02
0.76 + 0.009
1.17 + 0.01
0.80 = 0.005
In general, pretraining usually helps, as does adding structure when comparing MIF and MIF-ST to CARP-640M, and adding sequence transfer when comparing MIF-ST and CARP-640M. However, different tasks, even those using the same underlying protein fitness landscape, are not best predicted by the same models, and the baseline models compare favorably on many tasks. Furthermore, there is no correlation between pretraining performance and out-of-domain performance, even when adding structure sequence transfer. This suggests a mismatch between the masked language model pretraining task and the sort of OOD performance desired for protein engineering.
Table 3: Performance the FLIP GB1 tasks. Uncertainties standard deviation over random seeds. Values for CARP-640M, the small CNN, and ridge are taken from Yang et al. 2022).
Spearman P
Model
I-vs-many
2-vs-many
3-vs-many
low-vs-high
pretrained
CARP-640M
0.19 + 0.26
0.73 0.03
0.87 + 0.004
0.43 + 0.04
MIF
0.10
0.71
0.88 0.004
0.38 + 0.06
MIF-ST
0.22 0.03
0.74
0
0.43 0.01
ESM-
0.11 0.11
0.67 = 0.07
0.18
0.42 0.09
GVPMIF
0.005 0.06
0.66 + 0.04
0.87 +
0.44 + 0.07
naive
CARP-640M
0.11+00.07
0.38-L0.26
0.68
0.23++0.26
MIF
0.03+
0.05 NaN
0.23-0.02
0.17+0.11
MIF-ST
NaN
NaN
0.18
ESM-Ib
0.05 + 0.28
0.133
0.10 - 0.13
-0,04 0.09
GVPMIF
0.17 + 0.09
0.45 + 0.03
0.83 + 0.01
0.25 + 0.10
baselines
small CNN ridge
0.15
0.39+0004
0.8140
0.47
0.28
0.59
0.76
0.34
4.2 ZERO- SHOT MUTATION EFFECT PREDICTION
Large language models can also predict experimental measurements of protein function without further training on sequence- fitness measurements or of vvolutionarily-relatee sequences Hie
et al.
2022by Meier et al. (2021). Table reports results on five datasets, which are described in A.
We score sequences by masking every mutated position and computing the log odds ratio between the mutated and wild-type residues at each mutated position, assuming an additive model when sequence contains multiple mutations (the "masked marginal" method from Meier et al. (021 ) except for stability, where use the pscudolikelihood. Where possible, compare to ESM-Iv, which is a transformer masked language model trained on UniRef90, Structured Transformer, the model from McPartlon et al. (2022), GVP, and GVP-AF2. The ESM- 1x values for DeepSequence are taken from Meier et al. (2021); the ESM-1x values for RBD are taken from Hsu et al. (2022). We compute values for ESM-Iv on MSP, stability, and GB1 using only the second
7

---

Under review as a conference paper at ICLR 2023
Table 4: Zero-shot effect prediction. We report average Spearman correlation for DeepSequence, Spearman correlation for RBD and GB1, AUROC for MSP, and Pearson correlation for stability.
Task
Regime
Model
DeepSequence
MSP
RBD
stability
GB1
sequence
CARP-640M
0.493
0.53
-0.05
0.28
-0.08
ESM-1b
0.457
0.55
0.02
0.26
-0.04
ESM-Iv
0.508
0.54
0.03
0.22
-0.05
+structure
Structured Transformer
0.37
GVP
0.71
0.60
0,42
GVP+AF2
0.71
0.69
0,48
McPartlon et al. (2022
0.50
MIF
0.402
0.71
0.59
0.45
0.24
MIF-ST
0.509
0.64
0.55
0.47
0.23
model, not the full ensemble of five independent models. Values for GVP and GVP-AF2 are both taken from Hsu et al. (2022); we take the best reported value for each task across several tested model variants. Note that this GVP is trained on a different dataset and task than our GVPMLM model.
For all tasks except DeepSequence, MIF is better than sequence-only methods, and on DeepSequence, adding sequence transfer improves performance above that of the sequence-onle -ony methods. Within DeepSequence, MIF-ST beats CARP-6400 on 22 out of 41 datasets and MIF on 37 out of 41 datasets. [T results for each of the DeepSequence datasets. On the other tasks, MIF and MIF-ST achieve similar results, with sequence- transfer not consistently improving zero-shot performance despite improving pretraining performance. We suspect this is because fitness is unidentifiable from observational sequence data alone (Weinstein et al. 2022), and therefore improved density estimation does not necessarily lead to improved zero-shot fitness predictions. Table A2 shows that
MIF outperforms CARP-640M. 640M on all ten folds tested in the stability dataset, and MIF-ST outperforms MIF six out of ten folds. On both MSP and stability, MIF and MIF-ST comparable other inverse folding methods, but GVP+AF2 is the clear winner on RBD.
Table 5: AlphaFold structures vs PDB structures.
Model
Spearman
MIF
MIF-ST
PDB
AF2
PDB
AF2
DeepSequence
0,358
0.407
0.488
0,509
GB1
0.202
0.24
0.199
0.23
RBD
0.59
0.18
0.55
0.18
For DeepSequence, GBI, and RBD, we also compared the effect of using PDB or AlphaFold structures, as shown in T Surprisingly AlphaFold structures lead to better predictions for both GB1 and DeepSequence. (We were only able find PDB structures for 38 of the 41 DeepSequence datasets, so the results in Table listed in Table AT) As shown Figure 3al the AlphaFold strueture for GB1 nearly identical to its PDB structure. It is unclear why AlphaFold structures lead to better zero-shot predictions in these cases. However, using an AlphaFold-m multimeer prediction for RBD greatly degrades performance. Upon examining the structures, this is not surprising, AlphaFold places the RBD the wrong side of ACE2, as shown in Figure 3b
sightly from those in Table The PDB structures used are
Table A3 shows zero-shot performance the GB1 dataset separated by number of mutations using both PDB and AlphaFold structures. Without structural information, CARP-640M performs poorly for even single mutants. and no correlation all for triple and quadruple mutants. Adding structure allows MIF and MIF-ST to make much better predictions at all mutation levels, but the accuracy nevertheless falls very quickly the number of mutations increases.
8

---

Under review as a conference paper at ICLR 2023
(a) GB1
(b) RBD
Figure 3: Comparisons of PDB (blue)and AlphaFold (red) structures for GB1 (PDB:2G19) and SARS-Cov-2 RBD bound to human ACE2 (PDB:6M0J, orange & grey).
5 CONCLUSIONS
Protein structure is richer source of information than protein sequence, but the largest protein sequence databases contain billions of sequences, while the number of experimental structures is currently in the hundreds of thousands. In this work, we investigate masked inverse folding on 19 thousand structures and sequences pretrained model for variety of downstream protein engineering tasks. We then extend MIF by transferring information from model trained on tens of millions of protein sequences, improving pretraining perplexity and performance some downstream tasks. High-quality predicted structures from AlphaFold often improve zero-shot performance over experimental structures. However, improving pretraining perplexity does not always lead to better downstream performance, and no model consistently outperforms the others on out-of-domain prediction tasks. We suspect that this is due to mismatch between the masked language model pretraining task and out-of-domain fitness prediction
pretraining task. We observe that MIF is an effective
Limitations However, the MIF and MIF-ST pretraining schemes have several important limitations. First, they require structures as input during downstream tasks. This is ameliorated by the ability to predict high-quality structures for most protein sequences. In this work, we use single structure for each protein and its variants: we may be able to improve results by predicting structures for all variants. However, this would be computationally expensive for large datasets, and it is currently unclear how good AlphaFold is at predicting the effects of single mutations (Pak et al.) [022)) Some datasets, such as the the FLIP Meltome landscape, contain many unrelated sequences; collating or predicting structures for each sequence would be significant endeavor. Furthermore, because the structure held constant during pretraining, is unclear how to deal with insertions and deletions in downstream tasks. For example, this prevented us from evaluating on the FLIP AAV landscape.
Future work The related work section suggests improvements that should be composable with MIF and MIF-ST. Using more advanced GVP or SE(3) transsfrano architecture instead of Structured GNN as the base model would likely improve pretraining performance, as would augmenting with AlphaFold structures or adding noise to the input structures. Another obvious extension is to train with an autoregressive or span-masking loss, which should be more amenable to generation tasks, better handle insertions and deletions, and may generalize better complexes.
Potential negative societal impacts Machine learning on molecular data generally entails fewer societal risks than work on language, images, medical, or human data. Pretraining data comes from large, curated protein databases that compile results from the scientific literature, with no privacy concerns. However, large pretrained models incur significant energy and monetary costs to train. CARP- 640M and ESM are trained on hundreds of V100s for weeks at time, contributing to greenhouse gas emissions and putting retraining out of the reach of most academic labs.
Outlook Most work in protein pretraining has used methods borrowed from natural language processing on amino-acid sequences. However, leveraging information from structure, annotations, and even free text should improve performance. We hope that MIF and MIF-ST will lead to more investigations of multimodal protein pretraining tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, Douglas Renfrew, Vikram Mulligan, Kalli Kappel. et al. The Rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):3031-300 2017.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-basee deep representation learning. Nat. Methods, 16(12)1315-1322 December 2019.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushhoo Bafna, Christoffer Norn, Alex Kang, Asim K Bera, et al. De novo protein design by deep network hallucination. Nature, 600(7889::577-552, 2021.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: A universal deep-learning model of protein sequence and function. bioRxiv, 2021.
Patrick Bryant, Gabriele Pozzati, Wensi Zhu, Aditi Shenoy, Petras Kundrotas, and Arne Elofsson. Predicting the structure of large protein complexes using alphafold and sequential assembly. bioRxiv, 2022.
Yue Cao, Payel Das. Vijil Chenthamarakshan, Pin-Yu Chen, Igor V. Melnyk, and Yang Shen. Fold2Seq: A joint sequence( (d)-fold(3d) embedding- based generative model for protein design. Proceedings of machine learning research, 139:1261-1271, 2021.
Can Chen, Jingho Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. ArXiv, abs/2204.04213, 2022.
Aaron Chevalier, Daniel-Adriant Silva, Gabriel J. Rocklin, Derrick R. Hicks, Renan Vergara, Patience Murapa, Steffen M. Bernard, Lu Zhang, Kwok ho Lam, Guorui Yao, Christopher D. Bahl, Shin- Ichiro Miyashita, Inna Goreshnik, James T. Fuller, Merika Treants Koday, Cody M Jenkins, Tom Colvin, Lauren P. Carter, Alan J Bohn, Cassie M. Bryan, Daniel Alejandro Fernández- Velasco, Lance Stewart, Min Dong, Xuhui Huang, Rongsheng Jin, Ian Wilson, Deborah Heydenburg Fuller, and David Baker. Massively parallel de novo protein design for targeted therapeutics. Nature, 550:74-79, 2017.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Charlotte Rochereau, George M. Church, Peter K. Sorger, and Mohammed AlQuraishi. Single-sequence protein structure prediction using language models from deep learning. bioRxiv, 2021.
Bassil Dahiyat and Stephen I. Mayo. De novo protein design: fully automated sequence selection. Science, 278(5335):82-88) 87, 1997.
Christian Dallago, Jody Mou, Kadina Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinMPNN. Science, 778(6615):49-555 2022.
based
William F DeGrado, Daniel P Raleigh, and Tracey Handel. De novo protein design: what are we learning? Current Opinion in Structural Biology, 1(6):984-99,, 1991.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards cracking the language of life's s code through Self-Supervised learning. May 2021.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. A deep unsupervised language model for protein design. bioRxiv, 2022.
W Trent Franks, Benjamin Wylie, Sara Stellfox, and Chad M Rienstra. Backbone conformational constraints in microcrystalline U-15N-labeled protein by 3D dipolar-shift solid-state NMR spectroscopy. Journal of the American Chemical Society, 128(10):31543))
2006.
Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. trans-rranffoormrss: 3d ooto-translation equivariant attention networks. ArXiv, 0at/20006115000 2020.
Pehr B Harbury, Joseph Plecs, Bruce Tidor, Tom Alber, and Peter S Kim. High- protein design with backbone freedom. Science, 282(5393) 1998.
Daniel Hesslow, N. ed. Zanichelli, Pascal Notin, Iacopo Poli, and Debora S. Marks. RITA: a study on scaling up generative protein sequence models. 2022.
Brian Hie, Ellen D Zhong, Bonnic Berger, and Bryan Bryson. Learning the language of viral evolution and escape. Science, 371(6526):284-2885 January 2021.
Brian L Hie, Duo Xu, Varun R Shanker, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, and Peter S Kim. Efficient evolution of human antibodies from general protein language models and sequence information alone. bioRxiv, 2022a.
Brian L., Hie, Kevin K. Yang, and Peter Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 2022b. ISSN 24054712. doi: 2020066
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in Neural Information Processing Systems, pp. 15794-15805, 2019.
Justina Jankauskaite, Brian Jiménez-García Justas Dapkūnas, Juan Fernández-Recio. and Iain H. Moal. Skempi 2.0: An updated benchmark of changes in protein-protein binding energy, kinetics and thermodynamics upon mutation. bioRxiv, 2018.
Michael Jendrusch, Jan O. Korbel, and S. Kashif Sadiq. Alphadesign: A de novo protein design framework based on alphafold. bioRxiv, 2021.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structum co-design. arXiv preprint arXiv:2 10. 04624, 2021.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvumakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-5888 2021.
James E Lucas and Tanja Kortemme. New computational protein design methods for de novo small molecule binding sites. PLoS Computational Biology. 16, 2020.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. ProGen: Language modeling for protein generation. arXiv, 2020.
11

---

Under review as a conference paper at ICLR 2023
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, 2021.
Sanaa Mansoor, Minkyung Back, Umesh Madan, and Eric Horvitz. Toward more general embeddings for protein design: Harnessing joint representations of sequence and structure. bioRxiv, 2021.
Matt McPartlon, Ben Lai, and Jinbo Xu, A deep SE (3)equivariann model for learning inverse protein folding. bioRxiv, 2022.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations protein function. In M. Ranzato, A. Beygelzimer, K. Nguyen. P.S. Liang. J.W. Vaughan, and Y. Dauphin (eds.), Advances in Neural Information Processing Systems 34, 2021.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold making protein folding accessible to all. 2021.
Lewis Moffat, Joe G. Greener, and David T. Jones. Using alphafold for rapid and accurate fixed backbone protein design. bioRxiv, 2021.
Lewis Moffat, Shaun M. Kandathil, and David T. Jones. Design the dark: Learning deep generative models for de novo protein design. bioRxiv, 2022.
Christoffer Norn, Basile IM Wicky, David Juergens, Sirui Liu, David Kim, Doug Tischer, Brian Koepnick, Ivan Anishchenko, Foldit Players, David Baker, et al. Protein sequence design by conformational landscape optimization. Proceedings of the National Academy of Sciences, 118 (11):e2017228118 2021.
Sergey Ovchinnikov and Po-Ssu Huang. Structure-based protein design with deep learning. Current opinion in chemical biology, 65:136-144, 2021.
Marina A Pak, Karina A Markhieva, Mariia S Novikova, Dmitry S Petrov, Ily S Vorobyev, Ekate- rina S Maksimova, Fyodor A Kondrashov, and Dmitry N Ivankov. Using alphaFold to predict the impact of single mutations on protein stability and function. BioRxiv, 2021.
Alfredo Quijano-Rubio, Hsien-Wei Yeh, Jooyoung Park, Hansol Lee, Robert A. Langan, Scott E, Boyken, Marc J. Lajoie, Longxing Cao, Cameron M. Chow, Marcos C. Miranda, Jimin Wi, Hyo Jeong Hong. Lance J. Stewart, Byung-Ha Oh, and David Baker. De novo design of modular and tunable protein biosensors. Nature, 2021.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Picter Abbeel, and Yun Song. Evaluating protein transfer learning with TAPE, In Advances in Neural Information Processing Systems, pp. 9686 1898,9698,,.8 2019.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models unsupervised structure learners. In International Conference Learning Representations, 2020.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8844 8856. PMLR, 18-24 Jul 2021. URL https: httpp:://ppcceeednntt..
ml wtw.peesswv3//// 139 r/oll/aalll html
Adam J Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods. 15(10):816-3) 822, 2018.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sei. U. S. A., 118(15), April 2021.
12

---

Under review as a conference paper at ICLR 2023
Gabriel Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, et al. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, 357 (6347))168-117 18-175, 2017. doi: 10.1 1. 22////innccccecoeeecc aan0693. 2a//ciennee..an6693.
Monica Rosas-Lemus, George Minasov, Ludmilla Shuvalova, Nicole L Inniss, Olga Kiryukhina, Joseph Brunzelle, and Karla Satchell. High-resolution structures of the SARS-CoV-2 2' -O- methyltransferase reveal strategies for structure-based inhibitor design. Science signaling, 13(651); eabel 202, 2020.
Peter W Rose, Andreas Prlić, Ali Altunkaya, Chunxiao Bi, Anthony R Bradley, Cole H Christie, Luigi Di Costanzo, Jose M Duarte, Shuchismita Dutta, Zukang Feng. Rachel K Green, David Goodsell, Brian Hudson, Tara Kalro, Robert Lowe, Ezra Peisach, Christopher Randle, Alexander Rose, Chenghua Shao, Yi-Ping Tao, Yana Valasatava, Maria Voigt, John Westbrook, Jesse Woo, Huangwang Yang, Jasmine Y Young, Christine Zardecki, Helen M Berman, and Stephen K Burley, The RCSB protein data bank: integrative view of protein, gene and 3D structural information.
Nucleic Acids Research, 2016. doi: 10.1093/mar/gkw955.
Chence Shi, Chuanrui Wang, Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.082 2022.
Jung-Eun Shin, Adam Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Justin B. Siegel, Alexandre Zanghellini, Helena M. Lovick, Gert Kiss, Abigail R. Lambert, Jennifer L. St.Clair, Jasmine L. Gallaher, Donald Hilvert, Michael H. Gelb, Barry L. Stoddard, Kendall N. Houk, Forrest E. Michael, and David Baker. Computational design of an enzyme catalyst for stereoselective bimolecular diels-alder reaction. Science, 329:309 - 313, 2010.
Ian Sillitoe, Natalie Dawson, and Christine Orengo. CATH protein domain classification (version 4.2). 4 2019. doi: 10.5552204//9933330.l.. URL https://: ucl. ac. uk/articles. dataset/ CATH protein domain classification version 4 2 /7337330
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning proteins. In NeurlPS, 2021.
Tyler N. Starr, Allison Greaney, Sarah K. Hilton, Daniel Ellis, Katharine H. D. Crawford, Adam Dingens, Mary Jane Navarro, John E. Bowen, M. Alejandra Tortorici, Alexandra C. Walls, Neil P. King, David Veesler, and Jesse Bloom. Deep mutational scanning of sars-cov-2 receptor binding domain reveals constraints folding and ace2 binding. Cell, 182: 1295 1310.e20, 2020.
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 911):1-8, 2018.
Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nature methods, 16(7):603 2019.
Arthur G Street and Stephen Mayo. Computational protein design. Structure, ((5):R05-RII) 1999.
Alexey Strokach, David Becerra, Carles Corbi-Verge. Albert Perez-Riba, and Philip M Kim. Fast and flexible design of novel proteins using graph neural networks. bioRxiv, pp. 868935, 2020.
Baris E Suzek, Yuqi Wang. Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31669922-932,, 2015.
Felix Teufel, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Magnús Halldór Gíslason, Silas Irby Pihl, Konstantinos D Tsirigos, Ole Winther, Søren Brunak, Gunnar von Heijne, and Henrik Nielsen. SignalP 6.0 achieves signal peptide prediction across all types using protein language models. bioRxiv. 2021.
13

---

Under review as a conference paper at ICLR 2023
Vineet Thumuluri, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Henrik Nielsen, and Ole Winther. DeepLoe 2.0: multi-label subcellular localization prediction using protein language models. Nucleic Acids Research, 2022.
Raphael L. Townshend, Rishi Bedi, Patricia Suriana, and Ron O. Dror. End-to-end learning on 3d protein structure for interface prediction. In NeurlPS, 2019.
Raphael J. L. Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander S Powers, Yianni Laloudakis, Sidhika Balachandar, Brandon M. Anderson, Stephan Eismann, Risi Kondor, Russ B. Altman, and Ron O. Dror. ATOM3D: Tasks molecules in three dimensions. ArXiv, abs/2012.04035, 2021.
UniProt Consortium. UniProt: the universal protein knowledgebase in 2021. Nucleic Acids Res., 49 (D1): -D89, January 2021.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe. Gemma Wood, Agata Laydon, et al. AlphaFold protein structure database: Massively expanding the structural coverage of protein-sequence space with 2022.
high-accuracy models. Nucleic acids research, (O(D))
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
Connie Y Wang, Paul M Chang, Marie I. Ary, Benjamin D Allen, Roberto A Chica, Stephen I. Mayo, and Barry D Olafson. Protabank: A repository for protein design and engineering data. Protein doi: 10.1002/pro.3406
Science, 113-11113-1124, 2018.
Jue Wang, Sidney Lisanza, David Juergens, Doug Tischer, Ivan Anishchenko, Minkyung Back, Joseph I. Watson, Jung Ho Chun, Lukas F Milles, Justas Dauparas, et al. Deep learning methods for designing proteins scaffolding functional sites. bioRxiv, 2021.
Zichen Wang, Steven A. Combs, Ryan Brand, Miguel Calvo Rebollar, Panpan Xu. George Price, Nataliya Golovach, Emmanuel Oluwatobi Salawu, Colby Wise, Sri Priya Ponnapalli, and Peter M. Clark. Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific Reports, 12, 2022.
Eli N Weinstein, Alan N Amin, Jonathan Frazer, and Debora Marks. Non-identifiabiiity and the blessings of misspecification in models of molecular fitness and phylogeny. bioRxiv, 2022.
Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes is facilitated by indirect paths. Elife. 5:e16965, 2016. doi: 10.7554/eLife.16966
Zachary Wu, S. B. Jennifer Kan, Russell D. Lewis, Bruce Wittmann, and Frances H. Arnold. Machine learning-assiste directed protein evolution with combinatorial libraries. Proceedings of the National Academy of Sciences USA, 2019.
Zachary Wu, Kadina E Johnston, Frances H Arnold, and Kevin K Yang. Protein sequence design with deep generative models. Current opinion in chemical biology, 65:18 27, 2021.
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 17933):1466113 2020.
Kevin K Yang, Alex X Lu, and Nicolo K Fusi. Convolutions are competitive with transformers for protein sequence pretraining. bioRriv, 2022.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurélie C. Lozano, Payel Das, and Jian Tang. Protein structure representation learning by geometric pretraining. 2022.
Jianfu Zhou, Alexandra E Panaitiu, and Gevorg Grigoryan. A general-purpose protein design frame- work based on mining sequence-structin structure relationships in known protein structures. Proceedings of the National Academy of Sciences, - 2020.
14

---

Under review as a conference paper at ICLR 2023
A ZERO-SHOT FITNESS PREDICTIONS
A.1 DATASET
1. DeepSequence: 41 deep mutational scanning datasets originally compiled by RResselman et al. (2018). These datasets each measure the effects of thousands of mutations or combina- tions of mutations to a parent sequence. Models are evaluated on their average Spearman p across the datasets. We ensemble results from five Alphafold structures obtained using ColabFold (Mirdita et al. 202T) default parameters for each wild-type protein.
2. Mutation Stability Prediction (MSP): Townshend et al. (202T) curate single mutants from SKEMPI (Jankauskaite et al. 2018), which measures whether mutant proteins display better binding than wildtype. Models are evaluated using area under the receiver operating curve (AURÓC) on the 893 positive and 3255 negative examples in the test set.
3. Stability: Rocklin et al. (2017) used deep mutational scans to measure protease stability for point mutants of set of de novo designed miniproteins with 10 different folds. Models are evaluated on their average Pearson correlation across the folds.
4. RBD:
Starr et al. 2020) performed a deep mutational scan to measure how all amino-acid mutations the receptor binding domain (RBD) of the SARS-CoV-2 spike protein affected its affinity to the human ACE2 receptor. Models are thus tasked to predict binding affinity for all 1311 mutant RBD sequences. We use PDB 6MOJ (Rosas-Lemus et al. 2020) as the native structure.
5. GB1: Wu et al. (2016) performed 4-site combinatorial deep mutational scan on protein G domain B an gumunoglobuii--indiin protein expressed in Streptococcal bacteria. Models are evaluated on Spearman across all measurements using an AlphaFold structure of the 56-amino acid GB1 domain.
A.2 ADDITIONAL RESULTS
model
MIF-ST
MIF
CARU--40O
ESM lb
ESM Iv
0.6
0.5
Spearman
0.4
0.3
0.2
0.1
0.0
BLAT ECout angganathan20015 5
PRUe YEASY fielt Perpaaroo idouth les
AMIE PIRAE Pideeeadd
MaA KLEPN Miterioan7001
DLGR Kansanatiio Iepaanationss 222122
usi MMMN Ro17
to
GRL4 YEAST Pennureee 2014
MTH HABP Pot_ HV C.. Heturauz Sunz0114
BLAY CCOL1 Tonallon201 2012
HaE HAEXEESTAII dESTTALLLEEE CDD DDwfli2015
Forth findiz01 senlee
POLO 426 boie20016
TPMI NURAA aoneer211
IF Ecou ristony
BRCAL NUMAR FING
Lme448 eOUUE bedk anv Doomz0le Br520 TIM TIEMA bu
yarl, NUMAN picce HURMAN BRCT
RuRAN, cowlerza 2012
RL TPE) RRMAN 20th22112
YEASY 2olnn2012
RL401 TEAST 2015
CAURi HilAn Reth2012
281 YEAST Chrmaarzin
MKTO HiRAN iptarresteen
TIM
BC200 HURAH
HESPg YEASY
BLAT HO FU Soonn
SCOLY, pai
STA50 (15522011 25155
MIEN RRACH RAMAH
Baviss (PPST Whi
PR Fu
BLAY ECOLE
H57 e5
Figure A1: shoo prediction on deep mutational scanning datasets in DeepSequence.
15

---

Under review as a conference paper at ICLR 2023
Table Al: Mappings from DeepSequence alignments PDB structures.
Alignment
PDB
chain idx
aln
aln
pdb s
pdb_g
AMIE PSEAE b.
2uxy
0
341
0
341
B3VI55 LIPSTSTABLE b_
4zfv
1
439
0
430
B3VI55 LIPST __b0
4zfv
1
439
0
430
BG STRSQ_ 1 b0
Ignx
0
14
478
0
464
BLAT_ECOLX.2. 1_b0
1sOw
0
263
0
263
BRCAI LAAAILHUMNNLLLL bO
1jm7
0
103
0
103
BRCAI SRCAI_HUMANJRRH
4u4a
0
21
235
0
214
CALMI HUMAN 1_b0
3sjq
148
0
147
DLG4 RAT 2_b0
2xkx
101
296
397
DYR ECOLL I_b0
5uio
159
160
F7YBW7 MESOW 1_b0
TYBW7_MESOW
5ceg
1
103
0
103
FYN_ HUMAN I_b0 b0
3uf4
0
0
SAL4_YEAST b0
3coq
0
75
0
68
HG_FLU
6mya
4
16
508
0
493
HIS7 YEAST b0
6ezm
0
219
0
217
HSP82 YEAST 1 b0
2cg9
0
216
0
215
IF1 FFL_FCOLLL b0
lah9
72
71
KKA2_KLEPN, 1_b0
nd4
0
264
0
255
MKOI HUMAN 1 b0
7opm
360
2
362
MTH3 HAEAESTABILIZED
3ubt
0
328
0
328
P84126 THETH 60
Ive4
254
254
PABP YEAST 1_
6r5k
96
77
173
AA.FLU b0
7nj7
716
O
716
3fqq
0
32
114
83
POI_HVINS-CA_11 b0
6wap
0
231
PTEN HUMAN 1 b0
7jvx
0
351
0
345
PYP HALHA
4bbv
0
125
0
125
RASH HUMAN b0
4q21
169
0
169
RL401 YEAST 1 b.
6zqh
1
76
0
76
SUMO1 LAOOI_HHMANLLLA I_b0
la5r
101
2
103
[IU__HMAN__LA
3s4y
0
15
242
0
227
TPMT SANT__UMAN_]
2h11
16
245
0
229
TRPC SULSO 1 bO
ligs
0
248
0
247
TRPC THEMA 1 b0
li4n
252
251
TRY2 RAA_RAA_R
3fp6
0
O
223
0
233
UBC9 HUMAN_
2xwu
0
158
0
158
UBE4B MOUSE_L b_
2kre
0
104
4
100
YAPI SEPI_HUAA__LE
2ltw
0
0
36
Table A2: Zero-shot performance on novo miniproteins from Rocklin et al. 2017). broken out by
fold.
Fold
Pearson correlation
CARP-640M
MIF
MIF-ST
HHH 138
0.41
0.46
0.52
HHH
0.36
0.45
0.48
HEEH
0.23
0.37
0.41
HEEH,26
0.21
0.26
0.25
HEEHa23
0.22
0.53
0.56
HEEH,79
0.50
0.55
0.62
EEHEE1495
0.10
0.36
0.32
EEHEE
0.41
0.65
0.68
BEHEE1710
0.22
0.60
0.60
EEHEE
0.03
0.26
0.24
16

---

Under review as a conference paper at ICLR 2023
Table A3: Zero-shot Spearman rank correlation GB1 broken out by number of mutations.
PDB
AF2
Model
1
2
3
4
1
3
4
CARP-640M
0.19
0.12
-0.01
-0.01
0.19
0.12
-0.01
-0.01
MIF
0.83
0.64
0.42
0.11
0.84
0.66
0.44
0.15
MIF-ST
0.76
0.62
0.41
0.11
0.83
0.67
0.44
0.15
17


Reviews:
[{"review_id": "wCqONlaJo9", "reviewer": "Reviewer_vWP7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using protein structure and sequence information as a pretraining task for representation learning of proteins. They then apply these methods to two downstream applications: (1) zero-shot prediction and (2) OOD prediction.\n", "summary_of_the_review": "I think the paper has good work that pushes the research field forward for protein engineering applications. However, I think the focus on zero-shot prediction is mostly a distraction from the more useful aspects of combining sequence and structure information (as opposed to just structure and just sequence). More thorough experimentation and benchmarking along with a rework of the exposition to highlight this fact would greatly improve the quality of this paper.", "strength_and_weaknesses": "Strengths:\n- Combining sequence and structural information and pretraining hasn't been done before which is definitely a natural progression for the progress in the protein + ML space in recent years.\n- Finetuning of structural/inverse folding models hasn't been studied as much compared to sequence-based finetuning.\n\nWeaknesses:\n- The framing and experiments of the paper could be improved to highlight the core contributions. Hsu et al (2022) had demonstrated the impact of structure on zero-shot fitness prediction. The takeaway from the section on zero-shot prediction largely recapitulated the results from Hsu et al while demonstrating that adding sequence information atop structure did not seem to impact performance. While this is somewhat interesting, it seems like this should not be such a dominant section of the paper. Rather the pretraining and finetuning with structure and sequence for OOD prediction seems much more novel. However, the experiments in this section are much less explored and don't benchmark against simple baselines (CNNs etc) or other methods. Experiments focused on finetuning and pretraining strategies for structural data in combination with sequence data would be interesting but seemed less prioritized than the zero-shot section. Overall, the exposition and experiments seem to highlight the less novel application while providing less thorough experiments that would push the field forward for the OOD prediction task. Getting fine-tuning to work for these LLMs can be challenging so guidelines on how to do this for various representations(sequence-only, structure-only, combined) would be extremely valuable in my opinion.\n- The point about the lack of structure data causing the need for sequence data for pretraining has been somewhat obviated given AlphaFoldDB.\n\nMinor Comments:\n- Citing ProteinMPNN (Daupras et al 2022) seems relevant here even if its not benchmarked against.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iJMHNAFm29", "reviewer": "Reviewer_mxA7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use masked inverse folding to pre-train (1) a protein structure-based encoder (MIF) and (2) a structure-based encoder enhanced by protein language model representations (MIF-ST). (1) MIF encodes the information of protein backbone structure and masked protein sequence into node and edge features, and the model is trained to recover the masked residues based on the encoded information. (2) MIF-ST further improves the performance of sequence recovery by feeding protein language model representations into the structure-based encoder. On zero-shot and out-of-domain protein landscape prediction benchmarks, MIF and MIF-ST achieve competitive empirical performance against previous methods. ", "summary_of_the_review": "To summarize, this paper shows some merits of boosting future research on structure-based pre-training for protein engineering, while more comprehensive performance comparisons and ablation studies are required to make it a more solid contribution. I think it is currently on the border and expect authors' efforts during the response period. ", "strength_and_weaknesses": "Strength:\n1. The proposed methods are well motivated. MIF is proposed to improve sequence-only masked language modeling with backbone structural information, and MIF-ST is proposed to further improve MIF by pre-trained protein sequence representations. \n2. The advantages of pre-trained structure-based encoders are demonstrated on a comprehensive set of protein engineering related tasks, and the ablation study analyzes the effect of using experimental and predicted structures for downstream evaluation. \n\n\nWeakness:\n1. For performance comparison, some important protein structure pre-training baselines are lacked. In a closely related work [a], several structure-based encoders pre-trained by contrastive learning and self-prediction methods are proposed, and the source codes are released. It will be nice to compare with these baseline models on protein engineering tasks. \n2. Some ablation studies are lacked. According to Table 2, ESM-1b and ESM-1v are superior protein language models against CARP-640M. Why not using their representations as inputs of MIF? In this way, we can expect stronger MIF-ST models than the current one based on CARP-640M representations. \n3. Some notations are not consistent across figures and texts. In Figure 2, authors use $\\varphi$ and $\\theta$, while $\\phi$ and $\\psi$ are used in the equations (3) and (4).\n\n\n[a] Zhang, Zuobai, et al. \"Protein representation learning by geometric structure pretraining.\" arXiv preprint arXiv:2203.06125 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hQE29vXn5Uv", "reviewer": "Reviewer_ABwW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "BERT-style pretraining has become popular for protein modeling, since there are very large databases of natural protein sequences available. A separate body of work on inverse folding has explored structure-conditioned generative models of sequences.This paper explores structure-conditioned BERT, where at both train and test time the structure is assumed to be known for any query sequence. The authors demonstrate that conditioning on the structure improves performance on a variety of zero-shot and few-shot protein function prediction tasks.\n", "summary_of_the_review": "I work in this exact research area and am in general supportive of this kind of work at ICLR. It can have large real-world impact and these protein modeling problems are also a great place for cutting-edge methods to be applied. My primary hesitation about this paper, however, is the nature of the experimental results.\n\nI didn't understand why there were few results for ESM in Table 2. Without these, how can you make the claim \"For all tasks except DeepSequence, MIF is better than sequence-only methods?\" There is a good github package for ESM. It shouldn't be too hard to run on these datasets. Can you add these?\n\nAlong these lines, to me the take-away point from table 2 is that structure-conditioned models are good at zero-shot prediction, not that the particular model proposed by this paper provides a breakthrough. GVP, for example, is quite strong. Am I missing something? Where is there convincing evidence that MIF-ST improves over existing work?\n\nI was disappointed by the complete lack of baselines that are not based on large neural networks. What about, for example, profile HMMs or Potts models for the zero-shot experiments? What about a simple linear model or lightweight non-linear model (like a shallow MLP) for the few-shot experiments? Adding all of these to the paper would be a lot of work. Can you provide baselines for at least some of the experiments, though?\n\nThe paper's propose model combines two orthogonal ideas: (1) using a CNN instead of a transformer and (2) conditioning on the structure. (1) appears in current work under review. It's not required for acceptance, but the paper would have been considerably improved if there were experiments that isolate (2) by using a transformer.\n\nI was really surprised by how strong the RBD results are for structured-conditioned models. Do you have any intuitions for why conditioning on the structure makes such a big difference here?\n", "strength_and_weaknesses": "=strengths=\nWell written\nExperiments compare to recent work\nExperiments demonstrate that conditioning on structure can provide good performance improvements (e.g., on zero-shot modeling of the covid RBD).\n\n=weakenesses=\nThe experiments don't clearly demonstrate a regime where the paper's proposed method improves over existing work. The experiments demonstrate that structure-conditioned models are good at zero-shot prediction, but existing structure-conditioned models can do this well too.\n\nExperiments don't compare to any non-neural-network baselines\n\nThe paper is frank about the fact that there are lots of opportunities for follow-up work, such as by pretraining at scale on alphafold-predicted structures. The paper is good enough for acceptance in its current form, but it wouldn't have been a terrible idea to slow things down, run some of the proposed experiments, and submit when the full setup is available.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XcQcQdoGd-", "reviewer": "Reviewer_u96a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For the first time, this study proposes to apply masked language modeling (MLM) to the inverse folding problem of proteins. The protein language model is parameterized as a bidirectional version of the structured GNN (Ingraham et al, 2019). The authors also showed that using the outputs from a pretrained sequence-only protein MLM as input can further improve pretraining perplexity. The proposed models are evaluated on downstream protein engineering tasks, including zero-shot mutation effect prediction and out-of-domain generalization.", "summary_of_the_review": "I lean upon a weak reject due to the unsatisfactory performance compared to GVP on MSP, RBD, and stability. Moreover, ", "strength_and_weaknesses": "$Strengths$:\n\nApplying MLM to the protein inverse folding problem sounds interesting. It is valid to use graph neural networks to incorporate the structure priors from folded proteins. Besides, using the learned amino acid representations from a pretrained sequence-only protein MLM as inputs to the graph neural networks is a resonable and meaningful trial.\n\n$Weaknesses$:\n\n1. Performance. \n\nSince this paper presents a straightforward pre-training methodology that uses masked inverse folding as a pretext task, it is important that the pre-trained model shows superior performance on downstream tasks. However, in Table 2, performance of the proposed MIF-ST is lower than GVP+AF2 on three selected tasks, i.e., MSP, RBD, and stability. Moreover, when compared to GVP only, MIF-SF still produces much worse results on MSP and RBD. Also, explanations behind these failures are missing. \n\nBesides, when evaluated on the out-of-domain generalization tasks, CARP-640M is employed as the only baseline. However, MIF-ST already includes CARP-640M as the feature extractor for amino acids. Thus, the reported comparisons are kind of unfair because MIF-ST is expected to achieve better results than CARP-640M. The authors should at least include another recent baseline in Table 2. If there are no baselines available, the authors should clarify the reasons in the manuscript.\n\n2. Some statements are either confusing or inappropriate. \n\nIn the abstract, the author claimed \"inverse folding methods do not take advantage of sequences that do not have known structures.\" After reading the paper, I failed to find how the authors addressed this point besides incorporating the predicted structures from PDB and AF2 (as shown in Table 3). If this is the truth, the above statement should be revised because GVP already includes structures from AF2 as training data. \n\nIn the last paragraph of sec. Introduction, the sentence \"Figure 1 compares our previous sequence-only dilated convolutional protein MLM (CARP), MIF, and MIF-ST\" is not appropriate (from my perspective). The citation of CARP refers to the anonymous submission but the authors did not provide the anonymous draft. As a result, how to implement the CNN model in MIF-ST is missing (well, this is another problem that needs to be addressed in the rebuttal), and I have to search CARP online and find the draft on bioRxiv. I'm not sure if the use of words like \"our previous xxx\" violates author guidelines of ICLR'23 but personally, I recommend using other words instead.  \n\n\nMinor points:\n\n1. Does formula 4 lack a concatenation notation? Otherwise, it should be $\\mathcal{E}_{i,j}$.\n\n2. In Figure 2, it seems that $w$ stands for the angle instead of coordinates.\n\n3. Why do the authors take the best reported value instead of mean value for each task across several tested model variants?\n\n4. Based on Table 3, I assume the authors already tried to use the predicted structures from AlphaFold2? If this is the truth, it is necessary to explain why GVP can utilize the predicted results from AF2 in a better way.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject