HotProtein: A Novel Framework for Protein Thermostability Prediction and Editing
Authors: Tianlong Chen
Chengyue Gong
Daniel Jesus Diaz
Xuxi Chen
Jordan Tyler Wells
qiang liu
Zhangyang Wang
Andrew Ellington
Alex Dimakis
Adam Klivans
Venue: ICLR 2023 poster (2023)
Keywords: Protein Thermostability
Protein Editing
Dataset
Structure-aware Pre-training
Factorized Sparse Tuning

Abstract: The molecular basis of protein thermal stability is only partially understood and has major significance for drug and vaccine discovery.  The lack of datasets and standardized benchmarks considerably limits learning-based discovery methods. We present \texttt{HotProtein}, a large-scale protein dataset with \textit{growth temperature} annotations of thermostability, containing $182$K amino acid sequences and $3$K folded structures from $230$ different species with a wide temperature range $-20^{\circ}\texttt{C}\sim 120^{\circ}\texttt{C}$. Due to functional domain differences and data scarcity within each species, existing methods fail to generalize well on our dataset. We address this problem through a novel learning framework, consisting of ($1$) Protein structure-aware pre-training (SAP) which leverages 3D information to enhance sequence-based pre-training; ($2$) Factorized sparse tuning (FST) that utilizes low-rank and sparse priors as an implicit regularization, together with feature augmentations. Extensive empirical studies demonstrate that our framework improves thermostability prediction compared to other deep learning models. Finally, we introduce a novel editing algorithm to efficiently generate positive amino acid mutations that improve thermostability. Codes are available in https://github.com/VITA-Group/HotProtein.

Published as a conference paper at ICLR 2023
HOTPROTEIN: A NOVEL FRAMEWORK FOR PROTEIN THERMOSTABILITY PREDICTION AND EDITING
Tianlong Chen', Chengyue Gong', Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, Qiang Liu, Zhangyang Wang, Andrew Ellington, Alex Dimakis, Adam Klivans
The University of Texas Austin
ftianl ong. chen, cygong17, danny diaz, xxchen, jordantwe ltstantweellseueeee weteeleeeeeeueeeueeeo {lqiang, at laswanng andy. e llington, dimakis, klivans edu
ABSTRACT
The molecular basis of protein thermal stability is only partially understood and has major significance for drug and vaccine discovery. The lack of datasets and standardized benchmarks considerably limits learning-based discovery methods. We present HotProtein, a large-scale protein dataset with growth tempera- annotations of thermostabilliyy containing 182K amino acid sequences and 3K folded structures from 230 different species with a wide temperature range -20 120%c. Due to functional domain differences and data scarcity within each species, existing methods fail to generalize well on our dataset. We ad- dress this problem through a novel learning framework, consisting of (1) Pro- tein atructure-war pre- pre-training (SAP) which leverages 3D information to en- hance sequence- based pre-training (2) Factorized sparse tuning (FST) that uti- lizes low-rank and sparse priors as an implicit regularization, together with fea- ture augmentations. Extensive empirical studies demonstrate that our frame- work improves thermostability prediction compared to other deep learning mod- els. Finally, we introduce a novel editing algorithm to efficiently generate pos- itive amino acid mutations that improve thermostability. Codes available in bttps://githun com/ SIITA-Group. Grrrgooooorrrteeeeeeee HottPateinn
INTRODUCTION
Proteins are the bio-polymers responsible for ex- ecuting most biological phenomena and, through evolution, have had their sequences optimized carry out specific functions within specific cellu- lar environments. A protein's 's stability is multi- dimensional property that depends on a series of factors (Pucci al. 2017: Cao al. 2019) such as pH, salinity, and temperature (thermostability shown in Figure [ making hard adapt a pro- tein to function outside of its endogenous cellular environment. Protein engineering is the field where natural proteins are mutated to improve their stability exogenous environments and their overall fitness for particular function. In protein engineering, one of the initial goals for most engineering campaigns is to improve the thermal stability of protein (Haki & Rakshit, 2003 Bruins et al. 2001 Frokjaer & Otzen 2005). Thermally stabilized proteins are more robust and therefore enable downstream applications in the food (Kapoor et al. 2017). biofuel (Huang et al. 2020). deter- gent (Von der Osten et al. 1993), chemical (Cho et al. 2015), and pharmaceutical industry (Amara 2013). drug design (De Carvalho 2011 Mora & Telford 2010). and bioremediation of environmen- tal pollutants (Lu et al., [022] Alcalde al. 2006). Thus, to accelerate the engineering of target protein it is critical to understand and accurately predict thermal stability changes of mutations. There has been substantial effort from the community quantitatively understand and model pro- tein thermostability (e.g., Pucci et al. 2017 Cao et al.) 2019 Pucci & Rooman 2014] Li et al.) 2019 Pucci & Rooman 2017; Pouyan et al. 2022). However, the generalizability of them still unsatisfactory, and laborious experimental methods such as directed evolution are often preferred.
Overheat unfolds and deactivates Proteins.
Figure 1; Overhcat unfolds and deactivates pro-
teins (Paci & Karplus 2000).
"Equal Contribution.

---

Published as a conference paper at ICLR 2023
To enhance the capabilities of learning-based approaches, present large-scale, standardized protein benchmark, i.e, HotProtein, with organism-level temperature annotations which is a lower bound of protein's melting temperature (Jarzab et al. 2020). It consists of 182K protein sequences and 3K folded structures from 230 different species, covering a broad temperature range of - C2 However, similar to Cao et al. (201)), naively trained deep models on our dataset do not enable generalization to unseen proteins. The presumed reasons are (1) the considerable functional heterogeneity in proteins that arise from the environmental conditions and evolutionary history and (2) the scarcity of high-quality thermostabilityy experimental data due to the massive cost and labor required to generate such data.
To tackle these pain points, we introduce a novel algorithmic pipeline to improve thermostabil- ity prediction. First, we enrich our sequence embeddings by infusing 3D structural information in contrastive manner-we call this structure-aware pre-training (SAP). Then, we further fine-tune our model with a factorized sparse tuning (FST) approach. Here, we utilize factorized low-rank and sparse priors as implicit regularizers and leverage feature augmentation, such as mix-up (Vermal (ct al. 2019) and worse-case augmentations (Chen et al. 2021d). FST greatly boosts the performance of tuned predictors, suggesting improved data efficiency and robustness against domain shifts (Li) jet al. 2022b Chen et al. (202cc). Extensive evaluations on both Hot and the other exist- ing protein datasets (i.e., FireProtDB (Stourac al., 202T) verify our proposals' effectiveness.
Finally, to identify the top mutational predictions likely to improve thermal stability for a target aptimization-based editing framework top of classifier regressor, that attempts to mimic the process of directed evolution while limiting the stochasticity (Pucci & Rooman| 2014). Unlike existing protein engineering approaches (Eijsink al.) 2005 Couñago et al.] 2006 Wijma et al., (2033) that directly utilize the predictions to generate mutational designs, our proposal maximizes the model's 's objective to approach a more thermostable label to identify input mutated sequences. Our contributions can be summarized as follows:
protein, we develop
We collect and present large-scale protein dataset, i.e., Hot with organism-leve tem- perature annotations. We use the organism's environmental growth temperature label and clas- sify all proteins within each organism, which we use for thermostability prediction and editing. It contains 182K amino acid sequences and 3K folded 3D structures of proteins from 230 dif- ferent species, covering five thermostability types, e.g.. Cryophilic, Psychrophilic, Mesophilic, Thermophilic, and Hyperthermophilic.
We introduce a protein structure-aware pre-training by injecting 3D structural information into sequence embeddings in a contrastive fashion. It enhances the diversity and expressivity of the protein representations, resulting in improved thermostability predicting performance.
We introduce robust and data-efficient tuning framework that performs weight updates in the fac- torized and sparse subspace together with augmented feature embedding. This leads to substantial performance improvements against data scarcity and severe distribution shifts.
We formulate the search for thermal stabilizing mutations as optimization problem: for target protein and a trained predictor, we customize an editing framework that optimizes the input protein sequences identify thermostabilizing mutations.
Extensive experiments conducted on both thermostability prediction and protein editing tasks, consistently demonstrate the superiority of our proposals over various existing approaches (Rives) let al.) (02TT. For example, when fine-tuned on experimentally determined T,, dataset, Fire- ProtDB, our editing suggester achieves 53.93% (t 8.96%) precision in positive mutation clas- sification, 50.79 (f 6.54) Spearman P correlation coefficient in the temperature regression, and 54.24% (t 1.83%) successful rate in generating positive single mutations.
RELATED WORKS
Protein Thermostability Prediction. To enhance a protein's 's stability, AAG and ATm are common metrics by molecular biologists, enzymologists, and protein engineers. AAG evaluates the changes in free energy between a protein and a mutated variant. While ATm, evaluates the change in thermal tolerance between two protein variants. The two are related through the Van Hoff equation (Wright eet al. 2017) and it is common to obtain AAG from T,,, measurements (e.g. Chen et al. 2013 Capriotti et al. 2005 Rodrigues et al. 2018 Pires et al. 2014a Parthiban al. 2006).
2

---

Published as a conference paper at ICLR 2023
Most studies lack accurate large-scale thermostability data. For example, deepDDG (Cao et al. 2019) is trained on 5, 766 manually-curaae A measurements across 242 proteins, while one of their test set contains 173 experimental melting temperature changes (ATm) assess how well deepDDGG correlated with ATm (Cao et al. 2019). Previously, researchers use empirical physics- based energy contributions (Kellogg al.] 2011), torsion angles (Parthiban et al. 2006), or graph- based distance patterns (Pires et al.) (0144) as features and apply different models, e.g. physical models, residue interaction networks (Giollo et al., 2014), SVMs (Chen et al. 2013), predict the thermodynamics AAG. However, none of these methods have shown strong generalization.
Protein Engineering. Protein engineering with machine learning usually formulated as an energy-guided refinement process by maximizing pre-defined energy function with changing imput data (e.g., torsion angle, 1D amino acid sequence, or 2D contact map) (AIQuraishi 2019 Kuhlman et al. 2003, Huang et al. 2016]] Kuhlman & Bradley 2019). A group of works trains generative model or autoencoder and then optimizes the continuous hidden representation to maximize some given objectives (e.g., Gligorijevic et al. 2021 Shuai al. 2021 Hawkins-
et al., I02 Hoffman et al. 2022). Another category of works train models to map the input data to the target property (e.g. temperature, energy, etc.), and then optimize the discrete input space with combi- natorial optimization (Nom et al. (021). These methods have been applied to different kinds of
input data, e.g., 1D structural features (Norn al. 2021] [Wang al. 2017 Karplus 2009). torsion angles, and contact maps (Jones & Kandathil 2018 del Alamo et al. 2021)
Description about more related works. Due space limitations, we place discussions about protein engineering background, directed evolution, guided directed evolution, other protein ther- mostability datasets, sequencce-ssrructuue-baaae protein models, and sparse and low-rank subspace fine-tuning in our Appendix A
THE HOTPROTEIN DATASET
To obtain thermostable labels for an organism's proteome, we collect the raw data from the NCBI BioProject (Barrett et al., 2012), which offers an organizational framework to access the (meta-)dt )data about research projects, which is deposited or planned for deposition, into archival repositories.
Preprocess. From the NCBI bioproject XML file!] we filter organisms where the environmental data (i.e., Temperatuur" and "TeemeeratureRange") is available. After removing duplicate organism entries (keeping the first entry), this provides us with 1, 733 unique entries. Next, we pro- ceed to download these organisms' proteomes from UniPro via their taxids and bin the proteomes classification of that organism. Finally, we remove all proteins
based on the "TemperatureRange over 1, 500 amino acids in length and utilize CD- CD-Hifj to cluster protein sequences across organisms within a "TemmeraaureRange"" class at a sequence similarity threshold of 50%. In each cluster, we only keep proteins whose sequence lengths are between 200 550.
Annotation and Folding, To further increase the fidelity of our annotations, we remove organisms from each "TemperatureRange" bin where the COptimunTeemperuure responding limits: 1 Hypertthrrmohiilic (> 75 Celsius), @ Thermophilic (45 ~ 75 Celsius), @ Mesophilic (25 45 Celsius), O Psychrophilic (5 25 Celsius), and Cryophilic (-20 5 Cel- sius). The filtered proteomes and their corresponding annotations form the HotP rotein dataset and are utilized throughout the classification and regression tasks in our study.
does not fall within the cor-
For random subset of the HotProtein dataset (~ 3K), we predict structure with Al- phaFoldV2 (AIQuraishi] 2019) Jumper et al.] (02T)) to obtain their 3D coordinates. The official implementation is adopted. We report the predicted template modeling (P-TM) scores for each folded structure in Figure 2 A.3) and B.3). The P-TM score is rough approximation of the folding quality (ATQuraishi] 2019), and we kept structures only with a P-TM 2 0.8.
Properties. As described in Figure 2 we generate four distinct testbeds from the Pottrin dataset that differed in scale: (1) HP HPc has 1026 "hot" (2 45-C) and 939 "cold" (< 45"C) proteins from 61 and 4 species, respectively. Both sequence and structure statistics these proteins provided. (2) HP-S2c5 consists of both sequences and structures for {73, 387, 195, 196, 189)
httoas//ww...miproo..oo////////p http://wei2hhnggllbbueuaa
3

---

Published as a conference paper at ICLR 2023
(439)
(rt)
(187)
Pro and cod
(295)
Prydunghiie
(106)
Thoropile
Cold
(4)
Inpertenngtt:
Ipperthemophiic
Moaspiile (O
Thermophlle
Cysphiie
A.A)
(aa)
Steerphills (yr)
Paydhrnndi
(scees)
Themoghille
Pryshrophille (ge)
Oryophiie
Therophite
Hyperthemmpplii (4li)
The overview of HotProteis dataset. Figures A.1 ~ 3), B.1 ~ 3), and C.1 2) collect the statistics of HP-s HP-s$c5, and HP-S respectively. The left figure records the density distribution of each category over the protein's ' growth (organism-level) temperature. The density in bin is computed proteins within the hin [1026] indicates that there are 1026 proteins the corresponding class "Hot". The upper
Figure 2:
prolciin the cntegory right figure is the predicted template modeling (P-TM) score from AlphaFoldv2 (AIQuraishi] [201]] Jumper
eet al. 2021), reflecting the quality of folded protein structures. A larger P-TM suggests a better quality and P-TM 2 0.8 (red lines) is normal threshold for satisfied folded structures. The bottom right figure presents the species distribution of the three datasets, and # species in certain categories included in the brackets.
proteins sampled from the five categories, from Cryophilic to Hyperthermophilic. (3) HP-S entire sequence HotP rotein dataset. It contains {6390, 34946, 30333, 79087, 31549) sequences from (3. 32, 31, 116, 48} different species, of five classes ordered from Cryophilic to Hyperther- mophilic. (4) Moreover, HP SSP-SC2 as 2-class variant is created by merging Hyperthermophilic and Thermophilic as "hot" class and the other three as "cold" class. All temperature annotations are organism-level, serving c2/ / C5 and HP-S/ SSC are regarded as small- and large-scale datasets.
lower bound of for protein's melting temperature. Given their
number samples, HP-S 22cc
4 METHODOLOGGY
Denote the thermostability dataset by D = (æ, W where stands for the input while is the thermostability critical temperature (either real value or class index). Here we describe 1) how we pretrain the model F, and 2) how we finetune to pretrained model to fit DD
4.1 PROTEIN STRUCTURE- AWARE PRE-TRAINING
Previous protein pre-trained models (e.g., Rives et al. 2019 Vig et al. 2020 Rao et al.) 2021 Meier
al. 202l) mainly focus masking prediction tasks and amino acid representations. These pre- trained models achieve considerable inprovementscomppan
to traditional computation methods (e.g., [.g.[Sccymkooizz et al. 22055 Montanucei et al. 2019 Chen et al. (2020) on amino acid prediction tasks, e.g., contact prediction. mask prediction (Brandes et al. (002). mutational effect prediction (Meier et 202T Notin et al. 2022 et al., 2022a). On the other hand, directly learning universal protein embedding is potentially useful for protein prediction tasks. Inspired by recent sentence representation learning works (Gao et al. 2022I and 3D structure-aware pre-training (Hsu et al, 2022), adopt contrastive loss for representation learning.
Contrastive Loss Design. Denote the model F, our contrastive loss function is defined as:
exp(F'(x:) F" (x:))/)
(1 = log exp(F((xs)) (x) *Z Neg(x))
(1)
where Z ((,,, Neg(z:) En, E N exp(F((x.) Jr (zx)) //)) Here, Fx, and Fx,)) are two copies randomly perturbed models by injecting independent dropout noises into F(x;), and Neg(.) presents set of negative examples, and 5rmom is slowly updated momentum model
4

---

Published as a conference paper at ICLR 2023
Sequence Encoder
Pre-trained Weights
Sepaser
Ingert Sequencc
W Ru
Podlive
Negalve
Impor gD data
3D Encoder
Memory
Bank
U E guax
g
Pelsovttc Aeaise
Trateable Weights
Flod Weigln 2 Prence Wedght
Figure 3: The overall pipeline of our proposals. The left describes the protein structure-awar pre-training (SAP): The right presents the factorized sparse tuning (FST).
(see below) and 7 is the temperature value. Notice that random mutating amino acid could introduce unknown changes (Shortle 2009 Resch et al. 2008). we add perturbations to hidden representations and inject layer-wise dropout noise (Srivastava et al. 2014) with rate 0.05 in practice, and follow
the hyperparameter configurations in He et al. (2020).
Leverage 3D Information. We combine an additional 3D structure model using 3D inputs with the sequence-based model, to enhance the performance. We therefore slightly change Equation into,
exp(FF(x;;) F Jih (T:)TT)
(an N))( log (x))) F (a:)/T1
+ ((x;, Neg(x:))
Z(2,, =
Σ
exp (x:)) F (//,/////))))))))())) + (xp(FF -F F. (2, (/))
Neg((.))
where FAD stands for the 3D structure model which uses 3D coordinates inputs, Fmom is its momentum model. The memory bank logs the negative examples from both the sequence and 3D models. In short, pair for sequence and 3D model representation is regarded the positive pair, and we aim at injecting 3D representation information into the sequence model.
4.2 FACTORIZED SPARSE TUNING UNDER DATA SCARCITY AND DISTRIBUTION SHIFT
In real-world application domains, it is extremely challenging even infeasible collect a suffi- cient large-scale (engincered) protein dataset, due to the massive time and resource cost of trans- formation, protein expression, and purification. Therefore, data scarcity is one of the crucial bot- tlenecks in predicting protein properties. Another obstacle lies in the substantial data distribution shifts. Proteins with different (sometimes even the same (Xial 202T)) functionalities have distinc- tive and idiographic structures, sharing few common characteristics. To tackle these two issues, we introduce factorized sparse tuning pipeline (FST). It leverages the low-rank and sparse priors as implicit regularizations for enhanced data-efficiency (e.g., Khan & Stavness 2019 (Shlev-SSwartz Shwartz & David 2014: Rasmussen & Ghahramani 20000 Zhou et al. [018]| Arora et al. 2018 Zhang al. (021) and robustness to domain shifts (Li et al. (0226)).
Enforcing the Low-rank Prior through Factorization. Given a pre-trained model W, E Rmxx perform low-rank decomposition its weight update AW represented as W, A AW W + UV, where U Rx V e R and the rank r < min (d, k}. Usually, r = or 8 (i.e., 0.6% of total parameters) is sufficient to achieve great performance in our case. During forward, the input is fed into both dense pre-training W, and its low-rank representation UV. The obtained features are summed in a coordinate-wise manner, as demonstrated in Figure 3 (right). Take the original feature h = wy as an example. Our modified features can be described as below:
h h- - + a W, UV
As for the backpropagation, W, is frozen and low-rank matrices V) receive gradient updates. We adopt Xavier normal (Glorot& & Bengio 2010) and zero initialization for U and V respectively, and therefore the low-rank update is zero in the beginning. As suggested Hy et al. (021). UVæ can be scaled via an extra hyperparameter a, which works similarly to the learning rate. In our case, find a is not sensitive and set it as 4, i.e., the default value used in Hu et al. (202T).
We introduce the sparsity into tuning processes as an implicit struc- tural prior by modeling sparse weight update with S p As indicated in Figure 3 our refined features h are depicted follows:
Enforcing the Sparsity Prior.
h = W + (UV + +Sax S = 8igh (i,j) En
(4)
0, (i,j) 2
5

---

Published as a conference paper at ICLR 2023
where a "residual" feature Sæ is point-wisely added to from Equation i {1, .... .m) and j E f1,2,... n). the latter is 0 across the whole training. We compute the initial sparse matrix of S by i) first solving a robust principal component decomposition (Candès et al. (011) of Wp with an efficient algorithm (i.e., GreBsmo (Zhou& & Tao 2013)), and (x) then eliminating the elements with the least magnitude of obtained sparse solutions. In this way, step i) produces the initial values of st and step ti) constructs the set S2, where we observe [9]] 64 (i.e., 0.01% of total parameters) good enough in our case. As also revealed in Yu et al. (2077), combining the low-rank and sparse weight updates is capable of delivering superior performance, compared to either of them.
The set S determines the position of trainable St,, and pruned elements, where
Feature Augmentation. Another group of common fixes to data scarcity and domain shift prob- lems is data augmentation (Shorten & Khoshgoftaar 2019). However, most of existing data- level augmentations are unrealistic to protein sequences, including regional dropout (Zhou & Tao, 2013 Zhong et al.) 2020) and CutMix (Yun et al., 2019) in computer vision; synonym replace- ment (Kolomiyets et al. [201] Zhang et al. 2015 Wang & Yang 2015), random insertion, swap, and deletion (Wei & Zou, 2019) in natural language processing. The reason is that even a single amino acid mutation for the protein sequence may dramatically change its functionality (Resch et al.) 2008 Shortle| 2009). To avoid ambiguity. we utilize feature-level augmentations which manipulate the model's 's intermediate feature embedding. Specifically, we examine two effective mechanisms: (1) Mixup feature augmentation. It creates fused feature haug X x hn + (1 X) h2 and its associated soft label Yang À x y1 + (1 - A) y2, where [h1,91}, {h2, y2) are {feature em- bedding, }abel) of two different sequences and X = 0.2 in our experiments. (2) Worst-case feature augmentation. It injects worst-case noises 5 and builds augmented feature hang h + 6, where ô is generated by maxx h + +6). y) with the gradient ascent (Chen al., 2021dh. C, F, and denote the objective function, model, and input sequence, respectively.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Metrics. The performance is evaluated on the test splits. {Accuracy, Precision] and (Spearman, Pearson] correlation coefficients are used for classification and regression tasks. For HP sacc and HP-S?C5, 10-fold evaluation is conducted; while on HP-S and HP-SC2, run three replicates with different random seeds. Average performance and its 95% confidence intervals are reported.
Training Details. Baselines. 3D GCN (Gligonijevćć et al.) 2021) is trained for 20 epochs. with an initial learning rate of 1 x 10-4 that decays by 0.1 at the 10th epoch. For TAPE (Rao et al. 2019). train it for 4 epochs, with an initial learning rate of 1 10-4 and linear decay schedule. As for ESM-IB, we follow (Rives et al.) 2021) and only train a linear classification head on the top of ESM-IB backbone. The head tuning consists of 4 epochs with an initial learning rate of 2 x 10-2 and an OneCycle (Smith & Topin, 2019) decay scheduler. A training batch size of 4 is used across all experiments. Since we start tuning from pre-trained models (Rao et al. 2019 Rives al. 2021), the performance of TAPE and ESM- B are usually saturated after 2 ~ 3 epochs.
>> SAP. We use AlphaFoldDB (Jumper al. 202T) for SAP protein pre-training. We filter the data with sequence length and data quality and finally get 270K data. ESM-1B (Rives et al.) 2019) and ESM-IF (Hsu al., (022) backbone are used to process the sequence and 3D coordinate inputs, and an average pooling layer is applied to the final-layer token representations of ESM models and get protein embeddings. A momentum encoder with 7 = 1.0, momentum encoder coefficient 0.9999 and memory bank of size 65, 536 is used and the model is trained for 4 epochs, with Adam W optimizer. weight decay 10-12, 12, batch size 512 and an initial learning rate 10 decayed with OneCycle (Smith & Topin 2019) decay scheduler.
> FST. For our FST, we choose an initial learning rate of 1 x 10-2 for the linear classification head, and an initial learning rate of 1 x 10-3 for training the low-rank and sparse components in ESM-IB. Other training configurations inherit the same ones from tuning ESM-IB. As for the hyperparameters of rank and the number of non-zero elements |S2| in FST, we perform screenings
on r E {4, 8, 16) and [ E [16, 32, 64, 128), where we choose (r, (2]) = (4, 64) on HP-s*c2 s*c2/C5 (8, 64) on HP-S and HC-SC2. Mcantime, we adopt a one-step gradient ascent with a step size of 1 x 10 5 to generate worst-case feature augmentations, and apply them to the last two
and (r. [2])
layers of ESM- B, as suggested in Chen al. (021d).

---

Published as a conference paper at ICLR 2023
Table I: Performance of predicting thermostability with classification. Accuracy (%) is reported for all three datasets, and Precision (%) is calculated for the 2-class classification HP- C2 and HP-SC2. "FST" "Aug.". and "SAP" denote factorized sparse tuning, feature augmentation, and protein structure- aware pre-training, respectively. FST adopts (r, [2]) = (4, on HF cF-s$c2/C5 and (r, [[2 (8, 64) on HP HP-S/SC2. N.A. A. means "not applicable". 95% confidence interval are computed via the 10-fold evalua- tion on HP HP- and 3 replicates on HP-S/SC2
Methods
AP- 42-52o2
HP HP-sPc5
HP-S
SP-9C2
Accuracy
Precision
Accuracy
Accuracy
Accuracy
Precision
3D GCN (Glioorievi et al. 2021
78.881157
T3. 23.39
67 10
K.A.
N.A.
TAPE: |Rag 2019
83.31#110
76.42:330
66.44:2230
64.75a015
76.37
60.6120000
ESM-IFT (Hsu ora 2022
79.08.0.85
76. 19.596
58, 7512.46
K.A.
10 M.A..
N.A.
ESM-1B (Raves et
91.19s0.47
84.18:177
84. 81.2621154
69.5010016
86,2410.22
88.14
ESM. Aug.
91.74+077
$6.09-2.12
84.32+1.41
69. 69.54+0.39
80, 60.26+0022 72
88, 68.27-1143
ESM- FST
91.85++0.5
84.85-1.04
85.96=1.11
72.97=028
$7.50+0.12
68.71=1.78
ESM- 1B4-1B-+ FST Aug.
91 01.911/
86. 10::1.14
16.08811.3
73300820000
87.3 87.51120338
88, 83:11277
Aug. SAP
92.36=0.58
16.5771.07
86. 1..225=:..
73.21=0.18
87.57=0.10
89.07+1.29
Table 2: Performance of thermostabillty regression. Correlation coefficients are reported for all three datasets. 95% confidence interval are computed via the 10-fold evaluation on HP - SPc-scc//5 and 3 replicates on HP-
Methods
HP-8Pc2
HP- RP-$PC5
HP-S
Spearman
Pearson
Spearman
Pearson
Spearman
Pearson
GCN (ligorijevii et al. 2021
0.490
0.469:10.019
0.291 20006
0.301
N.A.
N.
TAPE (Rao et al. 2019
0.432.0..61
0.386100065
20.07
0.36-110.847
0,50110011 N.A.
0.153
ESM-IFT (Hs et al 2022
0.589=0.000
0.547:+0.06
0.373=00006
0.377:+0.055
N.A.
ESM-IB eti 202
0.89010.018
0.89310.0238
0.71210041
0. 0.800110.025
0,807.10001
0.809_0.001
ESM-IB Aug.
0. 1.893=0.014
0.909
0.714
18811
0.80
0.809=6.001
ESM-IB + FST
0. 0.89810,088
0,900.0.000
0.742200009
0.8 0.81510..244
0.81910:000
0.825_0.004
ESM-IB FST Aug.
0.892=0.011
0.91240.033
0.747=002m
0.818:0..12
0,820=0
0. 0.825=00008
ESM-IB + FST Aug, + SAP
0. 0.000_00.00
0.923.00012
0.75410.033
0.837_0.011
0.82310.00 001
0.827
5.2 PREDICTING THERMOSTABILIT VIA CLASSIFICATION AND REGRESSION
Comparison to Existing Approaches. We evaluate our proposals and compare with existing approaches on HP- /C5 and HP- S/ SC2 for both classification and regression tasks. 3D GCN (Gliorijjevi et al. )021) and TAPE (Rao et al.) 2019) are classical structure- and sequence- based models, while ESM-IFI IF (ssu et al. 2022) and ESM-IB (Rives et al., 202T) emerges recently as current state-of-the-an approaches, dealing with structure and sequence inputs respectively. From results shown in Table | baselines. Our proposal, i.e., ESM-1E+FST+AN margin of [1.17% ~ 13.48% accuracy, 2.33% ~ 13.12% precision, 0.016 ~ 0.474 Spearman, and 0.030 0.537 Pearson correlation] on HP- c22, f2.99% 27.50% accuracy, 0.042 0.463 Spearman, and 0.033 ~ 0.536 Pearson correlation) on HP-s²c5, f3.71% ~ 8.46% accuracy, 0.016 ~ 0.319 Spearman, and 0.018 0.374 Pearson correlation) on HP-S, and {1.33% ~ 11.2% accuracy and 0.93% ~ 8.43% precision on HC-SC2. These generalization improvements on hold- out proteins validate the effectiveness of our methods in tackling the functional domain shift and data-scarcity issues, i.e., pre-training on a general-purpose dataset Un i Re (22M samples) and tuning on the specific-domain dataset Hot form much stably in general, evidenced by the reduced confidence interval of multiple runs. e Structure versus sequence-based models. On the classification task, sequence-based models like ESM-1B and TAPE show clear performance advantages in most cases. As for the regression task, ESM-IB achieves an overwhelming superiority among the four baseline approaches, while the next best model is ESM-IF1 which takes protein structures as inputs. It suggests that powerful mech- anisms for utilizing 3D structure information are still missing on our challenging Hot P Prote e ein dataset. We make a pioneer attempt by leveraging 3D information to enhance sequence-basee els. 8 Does SAP, FST, and Aug helps? We examine these three components in an incremental manner and we observe: i) the performance gains from SAP demonstrate the benefits of treating 3D) protein structures as auxiliary information; it) Both Aug. & FST strengthen the model tuning and combination of them enjoys extra improvements; úii Among these ingredients of our proposal, FST contributes the most to superior performance. Specifically, Aug consistently obtains improvements in terms of the average performance; FST usually leads to statistically significant improvements with respect to the 95% confidence interval, especially on the HP-S/ SC2 dataset. @ Small v.s. large datasets. Feature level augmentations are more beneficial at scall-scale datasets, while FST and SAP bring performance gains for both small (HP-S2c5) and large (HP-S) datasets.
and 2 several consistent observations can be drawn: 1 Compared with -SAP, greatly surpasses various baseline by
(182K samples). Morcover, our methods per-
mod-
7

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study on the components of our framework. Accuracy (%) for classification and Spearman
& Pearson correlation coefficients for regression are reported. 95% confidence interval are computed via the
10-fold evaluation on HP HP-S' c5 and 3 replicates on HP-S.
Methods
HP-SPc5
HP-
Accuracy
Spearman
Pearson
Accuracy
Spearman
Pearson
Based on ESM-IB Rives st 2021
83,26 14
0.712..0.043
0.0022
69. 09.50....6
0.807 20,07
0,8009 20,00
Random
62.97+1.46
0.708-0.043
0.814:0.00
63.46+0.38
0.007
0.809
Feature Aug.
Mixup
63,94+1445
0.720
0.810-0.07
00.277+0:2
0.805-0.001
0.806-60008
Worst case
84.32+1.41
0.714=0.034
0.811=0.004
69. 09..54:0.00
0.808+0.00 002
0.809
86. 96:1.13
0.742
0.815 024
72.76
0.818
0.82 1821:00,000 1:0.002
Rank FST
65.00+2.10
0.703-0.002
0.801
72.97-00.8
0.819-0.002
0.8225+
16
63.75 1.92
0.725
0.758=0.319
73. 13.20000.40 40
0.821=0.000
0.825::0.002
[3]
16
85, 19::1 02
0.702
0.794=0.027
72.82::043
0.816
0.822:0.000
Sparsity in FST
[
32
85,5741.16
0,706_.0.005
0.767
72 85..0.0
0.818
0 0,0000000022
64
0.742:00.090
0.815
72. 97.97...22
0.819 002
8.8255000000 0.000
[52] 128
65.79+ 008
0.720-0.012
0.807-0.009
72.75+0.19
0.818-0.005
0.822-60000
3D infor. in SAP
W.O.
83.89&11.10 .10
0.718_.0.23
0.815
09.880...48
0.808800001
0.811
85, 11.442
0.727 03505
0.817::0.0 022
71.52 02::2.20
0.815 001
0.8199:0.000 0.000
Ablation Study. A comprehensive ablation is presented in Table 3 where we inspect different feature augmentations, the rank number in FST, the sparsity in FST, and the necessity of 3D infor- mation in SAP, on top of the vanilla ESM-IB model. Diverse feature augmentations. Besides mixup and worst-case feature augmentations, a straightforward baseline ("Random")) that applies a augmen-
Gaussian noise from (N0 0.12) to features, is also implemented. We see: random feature a tation usually incurs a performance degradation; mixup benefits models tuned on the small-scale dataset (HP (HD-s²c5), while hurts on the large-scale dataset (HP- (S): worst-case feature augmenta- tions consistently improve the generalization of tuned models on both HP HP-S and HP S, which is adopted by default in all other experiments of Table 22 and 4 @ The # rank in FST. The larger dataset prefers FST with higher rank such as 8 16, compared to the smaller dataset in which FST with r = 4 works the best. The finding coincided with the ones in Hu et al. (2021): Chen et al. e) @ The sparsity in FST. FST with [S2] 64 is a "sweet point". Superfluous tuning elements in f may lead to inferior results. 1 3D infor: in SAP. Directly introducing a contrastive loss in Equation Io the pre-training, has already boosted ESM-IB's performance, implying an im- proved protein embedding learning. Coupling the 3D information of protein structures, obtains an additional quality bonus for the pre-training.
More ablation studies. We summarize the results of additional ablation studies here and refer readers to the Appendix D for details: O We achieve the best results on extra test benchmarks and balsnee test sets. SAP outperforms other approaches to inject structure information. @ Training from scratch yields worse results than pretraining. @ On ESM-1B, we test several other baselines. We notice that additional l or & regularization, end-to-end fine-tuning, partially frozen tuning, and other feature aggregation methods all come to worse results than our current ESM-IB.
5.3 PROTEIN EDITING TOWARDS IMPROVED THERMOSTABILITY
Based on the models we trained in previous sections, three approaches for protein editing are pro- posed. We first describe the setting we use to edit proteins, and then report the performance of different models for three settings, classification, regression, and Editing. To demonstrate the effectiveness of protein editing towards improved thermostability, we further evaluate our proposals on FireProtDEB which is a manually curated database of the protein stability data for single-point mutants. It contains over 200 natural protein amino acid sequences, i.e., P = {p(0)
J. i (1,-... ,z), and their 3.9K mutated sequences, i.e,, P = (r/)), }, E (1,.... .#) and j E (1..... ,n((),, where (9) is the number of mutated variants for the original protein sequence (f)) We take P as the train- ing set and P as the hold-out testing set. Detailed configurations are referred to the Appendix C
Editing Suggestions via Classifiers. We deliver the protein editing suggestion via proposed clas- sifiers. The five-class classification is performed in both fine-tuning and testing stages with P and P. respectively. Furthermore, we regard a mutation as positive if (()) is predicted to class that has higher temperatures than the category of its original counterpart p(0) (e.g., from Psychrophilic
Petps://1cschen (tchhmidtttt chemi muni ww. fi/eottttt////////////
8

---

Published as a conference paper at ICLR 2023
Table 4: Evaluation of protein editing suggestions. Accuracy (%) and Precision (%) for classifiers Kendall
T & Spearman p rank correlation coefficients for regressors and successful rate (%) ) for adversarially learned mutations of protein sequences are reported. 95% confidence interval computed via three trials.
Classifier
Regressors
Leamndd Mutations
FireProtDB
Accuracy
Precision
Kendall
Spearman
Successful Rate
TAPE ESM-IB
44.75
30.18
0.07
5.64
34.78
Zero shot
62. 02.26
41.73
8.31
9.19
40.39
Ours
66.18
43.60
13.65
19.96
43.46
TAPE
55.18+0.56
38. 06335
1.335
1.77
44.59+0045
Fine-tune on P
ESM-IB
69.3010..3
14.97_047
34. 36..441442
11..25.1.89
10.41
Ours
71.42=0.34
53.93+0.39
36.97=0.7
50.79=1.22
54.24=0.41
to Mesophilic);; otherwise, we label the mutation as negative. Then, we compute the associated accuracy and precision to evaluate the quality of editing suggestions from our proposals.
Editing Suggestions via Regressors. We can also suggest possible editing via regressors. Specifi- cally, we fine-tune models to regress the protein's temperature in P, predict the possible temperature of their mutated variants in P, and measure the ordinal association between prediction and ground truth temperatures of P.
Optimizing Editing Suggestions. Instead of clas- sifying whether the mutation increases or decreases the thermostability, we introduce an efficient editing algorithm, by optimizing towards an improved ther- mostability, as depicted below:
Rephoctlacuning and
D T S3DC
max [M|Sn
(5) Figure 4: Visualization of an AI design that thermostabilizeed a protein. The amino acid "N" was predicted by the MutCompute web server to he mutated to a "k", which was experimen- tally shown to improve thermostability by 8.6 and was critical for the engincering of FAST- PETase(Lu et al. (2022]. ThermoPETase structure (left) (PDB: 6i 6i)6) and the mutated one (right)
where x the input, 1/8 denotes the target label of the class with higher temperature (e.g.. from Mesophilic to Thermophilic). M stands for the mu- tation function whose number of mutated amino acids |M] is constrained by n, F and & is our clas- sifier and the objective function. In practice, we class, Hyperther- mophilic. We set C in Equation 5 as probability change times the saliency score following PWWS (Ren et al. 2019). See Appendix C. for details.
was obtained via (Waterhouse el al., 2018).
set as the bighest-
Experimental Results. As demonstrated in Table 4 ESM- IB outperforms TAPE when it is zero- shot transferred to test data. Ours performs the best. We achieve 66. 13% accuracy, 13.65% Kendall correlation, and 19.96% Spearman correlation, which is 3. 3.86%, 5.31%, 10.47% relative higher than the BB baseline. After finetuning on P. EB obtains slightly better results than TAPE on both regression and classification metrics. Our model still outperforms both in all the tested cases. O When we do protein editing after fine-tuning, ours achieves the best (54.24% successful rate, 3.49% relative improvements than ESM-1B) B). It indicates that a better classifier also benefits protein engineering. Visualization of an edited protein is displayed in Figure
CONCLUSION
Predicting thermostabillzing mutations is a primary goal most protein engineering campaigns. However, the lack of thermally annotated protein data and effective algorithms has hindered the de- velopment of a thermostability prediction model that can generalize across the protein space. This work provides attempts to address both points via large-scale protein dataset (HotProtein) with species species-specific, lower-bound thermostability annotations and a novel algorithmic framework designed tackle the intrinsic challenges of functional domain shifts and data scarcity in ther- mostability protein engineering. Extensive results validate that our dataset and algorithmic frame- work provide meaningful improvements over baseline models. Lastly, based on established superior predictors, we search a protein's 's singlc-point mutation landscape towards identifying thermostabi- lizing variants. We will keep updating HotProted by collecting more sequences, improving our sequence-clustering and filtering. and folding more structures.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), the Defense Threat Reduction Agency [HDTRAI201001] and the Welch Foundation (F-1654). We would like to thank the Reviewers for taking the time and effort necessary to review the manuscript. We sincerely appreciate all valuable comments and suggestions, which helped us improve the quality of the manuscript.
REFERENCES
Miguel Alcalde, Manuel Ferrer, Francisco J Plou, and Antonio Ballesteros. Environmental biocatal- ysis: from remediation with enzymes to novel green processes. TRENDS Biotechnology. 24 (6):281-287, 2006.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):135--122, 2019.
Mohammed AlQuraishi. Alphafold at casp13. Bioinformatics, 35(22):486-44865, 2019.
Amro Abd-Al-Fattah Amara. Pharmaceutical and industrial protein engineering: where we are? Pakistan Journal of Pharmaceutical Sciences, 26(1), 2013.
Frances H Amold. Directed evolution: bringing new chemistry to life. Angewandte Chemie Inter- national Edition, 57(16):4143 4148. 2018.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep via compression approach. In International Conference Machine Learning, pp- 254-263, PMLR, 2018.
Tanya Barrett, Karen Clark, Robert Gevorgyan, Vyacheslav Gorelenkov, Eugene Gribov, Ilenc Karsch-Mizrachi, Michael Kimelman, Kim D Pruitt, Sergei Resenchuk, Tatiana Tatusova, et al. Bioproject and biosample databases at ncbi: facilitating capture and organization of metadata. Nucleic acids research, 40(D1):D57-D63 2012.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: A uni- versal deep-learning model of protein sequence and function. Bioinformatics, 280))2102--2 2022.
Maricke E Bruins, Anja EM Janssen, and Remko M Boom. Thermozymes and their applications. Applied biochemistry and biotechnology, 90(2):155-186, 2001.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Huali Cao, Jingxue Wang. Liping He, Yifei Qi, and John Zhang. Deepddg: predicting the stability change of protein point mutations using neural networks. Journal of chemical information and modeling, (9(4):1508-155) 2019.
Emidio Capriotti, Piero Fariselli, and Rita Casadio. 1-mutant2. 0: predicting stability changes upon mutation from the protein sequence or structure. Nucleic acids research, (Xsuuppl_):W3366 W310, 2005.
Beidi Chen. Tri Dao, Kaizhao Liang. Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. In Interna- tional Conference on Learning Representations, 2021a.
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Uni- fying sparse and low-rank attention approximation, arXiv preprint 10X b021b.
10

---

Published as a conference paper at ICLR 2023
Chi-Wei Wei Chen, Jerome Lin, and Yen-We Wei Chu. istable: off-the-shelf predictor integration for pre- dieting protein stability changes. In BMC bioinformatics, volume 14, pp. 1-14. BioMed Central, 2013.
Chi-Wei Chen, Meng- Han Lin, Chi-Chou Liao, Hsung-PPy Pin Chang, and Yen-Wei Wei Chu. istable 2.0: Predicting protein thermal stability changes by integrating various characteristic modules. Com- putational and structural biotechnology journal, 18:622-666 6300, 2020.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-cfficient gan train- ing beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, 34, 2021c.
Tianlong Chen, Yu Cheng, Zhe Gan, Jianfeng Wang, Lijuan Wang. Zhangyang Wang, and Jingjing Liu. Adversarial feature augmentation and normalization for visual recognition. arXiv preprint arXiv:2103. 12171, 2021d.
Xuxi Chen, Tianlong Chen, Yu Cheng, Weizhu Chen, Zhangyang Wang, and Ahmed Hassan Awadallah. Dsee: Dually embedded efficient tuning of pre-trained language models. arXiv preprint arXiv:2111.00160) 2021e.
Changhee Cho, So Young Choi, Zi Wei Luo, and Sang Yup Lee. Recent advances in microbial production of fuels and chemicals using tools and strategies of systems metabolic engineering. Biotechnology advances, 33(7):1455-1466 2015.
Rafael Couñago, Stephen Chen, and Yousif Shamoo. vivo molecular evolution reveals biophysical origins of organismal fitness. Molecular cell, 22(4):441-449, 2006.
Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks in fitness landscape infer- ence for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https:: /tttp:://openreview.. net. fwrmmmrrrmi p2dMLEWL8tE)
Carla CCR De Carvalho. Enzymatic and whole cell catalysis: finding new strategies for old pro- cesses. Biotechnology advances, 29(1):75-83 2011.
Ramin Dehghanpoor, Evan Ricks, Katie Hursh, Sarah Gunderson, Roshanak Farhoodi, Nurit Haspel, Brian Hutchinson, and Filip Jagodzinski. Predicting the effect of single and multiple mutations on protein structural stability. Molecules, 23(2):251, 2018.
Yves Dehouck, Aline Grosfils, Benjamin Folch, Dimitri Gilis, Philippe Bogaerts, and Marianne Rooman. Fast and accurate predictions of protein stability changes upon mutations using statisti- cal potentials and neural networks: Popmusic-2.0. Bioinformatics, 25((9)25377-254,,000 2009.
Diego del Alamo, Davide Sala, Hassane Mchaourab, and Jens Meiler. Sampling the conformational landscapes of transporters and receptors with alphafold2. bioRxiv, 2021.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint 1 2018.
Vincent GH Eijsink, Sigrid Gáseidnes, Torben V Borchert, and Bertus Van Den Burg. Directed evolution of enzyme stability. Biomolecular engineering, 22(1-3):21 2005.
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean R Eddy, Aurélien Luciani, Simon C Potter, Mat- loob Qureshi, Lorna J Richardson, Gustavo A Salazar, Alfredo Smart, al. The pfam protein families database in 2019. Nucleic acids research, 47DD))D427-D4433 2019.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: towards crack- ing the language of life's s code through sulf-supe deep learning and high performance com- puting. arXiv preprint 2000020007.00225, 2020.

---

Published as a conference paper at ICLR 2023
Brandon Frenz, Steven M Lewis, Indigo King, Frank DiMaio. Hahnbeom Park, and Yifan Song. Prediction of protein mutational free energy: benchmark and sampling improvements increase classification accuracy. Frontiers in bioengineering and biotechnology, pp. 1175, 2020.
Sven Frokjaer and Daniel E Otzen. Protein drug stability: a formulation challenge. Nature reviews drug discovery, 4(4):298-306, 2005.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Manuel Giollo, Alberto JM Martin, Ian Walsh, Carlo Ferrari, and Silvio CE Tosatto. Neemo: method using residue interaction networks to improve prediction of protein stability upon muta- tion. BMC genomics, 15(4):1-11, 2014.
Vladimir Gligorijevic, Daniel Berenberg, Stephen Ra, Andrew Watkins, Simon Kelow, Kyunghyun Cho, and Richard Bonneau. Function-guided protein design by deep manifold sampling. bioRxiv, 2021.
Vladimir Gligorijevćć, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):1-14, 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long 2021.
Pupers), pp.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv: 18f2.0472...4 2018.
GD Haki and SK Rakshit. Developments in industrially important thermostable enzymes: review. Bioresource technology, 89(1):17-34, 2003.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014: 2015.
Alex Hawkins-
David T Jones, and Brooks Paige. Msa-conditioned generative protein lan- guage models for fitness landscape modelling and design. In Machine Learning for Structural Biology Workshop, NeurlPS, 2021.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kirk-Kiikpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110004366. 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2020.
Brian L. Hie, Duo Xu, Varun R Shanker, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, and Peter S Kim. Efficient evolution of human antibodies from general protein language models and sequence information alone. bioRxiv, 2022.
Samuel C Hoffman, Vijil Chenthamarakshan, Kahini Wadhawan, Pin-Yu Yu Chen, and Payel Das. Opti- mizing molecules using efficient queries from property evaluations. Nature Machine Intelligence, 4(1):21-31, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2023
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022. doi: 10.11001220222044144488777777770
2022/00/10/2022..0..
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Jie Huang, Peng Zhao, Xin Jin, Yiwen Wang, Haotian Yuan, and Xinyuan Zhu. Enzymatic biofuel cells based on protein engineering: recent advances and future prospects. Biomaterials science, 8 (19):5230-5240,, 2020.
Po-Ssu Huang, Scott Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):320-333 327-327, 2016.
Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, et Meltome atlas thermal pro- teome stability across the tree of life. Nature methods, 17(5):495-503, 2020.
Lei Jia, Ramya Yarlagadda, and Charles C Reed. Structure based thermostability prediction models for protein single point mutations with machine learning tools. PloS one, (0(9):c013800) 2015.
Emmi Jokinen, Markus Heinonen, and Harri Lähdesmäki. mgpfusion: predicting protein stability changes with gaussian process kernel learning and data fusion. Bioinformatics, 34(13)i174-i223, 2018.
David T Jones and Shaun M Kandathil. High precision in protein contact prediction using fully convolutional neural networks and minimal sequence features. Bioinformatics, 34(19):3308- 3315, 2018
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Swati Kapoor, Aasima Rafiq, and Savita Sharma. Protein engineering and its applications in food industry. Critical reviews in food science and nutrition, 57(11):2321-3322 2017.
Kevin Karplus. Sam-t08, hmm-hased protein structure prediction. Nucleic acids research, 37 (suppl 2): 2009. 2009.
Elizabeth H Kellogg, Andrew Leaver-Fay, and David Baker. Role of conformational sampling in computing -inuuue changes in protein structure and stability. Proteins: Structure, Func- fion, and Bioinformatics, 79(3):830-838, 2011.
Najeeb Khan and Ian Stavness. Sparscout: Controlling sparsity in deep networks. In Canadian conference on artificial intelligence, pp. 296-307. Springer, 2019.
Dhananjay Kimothi, Akshay Soni, Pravesh Biyani, and James Hogan. Distributed representations for biological sequence analysis. arXiv preprint arXiv: (608.05549) 2016.
Oleksandr Kolomiyets, Steven Bethard, and for textual temporal analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, volume 2, pp. 271-276 ACL; East Stroudsburg, PA, 2011.
Moens. Model-portabiln experiments
Felix Krueger and Simon R Andrews. Bismark: flexible aligner and methylation caller for bisulfite- seq applications. bioinformatics, 27(11):1511-157) 2011.
Brian Kuhlman and Philip Bradley. Advances in protein structure prediction and design. Nature Reviews Molecular Cell Biology, 20(11):681-697, 2019.
Brian Kuhlman, Gautam Dantas, Gregory C Ireton, Gabriele Varani, Barry L Stoddard, and David Baker. Design of a novel globular protein fold with atomic-level accuracy. science, 302(5649): 1364-1368, 2003.
13

---

Published as a conference paper at ICLR 2023
Julia Koehler Leman, Brian D Weitzner, Steven M Lewis, Jared Adolf Slolf-Bryfogle. Nawsad Alam, Re- becca F Alford, Melanic Aprahamian, David Baker, Kyle A Barlow, Patrick Barth, et al. Macro- molecular modeling and design in rosetta: recent methods and frameworks. Nature methods, 17 (7):6656600, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3055 2021.
Bian Li, Yucheng T Yang, John A Capra, and Mark B Gerstein. Predicting changes in protein thermodynamic stability upon point mutation with deep 3d convolutional neural networks. PLoS computational biology, 16(1 1008202 2020.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Leurning Reprexentations, 2018.
Gang Li, Kersten S Rabe, Jens Nielsen, and Martin KM Enggvist. Machine learning applied to pre- dicting microorganism growth temperatures and enzyme catalytic optima. ACS synthetic biology, 8(6):1411-1420, 2019.
Gang Li, Filip Buric, Jan Zrimec, Sandra Viknander, Jens Nielsen, Aleksej Zelezniak, and Mar- tin KM Engqvist. Learning deep representations of enzyme thermal adaptation. bioRxiv, 2022a.
Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13409-13418, 2022b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, 2021.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907.19977.19999 2019.
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-supervised contrastive learn- ing of protein representations by mutual information maximization. BioRxiv, 2020.
Hongyuan Lu, Daniel Diaz, Natalie J Czarnecki, Congzhi Zhu, Wantae Kim, Raghav Shroff, Daniel Acosta, Bradley R Alexander, Hannah O Cole, Yan Zhang, et al. Machine learning-aided engineering of hydrolases for pet depolymerization. Nature, 604(7907):662-667, 2022.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations protein function. bioRxiv, 2021. doi: 210.110/2222100009945555500 1101111/2000 2221000222222 07 09 450648v1
URL https:/ http://www..biiii..../../.. www.oooii..or//ooo.. wwt.://t////cttc////o/o
Ludovica Montanucci, Emidio Capriotti, Yotam Frank. Nir Ben- and Piero Fariselli. Ddgun: an untrained method for the prediction of protein stability changes upon single and multiple point variations. BMC bioinformatics, 100 2019.
Marirosa Mora and John L Telford. Genome-basee approaches to vaccine development. Journal of Molecular Medicine, 88(2):143-147, 2010.
Rahul Nikam, A Kulandaisamy, Harini, Divya Sharma, and M Michael Gromiha. Prothermdb: thermodynamic database for proteins and mutants revisited after 15 years. Nucleic Acids Re- search, 49(D1):D420 D424, 2021.
Christoffer Norn Basile IM Wicky, David Juergens, Sirui Liu, David Kim, Doug Tischer, Brian Koepnick, Ivan Anishchenko. Foldit Players, David Baker, et al. Protein sequence design by conformational landscape optimization. Proceedings of the National Academy of Sciences, 118 (11):e2012228118 2021.
14

---

Published as a conference paper at ICLR 2023
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transform- and anference-time retrieval. In International Conference Machine Learning, pp. 16990- 17017. PMLR, 2022.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392 2019.
Emanuele Paci and Martin Karplus. Unfolding proteins by external forces and temperature: the importance of topology and energetics. Proceedings of the National Academy of Sciences, 97 (12):6521-655 66226, 2000.
Inyup Paik, Phuoe HT Ngo, Raghav Shroff, Daniel J Diaz, Andre C Maranhao, David JF Walker, Sanchita Bhadra, and Andrew Ellington. Improved bst dna polymerase variants derived via machine learning approach. Biochemistry, 2021.
Stefano Panno, Slavica Matić, Antonio Tiberini, Andrea Giovanni Caruso, Patrizia Bella, Livio Torta, Raffaele Stassi, and Salvatore Davino. Loop mediated isothermal amplification: principles and applications in plant virology. Plants, 944):461, 2020.
Emmanuel Paradis and Klaus Schliep. ape 5.0: environment for modern phylogeneties and evolutionary analyses r. Bioinformatics, 35(3):526-528, 2019.
Hahnbeom Park, Philip Bradley, Per Greisen Jr, Yuan Liu, Vikram Khipple Mulligan, David E Kim, David Baker, and Frank DiMaio. Simultaneous optimization of biomolecular energy functions features from small molecules and macromolecules. Journal of chemical theory and computation, 12(12):6201-62 12122, 2016.
Vijaya Parthiban, M Michael Gromiha, and Dietmar Schomburg. Cupsat: prediction of protein stability upon point mutations. Nucleic acids research, 34(suppl D.W239--W242 2006.
Douglas EV Pires, David B Ascher, and Tom L Blundell. Duet: a server for predicting effects of mutations on protein stability using an integrated computational approach. Nucleic acids research, 42(W1):W144W319. 2014a.
Douglas EV Pires, David B Ascher, and Tom L Blundell. mcsm: predicting the effects of mutations in proteins using graph-based signatures. Bioinformatics, 30(3):335-342, 2014b.
Soroosh Pouyan, Milad Lagzian, and Mohammad Hossein Sangtarash. Enhancing thermostabiliza- tion of newly discovered a-amylase from bacillus cereus g196 by combining computer-aided directed evolution and site-directed mutagenesis. International Journal of Biological Macro- molecules, 197:12 22, 2022.
Fabrizio Pucci and Marianne Rooman. Stability curve prediction of homologous proteins using temperaaure-dependenn statistical potentials. PLoS computational biology, 10(7)=e003368, 2014.
Fabrizio Pucci and Marianne Rooman. Physical and molecular bases of protein thermal stability and cold adaptation. Current opinion in structural biology, 42:1 17-1288 2017.
Fabrizio Pucci, Jean Mare Kwasigroch, and Marianne Rooman. Scoop: accurate and fast predic- tor of protein stability curves as function of temperature. Bioinformatics, 33(21):3415-3422 2017.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural infor- mation processing systems, 32, 2019.
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Picter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp. 8844-8856. PMLR, 2021.
Carl Rasmussen and Zoubin Ghahramani. Occam's 's razor. Advances in neural information process- ing systems, 13, 2000.
15

---

Published as a conference paper at ICLR 2023
Sylvestre-Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 506-5066, 2017.
Shuhuai Ren, Yihe Deng, Kun He. and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085-1097, 2019.
Marcus Resch, Harald Striegl, Eva Maria Henssler, Madhumati Sevvana, Claudia Egerer-Sieber, Emile Schiltz, Wolfgang Hillen, and Yves A Muller. A protein functional leap: how a single mutation reverses the function of the transcription regulator tetr. Nucleic acids research, 36(13):
4390-4401, 2008.
Adam Riesselman, Jung-Eum Shin, Aaron Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew Kruse, and Debora Marks. Accelerating protein design using autore- gressive generative models. BioRxiv, pp. 757252, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, 2019. doi: 10.1101/622803, URL https:: httpt::t/ww..bbbttttttt http://wwwwbirr.e wiwrxivi.......o org/ wwwpg//ooee/////eeeeee / 100. 11101662280v4
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Carlos HM Rodrigues, Douglas EV Pires, and David B Ascher. Dynamut: predicting the impact of mutations on protein conformation, flexibility and stability. Nucleic acids research, 46(W1): W1350-W355,222 2018.
Carol A Rohl, Charlie EM Strauss, Kira MS Misura, and David Baker. Protein structure prediction using rosetta. In Methods in enzymology, volume 383, pp. 66-93. Elsevier, 2004.
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Ser- rano. The foldx web server: an online force field. Nucleic acids research, W388, 2005.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
David Shortle. One sequence plus one mutation equals two folds. Proceedings of the National Academy of Sciences, 106(50):20011-10011 2009.
Raghav Shroff, Austin W Cole, Daniel Diaz, Barrett R Morrow, Isaac Donnell, Ankur Anna- pareddy, Jimmy Gollihar, Andrew D Ellington, and Ross Thyer. Discovery of novel gain-of- function mutations guided by structure-based deep learning. ACS synthetic biology, 9(11): 2935, 2020.
Richard W Shuai, Jeffrey A Ruffolo, and Jeffrey Gray. Generative language modeling for antibody design. bioRxiv, 2021.
Leslie N Smith and Nicholay Topin. Super-convergence. Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-de domain operations applications, volume 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
16

---

Published as a conference paper at ICLR 2023
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 9(1):1-8, 2018.
Jan Stourac, Juraj Dubrava, Milos Musil, Jana Horackova, Jiri Damborsky, Stanislav Mazurenko, and David Bednar. Fireprotdb: database of manually curated protein stability data. Nucleic acids research, 19(DI))D319-D324 2021.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103. 15316, 2021.
Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consor- tium. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31 31(6):926-932, 2015.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438 0188-647. PMLR, 2019.
Jesse Vig, Ali Madani, Lav Varshney, Caiming Xiong, Nazneen Rajani, et al. Bertology meets biology: Interpreting attention in protein language models. In International Conference on Learn- ing Representations, 2020.
C Von der Osten, S Branner, S Hastrup, L. Hedegaard, MD Rasmussen, H Bisgaard- Carlsen, and JM Mikkelsen. Protein engineering of subtilisins to improve stability deter- gent formulations. Journal of biotechmology, 28(1):55-68, 1993.
Sheng Wang. Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLoS computational biology, 13(1):e100532) 2017.
William Yang Wang and Diyi Yang. That's annoying!!!: A lexical and frame-semante embed- ding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In Proceedings of the 2015 conference empirical methods in natural language processing, pp. 2557-25555555 2563,, 2015.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151-6162, 2020.
Andrew Waterhouse, Martino Bertoni, Stefan Bienert, Gabriel Studer, Gerardo Tauriello, Rafal Gu- mienny, Florian T Heer, Tjaart A P de Beer, Christine Rempfer, Lorenza Bordoli, et al. Swiss- model: homology modelling of protein structures and complexes. Nucleic acids research, 46 (W1):W296-W2) W303, 2018.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint /901v 2019.
Hein J Wijma, Robert J Floor, and Dick B Janssen. Structure-and sequence-analysis inspired en- gineering of proteins for enhanced thermostability. Current opinion structural biology, 23(4): 588-594, 2013.
Bruce Wittmann, Yisong Yue, and Frances H Arnold. Informed training set design enables efficient machine learning-assister directed protein evolution. Cell systems, 12(1 12(11))1026-1055 2021.
Catherine I. Worth, Robert Preissner, and Tom I. Blundell. Sdm-a server for predicting effects of mutations on protein stability and malfunction. Nucleic acids research. 32(s W222, 2011.
Thaiesha A Wright, Jamie M Stewart, Richard C Page, and Dominik Konkolewicz. Extraction of thermodynamic parameters of protein unfolding using parallelized differential scanning fluorime- try. The journal of physical chemistry letters, pp. 553-558, 2017.
Xuhua Xia. Domains and functions of spike protein in sars-cov-2 2 in the context of vaccine design. Virases, 139 2021.
17

---

Published as a conference paper at ICLR 2023
Kevin K Yang, Zachary Wu, and Frances H Amold. leareing directed evolution for protein engineering. Nature methods, 16(8):687-694 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019b.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7370-7379, 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- 2019.
ings of the IEEE/CVF international conference computer vision, pp.
Shuai Zhang, Meng Wang. Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. Why lottery ticket wins? theoretical perspective of sample complexity on sparse neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedingy of the AAAI conference on artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Tianyi Zhou and Dacheng Tao. Greedy bilateral sketch, completion & smoothing. In Artificial Intelligence and Statistics, pp. 650-658. PMLR, 2013.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous gen- eralization bounds at the imagenet scale: a pac-bayesian compression approach. In International Conference Learning Representations, 2018
18

---

Published as a conference paper at ICLR 2023
A MORE RELATED WORKS
Protein Engineering. In general, the factors that make one protein functional and simultaneously thermostable compared to another are complex and unknown for specific target protein. The lit- erature on computational models for in silico screening of the thermodynamic impact of mutation is exhaustive. In the development of biopharmaceuticals and biocatalysts, improving the thermody- namic properties of the target protein is common early goal in most protein engineering campaigns. A protein's 's thermodynamic stability is commonly represented by its Gibb's 's free energy and/or melt- ing temperature. The protein engineering community has been designing computational tools to enable in silico screening of mutations for several decades and these tools can be classified into bio- physical models and statistical/machine learning-batt -band models. Computational protein engineering models are used to guide the search for optimized sequences. Despite recent progress, these meth- ods have limited utility in reliably ranking sequences, especially at discerning small changes in thermodynamic properties. Although physics-based methods have been shown reach reasonable accuracy, they are computationally demanding and low-throughput making them intractable to apply to large-scale in silico screening of protein variants.
Recent Success in AI + Protein Engineering. Beyond promising performance in simulation ex- periments, in the last two years, unsupervised structure-based deep learning techniques have also shown their ability to learn from natural protein structures and then generalize to guide the engineer- ing of structurally diverse proteins (Kuhlman & Bradley, 2019, Shroff et al. 2020). For example, Paik et al. (202I) used MutCompute to improve the thermostability of a DNA polymerase in order to enable faster loop-mediated isothermal amplification (LAMP) (Panno et al., (2020) assays to be carried out at C3 temperature where the commercial counterpart enzyme. BST 2.0, is inactive. Additionally, [u et al. (2022) engineered two Polyethylene terephthalate (PET) hydrolases (PETase and Cutinase) with MutCompute to improve its thermal and pH stability, resulting in FAST-PETasc: a PET hydrolase that can fully degrade post-consumer PET within 48 hours.
Directed Evolution and Guided Directed Evolution. The most common strategy to engineer and stabilize a protein is directed evolution. Directed evolution leverages natural evolution where random mutagenesis is synergized with screening to identify variants with improved fitness for the desired phenotype. essentially performing a greedy local search to optimize protein fitness (Amold, 2018). For directed evolution, it is necessary to design a high-throughput assay that selects mutants enriched for your target phenotype, which is not always possible. Recent progress has been made in combining machine learning approaches with the data collection capability of directed evolution in order to accelerate iterative rounds of directed evolution (Yang et al. 2019a; Wittmann et al.) 2021]. Here, machine learning is used to ease the experimental screening burden by evaluating proteins in silico. Briefly, machine learning-guided directed evolution (MLDE) works by iteratively training an ML model on a small number of variants (10) from a combinatorial library and is then used to infer the remaining variance in the library where the variants with the highest predicted fitness are then experimentally evaluated and then added to the training set for the subsequent round. However, the need to consistently generate libraries and expand the training set does not completely alleviate the screening burden. Furthermore, |Wittmann et al. (2021) showed empirical evidence that eequence-based embeddings from transformer models can accelerate MLDE and that an MSA transformer (Rao et al.) 2021) embedding outperformed sequence embeddings of larger models such as ProtBert-BFD and ESM-IB (Rives et al. (021).
Protein Thermostability Dataset. In protein engineering, improving the stability of a protein is a goal in nearly all protein engineering campaigns and can be deciding factor in the commercial- ization of a biocatalyst. Thus, there is plenty of literature on computational algorithms attempting to model this phenotype. Over the last two decades, several algorithms that predict the thermody-
namic impact of mutations have been developed (c.g.. Park et al.] 2016 [Schymkowitz et al. 2005
Dehouck et al., 2009, Jokinen et al.) 2018 Worth et al. 201 1] Cao et al. 2019 Li et al., 2020 Paradis & Schliep 2019; Krueger & Andrews] 2011 Dehghanpoor et al. 20188 and can be classi- fied as either biophysical models or statisticallmachine learning models. Biophysical models utilize sampling of the protein structure to determine the changes in stability. The primary two biophysical models are Rosetta (Park et al. 2016) and FoldX (Schymkowitz et al. 2005). These methods have shown good performance and utility in sev- eral applications but fail to capture narrow changes in stability (< Ikcal/mol), which is commonly observed for many mutations. Furthermore, they are computationally demanding and have low
amino acid interactions and conformational/rooaan
A19

---

Published as a conference paper at ICLR 2023
throughput, hindering their application to large-scale in silico screening for identifying stabilizing point mutations (Li et al. [020 Paradis & Schliep 2019). Machine learning tools trained to evaluate the impact of a mutation
the stability of protein are growing in popularity (c.g., (c.g..Pokinen Pokinen et al. Dehouck et al. 2009. Worth et al. 2011 M Cao et al. 2019 Li et al. 2020; Paradis & Schliep,
2018
2019 Krueger& & Andrews 2011 DDhhhiegghaapoo et al. 2018). Some notable machine learning mod-
cls include PopMusic (Dehouck et al., 2209), DeepddG (Cao et al. 2019), ThermoNet (Li et al. 2020), SDM (Worth et al. (011)) and mGPfusion (Jokinen et al. 2018). These models are trained in supervised fashion to predict available experimental data, usually sourced from ProTTherm (Jia jt al.] 2015) or manually curated datasets from the literature, using either evolutionary or structural input features with some models utilizing both types of features. Thus, all use relatively small training set, with DeepDDG (Cao et al. 2019) having the largest at 5700 manually curated data points, due to the limited amount of experimentally validated point mutations. So far, we have yet to see a muabiltty-reddictnng algorithm that leverages a language model's learned representation to predict the stability effects of mutations.
Sequence- and sannctured-basee Protein Models. Motivated by self-supervised pre-training in natural language community (e.g., Liu et al. 2019 Yang al. 2019b), variety of recent works for- mulate the protein pre- pre-training tasks as sequence self-supervised learning, e.g., with auto eecoding
(Shuai et al. 2,02), auto-regressive (e.g.. Rives et al., 2019; Meier et al. 2021: Elnaggar et al., 2020 Riesselman et al. (201)), skip-gram language model (Kimothi et al.] 2016). mask prediction
(Vig al. 2020; Brandes et al. 2022) or amino acid contrastive learning objectives (Lu et al. 2020), similarity metric learning (Bepler& & Berger 2019; Alley et al. 2019). Although using differ- ent architectures (e.g.. LSTMs, transformers, graph networks), different pre-trained datasets (e.g., UniRef100 (Suzek et al. 2013), BFD (Steinegger & Söding 2018), Pfam (El-Gebali et al., 2019), and different objectives, these models convert input sequences into per amino acid representations. Most recently, thanks to AlphaFoldV2 and its folded dataset (Jumper et al. [022]) ESM-IF (Hsu jet al. 2022) proposes to learn the 3D model with inverse folding, which predicts sequences from backbone 3D structures. The input is 3D coordinate information for each residues backbone N, CA, and C atoms (present in every amino acid). Passing the input through the ESM-IF permutation invariant model, the model outputs probability distribution for amino acid at each residue position.
Sequence-based Pretraining Methods, ES (Rives et al. (021). MSA Transformer (Rao et al., 2021), and Vig et al. (2020) several popular pre- trained transformer protein language models, which regard one amino acid as a token. [Shuai et al. (202T) trains an antibody language model with antibody data. These works have been applied simulation experiments. For example, (Hie et al. (2022) performs ESM-guided affinity maturation of seven diverse antibodies, screening 20 or fewer variants of each antibody across only two rounds of evolution. [Shuai et al. (2021) demonstrates that their model can be applied to generate synthetic libraries that may accelerate the discovery of therapeutic antibody candidates in real experiments.
Tranception (Notin et al., 2022) proposes new architectures for sequence pretraining and offers new test benchmarks. Different from this work, we are interested in pre-training objectives and fine- tuning methods, instead of neural architectures. FLIP (Dallago et al. 2021) sets up benchmarks for fitness landscape inference for proteins, and theorem stability is one part of the benchmark. DeepET (Li et al., 2022a) first pre-trains their convolution network in sequence and then the model is fine-tuned on thermal prediction tasks. Compared to DeepET, which proposes to tne-tune the last layer or do end-to-end fine-tuning, we introduce the sparse and low-rank fine-tuning method. As shown in Table A6 our regularized fine-tuning reaches superior performance, compared to the last-layer end-to-end fine-tunings.
Sparse and Low-rank Subspace Fine-tuning, Sparsity aaware finettuing is typically leveraged to- wards the goal of parameter-efficinnc (Rebuffi et al. [017 HHoulsyy al. 2019 Li & Liang 2021 Lester et al., (021). which only tunes a few of model weights and keep the rest unchanged. Guo al. (2021) embeds the sparsity into fine-tuning with a differentiable approximation of to regular- ization. Chen et al. (2021e) introduces sparse tuning patterns via classical pruning methods (Han) al. 2015). (Chen et al.) 2021a) pre-defines the shape of sparsity patterns and learns their combi- nation. Another alternative solution for efficient fine-tuning is to constrain weight updates within a
low-rank subspace (Hu et al. 202T] Chen et al.) 2021e; He et al., (02T)). Numerous literature (Yu eet al. 2017 Li et al., 2018 Oymak et 2019 Aur-Ari et al. [0018 8) point out the intrinsic low-rank dimensionality of trained over-parameterizer models. Wang et al. (2020); Hu et al. (202T) focus on imposing explicit low-rank structures when transferring pre-trained models to diverse downstream
A20

---

Published as a conference paper at ICLR 2023
tasks, leading to considerable parameter efficiency. Recently, several pioneering studies (Chen al. 2021e a b) consider combining sparsity and low-rank decomposition for improved efficient training. However, investigate these two structural priors from distinctive perspective, i.e., their implicit regularization effects against data scarcity and domain shifts.
B MORE METHOD DETAILS FOR STRUCTURE-AWARE
During the pretraining, the language model only learns from context-token pairs, instead of learning global representation for one sentence (or one protein). Once we target the global representation of one sentence (or one protein), we should learn from a loss function that directly compares one sentence and another (Gao et al., 202I). Similar the sentence representations et 2018), we notice that the protein representations learned by sequence models are not uniformly distributed the latent space (Gao et al.] (02T). Therefore, we adopt contrastive learning in the protein embedding space to distribute the representations more uniformly and let the 3D model inject information into the sequence model. We refer readers to (Gao et al. 2021 Su et al. 2021) for details about motivations.
BERT (Devlin
Momentum Encoder to Construct Negative Examples. Following (He et al.
2020). the momen-
tum model Frmm is a slowly updated model with update rule: Frmom f- a.
(1 a)F,,
where a E [0. the momentum coefficient. During the optimization of Equation | to save the computation budget for negative examples, adopt the memory bank (He et al. 2020) to
struct enough negative examples. The Neg set is taken to be queue of (2,)) representations from previous mini-batches.
C MORE IMPLEMENTATION DETAILS
Computing resources. Experiments use Tesla V100- 12000-XM--3300 GPUs as computing re- sources. Each experiment can be run with a single V100 GPU.
C.1 EXPERIMENTAL DETAILS ABOUT PROTEIN EDITING
Search Space. Based on our fine-tuned classifiers, we generate 20 possible single point mutations at each position (i) in a natural protein sequence y'+)) and then evaluate whether the resulted fo (()) (k E {1,.... 20}) successfully leads to improved thermostability. The corresponding successful rate is used as evaluation metric.
Optimization Method. We adopt a recently- prroposd well-known method, PWWS Ren et al. 2019). PWWS replaces tokens base ranking score H/ (x,x;²) ((S( where x is the input sequence, 2F stands for replacing the amino acid at index i, A., denotes the change in classification probability after replacing with x;, and ((S(x)); is the saliency score.
Dataset Descriptions. Previously, numerous benchmarks have been proposed for thermostabillty or AAG prediction. Some of them use Rosetta (Rohl et al. (0044) predicted score, (Frenz et al.) 2020) and others create datasets, e-g., ProThermDB (Nikam et al., (021) based on experiment records. The curation of large high-quality thermostability datasets still ongoing, and protein stability experiments are time-consuming and expensive (Stourac et al., 2021). We set up our benchmark using FireProtDB (Stourac et al. 202T), which is superset of published experiments records and is the most up-to-date dataset to our knowledge. We download the latest version of FireProtDB, clean the duplicates, and extract the ATm values. We retain all the values with ||Tmm|| 2 1 to make sure that the temperature change is large enough and not random noise. Finally, we get 961 data instances with || 1.
Benchmark Quality. Rosetta is an academic framework for computational modeling and analysis of protein structures. We utilize the Rosetta Cartesian AAG application (Park et al. (016) to assess how well-established biophysical computational tools can recapitulate the thermostability dataset FireProtDB. The Cartesian A AAG application calculates the change in the folding energy of a mutation. First, relax the crystal structures of the proteins in FireProtDB with an unconstrained FastRelax as recommended in [emann et al. (2020), which allows the backbone and side-chain atoms move slightly to be better accommodated into the chosen Rosetta score, "ref2015 cart". Cartesian
A21

---

Published as a conference paper at ICLR 2023
Table A5: We report the correlation coeficciett between Rosetta AAG and FireProtDB AT AAG.
FireProtDB
Rosetta
Data Size
Spearman (%)
Pearson (%)
AAG
with backbone relaxation AAG
3,399
44.93
21.36
AAG
w/o. backbone relaxation AAG
3,248
28.59
3.31
ATm
with backbone relaxation AAG
981
-35.91
-3.98
ATm
w/o. backbone relaxation AAG
1,018
-13.54
0.26
AAG then mutates the residues as specified in the database, packs side-chain conformations, and does gradient-based minimization of the atomic coordinates. Using the values from the Rosetta score, we can then calculate the AAG of mutation. Any mutations that failed during the calculation for any reason
discarded.
We calculate the correlation between simulation A AAG and experimental AAG in Table A5 Allow- ing backbone relaxation during structure generation improves the Spearman AAG 44.93 from 28.59 and the Spearman ATm to -35.91 from - 13.54. Compared to Table 2 we observe that our proposal offers better candidate mutations terms of the spearman correlation coefficient.
Training Settings. For zero-shot transfer, we directly apply the models trained HP-S P. For fine-tuning, we train all the models with AdamW optimizer. 5 x 10 weight decay, and the OneCycle learning rate schedule. TAPE models are trained with batch size 8, learning rate 10" 10-3 and 10 epochs, while ESM models are trained with batch size 16, learning rate 10-3 and 20 epochs. For Deep Editing, we use models trained on P. When we finetune ESM models, we re-initialize the final-layer linear head.
D MORE EXPERIMENTAL RESULTS
Table A6: Head Tuning versus Full Tuning of ESM-IB on thermostabilitty classification and regression. 95%
confidence interval computed via the 10-fold evaluation HP-stc2/CS and 3 replicates HP-S.
HP HP-s²C2
HP c²c
HP S
ESM-IB
Accuracy
Spearman
Accuracy
Spearman
Accuracy
Spearman
Head Tuning
91.19
1..901
83.26=1.54
0.7 2.0043
69.5020.16
0 809 001
Full Tuning
88. 75:1.99
0. 0..97=0.029
77.9712.46 46
0.50310.058
65.8810.80
0.615
Head Tuning versus Full Tuning ESM-1B. In Table A6 only tuning the head ESM- 1B consistently achieves better performance than tuning the whole ESM-1L IB across all three datasets. A potential explanation is that the fully tuned ESM-1B tends to overfit due to the relatively small size of HotProtein (182K), where the pre-training dataset has around 22M protein sequences.
Table A7: Hyperparameter tuning of the Mixup aug- mentation. ESM-1B equipped with our proposals is adopted with HP- HP-s2c2 and -D- c55
Table A8: Evaluation results of our proposals with
ESM- 1B on the Meltome Atlas benchmark.
Methods
Spearman
Pearson
Settings/Ace. (%)
HP- HP-S?c2
HP HP-s2c5
ESM-1B Ours
0.4560
0.6866
À ((0.2, 0.11
0.8547
0.8638
ESM-IB
0.3874
0.5331
X B(0.1, 0.2)
0.8600
0.8527
TАPE
0.3076
0.3132
x B(0.2, 0.5)
0.8675
0.8596
Hyperparameter Tuning. general, we find that our method not very sensitive to hyperpa- rameter tuning to yield good results. Specifically: (1) Many hyper-parameters were left at default values. For example, for SAP, we used the hapar-parameters from He et al. (2020) without optimiz- ing. (2) For the other hyper-parameters (e.g. batch size, learning rate), used standard 10-fold cross-vate valsdaaiion for selection. (3) Our algorithm is not sensitive to the choice of papper-arammters, shown in Table 2 our FST achieves considerable improvements for various hyper-parametae
A22

---

Published as a conference paper at ICLR 2023
choices. (4) We provide more ablations for Mixup augmentation in Table A7 where À is driven from beta distributions B.
Our Proposal Other Benchmarks. To demonstrate the generalization ability of our methods, we further perform the zero-shot transferring on Meltome Atlas (we train the model on HotProtein and directly evaluate the model's performance on 8K Meltome Atlas data). We notice that our method still achieves better results in Table A8
Table A9: Comparison with baselines of MLP ESM-IB without any pre- trainin..
Table A10: More approaches inject the structure information with ESM-IB.
Settings/Ace. (%)
3P-3Pc2
HP-S²c5
Settins/Ace. (%)
HP sPcc
HP-Sac5
ESM-IB IB pre-training
0.7808
0.6652
Ours
0.9236
0.8625
ESM-IB IB pre-training
0.91 19
0.8326
Add
0.8873
0.8093
MI.P
0.6931
0.6378
Concat
0.9001
0.8186
More Baselines: Small Neural Networks and ESM-1B without Pre-training. We conduct extra experiments with 1) ESM-1B without the pretraining weights (with times larger number of training iterations than an MLP's) and 2) a small neural network, 3-layer MLP (256 Dimensional Embedding Layer - 3 Layer MLP + Average Pool) in Table A9 results than ESM- IB baseline.
These two approaches achieve much worse
Other Approaches to Inject Structure Information. As shown in Table A10 we notice that directly concatenating adding the final-layer representations of ESM-IF (structure information) and ESM-IB (sequence information) comes to slightly worse results.
More Comparisons: End-to-end Freezing Some Layers Tuning. We provide additional re- sults with ESM-IB on HP-S* cp2s$cc and HP cP- "First 1 / Layers" indicates the layers close to the input. To achieve better results, we tune the backbone learning rate with 10-fold valsdavai for these numbers. Table All us that only tuning the head leads to superior performance.
Evaluation on Balanced Test Sets. We perform extra evaluations on manually balanced test sets. Results are summarized in Table A12 where our methods consistently show superior performance.
Table All: Comparison with the end-to-end and par- tially frozen tuning with ESM-IB on HP HP-S"c2/S /5.
Table A12:
Evaluation on manually balanced
datasets.
Settings/Acc. (%)
HP HP-sPc2
EP-S2c5
Settings/Ace. (%)
HP-33Pc2
HP-sPc5
HP-S
End-to-End
0.8875
0.7797
3D GCN
0.7633
0.5004
Freeze First Layers
(188977
0.8018
TAPE
0.8117
0.5535
0.6370
FSM-IFI
0.7752
0.6032
Freeze First 2 / Layers
(1.9011
0.8195
ESM-1B
0,8605
0.8029
0.6980
Tuning Heads
0.9119
0,8326
ESM-1B Ours
0.9308
0.8294
0.7517
Tuning Heads Ours
0.9236
0.8625
Table A13: Ablation the feature aggregation methods with ESM-IB HP-S*c2 and HP- c
Table A Comparisons with the & and & regular-
ized tuning for ESM-1B #C-
Settings/Acc. (%)
HP-S?c2
HP-S?c5
Settings/Acc. (%)
HP HP-$$c5
Average Pooling 0.9119
0.8326
ESM-1B 1B
0.8326
Max Pooling
0.8819
0.3750
ESM- IB b2 (Ridge Regression)
0.7375
No Pooling
0.9084
0.7808
ESM- 1B L (Lasso)
0.3808
Ablation on the Feature Aggregation Methods. In Table A13 we conduct ablation studies on the feature aggregation methods with ESM-IB on HP-S2c2 and HP- c22, including average pooling, max pooling, and no pooling. The results are summarized in the below table. We observe that the average pooling outperforms other aggregation options.
A23

---

Published as a conference paper at ICLR 2023
Comparison to &g (Ridge Regression) and & (Lasso) Regularization. We conduct comparisons to ly (ridge regression) and & (Lasso) regularization with ESM-IB IB on HP-s²c5. Results are in Table A14
We observe that additional regularizers may lead performance degradation.
Additional Results on Editing In Table we use to 5-class classifier trained on HP-S, and provide additional results with 2-class classifier trained HP-SC2. We get zero-shot accuracy 65.66%, precision 42.77% and successful rate 42.14 14.. After fine-tuning, we get accuracy 70.82 10.8%% 0.32%, precision 48.31 + 0.29% and successful rate 53.05 + 0.26%. The result is slightly worse than the 5-class classifier results.
E MORE DATASET DETAILS
Data Collection and Process. All the data in the NCBI bioproject is accumulated from all scientists who publish data that makes it into the NCBI database. Thus, there is lot of duplication and variation in data entry. Due to the vast number of organisms and multiple strains for organisms, removed duplicates by taking the first observation in the NCBI bioproject that had consistent quantitative optimal growth temperature and temperature classification.
We remove all sequences that are greater than 1500 amino acids, since most proteins of interest for engineering fall within this range and it greatly simplifies clustering. During the clustering, cluster across all organisms in the same temperature bin in order remove redundancy across organisms and kept representative sequences from each cluster in order to not bias sampling to specific organism.
There are numerous organism-speeific idiosyncrasies present (such as sequence homology between organisms due to their evolutionary relationship) each organism proteome that has nothing to do with thermostability and instead their unique adaptation to their environment. However, there no clear-cut way to identify these features/sequences and remove them. Therefore, we expect there to be numerous organism-specific idiosyncrasies present (such as sequence homology between organisms due to their evolutionary relationship) each organism proteome that has nothing to do with thermostability and instead their unique adaptation to their environment. In practice, we do not further filter the data to reduce cvolutionary differences. Instead, we cluster proteins based on sequence similarity across all organisms within temperature bin and select the representative sequence of each cluster to avoid biasing our dataset to a particular organism.
Domain Shift. For domain shifts, it includes two perspectives: Between HotProtein and Fire- protDB. Annotating a proteome with the optimal growth temperature of the organism provides us a lower bound soft label for training nearly 200K sequences (i.e., HotProtein). However, expect to observe a domain shift when fine-tuning on experimentally curated stability datasets (i.e., Fire- protDB) since these labels accurately represent a protein variant thermodynamic properties, unlike our coarse, optimal growth temperature label. We use the classic tool, i.c., ed-hit to compute the protein sequence similarity (1) between FireprotDB and Hotprotein: (2) within FireprotDB. We find the similarity between FireportDB and Hotprotein (0.1928) is much lower than the one within Fire- protDB (0.2504). It suggests the substantial domain shifts, echoed with our paper's descriptions. e Within HotProtein. HotProtein covers 230 in different species where proteins have different func- tionality and structures. For example, we expect to observe domain shifts between proteins sampled from the eukaryotic species (primarily unicellular fungal) and the prokaryotic organisms.
5
Data Scarcity. For the data scarcity, we measure our framework on two kinds of data lata-limited scenar- ios: (I) We have evaluated our proposals on subsampled HP-S* -s2c and HP- s²c5, and demonstrated their effectiveness. HP- and HP- only have around 2K sequences ( 189 of the whole Hot- Protein dataset). (2) We further examine our approaches FireprotDB which is manually curated database of the protein stability data for single-point mutants. It only contains 0.2K natural protein amino acid sequences which serve the training set in our case. Table 4 shows that our methods lead to consistent performance improvements.
An Organism Lives is Correlates with the Thermostability of the Protein Sequences of that Or- ganism. The Meltome Atlas provides thermal proteome profiling (TPP) for 13 organisms (Jarzab al. 2020). Although TPP not the melting temperature of the protein since it is not measured
http:/ http://weiiie http:://wwizzoggieeeee 1ab. ucad. edu/ co-hit///
A24

---

Published as a conference paper at ICLR 2023
with purified proteins and requires the protein to become insoluble upon denaturation (proteins remain soluble even after denaturation), it provides empirical evidence that for mesophilic, and ther- mophilic prokaryotic organisms, the optimal growth temperature of organism is very close to the lower bound of the thermal stability of that organism. This not the case for eukaryotic and psychrophilic prokaryotes where the Meltome Atlas shows č 10C gap between the optimal growth temperature and the lower bound of protein stability in these organisms. Thus, given the available proteome experimental data, we assume that using the optimal growth condition for prokaryotic mesophiles and thermophiles provides an accurate lower bound melting temperature for their re- spective proteome. Furthermore, although our method utilizes the lower bound of a proteome's thermal stability as a label, we demonstrate that it provides sufficient learning signal to improve performance on the manually-curated experimental dataset: FireProtDB.
F DISCUSSION OF BROADER IMPACT
This research aims to predict the thermal stability of proteins from the sequence and structural data and predict thermostabilizing mutation designs. Improving protein's robustness to thermal chal- lenges can often be the deciding factor for the commercialization of a biocatalyst. Furthermore, the ability to rapidly thermostabilize protein will open the door for the engineering of a broader range of biotechnologically relevant enzymes and therapeutics, which can have a profound impact on the chemical, agricultural, food, and pharmaceutical industries. It is well documented that func- tional residues tend to be destabilizing residues and engineering function into a protein can quickly destabilize the protein. Thus, it is common to first improve a protein's 's stability prior introducing destabilizing mutations. Our work attempts to accelerate the initial stabilization target protein enable downstream functional protein engincering.
H2 H2-sPc2
HP c2c
HP
ESM-IB
Accuracy
Spearman
Accuracy
Spearman
Accuracy
Spearman
Full Tuning
88.75
0. 0.009
77.97:2.46
0.008
65.88:00.8
0.615:10.550
Head Tuning
91.19
D.000
83.26:11.54
0.712
69.50:10.16
0.009
Ours
91.91 10.64
0.892 0.011
86.08 1.33
0.747 -00026
73.0990.10
0.823 100001
A25


Reviews:
[{"review_id": "fiI2jPyB4S", "reviewer": "Reviewer_jaKq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework for protein thermostability prediction. A large-scale protein dataset with organism-level temperature annotations is curated, and one pretraining and one tuning module are proposed for prediction.\nThe main contributions are:\n1.\tA dataset of protein sequences and folded 3D structures data with temperature annotations.\n2.\tIntroducing a protein structure-aware pre-training module for protein-related tasks.\n3.\tProposing a factorized sparse tuning module for thermostability prediction.\n", "summary_of_the_review": "The paper is clearly written, and the experiments are comprehensive. However, it's hard to say whether the model would really be useful in resolving biological problems. ", "strength_and_weaknesses": "Strengths:\n1.\tThe amount of data for thermostability-related tasks is greatly expanded.\n2.\tThe writing is generally clear, and easy to read.\n3.\tEditing proteins to improve thermostability is interesting, biological experiments can be added in the future.\n4.\tThe experiments on the model are relatively comprehensive.\nWeakness:\n1.\tTheir pretraining module simply combines the sequence pretrained model and the 3-D structure-aware pre-trained model. No novelty on the loss function can be viewed.\n2.\tTwo reasons mentioned for using factorized sparse tuning for this prediction task are not convincing. The sentence ‘while COVID-19 related proteins usually contain more than 1,000 amino acids’ has no strong connection with the studied problem.\n3.\tFrom the result, the framework seems just introduce some training strategies based on ESM-1B, and there is no design in biology for the thermostability prediction task. Besides, no significant improvement in results is observed.\n4.\tThe description of biology in this paper is superficial, and the study lacks interpretability analysis.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Bbg-mI0Onv", "reviewer": "Reviewer_un7C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a new model for predicting protein thermostability based on a published dataset (Jarzab et al Nature Methods 2020) and create three derived different datasets annotating each of proteins with a class name that reflects the temperature that the organism lives at  (retrieved from NCBI). The task then becomes the ability to predict the correct class. They present two algorithmic ‘advances’: structure-aware pretraining and factorized sparse tuning  improve thermostability prediction. After evaluating compared other methods for prediction they also propose a method to generate sequences with increased thermostabilty. \n", "summary_of_the_review": "For the reasons given in the Weaknesses section I don't believe this paper achieves what the authors would indicate are their goals - thermostability prediction and design.  The fact that they do not test generalizability, limits the application of this work to where there are very large datasets  of experimental results on related proteins from environmentally related organisms. I would suggest that the authors rethink their evaluations, test with much more naive baselines. ", "strength_and_weaknesses": "A fundamental weakness of this paper is similar conceptually to many papers submitted to ML conferences on biological data -that is the way the train-test splits are made which is 10 fold cross validation. What this means is that they have ‘data leakage’ relative to what it seems they claim to predict -- maximally they show if you had 90% of the data with related  sequences in the split, you could predict the missing 10%. Not a very useful predictor Now imagine a test split that removes a family of sequences that has no sequence relation to training data - then one could say a load more about the ability of the model to generalize \n\nWith thousands of proteins many of which are homologous i.e. similar  and very few classes in theri stes (2 or 5 )  - it's hard to see why one even needs a language model when a simple sequences similarity search as a baseline could do the same. Have the authors tested a simple baseline?\n\nThe sentence in the abstract “We present HotProtein, a large-scale protein dataset with growth temperature annotations of thermostability..” strongly suggests  they are reporting this dataset here for the first time - when in fact it was published 2 years ago and is publicly available. Since they do in fact clear the data for more facile use, the statement should make that distinction clear.  \nRelated work seems to be missing - especially where methods have collected datasets but also run models to predict mutation effects - some of which are thermostablity related ; since some of these are even unsupervised, using only evolutionary sequence alignments - minimally a  discussion of this body of work is warranted eg - see ProteinGym ( Tranception ICML 2022)  plus older papers that use VAEs ( Riesselman Nature Methods 2018)  and even Potts Models (eg Hopf et al 2017 Nature Biotech) plus MAVEsdb.  In addition FLIP ICML 2021 and 2022- Learning deep representations of enzyme thermal adaptation. Why no comparison  to  ESM1v ( Meier 2021), Tranception ( Notin 2022), FLIP 202, DeepET (2022) ?\n\n\nMinor \nExplain why theory are not using the Meltome measurement as their thermostability label\nAlthough it’s fine to explore whether using structural information improves existing approaches , the sentence on page 4  is misleading for different reasons -- “These models achieve considerable improvements on amino acid prediction tasks, e.g., contact prediction, mask prediction (Brandes et al., 2022), mutational effect prediction (Meier et al., 2021).” Compared to what? Meier’s results that the authors reference are great  BUT are only very marginally better than a VAE ( DeepSequence 2018 Nature Methods) and not as good as Tranception ( Notion et al,  ICML 2022) and are zero shot!! Using only sequence info. \nThe structure barely improves the results - why do they think that is the case? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8IsBOoz6fbO", "reviewer": "Reviewer_Uvn5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the prediction of protein thermostability and the design of more thermostable proteins. After introducing a new benchmark to train and assess thermostability prediction models, authors describe a contrastive-learning framework to impart protein structure information on representations learned by large transformer networks usually trained on sequence alone (eg., ESM-1b). The paper also adapts ideas from other domains to carry out robust fine tuning and protein editing.", "summary_of_the_review": "Overall a very thorough and interesting work. Some concerns on validation as discussed in the strengths / weaknesses section. I am leaning accept overall given the several meaningful contributions, in particular the HotProtein dataset and the structure-aware pre-training. I would be willing to recommend acceptance more enthusiastically if the aforementioned concerns are addressed during rebuttal. \n\n--------------------------------------------------------------------------------------------------------\n[UPDATES POST REBUTTAL]\n\nThank you to the authors for their thorough responses during rebuttal. My main concerns have been adequately addressed (in particular regarding validation / potential data leakage). I have also read in detail the assessments from the other two reviewers, and believe their comments have also been properly addressed in reviews. I do believe that the newly introduced HotProtein benchmark, together with the methodological contributions (structure aware pre-training) and various adaptations from other domains to protein modeling (eg., factorized fine tuning and feature augmentations) represent a solid contribution. In my opinion this paper should be accepted to the conference, and I have increased both my score for correctness and overall appreciation accordingly. \n", "strength_and_weaknesses": "**Strengths**\n- The work is very thorough with many ablations to illustrate the benefits from the different modeling decisions suggested by the authors\n- There are several novel contributions, in particular the Hotprotein benchmark and the structure-aware pre-training. The work also borrows ideas introduced in other fields to address (in a novel way) problems in protein prediction models and protein editing.\n\n**Weaknesses**\n- The main flaw I see in this paper is with respect to model performance validation. Unlike in many other application domains, protein sequences are not iid samples from the underlying data distribution as there is a lot of dependency between sequences that share common ancestors. This calls for particular cross-validation schemes to mitigate data leakage between training and test sets (eg., ensuring that sequences are not too similar), especially in several of the (semi-)supervised settings discussed in this work. See for example [1] for some ideas on how to do this. Similarly, it would be helpful if authors could comment on the similarities / overlap between the introduced Hotprotein dataset and the other datasets used in fine tuning / editing experiments (eg., FireProt and Meltome Atlas) -- if no substantial overlap, could you please specify that in supplementary?\n- The paper is not very clear at times. For instance, the reasoning / rationale that gave rise to the equations 1 and 2 in section 4.1 is absent\n- Not really a weakness, but rather a missed opportunity: since the SAP and fine-tuning frameworks are general and agnostic to thermostability, more extensive validation across protein families & fitness / protein attributes could be carried out to validate generalizability of the introduced ideas beyond thermostability. See for instance large-scale fitness benchmarks like ProteinGym [2].\n\n------------------------------------------------------------------------------------------------------------\n\n[1] Magńus Halld́or Ǵıslason,  Felix Teufel,  Jośe Juan Almagro Armenteros,  Ole Winther,  and Henrik  Nielsen.   Protein  dataset  partitioning  pipeline.   2021b.   URL https://github.com/graph-part/graph-part\n\n[2] Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A.N., Marks, D.S., & Gal, Y. (2022). Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. ICML.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster