Metalic: Meta-Learning In-Context with Protein Language Models
Authors: Jacob Beck
Shikha Surana
Manus McAuliffe
Oliver Bent
Thomas D Barrett
Juan Jose Garau-Luis
Paul Duckworth
Venue: ICLR 2025 Poster (2025)
Keywords: Meta-Learning
In-context Learning
Tuning
Protein
Fitness Prediction
Property Prediction
Protein Language Model
PLM
Large Language Model
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce Metalic, an approach that uses foundation models, in-context learning over related tasks, and fine-tuning for protein fitness prediction. Metalic achieves state-of-the-art results on ProteinGym with less computation and fewer parameters.

Abstract: Predicting the biophysical and functional properties of proteins is essential for in silico protein design. Machine learning has emerged as a promising technique for such prediction tasks. However, the relative scarcity of in vitro annotations means that these models often have little, or no, specific data on the desired fitness prediction task. As a result of limited data, protein language models (PLMs) are typically trained on general protein sequence modeling tasks, and then fine-tuned, or applied zero-shot, to protein fitness prediction. When no task data is available, the models make strong assumptions about the correlation between the protein sequence likelihood and fitness scores. In contrast, we propose meta-learning over a distribution of standard fitness prediction tasks, and demonstrate positive transfer to unseen fitness prediction tasks. Our method, called Metalic (Meta-Learning In-Context), uses in-context learning and fine-tuning, when data is available, to adapt to new tasks. Crucially, fine-tuning enables considerable generalization, even though it is not accounted for during meta-training. Our fine-tuned models achieve strong results with 18 times fewer parameters than state-of-the-art models. Moreover, our method sets a new state-of-the-art in low-data settings on ProteinGym, an established fitness-prediction benchmark. Due to data scarcity, we believe meta-learning will play a pivotal role in advancing protein engineering.

Published as a conference paper at ICLR 2025
METALIC: META-LEARNING IN- WITH PROTEIN LANGUAGE MODELS
Jacob Beck, Shikha Surana, Manus McAuliffe, Oliver Bent, Thomas D. Barrett, Juan Jose Garau Luis, & Paul Duckworth
InstaDeep https:: / thub. com/ /tsstaeeeea/mmttalllea Boston, MA, USA & London, UK
ABSTRACT
Predicting the biophysical and functional properties of proteins is essential for in silico protein design. Machine learning has emerged as promising technique for such prediction tasks. However, the relative scarcity of in vitro annotations means that these models often have little, or no, specific data on the desired fitness pre- diction task. As a result of limited data, protein language models (PLMs) are typically trained on general protein sequence modeling tasks, and then fine-tuned, applied zero-shot, to protein fitness prediction. When no task data is available, the models make strong assumptions about the correlation between the protein se- quence likelihood and fitmess scores. In contrast, we propose meta-learning over distribution of standard fitness prediction tasks, and demonstrate positive transfer unseen fitness prediction tasks. Our method, called Metalic (Meta-Learning In-Context), uses in-context learning and fine-tuning, when data is available, to adapt new tasks. Crucially, fine-tuning enables considerable generalization. even though it is not accounted for during meta-training, Our fine-tuned models achieve strong results with 18 times fewer parameters than -fate-of-the-art mod- els. Moreover, our method sets new state-of-the-ar in low-data settings on Pro- teinGym, an established fitness- ciness-predictio benchmark. Due to data scarcity, we believe meta-learning will play pivotal role in advancing protein engineering.
INTRODUCTION
The accurate prediction of functional and biophysical properties of proteins, collectively referred to here as fitness, is a critical challenge in the physical sciences with far-reaching implications for medical research, agriculture, and drug discovery. For example. fitness prediction can be used to optimize properties such as the binding affinity of a monoclonal antibody therapy to its target or the thermostability of enzymes functioning at high temperatures. While protein fitness can be measured in vitro, the process laborious and coneccnnsuming. Consequently, machine learning models have emerged as powerful tool to predict fitness directly from amino acid sequences in silico. However, due to the complex, high-dimensional relationship between protein sequences and fitness, and the limited availability of high-quality data, accurate fitness prediction is challenge.
Proteins can be encoded as sequences of characters representing amino acids, making protein lan- guage models (PLMs) effective for modeling them (Madani et al. 2020; Rives et al. 2021 Rao cct al., 2021 Lin et al. 2022 Notin et al., 2023 Truong Jr & Bepler, 2024). By predicting masked amino acids, subsequent amino acids, over known proteins at scale, PLMs can capture much of the structure and resultant properties deriving from the amino acid sequence. While PLMs are not directly trained to predict fitness, they are trained to model the likelihood of naturally occurring pro- teins, which has been found to correlate strongly with their fitness (Truong Jr & Bepler) 2024). In practice, large PLMs are fine-tuned on downstream protein fitness data for regression. While highly effective, PLMs provide limited utility given severely limited data, or no data. With limited data, learning the correct regression is difficult, even with informative pre-trained representations. With no data, it must be assumed that protein fitness is solely function of protein likelihood (Truong Jr & Bepler, 2024) Hawkins--oooer et al. 2024), and for masked language models, it also assumed that each amino acid contributes independently to fitness (Meier et al. 2021 Notin et al. 2024).
While limited or no data may be available for specific protein fitness prediction tasks, there are often other tasks that can serve as a valuable source of guidance. For example, rather than assuming
"Jacob Beck @ alummi.brown.cd..

---

Published as a conference paper at ICLR 2025
Pre-training Protein language modeling
Meta-training Fitness prediction
Fine-tuning Fitness prediction
NVVI SWA.STAATAONN TNAKA
My THAAK
10
Conventional protein fitness prediction approaches
ANGASTDAAAAC
21
GAEALEPHFLSE SF2
00
TIKTVENNITE SH
TIKTVEPNETE
AKCIT RIVE
SULEPADE TNUKA WNMCEE
ACAL MAMLNNNAA-VVO
0.1
MYL TNAYA
MINAA 54 501
21
Our approach
Un TIKTYE SH GSADVMSHOKRY
BHCLIVTAAHLPA
FETRAVIASI DNF1
Multiple tasks
Amount data needed
Figure 1: An overview of meta-learning for protein fitness prediction. PLMs are trained over massive quantities of unlabeled data. Using meta-learning, we also train over smaller quantity of labelled fitness data. Using this extra data is critical given limited data for fine-tuning at test time.
that fitness is solely function of protein likelihood, when data is available for given task, we can use data from other tasks to learn the relationship between PLM embeddings and fitness. Due to advances in high-throughput assays, such as deep mutational scanning Fowler & Fields (2014). data from other tasks is available to learn this relationship. One example is ProteinGym (Notin al. 2024),, which compiles over one hundred distinct fitness prediction tasks. Training over such a distribution of related tasks, to reason about new tasks, is referred to as meta-learning (Huisman
jet al. 2021 Hospedales et al. 2021; Beck et al., 2023a).
To address the challenge of limited data in protein fitness prediction, we propose Metalic, integrat- ing in-context meta-learning, protein language models, and fine-tuning. Metalic builds on top of PLM embeddings and fine-tuning methods, but additionally, unlike baseline methods, adds meta- learning phase over other protein prediction tasks. While some PLMs use in-context data (Rao et al.) Truong Jr & Bepler 2024), these methods do not meta-learn how to use their limited context for protein fitness prediction. The meta-leaming phase is depicted in Fig. [. and is critical to learning the relationship between PLM embeddings and fitness, given the limited fit- ness data available in each task at test time. While some meta-learning methods combine in-context learning and fine-tuning (Rusu al. 2018 Vuorio al. 2019), they do so by using computationally expensive higher-order gradients to account for tine-tuning during meta-learning Crucially. Metalic leverages in-context meta-learning, and then subsequent fine-tuning. without accounting for fine- tuning during meta-learning. This novel combination is particularly computationally efficient, and enables Metalic to outperform more complicated methods for meta-learning.
2021 Notin et al. 2022 2023
We present Metalic, an in-context meta-learning approach to tackle the problem of protein fitness prediction low-data settings, and make the following contributions:
We introduce a method that efficiently combines in-context meta-learning with PLMs and fine- tuning for protein fitness prediction.
We advance state-of-the-art (SOTA) for zero-shot protein fitness prediction on the ProteinGym benchmark (Notin al. 2024).
We attain strong performance for few-shot fitness prediction with 18 times fewer parameters.
We ablate each component of our method to demonstrate the contributions of each part and underscore their necessity.
We empirically validate the superiority of our method to alternative forms of meta-lcarning.
2 RELATED WORK
Meta-Learning Meta-leamning aims to create sample-efficient learning algorithm by training
over a distribution of tasks. The goal is to learn algorithms such that they can rapidly adapt to new tasks during inference. This inference-time adaptation is often called the inner loop, for which there are two primary forms found in the literature: gradient- gradient-bassed meta-learnigg (Finn et al. 2017 Zintgraf et al. 2019) and in-context meta-learning (Mishra et al. 2017 Nguyen & Grover 2022)
2

---

Published as a conference paper at ICLR 2025
tradient-bassee and in-context algorithms differ both in their computationnl efficiency and capacity for -iittr--iitiitttinnn generalization. Gradient-baie based approaches explicitly adapt model parameters within in the inner loop using standard gradient-based learning. Commonly, parameter initializa- tion is learned that can be adapted to new tasks with only a few gradient steps (Finn et al. 2017 Zintgraf et al., [209 Vuorio et al., 2019). However, this comes with considerable computational overhead - due to meta-gradients when differentiating the inner loop which makes gradient- meta-learning less suitable for large models. Alternatively, in-context meta-learning adapts by con- ditioning a sequence model on task-specific dataset in context. These methods condition on the data points over which gradient- based approaches would train (Santoro al.] 2016 Mishra et al.] 2017 BBek et al. 2024 2023b). Such methods are typically more sample- and compute-efficient than gradient-based methods, but perform worse on out-of-distribution tasks given the lack of ex- plicit gradient-based learning in the inner loop (Beck et al. 2023a). Rather than training for in- context learning, the in-context learning of pre-trained large language models can also be used to perform meta-learning (Coda-Forno et al. 2023); however, this lacks generalization guarantees and requires fitting all tasks in context simultas Itaneousl, which not possible given our data (Section (.1).
In Metalic, we only train for in-context meta-learning, but find this is still compatible with task- specific tine-tuning. In the meta- reitforeinorr learning setting, this combination has been shown to be possible by increasing task-specific data for fine-tuning (Xiong al. 2021). In contrast, uate in the supervised setting and do not give increased data at inference time. While prior work has combined gradient- gradient-based and in-context meta-learning (Rusu et al., 2018:) Vuorio et al. 2019). these works compute expensive meta-gras meta--radients to learn how to account for fine-tuning. Despite not ex- plicitly meta-learning gradient-based adaptation, we find in-context meta-learning alone provides a strong foundation for subsequent fine-tuning and that both aspects are critical for high performance.
eval-
Likelihood-Basee Fitness Prediction with PLMs Leveraging pre- pr-itrined PLMs standard prac- tice in protein fitness prediction (Rives et al., 2021] Notin et al. 2023: Rao et al. 2021 Truong Jr & Bepler] 2024). the few-shot setting, PLMs intended for sequence generation are repurposed by fine-tuning for protein fitness prediction (Rives et al.
202T). In the zero-shot setting, it is assumed that the fitness correlates with the likelihood of the proteins associated sequence of amino acids, as predicted by PLM (Meier al.) [021 Truong Jr & Bepler 2024). Furthermore, if using a masked PLM, it is often assumed that each amino acid contributes independently to the fitness (Meier et al., 2021. In this work, we likewise leverage PLMs for protein fitness prediction. However, in contrast, we make use of additional data in the form of additional fitness prediction tasks on other proteins. Specifically, we meta-learn how to use PLM for protein fitness prediction, rather than relying on assumptions. Only after meta-learning, do we fine-tune our model, as depicted in Fig. T Meta- learning across tasks lets us avoid restrictive model constraints and achieve SOTA performance. We will show that in-context meta-lcarning is necessary to achieve strong results in low-data settings.
In-Context PLMs We build upon existing PLMs that make use of in-context data for protein fitness prediction (Notin et al. [022] Truong Jr & Bepler 2024 Notin et al. 2023 Rao et al.. 2021) However, these methods do not meta-learn how to make use of their context. These methods either learn to use the context only for protein language modelling, and then assume that the likelihood from the generative model correlates with fitness (Truong Jr & Bepler [024]] Notin et al., 2022 Rao et al.) 202T), or they use the context for protein fitness prediction, but not by meta-learning over protein tasks (Notin et al., 2023). Of these ProteinNPT (Notin et al. 2023) is the most related to our method, since we use the same attention architecture condition related proteins in-context, and it uses gradient steps to fine-tune to the target task. In comparison, method meta-learns over many tasks how to make use of the fitness information, which we find to be critical (Section 4). Additionally, our method is the first to use the aforementioned procedure to allow fine-tuning and in-context conditioning on the very same context at inference time.
fitness information about
3 METHODS
3.1 PROBLEM SETTING
We consider a fitness prediction task, T. to be defined by a dataset of the form Dr ((2, ((2.,,=ff(2.) (x where I; is sequence of amino acids, and y, E R is the associ- ated scalar fitness value assigned by the (unknown) underlying fitness function fT. In the few- shot setting, the task-specific data is typically split into non-overlapping support and query sets:
3

---

Published as a conference paper at ICLR 2025
Finress predictiona for the query set
Alrrseee T
Aed dttrition
B
DAP PUPEA oFF
Pheckdaree ceet the orggial query
Aucal attentoon
crboodd gg
The freeerting support and quary ants ceed in secwal gradient
Bct attentio computed over rows
FLM
PLM
Support
the zero shot seling nn iuppert
Support
Muliple fine fenieeunnngg support from the support set
mimihiiouss S1
sct
the model
Muttal protch
prodation
(a) Meta- Meta-training
(b) Fine-tuning
Figure 2: An overview of Metalic. Meta-training our model over many protein prediction tasks enables in-context learning (a). Fine-tuning the in-context learning the support set requires sub- sampling smaller support and query sets, and enables generalization at test time (b).
D. (( Di ) and D = {(x:-) V, ) such that D(S) n D( = 0, with sup- port and query set sizes, NS)) and N(2), respectively. The support set provides data for task- specifce adaptation, and the query set provides data for evaluating the adapted performance. The size of the support set is called the shot. Note, even with an empty support set (zero-shor), the model can still adapt to the task using information from the query set, i.e., related sequences without fitness scores.
Meta-learning for protein fitness prediction requires not just a single task, but multiple tasks, D Di,... DJ over which to learn. The full dataset of tasks, D, can be seen as defining a distribution of tasks which can be split into training and test tasks in the usual way. Concretely, for meta-learning in-context, the goal is to learn a function with parameters conditioned on the full support set and (1) N( D(S)). ). In our case, rather than directly predicting fitness values, we instead follow prior works that use a preference-based objective to rank the query
unlabelled inputs from query set, fo((2; Ji=1
set in order of fitness (Krause et al. 2022 Brookes et al. 2023 Hlawkins-Hooke et al. 2024).
3.2 METALIC
Architecture Our work leverages the ProteinNPT architecture proposed by Notin et al. (0023) as an in-context PLM for fitness prediction. Here briefly summarize the key elements, but defer the reader to Appendix A and the original paper for full details. The architecture, along with the data use for meta-leaning, are illustrated in Fig. 2a A detailed illustration give in Appendix A.6
Protein sequences in both the support and query set are converted to per-residue embeddings us- ing a pre-trained PLM; our case we take the third layer of ESM2-8M (Lin al.) 2022). Fitness scores in the support set are projected to match the dimensionality of the residue embeddings using linear layer. while the query set fitness embeddings share single learned embedding. The protein sequence embeddings and fitness embeddings are then concatenated along the sequence dimension. Optionally, zero-shot fitness predictions from an auxiliary PLM can be embedded and concatenated in the same way, which we explore in Section 4.3 This tensor is then processed via axial attention blocks (Ho et al. 2019), each of which applies self-attention separately along and across the se- quences. Axial attention reduces the computational complexity of self-attention from 3(K2 to O(K² + L2), where K is the shot and L is the length of a protein. Finally, a multi-layer perceptron conditions the fitness embedding and mean-pooled sequence embedding to predict each query value, i.e. (v! ( = fo((a, (Q) ( D(S)), which is then used to rank the proteins by fitness.
Meta-Training Following prior works that use a preference-based objective (Krause et al. 2022 Brookes et al. 2023 -lakkins-Hooker Hooker et al. 2024), we reframe the relative score prediction of two
sequences as binary classification, predicting whether sequence x;
(Q) has higher fitness than (Q)
yi ( > yj (2)
= a (1) 1/
( (Q)
(1)
4

---

Published as a conference paper at ICLR 2025
where is a sigmoid function and the dependency of the query values 0 has been dropped for brevity. This classifier is optimized with respect to every pairwise comparison between sequences in the query set corresponding the loss function:
Ni (C
[(0, "0. - - Σ Σ
( 1i
11
100) logo (110) -))))
(2)
J
where I is an indicator function. Intuitively, this optimising ( x (N 1) binary clas- sification problems. Note that we only compute the loss over the query set to avoid encouraging memorization of the support set. Adapting this to meta-learning (Fig. 2ap. the objective becomes to find the parameterization that minimizes the loss across the task distribution,
J(0,D) = - EED D JDD, C(O, DJ Q .D.)
(3)
Fine-Tuning Metalic uses fine-tuning, during inference, in order to enable generalization, without having to account for fine-tuning during meta-training. This process is depicted in Fig. 2b
The combination of in-context learning and fine-tuning creates a unique problem. Since fine-tuning occurs at inference time, labels for the query set, {u. (Q) J are not available for training. While
labels for the support set, {v. (,,))) ) are available, propagating gradients from the support set would encourage memorization of the support set, since the labels are also passed as input in-context. While prior methods that combine in-context and gradient-based Vuorio et al. 2019) would encounter this issue, this problem is exacerbated for Metalic. Whereas prior methods compress inputs to a representation with a constant number of dimensions, Metulic uses self-attention, which scales with the number of inputs, allowing them to be stored without compression. Moreover, whereas prior methods use meta meta-gradies graddgradsnnss that could adjust the gradient update procedure so to be useful for generalization and not memorization, Metalic does not take into account the fine-tuning process during meta-training.
meta-learning (Rusu et al. 2018
We address the issue of memorization by sub-sampling from the support set. The fine-tuning pro- cedure is the same as during meta-training, with the exception that the support set is sub-sampled. In order compute updates on single support set, the support set is sub-sampled into multiple
smaller support and query sets, D(S') C DDS)) and D(Q') C D Concretely, this corresponds to fine-tuning on unseen data using the objective:
J(0, (8) -B (Df"" "ipp",pp JED() [(0, L(0. "D("))
(4)
After fine-tuning, Metalic conditions on the complete support set, allowing no data to go waste.
Using Metalic's unique combination of in-context meta-learning followed by fine-tuning, we enable the generalization of extensive fine-tuning, while also precluding expensive computation. If a typical gradient-based meta-learning method requires 0(m) meta- meta-gradients and 0(mn) regular gradients for meta-training, Metalic requires no meta-gradients and ((m) regular gradients, constituting linear reduction with superior performance to efficient alternatives, as demonstrated in Section 4.6
4 EXPERIMENTS
In this section, we evaluate Metalic on fitness prediction tasks from the ProteinGym benchmark (Notin et al. 2024). We evaluate in the zero-shot setting with no support data, and the few-shot setting with limited support data. To establish SOTA results in the zero-shot setting, we first compare to the predictions provided by ProteinGym for the baseline models. To establish strong performance in the few-shot setting, since predictions are not provided, we train baselines from Hawkins- Hooker jet al. (2024). While Metalic does not achieve SOTA results evaluations on proteins that have multiple mutations (multi-mutant proteins), we demonstrate that the performance grows as we add more meta eeta-ttaining tasks, providing a path forward for Metalic in the mutti- setting in the future. We perform ablations of Metalic, to show the benefits of meta- vecaa-le in-context learning, and monintunng. Finally, we compare to the gradient- based method, Reptile (Nichol et al., 2018). to show that taking account of gradients during training an unnecessary computational burden.
5

---

Published as a conference paper at ICLR 2025
4.1 EXPERIMENTAL SETUP
In our experiments, we focus on ProteinGym deep mutational scans. Each task in ProteinGym each measures one property on a set of proteins that all differ by one amino acid, or multiple amino acids, from a reference wild-type protein. We have 121 single-mutant tasks and 68 multi-mutan tasks from ProteinGym. From these, we evaluate over eight held-out single-muten mutant tasks, and five held- out multi-mutant tasks, DollowingNNotin et al. (2023); Srewkins-Hooker et al. (2024). This leaves 113 single-mutant and 68 multi-mutant tasks for training when evaluating single mutants (Sections 4.2 and 4.3). and 121 single-mutant and 63 multi-mutant tasks for training when evaluating multiple mutants (Section (.4). All fitness values are standardized by subtracting the mean and dividing by the standard deviation by task. Note that there are additional tasks in ProteinGym we do not consider. Specifically, we do not consider multi multi-mutant tasks that have overlapping proteins with single single-mutant mutant tasks, to make evaluation more difficult. We also ignore additional tasks in which the maximum protein length is > 750, to fit the backward pass on an Nvidia A A100-80Gb device.
We use a query set size of NQQ = 100, and the size of the support set determined by our evaluation setting and is one of three sizes: N(8) 0, 16, or 128. We also use an additional set of 128 points just for early stopping of the fine-tuning process, for all models, following the implementation of Hookins-Hookke et al. (2024). We then evaluate remaining points in the task, with a maximum of 2,000 points total, by dividing the data into multiple query sets. If the model can fit a larger query size, as in non mon-meta-learning baselines, then we pass the remaining data as single query set. If the data is not divisible by the query set size, it is left out from evaluation. However, we sample the support data over three independent samples, avoiding systemic exclusion.
All evaluation uses the Spearman rank correlation, in line with prior work (Notin al. 2023 Truong Jr & Bepler 2024; Hawkins- Hooker et al. (2024). We compute the Spearman correlation per task, and then average over tasks. For all evaluations of our models, we compare over three seeds for training and report the mean and standard deviation. Fach context, consisting of a support and query set, consists of S 171 171,000 tokens. We meta-train for 50,000 updates and fine-tune for 100. Fine-tuning uses the same procedure, including an Adam optimizer and cosine learning rate scheduler, but fine-tuning skips the learning rate warm-up, so the scheduling has little effect. Using single Nvidia A100-80 100-80Gb, training our model takes roughly 2 to 8 days per seed, depending on support size and frequency of fine-tuned evaluation.
4.2 ZERO-SHOP SHOT
The first setting we evaluate is the zero-shot performance of our model, with no support set for fine-tuning. In Table we compare against predictions provided by ProteinGym for each baseline to compute the zero-shot Spearman correlation (p). We compare to provided predictions, on our data splits, to enable a fair comparison to the strongest models available without retraining each baseline from scratch ourselves. We include the best performing model, and notable models, as baselines. We find that Metalic outperforms every reported baseline and is SOTA at zero-shot prediction.
Table I: Spearman correlation in the zero-shot setting. Results are computed using predictions provided by ProteinGym. Using a single PLM, in the zero-shot setting, renders the baselines de- terministic. For comparison, we report our best model, and the mean and standard deviation, for quantifying the variation in meta-training. Metalic achieves SOTA performance in either case.
Model Name
Spearman Correlation
Metalic (max)
.484
Metalic (mean)
.482
+ .002
VESPA
.464
TranceptEVE-MMdiin
.457
ESM1-V-65M
.437
Tranception-Mediue
.427
Progen2-Medium
.419
ESM2-650M
.399
MSA Transformer
.398
ESM-IFI
3655
ESM2-8M
.121

---

Published as a conference paper at ICLR 2025
Our method significantly outperforms strong baselines with many more parameters, such as ESM1- v-650M. The 8 million parameter PLM used by our method, ESM2-8M, without Metalic, achieves a score of only .121 p. demonstrating the large contribution of our meta-learning procedure. The next strongest method after ours is VESPA (Marquet et al. 2022). VESPA optimizes PLM embeddings predict distinct set of binary annotations from 9,594 proteins, and uses these features predict protein fitness by comparing to a reference wild-type embedding. Note, unlike our method, VESPA relies on strong assumptions to generalize and specific the zero-shot setting.
The strong performance of our method in the zero-shot setting can be attributed to meta-learning. Since there is no data for fine-tuning, the zero-shot performance increase over ESM2-8M derives entirely from our mneta-traiiing procedure. In this case, other methods generally assume that the PLM likelihood of mutation correlates with fitness, whereas our model learns to make use of the information contained in PLM embeddings to make predictions zero-shot. Moreover, our method still conditions on an unlabeled query set, and the protein embeddings in that query set, which allow for meta-learning form of in-context unsupervised adaptation.
4.3 FINE-TUNING RESULTS
In Table 2 report Spearman correlation with support set of size N(S) 0, 16, and 128, and we compare to baselines that we train and evaluate ourselves over three random seeds. We re-train these methods using the models, following HHawkiss-Hooker over multiple seeds between these methods in a range of practical low-data settings. All models use the same preference-basee loss function as Metalic, for fair comparison, and none use ensembling.
al. (2024), provide comparison
We also train Metalic-AuxIf which allows Metalic to condition on zero-shot predictions from an additional PLM, as introduced in Section 3.2 We choose ESM-IFI (Hsu al. 2022) as the auxiliary input, since it contains embeddings derived from inverse folding that summarize protein structure. Note that Metalic-IFI uses 170 million parameters, whereas Metalic uses 36 million (including the ESM2-8M embedding), so the auxiliary predictions significantly increases the total parameters.
Again, we find that Metalic has the strongest performance in the 0-shot and 16-shot settings, and has comparably strong performance in the 128-shot setting, with 18 times fewer parameters. Moreover, Metalic also outperforms contemporary models, such as PoET (Truong & Bepler| [002]), that make of additional evolutionary information, in the form of multi-sequence alignment, and in- context learning. Note that ESM2-8M and ProteinNPT are the worst performing methods in the few-shot setting. This results suggests that the effectiveness of Metalic does not come from the ESM2-8M embedding, nor the ProteinNPT architecture, but rather from the meta-learning itself.
Additionally, we that Metalic- AuxlF, improves results. This result is significant because ESM2- 8M and ESM-IF1, the two PLMs used by Metalic-AuxIF, are the worst performing methods in the zero-shot setting (Table T) with FSM2--8M also weak in the few-shot setting. Thus, the effectiveness of Metalic- FuxIF comes entirely from meta-learning: We can learn how and when to rely on features from each of these weak predictors of protein fitness, to combine them into strong predictor.
Consistent with the motivation of meta-learning, results are strongest when the data is most lim- ited. Meta-learning adds an additional training stage learn prior beliefs and inductive biases from related data. The more limited, the more relying on prior data is useful.
Table Spearman correlation for the 0, 16, and 128-shot setting. Baseline results are re- computed. Standard deviation is provided over three seeds. Metalic matches or exceeds all baselines.
Model Name
0
n = 16
128
Metalic
.482
.002
.484
.001
.552 + .009
Metalic-AuxIF
.498 + .008
.500
= .002
.556 + .005
ESMI-v-650M
.384 .000
.422
= .000
.533 = .000
ESM2-8M
.105 .000
.266
.000
.406 + .000
PoET
.416 + .003
.455
= .026
.588 +. .006
ProteinNPT
N/A
.192 = .003
.443 + .003
7

---

Published as a conference paper at ICLR 2025
Table 3a: Multi-mutant Spearman correla- tion in the zero-shot setting, Results are com- puted using predictions provided by ProteinGym tasks with multiple mutations. Metalic is petitive, but not better than all baselines, due lack of sufficient datascts with multiple mutants.
Figure 3h: Multi-mutant Spearman Correlation by Shot. Spearman correlation for the 0, 16, and 128-shot, over three seeds, sorted by zero-shot performance. For complete results, including for Metalic-Auxlf, Ap- pendix(A.3
Multiples
I O-Shot
In 16-Shot
In 128-Shot
Model Name
O-Shot
.8
Metalic (max)
.450
Metalic (mean)
.436 + .011
.6
Metalic-AuxIF (max)
.548
Metalic-AuxIF (mean)
.533 + .012
ESM-IFI
.590
.4
FrancepEEVE-Meding
.529
Tranception-Mediig
.133
.2
MSA Transformer
.503
VESPA
.408
ESM2-650M
.355
PoET
Metalic
ESMI-V-650M
ESM2-8M
ProteinNPT
Progen2-Medine
.05
ESM2-8M
2889
ESM1-v-650M
.79
4.4 MULTIPLE MUTANTS
In this section we evaluate Metalic on tasks where proteins have multiple mutations. In Table we that Metalic has strong performance, but does not outperform all baselines. However, with iliary inverse folding predictions (Metalic-AuxII)) Metalic only outperformed by ESM-IF1 itself. These results indicate that ESM-IFI is particularly strong on the eight held-out evaluation tasks, and that we can recover most its performance by incorporating its predictions into our model. Note setting, is among the worst performing models
that while ESM-IF1 is strong in the in the single-mutant setting. In contrast, our method is strong both settings, since can learn to leverage ESM-IF1 predictions when they are helpful, and ignore them when they not.
By comparing across all shot settings, we can see that Metalic is competitive in the multi-mutant setting (Fig. 3b but no longer SOTA, which hypothesize is due limited multi-mutant training data. In Table 4a and Fig. 4b we explore this hypothesis. Specifically, we see the performance of Metalic increases as the amount of training data, measured in tasks, increases. This trend holds true even when adding single mutant tasks, which are significantly different from the testing data. Providing more data a path forward for strengthening the
results.
Table 4a: Multi-mutant Spearman correla- tion by training data. Results computed on eight tasks with multiple mutations with differ- ent amounts single- and multi-mutan training tasks. We see performance of Metalic increases with more data, giving path to improve results with future data collection. This trends holds even when we add single-mutant data, which signifi- cantly different from the mutant test data.
Figure 4b: Multi-mutant spearman correlation by number of tasks. Metalic with and without 121 addi- tional single- mutant training tasks. (With 121 single- mutant the default.) The performance of Metalic in- creases with more data, giving a path to improve results.
Spearman Correlation (n
Multiples
Single- Single-Mutant Multi-Mutan
n = 0
121
63
.436 .011
.2
121
30
.47 + .021
121
.83 + .010
0
63
.366 .018
Metalic (+121 Single)
30
.377 .022
Metalic (40 Single)
0
1
.047 + .056
30
63
Number of Multi-Mutan Training Tasks
8

---

Published as a conference paper at ICLR 2025
4.5 ABLATIONS
Here we report ablations of Metalic, evaluating on single-mutant, (Table Spearman correlation reported in the zero-shot and 128-shot settings. The results justify all components of our method.
Most detrimental to performance is removing meta-training from Metalic (NoMetaTTriin). This ablation is identical to Metalic, with the exception that there no additional meta-learning stage over multiple protein landscapes. We see that without meta-learning, the initial zero-shot predictions have near zero correlation with the fitness and that the 128-shot predictions are critically impaired.
We also ablate the ability to attend to the rest of the proteins in context by turning off the column attention in the axial attention layers (NoICL), we ablate the fine-tuning stage of training (NoFT), and we ablate the preference loss by using mean squared error instead (No- Pref). Note that when remove the fine- tuning, and have a non-zero support size, we allow the early stopping data be passed in- context, to not unfairly ad advantage fine-tuning with additional data. Ablating in-context learn-
Table 5: Ablations in the 0 and 128-shot set- ting. Results show the importance of fine-tuning, in-context learning, meta-training, and additional training tasks as an augmentation.
Model Name
= 0
n = 128
Metalic
.482 .002
.552 + .009
Metalic-NoFT
.482 + .002
.488 + .012
Metalic-NoPref
.465 .011
.520 + .006
Metalic-NoICL
.441 = .002
.529 = .002
Metalic-NoMetaTruin -006 = .018 .346 = .003
ing decreases performance in both settings, indicating an ability to adapt in an unsupervised fashion even from the query set alone. Ablating fine-tuning decreased performance in the 128-shot setting, indicating the need for fine-tuning for generalization to held-out data. Ablating the preference- based loss decreased performance in both settings, in line with recent literature (Krause et al. 2022 Brookes et al. 2023 Hawkins- et al. 2024). An additional ablation is given in Appendix A.3
4.6 GRADIENT- BASED META- LEARNING
In this section, we compare Metalic to the same architecture but trained with Reptile (Nichol et al.) 2018), an efficient method for gradient-based meta-learning. Unlike Metalic, Reptile does not use in-context learning and modifies the outer-loop during meca-trrining take into account the subse- quent fine-tuning. While accounting for the fine-tuning during meta- meta-trainin training comes with increased computational costs and can increase bias and variance (Vuorio et al.) 2021). Reptile provides sim- plified algorithm that can be run more efficiently. The primary differences between Reptile and Mer- alic are that Reptile adjusts the meta-learning loss to account for fine-tuning during meta-training, while Metalic performs in-context learning. Reptile details are provided in Appendix A.
Although Reptile is more compute-efficient than other gradient-based methods, it still performs gradient updates in the inner loop during meta-trainee meta-training, making it computationally expensive relative to Metalic. Our method uses 100 updates on the support data for fine-tuning. Reptile uses inner- loop gradient updates during both meta-training and fine-tuning. Due to compute limitations, we cannot use 100 steps for each forward pass of meta-training. Thus, we evaluate Reptile with inner-loop gradient steps during meta-training and 3 during fine-tuning, so that train and test match (Reptile-3-3), and we evaluate Reptile with 3 inner-loop gradient steps during meta- veta-training and 100 during fine-tuning, so that test time matches our method (Reptile-3-100), Note that even Reptile-3-3 uses three times more compute than Metalic. Finally, we evaluate Reptile-3- 100 with Metalic, to see whether they can be used in conjunction (Metalic-Reptill)). We train for 50,000 steps, and also report Metalic after 150,000 steps, to match the number of gradient computations as the Reptile models.
Table 6: Comparison to Reptile. Reptile (Nichol et al. 2018) is a gradient-based meta-learning method. Results are reported after 50,000 steps, with Metalic additionally reported after 150,000 steps, to allow for an equal number of gradient computations as the methods using Reptile. Results show that accounting for fine-tuning during meta-training using Reptile is unnecessary.
Model Name
Meta- Training Steps
Total Gradient Computations
n = 128
Metalic (150k)
150,000
150,000
.562 + .004
Metalic-Reptile
50,000
150,000
.562 + .004
Metalic (50k)
50,000
50,000
.522 + .009
Reptile-3-100
50,000
150,000
.599 + .001
Reptile-3-3
50,000
150,000
.499 + -008

---

Published as a conference paper at ICLR 2025
Results are reported in Table 6 We evaluate in the 128-shot setting. Note that Reptile is not ap- plicable in the zero-shot setting, as it requires some data for fine-tuning. We observe that without controlling for the amount of computation, Metalic (50k) outperforms both Reptile variants. We also observe that Metalic, in conjunction with Reptile (Metalic-Reptile) outperforms Reptile and Metalic independently. However, the improvement over Metalic is marginal, and the increased computation (by factor of 3) is large. Controlling for the total number of gradient computations, Metalic (150k) achieves Spearman correlation comparable to the combined Metalic-Reptile. Thus, Reptile performs poorly in isolation, and, while Reptile can be used with Metalic, the combination increases implementation complexity with no clear advantage. This result is in line with previous work showing that accounting for gradient updates during meta-training (in their case, without in- context learning) can be detrimental when there are a limited number of tasks for meta-training or a limited amount of data for the inner-loop (Gao & Sener, 2020)] (Trintafillou et al. 2020). Collec- tively, this evidence further justifies fine-tuning only after meta- vecta-training.
ANALYSIS
0.0123
0.0120
0.0123
0.0113
0.0133
0.011
010119
09U110
20
UUUL1
0.0001
0.0101
0.0001
00
5000
73
13
0,0092
73
04
0.0003
0,0032
04
0.0003
0.00 0009
10
0.000
0.009
Protes Indes
Protet Br0ms
Protet In0es
(a) Untrained Attention
(b) Half- Trained Attention
(c) Trained Attention
Figure 5: Attention maps. We present axial attention maps over the query set in the zero-shot setting. Each row shows attention to other proteins in context, normalized to one by row, averaged over layers and mutation location. Attention at step 1 (a) 25,0000 (b) and 50,000 (c). Each protein learns to pay attention to itself, while still attending to other significant proteins in context.
Here, we briefly investigate the in-context learning abilities of Metalic via attention maps. In Sec- [ion[4] we show that in-context learning is vital to Metalic by ablating the attention between proteins and showing decreased performance. Notably, the in-context learning was beneficial not only in the few-shot setting, but also in the zero-shot setting. This suggests an interesting phenomenon: The emergence of unsupervised in-context learning from the query set alone. To confirm this phe- nomenon, in Fig. 5 we show the attention maps in the axial attention layers between proteins in the observe the emergence of bright vertical lines, indicating
query set. Over the course of training, some significantly informative proteins to which all others proteins attend. Moreover, we observe no rows that are entirely dark off the diagonal entries. Thus, no protein attends only to itself. Together, these results further corroborate the necessity of in-context meta-learning.
6 CONCLUSION
In this paper, have demonstrated state-of -hr---rtttt-tttt results on standard protein fitness predic- tion benchmark in low-data settings. To do so, we proposed Metalic, which makes use of both in-context meta-learning and subsequent fine-tuning. Critically, we have demonstrated the ability of meta-learning to take advantage of additional data from other proteins fitness prediction tasks, while remaining computationally tractable by deferring fine-tuning to test time alone. Unique within the meta-lcarning literature, we show that in-context meta-learming provides a useful initialization for further fine-tuning, and can make use of test time data for both fine-tuning and in-context learn- ing. Metalic additionally demonstrates the ability to learn from the query set alone (zero-shot), performing unsupervised in-context learning. Future work could investigate leveraging Metalic's unique in-context learning ability to act an auto-regressive fitness model (i.e., a world model in the renforreinor learning setting (Beck et al.] 2023a)), for optimally trading off exploration and exploitation when designing novel proteins. Given its efficacy at leveraging additional data, believe that meta- leaa-learning in-context will play crucial role in advancing protein fitness prediction, with Metalic being foundational step in that direction.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shi- mon Whiteson. survey of meta-reinforreemntt learning. arXiv, 2023a.
Jacob Beck, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Recurrent hypernetworks are sur- prisingly strong in meta-rl. Advances in Neural Information Processing Systems, 2023b.
Jacob Beck, Matthew Jackson, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Splagger: Split aggregation for meta-reinforcemeen learning. Reinforcement learning conference, 2024.
David H Brookes, Jakub Otwinowski, and Sam Sinai. Contrastive losses as generalized models of global epistasis. arXiv, 2023.
Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta- in-context learning in large language models. Advances in Neural Information Processing Sys- tems, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning. 2017.
Douglas M Fowler and Stanley Fields. Deep mutational scanning: a new style protein science. In Nature Methods, 2014.
Katelyn Gao and Ozan Sener. Modeling and optimization trade-off in meta-learning. Advances in neural information processing systems, 2020.
Alex Hawkins-Hooker, Jakub Kmec, Oliver Bent, and Paul Duckworth. Likelihood-based fine- tuning protein language models for few-shot fitness prediction and design. ICML 2024 Workshop on Efficient and Accessible Foundation Models for Biological Discovery, 2024.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv, 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. In International feerence on machine learning, 2022.
Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning Artificial Intelli- gence Review, 2021.
Ben Krause, Nikhil Naik, Wenhao Liu, and Ali Madani. Don't throw away that linear head: Few- shot protein fitness prediction with generative models, 2022. URL https::/openreviee
net/forum? hd 5pss
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, and Alexander Rives. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv, 2020.
Céline Marquet, Michael Heinzinger, Tobias Olenyi, Christian Dallago, Kyra Erckert, Michael Bern- hofer, Dmitrii Nechaev, and Burkhard Rost. Embeddings from protein language models predict conservation and variant effects. Human genetics, 2022.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction the effects of mutations on protein function. Advances in neural information processing systems, 2021.

---

Published as a conference paper at ICLR 2025
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.0314,) 2017.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-awan meta learning via sequence modeling. In International conference machine learning, 2022.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv, 2018.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchenaa-Huttade Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transform- and inference-time retrieval. In International Conference Machine Learning, pp. 16990- 17017. PMLR, 2022.
Pascal Notin, Ruben Weitzman, Debora Marks, and Yarin Gal. Proteinnpt: Improving protein prop- erty prediction and design with non-parametric transformers. Advances in Neural Information Processing Systems, 2023.
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Neural Information Processing Systems, 2024.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning. PMLR, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 2021.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski. Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International conference on learning representations, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with ing, 2016.
neural networks. In International conference on machine learn-
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: dataset of datasets for learning learn from few examples. In International Conference Learning Representations, 2020.
Timothy Truong and Tristan Bepler. Poet: generative model of protein families as sequences- of-sequences. Advances in Neural Information Processing Systems, 2024.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. Advances neural information processing systems, 2019.
Risto Vuorio, Jacob Austin Beck, Gregory Farquhar, Jakob Nicolaus Foerster, and Shimon White- son. No dice: An investigation of the bias-variance tradeoff in meta-gradients. In Deep RL Workshop NeurIPS 2021, 2021.
Zheng Xiong, Luisa M Zintgraf, Jacob Austin Beck, Risto Vuorio, and Shimon Whiteson. On the practical consistency meta-reinforeement learning algorithms. Fifth Workshop on Meta- Learning at the Conference Neural Information Processing Systems, 2021.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In International conference on machine learning, 2019.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. REPTILE DETAILS
This section provides additional details on how Reptile (Nichol et al. 2018) works. In order to account for fine-tuning during meta-training, gradient-based methods generally compute a meta- gradient that requires the computation of higher order derivatives, which can be computationally intractable for a large model. The costs can be especially burdensome when many gradient steps are needed for out of distribution adaptation. For this reason, Reptile avoids meta mnta-ggadiintss by changing optimization during meta-training. Here, the new parameters updated, not by gradient descent, but rather by moving toward the mean, after each theta is adapted to a task, θr, by fine-tuning on that task. From time-step to + 1 of this outer-loop optimization process, Reptile can be written:
g++l 0 H.SEDEED 107 6').
(5)
We had to choose several implementation details for Reptile. First, note that Reptile sub-samples the support set during fine-tuning and has no distinct query set. In our implementation, we sub- sample mini-batches of size 50 to match the query size of Metalic for sub-sampling in the 128-shot setting. Reptile also has several options for the outer loop optimization. We choose to use the batched version with batch size of 4 to match Metalic. Rather than using the direct update given in Equation (5). we take the difference over the learning rate, (0f, 0²) as an approximation of a gradient to be used with the Adam optimizer, suggested by Nichol et al. (2018) and validated in Appendix A.2
A.2 HYPER- PARAMETERS
Since we build upon the axial attention of ProteinNPT (Notin et al. 2023), we follow their choice for most typer-parameters. with few exceptions. Most notably, in our experiments, use the third layer of ESM2 as an embedding for each protein, given the strong performance and reduced number of parameters. Additionally, we use ranking loss from Hawkins- et al. (2024), and do not use the CNN or additional inputs (such as zero-shot predictions) from ProteinNPT. We likewise found conditioning on the wild-type unhelpful. The same set of hyper-parameters setting: O-shot, 16-shot, and 128-shot, and for multi-mutant results. We used the same learning rate for fine-tuning Metalic as for meta-training We found the default ProteinNPT learning rate to be too large, and decreasing by a factor of 5 to be sufficient. Consistent with our baselines, we select hyper-parameters using the single-mutant results and then use the same hyper-parameters for the multi-mutant setting, following Hawkins--ooooooeo on hyper-parameters used are available in Table 7 We tuned relatively few of the hyper-parameters of our method, and mostly tuned over a single seed. There is likely room for improvement in the hyper-parameter selection of Metalic.
used for each
et al. (2024):[Notin et al.] (2023). Complete details
For the majority of baselines no tuning was required, other than Reptile. For the baselines, we used reference predictions for Table I and reference implementations for Table neither of which required tuning. For Reptile, the update in the outer-loop can be uses written in Equation (5) or (84--0), can be interpreted as a gradient for use with the Adam optimizer Nichol et al. (0118) We experiment with both and find the use of the Adam optimizer to be superior in performance. All results in the main body use the Adam optimizer for Reptile. In order to evaluate the method without the Adam optimizer, we re-tune in Equation (5). which is the outer-loop learning rate. We leave the inner-loop learning rate, a, fixed at the learning rate for Metalic, 3e -5, which uses the same learning rate for the inner- and outer-loop. Given the increased computational cost of Reptile, we use a single seed over three learning rates for 10,000 steps. We tune over the following learning rates (3): the learning rate for Metalic, which is 35; a learning rate of 1, which corresponds to the learning rate of Metalic if you interpret (0C, - 8a)/a as the gradient in tradient-basee update Nichol et al. 2018); and learning rate of n. where is the number of inner-loop updates (3 in this case), which corresponds to the learning rate of Metalic if you interpret (0z - 0²) / as the gradient in a gradient-batt based update"] We found a learning rate of I to perform best, in line with the
'The standard gradient update, 0--1 0r + v., with V (07 (')())n),, gives Ø++1 = 0² + o00 8')/ (am), which is the same update as Equation with 3 omitting the expectation for brevity.
13

---

Published as a conference paper at ICLR 2025
Table 7: Hyper-Paree Parameters for Metalic. Where different, an earlier set of hyper-parameters used for ablations and Reptile comparisons are given in parentheses.
Hyper-Parameter Training Steps Warm-Up Steps
Description
Value
The total number of training steps in meta-training
50,000
The number of training steps spent linearly warm- ing up. preceding cosine decay
5,000
Batch Size
The number of contexts evaluated per training step.
4
Note that gradient accumulation is used for each context in the hatch, this scales training time lin- early.
Weight Decay
Weight decay applied to non-bias parameters only
5e-3
Learning Rate
The learning rate meta training and tine-tuning
6e-5
Min LR Fraction
The minimum fraction of the LR maintained during the cosine decay in learning rate scheduling
le-5
Adam Eps
The epsilon value for the Adam optimizer
le-8
Adam Betal
The betal value for the Adam optimizer
0,9
Adam Beta2
The beta2 value for the Adam optimizer
0.999
Gradient Clip Value
The maximum norm allowed for the gradient The full name for the ESM2 model used
1.0
ESM embed model
esm 16 8M UR50D
FSM embed layer
The layer from the ESM2 model used emhed- ding
3
Number
Fine-tune
The number of gradient updates for fine-tuning after meta-training
100
Steps
Num ProteinNPT Lay-
The number of layers using axial attention, as in ProteinNPT
5
Condition on Pooled
Whether each sequence is pooled or ignored after axial attention
True
Sequence
MLP Layer Sizes
The number and size of fully connected layers after axial attention
(768,768,768,768)
Embed Dim
The embedding dimension for all inputs including the protein sequences and fitness values
768
Axial Forward Embed
The hidden size of the sorwwrd layer within the ProteinNPT layer
400
Dim
Attention Heads
The number of heads self-atee attention
4
Dropout Prob
The probability of dropout during training and fine-
0.0
tuning for layers other than axial attention
Attention Dropout
The probability of dropout during training and fine- tuning for axial attention layers
0.1
Num Single Tasks
The total number of single-mutant tasks available. These tasks are included for meta-tran meta-training even when testing multi mutants. Eight are held-out for evaluation when evaluating hingle-mutane per- formance.
121
Num Multiple Tasks
The total number of multi-mutane tasks available. These tasks are included for meta-training even when testing on single-mutants. Five are held-out for evaluation when evaluating multi-mutant perfor- mance.
68
Warm-up During Fine- tuning
Whether or not to use linear warm-up from the lcarning rate scheduler during fine-tuning.
False
outer loop learning rate of Metalic and the interpretation of the gradient from Nichol et (2018) For Adam, there is no 3, and we do not re-tune Adam's 's outer-loop learning rate. When using Adam, the gradient is interpreted as (0t- 3/a/ The results of tuning the learning rate without Adam, suggests learning rate of -6 with this gradient interpretation. Indeed, we confirm that when using Adam with this learning rate, and this gradient interpretation, it works better than using the alternative Reptile update rule with any learning rate. Results of the optimizer and learning rate tuning are presented table Table Note that these results used an earlier set of hyper- param-tarameter.
14

---

Metalic does use warm-up period for fine-tuning as does for meta-training. This decision was made since the warm-up normally occurs for 5,000 steps, but fine-tuning only occurs for 100 steps. compromise, we evaluate a warm-up period for fine-tuning, after taking a single gradient We still see that not using a warm-up for Published as a conference paper at ICLR 2025
Table 8: Reptile tuning results for the 128-shot setting. Results are trained over 10,000 steps for one seed each. The Adam optimizer performs the best. Both Adam, with an outer-loop learning rate of 3e-b, and Equation with 8 correspond the same scale the outer loop.
Model Name
Adam (3e
8
.333
8 3e
Reptile-3-3 Reptile-3-100
.452
.420
.350
.091
.472
.444
.396
.181
Metalic-Reptile
.483
.476
.411
.190
A.3 AUGMENTATION ABLATION
In this section we provide additional ablation. Specifically, we ablate the augmentation of single- mutant training data with multi-mutant data (NoAug). The result is reported in Table Ablating the multi-mutant augmentation decreased performance in the zero-shot setting, with comparable 128- shot performance, indicating the benefit of additional meta-training data when fine-tuning data is most limited.
Table 9: Augmentation Ablation in the 0 and 128-shot setting. Results show the importance of augmenting with multi-mutant data in the zero-shot setting.
Model Name
71 0
128
Metalic
.482 + .002
.552 + .009
Metalic-NoAug
.464 + .011
.558 .002
A.4 FINE-TUNING WARM-UP
step with the full learning rate PretalicFFTWamUJJ fine-tuning superior. Results can be see in Table I0
Table Spearman correlation with and without warm-up.
Model Name
128
Metalic
.552 + .009
Metalic-FF Frrulic-FTWamUpp
.539 + .007
15

---

Published as a conference paper at ICLR 2025
A.5 MORE MULTI-MUTAN RESULTS
In this section we present a table of additional results for tasks with multiple mutants, including the Metalic-AuxIF models.
In Table 12 we see the performance of Metalic increases as the amount of training data, measured in tasks, increases, providing more data is path forward for strengthening the multi-mutant results. Here, we also see that the trend changes for Metalic-AuxIF, since it performs better with more data, but only if that data is multi-mutant data. This makes sense, since the auxiliary ESM-IfI predictions are most effective for multi-mun mutants, and training on single-mttant could discourage relying on them.
Table 11: Spearman correlation for the 0, 16, and 128-shot setting for multi-mutant tasks.
Model Name
n = 0
n = 16
1 = 128
Metalic (Both)
.436 + .011
.484 + .001
.670 .003
Metalic-AuxIF (Both)
.480 + .007
.500 + .002
.693 + .006
Metalic-AuxIF (MultiTrain)
.533 + .012
.577 + .008
.692 .003
ESMI-V-650M
.426 = ,000
.557 = .000
.644 # .000
ESM2-8M
.368 + .000
.484 + .000
.596 + .000
PoET
.588 + .014
.638 + .001
.734 + .006
ProteinNPT
N/A
.367 + .003
.624 .003
Table 12: Multi-mutant Spearman correlation by training data. Results are computed using predictions provided by ProteinGym on five tasks with multiple mutations with different amounts of single- and multi-mutant training tasks. We performance of Metalic increases with more data, giving a path for future data collection. trained with only multi-mutant data. This makes sense, since the augmented predictions are most effective for multi-mutants, mutants, and training on single-mun mutants could discourage relying on them. An asterisk (*) represents the default training data regime for that method.
also see here that Metalic-Auxli performs better when
Multiples
Model Name
Mutant Tasks
Multi-Muteen Tasks
0
Metalic (Both)"
121
63
.436 + .011
Metalic (MultiTrain)
63
.376 + .018
Metalic (MultiTrainHalf)
30
.307 + .022
Metalic-AuxIF (Both)
121
.480 + .007
Metalic-AuxIF (MultiTrain)"
63
.533 = .012
Metalic-AuxIF (MultiTrainfee Haal
30
.517 + .021
16

---

Published as a conference paper at ICLR 2025
A.6 DETAILED ARCHITECTURE
Here, we present Fig. which gives a more detailed version of the architecture diagram.
Concat
5D
15
MLP
Fielten
Sequeecce dimension reduced to
Mean 500
Orly query
mibadidings
Axial attention
Feedlorward
LeyerNoom
Column ettention
LayerNom
How attention
LayerNorm
emboddings
Concal
PLM
Linear
Support set
TINEVEI EDUL
ARAINS GEACYNG-ONNY
Query set
GAL FRAFLATP
Protains
Fincssos
Figure 6: Detailed diagram of architecture. Note that fitness embeddings computed using leamned linear projection for the support set and shared learned embedding vector for the query set.
17


Reviews:
[{"review_id": "EhWBwMMQnM", "reviewer": "Reviewer_suA1", "summary": "This work introduces Metalic, a framework for performing protein sequence fitness prediction in both in-context and few-shot regimes. It proposes to use in-context-learning as a replacement for gradient updates used in the traditional inner loop of meta-learning. While using the smallest ESM-2 model, it achieves strong performance on the chosen fitness tasks.", "questions": "* How does performance change when training and evaluating on a separate set of proteins?\n* How do inference and training costs of Metalic compare to the Reptile ablations?\n* To clarify the architecture, would it be possible to have a Vaswani et al. 2017 style figure in the appendix of the exact computations that occur?\n\nOverall the work is quite strong, and would benefit from a simple experiment to further prove its robustness. \n\nVaswani et al. 2017, \"Attention Is All You Need\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Metalic achieves success in both providing a compute efficient alternative to meta-learning and also provides strong performance on downstream tasks. It’s also noteworthy the parameter efficiency that Metalic achieves in the process.\n\nThe work is also highly reproducible. Each experiment could be reproduced by a reader from the text alone (barring which datasets were and were not used). In the appendix, readers can find a comprehensive set of hyperparameters and training details used in each experiment.\n\nWhen comparing metalic, it uses relevant choices in baseline models and also analyzes where and how performance degrades (e.g. the case of multi-mutants). In addition, the ablations show the utility of the different components of the method, even displaying its ability to outperform traditional meta-learning methods like Reptile. The experiments are very thorough in justifying all choices made during modeling.", "weaknesses": "The only major weakness that appears in this work is the limited number of datasets used in evaluating the proposed methods. ProteinGym performance fluctuates a lot amongst tasks, hence it would be nice to decouple the comparatively small number of datasets, 13 in total, for evaluation from performance. A simple experiment selecting different splits and training replicates in a similar manner to a partial k-fold cross validation would be convincing.\n\nOn a more minor note, the exact implementation of the algorithm would benefit from a slightly more detailed treatment. Though it can be inferred from Figure 2a, it would benefit from some support either pictorially or mathematically in the exact computations that must be performed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FD2in7SbAP", "reviewer": "Reviewer_Q2KC", "summary": "This paper introduces Metallic, which performs few-shot (N=16 or N=128) or zero-shot fitness prediction using in-context meta-learning, using the ProteinGym benchmark. The model is meta-trained across many protein prediction tasks, to enable in-context learning, then finetuned on the support set, where the support set defines the number of \"shots\" for the few-shot task. Evaluations uses pairwise ranks (i.e. if model correctly prefers one mutant to the other). The meta-training procedure yields models that fare better in the finetuned few-shot setting than baselines (ESM1, etc.)", "questions": "* Are there some tasks that are easier to learn vs. harder to learn? It would be interesting to see a breakdown analysis.\n* Do authors have hypotheses for why the performance gap between Metalic and baselines is so much more pronounced for zero-shot vs. few shot tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea is relatively under-explored in protein ML despite ICL having shown strong performance in other fields of ML, and the importance of few-shot learning.\n* Paper is fairly clearly written\n* Attention map analysis is a great addition to the work! Would love to see follow up on this.", "weaknesses": "* In Table 1, 2, and 3a, only ESM2-8M is used for comparison. What happens if we compare to the larger ESM2 models? Can we get better zero-shot or few-shot performance via scaling pretraining parameters?\n* Results in Figure 3b and 3a don't seem too much better than baselines.\n* More rigorous examination of the effects of varying query set size, etc. might be quite insightful\n* Some papers have pointed out that protein likelihoods don't always indicate protein fitness, and might be due to the training data biases [1,2].\n\nMinor / meta comment:\nI think that though thinking about ICL for proteins is a good idea, and authors demonstrate thoughtfulness in how this should be done. However, there might be more interesting tasks to examine than likelihood & protein fitness. There has also been working showing that \"evotuning\" (finetuning on evolutionary neighbors) can improve performance [2,3]. In a real-world setting, if I have a protein for which my pretrained model is underperforming on, I'm not sure if I would prefer to use a meta-trained model or a evo-tuned model.\n\nFrom a writing perspective, authors might be able to increase impact by outlining real-world drug discovery scenarios where this is relevant.\n\n[1] Protein language models are biased by unequal sequence\nsampling across the tree of life. Ding et al., 2024\n[2] Protein Language Model Fitness Is a Matter of Preference. Gordon et al., 2024\n[3] Unified rational protein engineering with sequence-based deep representation learning. Alley et al., 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kobk5kllqA", "reviewer": "Reviewer_JKXe", "summary": "The paper titled introduces Metalic, a meta-learning approach for predicting protein fitness, which is a measure of protein properties like stability and binding affinity. Predicting these properties accurately is challenging due to the scarcity of labeled data. Metalic improves on this by applying meta-learning to a distribution of related protein fitness tasks, enabling positive transfer to unseen tasks. The approach combines in-context meta-learning and fine-tuning without accounting for fine-tuning during meta-training, which helps to generalize well even with limited data. Experiments show that Metalic achieves state-of-the-art performance in zero-shot and few-shot settings on the ProteinGym benchmark, using 18 times fewer parameters than current models.", "questions": "1-The authors assume that the correlation between PLM likelihood and protein fitness may not be reliable. How does Metalic perform when the assumed correlation is weaker? Would additional analysis on this correlation help refine the model's approach?\n\n2-In-context learning relies heavily on the distribution of tasks used during meta-training. How does Metalic handle tasks that are significantly different from the meta-training distribution, and are there strategies that could improve its generalization to such out-of-distribution tasks?\n\n3-Given that Metalic does not account for fine-tuning during meta-training, how does this omission affect the model's ability to adapt to tasks with extensive fine-tuning requirements? Would incorporating some level of fine-tuning awareness during meta-training improve results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1-By integrating in-context meta-learning with subsequent fine-tuning, Metalic can generalize well to new tasks, even though fine-tuning is not accounted for during meta-training. Moreover, the model is more computationally efficient than state-of-the-art approaches.\n\n2-Metalic achieves strong performance in zero-shot and few-shot protein fitness prediction tasks, which is particularly valuable given the limited availability of labeled data for many protein-related tasks.", "weaknesses": "1-I anticipate a more in-depth analysis of the correlation between PLM probability and fitness values in the context of meta-learning, especially when the authors assume that this correlation may not be reliable.\n\n2-Although in-context learning is a strength, its effectiveness diminishes for tasks that deviate significantly from the distribution of tasks used during meta-training, potentially limiting generalization to out-of-distribution tasks. Experimental results on some OOD tasks can help readers better understand the paper.\n\n3-The paper primarily compares Metalic to a few gradient-based meta-learning methods (e.g., Reptile) but does not extensively benchmark against other recent meta-learning approaches that incorporate different strategies for handling fine-tuning or leveraging unsupervised data.\n\n4-Metalic shows strong performance on single-mutant tasks but falls short of the state-of-the-art on multi-mutant tasks. The authors attribute this to the limited availability of multi-mutant training data, emphasizing the need for more diverse datasets to fully realize the method's potential. A more in-depth analysis of this limitation would be valuable.\n\n5-While the ProteinGym benchmark is a standard for protein fitness prediction, additional evaluations on other benchmarks would provide a more comprehensive understanding of Metalic's strengths and weaknesses across a broader range of protein-related tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xahMVt6brs", "reviewer": "Reviewer_ZGWy", "summary": "The authors present Metalic, a new approach to predicting properties of protein sequences. Metalic involves two phases: a \"meta-learning\" phase, during which a PLM backbone is fine-tuned to learn from data from multiple related fitness prediction tasks presented in-context, and then a per-task \"fine-tuning\" phase, where models are fine-tuned at inference time on a small amount of data specific to the task at hand. The authors show that Metalic outperforms baselines in the zero-shot setting for single mutations and is competitive for multiple mutations.", "questions": "While the paper is fairly clearly written, there are a few pieces of information I haven't been able to find:\n\n1. Why did you stop at 8M? How much more computationally intensive would it be to run experiments with the 35M model?\n2. How many tasks are presented in-context to generate the figures in e.g. Table 1? What data is used in-context at evaluation time? Is there overlap with the data used during the meta-learning phase?\n3. \"Moreover, our method still conditions on an unlabeled query set, and the protein embeddings in that query set, which allow for meta-learning a form of in-context unsupervised adaptation.\" -> Is there any evidence in the paper that this occurs? Would be cool to see an ablation of the data from other tasks to isolate this.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is clearly written, includes a range of baselines and ablations, and also attacks an important problem. In some sense, protein fitness prediction is an excellent application for meta-learning, in that it's one of the few settings I can think of where there is an almost arbitrarily broad distribution of tasks that are nevertheless closely related to one another.", "weaknesses": "I'm a bit conflicted about this paper. On the one hand, the method seems to work, and, as I mentioned, the authors already include most of the experiments that I'd think to ask for. On the other: Table 5 shows that the lion's share of the improvement from Metalic comes from the in-context meta-learning; the per-task fine-tuning phase only helps in the many-shot case. But Metalic is only the best method in the zero-shot case (albeit with fewer parameters than baselines), where the only possible improvement comes from meta-learning. If the authors provided parameter-matched versions of Metalic that showed a much clearer benefit to using the method even in the many-shot case, that might change my thinking, but that also obviously changes the viability of performing the fine-tuning step at inference time. As it stands, the paper seems to show that in-context meta-learning works well  for zero-shot protein fitness prediction. In-context meta-learning isn't new, and I just don't know if a simple application of a method that already exists to a task that---for ICLR readers---is ultimately fairly niche warrants acceptance. I'm obviously a bit conflicted and can likely be convinced, but at the moment I'm leaning reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)