One protein is all you need
Authors: Anton Bushuiev
Roman Bushuiev
Olga Pimenova
Nikola Zadorozhny
Raman Samusevich
Elisabet Manaskova
Rachel Seongeun Kim
Hannes Stark
Jiri Sedlar
Martin Steinegger
Tomas Pluskal
Josef Sivic
Venue: ICLR 2026 Poster (2026)
Keywords: proteins
generalization
self-supervised learning
model customization
test-time training
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Per-protein self-supervised customization improves generalization across models and tasks.

Abstract: Generalization beyond training data remains a central challenge in machine learning for biology. A common way to enhance generalization is self-supervised pre-training on large datasets. However, aiming to perform well on all possible proteins can limit a model’s capacity to excel on any specific one, whereas practitioners typically need accurate predictions for individual proteins they study, often not covered in training data. To address this limitation, we propose a method that enables self-supervised customization of protein language models to one target protein at a time, on the fly, and without assuming any additional data. We show that our Protein Test-Time Training (ProteinTTT) method consistently enhances generalization across different models, their sizes, and datasets. ProteinTTT improves structure prediction for challenging targets, achieves new state-of-the-art results on protein fitness prediction, and enhances function prediction on two tasks. We also demonstrate ProteinTTT on two challenging case studies. We show that customization via ProteinTTT enables more accurate antibody–antigen loop modeling and improves 19% of structures in the Big Fantastic Virus Database, delivering improved predictions where general-purpose AlphaFold2 and ESMFold struggle.

Published as a conference paper at ICLR 2026
ONE PROTEIN IS ALL YOU NEED
Anton Bushuiev 1.2+
Roman Bushuiev 1.2+
Olga Pimenova'
Nikola Zadorozhny'
Raman Samusevich 2.2.
Elisabet Manaskova'
Rachel Seongeun Kim Hannes Stärk?
Jiri Sedlar!
Martin Steinegger 9,45,,
Tomáš Pluskal2
Josef Sivic*
Czech Institute of Informatics, Robotics and Cyberneties, Czech Technical University,
Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences,
Sccooo of Biological Sciences, Scoul National University, "Interdisciplinary Program in Bioinformatics, Scoul National University, IIstitute of Molecular Biology and Genetics, Seoul National University, Artiffical Intelligence Institute, Scoul National University. CSAILL Massachusetts Institute of Technology, *Qminers
ABSTRACT
Generalization beyond training data remains a central challenge in machine Iearn- ing for biology. A common way enhance generalization is self- self-suupr pre-training on large datasets. However, aiming to perform well on all possible proteins can limit a model's capacity to excel on any specific one, whereas experi- mentalists typically need accurate predictions for individual proteins they study, often covered in training data. To address this limitation, we propose method that enables self-supervise customization of protein language models to one target protein time, on the fly, and without assuming any additional data. We show that our Protein Test-Time Training (Protein method consistently enhances gener- alization across different models, their sizes, and datasets. ProteinTTT improves structure prediction for challenging targets, achieves new state-of-the- thr-rrr results on protein fitness prediction, and enhances function prediction on two tasks. Through challenging case studies, we also show that customization via ProteinTTT achieves more accurate cuubody-antiggen loop modeling and enhances 19% structures in the Big Fantastic Virus Database, delivering improved predictions where general-purpose AlphaFold2 and ESMFold struggle.
INTRODUCTION
ESMFold
ESMFold PrT
A comprehensive understanding of protein struc- ture, function, and fitness is essential for ad- vancing research in the life sciences (Subrama-
PLDDT
niam & Kleywegi 2022 Tyers & Mann 2003 Papkou et al. 2023). While machine learning models have shown remarkable potential in pro-
tein research, they are typically optimized for achieving the best average performance across
Perplexity (1): 13.0
Perplexity (1): 3.0
large datasets (Jumper et al. 2021; Watson et al. However, biolo- gists often focus their research on individual proteins or protein complexes involved in, for example, metabolic disorders (Ashcroft et al. 2023 Gunn & Neher 2023), oncogenic signal- ing (Hoxhaj & Manning 2020 Keckesova et al.
TM-score (f): 0.63
TM-score (f): 0.84
2023) Kouba al.] 2023).
Figure 1: Example of protein structure predic- tion after single-protein model customization via ProteinTTT. ESMFold poorly predicts the structure of the CASP14 target T1074 (white) be- cause the underlying language model ESM2 poorly fits the sequence, indicated by the high perplex- ity (left and Fig. 2E in [in et al. (2023).. ). Self- supervised test-time customization of ESM2 the single sequence of T1074 reduces the perplexity, resulting improved structure prediction (right).
2017), neurodegeneration (Gulen et al. [0233 oh Seo et al., ) and other biological phe- nomena (Gu et al. (022). In these scenarios, detailed insights into single protein can lead to significant scientific advances.
"These authors contributed equally.

---

Published as a conference paper at ICLR 2026
However, general machine learning models for proteins often struggle to generalize to practically
interesting individual cases due to data scarcity (Bushuiev et al. [023]] Chen & Gong 2022) and
distribution shifts (Skrinjar et al. 2025 Tagasovska et al. 2024 Feng et al. 2024). ). Bridging the gap between broad, dataset-wide optimization and precision needed to study single proteins of practical interest remains a key challenge in integrating machine learning into biological research (Sapoval et al. 2022). This challenge is particularly acute in computational biology, where accurate predictions for individual proteins are essential to guide resource- resource-intensive wet-lab experiments, in contrast to domains such natural language processing or computer vision, where models are typically expected to flexibly handle diverse prompts from many users in real time (Brown, 2020
Ramesh et al., (021).
To address this challenge, we propose test-time approach for generalization one protein at time, effectively enabling more accurate predictions for individual targets, particularly those poorly represented in training data. Our Protein Test-Time Training (ProteinTTT) method customizes protein language models (PLMs) to individual proteins on the fly and without assuming additional data. Our approach is based on simple yet powerful premise: if language model less perplexed (surprised) by a protein sequence-or if it "understands" its unique patterns better-it will generate a more accurate representation for predicting its structure and function. Given a model pre-trained via masked language modeling, our method effectively minimizes perplexity on a target protein or its multiple sequence alignment (MSA) through sele-supe customization, improving downstream performance without updating the downstream task head. The widespread use of masked modeling as pre-training paradigm makes ProteinTTT broadly applicable in computational biology.
In summary, this work demonstrates the surprising effectiveness of protein model customization and lays the foundation for exploring other rest-time strategies and broader biological applications. The key contributions are: (1) We introduce ProteinTTT, the best of our knowledge the first customization method in machine learning for biology. We provide a user-friendly and easily extensi- ble implementation [ and provide insights into the effectiveness of protein model customization by linking it to perplexity minimization. (2) We empirically validate ProteinTTT, showing improvements in protein structure prediction with well-established models, achieving state-of-the-ard results in protein fitness prediction, and enhancing protein function prediction on terpene synthase substrate classification and protein localization prediction. (3) We demonstrate the practical utility of focusing on one protein time through two challenging case studies. ProteinTTT enables more accurate prediction of antibody-antiget loops and improves 19% of structures in the Big Fantastic Virus Database, delivering accurate predictions where general-purpest AlphaFold2 and ESMFold struggle.
2 BACKGROUND AND RELATED WORK
The broad adoption of Y-shaped architectures relying on masked modeling enables the development of general method for customizing protein models at test time via masking- masking-based self-supervision.
The Y-shaped paradigm of learning. In machine learning applied to proteins, architectures often follow a Y-shaped paradigm (Gan delsman et al. 2022), consisting of a backbone feature extractor f operating on protein tokens x, a sulf--uper head g. and an alter- native fine-tuning head h. During training, y I' is first pre-trained, and the pre-trained backbone / then reused to fine-tune o f toward a downstream task. Here, o denotes composition of two machine learning modules (c.g.. g is applied on top of f in g f). At test time, the final model h o ff is fixed. Generalization is achieved by leveraging the rich knowledge encoded in the backbone ff and the task-specific priors embedded in the fine-tuning head h. This paradigm enables overcoming data scarcity during fine-tuning and underlies breakthrough approaches in protein structure prediction (Lin et al.) 2023), protein design (Watson et al. 2023), protein function prediction (Yu et al. 2023). and other tasks (Hayes et al. 2024).
Pre-truiving
e.g. Lincar head
T Protein tokens
egt ESM2
Fine- toning
h
egg..
ESMPold
head
The backbone f is typically large neural network pre-trained in a self-supervised way on a large dataset using a smaller pre-training projection head g (Hayes et al. 2024). The fine-tuning head h, however, depends on the application. In some cases, h is a large neural network, repurposing the pre-trained model entirely (Watson et al. 2023 Lin et al. 2023); in others, h is a minimal projection
http::///tthh...coooaooooouuuuhr nTT
2

---

Published as a conference paper at ICLR 2026
with few parameters (Cheng et al. 2023) or even without any parameters all (i.e., zero-shot setup: |Meier et al.] (2021); Dutton et al. (2024)). The fine-tuning head h can also be a machine learning algorithm other than neural network (Samusevich et al. 2025).
Masked modeling. While the objective of fine-tuning h f is determined by the downstream application, the choice of pre-training objective for g o is less straightforward. Nevertheless, the dominant paradigm for protein pre-training is masked modeling, which optimizes model weights to reconstruct missing protein parts. This objective has proven effective across diverse tasks (Heinzinger
& Rost, 2025: Schmirler et al. 2024), including structure (Lin et al., [202] Jumper et al. (022))
fitness Meier et al. 2021 Su et al. (023), and function prediction (Samusevich et al. 2025 Yu et al. [023 Fnnggaar et al. )
well as protein design (Hsieh et al. 2025 Hayes et al. 2024
Nijkamp ct al. (023), , and has been successfully applied to various protein representations such as
sequences (Hayes et al., 2024 Elnaggar et al. 2023 graphs (Dieckhaus et al.
2024 Bushuiev et al.
2023). and voxels (Diaz et al. 2023).
Model customization. Several studies have shown that machine learning models for proteins benefit from being fine-tuned on protein-specific (Notin et al. [2024] Kirjner et al.) 2023 Rao et al.) 2019)
or protein family-specific (Sevgen et al. 2023) Samusevich et al. 2025) data. However, collecting
additional data may be resource-intensive, and for many targets, relevant datasets or proteins may be
limited or not available (Durairaj al. 2023 Kim al. 2025). In this paper, we propose a versatile method enabling customizing PLMs for a single target protein or its MSA in self- manner, on the fly, and without assuming any additional data. Customization methods have been developed in
computer vision (Chi et al. [2224, Wang al. 2023 Xiao et al. 2022 Karani et al. (021) and natural language processing (Hübotter et al.) 2024] Hardt & Sun 2023 David al. [022 Banerjee al. 2021). The paradigm of test- time training (TTT), developed to mitigate distribution shifts in computer vision applications (Gandelsman et al. 2022 [Sun et al. 2020), is the main inspiration for our work. We demonstrate that customization via test-time training enhances the accuracy of PLMs
across a wide range of downstream tasks even without the presence of explicit distribution shifts.
3 PROTEIN MODEL CUSTOMIZATION WITH PROTEINTTT
In this section, we describe the proposed Protein Test-Time Training (Protein approach (Sec- tion 3.1. followed by its applications to range of well-established models and datasets (Section 3.2)
3.1 SELF-SUPERVISED CUSTOMIZATION TO TARGET PROTEIN
At test time, we assume a Y-shaped model with a backbone f that has been pre-trained via the sulf-supervised track g o f, followed by task-specific fine-tuning through the supervised track h The goal of customization with ProteinTTT is to adapt the backbone f to a single protein æ before making a prediction on downstream task via the supervised track h f. To achieve this, customize the backbone ff to the single example æ:
Protein'TT (h f(:; x)) + h f(:0,))
where On denotes pre-trained parameters and Og parameters optimized for the target protein a using the self-supervised track g o f. while the supervised head h remains frozen. Figure 2a illustrates customization approach, which is summarized in the following sections. Appendix C describes the extension of our method to customization using MSA of protein, rather than its single sequence.
Customization training objective. We customize g ff to single target protein sequence r via minimizing the masked language modeling objective (Devlin) 2018: Rives et al. (02T):
C(x; 0) = EM -pees (M)
Σ LiE
- log pog
(2)
where x denotes a sequence of protein tokens (typically amino acid types), and Ey represents the expectation randomly sampled masking positions M. The objective function L(x; () maximizes the log-pre probaabbislsssss log p(a, ( m 8) = g.f(f. (): of the true (i.c., wild-type) tokens æ; at the masked positions E . in the partially masked sequence M where @ denotes the parameters of the
3

---

Published as a conference paper at ICLR 2026
Customization
Output
b
Inference
Imput QYYCQBVYY
Customized PLM L
Pretrained heads
Protein1'TT
Pretrained
Structure head ht
PC: B
Step
Step 5
Step
QYCQEVY
PLM perplexity High
Low
Customized PLM fC; 0,)
Fitness head ht
sengoeeicc spuce 0,
{8,
θ+}
Confidence
function
QYOQEVY
* Function
QY?QETY
QYCTEV?
Pretrained P
Linear
QYCQEVY QYCQEVY
head h,
QYCQ??Y
QYCQEVY
LORA
Figure 2: Overview of protein language model (PLM) customization with ProteinTTT. (a) Given protein sequence of interest and pretrained PLM f(:: ProteinTTT yields customized version of the PLM (=;00)) ) for that sequence. Customization is achieved by fine-tuning (fire icon) the pretrained parameters Op via masked language modeling solely the input sequence for T steps, selecting the optimal parameters 0g using a confidence function c. This procedure adapts the model specifically the imput sequence, improving its internal representation as measured by model perplexity. (b) Once customized, the PLM can be used with pretrained task-specific heads, such as structure, fitness, or function prediction modules, h1, h22 and h3, respectively, without modifying their parameters (snowflake icon). For example, the ESM2 PLM can be customized and then used with the pretrained ESMFold structure prediction head without modifying its 1.4-biliion task-specific parameters, resulting in improved structure prediction for the given sequence (e.g., Figure T
backbone J. and g is the masked language modeling head. While we focus classical bi-directional masked modeling, we also demonstrate that ProteinTTT can be similarly applied to autoregressive and discrete diffusion models (Appendix B).
To ensure consistency between customization and pre-training, ProteinTTT adopts the same masking and preprocessing strategies used during pre-training. Specifically, (Ma)s(A can follow different distributions, such as sampling a fixed ratio (e.g.. 15%) random tokens (Lin et al.] 2023). or dynamically varying the number of sampled tokens based on another distribution (e-g., beta distribution: Haaes et al. (2024). During customization, we replicate the masking distribution used in pre-training. We also replicate other pre-training practices, such replacing 10% of masked tokens
with random tokens and another 10% with the original tokens (Devlin) 2018 Lin et al. 2023 Su
al. 2023)
cropping sequences random 1024-token fragments (Lin al. 2023
Su 2023)
Optimization. Since customization with ProteinTTT does not assume than single protein, early stopping on validation data is not feasible. To address this, we first fine-tune the pre-trained pa- rameters of backbone f for fixed number of steps T. yielding parameters @ (Oo, 01, Vi..... 0r)- The final customized parameters 0g are selected as arg maxace (x; where is confidence function. is not available, we set 0g or. Appendix H.2 discusses how using PLDDT as the confidence function for structure prediction makes Protein TTT robust to hyperparameter selection and how the number of steps T can be fixed (e.g.. T 30) while optimizing learning rate and batch size effectively. Before customizing for the next target protein, the parameters are reset to fg.
To make ProteinTTT easily applicable large-scale models (e.g., the 3B-parameter ESM2 backbone), we leverage low-rank adaptation (LoRA; Hu et al. (2021)) and gradient accumulation during cus- tomization. Additionally, to improve the stability and predictability of customization, we use stochas- tic gradient descent SGGD:Ruder (2016) instead of the commonly used Adam optimizer (Kingma & Ba 2015), following (Gandelsman et al. (022). Further details are provided in Appendix
3.2 INFERENCE ON DOWNSTREAM TASKS
Once the backbone f is adapted to target protein via self-supervised customization, can be used in conjunction with pre-trained downstream head h, as h f. The key idea of customization with
4

---

Published as a conference paper at ICLR 2026
PLDDT
ESMFold
ESMFold ProteinITT (Step 3) Perplexity: 14.25
ESMFold +
ESMFold +
ProteinTTT (Step 4)
ProteinITT (Step 7)
Perplexity: 15.83
Perplexity: 13.11
Perplexity: 11.66
TM-score: 0.29
TM-score: 0.63
TM-score: 0.86
TM-score: 0.92
16.1
0.95
Perplexity
14.5
0.78
12.8
0.60
IM-S00
TM-sort
11.1
0.43
9.4
0.25
Customization
Figure 3: Customization with ProteinTTT improves protein structure prediction by reducing protein sequence perplexity. ESMFold fails to predict the structure of chain B from PDB entry 7EBL in the CAMEO validation set, as shown customization step where the perplexity is high and the TM-score is low. By applying customization with ProteinTTT for the single target sequence, the model iteratively improves the structure prediction quality, as demonstrated by the increasing TM-score, associated with reduced perplexity. At customization step 7, the predicted structure achieves the highest TM-score, as well as the highest predicted confidence metric pLDDT, enabling the selection of this step the final prediction the customized ESMFold ProteinTTT.
ProteinTTT is not to update the head h, but instead to leverage improved representations from (Figure 2). (opeendxx/Appovvicce justification for why these customized representations generally enhance performance downstream tasks by linking Protein to perplexity minimization.
Since Y-shaped architectures are prevalent in protein machine learning, Protein can be straight- forwardly applied to numerous tasks. In this work, consider three standard problems: protein structure, fitness, and function prediction, and apply our method to corresponding well-established models. For structure prediction, apply ProteinTTT to ESlFold (Figure Lin et al.) (2023). HelixFold-Single (Fang et al. 2023). DPLM2 Bit-based (Hsich et al. 2025). and ESM3 (Hayes et al.
2024;; for fitness prediction,
ESM2 (Lin al. 2023), SaProt (Su et al. 2023), ProSST (Li [Li al. 2024). MSA Transformer (Rao et al. 2021), and ProGen2 (Nijkamp et al.
2023 ; and for function
prediction, we apply ProteinTTT ESM-ly-based (Meier et al. [222] Exploer (Samusevich
et al. 2025) and ESM-lb-bbsed (Rives et al. 2021 Light attention (Stärk et al.) 202T).
In all models we consider, is Transformer encoder operating on protein tokens, and g masked language modeling head mapping embeddings to amino acid types. The downstream head h. however, varies strongly by task. For structure prediction, h is structure predictor: AlphaFold2-innpiree modules in ESMFold, HelixFold-Singl and DPLM2 Bit-wise (Jumper et al., (021). and VQ- VAE decoder in ESM3 (Razavi et al., 2019). For fitness prediction, outputs single score; methods perform zero-shot inference using h f via log likelihoods from g, with h acting simple, parameter-free adaptation of g. For function prediction, is a classifier: random forest in EnzymeExplorer (Samusevich al. 2025 and light attention module in (Stärk al. 2021.
4 EXPERIMENTS
In this section, we evaluate ProteinTTT on three well-established downstream tasks protein machine learning: structure (Section 4.1). fitness (Section 4.2) and function (Section 4.3) prediction.

---

Published as a conference paper at ICLR 2026
Table 1: Customization with ProteinTTT improves protein structure prediction. The metrics averaged across 18 ESMFold low-confidence targets the CAMEO test set, and standard deviations correspond to random seeds. CoT and MP stand for the chain of thought and masked prediction baselines.
Method
TM-score t
LDDT
ESM3 (Hayes al. 2024
0.3480 200057
0.3723
= 20085
ESM3 CoT (Hayes et al.
2024
0.3677 0.0088
0.3835
0.0024
ESM3 ProtemTTT (Ours)
0.3954
0.0067
0,4214
0.0054
DPLM2 Bit-based Hsieh et al.
2025
0.3701
0.0102
0,4681
0.071
DPLM2 Bit-based Protein
(Ours)
0.3796
0.0024
0.4742
= 0.0093
HelixFold-Single Fang et al.
2023
0.4709
0,4758
SeixFold-Singlee Protein
(Ours)
0.4839 +.00445
0.4840
0.0061
ESMFold (Lin et al.
2023)
0.4649
0,5194
ESMFold + MP [Ln et al.
2023
0.4862 0.0043
0.5375
0.0070
ESMFold + Protein TTT (Ours)
0.5047 0.0132
0.5478
0.0058
4.1 PROTEIN STRUCTURE PREDICTION
Protein structure prediction is the task of predicting 3D atom coordinates from an amino acid sequence. It is arguably one of the best-established problems in computational biology (Jumper et al. 2021).
Evaluation setup. To evaluate the performance of ProteinTTT, we employ CAMEO, a standard benchmark for protein folding. We use the validation and test folds from Lin et al. (022)) focusing only on targets with low-confidence predictions from the base ESMFold, as determined by pLDDT and perplexity (Appendix F.Ipp We use the standard TM-score (Zhang & Skolnick 2004) and LDDT (Mariani et al. 2013) metrics to evaluate global and local structure prediction quality, respectively.
As baseline methods, we use techniques alternative to Protein for improving the performance of the pre-trained base models. In particular, the ESMFold paper proposes randomly masking 15% of amino acids in protein sequence before inference, allowing for sampling multiple protein structure predictions from the regression ESMFold model (Lin et al. 2023). For each sequence, we sample number of predictions equal the total number of Protein TTT steps and refer this baseline as ESMFold + MP (Masked Prediction). As a baseline for ESM3, we use chain-of maain-of-thour thought iterative decoding, referred to as ESM3 CoT, proposed in the ESM3 paper (Hayes et al.) 2024).
Results. Customization with ProteinTTT consistently improves the performance of all the tested methods, ESMFold, HclixFold-Sine Single, and ESM3, outperforming the masked prediction (ESMFold + MP) and chain-of-thought (ESM3 CoT) baselines, as shown in Table [ Among the 18 challenging CAMEO test proteins, ProteinTTT significantly improved the prediction of 7, 4, 5, and 6 structures from ESMFold, DPLM2 Bat-based, HelixFold-Siin
and ESM3, respectively, while only moder- ately disrupting the prediction of 2, 1. 1, and structures, respectively (Figure (A6). Remarkably, ProteinTTT improves DPLM2 Bit-based despite the absence of confidence function (no trained PLDDT head available) and despite the model being pretrained via discrete diffusion, while still using the same masked masked-modeling objective for customization as for the other methods.
Most notably, ProteinTTT enables accurate structure prediction for targets that are poorly predicted with the original models. For instance, Figure | presents a strongly improved structure predicted using ESSFoold + ProteinTTT for the target that was part of the CASP14 competition and shown as an unsuccessful case in the original ESMFold publication (Lin et al. (2023), Fig. 2E). Another example is shown in Figure 3 where Pro refined the structure prediction from a low-quality prediction (TM- = 0.29) to a nearly perfectly folded protein (TM-score = 0.92). Figure A4 shows that ESMFold + ProteinTTT maintains computational efficiency of ESMFold, being order of magnitude faster than AlphaFold2. Figure AIll additionall demonstrates the robustness of ESM3 + PTo to the choice of hyperparameters.

---

Published as a conference paper at ICLR 2026
Table 2:
Customization with Protein'TTT improves protein fitness prediction. The right section of the table presents performance averaged across individual proteins then across different protein phenotypes. as classified in the ProteinGym benchmark (Notin et al. 2024). The middle column shows the final performance. averaged across all five phenotype classes. In total, ProteinGyy contains 2.5 million mutations across 186 proteins. Standard deviations are calculated over 5 random seeds.
Spearman phenotype
Avg. Spearman 1
Organismal Fitness
Activity
Binding
Expression
Stability
ESM2 (35M) (Lin 2023
0,3211
0.3137
0.2907
0.3435
0.2184
0,4392
ESM2 (35M) Precem (Ours)
0.3407 20.00144
0.3407
0.2942
0.3550
0.2403
0,4733
ProGen2-smaal (151M) Nijkamp al. 2023
0.3255
0.3316
0.2681
0.3730
0.3283
0.3264
ProGen2-small (151M) Protem (Ours)
0.3591 aL00221
0.3827
0.2960
0.3875
0.3302
0.3992
ProGen2-large (2.7B) (Nijkump et 2023
0.3724
0.4001
0.2820
0,4184
0,3711
0.3904
ProGen2-luuge (2.78) Protem (OirS)
0,3817 a.00ss
0,4091
0.3113
0.4227
0.3661
0.3993
SaProt (35M) Su et al. 2023
0,4062
0.3721
0.3568
0.4390
0.2879
0.5749
SaProt (35M)
Protem (Qurs)
0.4106 20.00044
0.3783
0.3569
0.4430
0.2955
0.5795
ESM2 (650M)
Lin al. 2023
0,4139
0.4254
0.3366
0.4151
0,3691
0.5233
ESM2 (650M)
Proten (Ours)
0.4153 n.0000
0.4323
0.3376
0.4168
0.3702
0.5195
SaProt (650M)
Su et al. 2023
0,4569
0,4584
0.3785
0.4884
0.3670
0,5919
SaProt (650M)
Protein [TT (Our))
0.4583
0.4599
0.3790
0,4883
0.3754
0,5896
ProSST (K=2048)
e1. al. 2024
0,5066
0.4755
0.4448
0,5302
0.4306
0.6526
ProSST (K=2048) + Protemn ITT (Ours)
0.5087 20000054
0.4822
0.4470
0.5321
0.4315
0.6507
4.2 PROTEIN FITNESS PREDICTION
The task of protein fitness prediction to accurately order mutations of a protein based on their disruptive/favorable effects on protein functioning.
Evaluation Setup. We evaluate the models using ProteinGym, the sate--o-ttheart fitness prediction benehmark (Notin et al., 2024). focusing on its well-established zero-shot setup. Since the zero-shot setup only provides test set without any data split, we also validate ProteinTTT on independent data. To achieve this, we create a new fitness prediction dataset mined from MaveDB, a public repository of Multiplexed Assays of Variant Effect (MAVEs) (Esposito et al. 2019). Following ProteinGym, we report Spearman correlation between predicted and experimental fitness values.
Results. TTT consistently enhances fitness prediction performance of all the tested models across varying model scales (35M and 650M parameters for both ESM2 and SaProt; 110M for ProSST) and both datasets, i.c., test ProteinGym (Table 2 and validation MaveDB (Table (AS). Notably, ProSST + ProteinTTT sets new state of the art on the ProteinGym benchmark (Spearman correlation coefficients calculated for individual deep mutational scanning experiments (DMSs) have statistically significant difference according paired t-test with p < 0.05).
We observe that Pro primarily improves performance for proteins with low MSA depth (i.e., the number of available homologous sequences), suggesting that single-sequence customization enhances predictions for proteins with fewer similar sequences in the training data (Table A4.. The fact that Protein'TT more effectively improves the performance of smaller ESM2 and SaProt models compared to their larger variants may be result of the benchmark performance being saturated for larger models, consistent with recent observation (Notin) 2025). We provide a qualitative example showing how ESM2 (650M) + ProteinTTT significantly improves fitness prediction by capturing residues critical for protein stability (Figure A5). We also demonstrate that customization can be combined with evolutionary information from MSA to further boost fitness prediction (Appendix C).
4.3 PROTEIN FUNCTION PREDICTION
Finally, we demonstrate proof of concept for customization in the context of protein function prediction. We experiment with two tasks: predicting protein location within a cell (Stärk et al. 2021), and substrate classification for terpene synthases (TPS), enzymes producing the largest class of natural products (Samusevich et al. 2025). Appendix SeatiixDjshoww [Dssowws that per-protein customization with ProteinTTT consistently enhances the performance of representative models on both tasks.
7

---

Published as a conference paper at ICLR 2026
0 60
Antibody
0,46
Antigen
Andbbddy
Antibody CDRs
HI
5.
0.44
56
LODT
HB
0
0 52
0.5 50
ESMFold ESMFold
0.40
ESMFo
ESMFald
H3
ESMFold
H2
Antigen
Proteiit
+ProteinTTT
CDR LLDTT: 51.34
CDR pLDOT: 61.49
Figure 4: ProteinTTT improves modeling of antibody-antigen loops. (a) Average LDDT on the antibody complementarity -etermiing regions (CDRs, 175 structures) and antigens (814 structures) from the SAbDab dataset with ESMFold pLDDT < 70. Error bars indicate 95% confidence intervals estimated from 1000 bootstrap samples. (b) Example of improved structure prediction for CDRs in the 8K2W entry. The CDR regions HI, H2, and H3, i... the parts of the antibody that bind to the antigen, are highlighted with spheres, while black lines show the alignment error between the ground-truth CDR structure (white) and the predictions (colored).
5 CASE STUDIES
ProteinTTT can be incorporated into structure, fitness, function prediction pipelines with a few lines of code (Appendix Eb. Here, we demonstrate two challenging case studies: improving modeling of antibody-antign loops (Section 5.1 and expanding known structures of viral proteins (Section 5.2)
5.1 MODELING ANTIBODY-ANTIGEN LOOPS
Accurately predicting structures of antibodies (e-g., human defensive proteins) and antigens (e.g. viral proteins) enables rational design of new therapeutics (Bennett et al. (2025). However, the presence of highly variable loop regions makes modeling of these interactions long-standing challenge. Here, we show that ProteinTTT substantially improves structure prediction for these loop-formed pegplemennarity-detemmmin
regions (CDRs) of antibodies, i.e., the parts that bind antigens, as well as for antigens themselves, the well-established SAbDab dataset (Dunbar et al. 2014).
We take the structures from SAbDab that are not predicted well by ESMFold (pLDDT < 70) and show that ProteinTTT improves the LDDT score for 115 of 175 antibody CDR substructures (66%) and 487 of 814 antigen chains (60%). As shown in Figure 4.. ESMFold Protein'TTT achieves significantly higher average LDDT scores compared to general-purpose ESMFold (paired t-test p-value 0.05). Figure A illustrates how Pro enables accurate prediction of all three CDRs in an antibody chain, providing an improved understanding of its binding interface with the corresponding antigen.
5.2 EXPANDING KNOWN STRUCTURES OF VIRAL PROTEINS
Predicting the structures of viral proteins is vital for vaccine development, antiviral design, and understanding infection (Bravi 2024). Nevertheless, it remains challenging due to the high mutation rate, which often leaves viral proteins without close homologs or experimental structures in databases (Kim al. 2025). Here, demonstrate that per-protein customized predictions with ESMFold + Protein improve viral protein structure prediction, substantially expanding the Big Fantastic Virus Database the comprehensive repository of 351, 1,2,44 viral protein structures (Kim et al., 2025).
Among all the entries in BFVD, predicted with AlphaFold2 through ColabFold (Mirdita et al., 2022) using MSAs constructed from Logan (Chikhi al.) 2024), only 55% have high-quality structure predictions (pI.DTT> > 70). We apply ESMFold and ESMFoldd Protein to the BFVD entries to expand the database with higher-quality structures. This achieved by applying three methods to the specific protein and taking the predicted structure with the highest DLDDT. While ESMFold manages to improve the predicted structure (as measured by PLDDT) for 10% of the BFVD proteins, improvement for 19% of the dataset entries, substantially
ESMFold Protein leads increasing the quality of known viral protein structures (Figure 5l)
We validate that the improved PLDDT confidence values from ESMFold + ProteinTTT correlate with the quality of the predicted structures, as measured by LDDT against reference AlphaFold2 structures having pl PLDTT' > 90 (Pearson 0.875; Figure A9). Notably, the largest improvements
8

---

Published as a conference paper at ICLR 2026
100
19N1
LODT ESMFaldd Protein
58
PLODT Inorovementt with ProbeinTTT
PLOOT improvement emmeettet with ProteinTT
Mean
structures with PLDDT
80
10%
50
Medien
very low
werg high
60
high
0.6
36
ESMiFoid ProteinTTT
ESMiFldd
0.4-
23
Improvvene
improxent
20
20
Alphafoid2 (BFVD V2)
Alphafoid2 (BFVD V1)
0.2-
-10
Best PLDUT (no ProteinTTI)
Best PLDUT (with Protein
0.0
50 PLDDT
90
0.0
0.2
LDDT ESMFOId
0.4
0.6
0.8
1.0
100000
1200 3003
[400 1000-5000
1700-5000
100-120000
1200-1 1700)
2300 3200)
10
UP1000641889E
A0A345MMR4
(log scole)
AlphaFald2
ESMFold (LDOT: 0.65)
ESMFold E ProtelnTTT (LCDT: 0.79)
FDB 2N23
(LDOT: 0.65)
ESMFold
AOAGN3IKL3
ADA481ZAK9
AOAGG6XUX6
ESMFold ProteinTTT
AlphaFoldz
AlphaFold2
ESMFold
AlghaFold2
ESNFold
ESMFOId ProteinTTT
70 90 100
Figure 5: ProteinTTT expands the Big Fantastic Virus Database (BFVD). (a) ProteinTTT (light green) substantially improves the performance of ESMFold (yellow) on viral proteins, yielding better structures (pink) for 19% of BFVD entries compared the original predictions by AlphaFold2 (green). (b) Improvements in pLDDT for ESMFold after ProteinTTT correspond to improvements in LDDT, as benchmarked against BFVD AlphaFold2 structures with PLDDT 90. (c) ProteinTTT provides the largest PLDDT improvements (y-axis) for the most out-of disttrribbtiio proteins, i.c., those with the smallest MSAs (left on the x-axis) from the Logan database.
(d) Structural comparison for BFVD entry UPI00064188 1889E against the PDB structure 2N2J (100% sequence identity) shows that ESMFold + ProteinTTT yields a prediction closest to the ground truth (gray), as also measured by LDDT. (e-g) Additional examples of high-qualin viral structures (as measured by pLDDT) predicted with ESlFolld ProteinTTT but not with ESMFold or AlphaFold2. Higher pLDDT values are better.
in PLDDT align with the largest improvements in LDDT (Figure 5p). We find that the benefit of customization saturates with the number of homologs available for a protein, indicating that ProteinTTT is most effective for challenging, dustif-diittibuution proteins (Figure (p)). Finally, Figure 5lg shows examples where Pro enables high-coo high-confidence structure predictions in cases where ceneral-purpoct uncustomized AlphaFold2 and ESMFold struggle.

---

Published as a conference paper at ICLR 2026
6 DISCUSSION
We introduce ProteinTTT,, a method for customizing protein language models to individual targets. PTo consistently improves performance across various models, their scales, and downstream tasks. It excels on challenging, demonstrate its practical value through two case studies: enhancing the structural prediction of difficult antibody-antigen loops and improving 19% of low-confidence viral protein structures in the Big Fantastic Virus Database. Our work establishes per-protein customization as a powerful and practical tool for biological research.
-it-r-ddistriuution examples where general models often fail. We
ACKNOWLEDGMENTS
We thank Milot Mirdita for discussions and feedback about this work. This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through projects e-INFRA CZ [ID:90254], ELIXIR [LM2023055], CETOCOEN Excellence CZ.02.1.0l0 1.100../00.0///....... 0000_04430006632. ESFRI RECETOX RI LM2023069. This work was also supported by the CETOCOEN EXCEL- LENCE Teaming project supported from the European Union's Horizon 2020 research and innovation programme under grant agreement No 857560. This work was also supported by the European Union, ERC FRONTIER (No. 101097822). ELIAS (No. 101 120237)) and the Technology Agency of the Czech Republic under the NCC Programme from the state budget (No. RETEMED TN02000122) and under the NRP from the EU RRF (No. TEREP TN020001 2200000I,///,N),,, This work also supported by the EU's Horizon Europe Programme under the Grant agreement No. 101 101136607 (CLARA Project), and was co-funded by the EU from the Operational Programme Jan Amos Komenský (OP JAK) (project "Center for Artificial Intelligence and Quantum Computing in System Brain Research, reg. no. CZ.02.01. 00..010///003.1.111111111111 00110///2302//00000337777777 This work was also supported by the Czech Science Foundation (GA CR) grant 21-11563M and by the European Union's Horizon Europe program (ERC, TerpenCode, 101 70268). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. M.S. acknowledges sup- port by the National Research Foundation of Korea grants (RS-2020-NR04954) RS-2021 659 and AS-2021-NRO56571 RS- 2224000666)), Nordisk Foundation (NNF24S This work was supported with funding from Google.org via the Google PhD Fellowship (R.B.).
Researchers Program and Novo
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024. doi: 10. 541586-004-07487-w URL wttp:::ttts//tt org/ 10. 1038/ $ 586 020--004--0777777
Sarah Alamdari, Nitya Thakkar, Rianne Van Den Berg, Neil Tenenholtz, Robert Strome, Alan M Moses, Alex X Lu, Nicolò Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence all you need. BioRxiv, pp. 2023 09, 2023. doi: 1001/22// 09.11.556673. URL httpt:ttp///// wrw/10..00.........oo0oo 11011111222 220112023.0 222000230333 09. 5566633
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence based deep representation learning. Nature methods, 16(12):135-1122) 2019. doi: 10.10399441922-09--555---- URL https://///// //doi
requence-baseed
org/10. 1 0331/4441920011-555555
José Juan Almagro Armenteros, Casper Kaae Sønderby, Søren Kaae Sønderby, Henrik Nielsen,
and Ole Winther. Deeploc: prediction of protein subcellular localization using deep learning. Bioinformatics, 33(21))3387-3399 2017. doi: 10. cs/btx431
URL https
//doi, .aciiiorg//ie 11w.......10.i....
Frances M. Ashcroft, Matthew Lloyd, and Elizabeth A. Haythorne, Glucokinase activity in diabetes: too much of good thing? Trends in Endocrinology & Metabolism, 34 Feb 2023. ISSN 1043-2760. doi: t0.10166j.tem..022... 12.007. URL htt.httpps. /tttpa://oii.ooe http://....../11..0.....0.0.....
tem. 2022. 20007
10

---

Published as a conference paper at ICLR 2026
Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. Sulf-super test-time learning for reading comprehension. arXiv preprint arXiv:2103.11263, 2021. doi: 10.483500/rX1..2103.1 1263. URL
https:://bo..rr//111 2ttp:://...........2....2.......
Eyal Ben-David, Nadav Oved, and Roi Reichart. Pada: Example-based prompt learning for on-the-fly adaptation unseen domains. Transactions of the Association for Compulational Linguistics, 10:414-45 2022. doi: 10.48550VarXi.22102.2222 /tt.:///................././/... /tt.://::/.....//1.....0...0
URL BRLhttps:
arXiv. 2102 12206
Nathaniel R Bennett, Joseph I. Watson, Robert Ragotte, Andrew J Borst, DéJenáé I. See, Connor Weidle, Riti Biswas, Yutong Yu, Ellen Shrock, Russell Ault, al. Atomically accurate de novo design of antibodies with rfdiffusion.. bioRxiv, pp. 2024 03, 2025. doi: 00.1001/2024.0331455855100 14. 585103
URL
Barbara Bravi Development and use of machine learning algorithms in vaccine target selection. npj Vaccines, 9(1):15, 2024. doi: 10. 00.1000//4111--033000000
URL httpa:///oi..rr//11
1038/ 0036/4415410023-0775555
Tom B Brown. Language models are few-shot learners. arXiv preprin ar1i:00mmm.. 2020. doi: (0.48550/rrxi.22055.1111 htt/:ttpss///ool.o///1..... 2ttp:/10...55//raaxxx/// 2
Anton Bushuiev, Roman Bushuiev, Anatolii Filkin, Petr Kouba, Marketa Gabrielova, Michal Gabriel, Jiri Sedlar, Tomas Pluskal, Jiri Damborsky, Stanislav Mazurenko, al. Learning to design protein- protein interactions with enhanced generalization. arXiv preprint arXiv:2310. 2023. doi: URL
1..48550/arXii.2310..
18515
Andrew Camphell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXivv24022.04997, 2024. doi: 10.48550/uxXiv.2422.4400 URL https:
org 20tg/0004855//222..22222
04997
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress statistical language modeling. arXiv preprint arXiv:1312.3005. 2013. doi: 14445500/rXii.1112..33 URL https
Tianlong Chen and Chengyue Gong. Hotprotein: A novel framework for protein thermostability prediction and editing. NeurlPS 2022, 2022. URL htt.atpps/oooeeeee..e....eeee/ee
Tdd=YDJFFBMIN
Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilè Žemgulyté, Taylor Applebaum, Alexan- der Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, et al. Accurate proteome-wide wide missense variant effect prediction with alphamissense. Science, 38 38666666))) 2023. doi: 10.1126/science.. adw.eeenne...d.dg..... 7497 URL
adg7492
Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, and Yang Wang. Adapting distribution shift by visual domain prompt generation. arXiv preprint
MrXiv:2405.02799 2024. doi: 10.48550/r2ivv.4455.00 48550/arXiv 2405. 2400.0277777777 02797
URL
Rayan Chikhi, Téo Lemane, Raphaël Loip-Krippleber, Mercè Montoliu-Nerin, Brice Raffestin, Antonio Pedro Camargo, Carson Miller, Mateus Bernabe Fiamenghi, Daniel Paiva Agustinho, Sina Majidian, et al. Logan: planetary-scale genome assembly surveys life's diversity. bioRxiv. pp. 07,4-07, 2024. doi: 10. ww..110/2224407.30... 6068888. URL httphhttss/....rggg//// 2024 07. 30 605881
Yehlin Cho, Martin Pacesa, Zhidian Zhang, Brumo E Correia, and Sergey Ovchinnikov. Boltzdesignl: Inverting all-atom structure prediction model for generalized biomolecular binder design. bioRxiv, pp. 2025 04, 2025. doi: 10.1101//025.04.06.00 2025. 04 06. 647261
URL httpt::t://// //doi. ottp....gg/01111111111
11

---

Published as a conference paper at ICLR 2026
Cyrus Chothia and Arthur M Lesk. Canonical structures for the hypervariable regions of immunoglob ulins. Journal of molecular biology. (1144))901-991)) 1987. doi: 10.101660022-22366677)/88888
90412-8
David W. Christianson. Structural and chemical biology of terpenoid cyclases. Chemical Reviews,
117(17):11500-11648, Sep 2017. ISSN 0009-2665. doi: 10. 001101//cc.chhmmrex7702222 URL
https://aoi.org//11 1021/ace chemrev. 7b00287
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2023. Nucleic acids research, 51(D1))D523-D531 2023. doi: 10. 10.1093/nar/kkac10053 URL httph:ttp:///ttttm http::///doooorggg..
/
10. 10.1093/nar/gkacl.n
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Rohert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep dearning-base protein sequence design using proteinmpnn. Science, 378(6615):49-56, science.add2187.
2022. doi: 10.1126/
URL
acd2187
Jacob Devlin. Bert: Pre-training of deep bidirectional transformerss for language understanding. arXiv preprint arXiv:1810.044805, 2018, doi: 10.485500arXi..1111.008888 URL https:///////oorr 10. 20.48550/arxi11 1810 04805
Daniel J Diaz, Chengyue Gong, Jeffrey Ouyang -hanng James M Loy, Jordan Wells. David Yang, Andrew D Ellington, Alex Dimakis. and Adam Klivans. Stability oracle: aructure-based graph-transformer for identifying stabilizing mutations, BioRxiv, pp. 2023-05, 2023, doi: 10.1038/ s41 4414470024-49780- URLhttts:://dot org/10. 1038/1103334441 http::///......ooo../o/0///11110
Henry Dieckhaus, Michael Brocidiacono, Nicholas Z Randolph, and Brian Kuhlman. Transfer learn- ing to leverage larger datasets for improved prediction of protein stability changes. Proceedings of the National Academy of Sciences, 121 ()) 2024. doi: 10.1073/pna..23148855
URL /eEtps://do::. http:///oooooo//.o./o0//
James Dunbar, Konrad Krawczyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42 doi: 10. 10.1093/nar/kkt1044
(0):D1140-D1144 2014.
nar/ /az/g/
Janani Durairaj, Andrew M Waterhouse, Toomas Mets, Tetiana Brodiazhenko, Minhal Abdul- lah, Gabriel Studer, Gerardo Tauriello, Mehmet Akdel, Antonina Andreeva, Alex Bateman, et al. Uncovering new families and folds in the natural protein universe. Nature, 622(7983): URL btttts:://di..rgg///
646-653, 2023. doi: 10.
s41586 02 0662--662222
Oliver Dutton, Sandro Bottaro, Istvan Redl, Michele Invernizzi, Albert Chung, Carlo Fisicaro, Falk Hoffmann, Stefano Ruschetta, Fabio Airoldi, Louie Henderson, et al. Improving inverse folding models at protein stability prediction without additional training or data. bioRxiv, pp. 2024-06 06. 2024. doi: 10.1101/2024.06.155.99 599145
URL htttt:://////// www.oogg......... 11tp://22..1112221... 06.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):1112-112) 2021. doi: 10. 10911000mmiim0002 2021339533.. URL
https: /ww..///ii..o/ wttp://...111/pppaa..e 2021. 3095381
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Char- lotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general- purpose modelling. arXiv preprint arXiv:2301.06568 2023. doi: 10448550/a22w..23100066
URL wttp::////oo....00......0 100.:.000001....5///aaa.. UtT.1//pe////oooorr//////5222222 www.aaiwiv.......o. 06568
Daniel Esposito, Jochen Weile, Jay Shendure, Lea M Starita, Anthony Papenfuss, Frederick P Roth, Douglas M Fowler, and Alan F Rubin. Mavedb: open-source platform to distribute and interpret data from multiplexed assays of variant effect. Genome biology, 20:1 2019. doi: 10.1 3059
s13059-019- 111-09999-1----- 1845-6. URL https:://///......p.m / ://o: .otg//0.
12

---

Published as a conference paper at ICLR 2026
Richard Evans, Michael O'Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Zidek, Russ Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with alphafold-muline mullemer.. biorxiv, pp. 2021-10, 2021. doi: dbo..or//0.1111/021.........
URL
https: http:::/dooo.ooo/ org/ w1w..11111110001 1001/022111111 10. 04 463034
Xiaomin Fang, Fan Wang. Lihang Liu, Jingzhou He, Dayong Lin, Yingfei Xiang, Kunrui Zhu, Xiaonan Zhang, Hua Wu, Hui Li, et al. A method for muuliple-sequencc-aliinn free pro- tein structure prediction using protein language model. Nature Machine Intelligence, 5(10): 1087-1096, 2023. doi: 10. 001008//42256-003-0002222 /ttp/
s42256- 023-
Tao Feng, Ziqi Gao, Jiaxuan You, Chenyi Zi, Yan Zhou, Chen Zhang, and Jia Li. Deep reinforcement learning for modelling protein complexes. arXiv preprint wwt.::::20000029999.. 2024. doi: 10.48550/ .00.0.4485500 48550/ arxiv 2405. 02299
arXiv.2405.02299 URL /tRptttts://o..ccoo
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked au-
toencoders. Advances in Neural Information Processing Systems, 35:29374-293855 2022. doi: 10.448500\arXi.22200.7755
URI.https:: http://ooo.orgggg// wwwg/1.......o.. 48550
Cade Gordon, Amy X Lu, and Pieter Abbeel. Protein language model fitness a matter of preference. bioRxiv, pp. 2024-10, 2024. doi: 10.1101/2024.0.03..666666 URL httpp:://oi.org/
1101/2024. 10. 03. 6122
Jan Gorodkin. Comparing two k-category assignments by a k-category correlation coefficient. Computational biology and chemistry, 28(5-6):367-374, 2004. doi: (0.0016/.compbiolchenn 2004. 09.006. URL https: doi ottp://..r...........ooooo...a..
2004. 09.006
Xin Gu, Patrick Jouandin, Pranav V. Lalgudi, Rich Binari, Max L. Valenstein, Michael A. Reid, Annamarie E. Allen, Nolan Kamitaki, Jason W. Locasale, Norbert Perrimon, and David M. Sabatini. Sestrin mediates detection of and adaptation to low-leucine diets in drosophila. Nature, 608(7921):209-216, Aug 2022. ISSN 1476-4687. doi: 10.10388/44158--22--044499 btttp::/ddi..rr/
URL
Muhammet F. Gulen, Natasha Samson, Alexander Keller, Marius Schwabenland, Chong Liu, Selene Glück, Vivek Thacker, Lucie Favre, Bastien Mangeat, Lona J. Kroese, Paul Krimpenfort, Marco Prinz, and Andrea Ablasser. egas-sting drives ageing-related inflammation and neurodegeneration. Nature, 620(7973):374-339, Aug 2023. ISSN 1476-4687. doi: 10. 10010087441186--033006633----11
URL htRl:htts::///00 .rgg 11111111111111111111111 1486 011-8//2415666222-0653333 023-06373-
Kathryn H. Gunn and Saskia B. Neher. Structure of dimeric lipoprotein lipase reveals a pore adjacent to the active site. Nature Communicationns, 14(1);2569, May 2023. ISSN 2041-1723, doi: 1031038/ 241167-022-382444
URL/https:/ /ttp::::///oooioo
10..//0111000 1038/ 01////44147-2238822888
Sarah Gurev, Noor Youssef, Navami Jain, and Debora S Marks. Variant effect prediction with
reliability estimation across priority viruses. bioRxiv, pp. 1025-08. 2025. doi: 10.1101/2025.08 04.668549. URL hRttht:ts::/ee /ttp:////////////....///////0//0
08. 6468844999999999
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv htLp:::s: /rg.
preprint Xrvv 18466, 2023. doi: 10448550/arxiw.235..8466
10. 45.48800/arx/0 wwt.550/axiiv...
18466
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. bioRxiv, pp. 2024 07, 2024. doi: 10. wtw/ssiinnce.as0000 URL https: www www.ssciencee org/ doi 10. 1121 stt///cienccs ads 0018
Michael Heinzinger and Burkhard Rost. Teaching ai to speak protein. Current opinion in structural
biology. 91:102986, 2025. doi: 10.1016/j.sbi.20223 102986. URL hattps://dii.org///
1016 sbi 0221..22555 2025. 102986
Lucas Torroba Hennigen and Yoon Kim. Deriving language models from masked language models.
arXiv preprint arXiv: 15501, 2023, doi: 10.448500/rxxiv2205.5555 URL https::/aa //doi
org hTg/10048550/ara 2305. 15501
13

---

Published as a conference paper at ICLR 2026
Thomas A Hopf, John B Ingraham, Frank Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora S Marks. Mutation effects predicted from sequence co-variation. Nature biotechnology. 35(2):128-13) 135, 2017.
doi: 10.1038/nbt.379. URL httppa://dbl..og///22
1038/nbt.
Gerta Hoxhaj and Brendan D. Manning. The pi3k-akt network at the interface of oncogenic signalling and cancer metabolism. Nature Reviews ( Feb 2020. ISSN 1474-1768. doi: 10.
0058//41558--19-0216--
019-0216-7
Cheng- Yen Hsich, Xinyou Wang. Daiheng Zhang, Dongyu Xue, Fei Ye. Shujian Huang, Zaixiang Zheng, and Quanquan Gu. Elucidating the design space multimodal protein language models.
arXiv preprint arXiv::5044 /454, 2025. doi: 10.48550// 11444. URL https: 11454 11454
org/10. 48550/ wwt.://axxv......
org/10.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint doi: 1..48500arXxi.2106.066666 URL https:: settps:/ddiiorr/111
arXiv:2106.0666 2021.
48550/arXiv. 2106. 09685
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning test-time: Active fine-tuning of llms. XrXiv preprint arXiv:2410.080020, 2024, doi: 00.48550//rxiv.24100 08020. URL httpt:t://// httt:p::/doii.rgge http:po.rrrr.........aa..2...... www.:riww...........
48550/
Jonas Hübotter, Patrik Wolf, Alexander Shevchenko, Dennis Jüni, Andreas Krause, and Gil Kur. Specialization after generalization: Towards understanding test-time training in foundation models.
arXiv preprint arxih:2509.245510 2025. doi: 10.485550aaXii.250922255 24510. URL https::/dooi
org 0111/10.485550 /
24510
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873),583- 2021. doi: 10.
s41586-021-03811
URLpttps: http://///...a.coo......///////. 10. 1010111..1111111
Pranav Kantroo, Gunter Wagner, and Benjamin Machta. Pseudo- Pseudo-perplexity in one fell swoop for protein fitness estimation. bioRxiv, pp. 2024 07, 2024. doi: 10,48550%arxiv.2407.00m
URL
https: wor..orggiil 18550/arXiv.... www.rxivv.40...00 2407 07265
Neerav Karani, Ertune Erdil, Krishna Chaitanya, and Ender Konukoglu. Test-time adaptable neural networks for robust medical image segmentation. Medical Image Analysis, 68:101907, 2021. org 10. 00.00.1111155 wwtp://..meeia...mm medim. 2020
doi: 10. we00://.mmdia.2220.....
URL
101907
Zuzana Keckesova, Joana Liu Donaher, Jasmine De Cock, Elizaveta Freinkman, Susanne Lingrell, Daniel A. Bachovchin, Brian Bierie, Verena Tischler, Aurelia Noske, Marian C. Okondo, Ferenc Reinhardt, Prathapan Thiru, Todd R. Golub, Jean E. Vance, and Robert A. Weinberg. Lactb is tumour suppressor that modulates lipid metabolism and cell state. Nature, 543(7647)6881-6666 Mar 2017. ISSN 1476-4687. doi: 10. 10.1038/natrre24400 URUJattps:///oi.or/////// nature21408)
Rachel Seongeun Kim, Eli Levy Karin, Milot Mirdita, Rayan Chikhi, and Martin Steinegger. Bfvd large repository of predicted viral protein structures. Nucleic Acids Research, 53(D1):D340 2025. doi: 10. 10.1a///ar/g 19. URL https: www.oo.oo..o....oo 1093 htt./////......00/////////.111..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: wwwaa.avvvo... /a/////a///a 1411411222 6980
Andrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi Jaakkola, Regina Barzilay, and Ila R Fiete. Improving protein optimization with smoothed fitness landscapes. In The Twelfth International Conference on Learning Representations, 2023, URL TRLattps. /ttl/sttt////peeeiiiii net. http://oummmoxxxx1222222
14

---

Published as a conference paper at ICLR 2026
Petr Kouba, Pavel Kohout, Faranch Haddadi, Anton Bushuiev, Raman Samusevich, Jiri Sedlar, Jiri Damborsky. Tomas Pluskal, Josef Sivic, and Stanislav Mazurenko. Machine learning-guided protein engineering. ACS catalysis. 13(21): URL htt..httts:///oi..rrr
2023. doi: 10.1022/aacccaaal33307774
1021 cscatal 1
3c02743
Elodie Laine. Yasaman Karami, and Alessandra Carbone. Gemme: simple and fast global epistatic model predicting mutational effects. Molecular biology and evolution, 36(11):2604-2611 2019.
doi: 10. 10.199/////ee//ms21799
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Liang Hong, and Pan Tan. Prosst: Protein language mod- eling with quantized structure and disentangled attention. bioRxiv, pp. 2024-04, 2024. URL ht httpp::/ppooeeeinnggss.... neur
File/ htee:/dd5522222 http:///ppppooeeenneeeeeeeeeeecs http:/paapr_flleeppppppeea
Pape-eaaaerr-oeeecceeeeeee-e Conference pdf
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi Tom Scrcu, Salvatore Candido, and Alexander Rives. Evolutionary-ssale prediction of atomic-level pro- tein structure with language model. Science, 37966637 2023. doi: 10. 126/ science.ade2574. URL https: /www.wwww...m...o wwtp:/cccnnnc...oooeoeeeoe.ee//e ade2574
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. TTT++: when does self-supervised test-time training fail or thrive? In Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 21808 21820, 2021, URL b61 bb222222061111120000102222222222
wwwrips. ottp://daa.a.a.ooooooooooooeep/d
html
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on informationn theory, 28(2):
129-137, 1982. doi: 10.1 10.1109/TIT. 1982.1119922.0054488 /BLLittps://dd.org//10000 1982.105C
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URLhttis:://ppeeevieee www:forumm 2ttp:/orrrmm1.ddB...........
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddt: local superposition- score for comparing protein structures and models using distance difference tests. Bioinformatics, 29(21))2722-22723 2013. doi: 10.1093bbionfommatcchht11..... URL
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations pro- tein function. Advances in neural information processing systems, 34:29287-29303, 2021.
751338d736f9555 11223327776353522222222223333333 http:/o000000779----atttaaatattd httpa//ppp..ccppppp/////////se/
URL httpp:://proceedingg.eeet
Abstract.
Peter Mikhael, Itamar Chinn, and Regina Barzilay. Clipzyme: Reaction-conditionee virtual screening of enzymes. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Aus- Iria, July 21-27, 2024. OpenReview.net, 2024. URL www.ltttpp////m http::s:///peereeieeeee net forum?? 1d= 0mYAK6
Milot Mirdita, Konstantin Schlitze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Mar- tin Steinegger. Colabfold: making protein folding accessible all. Nature methods, 19(6): doi: 10.
682, 2022.
URL httptt//p// /rt.::::////oi..///00... 1038/
s41592- 022 0248-128---1--1
Erik Nijkamp, Jeffrey A. Ruffolo, Eli N. Weinstein, Nikhil Naik, and Ali Madani. Progen2: Exploring the boundaries of protein language models. Cell Systems, 14(11):968 978.e3, Nov 2023. ISSN 2405-4712. doi: 10.1016fj.e13.2223.00000 URL htt.h:tps://ooo org. 10000..11166110
2023 002
15

---

Published as a conference paper at ICLR 2026
Pascal Notin. Have we hit the scaling wall for protein language models? Sub-
stack blog post. May 2025. URL htttps:/paascaanttee substack com/p/
have-we-hit-the--ee www.ssacaiinaaaaaaaaa-a----
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Nickerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, al. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36. 2024. doi: 10. 10.1101/2023.1.....5770000
URL Utt.1tttss///oo..r////111011///0
07 17.577727
Dong oh Seo, David O' Donnell, Nimansha Jain, Jason D. Ulrich, Jasmin Herz, Yuhao Li, Mackenzie Lemieux, Jiye Cheng, Hao Hu, Javier R. Serrano, Xin Bao, Emily Franke, Maria Karlsson, Martin Meier, Su Deng, Chandani Desai, Hemraj Dodiya, Janaki Lelwala-Guruge, Scott A. Handley, Jonathan Kipnis, Sangram S. Sisodia, Jeffrey Gordon, and David M. Holtzman. Apoe isoform- and microbiota-deppedent progression of neurodegeneration in a mouse model of tauopathy. Science, 37966628))aadd11336 2023. doi: 10. 10.11126/ccemce
URL https
Andrei Papkou, Lucia Garcia- José Antonio Escudero, and Andreas Wagner. A rugged yet cas- ily navigable fitness sciee htt.:///eeeeeee.e2266633aaaaaaaa 2023. doi: 10.1 10.11126/ciemce.adh338883 adh3380
Saro Passaro, Gabriele Corso, Jeremy Wohlwend, Mateo Reveiz, Stephan Thaler, Vignesh Ram Somnath, Noah Getz, Tally Portnoi, Julien Roy, Hannes Stark, et al. Boltz-2: Towards accurate and efficient binding affinity prediction. BioRviv, 2025. doi: 10.1101/2025.06 1.............7... URL
hittps://dos /tl.ggg......... 1101111122222 06. 14. 659707
A Paszke. Pytorch: An imperative style, high-performaneee deep learning library. arXiv preprint 11Xfv:1912.01703, 2019. doi: 10448550aaXi..11122.117 URL hattp:://oi..org
98550/ar2iv.1112.00
Predrag Radivojac and et al. large-scale evaluation of computational protein function prediction.
Nature Methods, 10(3):221-221) Mar 2013. ISSN 1548-7105. doi: 10. 10.10088mmethh2344 URL
https://aol. http:////ooooo//////// 10. 10381018810000 a.meth.2340
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021. doi: 10448550/axXx:2122.22999 URL
htttps://do wooll.org/000 00.550 48550/ arxiv.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas. Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019. doi: 10.48550/axxi..1966..88 08230. URL https
//doi.org/ 48550/45555... 1906. 08230
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. Biorxiv, pp. 2020-12, 2020. doi: 10.1101/2020.1 12. 4227711 URL.nttp::://o..org//100
12. 15. 12.
422761
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sereu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp. 8844- 8856. PMLR, 2021. URL httttts::
ml. htt....ppeees/11///a2laa... html html
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC,
Canada, pp. 14837-14847, 2019. URL pttp:::/pprcceetnngggg nwwrips. cttppperrcpppp// Abst .tmi
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):c006623111 2021. doi: (0.1073/maas.01122999 URL https 7/doi mooii.rgg/ie 10. 1073/ 2016239118
16

---

Published as a conference paper at ICLR 2026
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo Tauriello, and Torsten Schwede. Continuous automated model evaluation (cameo) (oneo>)pprspeectvve on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioin- formatics, 89(12)1197-11986 2021. doi: 10. 00.1002/pro..22133 U htttps://doi.org//// 26213
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv: 2016. doi: 10.48550/arXxi.1609..47777 URL
48550/arXiv. 2509.0474
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model 2019, doi: 18.18653/v 1111663//1122220a111laa 18w.1/1663vvvv
scoring. arXiv preprint arXiv:
acll URL maae-mmain.. ww..aann..... 240
https::/doi.orgg
Raman Samusevich, Téo Hebra, Roman Bushuiev, Martin Engst, Jonáš Kulhánek, Anton Bushuiev, Joshua D. Smith, Tereza Čalounová, Helena Smrčková, Marina Molineris, Renana Schwartz, Adéla Tajovská, Milana Perković, Ratthachat Chatpatanasiri, Sotirios C. Kampranis, Dan Thomas Major, Josef Sivic, and Tomáš Pluskal. Structure-enabled enzyme function prediction unveils elusive URLhttps. 577750,
terpenoid biosynthesis in archaea. bioRxiv, 2025. doi: 10.1100/2002.0012295553
www..biorxs org/ http:////..o.aaeaaaaa///////////
Nicolae Sapoval, Amirali Aghazadeh, Michael G. Nute, Dinler A. Antunes, Advait Balaji, Richard Baraniuk, C. J. Barberan, Ruth Dannenfelser, Chen Dun, Mohammadamin Edrisi, R. A. Leo Elworth, Bryce Kille, Anastasios Kyrillidis, Luay Nakhleh, Cameron R. Wolfe, Zhi Yan, Vicky Yao, and Todd J. Treangen. Current progress and open challenges for applying deep learning across the biosciences. Nature Communicationns, 13(1):1728, Apr 2022. ISSN 2041-1723, doi: 1038/ doi http:///ooooorgg/////// 002-117011111111111------------1
241466-022-29268-- URL.ttpss:
Robert Schmirler, Michael Heinzinger, and Burkhard Rost.
fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications, 15(1):7407, 2024. doi: 10310 s41467-024-51844 24 URL htl.https://// htt./oo.oooooooo http:/oooo..rgg/l/////o 10. 1038/ 5 4114446--444 51 51844-2
Emre Sevgen, Joshua Moller, Adrian Lange. John Parker, Sean Quigley, Jeff Mayer, Poonam Srivastava, Sitaram Gayatri, David Hosfield, Maria Korshunova, et al. Prot-vae: protein transformer variational autoencoder for functional protein design. bioRxiv, pp. 2023-01, 2023. doi: 10.1073/ pnas.2408737122. htRtsstts::/// www.ioorg11... 1013111..333 / pnas. 2408737122
Richard W Shuai, Talal Widatalla, Po-Ssu Huang, and Brian L. Hie. Sidechain conditioning and modeling for full-atom protein sequence design with fampnn. bioRxiv, pp. 2025-02, 2025, doi: /doiip org /10 www.1/121222... 02. 13. 637498
10.1101/2025.09 02........1. 13.637498.
Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language models via sparse autoencoders. Nature Methods, pp. 1-11, 2025. doi: 10. 00.10088/411192-255-028666 URL
025 02836-7
Peter Škrinjar, Jérôme Eberhardt, Janani Durairaj, and Torsten Schwede. Have protein-ligand co- folding methods moved beyond memorisation? BioRxiv, pp. 2025-02, 2025. doi: 10. 1001/222. 02.03.636309. URL sattps://dblt org/10...001...00 1101:/200..000222......02 02. 03. 636309
Yidong Song, Qianmu Yuan, Sheng Chen, Yuansong Zeng, Huiying Zhao, and Yuedong Yang. Accurately predicting enzyme functions through geometric graph learning on esmfold-predicted structures. Nature Communications, 8180 2024. doi: 10. 40.10388441447-0245555555 w. URL https://doi. http:///oooooog/1.... www.oogg/....... 0000000/844440 024 52533-22333-
Hannes Stark, Felix Faltings, MinGyu Choi, Yuxin Xie, Eunsu Hur, Timothy John O'Donnell, Anton Bushuiev, Talip Uçar, Saro Passaro, Weian Mao, et al. Boltzgen: Toward universal binder design. bioRxiv, pp. 2025 11, 2025. doi: 00.1100/022.11....668880 //doi.org/10..... 1tt..oor//....00/2225..... 10.20. 689494]
URL h
Hannes Stärk, Christian Dallago, Michael Heinzinger, and Burkhard Rost. Light attention predicts protein location from the language of life. Bioinformatics Advances, 1(1):vbab035, 11 2021. ISSN 2635-0041. doi: 10. /i..0///ioodd//aab03... URL https: /oi, 10t.::////ol..og1/0.0000//////. vbab6035
17

---

Published as a conference paper at ICLR 2026
Jin Su. Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-awar aware vocabulary. bioRxiv, pp. 2023-10, 2023. URL https
//openreview.oo net/ 60
Sriram Subramaniam and Gerard J. Kleywegt. A paradigm shift in structural biology. Nature Methods, 19(1):20-23, Jan 2022. ISSN 1548-7105, doi: 10. 1011008/415922-011-0161--777 URL
fttps://doi..rr .tp.doio.rg..oo 1038/ 021-19222221-111---1
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229 9248. PMLR, 2020. URL /ttpss:/pprceeeingss mlr
press/vl sun sun20b. pdd
Nataša Tagasovska, Ji Won Park, Matthieu Kirchmeyer. Nathan C Frey, Andrew Martin Watkins, Aya Abdelsalam Ismail, Arian Rokkum Jamasb, Edith Lee, Tyler Bryson, Stephen Ra, et al. Antibody domainbed: -ut-of-distribution generalization in therapeutic protein design. arXiv preprint arpp:raanarxii::2247222000000 2024. doi: 10:48550/arXiw..400.2200 URL htttts::///i.....o
40.48550/arX
21028
Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan Weinstein, Niall M Mangan, Sergey Ovchinnikov, and Gabriel J Rocklin. Mega-scale experimental analysis of protein folding stability in biology and design. Nature, 620(7973):
434 444, 2023. doi: 10.1038/s41 (0038/441886-255 0886-023--6328 541586- 023-6660230663-
URL
Mike Tyers and Matthias Mann. From genomics to proteomics, Nature, 422(4449)2222219119999 Mar 2003. ISSN 1476-4687.
doi: 10. 1038//aauure01510. URL
nature01510
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pp. 2022 02, 2022. doi: 10. 10.1088/441588.023--1773..0 URL /
1038/ 1038/
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of sequence space with high-accuracy models. Nucleic acids research, S0(D1))D439-D444 2022. doi: 10.1093/nai gkab1061. URL httpstt:: http::://doo.ooooo/oooo
10.48550/arxiv.177666 /ttp:////..ogo///0100///////////
Vaswani. Attention all you need, Advances Neural Informaaiioo Processing Systems, 2017. doi: 170..03 037622
Renhao Wang. Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efrros, and Xiaolong Wang. Test- time training on video streams. arXiv preprint arXivv2337.005014, 2023, doi: 10.48550/arX1w.2307. 05014. htttppp:/ool.ooog00.0455555/////
05014
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402 2024a. doi: 10.48550kaxxiv2442211666
2tt50/100.5555////// 240.li.w.2.2..... 18567
Xinyou Wang. Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang. and Quanquan Gu. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410..3377 2024b doi: 2024b. 2410..11111... 13782
10.44550WarXX..2410.1.777
48550/ http:////...rr/0..55550///////// WRl.httpss///oi..oo
Joseph Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with ridiffusion. Nature, (080-)99)) org/
2023. doi: 1038/ 023 -223-06415-8
541586-023-00415--
Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning generalize across domains on single test samples. arXiv preprint arX.i:2200..804442 2022. doi: 10.48.55500 10.485500/axxx.2200.8880
U http.o/ogg......../.....ooooo
18

---

Published as a conference paper at ICLR 2026
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. En- zyme function prediction using contrastive learning. Science, 379(6639):1388 1363, 2023. doi: 10.1 10.. URL http://wwww...cceee/e org
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatiis, 57(4):702-710, 2004, doi: 20264
10.1002/prot.20264.
Zhidian Zhang, Hannah K. Wayment-Steele, Garyk Brixi, Haobo Wang. Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics interacting sequence motifs. Proceedings of the National Academy of Sciences, 1211455)e2200628512,, 2024. doi: 10. 110073/pnus22402288111 URL http:///ww...pss..rrr///////a//0 pnaa/ 2406285121
Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 42058-42080. PMLR, 2023.
URL htt.hhtsss///ppooeeeee ngs. ml www.presssvvv. / https///222222hh222d2d......
19

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Justification of customization via perplexity minimization
20
Customization beyond masked language modeling
20
Customization with multiple sequence alignment (MSA)
21
D Customization for protein function prediction
22
Implementation details
23
Experimental details
25
Protein structure prediction
25
F.I.
Datasets
25
E.1.2
Metrics
26
3
Models
26
F22
Protein fitness prediction
27
F2.1
Datasets
27
F2.2
Metrics
29
F2.3
Models
29
Protein function prediction
E.3.
Datasets
31
E3.2
Metrics
E3.3
Models
32
Case study details
32
G.1
Modeling antioody-antigen loops
................................
32
G.2
Expanding known structures viral proteins
33
Extended results
33
I
Detailed test performance
33
H.2
Validation performance
34
H.3 Runtime performance
34
Limitations and future work
34
20

---

Published as a conference paper at ICLR 2026
A JUSTIFICATION OF CUSTOMIZATION VIA PERPLEXITY MINIMIZATION
While the paradigm of test-time customization has been investigated in other domains, the reasons behind its sur- prising effectiveness are not completely clear LLu et al. 2021 Zhao et al. 2023). Here, we offer a potential justi- fication for the effectiveness of ProteinTTT by linking it to perplexity minimization.
Method (Correlation)
ESMFo (Pearson=-0.65)
ESM3 (Pearson=-0.77
0.8
TM-score
0.6
Perplexity has traditionally been used in natural lan- guage processing evaluate how well models compre- hend sentences (Brown, 2020lChellba et al.] 2013). Pro- tein language modeling has adopted this metric to as- sess how effectively models "understand" amino acid se- quences (Hayes al. 2024 Lin al., 2023). For bidirec- tional, random masking language models, which are the focus of this study, we consider the following definition of perplexity
0.4
+5
0.2
10
15
Perp exity
Figure Al: Quality of protein structure prediction, as measured by TM-score, correlates with perplexity of the under- lying language model on the challeng- ing targets from the CAMEO validation set. Higher TM-scores are associated with lower perplexity, indicating that bet- ter predictions are linked to lower un- certainty in the language model's 's under- standing of the protein sequence.
Perplexity(A) (x)) = exp
Σ 2=1
log Dlgg(aalaae p(asll
(3)
where [ar] is the length of the input protein sequence and p ()) represents the probability that the model cor- rectly predicts the token 2; position i when it is masked on the input Z.g. Perplexity ranges from I to infinity (the lower, the better), providing an intuitive measure of how well model fits, on average, tokens in given sequence. A perplexity value of indicates that the model perfectly fits the sequence, accurately predicting all the true tokens.
Several studies have shown that lower perplexity on held-out protein sequences (calculated through the self-supervised track g o f) correlates with better performance on downstream tasks (via the supervised track h o f), such predicting protein contacts (Rao et al. (2020), structure (Lin et al. 2023), or fitness (Kantroo et al. 2024). To give an example, we analyze the correlation between perplexity and structure prediction quality (Figure AA]] see Section I for experimental details). A notable correlation suggests that reducing a model 's perplexity on a single target sample æ (applied independently to all test samples) can lead to improved predictions on the downstream task (Figure Figure A10).
Since we assume only single target example x, the minimization of the masked language mod- eling loss [(x;6) 8) (Equation (2) on this example is directly linked to minimizing the perplexity Perplexity(z) (z) (Equation (3). For instance, in the case of single masked position (i.e., [M 1), the loss is equal to the logarithm of perplexity. More generally, it can be shown formally that by minimizing the masked language modeling objective, the model learns to approximate the condi- tional marginals of the language (of proteins), including the leave-one-out probabilities evaluated in perplexity (Hennigen & Kim 2023). As result, applying self-supervised test-time customization on through g enhances the representation of the target protein in the backbone f, leading to improved downstream performance via the fine-tuning track f.
B CUSTOMIZATION BEYOND MASKED LANGUAGE MODELING
In this work, we primarily focus on protein language models pretrained with masked language model- ing (MLM), where fixed proportion of randomly selected tokens (c.g.. 15%) are masked for training. To date, MLM has been the dominant paradigm in protein representation learning. Nevertheless, we also provide a proof of concept showing that ProteinTTT can be applied to autoregressive and discrete
2 Pleasee note that this is an approximation of perplexity, which is computationally intractable for hidirectional models, and is often referred pseudo-perplexit (I.n et al. 2023 Salazar al. 2019).
21

---

Published as a conference paper at ICLR 2026
diffusion-baast protein language models, with details provided in the corresponding paragraphs below. Furthermore, in Appendix [] we discuss how ProteinTTT could be extended beyond protein language models.
Autoregressive customization objective. To perform single-sequence customization in an autore- gressive setting (i.e.. customization of ProGen2 (Nijkamp et al. 2023)), we apply a standard teacher forcing procedure (Vaswani, 2017) with batch size of one. Specifically, each ProteinTTT step optimizes next token prediction across the whole sequence in parallel via the following loss function:
|
1
CAR 0)
log xee;
where denotes sequence of protein tokens, and p(a;; < 0) = (3 0)),; is the probability assigned by the model to the true token given all preceding tokens Teg. Here, we use the notation consistent with Equation
Discrete diffusion customization objective. Recently, discrete diffusion protein language models have emerged as an extension of MLM-based protein language models. Instead of masking a fixed ratio of tokens, discrete diffusson approaches vary the masking ratio during training according to diffusion schedule (Hsieh et al. 2025| Wang et al. 2024b a Campbell et al. 2024; Alamdari et al.) 2023). This has been shown to improve representation learning and to enable sequence generation by starting from a fully masked sequence and gradually denoising it (Wang et al.) 2024a).
In this work, experiment with the DPLM2 Bit-based discrete diffusion model (Hsieh al. 2025 for protein structure prediction. Interestingly, we find that using standard MI objective with fixed 155 masking ratio for customization (Fquation (2) already improves performance. Exploring modifications of the customization objective tailored specifically to discrete diffusion models presents an exciting direction for future work.
C CUSTOMIZATION WITH MULTIPLE SEQUENCE ALIGNMENT (MSA)
Table A1: ProteinTTY can be used with MSA when available. Please see Table for evaluation details.
Method
Avg. Spearman 1
ESM2 (Lin al. 2023
0.4139
FSM2 + Protein
MSA (Ours)
0.4299 0.00099
MSA Transformer (Rao et al. 2021)
0.4319
MSA Transformer + Protemn (Ours)
0.4326 0.00003
Customization training objective. Since many target proteins may not have homologous se- quences (Rao al. 2021) and finding such homologs may be time-consuming (Lin al. (023)..
the Protein''T' customization objective (Equation (2)) only assumes a single target sequence for customization. However, we also extend the loss function to the case when multiple sequence alignment (MSA) is available:
CMSA (2; = Eg' [C]
(5)
where PMSA (x'l (a'|ær is the distribution of sequences x' homologous to the target protein x, L is the single-sequence loss function defined in Equation (2) and 0 denotes the tunable parameters of the model backbone f. We refer to customization using Equation (5) as ProteinTTTwss
Results for fitness prediction. is known that evolutionary information is important for protein fit- ness prediction (Laine et al. 2019). Therefore, we demonstrate how ProteinTTTMs, and Pro can enhance the performance of PLMS on the ProteinGym benchmark (Notin et al., 2024). Table A1 shows that using Protein MSa with high-quality MSAs curated by Notin et al. (2024) strongly enhances the performance of ESM2, approaching that of MSA Transformer, pre- pre-trained on MSAs. Moreover, we find that MSA Transformer slightly benefits from single-sequence customization
22

---

Published as a conference paper at ICLR 2026
Correct TPS substrates
Incorrect TPS substrate
TPS enzyme (UniPrat A0A075FBG7)
Predicted probability
0.3
Predicted prebabilty
0.3
Predicted prebabiltt
0.3
Predicted probability
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.0
TPS substrates
substrates
TPS substrates
substrates
customization)
ProteinTTT customization step
Figure A2: Customization with Pro enables the correct substrate classification for a terpene synthase (TPS) enzyme. With progressive customization steps of EnzymeExplorer + ProteinTTT, the probability of the initially misclassified substrate (red) decreases, while the probabil- ity of the true substrates (green) increases. The bar plots also display the predicted probabilities for other substrates with non-zero values (grey).
with ProteinTTT, while customization whole or subsampled MSAs disrupts the performance (Ta- ble A3 in Appendix H.2). Please note that similar results were previously demonstrated in (Gordon ct al. 2024) and (Alley et al. 2019 by fine-tuning protein language models on MSA, referred as "evotuning".
D CUSTOMIZATION FOR PROTEIN FUNCTION PREDICTION
Protein function prediction essential for understanding biological processes and guiding bioengi-
neering, but is challenging due to its vague definition and limited data Yu et al. 2023 Radivojac
& et al. 2013 Stärk et al. 2021 Mikhael et al. 2024 Samusevich et al. 2025). While improved
structure prediction with Protein ITT (Section 4.1h already enhance function prediction (Song
et al. 2024). we also evaluate our customization method directly on two function classification tasks:
subcellular localization, predicting protein location within cell (Stärk et al.] 2021), and substrate classification for terpene synthases (TPS), enzymes producing the largest class of natural products (Christianson, 2017 Samusevich et al. 2025). Using ProteinTTT with EnzymeExplorer (Samusevich et al.) (025) for TPS detection and Light attention (Stärk et al. 2021) for subcellular localization, we achieve consistent performance gains.
Evaluation setup. For the terpene substrate classification, we use the largest available dataset of characterized TPS from Samusevich et al. (2025) and reuse the original cross-validation schema. In the case of protein localization prediction, we use a standard Deepl.oc dataset (Almagro Armenteros et al. 2017 as validation set and setHard from Stärk et al. (202T) as the test set.
Given a protein, the goal of function prediction is to correctly classify it into one of the predefined functional annotations. We assess the quality of the TPS substrate prediction using standard multi- label classification metrics used in the EnzymeExplorer paper (Samusevich et al. 2025): mean average precision (mAP) and under the receiver operating characteristic curve (AUROC). In the case of protein localization prediction, we similarly use the classification metrics from the original paper (Stärk al. 2021): accuracy, multi-class Matthews correlation coefficient (MCC), and F1-score.
Results. Customization with ProteinTTT improves model performance on both of the protein func- tion prediction tasks and across all considered metrics (Table A2). Figure A2 provides qualitative
23

---

Published as a conference paper at ICLR 2026
Table A2: Customization with ProteinITT improves protein function prediction. For the terpene syntase (TPS) substrate classification task, the metrics are computed on the 512 TPS sequences based on the cross- validation schema of the TPS dataset (Samusevich et al. 2025). Subcellular localization prediction performance reported for 432 protein sequences from the setHard test set (Stärk et al. 202T). The error bars show standard deviations across five random secds.
TPS substrate classification
Subcellular localization prediction
Method
mAP1
AUROC
Accurasy
Fl-scrcc
Eoploeeixplorer
0.805
0.948
attentios
(Ouns)
Enzy mxceiiee PIREIITT
0.811 + 0.0011
0.950
0.0002
result, where customization with ProteinTTT iteratively refines the prediction of EnzymeExplorer toward correct TPS substrate class. We hypothesize that improvement with customization is more challenging in classification tasks, as opposed to regression problems, because a larger change latent space is required to shift the top-class probability.
E IMPLEMENTATION DETAILS
import esm
from wwwteinttt.nnnneeeeeeee
import ESMFoldITT, DEFAULT ESMFOLD TTT CFG
4 Set protein sequence
5 sequence =
8
9
Load mode_ model www.ppeetraidd.eettt.... model = model.eval () ouda ()
13
(model, sequence)
PLDET: 38. 43025
18
ProteinTTT
Customize model to sequence
model ESMFoldTTI. tearrrrttt__foo_pp_ fro___frroooooieedddd
model, BWW.DDDAAAUTTTLMMLLL.C III__FG, confold.coffigrmoel..ccc
22
model.tte (sequence)
predict_structuue structure (model, sequence)
27
PIDDI: 78. 69619
ProteinTTT
Reset model to original state (after this model.ttt can be called with another protein) model.ttt_resett
Code snippet I: Incorporation of ProteinTTT into an ESMFold structure prediction pipeline using the proteinttt package.
24

---

Published as a conference paper at ICLR 2026
import torch
2 import esm
from esm.model, www.e..eellee.......mmmmmmm import ESM2
from proteinttt.base import TTTMcdule
6 class ESM2ITT (TTTModule,, ESM2)
def __iii____eee____________________ ttt_cfq: TTTConfig, **kwargs):
(self, **kwargs)
ITTMcdule. intt int
(tt_cfg=ttt_ofg)
10
self.ttt_alphabet -tw.aaaaeeaeeeeeeooooo.ooaooeae.
("ESM-1b")
11
self..tt_patcch.coneeeet = self. aet.
()
12
def pett_tooenizessell seq: str, #*waargs):
latch_labels, batch_strs, batch_tokens self.ttt_ f...tttt_atcchhcoveertee (
(None, seq)
16
return batch tokens
def _tttt_gtt_froooonnodll (sclf) 1ist [torch. Module]
20
return (solffcmbed_ttoknn))
def ttt_mask__toenn(stt (self, tcken: int) -> int:
return self sel...ttt_alplaaeet.ask..
25
def tettttte__pppeeeeeeeeeeee Btt__get_paddnggteeee ptwpppaadddddeennnepepppg -> int: return self..ttt_lphhaet... padding _ix
def _ tcken: int) str: return
29
[token]
def att__ge__all_ooeee (self) (self)
list [int]:
return
stl.ttt..llpaot..ckkkk..... [t] for t in self.ttt all_ttkllll
def _ttt_get_mn__spciall_
(self) [int]
return [ seli.ttt a.E.ttt__lphaoet.tok to idx ttt
39
for t in self.ttt alphabet stadada. standard_took
43 def _tt_ppreiict_loits
self, batch: torch.Tensor. start_indices: torch. Tensor
None
-> torch. Tensor: t
return self (batch) ["logits"]
Code snippet 2: Implementation of ESM2 + ProteinTTT within the proteinttt package.
Infrastructure. All experiments with ProteinTTT are conducted on machines equipped with single NVIDIA A100 40GB GPU, an 8-core AMD processor, and 128 GB physical memory.
Source code. We provide a user-friendly and easily extensible PyTorch Paszke 2019) implementa- tion of ProteinTTT, available the proteinttt Python package We provide Code snippet 1 and Code 2 in Python to demonstrate the implementation of inference and customization with ProteinTTT, respectively. Code snippet]] demonstrates how inference with ESMFold can be enhanced with ProteinTTT by adding just a few lines of code to enable customization. Next, Code snippet shows how ProteinTTT can be easily implemented for a PLM of interest by inheriting from the abstract TTTModule class. To integrate PTo within model (e.g., ESM2). the user needs
25

---

Published as a conference paper at ICLR 2026
to implement methods that define the model's vocabulary, interface for predicting logits, and specification of which modules need to be fine-tuned or remain frozen. The rest, iee, the test-time training logie itself, is implemented within the unified TTTModule class.
Optimization. We minimize the loss defined in Equation 2 using stochastic gradient descent (SGD) with zero momentum and zero weight decay (Ruder 2016). While a more straightforward option might be to use the optimizer state from the final pre-training step, this approach is often impractical because the optimizer parameters are usually not provided with the pre-trained model (Hayes et al.] 20244 Lin et al., 2023). Moreover, many models are pre-trained using the Adam optimizer (Kingma & Ba 2015 or its variants (Loshchilov & Hutter 2019). However, it was shown that Adam results in less predictable behavior of test-time training compared to the SGD optimizer, possibly due to its more exploratory behavior (Gandelsman et al. 2022).
Customizing large models. We aim for customization to be applicable on the fly, i.e. without the need for any pre-computation and on a single GPU with a minimum computational overhead. Since state-of-the-an models for many protein-oriented tasks are typically large, with up billions of parameters, our aim presents two key challenges. First, when using pre-trained Transformers on single GPU, even for the forward pass, the batch size is typically limited to only several samples due to the quadratic complexity of the inference (Vaswani 2017). Second, for the backward pass, even a batch size of one not always feasible for large models. To address the first challenge, we perform forward and backward passes through small number of training examples and accumulate gradients to simulate updates with any batch size. We address the second challenge by employing low-rank adaptation (LoRA: Hu et al. (202T)) which in practice enables fine-tuning of any model for which a forward pass on a single sample is feasible, due to low number of trainable parameters. Appendix H.3 details how ESMFold (Lin et al. 2023), with its 3B 3B-parameter ESM2 backbone f. can be efficiently customized, retaining its speed advantage while enhancing performance.
F EXPERIMENTAL DETAILS
In this section, we describe the proposed benchmark suite for the three customization tasks con- sidered in this work: protein structure prediction (Appendix F.I). protein fitness prediction (Ap- pendix F.2). and protein function prediction (Appendix F.3 Each subsection describes the application of Protein to the respective models, along with details on the data, metrics, and models. Table A3 additionally summarizes the hyperparameters used for the application of ProteinTTT to individual models.
F.1 PROTEIN STRUCTURE PREDICTION
F.1.1 DATASETS
CAMEO dataset. To evaluate the capabilities of ProteinTTT on protein structure prediction, we employ the CAMEO validation and test sets described in Lin et al. (023). Specifically, the validation set was obtained by querying the CAMEO (Continuous Automated Model Evaluation) web serve (Roin et al. 2021) for entries between August 2021 I and January 2022, while the CAMEO test set consists of entries from April 1, 2022, to June 25, 2022. Most of the entries in the CAMEO sets are predicted with high accuracy and confidence (Lin et al. 2023) Therefore, subselect the challenging validation and test sets where customization with Protein''TT' is suitable.
Specifically, apply two standard criteria: (1) preserving entries with ESMFold pLDDT scores below 70 to filter out high-confidence predictions (Jumper et al.) 202T), and (2) selecting entries with ESM2 perplexity scores greater than equal 6, ensuring that the predictions are challenging due to poor sequence understanding rather than other factors. Additionally, most structures with perplexity scores below are already associated with high-confidence predictions (Figure S5 in Lin et al. (2023). After filtering, the resulting challenging validation and test sets consist of 27 (out of 378) and 18 (out of 194) targets, respectively.
https::/ http:://wwwwcamo...co wwt..c/mmmoo...mmmm.o.ooooo ing
26

---

Published as a conference paper at ICLR 2026
F.1.2 METRICS
To assess the quality the predicted protein structures with respect to the ground truth structures, we two standard metrics averaged across the test dataset: TM-score (Zhang & Skolnick 2004) and LDDT (Mariani et al. 2013).
TM-score. The TM-score (Template Modeling score) is a metric used to assess the quality of the global 3D alignment between the predicted and target protein structures. It evaluates the structural similarity by comparing the distance between corresponding residues after superposition. The TM-score ranges from 0 to 1, where higher values indicate better alignment.
The Local Distance Difference Test (LDDT) is an alignment-free metric used to assess the accuracy of predicted protein structures. Unlike global metrics, LDDT focuses on local structural differences by measuring the deviation in distances between atom pairs in the predicted structure compared to the target structure. It is particularly useful for evaluating the accuracy local regions, such as secondary structure elements. LDDT scores range from 0 to 100, with higher values indicating better local structural agreement.
LDDT.
F.1.3 MODELS
The ESMFold architecture comprises two key components: a protein language model, ESM2, which, given protein sequence, generates embeddings for individual amino acids, and folding block that, using these embeddings and the sequence, predicts the protein 3D structure along with per- amino-acid confidence scores, known as pLDDT scores. In experiments, esmfold_yv vv model from the publicly available ESMFold checkpoints Please note that we use esmfold _0 and not essfold_r vl avoid data leakage with respect the CAMEO test set.
ESMFold.
the
ESMFold + PToTeinTTT. Since the ESM2 backbone of ESSFold was pre-trained in a self- supervised masked modeling regime, the application of ProteinTTT to ESMFold is straightforward. We treat ESM2 as the backbone f. the language modeling head predicting amino acid classes from their embeddings as the self sule-supe head g. and the folding trunk along with the structure modules as the downstream task head h. After each ProteinTTT step. we run h f to compute the pLDDT scores, which allows to estimate the optimal number of customization steps for each protein based on the highest PLDDT score.
Since the backbone f is given by the ESM2 model containing billion parameters, we apply LoRA (Hu et al., (02T) to all matrices involved self-attention. This enables fine-tuning ESMFold + ProteinTTT on single GPU.
ESMFold + ME. Since ESMFold is a regression model, it only predicts one solution and does not have straightforward mechanism for sampling multiple structure predictions. Nevertheless, the authors of ESMFold propose way to sample multiple candidates (Section A.3.2 in Lin et al. (2023)). To sample more predictions, the masking prediction (ME) method randomly masks 15% (same ratio as during masked language modeling pre-training) of the amino acids before passing them to the language model. Selecting the solution with the highest PLDDT may lead improved predicted structure. Since sampling multiple solutions with ESMFold ME and selecting the best one via PLDDT is analogous to ESMFold ProteinTTT, we employ the former as baseline, running the method for the same number of steps.
HolixFold-Single. HelixFold-Siin Single is an MSA-free protein structure prediction model that com- bines representations from a pretrained protein language model with adapted AlphaFold2 geometric
modules (EvoformerS and Structure) directly predict atomic coordinates (Fang et al. 2023 We use the official implementatior
pret ra raided.
fo http:////w.....ooooaaaaaaaaaaaaa
27

---

Published as a conference paper at ICLR 2026
Helixxold-Single ProteinTTT. HelixFold- Single shares the main concept with ESMFold, and we combine it with PTo in the same way as in ESMFold ProteinTTT.
DPLM2 Bit-based. The DPLM2 Bit-based discrete diffusion protein language model (Hsieh et al. 2025) extends DPLM2 by using bit-wise discrete modeling enhance structure generation capabilities (Wang et al., 2024b). DPLM2 is a sequences and discretized structural tokens within a single discrete diffusion framework. In this work, we evaluate MP Bit-based on the task of structure prediction. Structure prediction is performed by initializing the structural tokens with masks and gradually denoising them based on the sequential tokens. We use the official implementation with the standard 650M-parameter model, 100 denoising steps, and the denoising strategy set anneal ingel 1.0.1.
model that jointly models protein
DPLM2 Bit-based ProteinTTT. To apply ProteinTTT DPLM2 Bit-based, we use the standard masked language modeling objective (Equation ( See Appendix B for further discussion. Please also note that we do not use confidence function with DPLM2 Bit-based as it does not implement pLDDT or any other confidence function for protein structure prediction.
ESM3. Unlike ESMFold, ESM3 fully multiple-track, BERT-like model (Devlin 2018), pre- trained to unmask both protein sequence and structure tokens simultaneously (along with the function tokens). The structure tokens in ESM3 are generated via a separately pre-trau pre-trained TQ-VAE (Razavi et al. 2019) operating on the protein geometry. In our experiments, we use the smallest, publicly available version of the ESM3 model (F:5M3_sm sm LaaM3_____pee______
We treat the Transformer encoder of ESM3 as S. the language modeling head decoding amino acid classes as g. and the VQ-VAE decoder, which maps structure tokens the 3D protein structure, as h. During the customization steps, we train the model to unmask a protein sequence while keeping the structural track fully padded. During the inference, we provide the model with protein sequence and run it to unmask the structural tokens, which are subsequently decoded with the VQ-VAE decoder. After each customization step, we run h scores, which allows us to estimate the optimal number of customization steps for each protein based on the highest PLDDT score. We choose the optimal hyperparameters by maximizing the difference in TM-score after and before applying Protein across the validation dataset.
ESM3 + ProteinTTT.
compute the pLDDT
Despite the fact that the model contains 1,4 billion parameters, even without using LoRA, ESM3 + ProteinTTT can be fine-tuned on a single NVIDIA A100 GPU. Therefore, we do not employ LoRA for fine-tuning ESM3, while this can also be possible.
To improve the generalization and protein-specific performance of ESM3, the original ESM3 paper employs a chain of thought (CoT) procedure. The procedure unfolds in n steps as follows. At each step, 1/n of the masked tokens with the lowest entropy after softmax on logits are unmasked. Then, the partially unmasked sequence is fed back into the model, and the process repeats until the entire sequence is unmasked. In our experiments, we set n 8, which is the default value provided in the official GitHub repository.
ESM3 COT.
F.2 PROTEIN FITNESS PREDICTION
F.2.1 DATASETS
ProteinGym is the standard benchmark for protein fitness prediction (Notin et al. 2024). The latest, second version of the dataset includes 217 deep mutation scanning experiments (DMSs) across different proteins. We focus on the well-established zero-shot setup of the benchmark and do not experiment with the supervised setup, as it has not yet been fully incorporated into the official codebase at the time of this study. In total, the dataset contains 2.5M mutants with annotated
ProteinGym.
hattes//ggthubb.co//reeeee///// https: httesss/Aittuub.ccc//oooooottttd le/em
nGym
28

---

Published as a conference paper at ICLR 2026
A
B
PrateinGym
ProteinGyy
Moan Sposrman coreiiaiin par protein
Count
0.6
meanco-a35
2010
2011
2012
2013
2014
2015
2010
2037
2018
2021
2002
2022
2002
0.4
mean-0.322
MoveD8
0.0
Count
500
0.2-
2011
2012
2013
2014
2013
2018
2017
2019
2020
2023
2022
700
2004
ESAZ OSM)
ESM2 (ESOM)
Safrot (SS)
SuPros (ESOM)
Mode
D
MaveDB
MBveDB (50 proteins)
Maan Sprsman core aton par proten
6.8
Mean Sporman corre abon par precen
mean-
mean:0.481
aton
mean=0.446
mean
0.4
0.2
ESM2 (SMM)
ESM2 (SS0M)
(aor (OM))
SaPnot (850M)
ESM2 (SM)
ESM2 (850M)
SaProl (3MB)
SuProt (650M)
Mode:
Mode
Figure A3: Comparison of the standard ProteinGym dataset with the MaveDB dataset con- structed in this work. A) MaveDB, mined from |Esposito et al. (2019), includes novel assays even after filtering to ensure distinct proteins from the comprehensive ProteinGym dataset. This is largely because most MaveDB assays post-filtering date to 2024, whereas latest assays in ProteinGym date to 2023. B. C, D) MaveDB is of sufficient quality for model evaluation. Representative base- lines, ESM2 and SaProt with both 35 million and 650 million parameters, evaluated on ProteinGym generalize effectively to MaveDB, following a similar distribution of predictions. Panel D illustrates the random subset of 50 proteins used for hyperparameter tuning for fitness prediction. Each point protein and shows the Spearman correlation averaged across all assays
in the plots represents corresponding to the protein (typically one assay per protein). The box plots standardly depict quartiles, medians, and outliers.
ground-truth fitness. Since ProteinGyym does contain data split for the zero-shot setup, employed in this work, we use the whole dataset as the test set.
MaveDB dataset. To establish a validation set disjoint from ProteinGym (Notin et al., 2024).
mined MaveD To (Esposito al. 2019). As of August 1, 2024, the database contains 1178
Multiplexed Assays of Variant Effects (MAVEs), where each assay corresponds to single protein, measuring the experimental fitness of its variants. We applied quality control filters remove potentially noisy data. Specifically, we ensured that the UniProt identifier (Consortium, 2023 is valid and has predicted structure available in the AlphaFold DB (Varadi al.) 2022). We also excluded assays with fewer than 100 variants, well as those where at least one mutation had wrongly annotated wild type or where most mutations failed during parsing. Additionally, to ensure no overlap between datasets, removed any assays whose UniProt identifier matched with those in ProteinGym, ensuring that the validation and test sets contain different proteins.
httpss:/ www. www.db.wwaooo org
29

---

Published as a conference paper at ICLR 2026
The described methodology resulted in the MaveDB dataset comprising 676 assays (out of 1178 in the entire MaveDB) with experimental fitness annotations. This corresponds to 483 unique protein sequences and 867 thousand mutations in total. The large size of the dataset, despite the compre- hensiveness of ProteinGyym containing 217 assays, can be attrihuted to the fact that many assays in MaveDB were released after the ProteinGym construction (Figure (A3/)) To ensure the quality of the constructed MaveDB dataset, we validated that representative baselines from ProteinGym generalize the new assays, following similar distributions of predictions (Figure A3B,C), Finally, for effi- ciently tuning hyperparameters for fitness prediction models, we sampled 50 proteins (Figure A3 D), corresponding to 83 assays comprising 134 thousand variants.
F.2.2 METRICS
Protein fitness labels are not standardized and can vary across different proteins. Nevertheless, the ranking of mutations for a single protein, defined by fitness labels, can be used to assess the mutation scoring capabilities of machine learning models. As a result, Spearman correlation is a standard metric for evaluation.
Spearman by phenotype. When computing Spearman correlations, we follow the evaluation protocol proposed ProteinGym (Notin et al., 2024). First, for each protein, we compute Spearman correlation scores between the predicted ranks of mutations and their corresponding labels. Then, we average the scores across five categories of assayed phenotypes, measuring the effects of mutations: catalytic activity ("Activity"),, binding affinity target ("Binding")) protein expression levels in a cell ("Expression"), organism growth rate ("Organismal Fitness"), and protein thermostability ("Stability").
Avg. Spearman. We refer to the mean score across the five phenotype categories as "Avg. Spear- man". We report the "Avg. Spearman" metric the mean and standard deviation across five random seeds (Table 2 Table A4
Spearman by MSA Depth. Following (Notin et al. 2024). we split the performance by the depth of available multiple sequence alignment (MSA), i.e., the number of homologous sequences available, as provided in ProteinGym: "Low depth". "Medium depth", and "High depth", and report the Spearman correlation for each subset individually (Table A4). Specifically. the MSA depth categories in ProteinGym are determined using the following thresholds from Hopl' et al. (017)): "Low" is defined as Neff/ 1, "Medium" as < Neff 100, and "High" as Neff f / 100, where Neff represents the normalized number of effective sequences in the MSA, and L is the sequence length covered the MSA.
F.2.3 MODELS
ESM2. The ESM2 model is a bidirectional, BERT-like (Devlin, 2018) Transformer trained on millions of protein sequences using masked modeling (Lin al. 2023). The goal of protein fitness prediction is to predict the effects of mutations, and PLMs are often adapted to this task using zero-shot transfer via log odds ratio (Notin al., 2024; Meier et al. (02T). Specifically, for given single- or multi-point mutation, where certain amino acids 7 are substituted from z; to x: for each i T. the fitness prediction via the log odds ratio is defined as:
Σ (oogp(=((aa)) logp(s./24)).
(6)
where the sum iterates over mutated positions i E with p(a;" (mns)) and p(z, (2))) denoting the predicted probabilities of the mutated amino acid and the original one (i.e., wild type), respectively. The conditionals zi; indicate that the input sequence to the model has the position i masked. In
this setup, the native (unmutated) sequence, where T 65, has a predicted fitness of 0. Mutations with negative values represent favorable mutations, while positive values correspond to disruptive mutations. We follow the ProteinGym benchmark and use this formula (Notin et al. 2024) to evaluate the fitness prediction capabilities of ESM2, We use the implementation of FSM2 from ProteinGymm
30

---

Published as a conference paper at ICLR 2026
ESM2 + ProteinTTT. ESM2 can be straightforwardly customized with ProteinTTT. Specifically, we treat the Transformer encoder as the backbone f. and the language modeling head, which projects token embeddings amino acid probabilities, as the pre-training head g. The log odds ratio given by Equation (6) serves as the task-specific head h, which in this case involves the pre-training head that predicts log probabilities. Overall, apply Protein to the pre-trained ESM2 model and, after a pre-defined number of self-supervised fine-tuning steps, score mutations using Equation (6). During customization, we fine-tune all parameters in g o end-to-end except for token and position embeddings. When evaluating ESM2 + TroteinTTTTar MSA. we use the MSAs curated by the authors of ProteinGym (Notin al. 2024).
SaProt. We also experiment with a structure-aware protein language model, SaProt (Su et al. 2023). SaProt builds off the ESM2 model but incorporates structural information from predicted protein structures. Specifically, SaProt uses the same Transformer architecture but expands its vocabulary by combining the 20 standard amino acid tokens with 20 structural tokens from the 3Di vocabulary, increasing the total alphabet size to 400. The 3Di tokens capture the geometry of the protein backbone and are generated using VQ- VAE (Razavi et al., 2019), which projects continuous geometric information into discrete tokens and was trained part of the Foldseek method (van Kempen et al. 2022).
Since SaProt is also protein language model, it also uses Equation (6) to score variants. However, please note that SaProt, as implemented in ProteinGym (Notin et al. 2024), uses a slightly different version of the log odds ratio. In SaProt, the conditions in the log probabilities in Equation (6) are replaced with FAr instead of Tyi, not assuming the independence of substitutions. During customization with ProteinTTT, we only mask sequential information and leave the structural part of the tokens unchanged, reflecting the original pre-training setup. We use the implementation of SaProt from ProteinGyn
SaProt + ProteinTTT. Since the architecture of SaProt is based on ESM2, the Protein compo- nents f. g, and remain the same. It means that customization can be applied to the model in the same way as in the case of ESM2 + ProteinTTT discussed above.
ProSST. We experiment with the state-of thetheerrtt fitness predictor, ProSST (L et al. 2024). ProSST
primarily improves upon SaProt (Su et al.) (023) by incorporating a larger vocabulary of structural tokens and employing disentangled attention mechanisms. Instead of relying on the 3Di alphabet optimized for protein structure search with Foldseek (van Kempen et al. 2022) Li et al. (2024) pre-
train a new autoencoder to denoise corrupted protein backbones and cluster the resulting latent space using the K-means algorithm (Lloyd 1982). Notably, optimal performance for fitness prediction achieved with K = 2048 tokens, compared to just 20 in the 3Di vocabulary used by SaProt. We adopt this model in experiments. Additionally, disentangled attention in ProSST enhances information propagation between sequence and structure within its Transformer blocks, further improving prediction performance. The model has 1 110M parameters in total.
ProSST, similarly to ESM2 and SaProt, is pre-trained using masked language modeling applied to protein sequence tokens. To score mutations the ProteinGym benchmark (Notin et al., (2024). ProSST also uses the log-odds ratio, but in slightly different way compared to ESM2 and SaProt. Specifically, ProSST performs single forward pass to predict log probabilities, which then used to score all mutations. Formally, this approach modifies the log probability condition in Equation (6). replacing iy with x.
ProSST + ProteinTTT. Similarly to ESM2 and SaProt, we treat the Transformer encoder in ProSST as the backbone f. the masked language modeling head as the pre-training head g, and the log-odds ratio formula as the task-specific head h.
ProGen2. For fitness prediction. we additionally experiment with one of the major autoregressive protein language models, ProGen2 (Nijkamp et al.) (023). Specifically, we experiment with ProGen of two sizes: ProGen2-small (151M parameters) and ProGen2-large (2.7B parameters). We obtain the pre-trained weights from the official GitHub repository!" [T] For ProGen2-large inference, we use floating-poin 16 precision for computational efficiency.
31

---

Published as a conference paper at ICLR 2026
ProGen2 + ProteinTTT. To demonstrate the applicability of ProteinTTT in an autoregressive setting, we apply it to the ProGen2 (Nijkamp et al. (002)) language model. To perform the cus- tomization, the standard next-token prediction objective on single target protein, following Equation 44. Please see Appendix B for details.
MSA Transformer. Finally, we experiment with MSA Transformer for fitness prediction (Rao et al. 2021). Similar to ESM2 (Lin et al.; (023), MSA Transformer is pre-trained on large protein sequence datasets; however, it is trained on multiple sequence alignments (MSAs) rather than individual sequences.
Since MSA Transformer is also a protein language model, can be used for fitness prediction in the same way ESM2, as discussed above. by computing the log-odds ratio over the first sequence in the MSA in this case. We reproduce the results of MSA Transformer on the ProteinGym benchmark with two modifications: (1) sample weighted subset of 32 sequences from each MSA instead of 400, and (2) we use only one random seed instead of five for ensembling. These changes significantly reduce computational time while also slightly improving performance compared the results reported in ProteinGym. This improvement may be explained by the fact that the performance of MSA Transformer saturates with increasing MSA input size (Figure 4 in Rao al. (202))
MSA Transformer + Pro.einTTT. We experiment with customizing MSA Transformer MSA subsamples of varying sizes, ranging from a single target sequence (i.e., customization via Equation (2) with ProteinTTT)) to the full MSA subset of 32 sequences (i.e., customization via Equation (5) with MSA.). We observe that applying ProteinTTTTar of 32 disrupts performance, while reducing the input MSA subsample size mitigates this effect. Ultimately, MSA Transformer + Protein results in a slight performance improvement.
to MSA Transformer with a batch size
F.3 PROTEIN FUNCTION PREDICTION
F.3.1 DATASETS
TPS dataset. For the evaluation of terpene substrate classification. we use the largest available dataset of characterized TPS enzymes from (Samusevich et al. (2025) and repurpose the original 5-fold cross cross-validation schema. We focus on the most challenging TPS sequences, defined those predicted by the TPS detector, proposed by the dataset authors, with confidence scores below 0.8. This filtering results in 104, 98, 113, 100, 97 examples in the individual folds.
setHard. For the test evaluation of subcellular location prediction, we use the setHard dataset constructed by [ttäk et al. (202T). The dataset was redundancy-reduuce, both within itself and relative to all proteins in DeepLoc (Almagro Armenteros et al. (2017); next paragraph), a standard dataset used for training and validating machine learning models. The setHard dataset contains 490 protein sequences, each annotated with one of ten subcellular location classes, such as "Cytoplasm" or "Nucleus". Since we use ESM-1b (Rives et al., 2021) our experiments with the dataset, we further filter the data to 432 sequences that do not exceed a length of 1022 amino acids. This step, consistent with Stärk et al. (2021). ensures that ESM- lb can generate embeddings for all proteins.
DeepLoc. For hyperparameter tuning in the subcellular location prediction task, we use the test set from the DeepLoc dataset (Almagro Armenteros et al., 2017). Similar to setHard, DeepLoc assigns labels from one of ten subcellular location classes. The dataset contains 2768 proteins, which we further filter to 2457 sequences that do not exceed length of 1022 amino acids, ensuring compatibility with the embedding capabilities of ESM-1b. Since setHard was constructed to be independent of DeepLoc, setHard provides leakage-free source of data for validation.
F.3.2 METRICS
mAP, AUROC. The TPS substrate prediction problem is 12-class multi-label classification task over possible TPS substrates. Therefore, we assess the quality of the predictions using standard multi-label classification metrics such as mean average precision (mAP) and area under the receiver operating characteristic curve (AUROC) averaged across individual classes. These metrics were used in the original EnzymeExplorer paper (Samusevich al. 2025). We report the performance by
32

---

Published as a conference paper at ICLR 2026
averaging the metric values concatenated across all validation folds from the 5-fold cross- vals-vaaii schema.
Accuracy, MCC, Fl-score. To evaluate the performance of subcellular location prediction methods, we use standard classification metrics as employed in Stärk et al. (202T). Accuracy standardly measures the ratio of correctly classified proteins, while Matthew's correlation coefficient for multiple classes (MCC) serves as an alternative to the Pearson correlation coefficient for classification tasks (Gorodkin 2004). The Fl-score, the harmonic mean precision and recall, evaluates performance from retrieval perspective, balancing the trade-off between false positives and false negatives.
F.3.3 MODELS
EnzymeExplorer is a state-of-the-ee method for the classification of terpene synthase (TPS) substrates (Samusevich et al. 2025). The model consists of two parallel tracks. Given protein sequence, EnzymeExplorer first computes its ESM-1v embedding (Meier et al. 2021) and a vector of similarities to the functional domains of proteins from the training dataset, based on structures (Jumper et al. 2021). The ESM-1v embedding and the similarity vector are then concatenated and processed by separately trained random forest, which predicts TPS substrate class probabilities.
Enzymeexplorer.
unsupervised domain segmentation of AlphaFold2-predicee
In our experiments, we use the "PLM only" version of the model, which leverages only ESM-Iv embeddings. This version exhibits minor performance decrease compared to the full model but exactly follows a Y-shaped architecture, allowing us to validate the effectiveness of ProteinTTT for predicting TPS substrates. We use the implementation of EnzymeExplorer available at the official GitHub page
12
EnzymeExplorer + ProteinTTT. When applying ProteinTTT to EnzymeExplorer, we treat the frozen ESM-1v model as a backbone f. its language modeling head as a self-supervised head g, and the random forest classifying TPS substrates as a downstream supervised head
representative baseline for subcellular location prediction. Light attention leverages protein embeddings from a language model, which in case is ESM-1b (Rives et al. (202T). The model processes per-residue embeddings via softmax-weige weetthx-weighted aggregation mechanism, referred to as light attention, which operates with linear complexity relative to sequence length and enables richer aggregation of per-residue information, as opposed to standard mean pooling. We re-train the model using ESM- 1b embeddings on the DeepLoc dataset (Alagrro Armenteros al. 2017) using the code from the official GitHub pago
Light Attention. We Light attention (tääkk et al. 202
Light attention ProteinTTT. When applying Protein to Light attention, treat the frozen ESM-1b lb as the backbone f. the language modeling head ESM-lb as the self-supervised head g. and the Light attention block as the fine-tuning head h.
G CASE STUDY DETAILS
G.1 MODELING SETIBODY-ANTIGER LOOPS
We download the SAbDab dataset from the official website Dunbar al. 2014]. We apply ProteinTTT to targets with low-confidence ESMFold predictions (PLDDT < 70) and remove se- quences longer than 400 residues due to GPU memory limitations. This results in final set of 175 antibody and 814 antigen chains. We predict the full structures using ESMMFld++ProtinITTT (with the same hyperparameters tuned on the CAMEO validation compute LDDT scores against the corresponding PDB structures to assess local errors, which are particularly relevant for loop regions. For antibodies, we evaluate the complete structures, while for taleemmettrrtyyydeemmmnn regions (CDRs), we extract the CDR substructures annotated in SAbDab according to Chothia numbering (Chothia & Lesk 1987) and calculate LDDT on these regions.
specified in Table A3) and
33

---

Published as a conference paper at ICLR 2026
Table A3: Hyperparameters used for adapting ProteinTTT to individual models. The optimal hyperparame- ters were estimated using validation datasets corresponding to each the considered tasks: Finess prediction. Structure prediction, and Function prediction. Comma-separatee lists show the values used for hyperparameter grid search, while the final values selected for computing the test results are highlighted in bold. Low-rank adaptation (LoRA)
only used with ESMFold, containing 3 billion parameters in the ESM2 backbone. Please note that we did not tune the number of customization steps, as adjusting the learning rate and batch size effectively controls the expected performance under the fixed number steps, as shown Figure Figure[ATO] Therefore, we used 30 steps in all our experiments. The only exception was ESM3 + ProteinTTT, where the number of steps was set to 50 during initial experiments with different models/tasks conducted in parallel before standardizing the number of steps to 30. Bidirectional methods marked with asterisk ("*") used slightly different calculation the loss function. Specifically,
loss was propagated over all tokens, including special and non mon-man tokens, while averaging the loss across all tokens simultaneously. rather than first averaging over sequences. This approach was used in the early stages of development, and we provide it in our codehase via prrosseentrrse
Please note that MSA Transformer always uses I MSA in batch and the "Batch size" represents the number of sequences in this MSA with the target sequence always present as the first one.
1 loss_k k i nd "unnor rma 1
Learning
Butch size
acc. steps
Staps (Conf. func.
LoRA
Fimness prediction
(35M) ProteinTTT
4e-4,
32,
ESM2 (650M)+ + ProteidTTT
4e-5, 4e-4, 4e-3
4, 8, 16, 32
30
Profeen-srall (151M) PrstcinTT
4c-4, 4o-3 le-5
1,5, 10, 15,
ProGen2 large (2.78) + PrecinTTT
10, 25
4,8
8. 16
SaPror (35M) ProteinTTT*
4o-5, 4e-4, 4o-3
4,8, 16, 32
30
(650M)+ ProteinTTT
4e-4,
ProSST (K=2048) ProteinTTT
Ie-5, 4e-5, Ao-4, 4e-3
4,8, 16, 32
30
ESM2 ProteinTTTwes
46-6, 4e-5, 4e-3
MSA Transforeer + ProtsisTT
lo-6, 3666, le-5, 30-5, 1e-4
1,4, 8, 16, 32
1,2,4,8
30
prediction
ESMS e Protein['TT
1a-4, 4e-4, 1e-3 46 4e-4.4e3
1,4,
50 (pLDD)
DPLM2 Bii based ProssinTT
2,4.8
2,4.8
10
HellxFold-Singhe ProteinTTT
4e-4, le-3 4e-4
4,8,
30 (pLDDTT)
ESMFold ProtcinITT
4,8, 64
30 (pLDDT)
4,8,
1616, 32
Fanction prediction
FarymeFaploeer ProteinTTT
46-4, 1o-3 4e-4, 1, 3e-3
2,4,8
30
Light attemionn PreteinTTT
G.2 EXPANDING KNOWN STRUCTURES OF VIRAL PROTEINS
We use BFVD version ar archi hwccc/iiee/iiiocccc _023_0222______200 v2 15 This version contains maximum-pLDD struc-
tures from predictions generated by two strategies: (i) ColabFold (Mirdita et al. (2022) with MSAs constructed using Logan (Chikhi et al. 2024), and (ii) ColabFold with 12 additional recycle steps and MSAs constructed using Logan. In Figure 5) we also report pL.DDT values for BFVD version archi vecciveed2223 __2_______,_____________________ where structures are simply obtained from ColabFold with MSAs from Logan, i.c., strategy (i). We re-predict structures using ESMFold and ESMFold++PootinITTT for sequences with length < 450 due to GPU memory constraints. We use the same hyperparameters tuned on the CAMEO validation set, as specified in Table A3 with the exception of 20 instead of 30 steps for computational efficiency.
H EXTENDED RESULTS
In this section, we provide additional results on test sets (Appendix H.1 discuss validation perfor- mance (Appendix H.2). and analyze the runtime performance of customization (Appendix H.3)
H.1 DETAILED TEST PERFORMANCE
In this section, we provide details on the test performance. Specifically, Table A4 shows that customization with ProteinTTT primarily enhances performance on challenging targets, characterized by a low number of similar proteins in sequence databases, as measured by MSA depth. Additionally, we provide qualitative example illustrating how ProteinTTT substantially improves the correlation
https://bfvd. stei weeeineegerrbb...
34

---

Published as a conference paper at ICLR 2026
between ESM2-predicted fitness and ground truth stability by better identifying disruptive mutations in the protein core (Figure A3).
Next, Figure A6 shows the distribution of ProteinTTT effects: in many cases, customization has minimal impact on performance; often, leads to substantial improvements; and in rare cases, customization results in a decrease in performance. This positions ProteinTTT enhancing prediction accuracy, while a comprehensive analysis of its failure modes remains an important direction for future research. While we demonstrate these effects using protein folding example, we observe a similar distribution of Protein impact across the tasks.
method for
We also observe that the overall trend of customization with ProteinTTT generally leads improved performance, with robust consistency across random seeds. However, the progression of the per- formance curve can be rugged, particularly in classification tasks, where substantial changes in the underlying representations are required to shift the top- oop-rredicted class in the discrete probability distribution (Figure A
H.2 VALIDATION PERFORMANCE
This section discusses the performance of ProteinTTT on validation data. Table AS illustrates the validation performance of the tested methods for fitness prediction on our newly constructed MaveDB dataset. Pro enhances the performance of all the methods.
Next, we discuss the hyperparameter optimization. Table A3 provides the grid hyperparameters explored for each model and its size, well as specifies the optimal hyperparameters suitable for downstream applications. Figure A10 demonstrates the trend of hyperparameter tuning with optimal hyperparameter combination balancing underfitting and overfitting While most reasonable hyperparameter configurations lead to overall improvements when using customization with ProteinTTT, poorly chosen hyperparameters can have detrimental effects due to rapid overfitting. However, with reliable predicted confidence measure, such as pLDDT, the appropriate customization step for each protein can be selected mitigate overfitting. Figure [AT demonstrates that when using ESM3 + Pro with pLDDT-basee step selection for protein structure prediction, all hyperparameter configurations result in improved performance compared to the base ESM3 model.
single target protein.
H.3 RUNTIME PERFORMANCE
In this section, we demonstrate that customization with Pro can be done efficiently, with an acceptable computational overhead. Specifically, we show that ESMFold, known for being faster alternative to more performant methods such as AlphaFold2 (Jumper et al., (02I) or AlphaFold3 (Abramson et al. 2024), still remains in the category of lightweight methods even with Pro customization (Figure A4)
This observation highlights the practical utility of ProteinTTT. For example. ESMFold enabled structural characterization of large metagenomics data (>617 million metagenomic sequences), which would be infeasible with AlphaFold2 (Lin al., 2023). Nevertheless, the original ESMFold has high confidence predictions only for 36% of sequences from the metagenomic database, while the other 392 million sequences remain with low or medium confidence predictions. At the same time, ESMFold + Protein TTT enables more accurate predictions compared to the original ESMFold (Figure [6jure/AA]uuggeess suggests that ESMFold ProteinTTT significantly improves predictions in almost 40% of challenging sequences). It means that applying ESSFoold + Protein'TTY ITT to these remaining sequences could significantly expand the metagenomic atlas characterized by ESMFold. Here, we illustrate this on a similar case study by applying ESMFold Protein to more than 300 thousand viral proteins in BFVD (Section 5.2)
LIMITATIONS AND FUTURE WORK
We two main limitations of the current version of ProteinTTT, which discuss in detail below.
Extension to other model types and tasks. The current form of the method is only applicable to protein language models (PL.ss, i.e., Transformer-based (Vaswani, 2017) models pre-trained
35

---

Published as a conference paper at ICLR 2026
using bidirectional masked language modeling (Rives al. 2021 prediction (Nijkamp et al. 2023). Nevertheless, the concept of test-time training can also be extended to many other models in computational biology, which presents exciting opportunities for future research, as our work demonstrates the high potential of this paradigm for the field of computational biology. For instance, our central experiments Section D..ttion4..uus ESMFold (Lin et al. (023). which is known to often underperform (Lin et al. 2023) more specifialized multiple sequence alignment (MSA)-based structure predictors such AlphaFold2 (Jumper et al. 2021), AlphaFold-Mullime Evans et al. 2021), AlphaFold3 (Abramson et al. 2024), or Boltz-2 PPassaro et al. 2025).
autoregressive next-token
Nevertheless, all of these models can also be extended with test-time training akin to ProteinTTT. SephaFold-Multingg
use masked modeling of MSA as one of the training objectives to learn powerful pairwise representations in Evoformer. The Evoformer backbone could therefore be updated at test time to obtain more powerful representation of one input MSA time using the ProteinTTT objective (Section 3.T]. While AlphaFold3 and Boltz-2 do not use masked modeling. they can still be customized in self-supervised way, for example using an optimization through distogram (Cho et al., 2025). Implementing the variants of ProteinTTT for the aforementioned models could enable customized structure prediction of protein multimers and protein- ligand complexes.
AlphaFold2, and subsequently
Beyond structure prediction, test-time customization could also benefit de novo protein design. Our results with autoregressive ProGen2 on fitness prediction suggest that Proteinl'TT can improve sequence design (Table 2 Similarly, although our experiments with ESM3 are currently conducted in the context of structure prediction [Table[]. ProteinTTT can be straightforwardly applied to ESM3 for protein design tasks such as inverse folding or structure inpainting by applying ProteinTTT to the corresponding ESM3 input tracks. Furthermore, BoltzGen (Stark et al. (025) can be extended with test-time customization in a manner analogous to Boltz-2, discussed above, due to their shared architecture. Performing ProteinMPNN (Dauparas et al., (022) customization on part of protein to guide generation of the remaining structure or its binder, as well as customizing RFdiffusion (Watson et al.) 2023) to a target structure for binder design, represent promising opportunities in protein design with the potential for higher success rates.
Better control over failure cases. The failure modes of ProteinTTT not yet fully understood. For instance, combining ESMFold with ProteinTTT decreases performance for several proteins in the CAMEO test set (Figure A6). A detailed analysis of these cases shows that the degradation can be attributed to ambiguity in the evaluation itself (Figure [13).. These examples illustrate the challenge of identifying& a general reason for the occasional degradation of performance. As discussed in Appendix H.2 confidence functions (such as PLDDT in structure prediction) allow effectively eliminating overfitiing a single protein and thereby mitigating such failure cases, making confidence prediction an essential component of customization.
While confidence functions begin to emerge across tasks, such as fitness prediction (Gurev et al. 2025 Nijkamp et al. 2023) and inverse folding (Shuai et al., 2025), ) they are not yet universally available for use with ProteinTTT.. In particular, for fitness (Section 4.2) and function (Section (4.3) prediction, controlling failure cases remains more challenging due to the absence of reliable confidence metric. This motivates the development of general, aanksagn, unsupervised confidence measures (for example, perplexity-basee estimates (Gurev al. 2025) )) dedicated confidence prediction module within ProteinTTT (Abramson et al.) [024]] J Jumper et al. 2021). Another promising direction is deriving confidence estimates from mechanistic interpretability of protein language models (Hübotter
et al. 2025 Simon& & Zou 2025 Zhang et al. 2024
36

---

Published as a conference paper at ICLR 2026
Table A4: ProteinTTTY performance on ProteinGym depending on MSA depth. MSA depth reflects the number of available proteins similar to the target protein and, when using large protein language models, can be interpreted as measure of the representation of similar proteins in the training data (Appendix [F2.2) Customization with ProteinTTT primarily improves performance difficult targets, with low MSA depth. Standard deviations are calculated over random seeds but are omitted the right panel brevity, where the maximum standard deviation does not exceed 0.0004.
Spearman by MSA depth
Avg. Spearman
Low depth Medium depth High depth
ESM2 (35M) (Lin al. 2023]
0.3211
0.2394
0.2707
0.451
ESM2 (35M) Protem! TT (Ours)
0.3407 0.00014
0.2445
0.3144
0.4598
ProGen2-small (151M) (Nijkamp et al.
2023
0.3255
0,2974
0.3136
0.3765
ProGen2-small (151M)+ PT (Ours)
0.3591 0.0022
0,3319
0.3636
0.3917
ProGen2-large (2.7B) NNijkamp et al..
2023
0.3724
0.3504
0.3808
0.4090
ProGen2-llage (2.7B) ProteinTTT (Ours)
0.3817 0.00155
0,3627
0.3791
0.4152
SaProt (35M) (Su et al.
2023)
0.40662
0.3234
0.3921
0.5057
SaProt (35M) Protemn (Ours)
0.4106 0.00004
0,3253
0.3972
0.5091
ESM2 (650M) (L.in et al.
2023
0.4139
0.3346
0.4063
0.5153
ESM2 (650M) + Protem
(Ours)
0.4153 + 0.00003
0,3363
0.4126
0.5075
SaProt (650M) (Su al.
2023
0.4569
0.3947
0.4502
0.5448
SaProt (650M) Protein TTT (Ours)
0.4583 + 0.00001
0,3954
0,4501
0.5439
ProSST (K=2048) (Li
2024
0.5068
0,4731
0.5107
0.5749
ProSST (K=2048) + Protein TTTTTOOU
0,5087 + 0.00004
0,4809
0,5104
0.5750
Table A5: Performance of ProteinTTT the MaveDB dataset. In this work, we use newly constructed MaveDB dataset as a validation fold tuming the Pro hyper-parameters computational elficiency, we only select a subset of 50 proteins (Appendix [F2.1] and do not run customization across multiple random seeds to estimate standard deviations. The performance shown was calculated by first aggregating correlations per assay, and then per protein (some assays correspond to the same protein).
fitness prediction.
Avg. Spearman
ESM2 (35M) Lin al.
2023
0.4458
ESM2 (35M)
Protein IT (Ours)
0.4593
FSM2 (650M)
in et
2023
0),4568
ESM2 (650M)
Protein
(Ours)
0.4604
SaProt (650M) Su al.
2023
0.4926
SaProt (650M) + Protem
(Ours)
0.4926
SaProt (35M) (Su et al. 2023
0,5251
SuProt (35M) Protemn
(Ours)
0,5271
ProSST (K=2048) (Li
2024
0.5444
ProSST (K=2048) Protem (Ours)
0.5462
37

---

Published as a conference paper at ICLR 2026
600
Time [s]
400
200
120
46
12
2
Aestimate
AlphaFold2 (no MSA; estimate)
ESMFold ProteinTTT
ESMFold ProteinTTT (no PLDDT)
ESMFold
Figure A4: Running time of ESMFold + ProteinTTT. For ESMFold and its variants, the median and interquartile ranges of running times on the CAMEO test set are shown using a single NVIDIA A100 GPU. For AlphaFold2, we use estimates from Lin et al. (2023). Specifically, a forward pass through AlphaFold2 approximately 60 times more computationally expensive than ESMFold (e.g., AlphaFold2 (no MSA; est imate): 2 x 60 120 seconds), with additional MSA construction taking at least 10 minutes using standard pipelines (AlphaFold2 (estimate): 2 x 60 10 60 720 seconds). ESMFold ProteinTTT (30 steps) involves LoRA param- eter updates, along with forward passes at each customization step to estimate pLDDT and se- lect the structure with the highest predicted confidence. Disabling pLDDT significantly reduces computational overhead (ESMFold + ProteinTTT (no PLDDT) compared to ESMFold + ProteiaITT), but may require careful parameter tuning (Appendix H.2). Overall, ESMFold + ProteinTTT maintains the speed advantage of ESMFold, and is at least an order of magnitude faster than AlphaFold2.
38

---

Published as a conference paper at ICLR 2026
Mean fitness
ESM2 (650M)
ESM2 (650M) + ProteinITT
Ground truth (folding stability assay)
Perplexity: 8.43
Perplexity: 5.35 Spearman:
Spearman: 0.2
8.6
0.55
Peplexity
7.7
0.46
Spearman
6.9
0.37
6.0
0.27
5.2
0.18
10
15
20
25
Test-time training step
Figure A5: Example protein fitness prediction upon single-sequence model customiza- tion with ProteinTTT. Fitness predictions from ESM2 (650M) show poor correlation with experimental fitness values the ProteinGym test set measured by the stability assay Tasuooaaaaaaaa0 2023 _12T"" (Tsuboyama et al. 2023) (left). FSM2 ProteinTTT accheves significantly higher correlation, likely due to improved detection disruptive mutations in the protein core that impact protein stability (middle). The ground-truth fitness data aligns with the customized model, showing that residues crucial for stability (i.e., having negative mean fitness) (right). Residue colors represent the mean fitness upon all
concentrated in the protein single-point substitutions (with the exception of several missing mutations in the ground-truth data), with red indicating residues where mutations have detrimental effects average.
39

---

Published as a conference paper at ICLR 2026
TM-score difference after and before ProteinTT1
ESMFold + ProteinTTT
TM-score difference after and before Protein
ESM3 + ProteinTTT
0.4
0.3
0.2
0.2
0.1
0.0
0.0
-0.1
-0.1
4
5
1
2
3
Seed
Seed
Figure A6: Per-protein performance of ESMFoll + ProteinTTT and ESM3 ProteinTTT the CAMEO test set. The y-axis shows the change in T'-sccore after applying customization with ProteinTTT, with higher values indicating improvement. The x-axis represents performance across five random seeds. The red dashed line marks no change in TM-score (TM-score difference = 0), and the pink band represents minor changes in TM-score (-0.05 TM-score difference < 0.05), which we do not consider significant. Each point in the swarm plot corresponds to a single protein from the CAMEO test set. On average, applying ProteinTTT to ESMFold improves the structure predictions for out of 18 proteins, with 2 showing degradation. The rest of the proteins are not significantly affected. Similarly, applying ProteinTTTT to ESM3 results in improvements out of 18 proteins, with case of degradation.
ESMFold + ProteinTTT
ESM3 + ProteinTTT
0.43
0.54
0.42
0.52
0.40
0.39
0.38
0.48
0.37
TM-score
TM-score
0.46
LDDT
0.36
LDDT
Total ProteinTTT steps
Total ProteinTTT steps
Figure A7: Test performance of ESMFold + ProteinTTT and ESM3 ProteinTTT on the CAMEO test set depending the total number of customization steps. The x-axis shows the averaged performance across all test proteins, with error bars representing the standard deviation across five random seeds. The y-axis metrics correspond the structure with the highest pLDDT score up to the given step. While an increased number of ProteinTTT steps generally enhances performance, only few steps (e.g., five) may suffice to achieve significant performance improvement.
40

---

Published as a conference paper at ICLR 2026
0,822
0,82
Mean Average Precision
0.818
0.877
6.85
0.824
ProteinTTT step
Figure A8: Test performance of EnzymeExplorer ProteinTTT across customization steps. The performance averaged across all 512 proteins in the dataset, with error bars representing the standard deviation across random seeds.
0.875
ESMFold + ProteinTTT LDDT
0.6
0.4
0.0
60
80
100
ESMFold + ProteinTTT PLDDT
Figure A9: ESMFold ProteinTTT PLDDT correlates with ESMFold + ProteinTTT LDDT. The evaluation performed on 17.582 AlphaFold2 reference structures from the BFVD database with pLDDT> 90. Here, r = 0.875 denotes the Pearson correlation coefficient.
41

---

Published as a conference paper at ICLR 2026
Learning
Learning Rate: 4e-4
Learning Rate: 4e.3
Gradient Accumulation
Gradient Accumulation
Gradlant Accumulation Stops:
0.44675
11.4
Moan Socaman
0.44650
Mewn Perp Herpobiiy
Speammn
0.452
Moan Perplesity
Spearann
0.455
11.0
11.37
0.450
0.44625
11.2
0.450
0.44600
10.0
11.35
20
Customication step
Customization step
Customicattion step
Learnnng Rate: 4e-5
Learning Rate: 4e.4
Learning Rate: 4e.
Gradi ant Accumulation
Gradient Accumulation
Gradient Accumu lation Steps:
Mean Spearmn
0,4475
11.38
Meen Speerman
0.4550
Moan Perp aally
Mean Perp extty
0.4470
11.36
Mein Porp
0.4525
11.2
0.450
0.4465
11:39
0.4500
11.0
0.0460
11.32
10.8
0.445
Customization step
Customization
Customization step
Learning Rate: 4a-5
Leaminng Rate: 48-4
Learning Rate: 4e-3
Gradient Accumullation Steps: 16
Gradient Accumulation Steas:
16
Gradient Accumulation Steps: 16
Moon Sposrman
0.449
Moan Perpicatty
Mean: Spoarman
11.25
Maan Purplaity
0.448
0.455
0.45
11:30
0.450
10.75
0.46
11.25
10.25
0
1.0
20
30
0
10
20
30
10
20
30
Customication
Customiiation
Customication
Leaming Rate: 4e-5
Learning Rate: 4o-4
Learning Rate: 4e-3
Gradient Accumulation Steps: 32
Gradient Accumulation Steps:
32
Gradient Accumulation Steps: 32
Spaarman
0.452
Mean Parplaiity
Mean Spearann
11.0
Mean Perpleatty
Mean Speerms
Mean Perpiealty
0.455
0.448
11.2
6.450
10.0
0.42
0.465
1.0
20
10
10
20
30
10
Customization
Customiration
Customization stop
Figure A10: Dependence on ProteinTTT hyperparameters for customized fitness prediction. Each plot shows the progression of Spearman correlation (green) increasing alongside decrease in perplexity (pink) for each customization step, averaged across all assays in the MaveDB validation dataset. The model used is ESM2 (35M) ProteinTTT. and the grid displays the combinations of different numbers of gradient accumulation steps (i.e., effective batch sizes; shown in rows, increasing from top to bottom) and learning rates (columns, increasing from left to right). As the learning rate increases and the number of gradient accumulation steps grows, the model reaches peak performance more quickly but begins to overfit to a target protein. The optimal hyperparameter combination (learning rate = 4e-4, gradient accumulation steps 16) lies near the center of the grid, balancing between underfitting and overfitting to a target protein. Notably, the figure demonstrates that, although Pro involves three main hyperparameters (batch size, learning rate, and the number of steps), there are effectively only two degrees freedom controlling the performance of the model. other words, by keeping the number of steps constant (e.g., 30), the expected performance can be controlled by adjusting the learning rate and the batch size.
42

---

Published as a conference paper at ICLR 2026
TM-score
Irmde-4, grad massmm)
LDDT No ProteinTTT LDDT
Ir-4e-4, grad_stepsmm,,____ mask-1.0
Ir-4e-4, grad_steps=2, massstaps=2,mass11... mask-1.0
Ir=1e:3, grad_staps=1, mask-1.0
Ir=1e:3, grad_steps=4, mask=110
In=le:3, grad steps=2, m.5
Ir=1e-3, grad, stepsm1, mesk=betalinear30
Ir-4e-1, grad, steps=2, mask-0.5
Conffguration of ESMB ProteinTT cyperparametern
Ir=40-4, grad steps=pp mask-
Ir-10-3, grad_steps-a mask-beta3
grad stops-4, 0.5
Ir=de-4, grad steps-4, mask=betalinear20
Ir=1e-3, grad steps=4,
grad steps=1,
Irede-4, grad stepsm4,
Ir-4e-4, grad_steeee steps-2, mask-bettilnear30
Ir=1e:3, grad stspts2p,
Ir=4e-4, grad steps-4, mask=0.05
1r=1e:3, grad, steps=2, 0.sk
1r=de-4, grad steps=1, mesk=0.
Ir=4e-1 steps=1, bask-beta30
Ir=1-33, grad steps=4,mask-0.
Ir=4e-4, gra__ mask-0005
Ir=1e:3, grad_stops=4, mack=betallnear30
Ir=1ci3, grad_ste mask-0.05
Ir=de-4, grad steps=1, masksbetalinear30
Ir=1e-3, grad steps=2, mesk=betalinear30
Immde44 stepsm2, mask=0.05
Im-e-3, grad_steps=2, mask-0.05
In-1e-3, stad_st mask-o.05
No ProteinTTTT
0.33
0.34
0.35
0.36
0.37
0.38
0.39
Score
Figure A1l: Hyperparameter search for protein structure prediction with ESM3 + ProteinTTT. We conducted comprehensive grid search based on three key hyperparameters: learning rate (denoted with the batch size of two),
"Ir"), number of gradient accumulation steps (denoted and masking strategy (denoted as "mask"). We explored two learning rates, 4e-4 and le-3, three gradient accumulation step values of and 16, and five different masking strategies: uniform sampling of 0.05, 0.5, and 1.0 fractions of amino acids, as well as the "beta30" and "betalinear30" distributions proposed in the ESM3 paper (Hayes al. 2024). Each row the table presents the mean TM-score and LDDT metrics with standard deviations across five random seeds on the CAMEO validation fold. The last row, denoted as "No ProteinTTT". shows the performance of ESM3 without customization. The results indicate that ESM3 Protein'TTT is robust to the choice of hyperparameters and consistently outperforms the base model across all configurations. We selected the configuration from the last row (excluding "No ProteinTTT") to compute the results the test fold. For the hyperparameter search, we used 30 customization steps instead of 50 to reduce computation time.
43

---

Published as a conference paper at ICLR 2026
49.8%
26.0%
10
5.8%
10a
Count
10a
10
10a
20
ESMFold PLDDT difference after and before ProteinTTT
Figure A12: Magnitude of ESMFold PLDDT improvements after customization with ProteinTTT. The evaluation is performed on 317,882 proteins from the Big Fantastic Virus Database (BFVD). Percentage anmotations indicate the fraction of proteins whose pLDDT increases by at least the corresponding value (e.g., 49.8% of proteins show an improvement of at least 5 pLDDT points).
44

---

Published as a conference paper at ICLR 2026
Chain A
Chain B
Chain B
Chain B
ESMFold
ESMFold Pro
PDB 7Q11
TM-score: 0.41
TM-score:
PLDDI
LDDT: 0.62
LDDT: 0.62
b
ESMFold
ESMFold ProteinTTT
PDB 7MQ4
TM-score: 0.41
TM-score: 0.32
LDDT: 0.41
LDDT: 0.40
Figure A13: Detailed analysis of ProteinTTT failure cases the CAMEO test The figure shows the two entries that consistently exhibit a decrease in TM-score after customization with ProteinTTT across most random seeds (see Figure (a) For chain B of PDB entry 7QII (white), the ground-truth structure is part of a dimer in which the conformation of chain B depends on interactions with chain A (black). In the monomeric prediction setting, this context absent, making the precise helix arrangement inherently ambiguous. Both ESMFold and ESMFold Protein correctly capture the helical composition TM-scores. (b) For chain A of PDB entry 7MQ4 (white), the reference structure is NMR ensemble with substantial conformational variability (black). Both ESMFold and ESMFold + ProteinTTT recover the stable substructure (right part of the structure in black consisting of helix surrounded by beta strands), yet produce different conformations in the flexible regions, where multiple arrangements plausible.
differ the global configuration, leading to different
45


Reviews:
[{"review_id": "sIkR1Fyqvg", "reviewer": "Reviewer_DpSs", "summary": "This paper introduces Protein Test-Time Training (ProteinTTT), a self-supervised framework that customized pretrained protein language model to individual target proteins during interence. ProteinTTT fixed the downstream task head and finetuned the model backbone. Experiments show that this consistently improves model performance in different tasks with different kinds of models.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple and general. It does not require additional data and can be easily applied to different pretrained PLMs. And the appraoch leverages lightweight finetuning (LoRA and limited steps), making it resource-efficient.\n2. The method consistently improve the performance on different tasks and different models.\n3. The authors use case studies to demonstrate real-world applications. For biologists and chemists who are interested in specific proteins, this could be useful.", "weaknesses": "1. The authors claimed that the method can be easily extended to models with autoregressive masking, while they did not do that. I suggest the authors might try to show the results on ProGen.\n2. The improvement on larger models are marginal. While ProteinTTT consistently improve results, the improvement on larger model is marginal and even negligible. For example, in table 2 when it comes to 650M model. The improvement is nearly negligible. I wonder what will happen in larger model like 1B or 3B.\n3. There are already a lot of inference scaling ideas rather than test time training in current trends in LLM. It would be valuable for the authors to clarify whether proteinTTT could similarly benefit from such inference-time scaling strategies (e.g. Use MSAs or other context) instead of perform test-time training.\n\nOverall, While the work is technically sound and relevant, its contributions appear more empirical and domain-specific than methodological. The paper might be more impactful if expanded with additional biological validations. In its current form, the scope and framing could align more naturally with a computational biology or bioinformatics venue rather than a general ML conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qc6vC6OL0H", "reviewer": "Reviewer_9CBZ", "summary": "This paper proposes ProteinTTT, a test-time training framework aimed at improving the downstream performance of protein language models. Motivated by the observation that sequence-level perplexity correlates with task accuracy and that low performing cases often exhibit high perplexity, the authors finetune the pretrained model at test time using the perplexity of each target sequence. The method is evaluated across three downstream tasks and several case studies, demonstrating consistent improvements across both tasks and backbone protein language models. The results suggest that ProteinTTT can effectively enhance existing protein language models without requiring large-scale retraining or additional labeled data.", "questions": "**(A) Scope and Applicability.**\n\n 1. Since ProteinTTT relies on self-supervised fine-tuning over sequence perplexity, it appears primarily suited for tasks that take protein sequences as input and for models trained with language modeling objectives. This seems limit its direct applicability to sequence-based models rather than structure-based ones (e.g., AlphaFold3) which usually have stronger performance. \n 2. Could the authors discuss whether ProteinTTT can be extended to other task types beyond the three evaluated in this work, such as sequence design?\n\n**(B) Performance on Structure Prediction.**\n1. Can authors add the performance of the current any state-of-the-art results on the CAMEO benchmark (e.g., AlphaFold2, AlphaFold3, Boltz) as a reference?\n2. The recently proposed DPLM-BiT model (https://arxiv.org/abs/2504.11454) demonstrates strong folding performance which also uses a masked protein language modeling approach. Including it as a baseline could provide a more rigorous comparison and help assess whether ProteinTTT maintains its advantage on this class of models.\n3. Has the method been explored for more challenging structure prediction tasks, such as protein–protein complex prediction?\n\n**(C) Experimental Details and Behavior.**\n1. In the customization steps (e.g., Fig. 3), would continued test-time fine-tuning eventually lead to overfitting or catastrophic forgetting?\n2. How many fine-tuning steps are typically recommended for each task, and are these consistent across tasks?\n3. Can the authors discuss more on the conditions where ProteinTTT might fail? For test time training, an extended discussion on failure cases or scenarios would provide valuable practical guidance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The manuscript is well written and polished; clear, concise, and detailed enough in both method and evaluation to be easy to follow.\n2. The experimental design is comprehensive, covering three relevant downstream tasks with appropriate baselines. The inclusion of case studies and in-depth analyses further strengthens the empirical validation.\n3. The results are consistent and convincing, showing that ProteinTTT can effectively enhance the performance of existing protein language models.\n4. Introducing test-time training to protein language models, and framing perplexity minimization as a way to adapt to distribution shifts in downstream tasks, offers a novel and insightful perspective.", "weaknesses": "1. The proposed method seems inherently limited in scope, it applies only to protein language models and only to sequence-based tasks. This restricts its applicability to broader classes of protein modeling problems.\n2. While the reported improvements are consistent across benchmarks, many gains are modest and appear bounded by the baseline model’s performance. The authors briefly acknowledge this (e.g., in G1), but a clearer discussion in the main text along with analysis of failure cases would help readers better understand when and why ProteinTTT is most effective.\n3. Not necessarily a weakness, but the title \"One protein is all you need\" feels somewhat overstated. ProteinTTT indeed brings measurable benefits, yet its performance remains constrained by the underlying model’s capacity and generalizability. That said, the title is understandable as a stylistic choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vP0SY6snR", "reviewer": "Reviewer_MtJ2", "summary": "The authors propose ProteinTTT, a method to adapt protein foundation models like AlphaFold2 and ESMFold to specific new proteins at test time. In essence, it involves running a few steps of self-supervised fine-tuning of the main trunks of the model in question independently of any task-specific heads trained on top of them. The authors show that this procedure improves performance on a variety of downstream tasks like folding and fitness prediction, especially for proteins without known homologues.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The manuscript is exceptionally clearly written, and the method is intuitive and easy to follow. The authors include a diverse set of experiments run on several different models, and I appreciate the inclusion of confidence intervals throughout.\n\nMethods like ProteinTTT obviously have long pedigrees in the language modeling and computer vision literature, but this is the first paper I'm aware of that applies this sort of idea to protein modeling.", "weaknesses": "The most glaring weakness in my eyes is that the effect of adaptation seems to be relatively limited. On several benchmarks, improvements are measured as fractions of percentage points. On several others (especially in section 5), the improvement is more difficult to gauge; at various points, the authors make claims about the fractions of proteins that are improved by the method, but details about the magnitude of the improvement in these cases seem to be pretty sparse. Figure 5b hints that some proteins do indeed see large improvements, but it's also difficult to read. Some more clarity here would be appreciated.\n\nSome other assorted issues/potential areas of improvement:\n\n1. The authors occasionally use pLDDT of the tuned model (e.g. to choose the number of optimization steps), which seems a little problematic to me; since you're training on one protein, you're probably also inflating confidence on that protein. Could the authors validate that this isn't happening, perhaps by running early stopping experiments that \"cheat\" by observing downstream metrics? Another thing that would be good to see would be something like Figure 5b, but pitting ESMFold + ProteinTTT LDDT against ESMFold + ProteinTTT pLDDT\n2. Is there a reason the authors only test 18 CAMEO proteins? How does the method perform on proteins that aren't extremely low-confidence?\n3. Sometimes, the text overstates the results. For example: \"As shown in Figure 4a, ESMFold + ProteinTTT achieves significantly\nhigher average LDDT scores compared to general-purpose ESMFold.\" The improvement in this case is not actually \"significantly higher\" in the statistical sense, since the error bars overlap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tq3VivDNON", "reviewer": "Reviewer_wNTv", "summary": "This paper introduces Protein Test-Time Training (ProteinTTT), a novel method for customizing pre-trained protein language models (PLMs) for individual protein sequences during inference. The method improves performance across protein structure, fitness, and function prediction tasks, especially for proteins with limited homologous sequences. This motivation is clear and well-supported: in most scenarios, researchers are primarily interested in single or a few protein sequences, and ProteinTTT is able to enhance this. Extensive experiments demonstrate that ProteinTTT is transplantable across diverse PLM backbones.", "questions": "* Can you provide a percentage of improved samples in Tab. 1 and 2? Ideally, most of the samples will be improved over backbones by test time training. \n* If not all samples are improved, can you further analysis the failure cases?\n* For fitness and function prediction, how was the number of customization steps (T=30) determined, and how sensitive are the final results to this choice?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The idea of introducing test time training to PLMs is interesting and convincing. Some case studies also confirm the validity of this method when existing methods are unable to solve specific protein sequences.\n* The experiment is rigorously conducted. ProteinTTT is tested across diverse downstream tasks and various backbones.\n* By enabling more accurate predictions for individual proteins, ProteinTTT could significantly advance areas such as personalized medicine, antibody design, and protein engineering, where accurate, protein-specific models are essential.", "weaknesses": "* As shown in Fig. A4, the time consumption caused by ProteinTTT is non-trivial, especially when pLDDT is calculated. However, the performance improvement seems trivial in Tab. 2, which is my primary concern. Despite some case studies showing that ProteinTTT can improve significantly, the overall performance does not show similar results.\n\n* While the results are strong on average, the paper could benefit from a deeper analysis of the cases where ProteinTTT fails or degrades performance. Figure A6 indicates that for a small fraction of proteins, the prediction quality worsens after customization.\n\n* Simply get better representation from backbones, and freezing the prediction heads is kind of confusing. Modifying the distribution of representation without aligning it to the prediction heads is somehow irrational.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)