One protein is all you need

## Abstract

Abstract Generalization beyond training data remains a central challenge in machine learning for biology. A common way to enhance generalization is self-supervised pre-training on large datasets. However, aiming to perform well on all possible proteins can limit a model’s capacity to excel on any specific one, whereas experimentalists typically need accurate predictions for individual proteins they study, often not covered in training data. To address this limitation, we propose a method that enables self-supervised customization of protein language models to one target protein at a time, on the fly, and without assuming any additional data. We show that our Protein Test-Time Training (ProteinTTT) method consistently enhances generalization across different models, their sizes, and datasets. ProteinTTT improves structure prediction for challenging targets, achieves new state-of-the-art results on protein fitness prediction, and enhances function prediction on two tasks. Through two challenging case studies, we also show that customization via ProteinTTT achieves more accurate antibody–antigen loop modeling and enhances 19% of structures in the Big Fantastic Virus Database, delivering improved predictions where general-purpose AlphaFold2 and ESMFold struggle.

## 1 Introduction

Figure: Figure 1: Example of protein structure prediction after single-protein model customization via ProteinTTT. ESMFold poorly predicts the structure of the CASP14 target T1074 (white) because the underlying language model ESM2 poorly fits the sequence, as indicated by the high perplexity (left and Fig. 2E in Lin et al. (2023)). Self-supervised test-time customization of ESM2 to the single sequence of T1074 reduces the perplexity, resulting in improved structure prediction (right).
Refer to caption: x1.png

A comprehensive understanding of protein structure, function, and fitness is essential for advancing research in the life sciences (Subramaniam & Kleywegt, 2022; Tyers & Mann, 2003; Papkou et al., 2023). While machine learning models have shown remarkable potential in protein research, they are typically optimized for achieving the best average performance across large datasets (Jumper et al., 2021; Watson et al., 2023; Kouba et al., 2023). However, biologists often focus their research on individual proteins or protein complexes involved in, for example, metabolic disorders (Ashcroft et al., 2023; Gunn & Neher, 2023), oncogenic signaling (Hoxhaj & Manning, 2020; Keckesova et al., 2017), neurodegeneration (Gulen et al., 2023; oh Seo et al., 2023), and other biological phenomena (Gu et al., 2022). In these scenarios, detailed insights into a single protein can lead to significant scientific advances.

However, general machine learning models for proteins often struggle to generalize to practically interesting individual cases due to data scarcity (Bushuiev et al., 2023; Chen & Gong, 2022) and distribution shifts (Škrinjar et al., 2025; Tagasovska et al., 2024; Feng et al., 2024). Bridging the gap between broad, dataset-wide optimization and precision needed to study single proteins of practical interest remains a key challenge in integrating machine learning into biological research (Sapoval et al., 2022). This challenge is particularly acute in computational biology, where accurate predictions for individual proteins are essential to guide resource-intensive wet-lab experiments, in contrast to domains such as natural language processing or computer vision, where models are typically expected to flexibly handle diverse prompts from many users in real time (Brown, 2020; Ramesh et al., 2021).

To address this challenge, we propose a test-time approach for generalization to one protein at a time, effectively enabling more accurate predictions for individual targets, particularly those poorly represented in training data. Our Protein Test-Time Training (ProteinTTT) method customizes protein language models (PLMs) to individual proteins on the fly and without assuming additional data. Our approach is based on a simple yet powerful premise: if a language model is less perplexed (surprised) by a protein sequence–or if it “understands” its unique patterns better–it will generate a more accurate representation for predicting its structure and function. Given a model pre-trained via masked language modeling, our method effectively minimizes perplexity on a target protein or its multiple sequence alignment (MSA) through self-supervised customization, improving downstream performance without updating the downstream task head. The widespread use of masked modeling as a pre-training paradigm makes ProteinTTT broadly applicable in computational biology.

In summary, this work demonstrates the surprising effectiveness of protein model customization and lays the foundation for exploring other test-time strategies and broader biological applications. The key contributions are:
(1) We introduce ProteinTTT, to the best of our knowledge the first customization method in machine learning for biology. We provide a user-friendly and easily extensible implementation
(^1^11[https://github.com/anton-bushuiev/ProteinTTT](https://github.com/anton-bushuiev/ProteinTTT))
and provide insights into the effectiveness of protein model customization by linking it to perplexity minimization.
(2) We empirically validate ProteinTTT, showing improvements in protein structure prediction with well-established models, achieving state-of-the-art results in protein fitness prediction, and enhancing protein function prediction on terpene synthase substrate classification and protein localization prediction.
(3) We demonstrate the practical utility of focusing on one protein at a time through two challenging case studies. ProteinTTT enables more accurate prediction of antibody–antigen loops and improves 19% of structures in the Big Fantastic Virus Database, delivering accurate predictions where general-purpose AlphaFold2 and ESMFold struggle.

## 2 Background and related work

The broad adoption of Y-shaped architectures relying on masked modeling enables the development of a general method for customizing protein models at test time via masking-based self-supervision.

##### The Y-shaped paradigm of learning.

[Uncaptioned image]: x2.png

In machine learning applied to proteins, architectures often follow a Y-shaped paradigm (Gandelsman et al., 2022), consisting of a backbone feature extractor $f$ operating on protein tokens $x$, a self-supervised head $g$, and an alternative fine-tuning head $h$. During training, $g\circ f$ is first pre-trained, and the pre-trained backbone $f$ is then reused to fine-tune $h\circ f$ toward a downstream task. Here, $\circ$ denotes a composition of two machine learning modules (e.g., $g$ is applied on top of $f$ in $g\circ f$). At test time, the final model $h\circ f$ is fixed. Generalization is achieved by leveraging the rich knowledge encoded in the backbone $f$ and the task-specific priors embedded in the fine-tuning head $h$. This paradigm enables overcoming data scarcity during fine-tuning and underlies breakthrough approaches in protein structure prediction (Lin et al., 2023), protein design (Watson et al., 2023), protein function prediction (Yu et al., 2023), and other tasks (Hayes et al., 2024).

The backbone $f$ is typically a large neural network pre-trained in a self-supervised way on a large dataset using a smaller pre-training projection head $g$ (Hayes et al., 2024). The fine-tuning head $h$, however, depends on the application. In some cases, $h$ is a large neural network, repurposing the pre-trained model entirely (Watson et al., 2023; Lin et al., 2023); in others, $h$ is a minimal projection with few parameters (Cheng et al., 2023), or even without any parameters at all (i.e., a zero-shot setup; Meier et al. (2021); Dutton et al. (2024)). The fine-tuning head $h$ can also be a machine learning algorithm other than a neural network (Samusevich et al., 2025).

##### Masked modeling.

While the objective of fine-tuning $h\circ f$ is determined by the downstream application, the choice of pre-training objective for $g\circ f$ is less straightforward. Nevertheless, the dominant paradigm for protein pre-training is masked modeling, which optimizes model weights to reconstruct missing protein parts. This objective has proven effective across diverse tasks (Heinzinger & Rost, 2025; Schmirler et al., 2024), including structure (Lin et al., 2023; Jumper et al., 2021), fitness (Meier et al., 2021; Su et al., 2023), and function prediction (Samusevich et al., 2025; Yu et al., 2023; Elnaggar et al., 2021), as well as protein design (Hayes et al., 2024), and has been successfully applied to various protein representations such as sequences (Hayes et al., 2024; Elnaggar et al., 2023), graphs (Dieckhaus et al., 2024; Bushuiev et al., 2023), and voxels (Diaz et al., 2023).

##### Model customization.

Several studies have shown that machine learning models for proteins benefit from being fine-tuned on protein-specific  (Notin et al., 2024; Kirjner et al., 2023; Rao et al., 2019) or protein family-specific (Sevgen et al., 2023; Samusevich et al., 2025) data. However, collecting additional data may be resource-intensive, and for many targets, relevant datasets or proteins may be limited or not available (Durairaj et al., 2023; Kim et al., 2025). In this paper, we propose a versatile method enabling customizing PLMs for a single target protein or its MSA in a self-supervised manner, on the fly, and without assuming any additional data. Customization methods have been developed in computer vision (Chi et al., 2024; Wang et al., 2023; Xiao et al., 2022; Karani et al., 2021) and natural language processing (Hübotter et al., 2024; Hardt & Sun, 2023; Ben-David et al., 2022; Banerjee et al., 2021). The paradigm of test-time training (TTT), developed to mitigate distribution shifts in computer vision applications (Gandelsman et al., 2022; Sun et al., 2020), is the main inspiration for our work. We demonstrate that customization via test-time training enhances the accuracy of PLMs across a wide range of downstream tasks even without the presence of explicit distribution shifts.

## 3 Protein model customization with ProteinTTT

In this section, we describe the proposed Protein Test-Time Training (ProteinTTT) approach ([Section˜3.1](https://arxiv.org/html/2411.02109v2#S3.SS1)), followed by its applications to a range of well-established models and datasets ([Section˜3.2](https://arxiv.org/html/2411.02109v2#S3.SS2)).

### 3.1 Self-supervised customization to a target protein

At test time, we assume a Y-shaped model with a backbone $f$ that has been pre-trained via the self-supervised track $g\circ f$, followed by task-specific fine-tuning through the supervised track $h\circ f$. The goal of customization with ProteinTTT is to adapt the backbone $f$ to a single protein $x$ before making a prediction on a downstream task via the supervised track $h\circ f$. To achieve this, we customize the backbone $f$ to the single example $x$:

$$ $\displaystyle\text{ProteinTTT}:(h\circ f(\cdot;\theta_{0}),x)\mapsto h\circ f(\cdot;\theta_{x})$ (1) $$

where $\theta_{0}$ denotes pre-trained parameters and $\theta_{x}$ parameters optimized for the target protein $x$ using the self-supervised track $g\circ f$, while the supervised head $h$ remains frozen.
[Figure˜2](https://arxiv.org/html/2411.02109v2#S3.F2)a illustrates our customization approach, which is summarized in the following sections. [Appendix˜B](https://arxiv.org/html/2411.02109v2#A2) describes the extension of our method to customization using a MSA of a protein, rather than its single sequence.

##### Customization training objective.

We customize $g\circ f$ to a single target protein sequence $x$ via minimizing the masked language modeling objective (Devlin, 2018; Rives et al., 2021):

$$ $\displaystyle\mathcal{L}(x;\theta)=\mathbb{E}_{M\sim p_{\text{mask}}(M)}\Biggl[\sum_{i\in M}{-\log{p(x_{i}|x_{\setminus M};\theta)}}\Biggl],$ (2) $$

where $x$ denotes a sequence of protein tokens (typically amino acid types), and $\mathbb{E}_{M}$ represents the expectation over randomly sampled masking positions $M$. The objective function $\mathcal{L}(x;\theta)$ maximizes the log-probabilities $\log{p(x_{i}|x_{\setminus M};\theta)}\ \dot{=}\ g(f(x_{\setminus M};\theta))_{i}$ of the true (i.e., wild-type) tokens $x_{i}$ at the masked positions $i\in M$ in the partially masked sequence $x_{\setminus M}$, where $\theta$ denotes the parameters of the backbone $f$, and $g$ is the masked language modeling head. Please note that here we focus on bi-directional masked modeling models, which employ random masking, but the method can be easily extended to models employing autoregressive masking.

To ensure consistency between the customization and pre-training, ProteinTTT adopts the same masking and data preprocessing strategies used during pre-training. Specifically, $p_{\text{mask}}(M)$ can follow different distributions, such as sampling a fixed proportion (e.g., 15%) of random amino acid tokens (Lin et al., 2023), or dynamically varying the number of sampled tokens based on another distribution (e.g., a beta distribution; Hayes et al. (2024)). During the customization, we replicate the masking distribution used during the pre-training. We also replicate other pre-training practices, such as replacing 10% of masked tokens with random tokens and another 10% with the original tokens (Devlin, 2018; Lin et al., 2023; Su et al., 2023) or cropping sequences to random 1024-token fragments (Lin et al., 2023; Su et al., 2023).

##### Optimization.

Figure: Figure 2: Overview of protein language model (PLM) customization with ProteinTTT. (a) Given a protein sequence of interest $x$ and a pretrained PLM $f(\cdot;\theta_{0})$, ProteinTTT yields a customized version of the PLM $f(\cdot;\theta_{x})$ for that sequence. Customization is achieved by fine-tuning (fire icon) the pretrained parameters $\theta_{0}$ via masked language modeling solely on the input sequence for $T$ steps, selecting the optimal parameters $\theta_{x}$ using a confidence function $c$. This procedure adapts the model specifically to the input sequence, improving its internal representation as measured by model perplexity. (b) Once customized, the PLM can be used with pretrained task-specific heads, such as structure, fitness, or function prediction modules, $h_{1}$, $h_{2}$, and $h_{3}$, respectively, without modifying their parameters (snowflake icon). For example, the ESM2 PLM can be customized and then used with the pretrained ESMFold structure prediction head without modifying its 1.4-billion task-specific parameters, resulting in improved structure prediction for the given sequence (e.g., [Figure˜1](https://arxiv.org/html/2411.02109v2#S1.F1)).
Refer to caption: x3.png

Since customization with ProteinTTT does not assume more than a single protein, early stopping on validation data is not feasible. To address this, we first fine-tune the pre-trained parameters $\theta_{0}$ of a backbone $f$ for a fixed number of steps $T$, yielding parameters $\Theta=\{\theta_{0},\theta_{1},\dots,\theta_{T}\}$. The final customized parameters $\theta_{x}$ are selected as $\operatorname*{arg\,max}_{\theta\in\Theta}c(h(f(x;\theta)))$ where $c$ is a confidence function. If $c$ is not available, we set $\theta_{x}=\theta_{T}$. [Section˜G.2](https://arxiv.org/html/2411.02109v2#A7.SS2) discusses how using pLDDT as the confidence function $c$ for structure prediction makes ProteinTTT robust to hyperparameter selection and how the number of steps $T$ can be fixed (e.g., $T=30$) while optimizing learning rate and batch size effectively. Before customizing for the next target protein, the parameters are reset to $\theta_{0}$.

To make ProteinTTT easily applicable to large-scale models (e.g., the 3B-parameter ESM2 backbone), we leverage low-rank adaptation (LoRA; Hu et al. (2021)) and gradient accumulation during customization. Additionally, to improve the stability and predictability of customization, we use stochastic gradient descent (SGD; Ruder (2016)) instead of the commonly used Adam optimizer (Kingma & Ba, 2015), following (Gandelsman et al., 2022). Further details are provided in [Appendix˜E](https://arxiv.org/html/2411.02109v2#A5).

Figure: Figure 3: Customization with ProteinTTT improves protein structure prediction by reducing protein sequence perplexity. ESMFold fails to predict the structure of chain B from PDB entry 7EBL in the CAMEO validation set, as shown at customization step 0, where the perplexity is high and the TM-score is low. By applying customization with ProteinTTT for the single target sequence, the model iteratively improves the structure prediction quality, as demonstrated by the increasing TM-score, associated with reduced perplexity. At customization step 7, the predicted structure achieves the highest TM-score, as well as the highest predicted confidence metric pLDDT, enabling the selection of this step as the final prediction by the customized ESMFold + ProteinTTT.
Refer to caption: x4.png

### 3.2 Inference on downstream tasks

Once the backbone $f$ is adapted to a target protein via self-supervised customization, it can be used in conjunction with a pre-trained downstream head $h$, as $h\circ f$. The key idea of customization with ProteinTTT is not to update the head $h$, but instead to leverage improved representations from $f$ ([Figure˜2](https://arxiv.org/html/2411.02109v2#S3.F2)b). [Appendix˜A](https://arxiv.org/html/2411.02109v2#A1) provides a justification for why these customized representations generally enhance performance on downstream tasks by linking ProteinTTT to perplexity minimization.

Since Y-shaped architectures are prevalent in protein machine learning, ProteinTTT can be straightforwardly applied to numerous tasks. In this work, we consider three standard problems: protein structure, fitness, and function prediction, and apply our method to corresponding well-established models. For structure prediction, we apply ProteinTTT to ESMFold ([Figure˜3](https://arxiv.org/html/2411.02109v2#S3.F3), Lin et al. (2023), HelixFold-Single (Fang et al., 2023), and ESM3 (Hayes et al., 2024); for fitness prediction, we use ESM2 (Lin et al., 2023), SaProt (Su et al., 2023), ProSST (Li et al., 2024), and MSA Transformer (Rao et al., 2021); and for function prediction, we apply ProteinTTT to ESM-1v-based (Meier et al., 2021) EnzymeExplorer (Samusevich et al., 2025) and ESM-1b-based (Rives et al., 2021) Light attention (Stärk et al., 2021).

In all models we consider, $f$ is a Transformer encoder that takes protein tokens as input, and $g$ is a masked language modeling head (a layer mapping token embeddings to amino acid types). The downstream task heads $h$ vary strongly across tasks. For structure prediction, $h$ is a protein structure predictor: in ESMFold and HelixFold-Single, it is an AlphaFold2-inspired module (Jumper et al., 2021), while in ESM3, it is a VQ-VAE structure decoder (Razavi et al., 2019). For fitness prediction, $h$ outputs a single score per sequence; ESM2, SaProt, and ProSST perform zero-shot inference using $h\circ f$ via log odds from $g$, with $h$ functioning as a simple adaptation of $g$ without introducing extra parameters. The function predictors are classification models: in EnzymeExplorer (Samusevich et al., 2025), $h$ is a random forest that outputs substrate probabilities, and in Light attention (Stärk et al., 2021), $h$ is a light attention module predicting protein localization classes within a cell.

## 4 Experiments

In this section, we evaluate ProteinTTT on three well-established downstream tasks in protein machine learning: structure ([Section˜4.1](https://arxiv.org/html/2411.02109v2#S4.SS1)), fitness ([Section˜4.2](https://arxiv.org/html/2411.02109v2#S4.SS2)), and function ([Section˜4.3](https://arxiv.org/html/2411.02109v2#S4.SS3)) prediction.

### 4.1 Protein structure prediction

Protein structure prediction is the task of predicting 3D coordinates of protein atoms given the amino acid sequence. It is arguably one of the best-established problems in computational biology (Jumper et al., 2021; Lin et al., 2023; Abramson et al., 2024).

##### Evaluation setup.

To evaluate the performance of ProteinTTT, we employ CAMEO, a standard benchmark for protein folding. We use the validation and test folds from Lin et al. (2023), focusing only on targets with low-confidence predictions from the base ESMFold, as determined by pLDDT and perplexity ([Section˜E.1](https://arxiv.org/html/2411.02109v2#A5.SS1)). We use the standard TM-score (Zhang & Skolnick, 2004) and LDDT (Mariani et al., 2013) metrics to evaluate global and local structure prediction quality, respectively.

As baseline methods, we use techniques alternative to ProteinTTT for improving the performance of the pre-trained base models. In particular, the ESMFold paper proposes randomly masking 15% of amino acids in a protein sequence before inference, allowing for sampling multiple protein structure predictions from the regression ESMFold model (Lin et al., 2023). For each sequence, we sample a number of predictions equal to the total number of ProteinTTT steps and refer to this baseline as ESMFold + MP (Masked Prediction). As a baseline for ESM3, we use chain-of-thought iterative decoding, referred to as ESM3 + CoT, proposed in the ESM3 paper (Hayes et al., 2024).

##### Results.

Customization with ProteinTTT consistently improves the performance of all the tested methods, ESMFold, HelixFold-Single, and ESM3, outperforming the masked prediction (ESMFold + MP) and chain-of-thought (ESM3 + CoT) baselines, as shown in [Table˜2](https://arxiv.org/html/2411.02109v2#S4.T2). Among the 18 challenging CAMEO test proteins, ProteinTTT significantly improved the prediction of 7, 5, and 6 structures from ESMFold, HelixFold-Single, and ESM3, respectively, while only slightly disrupting the prediction of 2, 1, and 1 structures, respectively ([Figure˜A6](https://arxiv.org/html/2411.02109v2#A7.F6)). Most notably, ProteinTTT enables accurate structure prediction for targets that are poorly predicted with the original models. For instance, [Figure˜1](https://arxiv.org/html/2411.02109v2#S1.F1) presents a strongly improved structure predicted using ESMFold + ProteinTTT for the target that was part of the CASP14 competition and shown as an unsuccessful case in the original ESMFold publication (Lin et al. (2023), Fig. 2E). Another example is shown in [Figure˜3](https://arxiv.org/html/2411.02109v2#S3.F3), where ProteinTTT refined the structure prediction from a low-quality prediction (TM-score $=0.29$) to a nearly perfectly folded protein (TM-score $=0.92$). [Figure˜A4](https://arxiv.org/html/2411.02109v2#A7.F4) shows that ESMFold + ProteinTTT maintains computational efficiency of ESMFold, being an order of magnitude faster than AlphaFold2. [Figure˜A11](https://arxiv.org/html/2411.02109v2#A7.F11) additionally demonstrates the robustness of ESM3 + ProteinTTT to the choice of hyperparameters.

### 4.2 Protein Fitness Prediction

The task of protein fitness prediction is to accurately order mutations of a protein based on their disruptive/favorable effects on protein functioning.

##### Evaluation Setup.

We evaluate the models using ProteinGym, the state-of-the-art fitness prediction benchmark (Notin et al., 2024), focusing on its well-established zero-shot setup. Since the zero-shot setup only provides a test set without any data split, we also validate ProteinTTT on independent data. To achieve this, we create a new fitness prediction dataset mined from MaveDB, a public repository containing datasets from Multiplexed Assays of Variant Effect (MAVEs) (Esposito et al., 2019). Following ProteinGym, we measure performance on both datasets using Spearman correlation between predicted and experimental fitness values.

##### Results.

ProteinTTTconsistently enhances fitness prediction performance of all the tested models across varying model scales (35M and 650M parameters for both ESM2 and SaProt; 110M for ProSST) and both datasets, i.e., test ProteinGym ([Table˜4](https://arxiv.org/html/2411.02109v2#S4.T4)) and validation MaveDB ([Table˜A5](https://arxiv.org/html/2411.02109v2#A7.T5)). Notably, ProSST + ProteinTTT sets a new state of the art on the ProteinGym benchmark (Spearman correlation coefficients calculated for individual deep mutational scanning experiments (DMSs) have statistically significant difference according to a paired t-test with p < 0.05).

We observe that ProteinTTT primarily improves performance for proteins with low MSA depth (i.e., the number of available homologous sequences), suggesting that single-sequence customization enhances predictions for proteins with fewer similar sequences in the training data ([Table˜A4](https://arxiv.org/html/2411.02109v2#A7.T4)). The fact that ProteinTTT more effectively improves the performance of smaller ESM2 and SaProt models compared to their larger variants may be a result of the benchmark performance being saturated for larger models, consistent with a recent observation (Notin, 2025). We provide a qualitative example showing how ESM2 (650M) + ProteinTTT significantly improves fitness prediction by capturing residues critical for protein stability ([Figure˜A5](https://arxiv.org/html/2411.02109v2#A7.F5)). We also demonstrate that customization can be combined with evolutionary information from MSA to further boost fitness prediction ([Appendix˜B](https://arxiv.org/html/2411.02109v2#A2)).

### 4.3 Protein function prediction

Finally, we demonstrate a proof of concept for customization in the context of protein function prediction. We experiment with two tasks: predicting protein location within a cell (Stärk et al., 2021), and substrate classification for terpene synthases (TPS), enzymes producing the largest class of natural products (Samusevich et al., 2025). [Appendix˜C](https://arxiv.org/html/2411.02109v2#A3) shows that per-protein customization with ProteinTTT consistently enhances the performance of representative models on both tasks.

## 5 Case studies

ProteinTTTcan be readily incorporated into structure, fitness, or function prediction pipelines by adding several lines of code ([Appendix˜D](https://arxiv.org/html/2411.02109v2#A4)). Here, we demonstrate two challenging structure prediction case studies: improving modeling of antibody–antigen loops ([Section˜5.1](https://arxiv.org/html/2411.02109v2#S5.SS1)) and expanding known structures of viral proteins ([Section˜5.2](https://arxiv.org/html/2411.02109v2#S5.SS2)).

### 5.1 Modeling antibody–antigen loops

Figure: Figure 4: ProteinTTT improves modeling of antibody–antigen loops. (a) Average LDDT on the antibody complementarity-determining regions (CDRs, 175 structures) and antigens (814 structures) from the SAbDab dataset with ESMFold pLDDT < 70. Error bars indicate 95% confidence intervals estimated from 1000 bootstrap samples. (b) Example of improved structure prediction for CDRs in the 8K2W entry. The CDR regions H1, H2, and H3, i.e., the parts of the antibody that bind to the antigen, are highlighted with spheres, while black lines show the alignment error between the ground-truth CDR structure (white) and the predictions (colored).
Refer to caption: x5.png

Accurately predicting structures of antibodies (e.g., human defensive proteins) and antigens (e.g., viral proteins) enables rational design of new therapeutics (Bennett et al., 2025). However, the presence of highly variable loop regions makes modeling of these interactions a long-standing challenge. Here, we show that ProteinTTT substantially improves structure prediction for these loop-formed complementarity-determining regions (CDRs) of antibodies, i.e., the parts that bind antigens, as well as for antigens themselves, on the well-established SAbDab dataset (Dunbar et al., 2014).

We take the structures from SAbDab that are not predicted well by ESMFold (pLDDT < 70) and show that ProteinTTT improves the LDDT score for 115 of 175 antibody CDR substructures (66%) and 487 of 814 antigen chains (60%). As shown in [Figure˜4](https://arxiv.org/html/2411.02109v2#S5.F4)a, ESMFold + ProteinTTT achieves significantly higher average LDDT scores compared to general-purpose ESMFold. [Figure˜4](https://arxiv.org/html/2411.02109v2#S5.F4)b illustrates how ProteinTTT enables accurate prediction of all three CDRs in an antibody chain, providing an improved understanding of its binding interface with the corresponding antigen.

### 5.2 Expanding known structures of viral proteins

Figure: Figure 5: ProteinTTT expands the Big Fantastic Virus Database (BFVD). (a) ProteinTTT (light green) substantially improves the performance of ESMFold (yellow) on viral proteins, yielding better structures (pink) for 19% of BFVD entries compared to the original predictions by AlphaFold2 (green). (b) Improvements in pLDDT for ESMFold after ProteinTTT correspond to improvements in LDDT, as benchmarked against BFVD AlphaFold2 structures with pLDDT $>$ 90. (c) ProteinTTT provides the largest pLDDT improvements (y-axis) for the most out-of-distribution proteins, i.e., those with the smallest MSAs (left on the x-axis) from the Logan database. (d) Structural comparison for BFVD entry UPI000641889E against the PDB structure 2N2J (100% sequence identity) shows that ESMFold + ProteinTTT yields a prediction closest to the ground truth (gray), as also measured by LDDT. (e–g) Additional examples of high-quality viral structures (as measured by pLDDT) predicted with ESMFold + ProteinTTT but not with ESMFold or AlphaFold2. Higher pLDDT values are better.
Refer to caption: x6.png

Predicting the structures of viral proteins is vital for vaccine development, antiviral design, and understanding infection (Bravi, 2024). Nevertheless, it remains challenging due to the high mutation rate, which often leaves viral proteins without close homologs or experimental structures in databases (Kim et al., 2025). Here, we demonstrate that per-protein customized predictions with ESMFold + ProteinTTT improve viral protein structure prediction, substantially expanding the Big Fantastic Virus Database—the repository of viral protein structures (Kim et al., 2025).

Among all the entries in BFVD, predicted with AlphaFold2 through ColabFold (Mirdita et al., 2022) using MSAs constructed from Logan (Chikhi et al., 2024), only 55% have high-quality structure predictions (pLDDT > 70). We apply ESMFold and ESMFold + ProteinTTT to the BFVD entries to expand the database with higher-quality structures. This is achieved by applying all three methods to the specific protein and taking the predicted structure with the highest pLDDT. While ESMFold manages to improve the predicted structure (as measured by pLDDT) for 10% of the BFVD proteins, ESMFold + ProteinTTT leads to an improvement for 19% of the dataset entries, substantially increasing the quality of known viral protein structures ([Figure˜5](https://arxiv.org/html/2411.02109v2#S5.F5)a).

We validate that the improved pLDDT confidence values from ESMFold + ProteinTTT correlate with the quality of the predicted structures, as measured by LDDT against reference AlphaFold2 structures having pLDDT > 90 (Pearson = 0.875; [Figure˜A9](https://arxiv.org/html/2411.02109v2#A7.F9)). Notably, the largest improvements in pLDDT align with the largest improvements in LDDT ([Figure˜5](https://arxiv.org/html/2411.02109v2#S5.F5)b). We find that the benefit of customization saturates with the number of homologs available for a protein, indicating that ProteinTTT is most effective for challenging, out-of-distribution proteins ([Figure˜5](https://arxiv.org/html/2411.02109v2#S5.F5)c). Finally, [Figure˜5](https://arxiv.org/html/2411.02109v2#S5.F5)d–g shows examples where ProteinTTT enables high-confidence structure predictions in cases where general-purpose, uncustomized AlphaFold2 and ESMFold struggle.

## 6 Discussion

We introduce ProteinTTT, a method for customizing protein language models to individual targets. ProteinTTT consistently improves performance across various models, their scales, and downstream tasks. It excels on challenging, out-of-distribution examples where general models often fail. We demonstrate its practical value through two case studies: enhancing the structural prediction of difficult antibody-antigen loops and improving 19% of low-confidence viral protein structures in the Big Fantastic Virus Database. Our work establishes per-protein customization as a powerful and practical tool for biological research.

#### Acknowledgments

We thank Milot Mirdita for discussions and feedback about this work.
This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through projects e-INFRA CZ [ID:90254], ELIXIR [LM2023055], CETOCOEN Excellence CZ.02.1.01/0.0/0.0/17_043/0009632, ESFRI RECETOX RI LM2023069.
This work was also supported by the CETOCOEN EXCELLENCE Teaming project supported from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 857560.
This work was also supported by the European Union (CLARA (No. 101136607), ERC FRONTIER (no. 101097822), ELIAS (No. 101120237)) and the Technology Agency of the Czech Republic under the NCC Programme from the state budget (No. RETEMED TN02000122) and under the NRP from the EU RRF (No. TEREP TN02000122/001N).
This work was also supported by the Czech Science Foundation (GA CR) grant 21-11563M and by the European Union’s Horizon Europe program (ERC, TerpenCode, 101170268). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
M.S. acknowledges support by the National Research Foundation of Korea grants (RS-2020-NR049543, RS-2021-NR061659 and RS-2021-NR056571, RS-2024-00396026), Creative-Pioneering Researchers Program and Novo Nordisk Foundation (NNF24SA0092560).

## References

- Abramson et al. (2024)
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al.
Accurate structure prediction of biomolecular interactions with alphafold 3.
*Nature*, pp. 1–3, 2024.
doi: 10.1038/s41586-024-07487-w.
URL [https://doi.org/10.1038/s41586-024-07487-w](https://doi.org/10.1038/s41586-024-07487-w).
- Almagro Armenteros et al. (2017)
José Juan Almagro Armenteros, Casper Kaae Sønderby, Søren Kaae Sønderby, Henrik Nielsen, and Ole Winther.
Deeploc: prediction of protein subcellular localization using deep learning.
*Bioinformatics*, 33(21):3387–3395, 2017.
doi: 10.1093/bioinformatics/btx431.
URL [https://doi.org/10.1093/bioinformatics/btx431](https://doi.org/10.1093/bioinformatics/btx431).
- Ashcroft et al. (2023)
Frances M. Ashcroft, Matthew Lloyd, and Elizabeth A. Haythorne.
Glucokinase activity in diabetes: too much of a good thing?
*Trends in Endocrinology & Metabolism*, 34(2):119–130, Feb 2023.
ISSN 1043-2760.
doi: 10.1016/j.tem.2022.12.007.
URL [https://doi.org/10.1016/j.tem.2022.12.007](https://doi.org/10.1016/j.tem.2022.12.007).
- Banerjee et al. (2021)
Pratyay Banerjee, Tejas Gokhale, and Chitta Baral.
Self-supervised test-time learning for reading comprehension.
*arXiv preprint arXiv:2103.11263*, 2021.
doi: 10.48550/arXiv.2103.11263.
URL [https://doi.org/10.48550/arXiv.2103.11263](https://doi.org/10.48550/arXiv.2103.11263).
- Ben-David et al. (2022)
Eyal Ben-David, Nadav Oved, and Roi Reichart.
Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains.
*Transactions of the Association for Computational Linguistics*, 10:414–433, 2022.
doi: 10.48550/arXiv.2102.12206.
URL [https://doi.org/10.48550/arXiv.2102.12206](https://doi.org/10.48550/arXiv.2102.12206).
- Bennett et al. (2025)
Nathaniel R Bennett, Joseph L Watson, Robert J Ragotte, Andrew J Borst, DéJenaé L See, Connor Weidle, Riti Biswas, Yutong Yu, Ellen L Shrock, Russell Ault, et al.
Atomically accurate de novo design of antibodies with rfdiffusion.
*bioRxiv*, pp. 2024–03, 2025.
doi: 10.1101/2024.03.14.585103.
URL [https://doi.org/10.1101/2024.03.14.585103](https://doi.org/10.1101/2024.03.14.585103).
- Bravi (2024)
Barbara Bravi.
Development and use of machine learning algorithms in vaccine target selection.
*npj Vaccines*, 9(1):15, 2024.
doi: 10.1038/s41541-023-00795-8.
URL [https://doi.org/10.1038/s41541-023-00795-8](https://doi.org/10.1038/s41541-023-00795-8).
- Brown (2020)
Tom B Brown.
Language models are few-shot learners.
*arXiv preprint arXiv:2005.14165*, 2020.
doi: 10.48550/arXiv.2005.1416.
URL [https://doi.org/10.48550/arXiv.2005.1416](https://doi.org/10.48550/arXiv.2005.1416).
- Bushuiev et al. (2023)
Anton Bushuiev, Roman Bushuiev, Anatolii Filkin, Petr Kouba, Marketa Gabrielova, Michal Gabriel, Jiri Sedlar, Tomas Pluskal, Jiri Damborsky, Stanislav Mazurenko, et al.
Learning to design protein-protein interactions with enhanced generalization.
*arXiv preprint arXiv:2310.18515*, 2023.
doi: 10.48550/arXiv.2310.18515.
URL [https://arxiv.org/abs/2310.18515](https://arxiv.org/abs/2310.18515).
- Chelba et al. (2013)
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson.
One billion word benchmark for measuring progress in statistical language modeling.
*arXiv preprint arXiv:1312.3005*, 2013.
doi: 10.48550/arXiv.1312.3005.
URL [https://doi.org/10.48550/arXiv.1312.3005](https://doi.org/10.48550/arXiv.1312.3005).
- Chen & Gong (2022)
Tianlong Chen and Chengyue Gong.
Hotprotein: A novel framework for protein thermostability prediction and editing.
*NeurIPS 2022*, 2022.
URL [https://openreview.net/forum?id=YDJRFWBMNby](https://openreview.net/forum?id=YDJRFWBMNby).
- Cheng et al. (2023)
Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilė Žemgulytė, Taylor Applebaum, Alexander Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, et al.
Accurate proteome-wide missense variant effect prediction with alphamissense.
*Science*, 381(6664):eadg7492, 2023.
doi: 10.1126/science.adg7492.
URL [https://www.science.org/doi/10.1126/science.adg7492](https://www.science.org/doi/10.1126/science.adg7492).
- Chi et al. (2024)
Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, and Yang Wang.
Adapting to distribution shift by visual domain prompt generation.
*arXiv preprint arXiv:2405.02797*, 2024.
doi: 10.48550/arXiv.2405.02797.
URL [https://doi.org/10.48550/arXiv.2405.02797](https://doi.org/10.48550/arXiv.2405.02797).
- Chikhi et al. (2024)
Rayan Chikhi, Téo Lemane, Raphaël Loll-Krippleber, Mercè Montoliu-Nerin, Brice Raffestin, Antonio Pedro Camargo, Carson J Miller, Mateus Bernabe Fiamenghi, Daniel Paiva Agustinho, Sina Majidian, et al.
Logan: planetary-scale genome assembly surveys life’s diversity.
*bioRxiv*, pp. 2024–07, 2024.
doi: 10.1101/2024.07.30.605881.
URL [https://doi.org/10.1101/2024.07.30.605881](https://doi.org/10.1101/2024.07.30.605881).
- Chothia & Lesk (1987)
Cyrus Chothia and Arthur M Lesk.
Canonical structures for the hypervariable regions of immunoglobulins.
*Journal of molecular biology*, 196(4):901–917, 1987.
doi: 10.1016/0022-2836(87)90412-8.
URL [https://doi.org/10.1016/0022-2836(87)90412-8](https://doi.org/10.1016/0022-2836(87)90412-8).
- Christianson (2017)
David W. Christianson.
Structural and chemical biology of terpenoid cyclases.
*Chemical Reviews*, 117(17):11570–11648, Sep 2017.
ISSN 0009-2665.
doi: 10.1021/acs.chemrev.7b00287.
URL [https://doi.org/10.1021/acs.chemrev.7b00287](https://doi.org/10.1021/acs.chemrev.7b00287).
- Consortium (2023)
The UniProt Consortium.
Uniprot: the universal protein knowledgebase in 2023.
*Nucleic acids research*, 51(D1):D523–D531, 2023.
doi: 10.1093/nar/gkac1052.
URL [https://doi.org/10.1093/nar/gkac1052](https://doi.org/10.1093/nar/gkac1052).
- Devlin (2018)
Jacob Devlin.
Bert: Pre-training of deep bidirectional transformers for language understanding.
*arXiv preprint arXiv:1810.04805*, 2018.
doi: 10.48550/arXiv.1810.04805.
URL [https://doi.org/10.48550/arXiv.1810.04805](https://doi.org/10.48550/arXiv.1810.04805).
- Diaz et al. (2023)
Daniel J Diaz, Chengyue Gong, Jeffrey Ouyang-Zhang, James M Loy, Jordan Wells, David Yang, Andrew D Ellington, Alex Dimakis, and Adam R Klivans.
Stability oracle: a structure-based graph-transformer for identifying stabilizing mutations.
*BioRxiv*, pp. 2023–05, 2023.
doi: 10.1038/s41467-024-49780-2.
URL [https://doi.org/10.1038/s41467-024-49780-2](https://doi.org/10.1038/s41467-024-49780-2).
- Dieckhaus et al. (2024)
Henry Dieckhaus, Michael Brocidiacono, Nicholas Z Randolph, and Brian Kuhlman.
Transfer learning to leverage larger datasets for improved prediction of protein stability changes.
*Proceedings of the National Academy of Sciences*, 121(6):e2314853121, 2024.
doi: 10.1073/pnas.2314853121.
URL [https://doi.org/10.1073/pnas.2314853121](https://doi.org/10.1073/pnas.2314853121).
- Dunbar et al. (2014)
James Dunbar, Konrad Krawczyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane.
Sabdab: the structural antibody database.
*Nucleic acids research*, 42(D1):D1140–D1146, 2014.
doi: 10.1093/nar/gkt1043.
URL [https://doi.org/10.1093/nar/gkt1043](https://doi.org/10.1093/nar/gkt1043).
- Durairaj et al. (2023)
Janani Durairaj, Andrew M Waterhouse, Toomas Mets, Tetiana Brodiazhenko, Minhal Abdullah, Gabriel Studer, Gerardo Tauriello, Mehmet Akdel, Antonina Andreeva, Alex Bateman, et al.
Uncovering new families and folds in the natural protein universe.
*Nature*, 622(7983):646–653, 2023.
doi: 10.1038/s41586-023-06622-3.
URL [https://doi.org/10.1038/s41586-023-06622-3](https://doi.org/10.1038/s41586-023-06622-3).
- Dutton et al. (2024)
Oliver Dutton, Sandro Bottaro, Istvan Redl, Michele Invernizzi, Albert Chung, Carlo Fisicaro, Falk Hoffmann, Stefano Ruschetta, Fabio Airoldi, Louie Henderson, et al.
Improving inverse folding models at protein stability prediction without additional training or data.
*bioRxiv*, pp. 2024–06, 2024.
doi: 10.1101/2024.06.15.599145.
URL [https://doi.org/10.1101/2024.06.15.599145](https://doi.org/10.1101/2024.06.15.599145).
- Elnaggar et al. (2021)
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al.
Prottrans: Toward understanding the language of life through self-supervised learning.
*IEEE transactions on pattern analysis and machine intelligence*, 44(10):7112–7127, 2021.
doi: 10.1109/tpami.2021.3095381.
URL [https://doi.org/10.1109/tpami.2021.3095381](https://doi.org/10.1109/tpami.2021.3095381).
- Elnaggar et al. (2023)
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost.
Ankh: Optimized protein language model unlocks general-purpose modelling.
*arXiv preprint arXiv:2301.06568*, 2023.
doi: 10.48550/arXiv.2301.06568.
URL [https://doi.org/10.48550/arXiv.2301.06568](https://doi.org/10.48550/arXiv.2301.06568).
- Esposito et al. (2019)
Daniel Esposito, Jochen Weile, Jay Shendure, Lea M Starita, Anthony T Papenfuss, Frederick P Roth, Douglas M Fowler, and Alan F Rubin.
Mavedb: an open-source platform to distribute and interpret data from multiplexed assays of variant effect.
*Genome biology*, 20:1–11, 2019.
doi: 10.1186/s13059-019-1845-6.
URL [https://doi.org/10.1186/s13059-019-1845-6](https://doi.org/10.1186/s13059-019-1845-6).
- Fang et al. (2023)
Xiaomin Fang, Fan Wang, Lihang Liu, Jingzhou He, Dayong Lin, Yingfei Xiang, Kunrui Zhu, Xiaonan Zhang, Hua Wu, Hui Li, et al.
A method for multiple-sequence-alignment-free protein structure prediction using a protein language model.
*Nature Machine Intelligence*, 5(10):1087–1096, 2023.
doi: 10.1038/s42256-023-00721-6.
URL [https://doi.org/10.1038/s42256-023-00721-6](https://doi.org/10.1038/s42256-023-00721-6).
- Feng et al. (2024)
Tao Feng, Ziqi Gao, Jiaxuan You, Chenyi Zi, Yan Zhou, Chen Zhang, and Jia Li.
Deep reinforcement learning for modelling protein complexes.
*arXiv preprint arXiv:2405.02299*, 2024.
doi: 10.48550/arXiv.2405.02299.
URL [https://doi.org/10.48550/arXiv.2405.02299](https://doi.org/10.48550/arXiv.2405.02299).
- Gandelsman et al. (2022)
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros.
Test-time training with masked autoencoders.
*Advances in Neural Information Processing Systems*, 35:29374–29385, 2022.
doi: 10.48550/arXiv.2209.07522.
URL [https://doi.org/10.48550/arXiv.2209.07522](https://doi.org/10.48550/arXiv.2209.07522).
- Gorodkin (2004)
Jan Gorodkin.
Comparing two k-category assignments by a k-category correlation coefficient.
*Computational biology and chemistry*, 28(5-6):367–374, 2004.
doi: 10.1016/j.compbiolchem.2004.09.006.
URL [https://doi.org/10.1016/j.compbiolchem.2004.09.006](https://doi.org/10.1016/j.compbiolchem.2004.09.006).
- Gu et al. (2022)
Xin Gu, Patrick Jouandin, Pranav V. Lalgudi, Rich Binari, Max L. Valenstein, Michael A. Reid, Annamarie E. Allen, Nolan Kamitaki, Jason W. Locasale, Norbert Perrimon, and David M. Sabatini.
Sestrin mediates detection of and adaptation to low-leucine diets in drosophila.
*Nature*, 608(7921):209–216, Aug 2022.
ISSN 1476-4687.
doi: 10.1038/s41586-022-04960-2.
URL [https://doi.org/10.1038/s41586-022-04960-2](https://doi.org/10.1038/s41586-022-04960-2).
- Gulen et al. (2023)
Muhammet F. Gulen, Natasha Samson, Alexander Keller, Marius Schwabenland, Chong Liu, Selene Glück, Vivek V. Thacker, Lucie Favre, Bastien Mangeat, Lona J. Kroese, Paul Krimpenfort, Marco Prinz, and Andrea Ablasser.
cgas–sting drives ageing-related inflammation and neurodegeneration.
*Nature*, 620(7973):374–380, Aug 2023.
ISSN 1476-4687.
doi: 10.1038/s41586-023-06373-1.
URL [https://doi.org/10.1038/s41586-023-06373-1](https://doi.org/10.1038/s41586-023-06373-1).
- Gunn & Neher (2023)
Kathryn H. Gunn and Saskia B. Neher.
Structure of dimeric lipoprotein lipase reveals a pore adjacent to the active site.
*Nature Communications*, 14(1):2569, May 2023.
ISSN 2041-1723.
doi: 10.1038/s41467-023-38243-9.
URL [https://doi.org/10.1038/s41467-023-38243-9](https://doi.org/10.1038/s41467-023-38243-9).
- Hardt & Sun (2023)
Moritz Hardt and Yu Sun.
Test-time training on nearest neighbors for large language models.
*arXiv preprint arXiv:2305.18466*, 2023.
doi: 10.48550/arXiv.2305.18466.
URL [https://doi.org/10.48550/arXiv.2305.18466](https://doi.org/10.48550/arXiv.2305.18466).
- Hayes et al. (2024)
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al.
Simulating 500 million years of evolution with a language model.
*bioRxiv*, pp. 2024–07, 2024.
doi: 10.1126/science.ads0018.
URL [https://www.science.org/doi/10.1126/science.ads0018](https://www.science.org/doi/10.1126/science.ads0018).
- Heinzinger & Rost (2025)
Michael Heinzinger and Burkhard Rost.
Teaching ai to speak protein.
*Current opinion in structural biology*, 91:102986, 2025.
doi: 10.1016/j.sbi.2025.102986.
URL [https://doi.org/10.1016/j.sbi.2025.102986](https://doi.org/10.1016/j.sbi.2025.102986).
- Hennigen & Kim (2023)
Lucas Torroba Hennigen and Yoon Kim.
Deriving language models from masked language models.
*arXiv preprint arXiv:2305.15501*, 2023.
doi: 10.48550/arXiv.2305.15501.
URL [https://doi.org/10.48550/arXiv.2305.15501](https://doi.org/10.48550/arXiv.2305.15501).
- Hopf et al. (2017)
Thomas A Hopf, John B Ingraham, Frank J Poelwijk, Charlotta PI Schärfe, Michael Springer, Chris Sander, and Debora S Marks.
Mutation effects predicted from sequence co-variation.
*Nature biotechnology*, 35(2):128–135, 2017.
doi: 10.1038/nbt.3769.
URL [https://doi.org/10.1038/nbt.3769](https://doi.org/10.1038/nbt.3769).
- Hoxhaj & Manning (2020)
Gerta Hoxhaj and Brendan D. Manning.
The pi3k–akt network at the interface of oncogenic signalling and cancer metabolism.
*Nature Reviews Cancer*, 20(2):74–88, Feb 2020.
ISSN 1474-1768.
doi: 10.1038/s41568-019-0216-7.
URL [https://doi.org/10.1038/s41568-019-0216-7](https://doi.org/10.1038/s41568-019-0216-7).
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2021.
doi: 10.48550/arXiv.2106.09685.
URL [https://doi.org/10.48550/arXiv.2106.09685](https://doi.org/10.48550/arXiv.2106.09685).
- Hübotter et al. (2024)
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause.
Efficiently learning at test-time: Active fine-tuning of llms.
*arXiv preprint arXiv:2410.08020*, 2024.
doi: 10.48550/arXiv.2410.08020.
URL [https://doi.org/10.48550/arXiv.2410.08020](https://doi.org/10.48550/arXiv.2410.08020).
- Jumper et al. (2021)
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.
Highly accurate protein structure prediction with alphafold.
*nature*, 596(7873):583–589, 2021.
doi: 10.1038/s41586-021-03819-2.
URL [https://doi.org/10.1038/s41586-021-03819-2](https://doi.org/10.1038/s41586-021-03819-2).
- Kantroo et al. (2024)
Pranav Kantroo, Gunter Wagner, and Benjamin Machta.
Pseudo-perplexity in one fell swoop for protein fitness estimation.
*bioRxiv*, pp. 2024–07, 2024.
doi: 10.48550/arXiv.2407.07265.
URL [https://doi.org/10.48550/arXiv.2407.07265](https://doi.org/10.48550/arXiv.2407.07265).
- Karani et al. (2021)
Neerav Karani, Ertunc Erdil, Krishna Chaitanya, and Ender Konukoglu.
Test-time adaptable neural networks for robust medical image segmentation.
*Medical Image Analysis*, 68:101907, 2021.
doi: 10.1016/j.media.2020.101907.
URL [https://doi.org/10.1016/j.media.2020.101907](https://doi.org/10.1016/j.media.2020.101907).
- Keckesova et al. (2017)
Zuzana Keckesova, Joana Liu Donaher, Jasmine De Cock, Elizaveta Freinkman, Susanne Lingrell, Daniel A. Bachovchin, Brian Bierie, Verena Tischler, Aurelia Noske, Marian C. Okondo, Ferenc Reinhardt, Prathapan Thiru, Todd R. Golub, Jean E. Vance, and Robert A. Weinberg.
Lactb is a tumour suppressor that modulates lipid metabolism and cell state.
*Nature*, 543(7647):681–686, Mar 2017.
ISSN 1476-4687.
doi: 10.1038/nature21408.
URL [https://doi.org/10.1038/nature21408](https://doi.org/10.1038/nature21408).
- Kim et al. (2025)
Rachel Seongeun Kim, Eli Levy Karin, Milot Mirdita, Rayan Chikhi, and Martin Steinegger.
Bfvd—a large repository of predicted viral protein structures.
*Nucleic Acids Research*, 53(D1):D340–D347, 2025.
doi: 10.1093/nar/gkae1119.
URL [https://doi.org/10.1093/nar/gkae1119](https://doi.org/10.1093/nar/gkae1119).
- Kingma & Ba (2015)
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*, 2015.
URL [http://arxiv.org/abs/1412.6980](http://arxiv.org/abs/1412.6980).
- Kirjner et al. (2023)
Andrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi S Jaakkola, Regina Barzilay, and Ila R Fiete.
Improving protein optimization with smoothed fitness landscapes.
In *The Twelfth International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=rxlF2Zv8x0](https://openreview.net/forum?id=rxlF2Zv8x0).
- Kouba et al. (2023)
Petr Kouba, Pavel Kohout, Faraneh Haddadi, Anton Bushuiev, Raman Samusevich, Jiri Sedlar, Jiri Damborsky, Tomas Pluskal, Josef Sivic, and Stanislav Mazurenko.
Machine learning-guided protein engineering.
*ACS catalysis*, 13(21):13863–13895, 2023.
doi: 10.1021/acscatal.3c02743.
URL [https://doi.org/10.1021/acscatal.3c02743](https://doi.org/10.1021/acscatal.3c02743).
- Laine et al. (2019)
Elodie Laine, Yasaman Karami, and Alessandra Carbone.
Gemme: a simple and fast global epistatic model predicting mutational effects.
*Molecular biology and evolution*, 36(11):2604–2619, 2019.
doi: 10.1093/molbev/msz179.
URL [https://doi.org/10.1093/molbev/msz179](https://doi.org/10.1093/molbev/msz179).
- Li et al. (2024)
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Liang Hong, and Pan Tan.
Prosst: Protein language modeling with quantized structure and disentangled attention.
*bioRxiv*, pp. 2024–04, 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/3ed57b293db0aab7cc30c44f45262348-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/3ed57b293db0aab7cc30c44f45262348-Paper-Conference.pdf).
- Lin et al. (2023)
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives.
Evolutionary-scale prediction of atomic-level protein structure with a language model.
*Science*, 379(6637):1123–1130, 2023.
doi: 10.1126/science.ade2574.
URL [https://www.science.org/doi/abs/10.1126/science.ade2574](https://www.science.org/doi/abs/10.1126/science.ade2574).
- Liu et al. (2021)
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi.
TTT++: when does self-supervised test-time training fail or thrive?
In *Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual*, pp. 21808–21820, 2021.
URL [https://proceedings.neurips.cc/paper/2021/hash/b618c3210e934362ac261db280128c22-Abstract.html](https://proceedings.neurips.cc/paper/2021/hash/b618c3210e934362ac261db280128c22-Abstract.html).
- Lloyd (1982)
Stuart Lloyd.
Least squares quantization in pcm.
*IEEE transactions on information theory*, 28(2):129–137, 1982.
doi: 10.1109/TIT.1982.1056489.
URL [https://doi.org/10.1109/TIT.1982.1056489](https://doi.org/10.1109/TIT.1982.1056489).
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019*. OpenReview.net, 2019.
URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7).
- Mariani et al. (2013)
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede.
lddt: a local superposition-free score for comparing protein structures and models using distance difference tests.
*Bioinformatics*, 29(21):2722–2728, 2013.
doi: 10.1093/bioinformatics/btt473.
URL [https://doi.org/10.1093/bioinformatics/btt473](https://doi.org/10.1093/bioinformatics/btt473).
- Meier et al. (2021)
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives.
Language models enable zero-shot prediction of the effects of mutations on protein function.
*Advances in neural information processing systems*, 34:29287–29303, 2021.
URL [https://proceedings.neurips.cc/paper/2021/hash/f51338d736f95dd42427296047067694-Abstract.html](https://proceedings.neurips.cc/paper/2021/hash/f51338d736f95dd42427296047067694-Abstract.html).
- Mikhael et al. (2024)
Peter Mikhael, Itamar Chinn, and Regina Barzilay.
Clipzyme: Reaction-conditioned virtual screening of enzymes.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=0mYAK6Yhhm](https://openreview.net/forum?id=0mYAK6Yhhm).
- Mirdita et al. (2022)
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger.
Colabfold: making protein folding accessible to all.
*Nature methods*, 19(6):679–682, 2022.
doi: 10.1038/s41592-022-01488-1.
URL [https://doi.org/10.1038/s41592-022-01488-1](https://doi.org/10.1038/s41592-022-01488-1).
- Notin (2025)
Pascal Notin.
Have we hit the scaling wall for protein language models?
Substack blog post, May 7 2025.
URL [https://pascalnotin.substack.com/p/have-we-hit-the-scaling-wall-for](https://pascalnotin.substack.com/p/have-we-hit-the-scaling-wall-for).
- Notin et al. (2024)
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al.
Proteingym: Large-scale benchmarks for protein fitness prediction and design.
*Advances in Neural Information Processing Systems*, 36, 2024.
doi: 10.1101/2023.12.07.570727.
URL [https://doi.org/10.1101/2023.12.07.570727](https://doi.org/10.1101/2023.12.07.570727).
- oh Seo et al. (2023)
Dong oh Seo, David O’Donnell, Nimansha Jain, Jason D. Ulrich, Jasmin Herz, Yuhao Li, Mackenzie Lemieux, Jiye Cheng, Hao Hu, Javier R. Serrano, Xin Bao, Emily Franke, Maria Karlsson, Martin Meier, Su Deng, Chandani Desai, Hemraj Dodiya, Janaki Lelwala-Guruge, Scott A. Handley, Jonathan Kipnis, Sangram S. Sisodia, Jeffrey I. Gordon, and David M. Holtzman.
Apoe isoform– and microbiota-dependent progression of neurodegeneration in a mouse model of tauopathy.
*Science*, 379(6628):eadd1236, 2023.
doi: 10.1126/science.add1236.
URL [https://www.science.org/doi/abs/10.1126/science.add1236](https://www.science.org/doi/abs/10.1126/science.add1236).
- Papkou et al. (2023)
Andrei Papkou, Lucia Garcia-Pastor, José Antonio Escudero, and Andreas Wagner.
A rugged yet easily navigable fitness landscape.
*Science*, 382(6673):eadh3860, 2023.
doi: 10.1126/science.adh3860.
URL [https://www.science.org/doi/abs/10.1126/science.adh3860](https://www.science.org/doi/abs/10.1126/science.adh3860).
- Paszke (2019)
A Paszke.
Pytorch: An imperative style, high-performance deep learning library.
*arXiv preprint arXiv:1912.01703*, 2019.
doi: 10.48550/arXiv.1912.01703.
URL [https://doi.org/10.48550/arXiv.1912.01703](https://doi.org/10.48550/arXiv.1912.01703).
- Radivojac & et al. (2013)
Predrag Radivojac and et al.
A large-scale evaluation of computational protein function prediction.
*Nature Methods*, 10(3):221–227, Mar 2013.
ISSN 1548-7105.
doi: 10.1038/nmeth.2340.
URL [https://doi.org/10.1038/nmeth.2340](https://doi.org/10.1038/nmeth.2340).
- Ramesh et al. (2021)
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation.
In *International conference on machine learning*, pp. 8821–8831. Pmlr, 2021.
doi: 10.48550/arXiv.2102.12092.
URL [https://doi.org/10.48550/arXiv.2102.12092](https://doi.org/10.48550/arXiv.2102.12092).
- Rao et al. (2019)
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song.
Evaluating protein transfer learning with tape.
*Advances in neural information processing systems*, 32, 2019.
doi: 10.48550/arXiv.1906.08230.
URL [https://doi.org/10.48550/arXiv.1906.08230](https://doi.org/10.48550/arXiv.1906.08230).
- Rao et al. (2020)
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives.
Transformer protein language models are unsupervised structure learners.
*Biorxiv*, pp. 2020–12, 2020.
doi: 10.1101/2020.12.15.422761.
URL [https://doi.org/10.1101/2020.12.15.422761](https://doi.org/10.1101/2020.12.15.422761).
- Rao et al. (2021)
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives.
Msa transformer.
In *International Conference on Machine Learning*, pp. 8844–8856. PMLR, 2021.
URL [https://proceedings.mlr.press/v139/rao21a.html](https://proceedings.mlr.press/v139/rao21a.html).
- Razavi et al. (2019)
Ali Razavi, Aäron van den Oord, and Oriol Vinyals.
Generating diverse high-fidelity images with VQ-VAE-2.
In *Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada*, pp. 14837–14847, 2019.
URL [https://proceedings.neurips.cc/paper/2019/hash/5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Abstract.html](https://proceedings.neurips.cc/paper/2019/hash/5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Abstract.html).
- Rives et al. (2021)
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.
*Proceedings of the National Academy of Sciences*, 118(15):e2016239118, 2021.
doi: 10.1073/pnas.2016239118.
URL [https://doi.org/10.1073/pnas.2016239118](https://doi.org/10.1073/pnas.2016239118).
- Robin et al. (2021)
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo Tauriello, and Torsten Schwede.
Continuous automated model evaluation (cameo)—perspectives on the future of fully automated evaluation of structure prediction methods.
*Proteins: Structure, Function, and Bioinformatics*, 89(12):1977–1986, 2021.
doi: 10.1002/prot.26213.
URL [https://doi.org/10.1002/prot.26213](https://doi.org/10.1002/prot.26213).
- Ruder (2016)
Sebastian Ruder.
An overview of gradient descent optimization algorithms.
*arXiv preprint arXiv:1609.04747*, 2016.
doi: 10.48550/arXiv.1609.04747.
URL [https://doi.org/10.48550/arXiv.1609.04747](https://doi.org/10.48550/arXiv.1609.04747).
- Salazar et al. (2019)
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff.
Masked language model scoring.
*arXiv preprint arXiv:1910.14659*, 2019.
doi: 10.18653/v1/2020.acl-main.240.
URL [https://doi.org/10.18653/v1/2020.acl-main.240](https://doi.org/10.18653/v1/2020.acl-main.240).
- Samusevich et al. (2025)
Raman Samusevich, Téo Hebra, Roman Bushuiev, Martin Engst, Jonáš Kulhánek, Anton Bushuiev, Joshua D. Smith, Tereza Čalounová, Helena Smrčková, Marina Molineris, Renana Schwartz, Adéla Tajovská, Milana Perković, Ratthachat Chatpatanasiri, Sotirios C. Kampranis, Dan Thomas Major, Josef Sivic, and Tomáš Pluskal.
Structure-enabled enzyme function prediction unveils elusive terpenoid biosynthesis in archaea.
*bioRxiv*, 2025.
doi: 10.1101/2024.01.29.577750.
URL [https://www.biorxiv.org/content/early/2025/04/29/2024.01.29.577750](https://www.biorxiv.org/content/early/2025/04/29/2024.01.29.577750).
- Sapoval et al. (2022)
Nicolae Sapoval, Amirali Aghazadeh, Michael G. Nute, Dinler A. Antunes, Advait Balaji, Richard Baraniuk, C. J. Barberan, Ruth Dannenfelser, Chen Dun, Mohammadamin Edrisi, R. A. Leo Elworth, Bryce Kille, Anastasios Kyrillidis, Luay Nakhleh, Cameron R. Wolfe, Zhi Yan, Vicky Yao, and Todd J. Treangen.
Current progress and open challenges for applying deep learning across the biosciences.
*Nature Communications*, 13(1):1728, Apr 2022.
ISSN 2041-1723.
doi: 10.1038/s41467-022-29268-7.
URL [https://doi.org/10.1038/s41467-022-29268-7](https://doi.org/10.1038/s41467-022-29268-7).
- Schmirler et al. (2024)
Robert Schmirler, Michael Heinzinger, and Burkhard Rost.
Fine-tuning protein language models boosts predictions across diverse tasks.
*Nature Communications*, 15(1):7407, 2024.
doi: 10.1038/s41467-024-51844-2.
URL [https://doi.org/10.1038/s41467-024-51844-2](https://doi.org/10.1038/s41467-024-51844-2).
- Sevgen et al. (2023)
Emre Sevgen, Joshua Moller, Adrian Lange, John Parker, Sean Quigley, Jeff Mayer, Poonam Srivastava, Sitaram Gayatri, David Hosfield, Maria Korshunova, et al.
Prot-vae: protein transformer variational autoencoder for functional protein design.
*bioRxiv*, pp. 2023–01, 2023.
doi: 10.1073/pnas.2408737122.
URL [https://doi.org/10.1073/pnas.2408737122](https://doi.org/10.1073/pnas.2408737122).
- Škrinjar et al. (2025)
Peter Škrinjar, Jérôme Eberhardt, Janani Durairaj, and Torsten Schwede.
Have protein-ligand co-folding methods moved beyond memorisation?
*BioRxiv*, pp. 2025–02, 2025.
doi: 10.1101/2025.02.03.636309.
URL [https://doi.org/10.1101/2025.02.03.636309](https://doi.org/10.1101/2025.02.03.636309).
- Song et al. (2024)
Yidong Song, Qianmu Yuan, Sheng Chen, Yuansong Zeng, Huiying Zhao, and Yuedong Yang.
Accurately predicting enzyme functions through geometric graph learning on esmfold-predicted structures.
*Nature Communications*, 15(1):8180, 2024.
doi: 10.1038/s41467-024-52533-w.
URL [https://doi.org/10.1038/s41467-024-52533-w](https://doi.org/10.1038/s41467-024-52533-w).
- Stärk et al. (2021)
Hannes Stärk, Christian Dallago, Michael Heinzinger, and Burkhard Rost.
Light attention predicts protein location from the language of life.
*Bioinformatics Advances*, 1(1):vbab035, 11 2021.
ISSN 2635-0041.
doi: 10.1093/bioadv/vbab035.
URL [https://doi.org/10.1093/bioadv/vbab035](https://doi.org/10.1093/bioadv/vbab035).
- Su et al. (2023)
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan.
Saprot: Protein language modeling with structure-aware vocabulary.
*bioRxiv*, pp. 2023–10, 2023.
URL [https://openreview.net/forum?id=6MRm3G4NiU](https://openreview.net/forum?id=6MRm3G4NiU).
- Subramaniam & Kleywegt (2022)
Sriram Subramaniam and Gerard J. Kleywegt.
A paradigm shift in structural biology.
*Nature Methods*, 19(1):20–23, Jan 2022.
ISSN 1548-7105.
doi: 10.1038/s41592-021-01361-7.
URL [https://doi.org/10.1038/s41592-021-01361-7](https://doi.org/10.1038/s41592-021-01361-7).
- Sun et al. (2020)
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.
Test-time training with self-supervision for generalization under distribution shifts.
In *International conference on machine learning*, pp. 9229–9248. PMLR, 2020.
URL [https://proceedings.mlr.press/v119/sun20b/sun20b.pdf](https://proceedings.mlr.press/v119/sun20b/sun20b.pdf).
- Tagasovska et al. (2024)
Nataša Tagasovska, Ji Won Park, Matthieu Kirchmeyer, Nathan C Frey, Andrew Martin Watkins, Aya Abdelsalam Ismail, Arian Rokkum Jamasb, Edith Lee, Tyler Bryson, Stephen Ra, et al.
Antibody domainbed: Out-of-distribution generalization in therapeutic protein design.
*arXiv preprint arXiv:2407.21028*, 2024.
doi: 10.48550/arXiv.2407.21028.
URL [https://doi.org/10.48550/arXiv.2407.21028](https://doi.org/10.48550/arXiv.2407.21028).
- Tsuboyama et al. (2023)
Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan J Weinstein, Niall M Mangan, Sergey Ovchinnikov, and Gabriel J Rocklin.
Mega-scale experimental analysis of protein folding stability in biology and design.
*Nature*, 620(7973):434–444, 2023.
doi: 10.1038/s41586-023-06328-6.
URL [https://doi.org/10.1038/s41586-023-06328-6](https://doi.org/10.1038/s41586-023-06328-6).
- Tyers & Mann (2003)
Mike Tyers and Matthias Mann.
From genomics to proteomics.
*Nature*, 422(6928):193–197, Mar 2003.
ISSN 1476-4687.
doi: 10.1038/nature01510.
URL [https://doi.org/10.1038/nature01510](https://doi.org/10.1038/nature01510).
- van Kempen et al. (2022)
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger.
Foldseek: fast and accurate protein structure search.
*Biorxiv*, pp. 2022–02, 2022.
doi: 10.1038/s41587-023-01773-0.
URL [https://doi.org/10.1038/s41587-023-01773-0](https://doi.org/10.1038/s41587-023-01773-0).
- Varadi et al. (2022)
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al.
Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models.
*Nucleic acids research*, 50(D1):D439–D444, 2022.
doi: 10.1093/nar/gkab1061.
URL [https://doi.org/10.1093/nar/gkab1061](https://doi.org/10.1093/nar/gkab1061).
- Vaswani (2017)
A Vaswani.
Attention is all you need.
*Advances in Neural Information Processing Systems*, 2017.
doi: 10.48550/arXiv.1706.03762.
URL [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).
- Wang et al. (2023)
Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang.
Test-time training on video streams.
*arXiv preprint arXiv:2307.05014*, 2023.
doi: 10.48550/arXiv.2307.05014.
URL [https://doi.org/10.48550/arXiv.2307.05014](https://doi.org/10.48550/arXiv.2307.05014).
- Watson et al. (2023)
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al.
De novo design of protein structure and function with rfdiffusion.
*Nature*, 620(7976):1089–1100, 2023.
doi: 10.1038/s41586-023-06415-8.
URL [https://doi.org/10.1038/s41586-023-06415-8](https://doi.org/10.1038/s41586-023-06415-8).
- Xiao et al. (2022)
Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek.
Learning to generalize across domains on single test samples.
*arXiv preprint arXiv:2202.08045*, 2022.
doi: 10.48550/arXiv.2202.08045.
URL [https://doi.org/10.48550/arXiv.2202.08045](https://doi.org/10.48550/arXiv.2202.08045).
- Yu et al. (2023)
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao.
Enzyme function prediction using contrastive learning.
*Science*, 379(6639):1358–1363, 2023.
doi: 10.1126/science.adf2465.
URL [https://www.science.org/doi/abs/10.1126/science.adf2465](https://www.science.org/doi/abs/10.1126/science.adf2465).
- Zhang & Skolnick (2004)
Yang Zhang and Jeffrey Skolnick.
Scoring function for automated assessment of protein structure template quality.
*Proteins: Structure, Function, and Bioinformatics*, 57(4):702–710, 2004.
doi: 10.1002/prot.20264.
URL [https://doi.org/10.1002/prot.20264](https://doi.org/10.1002/prot.20264).
- Zhao et al. (2023)
Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin.
On pitfalls of test-time adaptation.
In *International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA*, volume 202 of *Proceedings of Machine Learning Research*, pp. 42058–42080. PMLR, 2023.
URL [https://proceedings.mlr.press/v202/zhao23d.html](https://proceedings.mlr.press/v202/zhao23d.html).

## Appendix

## Contents

appendix.Aappendix.Bappendix.Cappendix.Dappendix.Esubsection.E.1subsubsection.E.1.1subsubsection.E.1.2subsubsection.E.1.3subsection.E.2subsubsection.E.2.1subsubsection.E.2.2subsubsection.E.2.3subsection.E.3subsubsection.E.3.1subsubsection.E.3.2subsubsection.E.3.3appendix.Fsubsection.F.1subsection.F.2appendix.Gsubsection.G.1subsection.G.2subsection.G.3

## Appendix A Justification of customization via perplexity minimization

Figure: Figure A1: Quality of protein structure prediction, as measured by TM-score, correlates with perplexity of the underlying language model on the challenging targets from the CAMEO validation set. Higher TM-scores are associated with lower perplexity, indicating that better predictions are linked to lower uncertainty in the language model’s understanding of the protein sequence.
Refer to caption: x7.png

While the paradigm of test-time customization has been investigated in other domains, the reasons behind its surprising effectiveness are not completely clear (Liu et al., 2021; Zhao et al., 2023). Here, we offer a potential justification for the effectiveness of ProteinTTT by linking it to perplexity minimization.

Perplexity has traditionally been used in natural language processing to evaluate how well models comprehend sentences (Brown, 2020; Chelba et al., 2013). Protein language modeling has adopted this metric to assess how effectively models “understand” amino acid sequences (Hayes et al., 2024; Lin et al., 2023). For bidirectional, random masking language models, which are the focus of this study, we consider the following definition of perplexity(^2^22Please note that this is an approximation of perplexity, which is computationally intractable for bidirectional models, and is often referred to as pseudo-perplexity (Lin et al., 2023; Salazar et al., 2019).):

$$ $\text{Perplexity}(x)=\exp\Biggl(\frac{1}{|x|}\sum_{i=1}^{|x|}{-\log{p(x_{i}|x_{\setminus i};\theta)}}\Biggl),$ (3) $$

where $|x|$ is the length of the input protein sequence $x$ and $p(x_{i}|x_{\setminus i};\theta)$ represents the probability that the model correctly predicts the token $x_{i}$ at position $i$ when it is masked on the input $x_{\setminus i}$. Perplexity ranges from 1 to infinity (the lower, the better), providing an intuitive measure of how well a model fits, on average, tokens in a given sequence. A perplexity value of 1 indicates that the model perfectly fits the sequence, accurately predicting all the true tokens.

Several studies have shown that lower perplexity on held-out protein sequences (calculated through the self-supervised track $g\circ f$) correlates with better performance on downstream tasks (via the supervised track $h\circ f$), such as predicting protein contacts (Rao et al., 2020), structure (Lin et al., 2023), or fitness (Kantroo et al., 2024). To give an example, we analyze the correlation between perplexity and structure prediction quality ([Figure˜A1](https://arxiv.org/html/2411.02109v2#A1.F1); see [Section˜4.1](https://arxiv.org/html/2411.02109v2#S4.SS1) for experimental details). A notable correlation suggests that reducing a model’s perplexity on a single target sample $x$ (applied independently to all test samples) can lead to improved predictions on the downstream task ([Figure˜3](https://arxiv.org/html/2411.02109v2#S3.F3); [Figure˜A10](https://arxiv.org/html/2411.02109v2#A7.F10)).

Since we assume only a single target example $x$, the minimization of the masked language modeling loss $\mathcal{L}(x;\theta)$ ([Equation˜2](https://arxiv.org/html/2411.02109v2#S3.E2)) on this example is directly linked to minimizing the perplexity Perplexity$(x)$ ([Equation˜3](https://arxiv.org/html/2411.02109v2#A1.E3)). For instance, in the case of a single masked position (i.e., $|M|=1$), the loss is equal to the logarithm of perplexity. More generally, it can be shown formally that by minimizing the masked language modeling objective, the model learns to approximate the conditional marginals of the language (of proteins), including the leave-one-out probabilities evaluated in perplexity (Hennigen & Kim, 2023). As a result, applying self-supervised test-time customization on $x$ through $g\circ f$ enhances the representation of the target protein in the backbone $f$, leading to improved downstream performance via the fine-tuning track $h\circ f$.

## Appendix B Customization with multiple sequence alignment (MSA)

##### Customization training objective.

Since many target proteins may not have homologous sequences (Rao et al., 2021) and finding such homologs may be time-consuming (Lin et al., 2023), the ProteinTTT customization objective ([Equation˜2](https://arxiv.org/html/2411.02109v2#S3.E2)) only assumes a single target sequence for customization. However, we also extend the loss function to the case when a multiple sequence alignment (MSA) is available:

$$ $\displaystyle\mathcal{L}_{\text{MSA}}(x;\theta)=\mathbb{E}_{x^{\prime}\sim p_{\text{MSA}}(x^{\prime}|x)}\big[\mathcal{L}(x^{\prime};\theta)\big],$ (4) $$

where $p_{\text{MSA}}(x^{\prime}|x)$ is the distribution of sequences $x^{\prime}$ homologous to the target protein $x$, $\mathcal{L}$ is the single-sequence loss function defined in [Equation˜2](https://arxiv.org/html/2411.02109v2#S3.E2), and $\theta$ denotes the tunable parameters of the model backbone $f$. We refer to customization using [Equation˜4](https://arxiv.org/html/2411.02109v2#A2.E4) as ProteinTTT${}_{\text{MSA}}$.

##### Results for fitness prediction.

**Table A1: ProteinTTT can be used with MSA when available. Please see [Table˜4](https://arxiv.org/html/2411.02109v2#S4.T4) for evaluation details.**
| Method | Avg. Spearman $\uparrow$ |
| --- | --- |
| ESM2 (Lin et al., 2023) | 0.4139 |
| ESM2 + ProteinTTT${}_{\text{MSA}}$ (Ours) | 0.4299 ± 0.00099 |
| MSA Transformer (Rao et al., 2021) | 0.4319 |
| MSA Transformer + ProteinTTT (Ours) | 0.4326 ± 0.00003 |

It is known that evolutionary information is important for protein fitness prediction (Laine et al., 2019). Therefore, we demonstrate how ProteinTTT${}_{\text{MSA}}$ and ProteinTTT can enhance the performance of PLMs on the ProteinGym benchmark (Notin et al., 2024). [Table˜A1](https://arxiv.org/html/2411.02109v2#A2.T1) shows that using ProteinTTT${}_{\text{MSA}}$ with high-quality MSAs curated by Notin et al. (2024) strongly enhances the performance of ESM2, approaching that of MSA Transformer, pre-trained on MSAs. Moreover, we find that MSA Transformer slightly benefits from single-sequence customization with ProteinTTT, while customization to whole or subsampled MSAs disrupts the performance ([Table˜A3](https://arxiv.org/html/2411.02109v2#A5.T3) in [Section˜G.2](https://arxiv.org/html/2411.02109v2#A7.SS2)).

## Appendix C Customization for protein function prediction

Protein function prediction is essential for understanding biological processes and guiding bioengineering, but is challenging due to its vague definition and limited data (Yu et al., 2023; Radivojac & et al., 2013; Stärk et al., 2021; Mikhael et al., 2024; Samusevich et al., 2025). While improved structure prediction with ProteinTTT ([Section˜4.1](https://arxiv.org/html/2411.02109v2#S4.SS1)) can already enhance function prediction (Song et al., 2024), we also evaluate our customization method directly on two function classification tasks: subcellular localization, predicting protein location within a cell (Stärk et al., 2021), and substrate classification for terpene synthases (TPS), enzymes producing the largest class of natural products (Christianson, 2017; Samusevich et al., 2025). Using ProteinTTT with EnzymeExplorer (Samusevich et al., 2025) for TPS detection and Light attention (Stärk et al., 2021) for subcellular localization, we achieve consistent performance gains.

##### Evaluation setup.

Figure: Figure A2: Customization with ProteinTTT enables the correct substrate classification for a terpene synthase (TPS) enzyme. With progressive customization steps of EnzymeExplorer + ProteinTTT, the probability of the initially misclassified substrate (red) decreases, while the probability of the true substrates (green) increases. The bar plots also display the predicted probabilities for other substrates with non-zero values (grey).
Refer to caption: x8.png

For the terpene substrate classification, we use the largest available dataset of characterized TPS from Samusevich et al. (2025) and reuse the original cross-validation schema. In the case of protein localization prediction, we use a standard DeepLoc dataset (Almagro Armenteros et al., 2017) as a validation set and setHard from Stärk et al. (2021) as the test set.

Given a protein, the goal of function prediction is to correctly classify it into one of the predefined functional annotations. We assess the quality of the TPS substrate prediction using standard multi-label classification metrics used in the EnzymeExplorer paper (Samusevich et al., 2025): mean average precision (mAP) and area under the receiver operating characteristic curve (AUROC). In the case of protein localization prediction, we similarly use the classification metrics from the original paper (Stärk et al., 2021): accuracy, multi-class Matthews correlation coefficient (MCC), and F1-score.

##### Results.

**Table A2: Customization with ProteinTTT improves protein function prediction. For the terpene syntase (TPS) substrate classification task, the metrics are computed on the 512 TPS sequences based on the cross-validation schema of the TPS dataset (Samusevich et al., 2025). Subcellular localization prediction performance is reported for 432 protein sequences from the setHard test set (Stärk et al., 2021). The error bars show standard deviations across five random seeds.**
| Method | mAP $\uparrow$ | AUROC $\uparrow$ |
| --- | --- | --- |
| EnzymeExplorer<br>(Samusevich et al., 2025) | 0.805 | 0.948 |
| EnzymeExplorer + ProteinTTT<br>(Ours) | 0.811 ± 0.0011 | 0.950 ± 0.0002 |

Customization with ProteinTTT improves model performance on both of the protein function prediction tasks and across all considered metrics ([Table˜A2](https://arxiv.org/html/2411.02109v2#A3.T2)). [Figure˜A2](https://arxiv.org/html/2411.02109v2#A3.F2) provides a qualitative result, where customization with ProteinTTT iteratively refines the prediction of EnzymeExplorer toward a correct TPS substrate class. We hypothesize that improvement with customization is more challenging in classification tasks, as opposed to regression problems, because a larger change in the latent space is required to shift the top-class probability.

## Appendix D Implementation details

##### Infrastructure.

All experiments with ProteinTTT are conducted on machines equipped with a single NVIDIA A100 40GB GPU, an 8-core AMD processor, and 128 GB of physical memory.

##### Source code.

We provide a user-friendly and easily extensible PyTorch (Paszke, 2019) implementation of ProteinTTT, available as the proteinttt Python package
(^3^33[https://github.com/anton-bushuiev/ProteinTTT](https://github.com/anton-bushuiev/ProteinTTT))
. We provide two Python code snippets [˜1](https://arxiv.org/html/2411.02109v2#Code1) and [˜2](https://arxiv.org/html/2411.02109v2#Code2) to demonstrate the implementation of inference and customization with ProteinTTT, respectively. [˜1](https://arxiv.org/html/2411.02109v2#Code1) demonstrates how inference with ESMFold can be enhanced with ProteinTTT by adding just a few lines of code to enable customization. Next, [˜2](https://arxiv.org/html/2411.02109v2#Code2) shows how ProteinTTT can be easily implemented for a PLM of interest by inheriting from the abstract TTTModule class. To integrate ProteinTTT within a model (e.g., ESM2), the user needs to implement methods that define the model’s vocabulary, an interface for predicting logits, and a specification of which modules need to be fine-tuned or remain frozen. The rest, i.e., the test-time training logic itself, is implemented within the unified TTTModule class.

Figure: Listing 1: Incorporation of ProteinTTT into an ESMFold structure prediction pipeline using the proteinttt package.

##### Optimization.

We minimize the loss defined in [Equation˜2](https://arxiv.org/html/2411.02109v2#S3.E2) using stochastic gradient descent (SGD) with zero momentum and zero weight decay (Ruder, 2016). While a more straightforward option might be to use the optimizer state from the final pre-training step, this approach is often impractical because the optimizer parameters are usually not provided with the pre-trained model (Hayes et al., 2024; Lin et al., 2023). Moreover, many models are pre-trained using the Adam optimizer (Kingma & Ba, 2015) or its variants (Loshchilov & Hutter, 2019). However, it was shown that Adam results in less predictable behavior of test-time training compared to the SGD optimizer, possibly due to its more exploratory behavior (Gandelsman et al., 2022).

##### Customizing large models.

We aim for customization to be applicable on the fly, i.e., without the need for any pre-computation and on a single GPU with a minimum computational overhead. Since state-of-the-art models for many protein-oriented tasks are typically large, with up to billions of parameters, our aim presents two key challenges. First, when using pre-trained Transformers on a single GPU, even for the forward pass, the batch size is typically limited to only several samples due to the quadratic complexity of the inference (Vaswani, 2017). Second, for the backward pass, even a batch size of one is not always feasible for large models. To address the first challenge, we perform forward and backward passes through a small number of training examples and accumulate gradients to simulate updates with any batch size. We address the second challenge by employing low-rank adaptation (LoRA; Hu et al. (2021)), which in practice enables fine-tuning of any model for which a forward pass on a single sample is feasible, due to a low number of trainable parameters. [Section˜G.3](https://arxiv.org/html/2411.02109v2#A7.SS3) details how ESMFold (Lin et al., 2023), with its 3B-parameter ESM2 backbone $f$, can be efficiently customized, retaining its speed advantage while enhancing performance.

Figure: Listing 2: Implementation of ESM2 + ProteinTTT within the proteinttt package.

## Appendix E Experimental details

In this section, we describe the proposed benchmark suite for the three customization tasks considered in this work: protein structure prediction ([Section˜E.1](https://arxiv.org/html/2411.02109v2#A5.SS1)), protein fitness prediction ([Section˜E.2](https://arxiv.org/html/2411.02109v2#A5.SS2)), and protein function prediction ([Section˜E.3](https://arxiv.org/html/2411.02109v2#A5.SS3)). Each subsection describes the application of ProteinTTT to the respective models, along with details on the data, metrics, and models. [Table˜A3](https://arxiv.org/html/2411.02109v2#A5.T3) additionally summarizes the hyperparameters used for the application of ProteinTTT to individual models.

### E.1 Protein structure prediction

#### E.1.1 Datasets

##### CAMEO dataset.

To evaluate the capabilities of ProteinTTT on protein structure prediction, we employ the CAMEO validation and test sets as described in Lin et al. (2023). Specifically, the validation set was obtained by querying the CAMEO (Continuous Automated Model Evaluation) web server(^4^44[https://www.cameo3d.org/modeling](https://www.cameo3d.org/modeling)) (Robin et al., 2021) for entries between August 2021 and January 2022, while the CAMEO test set consists of entries from April 1, 2022, to June 25, 2022. Most of the entries in the CAMEO sets are predicted with high accuracy and confidence (Lin et al., 2023). Therefore, we subselect the challenging validation and test sets where customization with ProteinTTT is suitable.

Specifically, we apply two standard criteria: (1) preserving entries with ESMFold pLDDT scores below 70 to filter out high-confidence predictions (Jumper et al., 2021), and (2) selecting entries with ESM2 perplexity scores greater than or equal to 6, ensuring that the predictions are challenging due to poor sequence understanding rather than other factors. Additionally, most structures with perplexity scores below 6 are already associated with high-confidence predictions (Figure S5 in Lin et al. (2023)). After filtering, the resulting challenging validation and test sets consist of 27 (out of 378) and 18 (out of 194) targets, respectively.

#### E.1.2 Metrics

To assess the quality of the predicted protein structures with respect to the ground truth structures, we use two standard metrics averaged across the test dataset: TM-score (Zhang & Skolnick, 2004) and LDDT (Mariani et al., 2013).

##### TM-score.

The TM-score (Template Modeling score) is a metric used to assess the quality of the global 3D alignment between the predicted and target protein structures. It evaluates the structural similarity by comparing the distance between corresponding residues after superposition. The TM-score ranges from 0 to 1, where higher values indicate better alignment.

##### LDDT.

The Local Distance Difference Test (LDDT) is an alignment-free metric used to assess the accuracy of predicted protein structures. Unlike global metrics, LDDT focuses on local structural differences by measuring the deviation in distances between atom pairs in the predicted structure compared to the target structure. It is particularly useful for evaluating the accuracy of local regions, such as secondary structure elements. LDDT scores range from 0 to 100, with higher values indicating better local structural agreement.

#### E.1.3 Models

##### ESMFold.

The ESMFold architecture comprises two key components: a protein language model, ESM2, which, given a protein sequence, generates embeddings for individual amino acids, and a folding block that, using these embeddings and the sequence, predicts the protein 3D structure along with per-amino-acid confidence scores, known as pLDDT scores. In our experiments, we use the esmfold_v0 model from the publicly available ESMFold checkpoints(^5^55[https://github.com/facebookresearch/esm/blob/main/esm/esmfold/v1/pretrained.py](https://github.com/facebookresearch/esm/blob/main/esm/esmfold/v1/pretrained.py)). Please note that we use esmfold_v0 and not esmfold_v1 to avoid data leakage with respect to the CAMEO test set.

##### ESMFold + ProteinTTT.

Since the ESM2 backbone of ESMFold was pre-trained in a self-supervised masked modeling regime, the application of ProteinTTT to ESMFold is straightforward. We treat ESM2 as the backbone $f$, the language modeling head predicting amino acid classes from their embeddings as the self-supervised head $g$, and the folding trunk along with the structure modules as the downstream task head $h$. After each ProteinTTT step, we run $h\circ f$ to compute the pLDDT scores, which allows us to estimate the optimal number of customization steps for each protein based on the highest pLDDT score.

Since the backbone $f$ is given by the ESM2 model containing 3 billion parameters, we apply LoRA (Hu et al., 2021) to all matrices involved in self-attention. This enables fine-tuning ESMFold + ProteinTTT on a single GPU.

##### ESMFold + ME.

Since ESMFold is a regression model, it only predicts one solution and does not have a straightforward mechanism for sampling multiple structure predictions. Nevertheless, the authors of ESMFold propose a way to sample multiple candidates (Section A.3.2 in Lin et al. (2023)). To sample more predictions, the masking prediction (ME) method randomly masks 15% (same ratio as during masked language modeling pre-training) of the amino acids before passing them to the language model. Selecting the solution with the highest pLDDT may lead to improved predicted structure. Since sampling multiple solutions with ESMFold + ME and selecting the best one via pLDDT is analogous to ESMFold + ProteinTTT, we employ the former as a baseline, running the method for the same number of steps.

##### ESM3.

Unlike ESMFold, ESM3 is a fully multiple-track, BERT-like model (Devlin, 2018), pre-trained to unmask both protein sequence and structure tokens simultaneously (along with the function tokens). The structure tokens in ESM3 are generated via a separately pre-trained VQ-VAE (Razavi et al., 2019) operating on the protein geometry. In our experiments, we use the smallest, publicly available version of the ESM3 model (ESM3_sm_open_v0)(^6^66[https://github.com/evolutionaryscale/esm](https://github.com/evolutionaryscale/esm)).

##### ESM3 + ProteinTTT.

We treat the Transformer encoder of ESM3 as $f$, the language modeling head decoding amino acid classes as $g$, and the VQ-VAE decoder, which maps structure tokens to the 3D protein structure, as $h$. During the customization steps, we train the model to unmask a protein sequence while keeping the structural track fully padded. During the inference, we provide the model with a protein sequence and run it to unmask the structural tokens, which are subsequently decoded with the VQ-VAE decoder. After each customization step, we run $h\circ f$ to compute the pLDDT scores, which allows us to estimate the optimal number of customization steps for each protein based on the highest pLDDT score. We choose the optimal hyperparameters by maximizing the difference in TM-score after and before applying ProteinTTT across the validation dataset.

Despite the fact that the model contains 1.4 billion parameters, even without using LoRA, ESM3 + ProteinTTT can be fine-tuned on a single NVIDIA A100 GPU. Therefore, we do not employ LoRA for fine-tuning ESM3, while this can also be possible.

##### ESM3 + CoT.

To improve the generalization and protein-specific performance of ESM3, the original ESM3 paper employs a chain of thought (CoT) procedure. The procedure unfolds in $n$ steps as follows. At each step, $1/n$ of the masked tokens with the lowest entropy after softmax on logits are unmasked. Then, the partially unmasked sequence is fed back into the model, and the process repeats until the entire sequence is unmasked. In our experiments, we set $n=8$, which is the default value provided in the official GitHub repository.

##### HelixFold-Single.

HelixFold-Single is an MSA-free protein structure prediction model that combines representations from a pretrained protein language model with adapted AlphaFold2 geometric modules (EvoformerS and Structure) to directly predict atomic coordinates (Fang et al., 2023). We use the official implementation(^7^77[https://github.com/PaddlePaddle/PaddleHelix/tree/dev/apps/protein_folding/helixfold-single](https://github.com/PaddlePaddle/PaddleHelix/tree/dev/apps/protein_folding/helixfold-single))

##### HelixFold-Single + ProteinTTT.

HelixFold-Single shares the main concept with ESMFold, and we combine it with ProteinTTT in the same way as in ESMFold + ProteinTTT.

### E.2 Protein fitness prediction

#### E.2.1 Datasets

##### ProteinGym.

ProteinGym(^8^88[https://github.com/OATML-Markslab/ProteinGym](https://github.com/OATML-Markslab/ProteinGym)) is the standard benchmark for protein fitness prediction (Notin et al., 2024). The latest, second version of the dataset includes 217 deep mutation scanning experiments (DMSs) across different proteins. We focus on the well-established zero-shot setup of the benchmark and do not experiment with the supervised setup, as it has not yet been fully incorporated into the official codebase at the time of this study. In total, the dataset contains 2.5M mutants with annotated ground-truth fitness. Since ProteinGym does not contain a data split for the zero-shot setup, employed in this work, we use the whole dataset as the test set.

##### MaveDB dataset.

Figure: Figure A3: Comparison of the standard ProteinGym dataset with the MaveDB dataset constructed in this work. A) MaveDB, mined from Esposito et al. (2019), includes novel assays even after filtering to ensure distinct proteins from the comprehensive ProteinGym dataset. This is largely because most MaveDB assays post-filtering date to 2024, whereas the latest assays in ProteinGym date to 2023. B, C, D) MaveDB is of sufficient quality for model evaluation. Representative baselines, ESM2 and SaProt with both 35 million and 650 million parameters, evaluated on ProteinGym generalize effectively to MaveDB, following a similar distribution of predictions. Panel D illustrates the random subset of 50 proteins used for hyperparameter tuning for fitness prediction. Each point in the plots represents one protein and shows the Spearman correlation averaged across all assays corresponding to the protein (typically one assay per protein). The box plots standardly depict quartiles, medians, and outliers.
Refer to caption: x9.png

To establish a validation set disjoint from ProteinGym (Notin et al., 2024), we mined MaveDB(^9^99[https://www.mavedb.org](https://www.mavedb.org)) (Esposito et al., 2019). As of August 1, 2024, the database contains 1178 Multiplexed Assays of Variant Effects (MAVEs), where each assay corresponds to a single protein, measuring the experimental fitness of its variants. We applied quality control filters to remove potentially noisy data. Specifically, we ensured that the UniProt identifier (Consortium, 2023) is valid and has a predicted structure available in the AlphaFold DB (Varadi et al., 2022). We also excluded assays with fewer than 100 variants, as well as those where at least one mutation had a wrongly annotated wild type or where most mutations failed during parsing. Additionally, to ensure no overlap between datasets, we removed any assays whose UniProt identifier matched with those in ProteinGym, ensuring that the validation and test sets contain different proteins.

The described methodology resulted in the MaveDB dataset comprising 676 assays (out of 1178 in the entire MaveDB) with experimental fitness annotations. This corresponds to 483 unique protein sequences and 867 thousand mutations in total. The large size of the dataset, despite the comprehensiveness of ProteinGym containing 217 assays, can be attributed to the fact that many assays in MaveDB were released after the ProteinGym construction ([Figure˜A3](https://arxiv.org/html/2411.02109v2#A5.F3)A). To ensure the quality of the constructed MaveDB dataset, we validated that representative baselines from ProteinGym generalize to the new assays, following similar distributions of predictions ([Figure˜A3](https://arxiv.org/html/2411.02109v2#A5.F3)B,C). Finally, for efficiently tuning hyperparameters for fitness prediction models, we sampled 50 proteins ([Figure˜A3](https://arxiv.org/html/2411.02109v2#A5.F3)D), corresponding to 83 assays comprising 134 thousand variants.

#### E.2.2 Metrics

Protein fitness labels are not standardized and can vary across different proteins. Nevertheless, the ranking of mutations for a single protein, as defined by fitness labels, can be used to assess the mutation scoring capabilities of machine learning models. As a result, Spearman correlation is a standard metric for evaluation.

##### Spearman by phenotype.

When computing Spearman correlations, we follow the evaluation protocol proposed in ProteinGym (Notin et al., 2024). First, for each protein, we compute Spearman correlation scores between the predicted ranks of mutations and their corresponding labels. Then, we average the scores across five categories of assayed phenotypes, measuring the effects of mutations: catalytic activity (“Activity”), binding affinity to a target (“Binding”), protein expression levels in a cell (“Expression”), organism growth rate (“Organismal Fitness”), and protein thermostability (“Stability”).

##### Avg. Spearman.

We refer to the mean score across the five phenotype categories as “Avg. Spearman”. We report the “Avg. Spearman” metric as the mean and standard deviation across five random seeds ([Table˜4](https://arxiv.org/html/2411.02109v2#S4.T4), [Table˜A4](https://arxiv.org/html/2411.02109v2#A7.T4)).

##### Spearman by MSA Depth.

Following (Notin et al., 2024), we split the performance by the depth of available multiple sequence alignment (MSA), i.e., the number of homologous sequences available, as provided in ProteinGym: “Low depth”, “Medium depth”, and “High depth”, and report the Spearman correlation for each subset individually ([Table˜A4](https://arxiv.org/html/2411.02109v2#A7.T4)). Specifically, the MSA depth categories in ProteinGym are determined using the following thresholds from Hopf et al. (2017): “Low” is defined as $N_{eff}/L<1$, “Medium” as $1<N_{eff}/L<100$, and “High” as $N_{eff}/L>100$, where $N_{eff}$ represents the normalized number of effective sequences in the MSA, and $L$ is the sequence length covered in the MSA.

#### E.2.3 Models

##### ESM2.

The ESM2 model is a bidirectional, BERT-like (Devlin, 2018) Transformer trained on millions of protein sequences using masked modeling (Lin et al., 2023). The goal of protein fitness prediction is to predict the effects of mutations, and PLMs are often adapted to this task using zero-shot transfer via log odds ratio (Notin et al., 2024; Meier et al., 2021). Specifically, for a given single- or multi-point mutation, where certain amino acids $T$ are substituted from $x_{i}$ to $x^{m}_{i}$ for each $i\in T$, the fitness prediction via the log odds ratio is defined as:

$$ $\displaystyle\sum_{i\in T}{\Bigl(\log{p(x^{m}_{i}|x_{\setminus i})}-\log{p(x_{i}|x_{\setminus i})}\Bigr)},$ (5) $$

where the sum iterates over mutated positions $i\in T$ with $p(x^{m}_{i}|x_{\setminus i})$ and $p(x_{i}|x_{\setminus i})$ denoting the predicted probabilities of the mutated amino acid and the original one (i.e., wild type), respectively. The conditionals $x_{\setminus i}$ indicate that the input sequence to the model has the position $i$ masked. In this setup, the native (unmutated) sequence, where $T=\emptyset$, has a predicted fitness of 0. Mutations with negative values represent favorable mutations, while positive values correspond to disruptive mutations. We follow the ProteinGym benchmark and use this formula (Notin et al., 2024) to evaluate the fitness prediction capabilities of ESM2. We use the implementation of ESM2 from ProteinGym.

##### ESM2 + ProteinTTT.

ESM2 can be straightforwardly customized with ProteinTTT. Specifically, we treat the Transformer encoder as the backbone $f$, and the language modeling head, which projects token embeddings to amino acid probabilities, as the pre-training head $g$. The log odds ratio given by [Equation˜5](https://arxiv.org/html/2411.02109v2#A5.E5) serves as the task-specific head $h$, which in this case involves the pre-training head $g$ that predicts log probabilities. Overall, we apply ProteinTTT to the pre-trained ESM2 model and, after a pre-defined number of self-supervised fine-tuning steps, score mutations using [Equation˜5](https://arxiv.org/html/2411.02109v2#A5.E5). During customization, we fine-tune all parameters in $g\circ f$ end-to-end except for token and position embeddings. When evaluating ESM2 + ProteinTTT${}_{\text{MSA}}$, we use the MSAs curated by the authors of ProteinGym (Notin et al., 2024).

##### SaProt.

We also experiment with a structure-aware protein language model, SaProt (Su et al., 2023). SaProt builds off the ESM2 model but incorporates structural information from predicted protein structures. Specifically, SaProt uses the same Transformer architecture but expands its vocabulary by combining the 20 standard amino acid tokens with 20 structural tokens from the 3Di vocabulary, increasing the total alphabet size to 400. The 3Di tokens capture the geometry of the protein backbone and are generated using VQ-VAE (Razavi et al., 2019), which projects continuous geometric information into discrete tokens and was trained as part of the Foldseek method (van Kempen et al., 2022).

Since SaProt is also a protein language model, it also uses [Equation˜5](https://arxiv.org/html/2411.02109v2#A5.E5) to score variants. However, please note that SaProt, as implemented in ProteinGym (Notin et al., 2024), uses a slightly different version of the log odds ratio. In SaProt, the conditions in the log probabilities in [Equation˜5](https://arxiv.org/html/2411.02109v2#A5.E5) are replaced with $x_{\setminus T}$ instead of $x_{\setminus i}$, not assuming the independence of substitutions. During customization with ProteinTTT, we only mask sequential information and leave the structural part of the tokens unchanged, reflecting the original pre-training setup. We use the implementation of SaProt from ProteinGym^[8](https://arxiv.org/html/2411.02109v2#footnote8).

##### SaProt + ProteinTTT.

Since the architecture of SaProt is based on ESM2, the ProteinTTT components $f$, $g$, and $h$ remain the same. It means that customization can be applied to the model in the same way as in the case of ESM2 + ProteinTTT discussed above.

##### ProSST.

Finally, we experiment with the state-of-the-art fitness predictor, ProSST (Li et al., 2024). ProSST primarily improves upon SaProt (Su et al., 2023) by incorporating a larger vocabulary of structural tokens and employing disentangled attention mechanisms. Instead of relying on the 3Di alphabet optimized for protein structure search with Foldseek (van Kempen et al., 2022), Li et al. (2024) pre-train a new autoencoder to denoise corrupted protein backbones and cluster the resulting latent space using the $K$-means algorithm (Lloyd, 1982). Notably, optimal performance for fitness prediction is achieved with $K=2048$ tokens, compared to just 20 in the 3Di vocabulary used by SaProt. We adopt this model in our experiments. Additionally, disentangled attention in ProSST enhances information propagation between sequence and structure within its Transformer blocks, further improving prediction performance. The model has 110M parameters in total.

ProSST, similarly to ESM2 and SaProt, is pre-trained using masked language modeling applied to protein sequence tokens. To score mutations on the ProteinGym benchmark (Notin et al., 2024), ProSST also uses the log-odds ratio, but in a slightly different way compared to ESM2 and SaProt. Specifically, ProSST performs a single forward pass to predict log probabilities, which are then used to score all mutations. Formally, this approach modifies the log probability condition in [Equation˜5](https://arxiv.org/html/2411.02109v2#A5.E5), replacing $x_{\setminus i}$ with $x$.

##### ProSST + ProteinTTT.

Similarly to ESM2 and SaProt, we treat the Transformer encoder in ProSST as the backbone $f$, the masked language modeling head as the pre-training head $g$, and the log-odds ratio formula as the task-specific head $h$.

##### MSA Transformer.

Finally, we experiment with MSA Transformer for fitness prediction (Rao et al., 2021). Similar to ESM2 (Lin et al., 2023), MSA Transformer is pre-trained on large protein sequence datasets; however, it is trained on multiple sequence alignments (MSAs) rather than individual sequences.

Since MSA Transformer is also a protein language model, it can be used for fitness prediction in the same way as ESM2, as discussed above, by computing the log-odds ratio over the first sequence in the MSA in this case. We reproduce the results of MSA Transformer on the ProteinGym benchmark with two modifications: (1) we sample a weighted subset of 32 sequences from each MSA instead of 400, and (2) we use only one random seed instead of five for ensembling. These changes significantly reduce computational time while also slightly improving performance compared to the results reported in ProteinGym. This improvement may be explained by the fact that the performance of MSA Transformer saturates with increasing MSA input size (Figure 4 in Rao et al. (2021)).

##### MSA Transformer + ProteinTTT.

We experiment with customizing MSA Transformer to MSA subsamples of varying sizes, ranging from a single target sequence (i.e., customization via [Equation˜2](https://arxiv.org/html/2411.02109v2#S3.E2) with ProteinTTT) to the full MSA subset of 32 sequences (i.e., customization via [Equation˜4](https://arxiv.org/html/2411.02109v2#A2.E4) with ProteinTTT${}_{\text{MSA}}$). We observe that applying ProteinTTT${}_{\text{MSA}}$ to MSA Transformer with a batch size of 32 disrupts performance, while reducing the input MSA subsample size mitigates this effect. Ultimately, MSA Transformer + ProteinTTT results in a slight performance improvement.

### E.3 Protein function prediction

#### E.3.1 Datasets

##### TPS dataset.

For the evaluation of terpene substrate classification, we use the largest available dataset of characterized TPS enzymes from Samusevich et al. (2025) and repurpose the original 5-fold cross-validation schema. We focus on the most challenging TPS sequences, defined as those predicted by the TPS detector, proposed by the dataset authors, with confidence scores below 0.8. This filtering results in 104, 98, 113, 100, 97 examples in the individual folds.

##### setHard.

For the test evaluation of subcellular location prediction, we use the setHard dataset constructed by Stärk et al. (2021). The dataset was redundancy-reduced, both within itself and relative to all proteins in DeepLoc (Almagro Armenteros et al. (2017); next paragraph), a standard dataset used for training and validating machine learning models. The setHard dataset contains 490 protein sequences, each annotated with one of ten subcellular location classes, such as “Cytoplasm” or “Nucleus”. Since we use ESM-1b (Rives et al., 2021) in our experiments with the dataset, we further filter the data to 432 sequences that do not exceed a length of 1022 amino acids. This step, consistent with Stärk et al. (2021), ensures that ESM-1b can generate embeddings for all proteins.

##### DeepLoc.

For hyperparameter tuning in the subcellular location prediction task, we use the test set from the DeepLoc dataset (Almagro Armenteros et al., 2017). Similar to setHard, DeepLoc assigns labels from one of ten subcellular location classes. The dataset contains 2768 proteins, which we further filter to 2457 sequences that do not exceed a length of 1022 amino acids, ensuring compatibility with the embedding capabilities of ESM-1b. Since setHard was constructed to be independent of DeepLoc, setHard provides a leakage-free source of data for validation.

#### E.3.2 Metrics

##### mAP, AUROC.

The TPS substrate prediction problem is a 12-class multi-label classification task over possible TPS substrates. Therefore, we assess the quality of the predictions using standard multi-label classification metrics such as mean average precision (mAP) and area under the receiver operating characteristic curve (AUROC) averaged across individual classes. These metrics were used in the original EnzymeExplorer paper (Samusevich et al., 2025). We report the performance by averaging the metric values concatenated across all validation folds from the 5-fold cross-validation schema.

##### Accuracy, MCC, F1-score.

To evaluate the performance of subcellular location prediction methods, we use standard classification metrics as employed in Stärk et al. (2021). Accuracy standardly measures the ratio of correctly classified proteins, while Matthew’s correlation coefficient for multiple classes (MCC) serves as an alternative to the Pearson correlation coefficient for classification tasks (Gorodkin, 2004). The F1-score, the harmonic mean of precision and recall, evaluates performance from a retrieval perspective, balancing the trade-off between false positives and false negatives.

#### E.3.3 Models

##### EnzymeExplorer.

EnzymeExplorer is a state-of-the-art method for the classification of terpene synthase (TPS) substrates (Samusevich et al., 2025). The model consists of two parallel tracks. Given a protein sequence, EnzymeExplorer first computes its ESM-1v embedding (Meier et al., 2021) and a vector of similarities to the functional domains of proteins from the training dataset, based on unsupervised domain segmentation of AlphaFold2-predicted structures (Jumper et al., 2021). The ESM-1v embedding and the similarity vector are then concatenated and processed by a separately trained random forest, which predicts TPS substrate class probabilities.

In our experiments, we use the “PLM only” version of the model, which leverages only ESM-1v embeddings. This version exhibits a minor performance decrease compared to the full model but exactly follows a Y-shaped architecture, allowing us to validate the effectiveness of ProteinTTT for predicting TPS substrates. We use the implementation of EnzymeExplorer available at the official GitHub page (^10^1010[https://github.com/pluskal-lab/EnzymeExplorer](https://github.com/pluskal-lab/EnzymeExplorer)).

##### EnzymeExplorer + ProteinTTT.

When applying ProteinTTT to EnzymeExplorer, we treat the frozen ESM-1v model as a backbone $f$, its language modeling head as a self-supervised head $g$, and the random forest classifying TPS substrates as a downstream supervised head $h$.

##### Light Attention.

We use Light attention (Stärk et al., 2021) as a representative baseline for subcellular location prediction. Light attention leverages protein embeddings from a language model, which in our case is ESM-1b (Rives et al., 2021). The model processes per-residue embeddings via a softmax-weighted aggregation mechanism, referred to as light attention, which operates with linear complexity relative to sequence length and enables richer aggregation of per-residue information, as opposed to standard mean pooling. We re-train the model using ESM-1b embeddings on the DeepLoc dataset (Almagro Armenteros et al., 2017) using the code from the official GitHub page(^11^1111[https://github.com/HannesStark/protein-localization](https://github.com/HannesStark/protein-localization)).

##### Light attention + ProteinTTT.

When applying ProteinTTT to Light attention, we treat the frozen ESM-1b as the backbone $f$, the language modeling head of ESM-1b as the self-supervised head $g$, and the Light attention block as the fine-tuning head $h$.

**Table A3: Hyperparameters used for adapting ProteinTTT to individual models. The optimal hyperparameters were estimated using validation datasets corresponding to each of the considered tasks: Fitness prediction, Structure prediction, and Function prediction. Comma-separated lists show the values used for hyperparameter grid search, while the final values selected for computing the test results are highlighted in bold. Low-rank adaptation (LoRA) was only used with ESMFold, containing 3 billion parameters in the ESM2 backbone. Please note that we did not tune the number of customization steps, as adjusting the learning rate and batch size effectively controls the expected performance under the fixed number of steps, as shown in [Figure˜A10](https://arxiv.org/html/2411.02109v2#A7.F10). Therefore, we used 30 steps in all our experiments. The only exception was ESM3 + ProteinTTT, where the number of steps was set to 50 during initial experiments with different models/tasks conducted in parallel before standardizing the number of steps to 30. Methods marked with an asterisk (“*”) used a slightly different calculation of the loss function. Specifically, the loss was propagated over all tokens, including special and non-masking tokens, while averaging the loss across all tokens simultaneously, rather than first averaging over sequences. This approach was used in the early stages of development, and we provide it in our codebase via loss_kind = ‘‘unnormalized_cross_entropy’’. Please note that MSA Transformer always uses 1 MSA in a batch and the “Batch size” represents the number of sequences in this MSA with the target sequence always present as the first one.**
|  | Learning rate | Batch size | Grad. acc. steps | Steps (Conf. func. $c$) | LoRA rank $r$ | LoRa $\alpha$ |
| --- | --- | --- | --- | --- | --- | --- |
| Fitness prediction |  |  |  |  |  |  |
| ESM2 (35M) + ProteinTTT * | 4e-5, 4e-4, 4e-3 | 4 | 4, 8, 16, 32, 64 | 30 | - | - |
| ESM2 (650M) + ProteinTTT * | 4e-5, 4e-4, 4e-3 | 4 | 4, 8, 16, 32 | 30 | - | - |
| SaProt (35M) + ProteinTTT * | 4e-5, 4e-4, 4e-3 | 4 | 4, 8, 16, 32 | 30 | - | - |
| SaProt (650M) + ProteinTTT * | 4e-5, 4e-4, 4e-3 | 2, 4 | 4, 8, 16, 32 | 30 | - | - |
| ProSST (K=2048) + ProteinTTT * | 1e-5, 4e-5, 4e-4, 4e-3 | 4 | 4, 8, 16, 32 | 30 | - | - |
| ESM2 (650M) + ProteinTTT${}_{\text{MSA}}$ * | 4e-6, 1e-5, 4e-5, 4e-4, 4e-3 | 4 | 2, 4 | 50, 100 | - | - |
| MSA Transformer + ProteinTTT | 1e-6, 3e-6, 1e-5, 3e-5, 1e-4 | 1, 4, 8, 16, 32 | 1, 2, 4, 8 | 30 | - | - |
| Structure prediction |  |  |  |  |  |  |
| ESMFold + ProteinTTT | 4e-4 | 4 | 4, 8, 32, 64 | 30 (pLDDT) | 4, 8, 32 | 8, 16, 32 |
| HelixFold-Single + ProteinTTT | 4e-4, 1e-3 | 4, 8, 16 | 1 | 30 (pLDDT) | - | - |
| ESM3 + ProteinTTT | 1e-4, 4e-4, 1e-3 | 2 | 1, 4, 16 | 50 (pLDDT) | - | - |
| Function prediction |  |  |  |  |  |  |
| EnzymeExplorer + ProteinTTT | 4e-4, 1e-3 | 2 | 2, 4, 8 | 30 | - | - |
| Light attention + ProteinTTT | 4e-4, 1e-3, 3e-3 | 2 | 2, 4 | 30 | - | - |

## Appendix F Case study details

### F.1 Modeling antibody-antigen loops

We download the SAbDab dataset from the official website(^12^1212[https://opig.stats.ox.ac.uk/webapps/sabdab-sabpred/sabdab](https://opig.stats.ox.ac.uk/webapps/sabdab-sabpred/sabdab))(Dunbar et al., 2014). We apply ProteinTTT to targets with low-confidence ESMFold predictions (pLDDT < 70) and remove sequences longer than 400 residues due to GPU memory limitations. This results in a final set of 175 antibody and 814 antigen chains. We predict the full structures using ESMFold+ProteinTTT (with the same hyperparameters tuned on the CAMEO validation set specified in [Table˜A3](https://arxiv.org/html/2411.02109v2#A5.T3)) and compute LDDT scores against the corresponding PDB structures to assess local errors, which are particularly relevant for loop regions. For antibodies, we evaluate the complete structures, while for complementarity-determining regions (CDRs), we extract the CDR substructures as annotated in SAbDab according to Chothia numbering (Chothia & Lesk, 1987) and calculate LDDT on these regions.

### F.2 Expanding known structures of viral proteins

We use BFVD version archived/2023_02_v2(^13^1313[https://bfvd.steineggerlab.workers.dev](https://bfvd.steineggerlab.workers.dev)). This version contains maximum-pLDDT structures from predictions generated by two strategies: (i) ColabFold (Mirdita et al., 2022) with MSAs constructed using Logan (Chikhi et al., 2024), and (ii) ColabFold with 12 additional recycle steps and MSAs constructed using Logan. In [Figure˜5](https://arxiv.org/html/2411.02109v2#S5.F5), we also report pLDDT values for BFVD version archived/2023_02_v1, where structures are simply obtained from ColabFold with MSAs from Logan, i.e., strategy (i). We re-predict structures using ESMFold and ESMFold+ProteinTTT for sequences with length < 450 due to GPU memory constraints. We use the same hyperparameters tuned on the CAMEO validation set, as specified in [Table˜A3](https://arxiv.org/html/2411.02109v2#A5.T3), with the exception of 20 instead of 30 steps for computational efficiency.

## Appendix G Extended results

In this section, we provide additional results on test sets ([Section˜G.1](https://arxiv.org/html/2411.02109v2#A7.SS1)), discuss validation performance ([Section˜G.2](https://arxiv.org/html/2411.02109v2#A7.SS2)), and analyze the runtime performance of customization ([Section˜G.3](https://arxiv.org/html/2411.02109v2#A7.SS3)).

### G.1 Detailed test performance

In this section, we provide details on the test performance. Specifically, [Table˜A4](https://arxiv.org/html/2411.02109v2#A7.T4) shows that customization with ProteinTTT primarily enhances performance on challenging targets, characterized by a low number of similar proteins in sequence databases, as measured by MSA depth. Additionally, we provide a qualitative example illustrating how ProteinTTT substantially improves the correlation between ESM2-predicted fitness and ground-truth stability by better identifying disruptive mutations in the protein core (Figure [A5](https://arxiv.org/html/2411.02109v2#A7.F5)).

Next, [Figure˜A6](https://arxiv.org/html/2411.02109v2#A7.F6) shows the distribution of ProteinTTT effects: in many cases, customization has minimal impact on performance; often, it leads to substantial improvements; and in rare cases, customization results in a decrease in performance. This positions ProteinTTT as a method for enhancing prediction accuracy, while a comprehensive analysis of its failure modes remains an important direction for future research. While we demonstrate these effects using a protein folding example, we observe a similar distribution of ProteinTTT impact across the tasks.

We also observe that the overall trend of customization with ProteinTTT generally leads to improved performance, with robust consistency across random seeds. However, the progression of the performance curve can be rugged, particularly in classification tasks, where substantial changes in the underlying representations are required to shift the top-predicted class in the discrete probability distribution ([Figure˜A8](https://arxiv.org/html/2411.02109v2#A7.F8)).

### G.2 Validation performance

This section discusses the performance of ProteinTTT on validation data. [Table˜A5](https://arxiv.org/html/2411.02109v2#A7.T5) illustrates the validation performance of the tested methods for fitness prediction on our newly constructed MaveDB dataset. ProteinTTT enhances the performance of all the methods.

Next, we discuss the hyperparameter optimization. [Table˜A3](https://arxiv.org/html/2411.02109v2#A5.T3) provides the grid of hyperparameters explored for each model and its size, as well as specifies the optimal hyperparameters suitable for downstream applications. [Figure˜A10](https://arxiv.org/html/2411.02109v2#A7.F10) demonstrates the trend of hyperparameter tuning with optimal hyperparameter combination balancing underfitting and overfitting to a single target protein. While most of reasonable hyperparameter configurations lead to overall improvements when using customization with ProteinTTT, poorly chosen hyperparameters can have detrimental effects due to rapid overfitting. However, with a reliable predicted confidence measure, such as pLDDT, the appropriate customization step for each protein can be selected to mitigate overfitting. [Figure˜A11](https://arxiv.org/html/2411.02109v2#A7.F11) demonstrates that when using ESM3 + ProteinTTT with pLDDT-based step selection for protein structure prediction, all hyperparameter configurations result in improved performance compared to the base ESM3 model.

### G.3 Runtime performance

In this section, we demonstrate that customization with ProteinTTT can be done efficiently, with an acceptable computational overhead. Specifically, we show that ESMFold, known for being a faster alternative to more performant methods such as AlphaFold2 (Jumper et al., 2021) or AlphaFold3 (Abramson et al., 2024), still remains in the category of lightweight methods even with ProteinTTT customization ([Figure˜A4](https://arxiv.org/html/2411.02109v2#A7.F4)).

This observation highlights the practical utility of ProteinTTT. For example, ESMFold enabled structural characterization of large metagenomics data (>617 million metagenomic sequences), which would be infeasible with AlphaFold2 (Lin et al., 2023). Nevertheless, the original ESMFold has high confidence predictions only for 36% of sequences from the metagenomic database, while the other 392 million sequences remain with low or medium confidence predictions. At the same time, ESMFold + ProteinTTT enables more accurate predictions compared to the original ESMFold ([Figure˜A6](https://arxiv.org/html/2411.02109v2#A7.F6) suggests that ESMFold + ProteinTTT significantly improves predictions in almost 40% of challenging sequences). It means that applying ESMFold + ProteinTTT to these remaining sequences could significantly expand the metagenomic atlas characterized by ESMFold. Here, we illustrate this on a similar case study by applying ESMFold + ProteinTTT to more than 300 thousand viral proteins in BFVD ([Section˜5.2](https://arxiv.org/html/2411.02109v2#S5.SS2))

**Table A4: ProteinTTT performance on ProteinGym depending on MSA depth. MSA depth reflects the number of available proteins similar to the target protein and, when using large protein language models, can be interpreted as a measure of the representation of similar proteins in the training data ([Section˜E.2.2](https://arxiv.org/html/2411.02109v2#A5.SS2.SSS2)). Customization with ProteinTTT primarily improves performance on difficult targets, with low MSA depth. Standard deviations are calculated over 5 random seeds but are omitted in the right panel for brevity, where the maximum standard deviation does not exceed 0.0004.**
|  | Avg. Spearman $\uparrow$ | Spearman by MSA depth $\uparrow$ |  |  |
| --- | --- | --- | --- | --- |
|  | Low depth | Medium depth | High depth |  |
| ESM2 (35M) (Lin et al., 2023) | 0.3211 | 0.2394 | 0.2707 | 0.451 |
| ESM2 (35M) + ProteinTTT (Ours) | 0.3407 ± 0.00014 | 0.2445 | 0.3144 | 0.4598 |
| SaProt (35M) (Su et al., 2023) | 0.4062 | 0.3234 | 0.3921 | 0.5057 |
| SaProt (35M) + ProteinTTT (Ours) | 0.4106 ± 0.00004 | 0.3253 | 0.3972 | 0.5091 |
| ESM2 (650M) (Lin et al., 2023) | 0.4139 | 0.3346 | 0.4063 | 0.5153 |
| ESM2 (650M) + ProteinTTT (Ours) | 0.4153 ± 0.00003 | 0.3363 | 0.4126 | 0.5075 |
| SaProt (650M) (Su et al., 2023) | 0.4569 | 0.3947 | 0.4502 | 0.5448 |
| SaProt (650M) + ProteinTTT (Ours) | 0.4583 ± 0.00001 | 0.3954 | 0.4501 | 0.5439 |
| ProSST (K=2048) (Li et al., 2024) | 0.5068 | 0.4731 | 0.5107 | 0.5749 |
| ProSST (K=2048) + ProteinTTT (Ours) | 0.5087 ± 0.00004 | 0.4809 | 0.5104 | 0.5750 |

**Table A5: Performance of ProteinTTT on the MaveDB dataset. In this work, we use our newly constructed MaveDB dataset as a validation fold for tuning the ProteinTTT hyper-parameters for fitness prediction. For computational efficiency, we only select a subset of 50 proteins ([Section˜E.2.1](https://arxiv.org/html/2411.02109v2#A5.SS2.SSS1)) and do not run customization across multiple random seeds to estimate standard deviations. The performance shown was calculated by first aggregating correlations per assay, and then per protein (some assays correspond to the same protein).**
|  | Avg. Spearman $\uparrow$ |
| --- | --- |
| ESM2 (35M) (Lin et al., 2023) | 0.4458 |
| ESM2 (35M) + ProteinTTT (Ours) | 0.4593 |
| ESM2 (650M) (Lin et al., 2023) | 0.4568 |
| ESM2 (650M) + ProteinTTT (Ours) | 0.4604 |
| SaProt (650M) (Su et al., 2023) | 0.4926 |
| SaProt (650M) + ProteinTTT (Ours) | 0.4926 |
| SaProt (35M) (Su et al., 2023) | 0.5251 |
| SaProt (35M) + ProteinTTT (Ours) | 0.5271 |
| ProSST (K=2048) (Li et al., 2024) | 0.5444 |
| ProSST (K=2048) + ProteinTTT (Ours) | 0.5462 |

Figure: Figure A4: Running time of ESMFold + ProteinTTT. For ESMFold and its variants, the median and interquartile ranges of running times on the CAMEO test set are shown using a single NVIDIA A100 GPU. For AlphaFold2, we use estimates from Lin et al. (2023). Specifically, a forward pass through AlphaFold2 is approximately 60 times more computationally expensive than ESMFold (e.g., AlphaFold2 (no MSA; estimate): $2\times 60=120$ seconds), with additional MSA construction taking at least 10 minutes using standard pipelines (AlphaFold2 (estimate): $2\times 60+10\times 60=720$ seconds). ESMFold + ProteinTTT (30 steps) involves LoRA parameter updates, along with forward passes at each customization step to estimate pLDDT and select the structure with the highest predicted confidence. Disabling pLDDT significantly reduces computational overhead (ESMFold + ProteinTTT (no pLDDT) compared to ESMFold + ProteinTTT), but may require careful parameter tuning ([Section˜G.2](https://arxiv.org/html/2411.02109v2#A7.SS2)). Overall, ESMFold + ProteinTTT maintains the speed advantage of ESMFold, and is at least an order of magnitude faster than AlphaFold2.
Refer to caption: x10.png

Figure: Figure A5: Example of protein fitness prediction upon single-sequence model customization with ProteinTTT. Fitness predictions from ESM2 (650M) show poor correlation with experimental fitness values in the ProteinGym test set measured by the stability assay “UBR5_HUMAN_Tsuboyama_2023_1I2T” (Tsuboyama et al., 2023) (left). ESM2 + ProteinTTT achieves significantly higher correlation, likely due to improved detection of disruptive mutations in the protein core that impact protein stability (middle). The ground-truth fitness data aligns with the customized model, showing that residues crucial for stability (i.e., having negative mean fitness) are concentrated in the protein core (right). Residue colors represent the mean fitness upon all single-point substitutions (with the exception of several missing mutations in the ground-truth data), with red indicating residues where mutations have detrimental effects on average.
Refer to caption: x11.png

Figure: Figure A6: Per-protein performance of ESMFold + ProteinTTT and ESM3 + ProteinTTT on the CAMEO test set. The y-axis shows the change in TM-score after applying customization with ProteinTTT, with higher values indicating improvement. The x-axis represents performance across five random seeds. The red dashed line marks no change in TM-score (TM-score difference $=0$), and the pink band represents minor changes in TM-score ($-0.05<$ TM-score difference $<0.05$), which we do not consider significant. Each point in the swarm plot corresponds to a single protein from the CAMEO test set. On average, applying ProteinTTT to ESMFold improves the structure predictions for 7 out of 18 proteins, with 2 showing degradation. The rest of the proteins are not significantly affected. Similarly, applying ProteinTTT to ESM3 results in 6 improvements out of 18 proteins, with 1 case of degradation.
Refer to caption: x12.png

Figure: Figure A7: Test performance of ESMFold + ProteinTTT and ESM3 + ProteinTTT on the CAMEO test set depending on the total number of customization steps. The x-axis shows the averaged performance across all test proteins, with error bars representing the standard deviation across five random seeds. The y-axis metrics correspond to the structure with the highest pLDDT score up to the given step. While an increased number of ProteinTTT steps generally enhances performance, only a few steps (e.g., five) may suffice to achieve significant performance improvement.
Refer to caption: x13.png

Figure: Figure A8: Test performance of EnzymeExplorer + ProteinTTT across customization steps. The performance is averaged across all 512 proteins in the dataset, with error bars representing the standard deviation across 5 random seeds.
Refer to caption: x14.png

Figure: Figure A9: ESMFold + ProteinTTT pLDDT correlates with ESMFold + ProteinTTT LDDT. The evaluation was performed on 17,582 AlphaFold2 reference structures from the BFVD database with pLDDT > 90. Here, $r=0.875$ denotes the Pearson correlation coefficient.
Refer to caption: x15.png

Figure: Figure A10: Dependence on ProteinTTT hyperparameters for customized fitness prediction. Each plot shows the progression of Spearman correlation (green) increasing alongside a decrease in perplexity (pink) for each customization step, averaged across all assays in the MaveDB validation dataset. The model used is ESM2 (35M) + ProteinTTT, and the grid displays the combinations of different numbers of gradient accumulation steps (i.e., effective batch sizes; shown in rows, increasing from top to bottom) and learning rates (columns, increasing from left to right). As the learning rate increases and the number of gradient accumulation steps grows, the model reaches peak performance more quickly but begins to overfit to a target protein. The optimal hyperparameter combination (learning rate = 4e-4, gradient accumulation steps = 16) lies near the center of the grid, balancing between underfitting and overfitting to a target protein. Notably, the figure demonstrates that, although ProteinTTT involves three main hyperparameters (batch size, learning rate, and the number of steps), there are effectively only two degrees of freedom controlling the performance of the model. In other words, by keeping the number of steps constant (e.g., 30), the expected performance can be controlled by adjusting the learning rate and the batch size.
Refer to caption: x16.png

Figure: Figure A11: Hyperparameter search for protein structure prediction with ESM3 + ProteinTTT. We conducted a comprehensive grid search based on three key hyperparameters: learning rate (denoted as “lr”), number of gradient accumulation steps (denoted as “grad_steps”; with the batch size of two), and masking strategy (denoted as “mask”). We explored two learning rates, 4e-4 and 1e-3, three gradient accumulation step values of 1, 4, and 16, and five different masking strategies: uniform sampling of 0.05, 0.5, and 1.0 fractions of amino acids, as well as the “beta30” and “betalinear30” distributions proposed in the ESM3 paper (Hayes et al., 2024). Each row in the table presents the mean TM-score and LDDT metrics with standard deviations across five random seeds on the CAMEO validation fold. The last row, denoted as “No ProteinTTT”, shows the performance of ESM3 without customization. The results indicate that ESM3 + ProteinTTT is robust to the choice of hyperparameters and consistently outperforms the base model across all configurations. We selected the configuration from the last row (excluding “No ProteinTTT”) to compute the results on the test fold. For the hyperparameter search, we used 30 customization steps instead of 50 to reduce computation time.
Refer to caption: x17.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See weakness
soundness: 3
strengths: 1. The method is simple and general. It does not require additional data and can be easily applied to different pretrained PLMs. And the appraoch leverages lightweight finetuning (LoRA and limited steps), making it resource-efficient.
2. The method consistently improve the performance on different tasks and different models.
3. The authors use case studies to demonstrate real-world applications. For biologists and chemists who are interested in specific proteins, this could be useful.
summary: This paper introduces Protein Test-Time Training (ProteinTTT), a self-supervised framework that customized pretrained protein language model to individual target proteins during interence. ProteinTTT fixed the downstream task head and finetuned the model backbone. Experiments show that this consistently improves model performance in different tasks with different kinds of models.
weaknesses: 1. The authors claimed that the method can be easily extended to models with autoregressive masking, while they did not do that. I suggest the authors might try to show the results on ProGen.
2. The improvement on larger models are marginal. While ProteinTTT consistently improve results, the improvement on larger model is marginal and even negligible. For example, in table 2 when it comes to 650M model. The improvement is nearly negligible. I wonder what will happen in larger model like 1B or 3B.
3. There are already a lot of inference scaling ideas rather than test time training in current trends in LLM. It would be valuable for the authors to clarify whether proteinTTT could similarly benefit from such inference-time scaling strategies (e.g. Use MSAs or other context) instead of perform test-time training.

Overall, While the work is technically sound and relevant, its contributions appear more empirical and domain-specific than methodological. The paper might be more impactful if expanded with additional biological validations. In its current form, the scope and framing could align more naturally with a computational biology or bioinformatics venue rather than a general ML conference.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: **(A) Scope and Applicability.**

 1. Since ProteinTTT relies on self-supervised fine-tuning over sequence perplexity, it appears primarily suited for tasks that take protein sequences as input and for models trained with language modeling objectives. This seems limit its direct applicability to sequence-based models rather than structure-based ones (e.g., AlphaFold3) which usually have stronger performance. 
 2. Could the authors discuss whether ProteinTTT can be extended to other task types beyond the three evaluated in this work, such as sequence design?

**(B) Performance on Structure Prediction.**
1. Can authors add the performance of the current any state-of-the-art results on the CAMEO benchmark (e.g., AlphaFold2, AlphaFold3, Boltz) as a reference?
2. The recently proposed DPLM-BiT model (https://arxiv.org/abs/2504.11454) demonstrates strong folding performance which also uses a masked protein language modeling approach. Including it as a baseline could provide a more rigorous comparison and help assess whether ProteinTTT maintains its advantage on this class of models.
3. Has the method been explored for more challenging structure prediction tasks, such as protein–protein complex prediction?

**(C) Experimental Details and Behavior.**
1. In the customization steps (e.g., Fig. 3), would continued test-time fine-tuning eventually lead to overfitting or catastrophic forgetting?
2. How many fine-tuning steps are typically recommended for each task, and are these consistent across tasks?
3. Can the authors discuss more on the conditions where ProteinTTT might fail? For test time training, an extended discussion on failure cases or scenarios would provide valuable practical guidance.
soundness: 4
strengths: 1. The manuscript is well written and polished; clear, concise, and detailed enough in both method and evaluation to be easy to follow.
2. The experimental design is comprehensive, covering three relevant downstream tasks with appropriate baselines. The inclusion of case studies and in-depth analyses further strengthens the empirical validation.
3. The results are consistent and convincing, showing that ProteinTTT can effectively enhance the performance of existing protein language models.
4. Introducing test-time training to protein language models, and framing perplexity minimization as a way to adapt to distribution shifts in downstream tasks, offers a novel and insightful perspective.
summary: This paper proposes ProteinTTT, a test-time training framework aimed at improving the downstream performance of protein language models. Motivated by the observation that sequence-level perplexity correlates with task accuracy and that low performing cases often exhibit high perplexity, the authors finetune the pretrained model at test time using the perplexity of each target sequence. The method is evaluated across three downstream tasks and several case studies, demonstrating consistent improvements across both tasks and backbone protein language models. The results suggest that ProteinTTT can effectively enhance existing protein language models without requiring large-scale retraining or additional labeled data.
weaknesses: 1. The proposed method seems inherently limited in scope, it applies only to protein language models and only to sequence-based tasks. This restricts its applicability to broader classes of protein modeling problems.
2. While the reported improvements are consistent across benchmarks, many gains are modest and appear bounded by the baseline model’s performance. The authors briefly acknowledge this (e.g., in G1), but a clearer discussion in the main text along with analysis of failure cases would help readers better understand when and why ProteinTTT is most effective.
3. Not necessarily a weakness, but the title "One protein is all you need" feels somewhat overstated. ProteinTTT indeed brings measurable benefits, yet its performance remains constrained by the underlying model’s capacity and generalizability. That said, the title is understandable as a stylistic choice.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 4
questions: See above.
soundness: 3
strengths: The manuscript is exceptionally clearly written, and the method is intuitive and easy to follow. The authors include a diverse set of experiments run on several different models, and I appreciate the inclusion of confidence intervals throughout.

Methods like ProteinTTT obviously have long pedigrees in the language modeling and computer vision literature, but this is the first paper I'm aware of that applies this sort of idea to protein modeling.
summary: The authors propose ProteinTTT, a method to adapt protein foundation models like AlphaFold2 and ESMFold to specific new proteins at test time. In essence, it involves running a few steps of self-supervised fine-tuning of the main trunks of the model in question independently of any task-specific heads trained on top of them. The authors show that this procedure improves performance on a variety of downstream tasks like folding and fitness prediction, especially for proteins without known homologues.
weaknesses: The most glaring weakness in my eyes is that the effect of adaptation seems to be relatively limited. On several benchmarks, improvements are measured as fractions of percentage points. On several others (especially in section 5), the improvement is more difficult to gauge; at various points, the authors make claims about the fractions of proteins that are improved by the method, but details about the magnitude of the improvement in these cases seem to be pretty sparse. Figure 5b hints that some proteins do indeed see large improvements, but it's also difficult to read. Some more clarity here would be appreciated.

Some other assorted issues/potential areas of improvement:

1. The authors occasionally use pLDDT of the tuned model (e.g. to choose the number of optimization steps), which seems a little problematic to me; since you're training on one protein, you're probably also inflating confidence on that protein. Could the authors validate that this isn't happening, perhaps by running early stopping experiments that "cheat" by observing downstream metrics? Another thing that would be good to see would be something like Figure 5b, but pitting ESMFold + ProteinTTT LDDT against ESMFold + ProteinTTT pLDDT
2. Is there a reason the authors only test 18 CAMEO proteins? How does the method perform on proteins that aren't extremely low-confidence?
3. Sometimes, the text overstates the results. For example: "As shown in Figure 4a, ESMFold + ProteinTTT achieves significantly
higher average LDDT scores compared to general-purpose ESMFold." The improvement in this case is not actually "significantly higher" in the statistical sense, since the error bars overlap.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Can you provide a percentage of improved samples in Tab. 1 and 2? Ideally, most of the samples will be improved over backbones by test time training. 
* If not all samples are improved, can you further analysis the failure cases?
* For fitness and function prediction, how was the number of customization steps (T=30) determined, and how sensitive are the final results to this choice?
soundness: 4
strengths: * The idea of introducing test time training to PLMs is interesting and convincing. Some case studies also confirm the validity of this method when existing methods are unable to solve specific protein sequences.
* The experiment is rigorously conducted. ProteinTTT is tested across diverse downstream tasks and various backbones.
* By enabling more accurate predictions for individual proteins, ProteinTTT could significantly advance areas such as personalized medicine, antibody design, and protein engineering, where accurate, protein-specific models are essential.
summary: This paper introduces Protein Test-Time Training (ProteinTTT), a novel method for customizing pre-trained protein language models (PLMs) for individual protein sequences during inference. The method improves performance across protein structure, fitness, and function prediction tasks, especially for proteins with limited homologous sequences. This motivation is clear and well-supported: in most scenarios, researchers are primarily interested in single or a few protein sequences, and ProteinTTT is able to enhance this. Extensive experiments demonstrate that ProteinTTT is transplantable across diverse PLM backbones.
weaknesses: * As shown in Fig. A4, the time consumption caused by ProteinTTT is non-trivial, especially when pLDDT is calculated. However, the performance improvement seems trivial in Tab. 2, which is my primary concern. Despite some case studies showing that ProteinTTT can improve significantly, the overall performance does not show similar results.

* While the results are strong on average, the paper could benefit from a deeper analysis of the cases where ProteinTTT fails or degrades performance. Figure A6 indicates that for a small fraction of proteins, the prediction quality worsens after customization.

* Simply get better representation from backbones, and freezing the prediction heads is kind of confusing. Modifying the distribution of representation without aligning it to the prediction heads is somehow irrational.


Decision: Accept (Poster)