Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models
Authors: Jonas Hübotter
Patrik Wolf
Aleksandr Shevchenko
Dennis Jüni
Andreas Krause
Gil Kur
Venue: ICLR 2026 Poster (2026)
Keywords: test-time training
linear representation hypothesis
specialization
continual learning
sparse autoencoders
compressed sensing
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We find empirically and theoretically that test-time training can improve predictions of underparameterized models, even for in-distribution inputs and while using only already-seen data.

Abstract: Recent empirical studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT), and have found it to yield significant performance improvements.
However, there is limited understanding of why and when TTT is effective.
Earlier explanations mostly focused on the observation that TTT may help when applied to out-of-distribution adaptation or used with privileged data.
However, the growing scale of foundation models with most test data being in-distribution questions these explanations.
We instead posit that foundation models remain globally underparameterized, with TTT providing a mechanism for *specialization after generalization*—focusing capacity on concepts relevant to the test task.
Specifically, under the linear representation hypothesis, we propose a model in which TTT achieves a substantially smaller *in-distribution* test error than global training.
We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet, showing that semantically related data points are explained by only a few shared concepts.
Finally, we perform scaling studies across image and language tasks that confirm the practical implications of our model, identifying the regimes where specialization is most effective.
Beyond TTT, our results provide additional strong evidence in support of the linear representation hypothesis.

Published as a conference paper at ICLR 2026
SPECIALIZATION AFTER GENERALIZATION: TOWARDS UNDERSTANDING TEST-TIME TRAINING IN FOUNDATION MODELS
Jonas Hübotter*. Patrik Wolf
Alexander Shevchenko
Dennis Jüni'
Andreas Krause'
Gil Kur"
'ETH Zürich, Switzerland
Max Planck Institute for Intelligent Systems, Tübingen, Germany
ABSTRACT
Recent empirical studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT), and have found it to yield significant performance improvements. However, there is limited understanding of why and when TTT effective. Earlier explanations mostly focused the observation that TTT may help when applied to adaptation or used with privileged data. However, the growing scale of foundation models with most test data being in-distribution questions these explanations. We instead posit that foundation models remain globally underparamettrrzze, with TTT providing a mechanism for specialization after ceneeraizatioon-occusinn capacity on concepts relevant to the test task. Specifically, under the linear repre- sentation hypothesis, we propose model in which TTT achieves a substantially smaller in-distribution test error than global training. We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet, showing that semantically related data points are explained by only a few shared concepts. Finally, we perform scaling studies across image and language tasks that confirm the practical implications of our model, identifying the regimes where specialization is most effective. Beyond TTT, our results provide additional strong evidence in support of the linear representation hypothesis.
INTRODUCTION
Since the "ImageNet moment" 2012 when AlexNet won the ImageNet challenge (Krizhevsky et al., 2012), scaling data, parameters, and compute have led to foundation models, which large-scale neural networks trained on vast datasets that can be applied across a wide range of use cases (Bommasani et al., 2021). This has spurred research on scaling laws, suggesting that scaling pre-training a single model on a broad data distribution is sufficient for good performance downstream tasks (Kaplan al., 2020; Henighan al., 2020; Hoffmann et al., 2022). With first-generation foundation models, fine-tuning was used primarily to adapt models out-of-distribution test data (i.e., with distribution shift) or to leverage fresh training data that was not seen during pre-training (so-called "privileged" data). Test-time training (TTT; Sun et al., 2020: Hardt & Sun, 2024; Akyürek al., 2025) emerged as pushing this mechanism the extreme: fine-tuning a separate model for each prediction. In recent years, foundation models have grown so large that most test data is effectively "in-distribution" meaning the model has encountered similar data during pre-training. This raises key question:
Can TTT improve predictions even in-distribution while using only already-seen data?
While foundation models typically have a large parameter count, extensive results on scaling laws show continuing improvements in performance when scaling model size (Kaplan et al., 2020; Bubeck & Sellke, 2021). Consequently, our work posits that today's foundation models are effectively "underparamettrrieed.. meaning that performance can be improved by further scaling. We hypothesize that due to this effective underparameterriztton,, even if test data is in-distribution,
"Equal contribution. Correspondence Jonas Hübotter humbottttetttr ethz..c..

---

Published as a conference paper at ICLR 2026
the model cannot simultaneously approximate the ground truth across the full data distribution. TTT offers a mechanism to specialize the model to local area around the test example. By temporarily "forgetting" irrelevant pre-trained knowledge, the model "frees up" capacity to learn the relevant concepts to the immediate task at a higher resolution. We refer to this mechanism as specialization after generalization. The mechanism of TTT temporarily reallocating capacity by "forgetting" irrelevant knowledge-
to concepts of capacity saturation and interference studied in continual learning (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017).
We propose to model this phenomenon under the linear representation hypothesis (LRH; Mikolov al., 2013: Park et al., 2024: 2025), which postulates that models represent high-level concepts-meaningiin semantic features-as as directions in a latent space. This latent concept space is typically assumed to be sparse conditioned on the input, meaning that any given input activates only a few possible concepts. For particular prediction task, the linear coefficients of the active concepts effectively characterize their "meaning" for that task. Because the number of real- world concepts far exceeds a model's dimensionality, these concepts are superimposed within the model's dense activations (cf'. Figure I; Elhage et al., 2022). The LRH has been used extensively in prior work on interpretabilityy (Kim et al., 2018) and activation
concept space
d1
embedding
Bom Anog
unembeedding
y
pustrpositinn
d2
0.1
0.4
0.2
di >> d2
activatious W/
Figure 1: Setting: large concept space Q superimposed on the feature space V.
steering (Bolukbasi et al., 2016; Templeton et al., 2024) of foundation models. In this work, we analyze model where TTT can learn the meaning of these superimposed concepts from data more efficiently than training a "global" model or non-parametric methods.
Setting: The linear representation hypothesis. The LRH can be formalized as follows. We assume the existence of an s-sparse concept space + X - Rdd which is approximated by learned feature map V + Ieda with d2 < d1. The ground truth be learned linear in the concept space, i.e., I' (x) (I(x), w.) for some unknown w, E Rd1 In Section 3, we leverage the LRH to develop a mechanistic understanding how TTT behaves, and make the following key observations:
01: The learned features V yield similar neighborhoods to neighborhoods in concept space +.
02: Among test point x= E X and its neighborhood (in V/-sacce, f can be approximated by an 8-Sparse linear function in the concept space h
03: TTT in V-space finds approximately the same task-specific model sparse TTT in the con- cept space, indicating that TTT implicitly adjusts coefficients based on only a few concepts relevant to the test task.
Based on the LRH and our key observations. in Sections and 5, we analyze when and why TTT be effective. We find that TTT improves predictions in the underparameterized regime, but its improvement diminishes as models become overparameterized. We support this finding empirically and theoretically:
When? Implications for real data ($4): We analyze TTT in a setting where we keep the learned superimposed concepts (i.e., v) fixed and only update last-layer weights. Through scal- ing studies in image classification and language modeling, we find that TTT improves accuracy when the model is underparameterrize, i.e. before the test loss saturates with increasing model size. Recent empirical findings also support this view (Lim et al., 2025; Doimo et al., 2024), showing that TTT learns the local meaning of concepts rather than discovering new concepts.
Why? Theoretical insights ($5): We analyze the in-distribution test error of TTT in compar- ison the test error of globally trained model. Under the LRH and key observations, we show that TTT generalizes at test-time even if the model is globally underparameterized (i.e., the feature space is exponentially smaller than the concept space: d2 log d,). In contrast, we show that if concepts are superimposed in an underparameterized feature space, the model cannot globally disentangle the meaning of all concepts.
In classification, the logits are canonically parameterized as (#)x), we) where is the class-specific weight vector. The class probability is then given by P(c a)) exp( (xpp((((7) (#)), we)).
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
TTT. In the classical machine learning paradigm, models are trained on a fixed training set and then kept frozen during evaluation. Despite this standard practice that was used for decades, early work suggested specializing the model at test-time to each prediction aask-such examples are local learning (Cleveland, 1979; Cleveland & Devlin, 1988; Atkeson et al., 1997) and local fine-tuning (Bottou & Vapnik, 1992). More recently, the idea of TTT (Sun et al., 2020; Wang et al., 2021) has regained attention in the context of fine-tuning large foundation models during evaluation (c.g., Krause et al., 2018; Hardt & Sun, 2024; Sun et al., 2025). TTT for few gradient steps on (supe--ssee losses has since shown success in domains such as control (Hansen et al., 2021), abstract reasoning (Akyürek et al., 2025; Zweiger al., 2025), language modeling (Hardt & Sun, 2024; Hübotter et al., 2025; Sun et al., 2025: Zhang et al., 2025; von Oswald et al., 2025; Yu et al., 2025), and video generation (Dalal et al., 2025). Many standard TTT methods train on carefully selected data from the pre-training dataset (i.e., do not add any new privileged information; Hardt & Sun, 2024; Hübotter et al., 2025), and several works studied how to optimally select data for imitation, e.g., the early seminal work of MacKay (1992) and recent extensions (Hübotter al., 2024; Bagatella et al., 2025b). TTT has also been extended from supervised learning to reinforcement learning (Zuo et al., 2025; Bagatella et al., 2025a; Binz-Bone et al., 2025).
So far it has not been well understood why and when TTT effective. While many different meth- ods have been proposed for TTT. we focus here on analyzing "semi "eei-pparaetrrc" TTT (e.g.. Hardt & Sun, 2024; Hübotter et al., 2025), where a pre-trained model is fine-tuned with a supervised loss on small neighborhood of the test point in the training data. This is different from some other methods for test-time "adaptation", which are commonly applied with distribution shifts (e.g.. Wang et al., 2021: Zhang et al., 2022; Durasov et al., 2025). Basu et al. (2023) consider similar setting to ours, but analyze it through the lens of non non-maramttric estimation, relying on the smoothness of the target function in the feature space V In contrast, our framework explicitly models the underlying sparse concept space Φ. This explains why TTT substantially outperforms thon-parametric methods even when the function is locally high-dimensional (s-sparse) in the concept space. Furthermore, while most prior theoretical work simply assumes the TTT gradient aligns with the gradient on the oracle label (e.g.. Sun et al., 2020), our work provides an idealized model where this alignment justified.
Sparse autoencoders (SAEs) and the LRH. Our theoretical framework is built upon the LRH, which posits that foundation models represent high-level concepts as linear directions in their acti- vation spaces. This idea has its roots in early word embedding models, which famously showed that semantic analogies could be solved with simple vector arithmetic (Mikolov et al., 2013; Pennington al., 2014; Arora et al., 2016). More recently, the LRH has been validated across a wide range of models and domains, with studies identifying linear representations for abstract concepts like senti- ment (Tigges et al., 2023), the state of a game board (Nanda al., 2023), and even fundamental axes of space and time (Gurnee& & Tegmark, 2024). A key tool for discovering and studying these concep- tual directions are SAFs (Makhzani & Frey, 2014; 4; Lieherum et al., 2024; Gao et al., 2025). SAES are auxiliary models trained to reconstruct foundation model's internal activations from sparse, over- complete dictionary of features. This process often yields features that are monosemantic, or aligned with single, human- interpnntrrreetalle concepts, thereby providing an empirical method to uncover the sparse concept space we consider in our work (Cunningham et al., 2024; Templeton et al., 2024).
How DOES SPECIALIZATION BEHAVE?
In this section, we begin by developing a mechanistic understanding of how TTT behaves. Since the "true" hypothesized concept space & is not accessible, we train SAFS to learn an approximate concept space & whose properties can be analyzed. We use a top-k SAE (Gao et al., 2025) to obtain sparse feature representations. After brief description of the experimental setup, we detail our key observations 01-03 from Section 1.
3.1 EXPERIMENTAL SETUP
SAE framework. The SAE encoder projects a dense input vector (x) E dd to learned high- dimensional, sparse representation (xr) Id:
((x) = top (xx)), E Rdxid
3

---

Published as a conference paper at ICLR 2026
where the top, operator retains the highest values and sets all others zero. A linear decoder then reconstructs the original vector from this sparse representation:
(x) = D (()), E
R4xx,,
The encoder E and decoder D (here for simplicity without bias terms) are optimized to minimize the reconstruction error:
(x) (x)
min E.D
To mitigate the issue "dead features" (elements of ((x) that are never activated), we incorporate ghost gradient auxiliary loss (Gao et al., 2025), which resulted in only % inactive concepts in our experiments.
ImageNet experiments. We use the ImageNet- K dataset (Deng et al., 2009). The dense vectors (xx) are normalized CLIP embeddings (Radford et al., 2021) of dimension d2 = 512 (we use only the <CLS> component). We set the sparse dimension to d1 = 8 x d2 = 4096 and the sparsity level to 16. For our analysis, we the SAE's reconstructions V (z) rather than the original CLIP embeddings T (x). This choice aligns our experiments more closely with our theoretical model and circumvents known challenges in training SAEs on raw, complex embeddings. This comes the cost of a mild 6 % drop in accuracy for a global linear classifier trained on the embeddings. In Appendix F.2, additionally validate the SAE framework MNIST (LeCun al., 1998b), where we use (x) directly rather than its reconstruction, as recovering an accurate (xx) is much easier on this simpler dataset. We include training details in Appendix F.3.
TTT baseline. We define the neighborhood of a test point x² denoted Dr,,, its k 50 nearest neighbors within the training set. Proximity is measured by the Læ-distance in given feature space. For example, D.d denotes the neighborhood found in the space of CLIP embeddings. Since the CLIP embeddings are normalized, this is equivalent to using cosine similarity. The TTT procedure involves training local linear classifier W. on the neighborhood of x*:
W == arg min WER x
1
Σ (x),),
T
(R.N)ED
where is the standard cross-entropy loss for the 1000 ImageNet classes. TTT in the estimated concept space is defined analogously using (xx) and neighborhoods D.
3.2 RESULTS
(
1.0
01: The SAF preserves local geometry. Our first hy- pothesis is that the SAE mapping preserves the angular relationships between point and its neighbors. To test this, we select a neighborhood for a test point x" three different spaces: the original CLIP space (V)) the re- constructed space (V), and the estimated concept space (.). We then measure the average cosine similarity in the estimated concept space between x² and points in each neighborhood. As shown in Figure the distributions of cosine similarities are nearly identical regardless of the space used for neighbor selection. This suggests that the SAE projection to the concept space preserves the local geometric structure, supporting our first key observation.
cosine similarity
0.5
0.0
random Figure 2: Average cosine similarity in the concept space (6) between a test point and its neighbors. Neighborhoods are selected in the original (V), recon- structed (V), and concept (.) spaces.
02: Neighborhoods are supported by few concepts. We hypothesize that the data within a lo- cal neighborhood can be explained by small subset concepts. To verify this, we train TTT classifier for ImageNet on a masked version of the concept vectors, & (x) = @ ((x), where m (0, 1jd a binary mask with m, = I[0, > 0} for some trainable parameter E Rdd The
Note that the SAF is trained in unsupervised way, without explicitly retaining classification accuracy, yet using the SAE's features leads only minor drop in accuracy.
4

---

Published as a conference paper at ICLR 2026
mask itself is learned for each neighborhood by optimizing the following objective with straight- through estimator for the mask's 's gradients:"
We x. arg min Wmm
1
Σ
L(W (x). + Almml2
(..g)E
With sparsity penalty of X = 0.2, the learned masks are highly sparse, activating on average only ||moloo 40 concepts. This is substantially smaller than the total num- ber of unique concepts active across the neighborhood. which is approximately 180. As shown in Table 1 (TTT column), this sparsely supported model performs on par with TTT on top of dense reconstructions (z)²* This suggests that a small, adaptively chosen set of concepts sufficient capture the relevant information within a local region. We obtain similar results for the Gemma Scope SAE (Lieberum et al., 2024) on MNIST data, which we present in Appendix F.2.
Global
TTT
((x) (xr
71.45 10211
72.64 10.20
71.26 #0.19
ImageNet accuracy of glob- ally trained linear models vs. TTT, with bootstrap standard errors.
Table 1:
Notably, a non-adaptive mask, such one that only includes concepts active in the test point x², performs poorly on ImageNet (71.51 %). The learned mask, in contrast, often excludes some of
the test point's active concepts (1 (xr=)) @ mllo ss 11 < 16 = s), likely identifying and removing
spurious features to improve generalization.
O3: TTT in feature space implicitly finds a sparse solution. While the adaptive masking in Equation (2) explicitly enforces a sparse solution, we find evidence that standard TTT in the feature space implicitly favors a solution that is sparse in the concept space. First, the dense reconstructions (z) and
relative TV distance
0.4
indicates functional equivalence
of TTT in and é space
0.2
TTT models trained sparse concepts 2m, (x) achieve nearly identical accuracy (ef. Table 1). Furthermore, their predictions agree in = 89 % of cases, indicating that they learn functionally equivalent classifiers (apart from pathological examples). Figure 3 reinforces this by showing that both models lead to closely matched predictive distributions over the top- 10 predicted classes. In Figure 3, we compare the ordered predicted probabilities for & to the corresponding probabilities for V matching the distributions' temper- atures, and averaging over all test points. Their strong correspondence suggests that TTT on reconstructed embeddings is implicitly biased towards sparse solution in the underlying concept space. This phenomenon may be linked to the implicit bias of optimization algorithms (e.g., SGD Adam), which are known to favor minimum- Belkin et al., 2019; Frei et al., 2022). When the feature map superimposes concepts, this implicit bias may favor sparse solutions in the underlying concept space (Vaskevicius al., 2019).
0.0
3
10
predicted class rank
Figure 3: Comparison of predicted class probabilities for TTT models trained on dense reconstructions (1) and sparse concepts (Q), relative to their magni- tude. The small relative TV distance (<1; defined in Appendix F.1) between both distributions indicates strong func- tional agreement between TTT in V and & space. We show 90 % bootstrap fidence intervals across 1000 test points.
solutions (Gunasekar et al., 2018;
When DOES SPECIALIZATION HELP?
After gaining some mechanistic understanding of TTT in Section 3, we next study when specializa- tion through TTT improves over globally trained model. In Section 5, we then relate our results from Sections 3-4, by providing theoretical support for why the LRH and our observations 01-03 may support our practically relevant findings from this section.
A We use the straight-through estimator Vo m = s8gmoid(8/r) with r 0.1.
'TTT in the concept space without extra masking achieves similar results. The fact that it also works with masking provides strong evidence that TTT adjusts only a few selected concepts in the neighborhood.
5

---

Published as a conference paper at ICLR 2026
global training
majority voting
TTT
MNIST
ImageNet
language modeling (Pilc)
0.10
0.24
0.90
error rate
0.05
0.23
0.85
0.22
0.80
0.02
0.01
0.21
0.75
0.01
0.20
0.70
10a
10a
109
10c
0.5
1.5
7
14
32
model parameters
model parameters
model parameters (B)
Figure 4: Model scaling. Error rates when scaling model size (classification error in image classifi- cation and bits per byte in language modeling). We evaluate a globally trained model (black) across different model sizes, well as TTT (red) and majority vote the neighborhood (gray). While majority vote leads to a poor predictor with many classes (i.e., complex tasks), TTT consistently outperforms global training, with the performance gap shrinking the model size increases. This supports our model's implication that TTT effectively recombines learned concepts, which is partic- ularly beneficial when many concepts are superimposed in an underparameterized model. 7B the largest model size we could train with LoRA on an NVIDIA RTX 4090. We do not evaluate majority voting for language modeling, since this has been shown perform poorly by Hardt & Sun (2024).
4.1 SETTINGS
We consider the following three tasks, and provide implementation details in Appendix F:
MNIST. We train global classifier following the LeNet-S (LeCun al., 1998a) architecture with a cross-entropy loss on the MNIST (LeCun et al., 1998b) dataset of handwritten digits. To distinguish learning of concepts from TTT's ability identify the correct weights for each locally relevant concept, we restrict TTT to updating only the last linear layer. This layer fine-tuned to minimize the cross-entropy loss on the neighborhood of each test point (cf. Equation (I)), while all earlier layers remain fixed. As shown by additional experiments in Appendix D.2, end-to-end TTT does not yield a meaningful performance improvement, validating this design choice. We report classification error on the test set.
ImageNet. We use a pre-trained CLIP VIT-B/32 vision transformer (Radford al., 2021) to compute 512-dimensional embeddings of images. We then train global linear classifier with cross-entropy loss on the ImageNet-IK (Deng al., 2009) dataset. As with MNIST, TTT updates only the last linear layer, and we report classification error on the test set.
Language modeling on the Pile. To validate implications of our model on real-world task, we consider language modeling on the Pile (Gao et al., 2020) dataset, restricting our use to data which is obtained and used in compliance with the terms of service of the data host. This version of the Pile contains 17 diverse and high-quality sub-datasets, including Wikipedia articles, academic papers, code, and more. We the open-source implementation of Hardt & Sun (2024) which uses the Pile training set containing 210M sequences of total size 1.3TB for selecting neighbors, and evaluates on the Pile test set As recommended by Gao et al. (2020), we report bits per byte, which is proportional to the negative log- liglliill loss. As baseline for global training, we evaluate the Qwen2.5 family of base models (Qwen et al., 2025). Analogously to Hardt & Sun (2024), we implement TTT by fine-tuning pre-trained L.M for a single gradient step each on 50 neighbors in the order that they are selected, from most similar to least similar. To perform TTT on single GPU, we use LoRA (Hu et al., 2022), fine-tuning around % of the model parameters.
Error bars correspond to 90 % confidence intervals computed via bootstrapping with 1000 samples.
'ee split sequences to avoid retrieval-evaluueee overlap (cf. Hardt & Sun (2024)), and evaluate % of the test set, corresponding 1796 sequences.

---

Published as a conference paper at ICLR 2026
global training
TTT
MNIST
ImageNet
ablating neighborhood size
0.26
0.22
0.05
error rate
0.24
0.21
0.22
0.02
0.20
0.20
10
2
10
10 10"
100
102
10
10
fraction of dataset
fraction of dataset
number of neighbors
Figure 5: Data scaling, Left & Middle: Classification error rate of different models, trained on varying fractions of the MNIST and ImageNet training dataset. Notably on MNIST, we find that TTT learns more effectively from larger sample sizes than global training. Right: We vary the neighborhood size for TTT on ImageNet. We find that the optimal neighborhood size trades off statistical variance due to "too few examples" and "too many examples with irrelevant concepts".
4.2 RESULTS
Scaling with model size. We conduct a scaling study by varying the model size and comparing the performance of TTT against global training well as majority vote baseline over the neighborhood (i.e., a simple non-parametric approach). The results are shown in Figure 4. For MNIST, we train convolutional neural networks of varying sizes, as summarized in Appendix F. For ImageNet, we train multi-layer perceptrons on top of CLIP embeddings, varying the hidden dimension. In language modeling, we evaluate Qwen2.5 base models of sizes ranging from 0.5B to 32B parameters. We find across all tasks that TTT outperforms global training and majority vote, with the performance gap shrinking as the model size increases. We hypothesize that at a larger model size, fewer concepts have to be superimposed in latent space, leading to less interference when globally mapping latent representations to predictions. While larger model size allows for better global disentanglement of concepts, TTT can compensate for limited model capacity by adapting the head to the specific concepts in a local neighborhood.
Scaling with dataset size. Next to performing a scaling study on model size, we also vary the dataset size. Figure 5 shows the results at fixed model scale. We subsample the training datasets of ImageNet and MNIST to fractions ranging from 1 to 100 % of the original training set size, ensuring that all subsampled datasets are balsnebalanced. We then train global models and evaluate TTT on the respective test sets. We find that TTT consistently outperforms global training, with the slight trend of the performance gap widening as the dataset size increases. We hypothesize that larger datasets provide richer neighborhoods for local adaptation, enabling TTT to specialize more effectively to the specific concepts relevant to each test point.
TTT improves predictions locally. To validate that TTT improves predictions locally, we globally evaluate some TTT heads. As shown in Table 2, while improving accuracy on the test point and neighborhood, the global accuracy of these fixed TTT heads is significantly lower than that of the model trained on the entire dataset without TTT. These results confirm that, while TTT can provide localized performance improvements when adapted individually test time, such benefits do not generalize when the same adaptation is applied globally. We further hypothesize that the neighborhood needs to be sufficiently large and diverse to span all relevant concepts. At the same time, the neighborhood needs to be sufficiently local to focus on only those concepts that are relevant the test point. In Figure (right), we support this hypothesis by varying the neighborhood size for TTT on ImageNet, and finding that the optimal neighborhood size trades off locality and diversity.
Takeaway 1
TTT locally improves predictions for underparameterrized models, but its improvement dimin- ishes as models become overparameterized.
7

---

Published as a conference paper at ICLR 2026
Global
TTT on Test Sample
TTT on Neighborhood
Global TTT
MNIST
98.57 2012
99.01 0010
100.00 + 0.00 95.19 0.00
36.38 0.16 77.04 0.06
ImageNet
78.33 0.19
79.39 0.18
Table 2: Accuracy of linear model trained on the full dataset (global), TTT evaluated the test sample, TTT evaluated on the neighborhood, and of ten randomly selected local TTT heads evaluated the entire test set (global TTT). We select ten TTT heads to keep the evaluation computationally tractable. The table reports bootstrap standard errors.
5
Why MAY SPECIALIZATION HELP?
Based mechanistic understanding gained in Section 3 through observations 01-03, we next provide theoretical evidence that the LRH supports our practical observations from Section 4, there- fore, potentially offering an understanding for why specialization effective. To simplify the analy- sis, we consider univariate regression setting with imput space X' and output space IR Based on the LRH, we posit the existence of an s-sparse high-dimensional concept space & X' + R and that the ground truth output is a linear combination of concepts, i.e., ($(x), w+) for an arbitrary weight vector w, E &R Let us denote the learne map by V X + Rd and assume that the feature map is "underparamettrized"" i.e., 1 S d2 < d1. We assume access to training set 'D of size N.
5.1 ANALYZING THE IN-DISTRIBUTION TEST ERROR OF TTT
Informally, the LRH states that the model represents high-level concepts linearly as directions in some concept space, with the unembedding w. defining the "meaning" of concepts. As well-known from compressed sensing (Candes & Tao. 2006; Donoho, 2006), the feature map V can represent exponentially many concepts in superposition (Elhage et al., 2022), i.e., di ~ 8 sexp(d2/s) /)) However, as we will see in this section, even if the underparameterizeed feature map encodes the structure of the concept space, it is not straightforward to learn a linear mapping of these superimposed concepts. In contrast, we show an idcalized setting based on our empirical observations, that TTT can efficiently learn the meaning of exponentially many concepts from data, by specializing the model to the concepts relevant to the test data. We begin by restating three key hypotheses based observations from Section 3. We formalize these hypotheses in Appendix B and include proofs in Appendix C.
The feature space preserves the geometry of the concept space. Based on 01, posit that the learned feature map V preserves the similarity structure of the concept space t. Let us denote by sim (I, x²) a similarity measure in V-space, which defines a neighborhood:
Hypothesis 1:
Definition 1 (Neighborhood). For a test point x² E X and a radius 2 0, the neighborhood of size ie in the training set is defined based on the similarity measure in terms of learned features:
B (r) = ((x, y) E simg (x, x²) 2 -),, k := I (r)l-
We assume that the neighborhood in feature space is contained within slightly larger neighborhood in concept space, i.c., 1º (r) C B (r + 5). For example, by the classical Boohnson-Lindensee lemma (Johnson & Lindenstrauss, 1984; Vershynin, 2018) this assumption is satisfied with high the lower-dimensional feature map V (x) is a random
probability for 8 < O( (log(N)/ds projection of (zx) and sim measures angles.
supported by few concepts. In Section 3, we have made the surprising observation that the neighborhood of a test point x" explained by only a few active concepts (cf. 02). Based on this, we assume that locally there exists (ss)-saass concept vector Wz- E IR:: that approximates (#xz), w.) over the test point's 'sss neighborhood. This assumption may seem unrealistic at first, since the neighborhood of x= has & samples and up l active concepts. Yet, as we observe empirically in Section 3, TTT learns a ((x)- SSSaree regressor that does well on the entire neighborhood if the neighborhood size is sufficiently small.
Hypothesis 2: Neighborhoods
sim may be any measure such that neighborhoods in feature and concept space approximately coincide.
8

---

Published as a conference paper at ICLR 2026
Hypothesis 3:
TTT implicitly regularizes towards sparsity in concept space. Based on O3, hypothesize that TTT finds sparse solutions in concept space, even without explicit regularization. Concretely, we assume that TTT in V-space implicitly finds to concept space.
solution when mapped
With these hypotheses, sparse recovery (Bickel et al., 2009; Van De Geer & Bühlmann, 2009):?
bound the in-distribution test error of TTT using techniques from
Proposition (informal, see Proposition in the appendix). Let X 12d1 be an s-sparse concept space and V X - Rd be learned feature map with d2 & d1- Let f(x) (Φ(x), w.) be the ground truth function and labels be o2 -usugaaussnan. Let x² be a test point with a neighborhood (in V-space) of sufficiently small size k such that Hypotheses 1-3 hold. Let the learned features Vp be sufficiently expressive to represent f locally (in particular, d2 2 Ui log d1) d.)). We denote by 2IT the local empirical risk minimizer on the neighborhood of xt.
Then, under standard regularity conditions for sparse recovery and with high probability over the sampling of the data,
a²s (oglog(d1/s))
(f(x*) (V 6T so
k
This is the standard minimax optimal rate from sparse recovery (Raskutii al., 2011, Theorem 1).
Takeaway 2
In this idealized model, TTT can locally learn a function from very few samples that activate similar concepts as the test point, even when the feature map underparameterrize.
We compare this error bound to the generalization error of training global model Gglobul on all data: Proposition (informal, see $C.3 in the appendix). We construct instance of our model with [(x) = where V is a random projection D. this instance, the error of the global model, when averaged over random realizations of the feature map v, is Ey ((f())) - (V)( jélobal) )) ====1 dz.
As one would expect, the global model not underparameterrzzd. i.e., d2 d1, the error of global training is zero. On the other hand, the trivial global model jelobal O has error 1. As the model size d2 shrinks, the error of global training increases towards 1. As the number of distinct concepts dj increases, the error of global training also increases, approaching as dj co.
Takeaway 3
The example illustrates that when concepts are superimposed in an underparameterized feature space, a linear head cannot globally disentangle the meaning of all concepts.
We expand on our theoretical results in the appendix as follows: First, we explore whether one can understand TTT under the LRH through the lens of statistical learning theory. Specifically, in Appendix A.I, we explore this direction using notions from low-degree polynomials and hypercontractivity (Klivans et al., 2008; Paouris et al., 2022; Damian et al., 2024; Bizeul & Klartag, 2025, and references therein). Additionally, in Appendix A.2, we contrast TTT to classical non-parametric methods (Fix & Hodges Jr., 1951; Nadaraya, 1964; Watson, 1964) such majority voting, which underperform in our experiments (cf. Section 4).
6 CONCLUSION
This work introduces a framework, supported by new empirical findings, for understanding the effec- tiveness of TTT on in-distribution data, based on the hypothesis that foundation models are globally underparameterizze. We hypothesize that TTT facilitates specialization after generalization, tem- porarily reallocating model capacity to concepts relevant to the immediate test task. We formalize
"Our results indicate that TTT may not converge quickly (or at all) if the label noise o² is large or any of Hypotheses 1-3 were to be violated. Notably, the neighborhood size k needs to be sufficiently small for Hypothesis 2 hold.

---

Published as a conference paper at ICLR 2026
this intuition under the linear representation hypothesis, and show how TTT can efficiently recover the local meaning of superimposed concepts ($5). Our trained sparse autoencoders reveal that local neighborhoods are indeed supported by few concepts and that TTT implicitly favors sparse solutions in the concept space ($3). Finally, scaling studies across vision and language tasks confirm that TTT yields the largest gains in the underparameterized regime ($4).
better understanding of specialization in foundation models opens up several exciting directions for future research. An interesting question is understanding what determines the optimal neigh- borhood size and whether it depends on the test point. Furthermore, it would be interesting to analyze the compute-efficienee trade-offs of TTT: estimating at which model scale and inference budget TTT becomes beneficial. The importance of specialization is also evident in online learning, where increasingly specialized experience may require ever-larger models to fit globally: TTT offers potential solution by enabling local adaptation. Particularly interesting would therefore be extending this framework beyond supervised learning to test-time online reinforcement learning, which has recently shown empirical success.
ACKNOWLEDGGENT
We would like thank Reese Pathak and Pierre Bizeul for helpful discussions. We also thank Bruce Lee, Celestine Mendler-Dünner, and Lars Lorch for feedback on early versions of the paper. JH was supported by the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545. PW was supported by the Max Planck ETH Center for Learning Systems. AS was supported by the Swiss National Science Foundation under grant 204439, GK conducted the initial part of this work during his visit to the IDEAL Institute, hosted by Lev Reyzin, which was supported by NSF ECCS-2217023.
REFERENCES
Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. In ICML, 2025.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4, 2016.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high- dighe-immnnional inference. JMLR, 22(106), 2021.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, 1997.
Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius, and Andreas Krause. Test-time of- fline reinforcement learning on goal-related experience. arXiv preprint arXiv:2507.18800 2025a.
Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of multi- task policies. In ICML, 2025b.
Soumya Basu, Ankit Singh Rawat, and Manzil Zaheer. statistical perspective on retrieval-based models. In ICML., 2023.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biri-variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 2019.
Ryo Bertolissi, Jonas Hübotter, Ido Hakimi, and Andreas Krause. Local mixtures of experts: Essen- tially free test-time training via model merging. In COLM, 2025.
Peter J Bickel, Ya' acov Ritov, and Alexandre В Tsybakov. Simultancous analysis lasso and dantzig selector. The Annals Statistics, 37(4), 2009.
10

---

Published as a conference paper at ICLR 2026
Pierre Bizeul and Boaz Klartag. Entropy and learning of lipschitz functions under log-concave measures. arXiv preprint arXiv:2509. 10355, 2025.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman to homemaker? debiasing word embeddings. In NeurIPS, 2016.
Rishia Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeanette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4(6), 1992.
Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. In NeurIPS, 2021.
Emmanuel J Candes and Terence Tao. Near-optimal signal recovery from random projections: Uni- versal encoding strategies? IEEE transactions on information theory, 52(12), 2006.
Sourav Chatterjee. Superconcentration and related topics, volume 15. Springer, 2014.
William Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American statistical association, 74(368), 1979.
William S Cleveland and Susan Devlin. Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American statistical association, 83(403), 1988.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. In ICLR, 2024.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.0600 2024.
Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298, 2025.
Alex Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks learn representations with gradient descent. In COLT, 2022.
Alex Damian, Loucas Pillaud-Vivee Vivien, Jason D Lee, and Joan Bruna. Computationaal-statiiiiiaal gaps in gaussian single-index models. arXiv preprint arXiv:2403.05529, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hicrarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. Discover: Automated
curricula for sparse-reward reinforcement learning. In NeurIPS, 2025.
Diego Doimo, Alessandro Serra, Alessio Ansuini, and Alberto Cazzaniga. The representation land- scape of few-shot learning and fine-tuning in large language models. In NeurlPS, 2024.
David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4), 2006.
Nikita Durasov, Assaf Shocher, Doruk Oner, Gal Chechik, Alexei A Efros, and Pascal Fua. It Idempotent test-time training. ICML, 2025.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dode Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition. Transformer Circuits Thread, 2022. URL htt::///ttaassfomeerrrrut
wwlpppubb020 2022/
model.
11

---

Published as a conference paper at ICLR 2026
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 23(120), 2022.
Evelyn Fix and Joseph Lawson Hodges Jr. Discriminatory analysis: nonparametric discrimination, consistency properlies, volume 1. USAF school of Aviation Medicine, 1951.
Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In COLT, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gh dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0003 2020.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In NeurlPS, 2018.
Wes Gurnee and Max Tegmark. Language models represent space and time. In ICLR, 2024.
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021.
Moritz. Hardt and Yu Sun. Test-time training nearest neighbors for large language models. In ICLR, 2024.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2009.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Jordan Hoffmman,, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 1555. 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Transductive active learning: Theory and applications. In NewrIPS, 2024.
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. In ICLR, 2025.
William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26, 1984.
Adam Tauman Kalai, Adam R Klivans, Yishay Mansour, and Rocco A Servedio. Agnostically learning halfspaces. SIAM Journal on Computing, 37(6), 2008.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
12

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiaoo Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 2017.
Adam Klivans, Konstantinos Stavropoulos, and Arsen Vasilyan. Testable learning with distribution shift. In COLT, 2024.
Adam R Klivans, Ryan O'Donnell, and Rocco A Servedio. Learning geometric concepts via gaus- sian surface area. In Annual IEEE Symposium on Foundations of Computer Science, 2008.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In ICML, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998a.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The mnist database of handwritten digits, 1998b. URL http://yann. Lecun. www.llewwc...eeeeeeeeeeee..o.l.e comm/exdb/miist/
Michel Ledoux. Four talagrand inequalities under the same umbrella. arXiv preprint arXiv:1909.0036: 2019.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse BlackboxNI.P.. 2024.
autoencoders everywhere all at once on gemma
Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In ICLR, 2025.
David JC MacKay. Information-basl based objective functions for active data selection. Neural compu- lation, 4(4), 1992.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. In ICLR, 2014.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24, 1989.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In NAACL, 2013.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1), 1964.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In BlackboxNLP. 2023.
Grigoris Paouris, Konstantin Tikhomirov, and Petros Valettas. Hypercontractivity and lower devia- tion estimates normed spaces. The Annals of Probability, 50(2), 2022.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In JCML, 2024.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hicrar- chical concepts in large language models. In ICLR, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurlPS, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
13

---

Published as a conference paper at ICLR 2026
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021.
Garvesh Raskutti, Martin Wainwright, and Bin Yu. Minimax rates of estimation for high- dimensional linear regression over 1 balls... IEEE transactions on information theory, 57(10), 2011.
Lev Reyzin. Statistical queries and statistical algorithms: Foundations and applications. arXiv preprint arXiv:2004.0055: 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. In ICLR, 2017.
Charles J Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, 1982.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with supersupe for generalization under distribution shifts. In JCML, 2020.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, al. Learning to (lear test time): Rnns with expressive hidden states. In JCML, 2025.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
//transsformr--crrcutt.. hwtprccuiss.ubbb/eee ptbp 2024/ haab//224/caaangg- moaliinggoooommmmttiiiiiii monosemanticity.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:23 151544 2023.
Sara A Van De Geer and Peter Büihlmann. On the conditions used to prove oracle results for the lasso. Electronic Journal of Statistics, 2009.
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. In NeurlPS, 2019. 2019.
Roman Vershynin. High-ddmenssonal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Johannes Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maxi- milian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, et al. Mesanet: Sequence modeling by locally optimal test-time training. arXiv preprint 20Xi::2506.05233, 2025.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR, 2021.
Geoffrey S Watson. Smooth regression analysis. Sankkyā: The Indian Journal of Statistics, Series A, 1964.
Johnathan Xie, Annie S Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating language models with adaptive temperature scaling. EMNLP, 2024.
Hongzhou Yu, Tianhao Cheng, Ying Cheng, and Rui Feng. Finemedlm-ol. Enhancing the medical reasoning ability of Um from supervised fine-tuning test-time training. In COLM, 2025.
14

---

Published as a conference paper at ICLR 2026
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. NeurlPS, 2022.
Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right. arXiv preprint arXiv:2505.23884 2025.
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. In NeurlPS, 2025.
Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self- adapting language models. In NewrlPS, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDICES
CONTENTS
A Discussion
16
Limitations of global learning under the LRRH
16
A.2 TTT vs. non-parametrie methods
17
B Formal assumptions
18
C Proofs
19
C.I Proof of Proposition 3
19
C.2 Concentration bounds
C33 Insufficiency of global training
23
C.4 Hardness of global learning under the LRH: main proposition and proof
26
D Additional experimental results
27
D.I Scaling with feature dimension
27
D.2 End-to-end test-time training
27
Additional ablations
28
F
Experiment details
29
F.1 Comparison of logits (Figure 3)
29
F.2 SAE on MNIST data
29
F3 SAE on ImageNet CLIP embeddings
30
F.4
Scaling experiments on MNIST
31
F5
Connection to MoEs
32
F66
Scaling experiments on ImageNet
33
F7
Scaling experiments for language modeling
34
A DISCUSSION
A.1 LIMITATIONS OF GLOBAL LEARNING UNDER THE LRH
Next, we demonstrate that under the LRH, obtaining global classifier requires quasi-polynomial number of samples in the dimension of the sparse concept space when analyzed within the "low- degree polynomial" framework. commonly used in learning theory (Kalai al., 2008; Klivans al., 2024: Damian et al., 2022; 2024, and references therein). In a nutshell, the idea is to think about the behavior of underparametrization the behavior of approximating low-degree polynomial.
16

---

Published as a conference paper at ICLR 2026
We consider the Gaussian measure Ya R and introduce the family of s-sparse, k-locally linear functions (with & "cells"), denoted by Fd,a,k k C {IR + R]. defined as
Fa,s, :aakk==( [(()) ( i
1x. 11 Si < l: 7a Kr (onnnk) K2) 0, 1 llo S s,
(3)
|wiil2 2 1, La(ta) x 1, Σ 110.f112 S a
The final condition defines the sparsity index derived from the Lı-L2 Talagrand's 's inequality (cf. the monograph of Chatterjee (2014) and the survey of Ledoux (2019)), and note that the cells Ki,-.., K may depend on the function f" For intuition, consider the function (x) = || here& k = d, 1, and the sparsity index condition holds with _ |10. 2 1/d.
We argue that when "high" frequencies of the Hermite polynomial basis. Specifically, by leveraging results Paouris al. (2022), we show that for some constants c, C 0, it holds that
and i is polynomial in d, the functions in .Fd.s.k predominantly lie in the
+ +
Σ
Pmn(5
E (0.1,0.9)
(4)
Sl log(d)
La(ta)
where (.) denotes the orthogonal projection onto the m-degree Hermite polynomial basis. The main result of this part the following:
Proposition 2 (informal, see Proposition 11 below). Assume that a S 1, k x Poly(d), and s x 1. Then Equation (4) holds. Furthermore, for Z exp(S2(log d)²)), there exists polynomial-time algorithm A such that
sup
- ( S S.1,
and under the low-degree polynomial conjecture, this bound is sharp for few classes of algorithms.
We refer to the survey of Reyzin (2020) for more details on the low-degree polynomial conjecture. We emphasize that our assumptions on Fa.,,k are much less restrictive than imposing a uniform bounded Lipschitz constraint (or average L2 Lipschitz constraint), the Lipschitz constant may be high on the boundary between two cells, and the results of Bizeul & Klartag (2025) show that Lipschitz functions can be learned with polynomial number of samples.
Roughly speaking, from a geometric view, this spectral concentration implies that most of the energy of the coefficients is localized at the decision boundaries between the cells {K,)- An "underpa- rameterized" global classifier, in this case, the best low-degree approximation, necessarily smooths these boundaries. Therefore, by isoperimetry in high dimensions, this smoothing leads to significant overlap between the decision boundaries of cells. Since most samples fall in the areas of these "dis- torted/smoothed" decision boundaries, we need many samples to learn such functions. The latter aligns with previous observations regarding the required complexity (and the lack of robustness) of foundation models (c.g., Bubeck Sellke, 2021).
A.2 TTT VS. NON-PARAMETRIC METHODS
While TTT utilizes local neighborhoods, superficially resembling majority voting (i.e., k-NN; Fix & Hodges Jr., 1951) or kernel regression (Nadaraya, 1964; Watson, 1964), its mechanism is fundamen- tally different. Non-parametric methods generally require the target function to be locally smooth or constant in the feature space (V)). When this assumption fails in high dimensions, their performance degrades rapidly-the so-called curse of dimensionality (e.g.. Hastie et al., 2009; Stone, 1982). Our framework explains this failure mode under the LRH. The superposition of concepts into the under- parameterized feature space leads to function that is locally complex and non-smooth within v, even though it is simple (sparse linear) +. This leads to ambiguous neighborhoods in V where
SIt is well-known that even for 1 1-Lipschitz function, its best low-degree polynomial (it terms of T2), is highly nos-hips,hhiz, cf. Bizeul Klartag (2025).
17

---

Published as a conference paper at ICLR 2026
samples share concepts but possess different labels. Simple averaging (like k-NN) cannot resolve this ambiguity as it relies on local smoothness that may not hold W.
In contrast, TTT performs local specialization by optimizing local parametric model. TTT ex- ploits the underlying sparse structure in the concept space ($3), effectively executing sparse re- covery ($5.1). This allows TTT to disentangle the local meaning of superimposed features, making substantially more effective than majority voting, as confirmed by our experiments ($4).
B FORMAL ASSUMPTIONS
Our key assumption is that for any given input, only a few concepts are active: Assumption 1 (linear representation hypothesis (sparse concept space)), For all x E X, the concept vector ((x) is s-sparse, i.e., [| S
Next, we formalize the series of hypotheses from Section 5.1.
Hypothesis 1: Feature space preserves the geometry of the concept space. We hypothesize that the learned feature map V preserves the similarity structure of the concept space Φ. Let us denote by simn (z, x²) a similarity measure in V-space such as cosine similarity.
Assumption 2 (Neighborhood preservation). The learned feature map preserves the similarity structure of the concept map t. There exists a distortion nang 2 0 such that B (r) is contained within the concept neighborhood B Mang).
Hypothesis 2: Neighborhoods are supported by few concepts. Experimentally, make the ad- ditional surprising observation that the neighborhood of a test point x² is explained by only a few ac- tive concepts. Let denote the corresponding feature matrices as Φ2, and the observation vector by yo E IR* We assume:
and a E IR Rxd,,
Assumption 3 (Local simplicity). Locally, the ground truth function is well-approximated by sparse model. There exists s'-sparse concept vector wz Rd with s' +b(s) such that the average approximation error over the neighborhood is bounded:
(1 xt W+ 112 S Tspp(
Learned features need to be sufficiently expressive. Next, quantify the expressivity of the learned features V. Naturally, features need to be sufficiently expressive for any linear model in feature space approximate the ground truth function. First, we make the relatively straightforward assumption that locally, the learned features are linear recombination of concepts.
Assumption (Local linearity of learned features). Locally, the learned features are a linear re- combination of concepts. That there exists some P.. E dedadd such that T VT()) P.. (x)) for all x E {x*} } B
Even if the global map from concepts ((x) to features (xx may be non-linear, A4 posits that local linear approximation is sufficient within a small neighborhood, where the local behavior can often be approximated by first-order Taylor expansion. This assumption is further supported by the effectiveness of linear decoders in sparse autoencoders (Elhage et al., 2022).
Assumption (Expressivity of learned features). The feature map V is expressive enough to rep- resent the local function Wzz. We consider the set of s' s'ssaarse weight vectors in 4-space that are linearly representable by the features V. By local linearity (A4), this means a vector w must lie in the image PT, (i.e., w PT for some t 1Rd2). Let w.- be the vector in this set that best approx- imates the predictions of Wzz over the neighborhood. We assume that the resulting representation error is bounded:
1
1 x" F Wa - w. < Nep.
If W.. is not in the image of PT., there is no corresponding vector in the feature space that can replicate its behavior. This means the compression defined by Pz. has discarded information nec- essary to represent wy.. Thus, A5 highlights the importance of pre-training: for the representation error Teep to be small, the feature map needs to learn sufficient structure to represent the ground truth function locally.
18

---

Published as a conference paper at ICLR 2026
Hypothesis 3: TTT implicitly regularizes towards sparsity in concept space. We find exper- imentally ($3) that TTT solutions often exhibit behavior consistent with sparsity in the concept space, even without explicit regularization. To facilitate theoretical analysis using sparse recovery frameworks, we analyze an idealized TTT estimator that explicitly enforces this observed sparsity:
Assumption 6 (Implicit regularization), We assume that TTT is implicitly regularized towards so- lutions which sparse in concept space:
GTT arg min U yo 112 subject to [|P vllo I s'. vER42
(5)
K
While standard TTT omits the explicit constraint, we use this formulation to analyze the specializa- tion mechanism that we observe empirically. It remains to state standard assumptions:
Assumption (Bounded concepts). Concepts are bounded in Loo and L2 norm, for all x E X, [x())ll>> S Co and [|0(z)]l2 I Co.2.
Assumption 8 (Linear model with homoscedastic noise). Data follows linear model in the concept space: y = (0))), w.) + E. The noise is i.i.d. zero-mean and o² subgaussian.
Finally, we assume that 22 satisfies the generalized restricted eigenvalue (GRE) condition. Combined with local linearity (A4), the GRE is simply the standard restricted eigenvalue condition on the local concept design matrix 2. a condition that is fundamental to guarantee stable recovery in sparse regression (Bickel et al., 2009; Van De Geer & Büihlmann, 2009).
Assumption (Generalized restricted eigenvalue (GRE) condition). The local design matrix T satisfies the GRE at order 2s' with respect to Px. There exists k > 0 such that for all v with [| vllo- < 2s':
1
2
2 R I.P.
(6)
With this, we are ready to state our result of this section.
Proposition 3 (Informal version of Proposition 4). Fix any test point x² E X and any 5 E (0, 1). Let AI-A9 hold. Define the inherent misspecification error of any sparse linear approxima- tion as Tinkerent == (9xx"),,, w, )²-)² and the misspecification error on the neighborhood as
Tmis == Topa( + Munz) Trep.
Then, with probability 1 8 over the sampling of the data,
0² +0 0 (og((d///) (o'slog(d1/s) ( + O Mmis) + Tinherent-
E B((ut - (V (,)), S 0²
k
We prove the result in Appendix C and briefly highlight several aspects of the error bound:
The fast rate Ös//k) explains why TTT can learn from few samples.
The optimal neighborhood size & depends on the tradcoff between variance (i.c., Ō(skk)) and bias (from T) Note that A3 assumes the neighborhood is described by a (ss)-spass model, which is only possible if < N and the neighborhood is sufficiently "local".
3. The error grows linearly with more active concepts and is only logarithmically dependent on the total number of concepts d1. In contrast, larger feature dimension dg can reduce misspecification error through Y/ng and Thep-
C PROOFS
C.I PROOF OF PROPOSITION 3
We begin by stating the formal version of Proposition
Proposition 4. Fix any test point x* X and any 8 E (0, 1). Let A1-A9 hold. Let k be the neighborhood size and s' the sparsity in concept space. Let concept vectors be bounded in Loo and La space with constants Ca Co O. Assume the GRE holds with O. Define the inherent misspecification bias of any sparse linear approximation @wheremt (x(x*), ), W. )
c2
19

---

Published as a conference paper at ICLR 2026
With probability 1 8 (over sampling of the data), the squared prediction enror is bounded
E [v' - (v"x"(((*") ( ) s²² 2 (Einherent + Eestimation)
where the squared estimation error E
estimation achieves rate ((s/k) with C universal constant:
CO
. (02 hgplogdd//'s/ + Mepa (r + Mang)) + Drep
estimaation
R²
k
mixspeciiication
Proof of Proposition 4. Step 1: Data decomposition and comparison to oracle. By A5, there exists some "oracle" uz E Rdda such that P. Ux
Wzr.
We decompose the observations yr- (ef. A8).
Yv 4 x W. +
= $ x. Dg + ④ W.
Du Wgr + ( Wxx
Dz W.y.
+
(total misspecification)
Note that using A2, A3, A5, the average squared magnitude of A is bounded by
I 2 S n/ sr, (r Tang s) + nepp Thpp((2)) (d2)).
Let = 0g. - G Since by A5, both v2. and TTT satisfy the constraint of Equation (5), the error vector h is sparse in the concept space: P hllo 2s'.
Step 2: By the optimality of GTTT (cf. A6):
1
- ya 113
V n. C.. yo yo.
h
Substituting y. V Can A + (using local linearity, A4, V u.. p, w.)):
w (A + +
A E|l
Expanding the left hand side:
(1 A12 2h WTT (A + 8) I| =e1B) 1 1 + s112-
Rearranging yields the basic inequality:
T 2 TT (A +
T V. 1112
--
(8)
Step 3: We analyze the right hand side of Equation (8). Using local linearity (cf. A4), h v (A + E) be the total concept score. Then,
(P.. h)T T.. Let Z TT
h
T (A
+ (P Z).
k
We bound the inner product using the sparse dual norm, since PTP rillo 2s'
IP.I. h, Z) S || h l|2 || 12.20
We condition on the high probability event (w.p. 8) of Lemma 6 and use Lemma (with
7 = (r, 8, d2))
1 < A + 7/4 == r.
Substituting this back into Equation (8) gives:
- 11l2 < 2r P h112.
(9)
k
20

---

Published as a conference paper at ICLR 2026
Step 4: Applying GRE. Since || hilo 2s', apply the GRE (cf. A9) Equation (9):
x| x11.2 S 21
This yields bound on the L2 estimation emror:
21
P v
(10)
Step 5: Bounding the prediction error. We next decompose the total prediction error,
ع ( (v(x=)) ))
(y* (M (,r"),00 Uz.) + (1((2',, ) Uz. TTT
(y* ( (x'),, )xz)) + (1)
(y² (+(x"), wx (D(x"), P.TA)
(local linearity, A4)
E + (2(x"), ) w. w (0(x') P.A.
(data model, A8)
Extimation
We next bound the estimation error &estimation using A7 (L2 bound):
Eestimation (((x"), (00 hll2 C...
Combining this with Equation (10) gives:
Eestimation I 2C
Hence, the expected squared error is B| 0² + (|Eintorent) esstimation 1)
Step 6: Finalizing the bound. Finally, we resolve the dependencies and compute E estimation Using
(a + b)2 2a² + 262:
4C2 R²
80 (A² + n22)
estimation
I
22
S
R²
Substituting the definitions from Lemmas and 7:
A C2 8'
(log(dh/8') log((16))
1o
V/2 = 2s' (r,
Therefore,
8C2 2 s' (c#o2 2og(d1/50) 41/ (r, 8, sd))))
C2
esstimation
x²
Combining the universal constants into O' yields the final result.
C.2 CONCENTRATION BOUNDS
We utilize the sparse dual norm analyze the correlation between sparse vectors and the noise / misspecification.
Definition (Sparse dual norm). We define the sparse Lg dual norm of a vector e Rd as:
| 1
sup (Mlaa
(u, 2) max S:|SI=m
||2s||2-
Lemma 6 (Sparse noise concentration). Under A7 (L bound) and A8 (subgaussian noise), there exists a universal constant CH > 0 such that for any 8 (0, 1), with probability at least 8:
ss' AA= Cи Ca. ov ( (log(d1/s)) + log(1/6))
$ k
2,28'
21

---

Published as a conference paper at ICLR 2026
Proof. Let Ze ; 2 and 2s'. The sparse dual norm is the supremum of stochastic process indexed by the set of sparse unit vectors, Um {u E RR: ||u/lo m, ||ul|2 1}.
1Ze sup (u, Ze). uethm
For any u E Um, the random variable X, = (u, Z = E, (Z(z)), u) a sum of independent centered subgaussian variables. Its variance is uniformly bounded:
1 k
Var(X
Var(s;) (b(x))
1=1
0²
(as)), u)²
k
k
σ² k2
mo² 4.00
(Ct, Vol S (k: m0
k2
k
Moreover, the process has subgaussian increments: for any u, v E Ums
1/2 oCo,00 [xx, Xollvz S ( o)))
o0o.so
I u112
||u
0//2-
Vh.
Vk
Hence, by Dudley's entropy integral (applied with the L2 metric on Um and diam)(mm) S 2),
oC0,00
dlam((km.)
E 2l
VlogA(umm: (122,,))
VTc
Using the standard bound N N(lm, |1 (12,,) I (3/E)
oCu.00
f V mlgggdd//// /m) + m log(3/e) de S oCo.co V
m mlogi d1 /m)
E [| 12
X
Vi:
k
We conclude using standard concentration inequality for the supremum W /Z of a sub- gaussian process: with probability at least 1 1- ô,
(V m mmog((d//m)
(og((/0)
log(1/5)
| < Ze + oC
2 00
The result follows by substituting m = 2s' and using Va + vb 2(aa+b) to consolidate terms under single universal constant Cи-
Lemma 7 (Misspecification correlation bound). Let A be misspecification vector such that | S n. Under A7 (Log bound), the correlation of A with sparse concept vectors is bounded by:
S = V25cCo.ov
k
2,25'
Proof. Let ZA = 1 A. We first bound the Loo norm.
By Cauchy-Schwand E. PiyAi <
V E.
+3
Σ, A? S /K02,.0VETH
Thus, 1 S
Co.co VTj The sparse dual norm is bounded by:
|| A 1م22 = max 5:18|226'
[I(Z))slz S 2 S VV25' 32.'Co.oovi
22

---

Published as a conference paper at ICLR 2026
C.3 INSUFRICIENCY OF GLOBAL TRAINING
We define specific construction that satisfies Al-A5 and models the superposition of con- cepts through a randomized feature map. This construction captures the challenge faced by an underparameterizee model learning complex environment.
Definition 8 (Globally non-lcarnable instance). Let d2 2 [ D(slogdd) We assume the following:
Data distribution and concepts: We partition the input space X into = d1 disjoint neighborhoods {Bm J=== with equal probability P(Bm) = 1/d1. The concept mup is constant locally: +(x) = em E RR (standard basis vector) for x Bm. This 1-sparse.
2. Ground truth: The observations are noiseless and constant y = everywhere. The global ground truth vector w. = 1. Locally, the function perfectly matched by the I-sparse model Wm Em-
Learned features random superposition: The feature map V + Teda is defined such that (xx) = Pm for E Bm. which implies local linearity of features. We model the learned drawn independently and uniformly features by assuming the representations {pm) 1 from the unit sphere Sda-1
We first verify that Definition 8 satisfies Assumptions 1 to 5, A7, A8, leading to a misspecification error of Mmis = 0. As a consequence and by Proposition 4, TTT is consistent. Notably, the dimension of the feature map d2 may be exponentially smaller than the number of concepts d1, yet TTT can still learn the ground truth perfectly.
Proof.
A1 (Sparse concepts): (xx) em I-sparse.
A7 (Bounded concepts): [ [22 = 1.
A8 (Linear model): y w. 1. 02 = =0
A2 (Neighborhood preservation): We require simg(ar,")) simp (z, x²)] S nang- Consider I E B;, E B, with i j. sime x') (e,, e,) 0. simg (æ,, x²) = PP). Since Pj are drawn uniformly from Sda-1, the inner product concentrates around O. By stan- dard concentration inequalities (related to the Johnson-LLindensrraus lemma), with high probability over the draw all pairs {p:}. we have max KPt,,;)| I S O ((ggl((()/d)))) Thus, A2 holds with Nang small if d2 is sufficiently large compared log(d1).
A3 (Local simplicity): In Bm, f(x) 1. wm = Cm is 1-sparse and achieves 7apa 0 since (T(x), w.) (cm. 1) = 1 and (0(x), wm) = (Cm: em) = 1.
To verify A4, A5, we need to define the local linear maps Pm.-
A4 (Local linearity): We require (z) Pm ((z) for Bn This means Pm. = Pm- We construct Pm Redxxi by setting the mith column to Pm and all other columns to zero:
Pm = (0,...,P3) .......pm.......
A5 (Expressivity): We require = to be in the row space of Pm- The row space of the constructed Pm is exactly span(em.. Thus, 'ep O. We also need to verify that the optimal local solution corresponds to a sparse concept vector. The local optimization is min,(1 (pm. e2)2 The minimum norm solution i,,, = Pon/ |lpm = Pm (since [Pmmll = 1). We check the sparsity of the corresponding concept vector P nr P Pm The j-th component of PT Pm is (col, (Pm), Pm). For j = m, it is (pm,Pm = 1. For j A m. it is (0, Pm) O. Thus, PT F = em. which 1-sparse. m
23

---

Published as a conference paper at ICLR 2026
We compare this to training single global model on all data,
N
Gglobal
arg min
1
Σ (y (V:)),, o))2.
N
2
minimizzed
Remarkably, global training fails learn this "simple" ground truth function, even as N oc. Due to being underparameterized. the model's 's features represent concepts in superposition. i.c., the features Pm are not orthogonal. Thus, adjusting the global model to fit one neighborhood inevitably interferes with the predictions in other neighborhoods. Global training therefore has to find compromise that minimizes the average error across all neighborhoods.
Proposition 9 (Interference error of global training). Consider the instance of Definition 8. The expected approximation error of the global model, averaged over the random realizations of the feature map V, Ey [(y
( (x), (selobel)) = de
d1
Remark 10. Note that if the global model underparameterized, i.e., d2 = dj, the error of global training is zero, one would naturally expect. On the other hand, the trivial global model oalobal has error 1. As the model size d2 shrinks, the error of global training increases towards 1. As the number of distinct concepts dj increases, the error of global training also increases, approaching as d1 + co. When increasing the number of distinct concepts, global training must compromise between more neighborhoods, leading to higher interference.
Proof of Proposition 9, We analyze the approximation error of the global model. The global loss is:
L Llaobal (v)
[y (V())
d1 (pm. u))
(11)
Let P E Ridi be the matrix whose rows are PTT- The loss can be written in vector form:
I.global (v)
d1 |1 Pull2.
(12)
This is a standard least squares problem. We assume d1 > d2. Since the vectors pm are drawn from a continuous distribution (uniform on the sphere), / has full column rank (d2) with probability 1.
The optimal global model is islobul = PPPTP)-1PT 1. The resulting approximation error is:
Eglobal = [1y - (Z)), relobal) ) = global (islobel)
= di - P(P
Let II = P(PT P)- Rd xd1 be the orthogonal projection matrix onto the column space of P.
The residual vector is 1 II1 (I II)1. Since (I IT) also an orthogonal projection matrix, (I - 11) (I II) (I II).
Eglobul
- -I m1
1 -
di
d
-1-1-1
m1)
dj (di - 1dm m
dj
1
1 mm.
d1
We want to calculate the expectation of this error over the random realization of P.
E(gglobal) = 1 di
1 IF m
(13)
We next analyze the term 11I1 E E Hy and the expected values of the entries E[Il,y].
Step 1: Diagonal elements (i = j). 11 the leverage score of the i-th data point p.. The trace of projection matrix equals its rank. Since P has rank d2 (w.p. 1), Tr(II) d2.
di
Tr(II)
(14)
24

---

Published as a conference paper at ICLR 2026
Taking the expectation:
d1 Σ
[[Tr(II)
I1
d2.
(15)
Since the vectors {pm] are drawn i..dd,, the distribution of P is invariant under permutation of the rows. Thus, [[[ must be the same for all i.
d2 d1
d, [ d2
E [1
(16)
Step Off-diagonal elements (i + j). We show that E[] using symmetry argument. The entry Iy given by pi (PT p)- Let S PTP.
Eksi P&PP and = S S_ P.P," Since d1 > d2. we have d1 - 1 2 d2. As the distribution is continuous, S_ invertible (rank d2) with probability
We examine the conditional expectation Ep. [T, (Pk.yyPpkiee | Let S-,
We use the Sherman-Morrioon formula to analyze 11; p. P-
S_.P.P S_1 -- 1 S_İP:
(17)
Applying P. from the left and py from the right:
(p. S- (:::(p)) (p. S
11.3 = p; S_PP,
1 1 p. S
P. S
(0) -TTT=)))
1
1 S:PPi
1 p. S_!Pp
Let A = S_ (which positive definite) and = S_!P,. Note that A and u are independent of p.- (p..a)
We define the function h(ps)
1+ 1+
We observe that h(p;) is an odd function of p::
(-pr. u)
- (pi, u)
h(-p.) = 1+ (-P, A(-pi))
= -h(p)).
(18)
1+ (pi, Api)
The distribution of pr (uniform on the sphere) is symmetric around the origin. The expectation of an odd integrable function over symmetric distribution is zero.
Ep, [1 HPN Jazil] Ep, [h(p:)] 0.
(19)
By the law of total expectation, E[II,]] = 0 for i j.
Step Finalizing, We combine the results for the diagonal and off-diagonal elements:
E11J111]= Σ EILL + Σ B[1l
di d2 + di (di 1) dz.
dl
Finally, the expected global error is:
= d2
gelobal
=
E |
(20)
d1
di
n
25

---

Published as a conference paper at ICLR 2026
C.4 HARDNESS OF GLOBAL LEARNING UNDER THE LRH: MAIN PROPOSITION AND PROOF
We begin by stating the formal version of Proposition 2:
Proposition 11. Let Ci 2 a S k d'), and s 1. Assume n 2 exp(S((log d)²). Then Equation (4) holds and there exists a polynomial-time algorithm A in its input, such that
sup FeFaak
ED]IA(D) In S 0.1.
Under the low-degree polynomial conjecture, this bound is tight, n. Z exp(S((log d)²) samples are required by any algorithm that relies on the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) frameworks. Furthermore, without the low-de polynomial conjecture, one can show that poly(n) samples are needed for any algorithu.
For further details on the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) frameworks, refer to Reyzin (2020).
Proof. Recalling the definition of Fd and using that s x 1 and
x 1, we obtain that x log log(I/d))lf
4
f S di
Paouris et (2022) showed
Var(Pa[f")) & exp(2 - 2t) Var(f*) exp(-ct log(d) S exp(-ctlog(d)), where P. is the Ornstein-Uhlenbed (OU) semigroup, i.e.
(21)
Px[f*)(x)= (fxxp_( + vI exp(-200 2t = exp(-tm)PPm.)
m=0
and here Pm
(f*) ) is the projection operator on the m-Hermite polynomials.
Using Equation (21), we conclude that for m E 1,..., cloo(c)) (by choosing 4m x 1/m)
|| (f (1) A exp(-c, log(d)/m))
Now, recall that we assume & is polynomial in and note that up to a measure zero, for A, P([f*] - f*, it holds for any t << 1 that
Va E Rdd A.(x) S 22/ogk()) svi Vlog(di).r,
where used our assumption that there are i cells, the definition of the OU group. and the maximal inequality of 2k-Gaussian. Therefore, we can choose t S c/ log(d), for small enough 2 0 and obtain that
| (I12(1))) 2 0.99.
If there were more than 0.1 L2(Y)), energy in the m-coefficients for m 2 C'lo((//n) and C' large enough, we would obtain contraction. As for t clog(d), holds that
exp(-mt)- ||Pm (f (f*))l & exp(-cm/ (o((cmmlog({d)] f
S 0.01 ||P (f*) (1
which cannot align with the previous equation and our assumption of || 1. In words, this property says that I is from being a "low degree" polynomial. Meaning that
0.1 <
E +
Σ
Pm(
$...
La(7a)
where c, E (0, 1), C 1 are absolute constants.
Therefore, obtain 0.1 approximation at least) the top (log(d)) basis of the Hermite polynomials. Using the result of Bizeul Klartag (2025) (or the classical work of Kalai al. (2008)), it can be done with
f*, need learn the coefficients of at most (and
8 log(k) d log(A) x² log(k) llogdd)=* exp((C log(d)))
26

---

Published as a conference paper at ICLR 2026
samples. By definition, Model (GIM). Since the generative component of our functions satisfies k* x (g((d/) n/) it follows that, under the low-degree polynomial conjecture, these classes require at least exp (log( (olllo/(g)))))) ))) samples to even learn the GIM, see Damian et al. (2024) and references within, and the seminal work of Arous et al. (2021).
easily that Fd,s,k .. is much harder to learn than the Gaussian Index
Without the low-degree polynomial conjecture, one may use the result of (Klivans et al., 2008, Thms. 26 and 27), which shows that the learnability of the subclass of
{Ix (z) K is polytope in Rd with Poly(d) facets}
{P< R.
requires at least exp exp((((oog()) x Poly(d) samples in the information theoretic sense. However, an algorithmic gap remains in these classical works Klivans et al. (2008); Kalai et al. (2008), and there also a gap in the corresponding upper bound for the sample complexity.
D ADDITIONAL EXPERIMENTIN RESULTS
This section provides additional experimental results that complement the findings presented in the main body.
D. SCALING WITH FEATURE DIMENSION
globul training
mujority voting
TIT
MNIST
ImageNet
0.10
0.24
cmror rate
0.05
0.23
0.22
0.02
0.01
0.21
0.01
0.20
102
10P
10a
hidden dimension
hidden dimension
Figure 6: Scaling model width. Classification error rates when scaling model width. We evaluate globally trained model (black) across different model sizes, as well as TTT (red) and majority vote on neighborhood (gray). While majority vote leads to a poor predictor with many classes (i.e., complex tasks), TTT consistently outperforms global training, with the performance gap shrinking as the model size increases.
For the scaling experiments MNIST and ImageNet shown in Figure 4, varied only the network width while keeping the depth fixed, and we reported the resulting error rates as a function of the total number of model parameters. Our theoretical analysis ($5) relies the expressivity of the feature space, which requires the feature dimension d2 to scale at least proportionally to log dj- To make this connection explicit, present the same results from the main body function of the feature dimension in Figure 6 (instead of the total parameter count). The resulting curves show similar qualitative and quantitative behavior: TTT consistently outperforms global training and majority vote, with the performance gap narrowing as the hidden dimension increases.
D.2 END-TO-END TEST-TIME TRAINING
Because full end-to-end training on the neighborhood set computationally expensive, it is common to fine-tune only the last linear layer (or the last few layers) as trade-off between computational cost and performance. An alternative to apply a parameter-efficient fine-tuning method such as LoRA (Hu et al., 2022). In the experiments in Figure 4 for the image tasks (MNIST, ImageNet), adopted the former approach and fine- tune only the classification head. For the language modeling experiments, used LoRA fine-tune the model. Both strategies introduce capacity constraint
27

---

Published as a conference paper at ICLR 2026
on the TTT model. To test whether our conclusions still hold when removing this constraint, conduct an end-to-end TTT experiment on MNIST.
Mirroring the scaling experiment in Figure 4, we train LeNet (LeCun et al., 1998a) convolutional neural net- works (CNNs) of varying sizes across five random seeds. Details about the global pre-training procedure are pro- vided in Appendix F.4. Based on these pre-trained mod- els, we compare two variants of TTT. First, we apply last- layer TTT. where all layers except the final one are frozen and only the classification head is fine-tuned on the neigh- borhood set. This matches the setup used Figure Second, we perform end-to-end TTT: neighbors are se- lected using the L2-distance in the hast-hidden-laeer rep- resentations of the pre-trained model (as before), but sub- sequently all model weights are fine-tuned. The resulting model is then used to predict the class label of the test sample. The tnn-ttunig hyperparameters for end-to-end TTT provided in Table 3, obtained by tuning the validation set.
0.10
global training
rute
0.05
TTT (linear head)
TTT (full model)
0.01
10'
10
10a
model parameters
Figure 7: Classification error rates
when scaling model width on MNIST. We compare last-layer TTT (red) against end-to-end TTT (blue) with the globally trained model (black) serving as the baseline. Both TTT variants yield nearly identical error rates.
Hidden dimension
25
50
128
176
240
320
512
832
1152
1792
Learning rate
5e-3
0.01
0.01
0.01
5e-3
1e-3
le-3
1e-3
1e-3
5e-4
5e-4
Epochs
200
200
300
100
200
200
200
300
300
300
200
Neighbors
300
50
200
300
300
50
50
300
300
50
300
Table 3: Hyperparameters for end-to-end TTT.
E ADDITIONAL ABLATIONS
While the improvement of TTT image classification may seem small in terms of classification error, find that TTT significantly reduce cross-entropy loss the test set, shown in Figure 8.
global training
TTT
test loss (emageNet)
test loss (ImageNet)
1.10
0.21
1.00
test loss
test loss
0.20
0.90
0.16
0.80
0.70
0.12
106
10
model parameters
model parameters
Figure 8: Left: The average cross-entropy loss of the test samples of the globally trained MLP head lower compared to the test-time training (TTT) model. Right: TTT substantially lowers cross- entropy loss on the test point compared to the global model when, to filter noisy labels, we filter to test points where either the global model or TTT predict the correct label. This suggests that TTT increases cross-entropy significantly on the noisy labels that are not predictable by the base model, while lowering cross-entropy the points which are predictable.
28

---

Published as a conference paper at ICLR 2026
F EXPERIMENT DETAILS
In this section, we provide additional details about our experimental setup. These include the exper- iments used to validate sparsity ($3) as well as those designed to analyze and illustrate the implica- tions of our theoretical results ($4).
We open-source our code implementation.
F.1 COMPARISON OF LOGITS (FIGURE 3)
We start by introducing the related notation. Define the top-10 probabilities for TTT in the sparse concept space for the test point x² as vector (x*) [0, 1 similarly, p* (xx)) for dense recon- structions. this notation, we implicitly assume that the probabílities p (x*) are scaled by some optimal factor Tz In particular, define the logits corresponding to 10 (z*) (i.e., before the softmax application) by logit (x*)) lxgit((**). Then, for each test point x², we "calibrate" (cf. (Xie et al., 2024)) the concept space probabilities (xx) by finding the optimal scale Tx* which aligns them closer. Namely, we choose Ta- to minimize the following KL-divergence term:
KL p (xx) 1" (x")) min,
logit + (x**)
x = softmax
We note that although the temperature Tg. is adjusted on a sample-by-sample basis, the variation not significant. Specifically, find the temperature has mean of around 0.8 and standard deviation around 0.1. This mean value (T.- < 1) indicates that the predictive distribution p (x*) in the dense space has slightly sharper profile than the baseline.
We now introduce a metric to quantify the remaining discrepancy between the calibrated sparse distribution and the dense distribution. The relative total variation at position {l...., 10) is defined as follows:
(1 pd (x+)
relTV, (V, )) = Eg-
Er ID (x*)) E. (pf (zo*))
where the absolute difference is normalized by the average magnitude of the corresponding proba- bilities. Incorporating both averages into the denominator ensures the robustness the quantity to noisy observations, since both the scale of ph and p. are taken into account.
F.2 SAE ON MNIST DATA
We evaluate the SAE setup the MNIST dataset (10k test samples, 60k training samples). To obtain a sparse concept vector (<x;) for each image E DMNIST. we employ Gemma-Scope- style SAE (Lieberum al., 2024). The only conceptual difference from regular SAE applications is that our encoder has a convolutional architecture, i.e., it is LeNet-like (LeCun et al., 1998a) and maps each image to a representation vector (xx,) E Rdd with de 256, which is then lifted sparse concept vector (Zzz). However, the decoder is still linear to force linearity of the concept space. We also directly reconstruct the inputs, i.e., images x, which means that reconstruction error takes the following form:
E || - (<))ll, D E x8.2
In this setup, ((x)) E Rdd where di = 1024 and the average sparsity of the concept vector is
Eg [x)|oo 18.9.
https: http://tttt...oerrroooo//ttttttt
29

---

Published as a conference paper at ICLR 2026
In the Gemma- Gemma-Scope-sty/e Scope-style SAE, the sparsity constraint is enforced via thresholding activation, Le., inputs v E IRd are passed through
= ReLU((e 0),
where the ReLU is applied component-wiss for some thresholds Rd independent of v. Con- sequently, all active components of the sparse vector i are positive. In addition to thresholding, the Semma-SScopestyyl SAE implicitly enforces the desired sparsity level via Lo penalty on concept vectors (x) and uses straight-through estimator for the gradient estimate.
The procedure of searching for the optimal mask m stays the same as per Section 3, however, we note that in this setup the penalty À is set to 10-3 For this experiment, the average resulting sparsity of the mask m is Eg- [[]m|]o] 24.5.
For dense base model (i.e., counterpart of CLIP embeddings in the case of ImageNet), we train a variant of LeNet (LeCun al., 1998a) with a scale of 0.5. The resulting feature dimension before the final linear classification layer is equal to 50. This model is clearly underparameterized with test accuracy 94.51 %.
For the base CNN TTT model, perform 100 full-batch steps of Adam with learning rate of 10-1 (increasing budget does not affect the performance). For TTT with the adaptive mask in the concept space, do 200 full-batch steps of Adam with learning rate 5 10-2. The neighborhood size for TTT is set to n = 100.
The resulting accuracies are:
LeNet CNN TTT: 97.62,
Masked SAE TTT: 97.72.
Auxiliary observations. We use the same notation for active sets for concept vectors in Sec- tion 3. However, for better clarity, we recap the notation. For each sparse vector ((x) E IR we define its active set as the set of vector components that non-zero, i.e.,
m( = {e (T())) > 0}.
In this spirit, we define the active set for the current test point x² as follows:
m2 m m()(x*))
Similarly, active components of neighbors are defined as
m: = mm m(((z))) T₁ E D
In this context, the "intersection" analysis reveals the following properties the adaptive mask m, active set of the test point m,, and neighbors' active m,:
Ex Im mm| 2 7.9. E., Us (m, (m)l s 7.1, Eg U/ (m, O m =s 8.3.
Ex U: mll ~ 144.5,
In particular, one might be tempted to draw the following simplifying conclusion:
m U.(m; O my
that is, to define the mask m so that it only selects indices appearing in both the test point and its neighbors. However, this approach fails on more complex datasets (e.g., ImageNet, see Section 3), because additional slack components in the mask are necessary to capture "non-spurious" test fea- tures.
F.3
SAB ON IMAGENET CLIP EMBEDDINGS
Top-k SAE training. Obtaining sparse autoencoder with meaningful features (with mild amount of non-active neurons) is a task with multiple caveats. We employ several common techniques to improve the training procedure, which describe below.
We use a learning rate warm-up to ensure that the concept space properly explored at the start of training, preventing neurons from deactivating and becoming trapped in suboptimal configurations.
30

---

Published as a conference paper at ICLR 2026
We warm up the learning rate linearly to the value 3 10 for To 5000 steps. After this,
employ a typical cosine decay with horizon of T = 10 In particular, let be the current step, then at each step the initial learning (in this case 3 10 computed as follows:
10-110+++000 +) is multiplied by the value of X₁ which is
A = 0.5 (1+ cos (2) with
(1-
To).
In addition to the learning rate schedule, also gently ramp up the sparsity of the concept vector to the desired value of k = 16 as follows: let ko = 128 be the imitial sparsity the concept vector and K 10000 be the number of warm-up steps, then
k == ko (ko k) 7 with m:
Once the target sparsity of k 16 is reached, the value remains the respective level until the end of the training.
We now describe the implementation of the ghost gradients (Gao et al., 2025) used in our ImageNet run. In a nutshell, ghost loss ensures that features that are not activated in the top-k are still getting learning signal during the training. This is very important as non-conee convextxy of the optimization land- scape often leads to suboptimal configuration for which considerable amount of units is inactive. The ghost gradient method aims at "shaking" these units up to make them active again. Thus, we first need to define which units are considered inactive during the training. Let ʃi E Rd denote the vector of unit activation frequencies after iteration where
(fi) = number of times unit j is active after i processed samples
Then the unit j is considered "almost inactive" if (fi)g S 10 Thus, define the corresponding "inactivity" binary mask (mz); = T(())) & 10-1). Ghost gradient uses these features to improve the current reconstruction
(xx (xx). Namely, the inactive features are decoded, i.e.. (x) D (m, o (E- (x))),, as if they were present and used to minimize:
(x) -
(22)
d2
We add the ghost loss (22) to the initial reconstruction objective with weight of 10
We also employ gradient clipping for more stable iterations to the norm range of [0, 1]. and introduce additional dropout with rate 0.5 for the are-activations E (xx) to foster the diversity of concepts. We use column- wise normalization for the decoder weights to enjoy more stable training. Note that, since CLIP embeddings have unit norm, such restriction does not hinder the expressivity of the decoder. We also initialize the decoder be the transpose of the encoder, which is common practice in SAFs. As for generic hyperparameters for the Adam optimizer (Kingma & Ba, 2015), fix them to the following values: batch size of 4096, weight decay of number of epochs 100, and Adam's 's 31 of 0.9 and 82 of 0.999.
Global training and TTT. To train the global concept space and CLIP reconstruction models, we perform 100 epochs of batch size 512 using Adam optimizer with learning rate 0.001 and weight decay of 5 10-9. For each TTT point, we do 80 full-batch steps (batch size 50) of Adam with learning rate 0.02 and zero weight decay. The latter hyperparameter set is used for TTT both in concept space and in CLIP reconstructions. Unless otherwise specified, Adam's parameters follow the default values in PyTorch (Paszke al., 2019).
F.4 SCALING EXPERIMENTS ON MNIST
In the scaling experiments reported in Section 4, we trained multiple LeNet (LeCun et al., 1998a) convolutional neural networks (CNNs) at different model seales. The architecture comprises two convolutional layers, each with ReLU activation and 2 nected classification head. The reference models of varying sizes were trained with the hyperparam- eters listed in Table 4. Optimization was performed with Adam (Kingma & Ba, 2015).
max-pooling, followed by a fully con-
For the model scaling plot in Figure 4, we trained each reference model across five random seeds and applied both TTT and majority voting. For TTT, all model parameters were frozen except for
31

---

Published as a conference paper at ICLR 2026
Model parameters
280
600
1268
1976
2985
4430
6386
11770
24294
40818
61342
85866
Learning rate
6e-3
2e-3
le-3
8e-4
6e-4
6e-4
2e-3
2e-3
2e-3
6e-4
2e-3
2e-3
Batch Size
500
200
400
600
300
300
400
300
300
100
300
500
Epochs
50
50
100
100
100
100
100
100
100
50
100
100
Table 4: Hyperparameters for globally trained CNNs.
the final linear layer, which was fine-tuned from its pre-trained initialization using the hyperparam- eters in Table 5. Majority voting was performed with neighborhood sizes specified in Table 6. All hyperparameters were obtained from hyperparameter optimization on the validation set. As the experiments were repeated over five seeds, we used 200 bootstrap iterations per seed to compute confidence intervals.
Model parameters
280
600
1268
1976
2985
4430
6386
11770
24294
40818
61342
85866
Learning rate
0.05
5e-3
0.01
0.01
5e-3
5e-3
le-3
5e-3
le-3
le-3
5e-3
le-3
Epochs
50
200
200
200
200
200
200
200
200
200
200
100
Neighbors
200
10
200
200
200
50
100
50
200
300
100
Table 5: Hyperparameters for TTT the linear head of the CNNs.
Model parameters
280
600
1268
1976
2985
4430
6386
11770
24294
40818
61342
85866
Neighbors
8
8
3
4
6
4
4
4
5
5
Table 6: Hyperparameters for majority voting based on last-hidden-layer features of the CNNs.
To generate the dataset scaling plot in Figure 5, we randomly subsampled the training set while explicitly ensuring a uniform distribution across the class labels. Here, TTT was performed by minimizing the cross-entropy loss over the k 80 nearest neighbors. Optimization was performed using Adam with a learning rate of 0.02 500 epochs.
F.5 CONNECTION TO MoEs
Given underparameterrized global model, natural alternative to specializing to each test-time task as in TTT is to instead specialize individual "expert" models to subsets of tasks, routing test-time tasks to few of these experts. Such mixture of experts (MoEs; Shazeer et al., 2017; Fedus et al., 2022: Bertolissi et al., 2025) have been shown to be an effective architecture for foundation models (e.g., Dai et al., 2024).
0.02
0.01
crror
global training
TTT
MaE
0.01
10a
10a
10a
10a
10a
To see whether our findings extend MoEs, we train multiple experts (each being different linear head) based on the MoE. architecture of Bertolissi al. (2025), and evaluate accuracy We find that a larger number of experts increases the ca- pacity of the model and improves accuracy, highlighting that MoEs are a promising approach to specialization without increasing inference cost.
number of experts
Classification error on MNIST. We compare the global classi- fier, TTT. and the MoE. Increasing the number of experts allows MoE to ap- proach TTT performance with inference cost comparable to the global model.
Figure
scale the number of experts.
We used a pre-trained CNN to extract last-layer embeddings from MNIST images. Following Bertolissi et al. (2025), for given number of experts, each expert associated with cluster
32

---

Published as a conference paper at ICLR 2026
Number of experts
3
10
30
100
300
1e3
3e3
10c3
20e3
50e3
Learning rate Epochs
6c-4
2c-4
6c-4
le-3
8c-4
4c-4
6c-4
4c-4
6e-4
4e-4
4c-4
2
1
1
2
3
10
30
20
50
40
Neighbors
60
50
30
60
30
40
30
20
30
30
20
Table 7: Hyperparameters for the mixture of experts (MoE) model based on last-hidden-layer fea- tures of the CNN.
centroid obtained by partitioning the training set into clusters via k-means. Expert training was per- formed by fine-tuning the pre-trained linear head on the nearest neighbors of the assigned centroid in embedding space. The fine-tuning hyperparameters are provided in Table 7, obtained by tuning on the validation set. At test time, inputs were first mapped through the CNN encoder, after which the single closest cluster centroid was selected based on Ly-distance. The corresponding expert head then produced the final prediction.
F.6 SCALING EXPERIMENTS ON IMAGENET
In our scaling experiments on ImageNet, we systematically vary the dimensionality and architecture of downstream classifiers, enabling a thorough analysis of scalability and representation quality. The following subsections detail our embedding extraction procedure, the dataset partitioning scheme, and the training protocols used for all baselines and scaling methods.
Embedding Extraction. For the ImageNet experiments, we use image embeddings derived from the CLIP vision-language model (Radford et al., 2021). Specifically, we employ the ViT-B/32 vari- ant of CLIP, as provided by the HuggingFace Transformers library. Images are first preprocessed by applying resizing, center cropping, and pixel normalization to match CLIP's training setup. The preprocessed images then passed through the CLIP vision encoder, yielding 512-dimensional representation vectors. To ensure stability and scale invariance, we normalize each embedding to unit length using the Ly norm.
Dataset Splits. Since ImageNet's official test labels are held out, we report results on the official validation set, treating it as our test set. For training purposes, we partitioned the 1.28M-image training set into a reduced training set and an artificial validation set. The artificial validation set was constructed by stratified sampling 50000 images to ensure a balanced class distribution, giving it the same size the official validation set. The remaining images were used for training. Unless otherwise stated, all reported results are based on evaluation on the official validation set.
Baseline Linear Classifier. As a first baseline, we trained a linear classifier on the 512- dimensional normalized CLIP embeddings. The linear head is fully connected layer mapping the embeddings to 1000 output classes, corresponding to the ImageNet labels. Training was performed with the Adam optimizer at a learning rate of 0.001, using batch size of 250 for 50 epochs. The model was trained with standard categorical cross-entropy loss, without additional regularization, and achieved test accuracy of 78.33 %.
TTT for Base Model. Building upon the baseline linear classifier, we apply test-time training procedure to adapt the model inference-time. Specifically, we fine-tune this linear head for each test sample by minimizing the cross-entropy loss over the set of & 600 nearest neighbors retrieved in the original CLIP embedding space. By optimizing the error rate on the validation set, we deter- mined that using k 600 neighbors with learning rate of 0.02. a batch size equal the number of neighbors, and 50 training epochs, combined with the Adam optimizer, perform near-optimally across all model scales. Consequently, adopted this universal set of hyperparameters for the ImageNet experiments.
Two-Layer MLP Projections. To assess the impact of embedding dimensionality. we trained two- layer multi- layer perceptrons (MLPs) classification heads top of the CLIP embeddings. The first hidden layer had variable size, ranging from 250 to 4000 neurons, followed by a ReLU acti- vation. The second layer maps the hidden representation to the 1000 ImageNet classes. Dropout is
33

---

Published as a conference paper at ICLR 2026
applied before the final layer to mitigate overfitting. Training conducted with the Adam opti- mizer, using the hyperparameters specified in Table 8, which were selected via optimization on the validation set.
Model parameters
3.8e5
7.6e5
1.1e6
1.5e6
1.9e6
2.3e6
3.0e6
3,8e6
4,5e6
6.1e6
Hidden dimension
250
500
750
1000
1250
1500
2000
2500
3000
4000
Learning Rate
4.0e-4
3.5e-4
3.0e-4
4.0e-4
3.5e-4
4.0e-4
4.0e-4
3.5e-4
4.5e-4
4.5e-4
Weight Decay
0
0
0
0
0
0
0
Batch Size
450
350
300
450
350
300
400
450
450
650
Num Epochs
50
50
50
50
50
50
50
50
50
50
Dropout Rate
0.05
0.25
0.3
0.35
0.45
0.55
0.6
0.65
0.7
0.7
Table 8: Hyperparameters for globally training the MLP heads across different model sizes.
After training, we freeze the first hidden layer and project the original 512-dimensional embeddings into this hidden space. The resulting hidden representations, taken after the ReLU activation, serve as our scaled embeddings for test-time training. Specifically, we fine-tune the pre- ceeitrinndd linear MLP head on the set of l nearest neighbors by minimizing the cross-entropy loss using Adam. This setup allows us to systematically examine how performance and representation quality vary with embedding dimensionality. The hyperparameters used for this procedure were obtained via hyperparameter optimization on the validation set, and we found that this universal set is nearly optimal across all model sizes, as reported in Table Loss optimizatiin was performed in a full- batch setting, with the batch size equal to the number of neighbors.
Hyperparameter
Value
Number of neighbors
100
Learning rate
5e-3
Batch size Epochs
100
50
Table Hyperparameters for TTT the linear head of the MLPs.
Majority Voting. As an alternative baseline, we leverage the learned feature spaces of the two- layer MLPs and apply simple majority voting protocol based on nearest neighbors. Specifically, first map the original 512-dimensional CLIP embeddings into the MLP's hidden feature space and then identify its k 10 nearest neighbors for any given test sample. Our empirical analysis showed that this neighborhood size performs well across all model scales. The predicted class is assigned the most frequent (majority) class label among these neighbors. This approach parallels the neighbor selection used test-time training (TTT) but replaces fine-tuning with a straightforward plurality vote. Majority voting thus serves simple, non-parametrie baseline to assess the quality of the scaled embeddings, providing insight into the clustering and class separability properties of the learned feature space.
F.7 SCALING EXPERIMENTS FOR LANGUAGE MODELING
We use the open-source implementation" 10 of Hardt Sun (2024) and evaluate the Qwen2.5 family of base models (Qwen al., 2025). We summarize hyperparameters in Table 10. During test-time training. we perform gradient steps sequentially for each sequence in the neighborhood. Sequences in the neighborhood that exceed the maximum sequence length are split into chunks of the maximum length. This means that a single neighbor can result in multiple gradient steps during TTT.
34

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Number of neighbors
50
Learning rate
2c-4
Adam's c-value
le-8
Max. sequence length in tokens
1024
LoRA rank
64
Table 10: Hyperparameters for language modeling on the Pile.
35


Reviews:
[{"review_id": "1rIBMU3meJ", "reviewer": "Reviewer_fGSc", "summary": "The authors conduct a comprehensive investigation into the potential of test-time training (TTT) to reduce in-distribution test error by leveraging only previously seen data. While TTT has traditionally been employed to address distribution shifts, this work explores its applicability to in-distribution scenarios. The authors first demonstrate that TTT encourages the model to converge to sparse solutions in the concept space. Building on this insight, they empirically show that TTT can effectively reduce in-distribution error rates on MNIST, ImageNet, and language modeling.", "questions": "Please refer to my weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of leveraging test-time training (TTT) to reduce in-distribution error is both novel and well-motivated, particularly given that most prior work on TTT has focused on addressing distribution shifts. The authors indeed demonstrate that TTT effectively reduces in-distribution test error compared to standard global training. \n- The analysis of the model’s representation space, grounded in the linear representation hypothesis, provides a solid conceptual foundation for understanding how models learn and encode semantic concepts.\n- The paper presents a rigorous theoretical analysis of the in-distribution test error behavior under TTT, which strengthens the overall contribution and supports the empirical findings.", "weaknesses": "- I believe the term “foundation model” requires clearer definition in the context of this paper. As shown in Figure 4, the effectiveness of test-time training (TTT) in reducing in-distribution error appears to diminish as model capacity increases. This observation raises concerns about the applicability of TTT to large-scale foundation models, which are typically defined as models with significantly larger parameter counts (e.g., exceeding 32 billion parameters). This concern is also acknowledged in Takeaway 1. However, in the introduction, the authors state that foundation models are underparameterized, which seems to contradict the commonly accepted understanding of the term. This inconsistency may lead to confusion for readers, and I believe a more precise and consistent definition of foundation model would help clarify the paper’s contributions and claims.\n- Although not a major concern, the experiment using MNIST with LeNet-5 appears somewhat outdated. While it is understandable that the authors aimed to demonstrate their method on a simple dataset and model, the use of a more recent dataset and architecture would have strengthened the empirical evaluation, particularly in light of the paper’s claims regarding foundation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U6iE1kJk11", "reviewer": "Reviewer_zPcj", "summary": "The paper studies the mechanisms of TTT in Foundation Models and its effectiveness in different contexts (the model capacity or dataset complexity). The work asks an important question of the TTT's effectiveness performed on the already seen data samples, which lets us focus on the process itself and not on the data. The claims made by the authors are clearly stated and supported by a thorough experimental section spanning experiments from image classification to NLP.", "questions": "1. Can we conclude that current LLMs, given their performance gains after TTT, are underparameterized models relative to their datasets? If so, why do the authors think that scaling the LLMs to even greater capacities ceased to be the main theme in LLM training, and we started to observe the diminished returns from scaling? \n\n\n2. How important is the metric used for generating the neighbourhood? The authors studied L2 distance, which was equivalent to cosine similarity, but there are multiple different ways to define the neighbourhood. Can we expect this metric to be, on average, the best approach, or would this choice be important when applying the approach to different datasets, modalities or architectures?\n\n3. Line 239-241: The Authors claim that the sparse TTT classifier is functionally equivalent based on 89% match (except for pathological cases). Could the authors elaborate on what \"pathological\" means? 11% seems to be relatively high and I don't understand where should I expect these two would deviate.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The work focuses on the extremely important topic, as the TTT plays an important role in the current cutting-edge research after showing its effectiveness in different domains; thus, I value the significance of the results highly. \n2. The idea to isolate the effect of TTT from the data (i.e. to study the effectiveness of TTT on already seen data) makes the analysis much more convincing to me. Still, I have to point out that I'm only vaguely familiar with this field, so my novelty judgment can be off. \n3. Both the presentation and the experimental design are thoughtful and help the reader to understand the main claims made by the authors, making it a solid piece of work with carefully balanced claims.", "weaknesses": "I did not find any important weaknesses in the paper. I have some questions which I will ask in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgfeev1keK", "reviewer": "Reviewer_1Gcj", "summary": "This paper investigates test-time training (TTT) and proposes that its effectiveness arises from enabling specialization after generalization, where models locally adapt to test data. Using the linear representation hypothesis, the authors model how TTT reweights a small subset of relevant concepts in an underparameterized feature space, achieving lower in-distribution error than global training. Empirical and theoretical results across vision and language tasks show that TTT offers the greatest gains for underparameterized models, with benefits diminishing as model capacity increases.", "questions": "Could the author elaborate on the definition of test-time training?", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a clear and formal connection between TTT and the linear representation hypothesis (LRH), providing an interpretable mathematical model for understanding why and when TTT improves performance.\n2. The use of SAEs to approximate the latent “concept space” offers a convincing and interpretable lens to study TTT’s behavior, even for readers (like myself) who are not deeply specialized in TTT.\n3. Experiments on both image (ImageNet, MNIST) and language (Pile) domains systematically support the theoretical claims, demonstrating consistent trends across scales and tasks.", "weaknesses": "As someone not specialized in test-time training, I found the exposition somewhat difficult to follow—the paper would benefit from a clearer introductory section that outlines core TTT mechanisms and variants before diving into the theoretical framing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQjSNVtHoh", "reviewer": "Reviewer_v2NY", "summary": "The paper attempts to answer why and when does test-time training (TTT) help foundation models even when test data are in-distribution? It assumes that foundation models are under-parameterized, and that TTT provides \"specialization after generalization\". Behind this is the Linear Representation Hypothesis (LRH): high-level \"concepts\" are sparse and linearly represented but superimposed onto a lower-dimensional feature space.\n\nThe authors reach 3 key observations: (O1) neighborhoods in learned features preserve the geometry from the concept space; (O2) each test neighborhood only depends on a few active concepts; (O3) standard TTT implicitly finds sparse solutions in concept space. Usinng sparse autoencoders and scaling studies on MNIST, ImageNet (with CLIP features), and The Pile (with LoRA), they show that TTT improves accuracy especially when models are small and under-parameterized, and that the benefit shrinks as models scale up in terms of number of parameters. Overall, the paper provides evidence that TTT’s success comes from reallocating model capacity locally.", "questions": "In the LoRA setup, the paper uses one gradient step. How was this chosen? Were multiple step sizes or LoRA ranks tested? Would they improve results?\n\nIn Appendix E.4, Table 4, different values are reported for the number of neighbors used for TTT on MNIST, depending on the model size. How were these chosen? In contrast, why was a fixed value of 100 used for ImageNet and of 50 for language modeling?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Comprehensive and helpful literature review. Text is well-written and structured. Figures are clear and support main claims.\n\nThe three observations (O1--O3) are backed up with both experiments and analysis.\n\nThe math connects with the experimental results and explains why global training can fail when features are mixed.\n\nUnderstanding when TTT helps (even with no distribution shift) is relevant for practical deployment of large models. The paper offers practical guidelines for scaling, such as how to choose neighborhood size and when TTT is likely to be worth using.", "weaknesses": "Several results depend on strong ideas like sparse concept spaces and preserved neighborhood geometry. It would help to test these assumptions on different architectures or layers to show they hold more generally.\n\nThe LoRA test-time training uses only one step and on a small subset (1%), 50 neighbors. Running more steps, larger splits, or other LLMs could show whether the findings scale.\n\nThe tests on images (LeNet-5 and pre-trained CLIP ViT) only update the last layer. Adding results with full end-to-end fine-tuning would make the findings more convincing.\n\nAlthough the paper does a good job at listing all its main assumptions, it mostly deals with them by giving arguments for why those assumptions might be reasonable. But the paper is lacking on discussing what could happen in scenarios where they do not hold. For example, TTT assumes neighborhoods correspond to a single uniform local concept. The authors should discuss what happens when that assumption breaks (e.g. due to label noise, or poor neighborhood size selection), since specialization in that case may lead to worse generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5GgLAvPdb", "reviewer": "Reviewer_V2Zv", "summary": "The authors analyze a model of test time training and argue that test-time training can provide benefits even in-distribution, so long as the model is capacity constrained.", "questions": "- In the theoretical framing, the relevant dimensionality seems to be the dimension of the latent space (compared to the number of \"concepts\" needed to solve the task). Despite this, in several places (e.g. Fig 4), the *total* number of parameters is used as a proxy for this dimensionality. Are there experiments you could run to disentangle these two different notions of dimensionality? For example, changing width/depth of MNIST models independently, do you see different behavior?\n- Can you clearly delineate the under/overparameterized regimes on Figs 4,5? It is not clear to me how much training data was used in each setting.\n- Is there a way to show that during TTT, the \"effective\" dataset size (during TTT we allow the model to perform poorly on data not in the current test set, so the effective dataset is small) becomes small enough that the model is again ~overparameterized? Can you connect this to double descent, at least in the MNIST setting? \n    - I am speaking loosely here, but the idea is that doing TTT in-distribution amounts roughly to retraining the model on a smaller dataset, which we know from the double descent literature has the potential to *improve* performance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The observation that TTT can help performance even for in-distribution inputs, and that this benefit is more pronounced in the underparameterized regime, is interesting, and to my knowledge, novel\n    - The authors show how the effect of TTT may be to 'disentangle' representations in an underparameterized representation space\n- The authors confirm their predictions with experiments across scales & domains: from MNIST to language models", "weaknesses": "- The authors consider TTT only in the regime where the last linear layer is retrained. While this does ensure that the representations learned during pretraining remain unchanged, it also puts a capacity constraint on the TTT model. Do the observations in the paper still hold up if we train more of the model at test time? \n    - I am particularly concerned that the phenomena in the paper may be a special case induced by the linearity of the final layer. I think even an experiment where TTT modifies the last 2 layers would go some way to mitigating this concern. \n- It is not clear to me where the under/overparameterized regimes begin and end. Could the authors note this on the figures? \n- In Figure 4, at least for the MNIST dataset, if the authors' hypothesis is correct, it should be possible to push to a regime where TTT yields no improvement. Is it feasible to extend the x-axis further, at least for the MNIST models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)