The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Authors: Ekin Akyürek
Mehul Damani
Adam Zweiger
Linlu Qiu
Han Guo
Jyothish Pari
Yoon Kim
Jacob Andreas
Venue: ICML 2025 poster (2025)
Keywords: Test-Time Training
In-Context Learning
Few-Shot Learning
ARC
BIG-Bench Hard
Primary area: deep_learning->large_language_models
TL;DR: Test-time training on in-context examples outperforms standard few-shot learning, boosting LM abstract reasoning and adaptability.

Abstract: Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks even when given a small number of in-context task examples. We investigate the effectiveness of test-time training (TTT)—temporarily updating model parameters during inference using a loss derived from input data—as a mechanism for improving LMs' reasoning and few-shot learning capabilities. On the Abstraction and Reasoning Corpus (ARC), performing TTT with in-context examples yields up to $6\times$ higher accuracy compared to fine-tuned baselines—reaching $53.0\%$ on the public validation set with an 8B-parameter LM and $61.9\%$ when ensembled with program-synthesis methods, matching average human performance. On BIG-Bench Hard (BBH), TTT on in-context examples surpasses standard few-shot prompting in the $10$-shot setting by $7.3$ percentage points ($50.5\%$ to $57.8\%$). Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Ekin Akyürek' Mehul Damani" 1
Adam Zweiger 1 Linlu Qiu Han Guo
Jyothish Pari
Yoon Kim
Jacob Andreas
Abstract
ARC
BBH
Language models (LMs) have shown impressive performance on tasks within their training distri- bution, but often struggle with structurally novel tasks even when given small number of in- context task examples. We investigate the effec- tiveness of test-time training ((TT)-temporanil updating model parameters during inference us- ing a loss derived from in-context examples- as a mechanism for improving LMs' reason- ing and few-shot learning capabilities. On the Abstraction and Reasoning Corpus (ARC), per- forming TTT with in-context examples yields up to 6x higher accuracy compared to fine-tuned baselines- reaching 53.0% on the public valida- tion set with an 8B-parameter I.M and 61.9% when ensembled with program-synthesis meth- ods, matching average human performance. On BIG-Bench Hard (BBH), TTT in-context ex- amples surpasses standard few-shot prompting in the 10-shot setting by 7.3 percentage points (50.5% to 57.8%). Our findings highlight the lim- itations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.
57.8%
50.5%
Accuracy
45.0 45.0%
Zero- "Shot
0.3-
0.2-
17.5%
0.1
0.0
FT
Base
TTT
Figure 1. Pass@2 accuracy on a subset of 80 randomly selected ARC validation tasks and overall accuracy on BIG- BIG-Bench Hard. The zero-shot baseline is 0 for ARC and 40.9% for BBH, indicated by the dashed line. TTT boosts the performance fine-tumed models (FT) ARC by 27.5 percentage points and increases accuracy on BBH by 7.3 percentage points.
trained-particularly tasks involving non-trivial reasoning, planning, and abstraction domains that differ significantly from their pre-training distributions. This question is funda- mental understanding how, and whether, LMS can exhibit the sort of flexible, novel-skill acquisition that has been proposed as measure of intelligence (Chollet, 2019).
Solving complex and novel tasks remains extremely chal- lenging for LMs, and simple sampling approaches often yield poor performance on such problems (Wu et al., 2024; McCoy et al., 2024). However, recent progress has shown that LMs can be substantially improved by adding extra test- time computation. Several methods fall into this category, such as chain-of-thought prompting (Wei et al., 2022), sam- pling with majority voting (self- self-consistencyy; Wang et al., 2023), code execution (Brown et al., 2024; Snell et al., 2025; Damani et al., 2025), and search (Yao et al., 2023).
1. Introduction
Large-scale neural language models (LMs) have demon- strated remarkable success few-shot learning of tasks related to those seen during pre-training, as well as ele- mentary variations or compositions of those tasks (Brown et al., 2020; Todd et al., 2024). When given natural lan- guage specifications or a small number of examples, LMs can often infer the desired task and generate appropriate out- puts. However, an open question whether these models can truly acquire new skills for which they have not been
The idea of updating model parameters using instance- specific data at test time has roots in the literature on local learning (Bottou & Vapnik, 1992) and transductive learn- ing (Joachims, 1999). In these methods, a learner refines its parameters or hypotheses after observing test inputs, adapt- ing to individual examples or small clusters examples. Such approaches inherently blur the line between training and inference, and can lead robust adaptation low-data
'quual contribution 'Massachusetts Institute of Technology. Correspondence to: Ekin Akyürek <akyurek @ mit.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
scenarios or under distribution shift.
ditioning on input examples or instructions provided. Given -. (Th yte and
sequence of input-output pairs (211 V1)....,
Modern versions of these transductive ideas for deep neural networks have been widely referred to as test-time train- ing. In TTT, model is updated at inference time using only the current test instance or a small batch of test in- stances, typically through explicit gradient steps. While test- time adaptation has been explored for vision models (Sun et al., 2020) and sequence architectures (Gandelsman al., 2022; Sun et al., 2024; Behrouz al., 2025), its interac- tion with other techniques for few-shot in-context learning-remains remains less understood.
new input 24+1, an LM can generate the corresponding output +k+1 by sampling from:
Wk+1 ~ LM(- F1, P1..1....... 2k, yk Tx+11
While the possibility of in-context learning (ICL) as im- plicit machine learning simulation is discussed in previous work (Akyürek et al., 2023), empirical evidence shows that in-context learning with language models does not always resemble standard machine learning algorithms (Zhao et al., 2024; Min et al., 2022b). Furthermore, ICL. often strug- gles with novel tasks "out-of-the-be For example, large language models exhibit poor performance on datasets like ARC (Opielka et al., 2024; Bober-Irizar & Banerjee, 2024).
In this paper, we investigate how to leverage TTT on top of standard in-context learning (ICL) to boost performance on challenging tasks that require reasoning or rule-based generalization. In-context learning is a powerful means of adaptation without parameter updates, guided by short, task-specific prompts. We show that combining ICI. with explicit gradient-based updates on test data can significantly improve performance on particularly difficult tasks. Specifi- cally, our main contributions' are:
2.2. Test-Time Training
Test-time training (TTT) enables parametric models to adapt during inference through dynamic parameter updates in re- sponse to each test input. This approach remains relatively unexplored in the era of large language models. The gen- eral TTT process is as follows: starting with initial model parameters Og. for each test input (or batch of inputs) d, we generate a temporary training dataset Drrr- We then optimize these parameters to minimize a loss function
1. A systematic analysis of the key components for effec- tive test-time training, including strategies for selecting training data at inference, training objectives, and how TTT interacts with an LM's pre-trained parameters and in-context learning.
2. An application of TTT to two challenging benchmark suites- The Abstraction and Reasoning Corpus (ARC; Chollet, 2019) and BIG-Bench Hard (BBH; Srivastava et al., 2023; Suzgun et al., 2023).
arg min
Σ L(LM(drr;;)),
dm
resulting in temporarily updated parameters 0J, which are subsequently used for prediction."
On ARC, our TTT approach outperforms existing open- source neural methods, attaining 53.0% accuracy with an 8B model and 61.9% when ensembled with a program- synthesis approach (comparable to human performance). On BBH, TTT yields 7.3% absolute improvement over few-shot prompting. achieving 57.8% accuracy. Gains are particularly large on tasks involving structural rules dis- tribution shifts (c.g., Dyck languages, Ruin names), where TTT yields 20-50 percentage points of improvement over standard in-context prompting.
In previous work (c.g., Sun et al., 2020), DrTT typically constructed by applying an unsupervised objective (e.g.. masked autoencoding) to the input alone. In this paper, we extend TTT to the few-shot learning setting, treating it as a form of transductive learning by leveraging few-shot demonstration examples to improve predictions. Although TTT can also be applied to chain of thought (CoT; Wei et al., 2022), we focus on direct transduction, where demonstra- tions consist of input-output pairs (x, y) without intermedi- ate reasoning steps explicit function descriptions.
Overall, our findings highlight that TTT drastically improves LM's few-shot learning ability on diitr---ittiittionn tasks.
The few-shot learning setting we consider provides richer context in the form of demonstration pairs (x1, y1).... (ZK. yk). One simple method for TTT is Di- rect I/O training, where we directly treat each input-output (I pair as training instances. Our key insight is that the few-shot examples can also be used to construct more
2. Preliminaries
2.1. In-context Learning
At a certain scale, many LMs exhibit the ability to adapt to new tasks without updating their parameters by simply con-
NNote that this use of "test-time training" is related but dis- tinct from the one used in recent line of work wherein an RNN's hidden state is treated as parameters and the update equation is interpreted as optimizing recall-based regression objective (Ravi & Larochelle, 2017; Sun al., 2024; Behrouz al., 2025; Wang et al., 2025).
'Code and data are available at https: thub. com/ekina kyurek/marc (ARC) and https: //gi thub. com/ adamzweiger /Fewshot-TTT (BBH).
2

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Data Generation
Test Task
3.1. Data Generation
Leave One Out
Direct I/O
LME Fu, Fai Fi
Given task with K training input-output pairs test-time training dataset DrTr can be created by either following an in-context learning setup or direct input-output (direct I/O) setup (top row in Figure 2):
LMI Ay, No Pa R.) LMI Fu. Fa.
LM
{( Ut ) k==1
Augmentations d((Fr,
LM
Horiiznnaal
Flip
Applied To
Selected
Leave-one-out tasks We begin with leave-one-out in- context learning tasks. For each pair (zz, Hj), we exclude it from the set of demonstrations and treat as "test" exam- ple within the newly formed synthetie task:
Loss oN:
Loss on: All Outputs
Loss on: Inputs Outputs
Test Output
Loss
1/1 12 1/2 f3 Ms
M/ 2/ 12 ff
Ms
3/1 d2g M/ 23
Ma
LM
LM
LM
27 M1 12
V2
IE3
1 91 22
V2 103
21 V1
22
V2 253
dic = ( (1(Z)) yk yk) Ty. vi).
Parameterization
Task Specific
Tasks
Shared
Tasks
Here, {(I }xys serves as the "in-context demonstra- tions," and (xy, Hi) is the "synthetic test example." " To in- crease the number of synthetic tasks, we additionally per- mute the order of the demonstrations in each dict.
Task 1
LMC | frm (D) g
Task 1 LM(- |
m) 8)
Task
LM prrm 199 )
Task 2 LM drrr en 0)
Task LM(: drr (8] (44)
Task LM(- 1777
(A3 B)
Direct onput-output (I/O) tasks Rather than constructing in co-toneet tasks, we treat each (Ix, Uk) pair independently as a single training instance:
Figure TTT design decisions. Data gencration: lest task consists of input-output pairs {(x., W.)). ). The Leave-One-Out strat- egy removes one example at a time to form in-context learning tasks, while augmentations further expand the dataset. An alter- native Direct I/O approach trains directly on the examples. Loss: The model is trained with loss computed the Test Output (only the test-time prediction), All Outputs (including demonstration outputs), or Imputs and Outputs (all tokens). Parametrization: The Task- Task-Specific approach trains a separate adapter per task while the Shared approach trains a single adapter across multiple tasks.
d (z))
In this setup, the model fine-tuned on these training pairs without in-context demonstrations. While this approach is more computationally efficient, our results (Sections 4 and 5) show that it underperforms methods that utilize in- context demonstrations.
Data augmentation For certain tasks with structured inputs (e.g., ARC), we can apply invertible transformations (e.g.. flips, rotations, color permutations) to further augment the TTT dataset. Let T be set of invertible transformations. For each IET, we have (1(x)) = 2, so we can apply / to each training and test instance in dj to yield a transformed task (dd)). Since these transformations preserve the core relationships in the data (c.g.. the input-output pattern is the same, just rotated), they effectively expand the training signal. If rule-based transformations are used, the final TTT dataset is: Drrr ULET U, t(dz).
robust and expansive DTTT of synthetic in-context learn- ing tasks, allowing for effective model adaptation during test time. Additionally, when task-specific knowledge is available, this structure can be leveraged to further expand ARC
the dataset, as demonstrated in our experiments (Section 4). We also explore the general case where no task- specific information is used, as tested on BBH (Section 5).
Our experiments in this paper characterize each compo- nent of the TTT pipeline, investigating different design choices across the following stages: (1) constructing an input-specific training dataset DrTT test-time; (2) fine- tuiing the LM by optimizing loss function L over the dataset Le ((LM(d; 0)); and (3) sampling from the updated model with an augmented inference strategy based on self-consistency to obtain a final prediction.
3.2. Loss Function
We optimize the standard LM loss on Drir. For the in- context leave-one-out setup, experiment with 3 different ways to take the loss (middle row in Figure 2):
3. TTT Design
Test output (no demonstration loss) The standard for- mulation where the loss is taken over ytest:
This section discusses the key design choices and challenges of applying TTT to LLMs, including how to best leverage their in-context learning capabilities, how to structure data for effective processing, what optimization objective to use, and how to efficiently update model parameters. We detail these considerations in the construction of the TTT dataset and the optimization setup (Figure 2).
label L
LIM( F1, 91 ... ......KKK yK Treati 0)
All outputs In addition to the loss on the test output,
'For ARC, we start the indexing at & 2 because the under- lying transformation of ARC task cannot he inferred without observing at least demonstration.
3

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
the loss is also taken over the outputs of the in-context demonstrations, which encourages the model to correctly predict the demonstration outputs after seeing the previous demonstrations:
In-Context Examples
Test
Model Predictions
ARC
K
LM
Couiputs
clabel
LIM(UR/Z1: TAi 0)
?
E
k=1
Loss on inputs and outputs The loss is taken over all tokens, encouraging the model to learn the structure of a well y::
(( >1 )>
(<(0g) H
< >)0(0
}1
BBH
K
Figure 3. Example of ARC and BBH tasks that the model success- fully solves only after applying TTT.
CLM
coutputs
+ LLM (1, Vi...., Vk-1; 0)
4.2. Experimental Details
This method, which requires learners to generate task imputs well outputs, is analogous to existing unsu- pervised TTT objectives (Sun et al., 2020).
Model architecture & optimization For our abla- tion experiments, we use the 1B-paramett Llama-3.2 model (Llama Team, 2024). For our final results in Sec- tion 4.6, we use the 8B Llama 3 model. We use Low-Rank Adaptation (LoRA; Hu et al., 2022) for parameter-efficient test-time training. More details are given in Appendix C.2.
We find in Sections 4.3 and 5.3 that the first method (taking the loss over both demonstration and test outputs) works best.
Fine-tuning before TTT While TTT offers task-specific adaptation, the initial capabilities of the base model signif- icantly influence its final performance (Section 4.4), We developed several approaches for generating synthetic train- ing data to enhance the base model's abstract reasoning capabilities through fine-tuning, exploring both automated and semi-automated methods for task generation. This is complementary to TTT the base model is fine-tuned on tasks distinct from those tested on, when TTT is applied. De- tails on our data generation strategies, as well as the effects of various data sources and model sizes on performance, are provided in Appendix B. The fine-tuned base model serves as the foundation for all subsequent experiments.
3.3. Parametrization
Once we have the test-time training dataset Drrr (con- structed via either the in-context or direct I/O approach), we perform a small number of gradient steps on task-specific LoRA adapters (Hu et al., 2022). This approach allows computationally efficient adaptation while maintaining the learn task-
model's general capabilities. By default, specific LoRA adapters for each ARC or BBH task at test- time. That is, we obtain K different LoRA adapters, where K is the number of test tasks. We also experiment with using a single shared LoRA adapter from the aggregated dataset of few-shot examples drawn from multiple tasks (bottom row in Figure 2 test-time version of meta-ICL. (Min al., 2022a). We find that the shared adapter degrades performance on ARC, whereas it improves performance BBH. We discuss this in more detail in Section 5.3.
The success criterion requires producing an exact match for all test outputs (no partial credit). Following the standard ARC scoring criteria, we the pass @2 met- ric and produce 2 attempts for each test input. The original training and validation sets consist of 400 tasks each. How- ever, for efficient evaluation purposes, we randomly pick 80 balanced ARC tasks from the ARC validation set, including easy, 20 medium, 20 hard, expert tasks according to the classification in (LeGris et al., 2024) (see Appendix A.2 for this task list). Except for our final results, we use this subset of ARC tasks throughout our experiments. We limit Drrr to have maximum of 250 examples per task for ef- ficiency reasons. Appendix C.2 provides additional details on the hyperparameters.
Evaluation
4. A Abstraction and Reasoning Corpus
4.1. Background
The Abstraction and Reasoning Corpus (ARC) aims to eval- uate the abstract reasoning capabilities of language models through their ability to solve visual puzzles. Each puzzle (henceforth referred to task) consists of input-output pairs of 2D grids (up to 30 x 30 in size) containing shapes or patterns in up to 10 different colors, as displayed in Figure 3. The output of each pair is obtained by applying an intuitive and shared transformation or rule y = f(x. Each task has 2-7 demonstration examples and 1-3 test examples.
Inference
One of the most common techniques to scale inference-time compute is to use temperature sampling to

---

The Surprising Effectiveness of Time Training for Few-Shot Learning
Hararchic Majonit
Data Ablations
Optimization Ablation
30-
29
26
Test Task
22
Tasks Solved
20-
13
10
Quelky
Pale Deved Augrerisees
11
FT TT
Ne Trat
Dorect
Sharel T77
Ne Dean
Figure 4. Augmented inference and hierarchical voting. We leave-one-out tasks and invertible geometric transformations to obtain multiple equivalent versions of the task for augmented inference. Predictions from these versions aggregated with hierarchical voting strategy: first, voting is performed within each transformation, and then the top candidates from each transforma- tion undergo global voting to yield the top two predictions.
Figure 5. Accuracy of different data and optimization ablations in TTT on ARC. Our data ablations reveal that the ICL data for- mat is crucial for effective TTT. and that applying transformations to augment the TTT dataset notably enhances performance. For optimization, learning task-specific adapters significantly outper- forms using single adapter and taking loss on the in-context demonstrations provides minor performance boost.
obtain multiple responses and select the best according to a ranker, called self-coonistency (Wang et al., 2023). How- ever, this is not viable in ARC (where the output grid is directly predicted) there way to directly enforce diversity across samples while ensuring coherence within samples. As an alternative self-consistency approach, we try an augmented inference strategy that combines greedy decoding with multiple versions of the input. Specifically. we generate multiple prediction candidates by using geomet- ric transformations. We then employ a hierarchical voting strategy to determine the final prediction from the of generated candidates. This approach involves two stages of voting progressively narrow down the best candidates: (I) Intra-transformatten voting: Group predictions by their corresponding transformation t. Within each group, select the top-3 most frequent predictions. (2) Global voting: Take the selected wwwafformation-sppeiff candidates from the previous step and select the top-2 most frequent predic- tions across all transformations. The augmented inference pipeline is summarized in Figure 4 and full details of the pipeline are in Appendix E.
35
PT
PI+
30
Tasks Solved
25
+22
20
+18
+24
15
10
11 14
5
0
Figure 6. Performance results across model sizes. Fine-tuned model performance improves with increasing size. However, the scaling behavior after TTT is less clear. For instance, the final performance the and models is identical after TTT.
portant; using the direct input-output data construct Drrr causes 1 11-task drop (38%). Removing transformations causes a 16 task drop (55%). Regarding optimization, per- task LoRA adapters outperform single shared adapter by 7 tasks (24%). Including losses on the demonstration outputs yields a modest but consistent gain (26% - 29%).
4.3. Impact of TTT Design
In this section, we compare the final implementation of our method with different design choices for TTT. FT serves as the baseline, using only the fine-tuned model with demon- strations in-context. No Transformations omits the aug- mentation step. Direct I/O Data replaces in-context tasks with the direct input-output task formulation (Section 3.1). Shared TTT uses single LoRA adapter across all tasks instead of learning one per task. No Demonstration Loss removes the loss on demonstration outputs (Section 3.)..
4.4. Impact of Model Size
We perform full fine-tuning of IB and 3B Llama 3.2 (instruction-tumm and 8B Llama 3 (instruction-tunel) us- ing synthetically generated data, as detailed Appendix B, and then use our default TTT implementation. We show results using different model sizes in Figure Increasing the model size consistently improves FT performance, with the 8B model achieving the highest accuracy of 36%. At all model sizes, TTT leads to significant improvements in performance. We also observe that for smaller model sizes, TTT effectively closes the performance gap, with the IB and 3B models achieving similar accuracy after TTT.
Results are presented in Figure 5. Our TTT method effec- tive, improving fine-tuned model accuracy approximately 6x (5% + 29%). In-context formatting is especially im-
5

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Individual
Aggregated
32
PS
Fine-tuned LM
TTT Method
Score
30
29
Tasks Solved
27
Ours
X
18.3%
Ours
Ours
47.1%
20
18
X BARC
BARC
Ours
53.0%
Ours
Ours
58.5%
BARC
BARC
Ours
62.8%
10
Avg. Human
60.2%
Best Human
97.8%
BARC (ensemble)
54.4%
Rotate
Transpone
Flip
Vanila
Flattoped
Hierarchical
Oracle
BARC (no synthesizer)
39.3%
Claude 3.5 Sonnet
21.0%
GPT-4o
9.0%
Figure? 7. Accuracy of different transformations and voting schemes, While individual transformations generally perform modest level and are comparable one another, aggregat- ing across them through voting yields substantial improvements. Notably, hierarchical voting strategy with voting stages sur- passes a flat voting approach. Our hierarchical method approaches oracle-level performance, demonstrating effectiveness accu- rately selecting the correct answer when present.
OpenAl ol -peeveew
21.0%
DeepSeck rl
20.5%
OpenAl 03-preview 82.8%
Table I. Pass@2 Scores of different systems the ARC vali- dation set. Our TTT pipeline improves base models consistently. We achieve 17. % accuracy when applied and 53.0% when applied to the BARC model (Li et al., 2025). We ensemble our method with program synthesis (PS) based models, where we achieve score of 61.9%, comparable to the average hu- man performance of 60.2%.
fine-tuned model
4.5. Impact of Augmented Inference
To analyze the impact of augmented inference and voting, we run several ablations: (1) Vanilla, which generates two predictions without transformations or advanced voting: (2) Transformed Inference, applying a single transformation (Rotate, Transpose, or Flip) to measure its isolated effect; (3) Hierarchical Voting, our full pipeline combining aug- mented inference and structured voting; (4) Flattened Vot- ing, which selects the top-2 predictions from a single voting round over all generated outputs; and (5) Oracle, an upper bound that selects the correct answer if present.
racy by combining neural and program synthesis approaches. While their fully neural approach shares similarities with our system, our TTT and inference pipeline has several ad- ditional components (per-task LoRA, more augmentations, hierarchical voting) that boost performance. To validate our improvements, we applied our TTT pipeline to BARC's fully neural model, achieving 53.0% accuracy- 35% im- provement over their original TTT method.
Building on these results, we explored combinations of our approach with BARC. Combining our TTT pipeline and neural model with BARC's synthesizer raised accuracy to 58.5% Combining our TTT pipeline with BARC's neural model and synthesizer raised accuracy to 61.9% 1.. This configuration matches average human performance of 60.2% (LeGris et al., 2024) on the benchmark.
As shown, individual transformations are modestly effective on their own (with Transpose performing worst), but their aggregation improves results markedly. Hierarchical voting further outperforms a flattened voting approach and closely approaches the oracle's accuracy, suggesting that our two- stage aggregation effectively identifies the correct solution when it is present.
Comparing program generation and end-to-end model- ing Li et al. (2025) found that program synthesis and fully neural predictors for ARC are highly complementary. Their end-to-end neural model can only solve 42.2% of the tasks solved by the program synthesis model. However, we find that when equipped with our TTT pipeline, BARC's fine- tuned fully neural model solves 73.5% of the tasks that solved by the program synthesis model. This suggests that our TTT pipeline significantly improves the neural model's ability to learn systematic reasoning patterns similar to those captured by program synthesis models.
4.6. Comparison to Other Systems
Following our experiments on 80 tasks, we present com- prehensive results on the full ARC public evaluation set, comparing our system against existing approaches. Our analysis focuses three key aspects: the impact of our TTT methodology, the benefits of combining our approach with existing methods, and the differences between fully neural and program synthesis methods.
TTT
We apply TTT and augmented inference procedure to our base fine-tuned model (fine-tuned 8B model). TTT significantly improves accuracy from 18.3% to 47.1%.
Semi-private evaluation ARC-AGI challenge provides a hidden "semi-private dataset" and performs external tests for submissions. We submitted our ensemble solution to the official ARC-AGI semi-private evaluation and observed 47.5% accuracy. This decline may be attributed to more
Integration with existing methods A concurren work by Li et al. (2025) introduced BARC, achieving 54.4% accu-
6

---

The Surprising Effectiveness of Time Training for Few-Shot Learning
significant distribution shifts in the semi-private evaluation dataset. A more detailed analysis of these performance differences can be conducted in the future once the semi- private set is publicly released.
Data Ablations
Optimization Ablations
60-
57.8
55.7
59.8
53.9
Accuracy (%)
50.5
31.5
40.9
-0
5. BIG-Bench Hard
20-
5.1. Background
10-
BIG-Bench Hard (BBH; Srivastava et al., 2023; Suzgun al., 2023) is benchmark comprising 27 challenging tasks across 23 task types, designed to evaluate large language models on reasoning, compositionality, and generalization. Unlike ARC, BBH features a broader natural language struc- ture and lacks a shared input format, making it unsuitable for invertible transformations. However, this broader scope offers valuable testbed for evaluating TTT's effectiveness in more generalized setting. Despite the absence of invert- ible transformations paee-ousevvvooeeee used in ARC to expand the TTT dataset and enhance inference-TTT still significantly improves performance on BBH.
Zone-hoot
rct.
r
Ne Perm iotonnationll
Direct kd
Shared PTT
No Dame Lons
Les on Toyaus and
Figure 8. Overall BIG-Bench Hard Results. TTT outperforms standard in- in-tontext learning by 7.3 absolute percentage points, from 50.5% to 57.8%. Our performance improvement over direct input-output data shows that using in- in-context leave-one-oue tasks is crucial. Not taking demonstration loss taking loss on inputs results in a performance decrease. Unlike with ARC, using a shared adapter across all tasks improves performance.
benefit the most from TTT (Section 5.4). Unlike with ARC, we do not have collection of invertible transformations to run augmented inference. Instead, we use greedy decoding. Further hyperparameter details and evaluation details are given in Appendix F.2.
5.2. Experimental Details
Model architecture & optimization We use Llama 3.1 (8B; Llama Team, 2024). For each task d, we train a separate set of LoRA parameters at test-time, with a LoRA rank of 64 over 40 random shuffles of the demonstration pairs to produce leave-one-out in-context tasks. More hy- perparameter details are given in Appendix F.I.
5.3. Impact of TTT Design
In this section, we evaluate our method and its ablations, primarily comparing the zero-shot baseline, ICL, and TTT. No Example Permutation updates the model on a single in-context prompt instead of multiple shuffled versions. Di- rect I/O treats each input-output pair as separate training instances. Shared TTT uses single adapter across tasks instead of task-specific adapters. No Demonstration Loss removes the loss applied to demonstration outputs. Loss on Inputs and Outputs extends the loss calculation to both inputs and outputs. These ablations are as detailed in Section 3. As these results are averages over 5 runs, the standard errors of the mean for each method are given in Appendix F. averaging 0.4%.
On BIG-Bench Hard, our base language model is able to achieve non-trivial scores out-of-the-box. Consequently, we do not perform any initial fine-tuning on synthetic tasks outside of BBH like we do for ARC. Furthermore, since models achieve nonzero performance in a zero-shot setting, we provide the zero-shot results and analyze how TTT and ICL improve upon them.
For the 27 tasks in BBH, we consider the 10- shot setting, where we select 10 random pairs from each task's dataset to be demonstration pairs and evaluate on the remaining data. Each of the 27 tasks is analogous to single ARC task, consisting of 10 labeled examples as demonstration pairs given at test-time. We report average results over five random seeds, where each seed specifies which 10 examples form the demonstration subset. For more control over the evaluation process with test-time training, we write our own evaluation function, which is available in our codebase (for more details, see Appendix F. The number of evaluation examples for each task is then 240 for all tasks except three: Causal Judgment, Penguins in Table, and Snarks, which have 177, 136, and 168 evaluation examples respectively. Note that the large number of evalua- tion samples for each task compared to ARC means we can do a task-specific analysis to analyze which types of tasks
Evaluation
The results in Figure 8 show that TTT achieves overall ac- curacy of 57.8%, outperforming standard ICL (50.5%) and Direct I/O learning (51.5%). This demonstrates that TTT's capabilities extend beyond ARC to more diverse and com- plex reasoning tasks, proving its effectiveness in a broader range of natural language problem-solving scenarios.
We observe that TTT without example permuta- tions- Mooss-peroorming multiple gradient steps on single in-context prompt before inference-reduce accuracy to a still-impressive 55.7%. Computing the loss only on the test output lowers accuracy to 54.4%, while applying it to both inputs and outputs achieves 55.9%.
7

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
100
align well with TTT's ability to adapt latent structural regularities during test-time.
Zeco Shot
31.0
Conversely, tasks requiring explicit step-by-step computa- tion show limited gains with TTT. For instance, Boolean Expressions declined from 85.7% to 80.4% under TTT. This task's algorithmic nature-dependent on sequential reason- ing rather than mattern-based transduction and its likely pre-training exposure suggest TTT's updates may not re- solve its specific demands. While these particular obser- vations align with our hypothesis, the reason certain tasks benefit more from TTT remains an open question.
Accuracy (%)
53.4
60.5
$3.2
20
18.3
3.5
0-
Dyck Lungung wx
Ruin Names
Movie Roco inmendation
Hlyperbaton
Boolean Ex preseions
6. Related Work
Figure 9. BIG-Bench Hard results for tasks with the largest TTT-CC score differences. The four tasks on the left show the most significant improvements with TTT over ICL, while the task on the right has the lowest TTT score relative to ICL, Full task- specific results are given in Appendix F.2.
Test-time training The idea of updating model param- eters test-time using instance-specific data traces back to early work on local learning (Bottou & Vapnik, 1992). More recently, Sun et al. (2020) propose simple test-time self-supervision scheme to adapt an image classifier when facing distribution shifts. In language modeling, Hardt & Sun (2024) fine-tune on retrieved neighbors at test-time for notable gains, while Hübotter et al. (2025) optimize retrieval via active data selection.
Shared adapter Unlike on ARC, using a shared adapter improves performance on BBH, indicating that tasks in BBH do not confound each other during training. On the ARC dataset, each puzzle has the same input format, so distin- guishing among multiple tasks is difficult, and we may have conflicting gradients with a single adapter. In BBH, however, distinguishing tasks is trivial (the instructions differ in plain text), and many tasks are mutually helpful. For instance, updating on Logical Deduction Five Objects also aids Logi- cal Deduction Three Objects, without hurting Word Sorting- Although this is no longer test-time training distinct tasks presented individually at test time, it can be interpreted as TTT on the entire dataset presented collectively at test time.
ARC challenge Abstraction and Reasoning Corpus (ARC; Chollet, 2019; Chollet et al., 2025) is collection of ex- tremely challenging few-shot visual reasoning problems. Most approaches to ARC fall into two main categories: program synthesis and fully neural. Program synthesis ap- proaches (Butt et al., 2024; Wang et al., 2024; Li et al., 2025; Greenblatt, 2024) first try to find the transformation function f. and then apply it the test example. Fully neural ap- proaches (Veldkamp al., 2023; Bober-Irizar & Banerjee, 2024) try to directly predict the output y yesr, only implicitly modeling f. In this work, we use fully neural approach, using an LM to predict the test outputs. Recent work has explored hybrid methods, leveraging inference scaling and program synthesis (Greenblatt, 2024; Li al., 2025). Similarly, we find that integrating our neural model with program synthesis improves performance.
5.4. Task-Specific Analysis
Our task-specific results show that performance improve- ments from TTT are highly dask-depen. Among the 27 tasks in BBH, TTT results in a performance decline of at least 2% compared to ICL in only 2 tasks. In contrast, 12 tasks show an improvement of at least 2%, with of these showing improvements of at least 5%. The four tasks with the most significant performance boost from TTT over ICL or zero-shot and the task with the most significant per- formance decrease are shown in Figure 9. These tasks in order of TTT's improvement over ICI. are Dyck Languages (parentheses matching). Ruin Names (humorous name modi- fications), Movie Recommendation (choosing similar films), Hyperbaton (adjective ordering), and Boolean Expression (evaluating a boolean expression). Detailed results for every task are given in Appendix F.2.
deep learning-guiee
7. Conclusion
We conduct an investigation of test-time training and demon- strate that it can significantly improve LM performance on abstract reasoning and few-shot learning tasks, namely the Abstraction and Reasoning Corpus (ARC) and BIG-Bench Hard (BBH). Our key contributions include a robust TTT framework with leave-one-out in-context task construction, the optimization setup, and the inference strategy after TTT. Our results reveal the potential of TTT to tackle novel rea- soning tasks, suggesting significant promise for test-time methods in advancing the next generation of LMs.
We hypothesize that improvements from TTT may be driven by tasks involving distribution shifts and structured patterns. For example, tasks like Dyck Languages and Hyperbaton fol- low clear grammatical or programmatic rules, which could
8

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Limitations
memorize at test time, 2025. URL https://arxiv.. org/ abs//5011 .00663.
Optimization bias In development ARC, we used a set of 80 tasks for validation/ablation experiments. Standard syper-parameters (learning rate, epochs) were optimized using this set, which might have introduced some bias.
Bober-Irizar, M. and Banerjee, S. Neural networks for abstraction and reasoning. Scientific Reports, 2024. ISSN 2045-2322. doi: 010008///41598--02-7733888888 URL https::/doi.org/// http://6600..////1110000///11133
Data leakage While the base Llama-3 performs poorly the public validation set of ARC, the public availability of the dataset introduces the possibility that these models may have seen these examples during pre-trate pre-training. Similarly, while the base model achieves reasonable performance on BBH, its public availability raises similar concerns.
Bottou, L. and Vapnik, V. Local learning algorithms. Neural Computation, 1992. ISSN 0899-7667. doi: 10.1162/neco .1992.4.6.888. URL
1162/neco
.1992.4.6. 6. 888.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https://arxiv wwwpxiiv.rgg/aae wttp/rr////aa/ae/oo 21787.
Acknowledgments
We sincerely thank the BARC team (Li et al., 2025) for their support and collaboration in ensembling our method with theirs, resulting in an official joint submission to the ARC public set. We thank Aniruddha Nrusimha for helpful discussions on parameter efficient training. This work was supported by MIT-IBM Watson AT Lab, the MIT Quest for Intelligence, and by the National Science Foundation under grants IIS 22,310, IIS IIS-2238244 2238240, and CCF-221 7064. JA is additionally supported by a Sloan Research Fellowship. This work also benefited from many conversations during the Simons Institute Program on Language Models and Transformers.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M. Kaplan, J., Dhariwal. P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C.,, McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, 2020. URL https: / //proceedings. neurips Abst444--bbttttt..... html. 7c0d6bfcb49674 4atteerfbb4477144bbbabcc
Impact Statement
Butt, N., Manczak, B., Wiggers, A., Rainone, C., Zhang, D. W., Defferrard, M., and Cohen, T. Codelt: Self- improving language models with prioritized hindsight replay. In Proceedings of the 41 International Confer- ence Machine Learning. PMLR. 2024. URL https: //dl. acm. wwt. 5555/3692770 3692267.
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
References
Chollet, F. On the measure of intelligence, 2019. URL https://arxivv wwt.ri..org/ab//19//// /
Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G. E., Nye, M. I., Tessler, M. H., and Tenen- baum, J. Communicating natural programs humans and machines. In Advances in Neural Information Pro- cessing Systems 35, 2022. URL http: http:://pppprr..iissss cc/paper httpe///iieee/ee////////////aaa/ chmarks.html.
Chollet, F., Knoop, M., Kamradt, G., and Landers, B. ARC Prize 2024: Technical report, 2025. URL https://arxi V. w.org/abs/24/ 246.24444..
Damani, M., Shenfeld, I., Peng, A., Bobu, A., and Andreas, J. Learning how hard to think: Input-adaptive allocation of LM computation. In The Thirteenth International Conference Learning Representations, 2025. URL https://openrevii http://ppeneeeeeeeeeeeerorrroooo www..net/oorum/ddd//gggwubbbb
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh Inter- national Conference on Learning Representations, 2023. URL. wtwphhttts://pprreeeee https//s//.pppeeei..eeee///////a wwt..e//pooPiiddagoxxxeea.. I.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMSs. In Advances in Neural Information Processing Systems 36,
2023. URL htt.htt::/ppperr..nnn cc/ cwtpppcr.pppppeeeeeeeeeeeee fflle//p
aprr//823/sss///f668888111331110 49b-Abstract--onferenee.
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. A. Test- time training with masked autoencoders. In Advances in Neural Information Processing Systems 35, 2022. URL http://ccccc22200444333666666762 act-Conference. html.
Li, W.-D., Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Zheng, W.-L.., Pu, Y., and Ellis, K. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL. https://op enreview. net/ wtw.wiewwnnt/oormmiiieeeeeee htt.//rommmmmmmmmoooooo tAVVe.
Greenblatt, R. Getting 50% (SoTA) on ARC-AGI with GPT- 4o, 2024. URL https: retps://redwoodresearchh substa ck. wtt.////ettingggggg//eee -on-innn-------o------- aacta---n-rr--ii---- gpt. Accessed 1--144.
Llama Team. The Llama 3 herd of models, 2024. URL https://arxiaa.
21783. 22772
Loshchilov, I. and Hutter, F. Fixing weight decay regular- ization in Adam, 2018. URL https: /oppnr/iee//..eiee.... (t/forum?id=rk6@ddee"
Hardt, M. and Sun, Y. Test-time training on nearest neigh- bors for large language models. The Twelfth Interna- tional Conference on Learning Representations, 2024. URL https://opeenevieww.ne wttp://wwwnee//ooomm//oooooo ra.
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., and Griffiths, L. Embers of autoregression show how large language models are shaped by the problem they are trained solve. Proceedings of the National Academy of Sciences, 2024. doi: 10. 00077///mas.22222212222 URL 1073/ 23
Hodel, M. Addressing the Abstraction and Reasoning Corpus via procedural example generation, 2024. URL https://arxiv. wttp://////....000/// 07353.
https: /wwww//wwwwwwwwwwtw www.wpppa.....mmmmm 22420121.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. URL https://openreviee www.eneeeeeeeeeeooom/iiiiiii hett/forum/dd=nZeeeeeeel
Min, S., Lewis, M., Zettlemoyer, L.,, and Hajishirzi, H. MetalCI: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2022a. doi: 10.18653/ http:///2222..nallnnneaa mai n.201. URL https::/acllanthoooog// org/202 naac 1-main. 201.
Hübotter, J.. Bongni, S., Hakimi, I., and Krause, A. Effi- ciently learning at test-time: Active fine-tuning of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttts:///ppereviiee net /forum?id-NS1G1Uhnyy)
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022b. doi: 10.18653/v 1/2022.eemllp-mall 759. URL https://aclanthotn . .org/2022, emn.2-2annm..-mii..........m 759.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. GPT-4o system card. ArXiv preprint, 2024. URL httph: //axiiv httpaaawww...oogbssss/sss aaaaaab//11 2410. 21276.
Joachims, T. Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., 1999 ISBN 1558606122
OpenAL. GPT-4 technical report, 2024. URL https://ar
xiv. siv.orr/abss/233. 08774.
Opielka, G., Rosenbusch, H., Vijverberg, V., and Steven- son, C. E. Do large language models solve ARC vi- sual analogies like people do?, 2024. URL https: //arxiv. wwexiv.orr/aae herrg//bb/2423... 09734.
Kwon, W., Li, 7., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serv- ing with PagedAAtention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23. Association for Computing Machinery, 2023. ISBN 9798400702297. doi: 10.1 141144//000006336133333 URL https://doi.org org/10. . 0144//660000636660 3613165.
Ravi, S. and Larochelle, H. Optimization model for few-shot learning. In The Fifth International Conference on Learning Representations, 2017. URL https://op enreview. www..neet/orrmmidddroo-o-oooo
LeGris, S., Vong. W. K., Lake, B. M., and Gureckis, T. M. H-ARC: A robust estimate of human performance on the Abstraction and Reasoning Corpus benchmark. ArXiv preprint, 2024. URL attps://aarkis http//axvvvogg//// http:///aaxvv..ooaa//////e/ org/abs//449.. 1374.
Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling test- time compute optimally can be more effective than scal- ing LLM parameters. In The Thirteenth International Conference on Learning Representations, 2025. URL http://ppeerreiiie.eee/rrrrr/aaa
10

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httph: //tps///veee/ee........ et St//oormmidduyyTL5BBess
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview. iwieerreieewnnet/Fooummiii
Wei, J.. Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, E., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, 2022. URL http:: nips cc/pap er eesffeeeeepppperr/2/22/h///fffff 4524eefff
Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning. PMLR. 2020. URL http://prrocediing. mlr. mln.press/ v119/sun20b. html.
html.
Sun, Y., Li, X., Dalal, K., Xu, J.. Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., and Guestrin, C. Learning to (learn test time): RNNs with expressive hidden states, 2024. URL https: //arxiv. .rgg /wwp///7..aae. 04620..
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2024. URL https: hatp::///alaathhllogyyy org /2024. naacl-long. 102.
Suzgun, M.. Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J. Challenging BIG-Bench tasks and whether chain-of-thougle can solve them. In Findings of the 2023 Conference of the Association for Computational Linguis- tics. Association for Computational Linguistics, 2023. doi: 18.15 1/2023. http:////.....iinnaaaa.2.222222. URL https: / //aclanthology wrwpeey..rr////// w02.. www...f....n.s--..e.e....aa findings-acll 824.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36, 2023. URL http: /tapers nips. cc cc/ pc/pappr/app//////cpee//ee/eeeep filees/pape///2/22/// hash/271db99222b33 f aae584e5a77733 act act-Conference.u html.
Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpss///ppereeeew..rr /forum?id=AwyytyyWWa..
Zhao, S., Nguyen, T., and Grover, A. Probing the decision boundaries of in-context learning in large language mod- els. In ICML 2024 Workshop on In-Context Learning, 2024. URL https: http:///ppeeeeeeneeeeeee/ wwwp:/e//foofffdfrrr.rrmo CtCcPuSt.
torchtune Maintainers and Contributors. torchtune: Py- Torch's finetuning library, 2024. URL https:
.com//ytoorcch/toccttnne.
Veldkamp, K., Rosenbusch, H., Thoms, L., and Stevenson, C. Solving ARC visual analogies with neural embeddings and vector arithmetic: A generalized method. OSF, 2023. doi: N0.17665/0SF..0//KKP5 URL /tots:://sf.... io /akp86/.
Wang, K. A., Shi, J., and Fox, E. B. Test-time regression: unifying framework for designing sequence models with associative memory. ArXiv preprint, 2025. URL https://arrxe orgo arg/abs/2501. 12352.
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. Hypothesis Search: Inductive reason- ing with language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviere wwweeeewwwwewweeeeeeeeeeeeeeeeee htt.://eene/orruu///de/////.ooo

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
A. ARC Dataset
present the tasks in the development set, the data format and evaluation details for the ARC dataset (available at this htips link.).
A.1. Data Format
We use numpy's array printing format for all experiments as shown in Figure 10.
A.2. List of 80 Tasks Used For Development
We use the following (Table 2) tasks validation tasks for our development.
Table 2. Selected development tasks and their hardness level based on (LeGris et al., 2024).
ID
Level
ID
Level
ID
Level ID
Level
0ald4efS
easy
762cd429
medium
e5c44e8f
hard
e99362f0
expert
692cd3b6
easy
e7639916
medium
604001fa
hard
lacc24af
expert
1da012fc
easy eld2900e
medium
4364c1c4
hard 19a67cb5
expert
66e6c45b
easy ace291af
medium
506d28a5
hard
ad7e01d0
expert
3194b014
easy c95c3d8c
medium
2037f2c7
hard
ca9794b1
expert
963f59bc
casy e0fb751
medium
d5c634a2
hard
58c15b12
expert
d37alef5
casy
ac58858c
medium
ac605ebb
hard
891232d6
expert
358ba94e
easy 93c31fbe
medium
27f8ce4f
hard
5833af48
expert
Bede58f
easy
27a77e38
medium
66f2d22r
hard
4ff4c9da
expert
55059096
easy 9bebae7a
medium
3ed85e70
hard 5b692c0f
expert
c7d4c6ad
easy 9ddd0or0
medium
8b28cd80
hard e2092c0c
expert
4b6b68e5
easy fe937213
medium
d19r7514
hard 47996f11
expert
00576224
easy
69889d6e
medium
dc2aa30b
hard 34b99a2b
expert
a04h2602
easy 15663ba9
medium
f5c89df1
hard 1c56ad9f
expent
e9c9d9a1
easy 17h80ad2
medium
50f325b5
hard efid6688f
expert
ef26cbf6
easy 16b78196
medium
08573cc6
hard fca12743
expert
7eelo6ea
easy 5b6cbet5
medium
3d31c5b3
hard 31d5bala
expert
e9ac8c9e
easy 40f6cd08
medium
94133066
hard
791b0314
expert
la2e2828
easy 505t1184
medium
136b0064
hard
87191442
expert
770cc55f
casy d017b73f
medium
90347967
hard a8610ef7
expert
A.3. Evaluation
We follow the competition rules that if any of the two pass @2 predictions of the system is correct, we consider that test correct. In the reported task-level accuracies, we did not give partial points if all tests are not solved, except the final table Section 4.6.
[[O 0 0
[[0 0 0
str
1 0
str
3 4 0
0 0 0]]
0]]
Figure Data Format: convert grids strings by representing them as numpy arrays of digits from 0 to where each digit corresponds to a different color.
12

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Examples Generators Annotations from the training set
Generated Generators Annotations
Sammory
Reccotig Unne
the bas we undomb paree be and wes
bescterse
LLM
of renermadessr flost, Lec (ltett ctet:
perrctionn rind, caff ant (lae) dictt
can Seance (2, 300
din wuso Ca
Blacke
calopta astorvalle, a6,
oc tan
sureen sape
ceirteliurt yn. Jin moondo
bacigriend
zanden
recure ("inacts "axtpet": 203
reaam ("least: E. 'eaipet" go1
Figure 11. LLM based synthetic tasks generation: Given some seed task descriptions and task generator functions in Python, generate more generator functions to produce novel tasks. We use three different approaches: (1) few-shot prompting with only generators, (2) few-shot prompting with generators and task descriptions, (3) two-stage approach: first generate free form descriptions, then condition on them to generate more generators (shown in Figure 12).
B. Fine-FTnuunn Tine-Tuning Before TTT
While test-time training facilitates task-specific adaptation, the base model's capabilities impacts the final performance. We developed several approaches for generating synthetic training data to enhance the base model's ' abstract reasoning capabilities through fine-tuning, exploring both automated and semi-automated methods for task generation. this section, we detail our fine-tuning data generation strategies and analyze the impact of different data sources and model sizes on final performance.
B.1. Preparing Fine-tuning Data
(Hodel, 2024) provides domain- speeiiic language (DSL), REARC, as well as the transformation fa that solves the task-i, and the data generation function y, that implemented in this DSL for each training task in the Durin dataset. These functions enable sampling of new input-output pairs that maintains the same underlying transformation principle:
d = (ax y) ~ cval(gs)
(I)
where d represents a newly generated input-output pair that can be solved using the same transformation function fi as the original task-?".
(a) Using Existing Generators by producing different instantiations of same tasks. We generate extra samples from these training tasks by running the code many times and randomly splitting these new examples (d ~ eval(g:)) to a set of train and test examples. These augmented examples are already provided with their DSL. release.
The generator functions g in REARC already provide effective data augmentation tool
(b) Fho-shot Prompting an LLM Additionally, we used several approaches to generate novel tasks using an LM (in our case, an ensemble of GPT4 and GPT4-o).
The simplest approach generates new task generators using few-shot examples:
g' LM(g1, 92...............
(2)
where g' is a new generator function and ..,..., Im are existing generator functions (shown in Figure 11). We sample different m examples by uniformly from existing training set. We repeat this process multiple times get a good amount of tasks.
We augment the generator functions with task descriptions and jointly generate both descriptions and generators:
(s'.g') ~ LM(S1, 91, $2, 92... Sm, 9m
(3)
where s; represents the description of task
*e can verify the generated examples hy asserting fi(x) = y-
13

---

The Surprising Effectiveness of Temt Training for Few-Shot Learning
Examples Cenerstors Amnotations from training set
frea
(1vo stogle generation))
Ganwretee Gonerators
Examples Annotations from the training set
Augmentation
THES FAACI
LLM
LLM
IMTIHIAANE AITANN COL BE EIL TOMON CAEY
Denerated Annotations
Figure 12. Two-stage generation using LLM: First, prompt the L.IM to generate a task description using few-shot prompting. Then, we generate the new generator based on existing task pairs and the newly created description.
To get the task descriptions, we manually created seed descriptions for 10 training tasks. These seed descriptions were then used to generate descriptions for the training and validation tasks through few-shot prompting. To increase diversity of tasks, we use task descriptions with hierarchical fields (category, summary, and description). The process of getting these descriptions is provided in Appendix D.I.
Instead of jointly generating task descriptions and function generations, additionally deployed two-stage approach (Figure 12) described as following:
(M1, 8m:
(4)
LM( 91: $2, $2, ...., Sms Im, s')
(5)
This approach first generates a task description s' and then conditions the generator creation both existing task pairs and the new description. In total we collected 6426 generators with these LLM based approaches. We provide qualitative samples from these LM generated tasks in Figure 16.
(c) Geometric Transformations Finally, our synthetic tasks are enhanced through various geometric transformations, such basic transformations (rotations, reflections, random shift and size scaling), pattern operations (random patching. tiling, and repetition), color permutations, and composite transformations involving sequential application of multiple basic transformations. These transformations are applied in three ways:
Input grids only: (z, v) + (f(x), v)
Output grids only: (z, v) (I, (g))
Both input and output: (I, + (t(z),t(y)
We use all the transformations given in Appendix C.I, and some additional transformations given in Table In the fine-tuning case, different from TTT, we apply augmentations only inputs, only outputs or both. These transformations are applied randomly to variants of tasks with 30% of the time.
Table 3. We provide the additional augmentations use in our data generation for fine-tuning with their function signature and description.
Augmentation Name
Description
Repeat (direction, n)
Rotates a grid in horizontal or vertical direction by times.
DropoutOutput DropoutInput
Randomly deletes some patches of the output grids.
Randomly deletes some patches of the input grids
14

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
35
FT
FT
30
29
Tasks Solved
Tasks Solved
25-
24
+22
20
+24
+24
15
+15
10
14
+9
No FT
All
No-Geam
No-LM
1B
3B
8B
Figure 13. Left: Accuracy when fine-tuning with different data sources. While all fine-tuned models perform similarly, their performance after TTT shows considerable variance. expected, removing geometric transformations from the fine-tuning data reduces performance compared to the model trained on the full dataset. Surprisingly, excluding LM- A.M-gnneratc data from finc-tuning actually outperforms the model trained on data. Right: Performance results across different model sizes. expected, performance of the base fine-tuned model improves with increasing model size, aligning with current scaling law trends. However, the scaling behavior after and models is identical after TTT. Full discussion Section B.3.
TTT less clear. instance, final performance
B.2. ARC Initial Fine-tuning Hyperparameters
We perform full fine-tuning on LLama-3 family models by using the torchtune library. We train each model up to 16000 steps. We use 2xNVIDIA A100 GPU for IB models, 4xNVIDIA A100 GPU for 3B and 8B models. We present hyperparameters in Table 4.
Table 4. ARC Initial Fine-tuning Hyperparameters
Hyperparameter Search Space
learning rate
2.5e-5
epochs
2
batch size
32
optimizer
A (Loshchilov Hutter, 2018)
scheduler
Cosine LR Schedule with 2k warmup
B.3. Results
We perform full fine-tuning IB, 3B Llama 3.2 instruction-tuned, and 8B Llama instruction tuned using augmented data. The format and training objective is same as the ones described for TTT in 3. Hyperparameter details are given in Appendix C.2. We do the following ablations for augmented data:
1. No FT: The original Llama 3 instruction-tumed model without any fine-tuning.
2. All: We use all methods described in Section B.I, including REARC, rule-based augmentation, and LM generation.
3. No-Geom: We remove geometric transformations from all tasks.
4. No-LM: We only REARC and rule-based augmentation, excluding tasks generated by the LM.
We show results using different model sizes in Figure 13. Increasing the model size consistently improves FT performance, with the 8B model achieving the highest accuracy of 36%. We also observe that TTT effectively closes the performance gap for smaller models, with the and 3B models achieving similar accuracy after TTT.
15

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
C. TTT Transformations for ARC
present the transformations used in TTT and the training details.
X
Test-Time Training Data
Idanity
Yo
V1
V4
Test Task
Xa
Hortzontal
?
TTT Label
X;
Vertical
Fiip
?
V1
Step-1:
Step-2:
Leave-one Tasks
Rule Based Augmentations
Figure 14. TTT dataset generation for test task (Section 3.1): We start by creating leave-one-out tasks from the given training examples of the task. These tasks then augmented through rule-based transformations obtain the full TTT dataset. Finally, we train specifc LoRA adapters top of the base FT model.
C.1. Transformations
We provide the augmentations used in TTT in Appendix C.I, please refer to code base for their implementations, After applying these augmentations, we additionally shuffle colors and shuffle training examples. Note that these transformations are applied to all input and output grids. The procedure for generating the dataset for TTT is shown in Figure 14.
C.2. Training Setup & Hyperparameters
We use the torchtune(torchuue models. We apply LoRA training to query and value projection weights of the self-atteniin layer, to the MLP weights and to the output projection layer (was only available for Llama-3 8B in torchtune). We present hyperparameters of this training in Table 6. We also found that using quantized LoRA adapters (Dettmers et al., 2023) instead of standard (full full-precision) LoRA leads to only small drop performance (29 - 26 tasks solved with the IB-parameter model), making a viable option in cemmory-onstrrained settings.
Maintainers & Contributors, 2024) library to train LoRA adapters Llama-3 family of
We resort to the vLLM (Kwon al., 2023) library for prediction it provides fast kernels and batched inference for models and LoRA inference. We just use greed decoding as we did not see improvements with temperature sampling in our early experiments. We use 90, 180 degree rotations, horizontal, vertical, and diagonal (transpose) flips our invertible transformations.
With that, the whole TTT and inference process takes approximately 12 hours for 100 randomly sampled validation tasks when using an NVIDIA A100 GPU.
16

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Table 5. We provide the augmentations used in TTT procedure with their function signature and description.
Augmentation Name
Description
Rotate(90)
Rotates a grid 90 degrees.
Rotate(270)
Rotates a grid -90 degrees.
Rotate(180)
Rotates grid 180 degrees.
Flip(0)
Flips grid horizontally
F1ip(1)
Flips grid vertically
Reflect(0, reverse=True)
Flips grid horizontally and prepends to the left of the
original grid
(1 reverse=True)
Flips a grid vertically and prepends to above the original
grid
Reflect(0, reverse=False)
Flips a grid horizontally and appends the right of the
original grid
Reflect(1, reverse=False)
Flips grid vertically and appends to below the original grid
RandomTraanslatexY))
Shifts a grid randomly in the borizontal and vertical direc- tions. The maximum shift size is
Transpose IncreasefResolution())
Reflects a grid on diagonal Upscales the grid by interleaving elements in both hori- zontal and vertical directions Upscales the grid by interleaving elements in vertical di- rection
(ncreasetHeiggh(())
Increasewidth((2
Upscales the grid by interleaving elements in horizontal direction
( (ocrreroonnnccssssssssttttt(111)
Sequential application
Rotate (90) and
IncreaseResolution()
( Inceaaaessesssssstiii((((()1)11)
Sequential application of Rotate(270) and IncreaseRes-
olution(2)
(hain((Rotate(11)) (harreceennnncceeeesessooottoooi
Sequential application of (otate(180)) and IncreaseRes-
olution(2)
Sequential application of Rotate (180) and IncreaseRes-
olution(2)
(hain((FFip(t), (hcrsiesil))nccasssssssslllll)))
Sequential application of Rotate (180) and IncreaseRes- olution(2)
, (); (
Sequential application of Rotate (180) and IncreaseRes- olution(2)
Table ARC TTT Hyperparameters. We find learning rate of Se-5 the best for IB and 3B models, and le-4 the best for 8B models.
Hyperparameter Search Space
r LoRA rank
128
a LoRA alpha
16
learning rate epochs
[5e-5, 1e-4)
2
batch size
[1.
optimizer
AdamW (Loshchilov & Hutter, 2018)
17

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
D. LM Data Generation
described three approaches Appendix B to use LM, we generated 6426 task generators by few-shot prompting GPT-4 and GPT-4o models (OpenAI, 2024: Hurst al., 2024).
D.I. Getting Descriptions for Tasks
This procedure is shown in Figure 15, We initially described 10 training tasks with the hicrarchical-style shown in Figure 11. Then, for other training tasks tasks, we obtained less quality crowd-wor worker annotations from LARC (Acquaviva al., 2022) project. By using our high-quality seed annotations and their LARC version, 10-shot prompt and LM produce high quality annotations for the other training tasks.
You are an intelligent agent that can induce task descriptions from examples. For Category, please *do not use generic terms like Transformation, Pattern Recognition.
Task: {stringified task inputs and outputs}
LARC Description: fdescription of the task-1 1 from LARC dataset) Good Description: (hierarchical description)
[truncated]
Task: fstringified task inputs and outputs for task-K}
LARC Description: !description of the task-K from LARC dataset)
Good Description: (hierarchical description]
Task: /stringified task inputs and outputs for query task]
LARC Description:
[description of the query task from LARC dataset)
D.2. Few-shot Prompting Details
We use the following simple prompting template with k-shot prompting for all data generation procedures, where numbers filled with examples sampled from seed set. In simple few-shot generation, we exclude examples. We use GPT-4 and GPT-4o generate the new scripts.
You are problem generator on 2D grids of colors. Here are some examples of such transformations, please follow the format:
Example:
(description of the generator function-1]
Script: (generator function-1)
[truncated]
Example: (description of the generator function-K( Script: {generator function-K)
Please generate more and make sure they are different:
18

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
10 seed Manually Created and Crowdisurced Amnotatonns
Exampe with Crowdsourced Annotations (from LARC)
Example with Itefined Armosations
Che MEC of
LLM
Wetere and oove
Figure 15. Generating quality seed descriptions: We use few-shot prompting to generate descriptions for given task, using 10 manually created seed descriptions along with crowd-worker annotations from Acquaviva et al. (2022) few-shot examples. given new task, we similarly provide the LM with examples and crowd- worker annotations (available only for training tasks).
Invalid
Valid
Figure 16. Example tasks generated by LM data augmentation procedure: We display three reasonable tasks that we can infer simple transformation (valid), and three tasks that could not infer simple transformation (invalid).
19

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
E. Augmented Inference Pipeline
E.1. Augmented Inference
Recent work has shown that scaling test-time compute can significantly improve the performance LMs. One of the most common techniques do this is by sampling multiple responses, and then selecting the best response using a ranker. However, while sampling is very effective in domains with multiple possible solutions (programs in code) or multiple possible paths the final answer (math), it can be detrimental when generating answers directly, there is no way directly enforce diversity across samples while ensuring coherence within samples. As alternative inference-time scaling, we use an augmented inference strategy that generates multiple prediction candidates by using geometric transformations, combined with a greedy decoding scheme.
For given task with training examples (zy, yk v=)k=n and test input Tiests we use invertible geometric transformations to produce equivalent transformed versions of the task, as shown in Figure 5. Let T be some set set of invertible geometric transformations ge. rotations and reflections). For each transformation t T. we apply all training demonstrations and the test input and run our model with these transformed inputs. We then apply the inverse transformation to obtain the final prediction for that transformation.
LM(+ ((ppt))))))) =
(((e))))
(6)
1/2 (
(7)
We further augment our predictions by permuting the order of training examples. For each transformation g. sample different permutations of the demonstration sequence, resulting in n [7] total predictions per task. This is to mitigate any bias in the model's processing of the demonstration sequence. (Bober-Irizze & Banerjee, 2024) also find transpose and rotation is helpful to produce extra prediction candidates.
E.2. Ensembling Predictions (Voting Strategy)
We employ a hicrarchical voting strategy to determine the final prediction from the set of candidates {v]1177 This approach involves two stages of voting to progressively narrow down the best candidates: first, by selecting the most frequent predictions within each transformation, and then by conducting overall vote across transformation-specffee candidates identify the t2p-2 most frequent predictions. The details of each stage are as follows:
1. Intra Transformation Voting:
We group predictions by their corresponding transformation and select the top-3 most frequent predictions within each group. If fewer than 3 unique predictions exist within a group, we supplement the candidates by computing additional predictions through:
Row-based majority: For each row in the predicted output grid, we take the most frequent row values across all predictions in the transformation group.
Column- majority: Similarly, for each column the predicted output grid, we take the most frequent column values across all predictions in the transformation group.
2. Global Voting: Using the selected transformation-speeiin candidates obtained from (1), we conduct an overall vote to select the top-2 most frequent predictions for submission. In case of tie, predictions with the identity transformation are given priority.
20

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
F. BIG-Bench Hard Details
F.I Further Experimental Details
We write our own evaluation function for BIG-Bench Hard available in our codebase. We found that existing evaluation frameworks did not properly measure zero-shot performance due to insufficient answer-extraction parsing and maswer-format prompting. We also wanted evaluation sets. For all results, we average results over different selections of the 10 few-shot examples with the following 5 random seeds: 42, 43, 44, 45, 46. The full TTT and inference process takes approximately 15 minutes on an NVIDIA A100 GPU.
control in splitting each individual task's dataset into demonstration examples and
The standard error of the mean for each method in Figure 8 over the seeds is given in Table 7.
Table Standard Error the Mcan for each method Figure
Method
Standard Error of the Mean
Zero-Shot
0.01
ICL
0.19
TTT
0.20
No Example Permutation
0.32
E2E
0.66
Shared TTT
0.72
No Demo Loss
0.69
Loss on Inputs and Outputs
0.35
We search the following hyperparametens:
Table 8. BBH TTT Fine-tuning Hyperparameters
Hyperparameter Search Space
learning rate
[le-5, 5e-5, 1e-4, 3e-4] [64, 128]
LoRA rank
LoRA alpha
[16, 32, 64, 128]
epochs
batch size
5
training steps
[20, 40, 60]
optimizer
AdamW
scheduler
Cosine LR Schedule
We similarly use the torchtune(torchtuee Maintainers Contributors, 2024) library for test-time training and the vLLM (Kwon al., 2023) library for inference.
F.2. Task-specific Results
The full results for all tasks over all methods and ablations are shown in Figure 17.
21

---

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
Figure 17. sask-specific 10-shot results for cach BIG-Bench Hard task, averaged over random sccds.
Range for TTT accuracy ICL Accuracy
Tasks (Count)
I task total:
x x -5
Boolean Expressions
1 task total:
<<<x < -2
Penguins In A Table
13 tasks total:
Causal Judgement, Disambiguation Qa, Formal Fallacies,
Logical Deduction Five Objects.
Logical Deduction Seven Objects, Logical Deduction Three Objects, Multistep Arithmetic Two, Navigate, Reasoning About Colored Objects,
<2<x<2
Salient Translation Error Detection, Sports Understanding, Web Of Lies, Word Sorting
3 tasks total:
<<
Object Counting. Tracking Shuffled Objects Five Objects.
Tracking Shuffled Objects Three Objects
9 tasks total:
Date Understunding, Dyck Languages, Geometric Shapes, Hyperbaton, Movie Recommendation, Ruin Names, Snarks, Temporal Sequences.
2 028 5
Tracking Shuffled Objects Seven Objects
Table 9, Tasks Categorized by the Difference TTT Accuracy ICL Accuracy.
22


Reviews:
[{"review_id": "qb21uzvRfy", "reviewer": "Reviewer_nbYh", "summary": "This paper proposes to use test-time-training as a method for scaling test-time capabilities of large models under the few-shot setting, and tested the model's performance on ARC and BBH bechmarks. The experiment results show positive performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Have authors tried domains such as coding or math?**\nThis paper produced a SoTA way to apply TTT to LMs in the few-shot learning setting using the challenging ARC-AGI and BB-Hard as our reasoning problem sets. Note that our implementations do not leverage the CoT capabilities of the models. We believe extending this to non-few-shot learning settings and to domains where CoT is crucial is a very exciting research problem and we’re currently exploring these extensions.\n\n**Q2. Recent findings show RL-trained models generalize better. Can TTT still be advantageous if the base model is strongly RL-trained?** \nThat is a great question! Recently, there has been a lot of interest in using RL to unlock long reasoning abilities in LMs. As mentioned in our previous answer, combining TTT with CoT models (both RL or no RL)  is a promising future direction! Similarly, another possible extension is **Test-Time-RL**, where RL is incorporated into the test-time training process ([Simonds and Yoshiyama](https://arxiv.org/abs/2503.00735), 2025). \n\n**Q3. Many benefits disappear when actual scaling happens, how will this affect TTT?**\nIn Section 4.4, we present scaling results for Llama models of 1B, 3B, and 8B parameters, where TTT improves performance by 480%, 163% and 157% respectively. Thus, TTT scales effectively with increasing model size. \n\n\n**Q4. Augmentations makes the method a little complicated**\nOur experiments on the Big-Bench Hard (BBH) suite of tasks show that the core method of fewshot TTT without any task-specific voting mechanisms or data augmentations still yields significant improvements. Moreover, our vanilla method (Fig. 7), which excludes any augmentations or voting procedures, still achieves a substantial performance boost—solving 22 tasks compared to just 5 (no TTT), marking a 440% improvement.", "claims_and_evidence": "Main claim: TTT helps few-shot performance compared to In Context Learning. The claim is very well supported by various ablation studies in BBH and ARC.", "methods_and_evaluation_criteria": "Methods\n\n(1) Three methods for TTT: leave-one-out TTT, direct train TTT, augmentated data for TTT. These three are ablated in the experiments\n\n(2) For training loss: test loss + all output loss + per-token loss.\n\nEvaluation: ablation on different methods, losses, and metrics come from the benchmark. So it's sound and solid.", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "The authors clearly discussed and ablated the model components.", "relation_to_broader_scientific_literature": "This would be interesting results to the \"general reasoning\" audience.", "essential_references_not_discussed": "No key missing references", "other_strengths_and_weaknesses": "Very well written paper.", "other_comments_or_suggestions": "The paper did use many data-aug or voting mechanisms to make it work better. It'll be cleaner and cooler if there's no need for those.", "questions_for_authors": "I wonder whether authors have tried other domains such as coding or algebraic / math questions? \n\nRecent findings show RL-trained models generalizes better, do the authors think TTT can still hold its advantage if the base model is strongly RL-trained, or the baseline is not FT but actually RL finetuned?\n\nAlso, many benefits disappear when actual scaling happens, not sure what will happen for this TTT finding.\n\n(I think the paper's finding is interesting overall. Just curious about the authors' thoughts on the above questions.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbIVI3x2Y7", "reviewer": "Reviewer_LZSp", "summary": "This paper investigates test-time training (TTT) for improving language models' few-shot learning capabilities, particularly on novel tasks that require reasoning and abstraction.\n\n- TTT significantly improves performance on challenging reasoning tasks, e.g. on ARC-AGI, TTT with in-context examples yields up to 6× higher accuracy (53.0%) compared to fine-tuned baselines, and reaches 61.9% when ensembled with program synthesis methods.\n- On BIG-Bench Hard (BBH), TTT improves performance by 7.3 percentage points over standard few-shot prompting (50.5% to 57.8%).\n\n\n## Update after rebuttal\n\nI maintain my rating as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review! We are happy to answer any new questions you have later during the rebuttal process.", "claims_and_evidence": "Nothing to report.", "methods_and_evaluation_criteria": "Nothing to report.", "theoretical_claims": "Nothing to report.", "experimental_designs_or_analyses": "Nothing to report.", "relation_to_broader_scientific_literature": "Nothing to report.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqySDIl8S7", "reviewer": "Reviewer_fvyu", "summary": "This paper presents a comprehensive analysis of the Abstraction and Reasoning Corpus (ARC) and BIG-Bench Hard (BBH) tasks. The authors train LoRA as an adapter. For training, the authors use flips, rotations, and color permutations to augment the original training data. During inference, they adopt intra-transformation voting and global voting strategies. Finally, they achieve improvements of 28% and 7% on ARC and BBH, respectively.\n\n## Update\nThanks for authors' detailed response. I appreciate authors re-state their contribution more clearly. This paper primarily focus on data augmentation and in-context fine-tuning. I believe that with further explanation and clarification, this paper will be much clearer. However, as for the submitted version, there are some missing important points, including augmented data statics, lora baselines comparison, more clearly statement of contribution, and so on (see above).\nI will maintain my score. I think this paper is not suitable for publication at this time", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and the feedback on our paper. \n\n**Q1. Definition and Application of Test-Time Training (TTT)**\n\nWe use the definition of TTT provided by ([Sun et al. (2020)](https://arxiv.org/abs/1909.13231)): self-supervised training a model using the unlabeled test sample $d_{\\textrm{input}}$. Our work applies this definition in the few-shot learning setting where $d_{\\textrm{input}}$ consists of instance-specific few-shot demonstrations and a query. The core idea remains: adapting model parameters at inference time based on information available for the specific test input being processed. \n\nFinally, LoRA is simply the means by which we implement TTT in this paper. it is a simple and efficient fine-tuning method that enables us to do TTT with a 8B-size model. In practice, our TTT framework can be used with other fine-tuning techniques as well. \n\nWe can clarify more if the reviewer has more questions.\n\n\n**Q2. Most of the improvements come from inference strategies**\n\nWe kindly disagree. Our experiments on the Big-Bench Hard (BBH) show that the TTT without any task-specific voting mechanisms or data augmentations still yields significant improvements. Moreover, our vanilla method (Fig. 7), which **excludes any augmentations or voting procedures**, still achieves a substantial performance boost—solving 22 tasks compared to just 5 (no TTT), marking a 440% improvement.\n\n**Q3. Can you clarify the contributions and significance?**\n\nThank you for asking for clarification, and we will update the text to reflect our contributions.\n\nWe presented a successful extension of the TTT paradigm to the few-shot learning setting and did a systematic study of design choices for this setting. To the best of our knowledge, we are the first to apply TTT within the in-context learning framework. \n\nWe showed substantial improvements (e.g., up to 6x on ARC, 7.3pp on BBH) over strong baselines on 2 popular benchmarks. The magnitude of these improvements highlight the limitations of standard ICL and opens up a new avenue for LM research with TTT. \n\n**Q4. TTT Dataset Statistics**\n\nThe size of the dynamically generated $D_\\textrm{TTT}$ depends on the number of demonstrations (K) and the number of augmentations/permutations used. \n\nFor ARC, we use $K$ demonstrations (typically 2-7), generate $K$ leave-one-out tasks, apply $|T|$ geometric transformations (Section C.1 lists 20 base transformations/compositions), and n=2 permutations (line 1066), leading to roughly $K * |T| * n$ synthetic examples per task, capped at 250 (line 202). For BBH, we use $K=10$ demonstrations and $n=40$ permutations (line 344), resulting in $10 * 40 = 400$ leave-one-out examples per task. \n\nWe will add these specific calculations/details to Appendix C.2 (for ARC) and F.1 (for BBH) to clarify the scale of the temporary training data used for TTT.", "claims_and_evidence": "There are some over-claims or unclear definitions here that need further clarification:\n1. Test-time training: creating a self-supervised learning problem based on this single test samplex, updating θ at test time before making a prediction (quoted from [1]). This paper adopt a lora to train on trianing data, then using several inference scaling methods to test. **The defintion of TTT is different with the main method in this paper.**\n2. The Lora is a parameter-efficient fine-tuning methods. Authors use this method as baselines. However, I believe more details should be provided regarding the inference methods used in the baseline approach. Most of the improvements come from the data augmentation and voting strategies presented in the paper.\n3. **It should be clearly stated in contribution that this paper employed existing techniques to conduct a comprehensive analysis on ARC and BBH, including data augmentation and voting strategies.  This paper does not propose novel methods, theoretical results, or insights.**\n---\n[1] Sun Y, Wang X, Liu Z, et al. Test-time training with self-supervision for generalization under distribution shifts[C]//International conference on machine learning. PMLR, 2020: 9229-9248.", "methods_and_evaluation_criteria": "This paper doesn't propose new methods. However, their empirical analysis is thorough and demonstrates significant effectiveness. Their evaluation criteria is reasonable.", "theoretical_claims": "This paper does not provide theoretical results for insights.", "experimental_designs_or_analyses": "Their empirical analysis is reasonable and comprehensive. My main concern is that the baseline should be compared using the same inference methods.", "relation_to_broader_scientific_literature": "This paper is not related to test-time training. They use LoRA, a parameter-efficient fine-tuning method, rather than conducting training during inference.", "essential_references_not_discussed": "This published paper proposed training a lora during the inference time, which is related to this paper.\n[2] Wang Y, Ma D, Cai D. With greater text comes greater necessity: Inference-time training helps long text generation[J]. arXiv preprint arXiv:2401.11504, 2024.", "other_strengths_and_weaknesses": "More theoretical analysis is necessary for out-of-distribution tasks, especially for the ARC and BBH tasks studied in this paper.", "other_comments_or_suggestions": "The formulation differences between LoRA and test-time training (TTT) should be explicitly clarified.", "questions_for_authors": "Please provide dataset statistics for LoRA training, such as the dataset sizes obtained after each type of augmentation, to better analyze the sources of actual performance improvements.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes using test-time training (TTT) to improve LLM reasoning and few-shot learning on tasks that require reasoning or rule-based generalization. The authors demonstrate that TTT, which involves updating model parameters during inference, enhances performance on the ARC and BBH benchmarks.

The paper shows strong empirical results, with TTT improving fine-tuned model accuracy on ARC by approximately 6x and increasing accuracy on BBH by 7.3 percentage points.  The paper is well-written and the claims are generally well-supported by the evidence provided.

One reviewer argues that the paper does not propose novel methods, theoretical results, or insights, and that the improvements mainly come from data augmentation and voting strategies. The authors replied that they are the first to apply TTT within the in-context learning framework, and that the combination and systematic study of TTT in this setting is novel.   

Overall, the paper demonstrates the effectiveness of test-time training in improving few-shot learning capabilities. The authors addressed reviewer concerns in their rebuttal. Hopefully, the authors can address Reviewer fvyu 's comments on "augmented data statics, lora baselines comparison, more clearly statement of contribution".

Overall, we recommend acceptance.
Decision: Accept (poster)