Let's (not) just put things in Context: Test-time Training for Long-context LLMs
Authors: Rachit Bansal
Aston Zhang
Rishabh Tiwari
Lovish Madaan
Sai Surya Duvvuri
Fnu Devvrit
David Brandfonbrener
David Alvarez-Melis
Prajjwal Bhargava
Mihir Kale
Samy Jelassi
Venue: ICLR 2026 Poster (2026)
Keywords: long-context language models
test-time training
inference-time scaling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We study the limitations of vanilla inference-time scaling approaches for long-context large language models and study test-time training as a promising alternative.

Abstract: Progress on training and architecture strategies has enabled LLMs with millions of tokens in context length. However, empirical evidence suggests that such long-context LLMs can consume far more text than they can reliably use. On the other hand, it has been shown that inference-time compute can be used to scale performance of LLMs, often by generating thinking tokens, on challenging tasks involving multi-step reasoning. Through controlled experiments on sandbox long-context tasks, we find that such inference-time strategies show rapidly diminishing returns and fail at long context. We attribute these failures to score dilution, a phenomenon inherent to static self-attention. Further, we show that current inference-time strategies cannot retrieve relevant long-context signals under certain conditions. We propose a simple method that, through targeted gradient updates on the given context, provably overcomes limitations of static self-attention. We find that this shift in how inference-time compute is spent leads to consistently large performance improvements across models and long-context benchmarks. Our method leads to large 12.6 and 14.1 percentage point improvements for Qwen3-4B on average across subsets of LongBench-v2 and ZeroScrolls benchmarks. The takeaway is practical: for long context, a small amount of context-specific training is a better use of inference compute than current inference-time scaling strategies like producing more thinking tokens.

LET'S S (NOT) JUST PUT THINGS IN CONTEXT: TEST-TIME TRAINING FOR LONG-CONTEXT LLMs
Rachit Bansal Aston Zhang
Rishabh Tiwari Lovish Madaan Sai Surya Duvvuri" Devvrit Khatri David Brandfonbrener' David Alvarez-Melis' Praj Bhargava" Mihir Kale Samy Jelasi!
Harvard University "Kempner Institute at Harvard 'OpenAl *UC Berkeley 6 Meta Superintelligence Labs UT Austin 'Anthropic Microsoof Research
ABSTRACT
Progress on training and architecture strategies have enabled LLMs with millions of tokens in context length. However, empirical evidence suggests that such long- context LLMS can consume far more text than they can reliably use. On the other hand, it has been shown that inference-time compute can be used scale per- formance of LLMs, often by generating thinking tokens, on challenging tasks in- volving multi-step reasoning. Through controlled experiments on sandbox long- context tasks, we find that such inference-time strategies show rapid diminish- ing returns, and fail at long context. We attribute these failures to score dilu- tion, phenomenon inherent to static self-attention. Further, we show that current inference-tine strategies cannot retrieve relevant long-context signals under cer- tain conditions. We propose query-only test-time-trainingg (qTTT) that, through targeted gradients updates on the given context, provably overcomes limitations of static self-attention. We find that this simple shift in how inference-time com- pute is spent leads to consistently large performance improvements across models and long-context benchmarks. qTTT leads to massive 12.6% and 14.1% points improvements for Qwen3-4B on average across subsets of LongBench-v2 and ZeroScrolls benchmarks. The takeaway is practical: for long context, small amount of context-specific training is a better use of inference compute than cur- rent inference-time scaling strategies like producing more thinking tokens. |
INTRODUCTION
Many ambitious LLM use-cases are rooted in long context: analyzing scientific corpora (Katz al. 2023 Taylor al., (022). synthesizing books (Kryscinski al. 2022), maintaining rich
multi-turn histories (Park et al. 2023 Zhou et al., 2024), and reasoning over large multi-file code repositories (Jimenez
2024 Zhang et al. 2023). Recent progress in pre-training and
architectural strategies have enabled context windows with millions of tokens (Yang et al.) 2025) Ding et al.) 2024; Reid al., 2024 Anthropic 2024). practice, however, persistent failure modes remain: models miss clauses buried in lengthy documents, overlook function definitions deep in repositories, or fail retrieve facts from prior turns even when the relevant content is present "in context" (Liu et al. 2024 Hsieh et al. 2024 Kamradi (024)).
Concurrently, there growing interest in using inference-tine compute to overcome limitations of vanilla transformer models. Methods such as thain-of-thought "thinking" tokens (Wei et al.) 2022b), best-of-n (Nakano et al.] 2021 Stiennon et al. 2020), and other "thinking" strategies (Zelikman et al. 2024) have shown promise. However, all these methods generate additional tokens with the same static attention mechanism that is already ander-allocating mass to the evidence.
Correspondence: rachithansal @ g. harvard.edu, az sr@@astonzangg.com "Wor done during an internship Meta Superintelligence Labs.

---

Gain ICL accuracy (%)
50
35
20
Accuracy (9))
+-17
32-
25
15
+-1
15.
10-
16
5-
04
70
10,000
250
Number of code lines
Number of transactions
Qwen3-1.78
Qwen3-8B
In-Context Only
With Thinking
With Query-only Test-Time Training (qTTT)
(a) Bug tracing in code repository
(b) Bug tracing in transaction logs
(c) LongBench-v2 ZeroScrolls
Query-only test-time training uses inference-time compute more effectively than "think- ing" tokens for long contexts. (a, b) We construct two tasks to perform controlled long-context analysis: (a) bug localization in large code repositories, and (b) anomaly detection in transaction logs. As context length T grows, in-context accuracy drops and thinking tokens show diminishing returns; with the same FLOP budget, qTTP consistently improves performance. (c) qTTT shows improvements across domains and model sizes on LongBench-v2 and ZeroScrolls benchmarks.
Figure 1:
We design two realistic sandbox tasks to perform controlled experiments and diagnose long-context failure modes. We identify that standard "in-context only" settings fail with growing context length (Figure I). We formalize this limitation of static, finite-precision self-attention, and score dilution: In presence of "distractor" tokens, logit on a "target" is insufficiently separated from the distractor logits. weakening the target probability mass. We establish that as context length 7' grows, the target-distractor logit margin must scale as M(log T") to avoid vanishing target probability. We extend this analysis show that vanilla compute-scaling strategies, such as "thinking" tokens, cannot retrieve the signal from buried target tokens.
Hence, natural question arises: How can we best use inference-ting compute to improve long- context retrieval and reasoning? We revisit test-time training (TTT) (Liu et al. 2021 Hardt & Sun 2024 Akyürek et al. 2024) as a way to adapt the model given long-context input rather than produce more text from unchanged model. Our key idea, query-only TTT (4T77), is computationally frugal approach: Perform a single prefill cache keys and values, followed by a few lightweight gradient updates exclusively the query projection matrices in the attention layers, keeping all other parameters fixed and reusing the key-value cache (Figure 2). We show the- oretically that this targeted adaptation directly increases the separation between target and distractor logits for the specific context at hand, counteracting the limitations of vanilla in-context learning.
We perform evaluations ZeroScrolls (Shaham et al.) 2023) and LongBench-v2 (Bai et al., (023b). with Qwen3 models spanning 1.7B-8B parameters. We observe consistently large performance gains across model sizes and datasets. Under
15+ real-world datasets from popular long-context benchmarks,
inference- compute budgets, qTTT consistently surpasses standard inference-time thinking strategies (Figure Ic) with more than 20% improvements on code comprehension, multi-document QA, and other multi-hop reasoning tasks. Our results call for reallocating inference-tee time budget from thousands of "thinking" tokens small number of query updates for long-context retrieval and reasoning without altering pre-training, architecture, or data.
Contributions.
We construct sandbox tasks demonstrate long-context failure modes ($2 We formalize score dilution in static, fmite-precisiie self-attention and prove a logarithmic margin requirement: the target-distree diittt-iittactor logit gap must scale as Q(log T') to avoid vanishing target probability ($2.3)
We show theoretically and empirically that current inference-time compute scaling strategies primarily scale decoding and cannot reliably meet the margin requirement; in particular, they cannot amplify the signal from buried targets beyond an e-fraction ($2)
We introduce query-only TTT (qTTT): compute-fruse frugal TTT procedure that performs prefill to cache K/V, then applies few gradient updates only to query projections while reusing the KV cache, directly increasing target-distractor separation ($3)
2

---

On 15+ real-world datasets from ZeroScrolls and LongBench-v2. using Qwen3 models query-only TTT consistently improves long-context performance and under FLOP-
matched budgets, outperforms intermediate thinking-toker baselines (Figure lc. $4.
Since qTTT takes place at inference-time, it can easily be applied on top of other existing strategies for long-context modeling: architectural changes such as sliding window attention (Dai et al.)
2019 Beltagy et al.) 2020), adaptive positional encoding (Press al. 2022; Su et al.] (024))
training tweaks for longer windows (Chen et al. 2023 Peng et al. 2024). or retrieval augmented
generation (Borgeaud al. 2022 Izacard al. 2022).
2
VANILLA COMPUTE-SCALING STRATEGIES FAIL FOR LONG CONTEXTS
In this section, we analyze how increasing context length T affects static sradratic-attention LLMs and common inference-time compute-scaling strategies. Using controlled synthetic tasks that mirror realistic long-context retrieval, observe sharp performance degradation as T grows, while generating intermediate "thinking" tokens yields rapidly diminishing returns. We then provide a the- oretical explanation: with static, knite-preeisiie self-attention, the target logit suffers score dilution as distractors accumulate, and avoiding this requires a logarithmic margin requirement- the worst- case iiget-distractov logit gap must scale as 22 T). Decoddnng-bssed inference strategies do not reliably meet this requirement; in contrast, small gradient-based adaptations can increase the margin, which motivates our methodology (developed in (3). All proof's are provided in Appendix
2.1 EMPIRICAL ANALYSIS ON SYNTHETIC LONG-CONTER TASKS
First, we empirically analyze the effect of context length on vanilla transformer models and current inference-time compute-scaling strategies. We study two synthetic retrieval tasks that mirror realistic long-context cases while allowing control over the context length T. For each example, the relevant evidence ("needle") is held fixed and only the surrounding "haystack" grows, isolating the effect of length on retrieval. We provide examples from our datasets in Anppendxx A
Bug Localization in a Code Repository. Starting from a large open-source repository we in- ject a single-line logical bug and ask the model to identify and fix it. Examples of bugs include missing softmax temperature scaling in the attention mechanism and layerorm misplacement Transformer block (see Appendix for details). We vary the context length by the number of lines exposed the model. For a given bug instance, we sample a span of lines around the bug, extending to other files in the directory for large L. We create splits of the dataset with L ranging from to 10000. Across length conditions, the bug location and content are held fixed; only the surrounding code (the "haystack") grows to introduce realistic, semantically relevant distractors.
the
Error in a Log of Transactions. We synthesize multi-account banking logs with initial state and a sequence of operations, each line recording old->new balances and indexed with a TX.ID. Valid logs must satisfy invariants: conservation of total funds, non-negative balances, and arithmetic correctness. We inject exactly one anomaly and consider the following bug types: (incorrect arithmetic), NEGATIVE_BAL (over-debit), LOST_UPDATE (stale write overwrites prior commit) and DUP LICATIETTXT (same payment applied twice). The model must output the bug type and offending TX ID. Context length is controlled by the number of operations nı; we sweep from 25 operations to 500 operations which varies the number of tokens from O(10²) to O(10)..
Findings. We evaluate Qwen3 models ranging from 1.7B to 8B parameters on these synthetic tasks. Figure I]shows the results for the Qwen3-4B model. For both tasks, we see clear consistent trends: (i) As the context lengths increases (number of code lines/transaction logs), the standard in-context performance (i.e., without any additional inference-time compute) decreases sharply. (ii) Further, using inference-tine compute via thinking tokens improves performance for shorter contexts, but shows clear diminishing returns as the context length increases, asymptotically converging close to the standard model performance for long contexts.
We use OL reference repository for the dataset: httpp:://githbb.com i /OLMO
3

---

Empirical Takeaway: Across both controlled tasks, holding the needle fixed and increas- ing the haystack length yields a sharp, monotonic drop in-context accuracy. Allocating inference-tinee time budget to "thinking" tokens offers only short-horizon gains with clear satura- tion at large T. These trends suggest a structural limitation of static attention in long contexts.
We now formalize this limitation as score dilution and derive the resulting logarithmie margin re- quirement, which explains why decoding-basse scaling fails to recover retrieval $2.3)
2.2 PRELIMINARIES
Recall, for a sequence of tokens with hidden representations (h:)7-1 E Rd, each Transformer layer commpues query, key, and value projections:
= W th his k/0 = w/ hy, u W by:
(2.1)
where (1(0) w Q K E R xd and w:(9 Rd. are learned projection matrices. Further, the scaled dot product between query qi and key ky gives the attention logits 2,, that are normalized via softmax to obtain attention weights az,. Finally, the output o, is a weighted sum of value vectors:
0. Kg dd
exp(z1.)
aij exp(2re)
O, Σ oi
(2.2)
In the autoregressive setting, causal masking enforces j S so that each position can only ag- gregate information from its past. Multi-head attention extends this computation across several subspaces, allowing the model to capture diverse forms of dependency.
In-Context Learning. This snvention-based retrieval the foundation of in-context learning (ICL; (Dong et al.] 2023)). By inserting task demonstrations, instructions, or relevant passages directly into the input, LLMs can adapt their outputs without parameter updates. For applications such as analyzing codebases, synthesizing long documents, sustaining multi-turn dialogues, the model must effectively identify and use information scattered across contexts of length 10--106 tokens.
Thinking Tokens. Given a prefix Ti-s and a target at position i+1, thinking-tokee methods (Wei
et al. 2022a Kojima al. 2022 Wang et al.) 2023a) append M 2 0 auxiliary tokens at indices E [++1,.... H before producing the final answer at a = i+ M+1 Each token is generated with static parameters and the same attention kernel as in Equation (2.2) yielding logits ze,j. weights Or,j, and outputs or over the augmented sequence of length T
Definition 2.1 (Retrieval). When predicting token Ii+1, the relevant information may lie in a specific key-value pair Uk.), (the 'needle') at some earlier position jr < i. For threshold T E (0, 1). we say that retrieval at position succeeds o; 2 т. Equivalently, in margin form define 7/2 == Zs.j" JogE efis, then retrieval succeeds iff
% 2 log
All other positions j i jr are distractors, contributing competing logits {2i, Diss".
2.3 THEORETICAL LIMITATIONS OF STATIC ATTENTION AND THINKING TOKENS
Informed by the empirical findings in 22 we now analyze single attention layer as in Equa-
tion (2.2) on the retrieval task (Definition 2.1). We formalize the fundamental challenge of score dilution, which arises when "near-tie" distractors inflate the softmax denominator, causing even unique maximal logit to receive vanishingly small attention mass.
Lemma 2.2 (Score dilution). If at least m distractor keys satisfy 21,5 2 21,3" A for some A 2 0, then
1 +-m--
In particular, m 2 cT for some > and A 0(1), then Oi. 0 0as' T +
4

---

This lemma formalizes simple intuition: When constant fraction of tokens are within O(1) logit of the needle, the attention budget cannot concentrate and the needle's 's mass vanishes with
This dilution effect imposes strict requirement how much the target logit must stand out from all other distractors. The following corollary quantifies this necessary separation, showing that the required margin between needle and distractor must grow with the context length.
Lemma 2.3 (Logarithmic margin requirement). Fix & E (0, 1). If
(T 1)(1 ))
min (Zl.) 24.5) 2 log 373"
then Oz.j* 21-E In particular, guaranteeing a fixed target mass against worst-case distractors requires gap that scales as Qilog T).
Achieving a margin that scales logarithmically is difficult for a model with static attention. Next, we evaluate the strategy of generating thinking tokens in satisfying the logarithmic margin requirement.
Proposition 2.4 (Needle-signal bound for generated tokens). For any thinking token t E (i+l,...,it i+M} and any u E IR.,,
(u, or) S ans (u, + (1-)) max (u, vz).
371
Corollary 2.5 (Specialization under small margin). 1fe margin at token t satisfies 7 S log(e/(11 E)) (equivalently, an. E by Definition 2.1 then
(u. 01) (u, + (1 max Uz).
Moreover, by Lemma 2.2 if at least m distractors satisfy Zt.j Zz.j" A, then oz, $ 1/(1 + me-A) yielding the same bound with c=1/(1 + me- AA).
Proposition 2.4 shows the fraction of needle signal any generated token can carry is at most its own attention mass the needle. Under dilution (small margin), this mass is provably tiny (Corol- lary (.55). so attending to thinking tokens cannot materially increase the final answer's effective margin unless some intermediate token first assigns non-trivial attention to the needle.
Takeaways: (i) With fixed weights, worst-case retrieval requires logit margin that grows like ('o failing to achieve this leads to score dilution and vanishing az (ii) Autore- gressively generating additional tokens with the same static attention does not repair missing access to the evidence. (iii) Any successful inference-tine strategy must change the similarity 9, kj (e.g., by updating queries) rather than sampling more tokens with unchanged parameters.
EFFICIENT TEST-TIME ADAPTATION VIA QUERY-ONLY UPDATES
Having established that existing inference-time scaling strategies vanilla transformer models fail for long contexts, we now investigate an alternate strategy of allocating inference-time compute via test-time training (TTT). First, we establish why standard TTT approach, involving several forward and backward passes over the model, is computationally infcasible for long contexts. We introduce query-only TTT (qTTT) that captures the benefits of TTT while minimizing the computational over- head by re-using the KV cache and only changing the query projections. We present theoretical ($3.2) and empirical ($4) evidence for the efficacy of qTTT over vanilla ICL and thinking tokens.
Naïve Test-Time Training is Infeasible for Long Contexts. A natural first-step is full- pal-merreettr TTT: update FFN and all attention projections (Wq, WK, Wv) on the long input æ1.7. We find that this is impractical for long-context regimes: every update alters keys/values across the sequence, invalidating the KV cache and forcing fresh each step, with prohibitive compute and activation memory.
passes over the entire context at
Compute-wise. our FLOP calculations (Appendix C) shows that even one such full call-parameter TTT step over T-token context is equivalent to generating about 1.2 x T decoding tokens. That is, for
5

---

Algorithm Query-Only Test-Time Training for Long Context
1: Input: model fo. long context IT.T, number of steps NITT. span length k. step size y/
2: (K(*) ] [.
+ FOLW..RDAASAANDCCCHEEE Z1.r)
b Siigle O operation
3: for 1 NTrr do
4: Sample a random span 2, = Tta++ from F1.T
5: Compute Lrrr(#;2, z.) using the frozen (K(9, V(0)
6: Update only the query parameters: {W?)) (W 1j CTT
7: end for
8: return adapted model før to generate the final answer
context of about T = 10 tokens, this makes single training step FLOOP equivalent generating full-parameter TTT untenable.
120K decoding tokens-rendents
These constraints motivate eache-preserving alternative. Our approach, query-only TTT (qTTT), performs single prefill to cache {K, V} and then adapts only the query projections on short spans, keeping the attention evidence pathway fixed while reshaping access it. This retains the benefits of TTT without repeated full-context passes; we describe and formalize this procedure next.
3.1 QUERY-ONLY TTT FOR LONG CONTEXT
Phase 2: query.only TTT
The core idea of query-only TTT is to avoid re- peated, costly forward and backward passes over the long context. Instead, we perform a single expensive prefill to cache the context's 's key and value represen- tations and then execute a series of much cheaper, targeted gradient updates. The procedure, also out- lined in Algorithm and Figure 2 as follows:
Cross enftopy Loss
Nerrx
Loghts
Phase 1: Prefill
Scalod Doi-Product Attuntion
KV secdurs
Query
Volne
Single-Pass KV Cache Generation. Given long context Ti.r, perform exactly one full forward pass with the pre-trained model fa- During this pass, for each layer & in the model, compute and store the Key and Value projection tensors, (()) E TR. Txdx and (f) E TT'd. These cached tensors represent the complete contextual infor- mation and remain frozen for the duration of the adaptation process.
KV wwightt Wx
Quecy weight Wa
29
34
Long conter:
Samplet span mrep cheap computed Nęrr times
expenssive computed only once
Figure Overview query-only TTT.
2. Span-Sampled, Query-Only Objective. With the KV cache held constant, we perform NrTr
steps of gradient descent. In each step, we update only the query projection matrices (w b The objective is the standard next-token prediction loss, computed over a small, randomly sampled contiguous span of tokens I, Ttitks where the span length < T:
+ Σ
LTTT xx)
log p0 aaa (K(0, v(n)
(3.1)
Crucially, the gradients VoLrrr are computed and applied only with respect to the parameters (w/(0) J. leaving all other model weights, including the now-static KV cache, unchanged.
3.2 WHY QUERY-ONLY TEST-TIME TRAINING EFFECTIVE
Section 2 showed that long-context failures arise from score dilution and the resulting need for
growing
margin. Query-only TTT targets this bottleneck directly: only adapt the query projections while holding keys/values fixed (from single prefill). This leaves the evidence (K, V) unchanged and instead reshapes query to it by modifying the similarity q," k, for given input (Proposition 3.1. Figure 3).

---

t* (Needle token)
t' (Needle token)
Distractors
Distractors
Gradient step
(new)
gradient step (Ag oe t" - p)
1t (centroid)
(centroid)
(i (Query)
qi (old)
Small logit margin A diffuse attention.
A grows to log(T)) > attention
concentrates on needle.
V x
gradient descent moves toward t* (needle) and away from js.
Figure 3: A visual representation of Proposition 3.1 showing how qTTT improves the logit margin. The gradient updates via TTTT directly move the query projection weights towards the target needles and counteracts score dilution.
Proposition 3.1 (Query update). For loss & - log Oij"* with fixed K, the gradient w.r.t. qi is
V l
aueke - kg.)
Vik
14
descent step 9i + -----1 -VV v,,e moves qi toward ky- and away from the sneettion-eighhte mean μ4i explicitly counteracting dilution. (The statement holds per head and aggregates across heads.)
Lemma 3.2 (Margin improvement). Let M;(gi) -((g:) denote the logit margin. For sufficiently small 7 > 0,
Mi(a vV, M.(a:) + V 9. 11 +12
Hence the margin strictly increases whenever q + O, with the gain proportional to ||k U4 l12 Improvements are therefore largest precisely when attention is most diffuse, i.e., in the long-conten regimes where score dilution is severe.
Takeaway: oner TTT reallocates inference -imen compute into margin-raising up- dates: with fixed {K, V} from a single prefill, each step moves qi toward ky- and provably increases the target-distractor logit margin. It thus directly mitigates score dilution, most when attention is most diffuse, without re-encoding the context or growing the KV cache.
3.3 FLOP EQUIVALENCE: THINKING TOKENS VS. QUERY-ONLY TTT
We compare two ways to spend inference-time compute after single prefill: (i) generate Thiik thinking tokens with frozen weights, (ii) run NqrT query-only updates on spans length k T while reusing the KV cache. For long T, FLOP equivalence (Appendix C) yields the rule of thumb
Tihikk 2 2 2 k
(long T, span <)).
(3.2)
Consider a dense model of about SB parameters on a long context = 10a and an inference-time budget budget decode 8K thinking tokens after the prefill. From (onotmmn (3.2) the FLOPs equate to about Ngrrr = 16 query-only TTT steps on spans of A 128, and Ngrrr = 8 for k= 512. both cases, thinking tokens grow the KV cache by thousands of positions without changing attention, whereas query-only TTT keeps the cache length fixed at T and uses the matched FLOPs to reshape queries against the existing keys/values, directly targeting the margin bottleneck from

---

60
Code Repositorie Long Dialogue
Long Structured
Performance (%)
50
40
Performance (%)
35
40
30
30
30
20
1.78
48
86
1.78
48
86
1.78
46
86
Long In-Context
Multi Document
Single Document
Performance (%)
50-
20
45
10
30
40
35
20
Code Repositories les
Dialogue
History
Long
Long StracTured Data
Long In-
Conteot
Mulli Document
QA
Single Document
1.78
Model Size (B)
48
86
1.76
Model Size (B)
48
8B
1.78
Model Size (B)
46
8B
In-Context Only
With Thinking
With Query-only Test-Time Training (qTTT)
(a) Comparison on LongBench-vz v2 subsets Qwen3- (b) Variation of performance across model size on
8B. Using qTTT consistently outperforms standard in- LongBench-v2 subsets. qTTT improves performance context and FLtP-d thinking settings.
consistently across model sizes.
LongBench-v2 Baai et al. 2023b) provides a testhed to evaluate long-context abilities across a diverse set of context types. Here, report evaluations across all six subsets of the bench- mark for Qwen3-[1.7/4/8D) models, qTTT shows consistent improvements over both standard in-context learning and FLOP-matched thinking tokens across the different context types.
Figure 4:
EXPERIMENTAL RESULTS
In this section, discuss experimental results across suit of long-context tasks. Firstly, we call- back the synthetic long-context setup from 2 Figure I shows that spending inference-time com- pute via query-only TTT results in significant performance improvements decoding. We observe that the improvements are consistent across context lengths unlike thinking tokens that show rapid diminishing returns. In the rest of this section, discuss our findings on long-context benchmarks that involve nuanced n--oop retrieval, reasoning, and comprehension.
top of just in-context
Further, empirically verify that these improvements with qTTT are indeed result of margin improvement and reducing score dilution. Appendix E (Table 2) shows an analysis of attention mass on the target tokens with and without TTT.. Particularly, we aggregate the attention scores for the target tokens (well defined for these synthetic tasks) across model layers to study the influence of TTTT against vanilla attention. We observe that as number of input tokens increases (hence the number of distractors), the performance as well as attention mass for vanilla attention goes down drastically. However, qTTT helps preserve attention mass significantly across context lengths.
Setup and Evaluation Protocol. We evaluate query-only TTT (qTTT) on long-context tasks against two baselines: (i) fh-context-standaaa decoding with intermediate tokens; and (ii) Thinking- chain-of-tee oftouhhhttthttttth variant whose extra tokens are compute- compute-matched matched to qTTT via the FLOP equiva- lence in $33 Our experiments
performed over Qwen3 models across 1.7B, 4B, and 8B pa- rameters, and cover all subsets of LongBench-v2 (Bai et al., 2023b) (six categories) and ZeroS- rolls (Shaham et al. 2023) (eight datasets). Unless stated otherwise, we use Tihint -192, 1-128, 32, and a common budget of 512 tokens to generate the final answe
Ngrr -2r
LongBench-v2. LongBench-v2 (Bai et al. 2023b) evaluates long-context reasoning across diverse context types. The benchmark probes whether models can locate and use dispersed evidence to answer multiple-choice questions across variety of context types: given multi-file project trees in the Code Repositories setting, to resolve arguments of a particular function; and given the context as set of related documents in the Multi-Document QA setting, synthesize spans across sources to answer a question. This allows us to assess the applicability of qTTT across forms input contexts.
We use the /think and /no_think tokens in the Qwen3 model to control for this. We elaborate on further details including decoding parameters and prompt templates in Appendix 1)
8

---

Report
MuS
NarrativeQA
Performance (%)
40
80-
30
40
30
30
Performance (%)
25
20
20
60-
20
10
1.78
48
1,78
88
1.78 4B
QASPER
QUALITY
SQUALITY
40
Performance (%)
40
17.5
15.0
20
12.5
10.0
0
MusiQue Narrative
Quality
Quality
1.78 48
1.78
48
88
1.78 48
Report
QA
Model Size (B)
Model Size (8)
Model Size (B)
In-Context Only
With Thinking
With Query-only Time Training (qTTT)
(a) Comparison ZeroScrolls subsets for Qwen3-
(b) Variation of performance across model size on Ze- roScrolls subsets. qTTT improves performance
8B. Using qTTT consistently outperforms standard in- context and FLOP-matched thinking settings.
sistently across sizes, often greater for larger models.
ZeroScrolls Shaham et al. 2023 evaluates diverse set of tasks and model abilities over long context inputs. We report evaluations across six subsets for (oven3((1.744/55) models. qTTT shows consistent improvements over both standard in-context learning and FLOP-matched thinking tokens, especially for retrieval-baa multi-hop reasoning and long form comprehension tasks.
Figure 5:
Figure 4|shows that, under compute-matchen budgets, qTTT delivers consistent and often substantial gains across model sizes. On Long Dialogue History and Multii-Dcuumen QA, where evidence is most diffuse, qTTT outperforms standard in-context and thinking by wide margins (e.g., for Qwen3- 4B: 30.8 - 43.6 on Long Dialogue History; 40.0 - 46.0 on Multi- Multi-Document QA). In Code Reposi- tories, qTTT scales especially well with model size (for Qwen3-8B: 30.0 + 44.0 + 52.0). Overall, the LongBench-v2 results indicate that qTTT fares well across markedly different context types.
ZeroScrolls. ZeroScrolls (Shaham al. 2023) evaluates long-contend reasoning across diverse tasks. We group the datasets into three categories: (i) Multi-hop reasoning (MuSiQue, QASPER, Narrative(A) which require locating and composing evidence spread across long documents; (ii) Long-form summarization (GovReport, QMSum, SQUALITY), which emphasize distilling lengthy inputs; and (iii) Long-passage comprehension (QAULITY), which measures multiple-choie accuracy over extended contexts. In contrast to LongBench-v2. this suite of tests evaluates the ability utilize some long context solve variety of different tasks.
Figure 9 shows that qTTT consistently outperforms vanilla thinking on multi-hop QA and com- prehension tasks, with gains that strengthen with model size. On summarization-style datasets, improvements are smaller and comparable to thinking, suggesting that when generation quality, not retrieval, is the primary bottleneck, reweighting attention yields limited returns. Overall, we see significant performance gains across datasets and model scales.
The full set of results on LongBench-v2 and ZeroScrolls are elaborated in Appendix Moreover, we include additional test-time compute baselines such as best-of-N and beam search in Appendix G We also perform a comprehensive latency and wall-clock time comparison qTTT with other approaches in Appendix H
Takeaways: (i) We see consistent gains in performance across benchmarks and model sizes, qTTT yields the best average performance under matched FLOPs (Figure 8 Figure 9p. (ii) Retrieval-driven tasks benefit the most, validating the score dilution diagnosis and the margin increase with qTTT 2 3.2.. (iii) Thinking tokens are not reliable substitute: they some- times help but can also trail In-context, especially in very long contexts. (iv) qTTT is more effective use inference-time compute; without changing architecture, data, or pre-training.

---

5 PRIOR WORK
Long-context LLMs. Context windows have expanded rapidly, with models reaching million- token scale (Reid et al. 2024). usually extending limits via RoPE scaling (Chen et al. 2023 Bai
et al.
2023a). Parallel efforts reduce quadratic attention with spanse/structured patterns (Beltagy
ct al.
2020: Zaheer et al. 2020p. Evaluation has coalesced around long-context suites such as (Bai et al. 2023b), ZeroScrolls (Shaham et al. 2023). RULER, and domain-specific code benchmarks like SWE-bench variants (Jimenez et al.) (024). However, these LLMs still exhibit strong position sensitivity, yielding the "lost in the middle" effect (Liu et al.) tests show that a single relevant span can be overwhelmed by many distractors, and this persists across languages and document structures (Kamradi) 2024). Our work targets this retrieval failure by addressing how attention mass is allocated over very long inputs.
2024).
Inference-time compute scaling. A common approach is to spend more compute at inference via chainoof----eeeeeeeeee ofothhttht (Wei et al. 2022c), self-consistency (Wang et al. 2023b), best-of-n selection (Nakano et al. 2021), other strategies (Zelikman et al., 2024, Zweiger et al., 2025 Kang et al., 2025). While often helpful, these methods scale decoding and can be compute-hean with dimin- ishing returns (Snell et al. 2024: Liu al. 2025). Anoohhr way to spend inference-compute is via test-time training (Sun et al. 2020 Liu et al., 2021) & Sun (222]]2244] Akyürek et al. (2024). While typically done handle distribution shift, recent work has started focusing on long-context LLM use cases (Sun et al. [0224] Zuo et al. (025). To our knowledge, our work is first re- re-purpose TTT to the micro-distribution of individual inputs via efficient, query-only variant tailored to long-context.
DISCUSSION
We identify score dilution in static quadratic attention core cause of long-context failures. We design synthetic tasks to study long-context behavior controllably and show that accuracy falls sharply with context length T and "thinking" tokens show diminishing returns ($2). We proposed query-only TTT (qTTT) to reallocate inference-time budget via few query-only updates that prov- ably increase the target-distractor margin ($3) Under matched FLOPs, qTTT consistently out- performs in-context and thinking on LongBench-v2 and ZeroSCROLLS, with the largest gains on retrieval and multi-hop reasoning ($4). In short, adapting queries a more effective use of inference- time compute than generating more tokens for long context tasks.
We evaluate single point on the (k. NrTT) trade-off; exploring budget schedules across span size and steps is immediate. (2) Our compute-matched baseline focuses on "thinking" tokens; extending to self-consistency and best-of-n within the same framework is future work. (3) Gains are task-dependent; developing simple predictors for when to prefer qTTT (vs. decoding-based scaling) is practical next step.
Future directions. (1)
ACKNOWLEDGMENTS
This work was done when RB, RT, SSD, and DK were summer interns at Meta. RB would like to thank other interns in the legacy GenAI team for the exchange of ideas and brainstorming that shaped this project. Namely: Irene Zhang, Winnie Yang, Julian Coda-Forno, Sriyash Poddar, Arushi Rai, and others in the Research Club. thank Sharan Narang, Prateck Yadav, and Mike Lewis for their guidance. RB would like to thank Yonatan Belinkov, Nihal Nayak, Lyndon Lam, Sunny Qin, Bingbin Liu, and other members of the ML Foundations group and the Kempner Institute at Harvard for their feedback the manuscript.
REFERENCES
Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning, 2024.
URL htlp:tt::: ptttts://arxive wwwpeiv..rr/abb///aeee 07279
10

---

Anthropic. Claude 3: Extended context windows for enterprise applications, 2024. Technical Re- port.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, al. Qwen technical report.
ArXiv preprint, abs/2309. 2023a. URL httpp///s///a abs 2309 2 16009
Yushi Bai. Xin Zheng, Tingyu Lv, Yuran Cui, and Wei Chen. Longbench: A bilingual, multi- task benchmark for long context understanding. ArXiv preprint, 0oss2330.11500 2023b. URL
https::/arxivv.. http://wwwggg///////////oo/b// ab3/ 14508
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. ArXiv preprint, abs/2004.031 2020. URL
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean- -apttttt Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Ste- fanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. https:/ http:s//proccedingss mlr. www.mmm.r....mmm.mmmmmm.mmm v1 62 wt/porggeaud2222 html
PMLR, 2022. URL
Shouyuan Chen, Sherman Ding, Chen Wang, Lulu Li, Dayi Zha, and Li Wang. Extending context window of large language models via positional interpolation. ArXiv preprint, abs/2306.15595, 2023. URL
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transforme-XL:: Attentive language models beyond fixed-length context. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Associa- fion for Computational Linguisties, pp. 2978-2988, Florence, Italy, 2019. Association for Com- putational Linguistics. doi: 10. (0.18653/11/P19-1228 1285. URL http::ttps///t http://///aaathhoggyy. .ooggooooogooooooooooooooooo.
P19- 119-1285
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending LLM context window beyond million tokens.
In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, net forum?io
July 21-27, 2024. OpenReview.net, 2024. URL
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in- context learning. 2023. hRtpps//s///xiii org/ abs/ 2301 00234
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In The Twelfth International Conference on Learning Representations, ICL. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: htt.ttttt//peeeeeeeee.e /wwp///eeeeeeeeeeeeeeeeee... net. forum?irroo CNL2bku4ra
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? ArXiv preprint, abs/2404.06654, 2024. URL /tttp:://axxi...ttt wwtpa/xwwwwo.g////////0/0 06654
Gautier Izacard et al. Atlas: Few-shot learning with retrieval augmented language models. ArXiv preprint, abs/2208.03299. 2022. http:ttt/////xa..ooooo/o////
Carlos E. Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL VTF8yNQM66
net wtt.o/ruetteeeureeeeemem forum?id-

---

Greg Kamradt.
Pressure testing llms: Needle in haystack.
https:
2024.
Accessed:
2025-09-04.
Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty, 2025.
Daniel Martin Katz, Michael Bommarito, Shang Gao, and Pablo Arredondo. Natural language processing in the legal domain. ArXiv preprint, abs/2302.12039, 2023. URL htttps:///axxi
arg/abs/2300 12039
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners.
Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /
html]
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. Yoav Gold- berg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computa- tional Linguisties: EMNLP 2022, pp. 6536-6558, Abu Dhabi, United Arab Emirates, 2022. URL
Association for Computational Linguistics. doi: 10.18653/v 122.333/1/222.11mm 11t.:///////......nnnneeeeeee
(in.iiii...iiiinnnnee.. 1eeeddinngseemppppppppmmmmmm. emnlp 488
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: a URL
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou, Can lb LLM surpass 405b LLM? rethinking compute-optimal test-time scaling, 2025.
URL https: / org/ abs 2502 06703
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. TTT++: when does self-supervised test-time training fail or thrive? In Mare" Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances Neural Information Processing Systems 34: Annual Confer- Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 2 21808-218822 2021. URL https: htttps:/ http:///oceciins...erirrrirr newrigg............p.. www.rrisssccpcppooe wttpppppppccpppp/////ppppppp http://ah11411222113333361222222
Reiichiro Nakano et al. Webgpt: Browserrasisseeddeessioo--aavee with human feedback. ArXiv
preprint, 12.b//2211.0932,, 2021. URL/attps://arxxxvvv..
Joon Sung Park, Joseph C O' Carrie Cai, Meredith Ringel Morris, Percy Liang, and
Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. ArXiv preprint, abs/2304.03442. 2023. URL
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https:://pperewwe
Ofir Press et al. Alibi: Attention with linear biases enables input length extrapolation. ICLR, 2022.
M. Reid, Gemini T. Team, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403.05530, 2024. URL https: ftttps://axxiv /arxiv..vv////.///////aa//aavv./ www.zv.og////............ abs/ 2403. 05530
12

---

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor. Juan Pino, and Kalika Bali (eds.),
Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, Singa- pore, 2023. Association for Computational Linguistics. doi: 00.18653/v1/0023..o 536.
a anthology.... wrg.//23g//0033 lotgooooooooorggg22233333 http://222....iinnneaeaeaammmaea 20nd://////000.dnnnnnneeemmmpp
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute op- timally can be more effective than scaling model parameters, 2024. URL https://arxx:
org ab///////// /tlb//440 2408. 03314
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning summarize with human feed- back. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. httphttsss://poceeeeii neurips............mm..........m oct/paperr2222/haaee (e18885556622eeeee1eee1566888888
wwt.:sssss-sstttttttooo.. html
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding. Neurocomputing, 368:127066 2024.
Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 2229-9248 PMLR, 2020. URL
httt::/prrceeedi:: ngs.. .mmm..r......m m.lr. press / vlv
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states, 2024. URL httth:s///axxxi> org/
abs/24077 04620
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085, 2022. URL
09085
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representattonss, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL adffid-1PLINING
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representationss ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL htttp:://openrevieww net/ pdf Saffid=1PLINIMAE
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28 December htt.le///pper//22222 html]
nips 9, 2022, 2022a. URL httttttp:////ppaaa wttp::/s..nn.......pppeeess ce/paper
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. thain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, IA, USA, November 28- December
13

---

9. 2022, 2022b. URL wtw.attp/pppperss.ssscccccc//ce/ http://ccocaappelleeaa//pp////// hash./ 44555615444444afffff55-4------00
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022. New Orleans, LA, USA. November 28 December 9, 2022, 2022c. URL http: hwtp://apperseeea n ps. cc. httpappprrrelleeeppee////// ht.l
Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy, 2025. URL
https://arxiv.. http///rrii....//oo//o////////// /200 18795
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 62, 2020, virtual, 2020. URL
33:
www. neurpp............e............. httpppp..cpppppp.peppppppppppppe Abstract.htm htm_
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models teach themselves to think before speaking. ArXiv preprint, abs/2403.09629, 2024. htLL/htss:///arettt
org/ abs/2403. 09629
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung. Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 2471-2484, Singapore, 2023. Association for Computational Linguisties. doi: 10. 10.11863/11/0000 http://.em.lmmmmmm..... w.e.eee-mmmm............. 151. URL http::://aaaattthh.o
org/ 2023. emnlp ma iiniin.. 151
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: Á realistic web environment for building autonomous agents. In The Twelfth International Conference on Learming Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL /tttps://openreviiie
net fooum? i id= www.:/eettoountme// 9x
Yuxin Zuo Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi. Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-time reinforcement learning, 2025. URL https://arxx:
org/abs/ 2500 16084
Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self- adapting language models, 2025.
10943
14

---

TX004, where the balance of account A becomes negative, violating Bug Description: The attention mechanism fails to properly normalize attention scores, leading to numerical instability and gradient explosion during training. The attention weights grow unbounded, causing immediate training divergence.
Code context:
olmo/moot htt//mmdee.pyy.
L335: def _ecaleel_dt_.prrcctttttteeeeien
L336:
self,
L337:
L338: q: torch.
torch.
L339: v: torch.
L340: drepout_p: float 0.0,
Optional [torch. Tensor] = None,
1341:
1342:
is_causal: bool False,
1343:
-> torch.Tensos
1.344:
1.345:
attn_xe' thtss torch. machch.matmllommmmmmm (g, k. k.transpose (-2, -1))
1346:
1347:
if is causal:
1348:
assert attn mask None
L349:
query_len, key len = q.shape [-2), k.shape [-2]
Target: Given the above code context, please identify the exact location of the bug.
Model output: olmo/modes L345
Figure example of the code bug localization synthetic task.
A SYNTHETIC TASKS
We illustrate two representative synthetic tasks used in study. 6 shows the code bug lo- calization task: the model receives a brief natural- languuel bug description together with minimal, line-numbered code context and must return the exact file-and-line of the offending statement. In the example, the model correctly identifies the line where attention scores are computed without proper normalization (olmo/motl
py: L345).
Figure 7 shows the transaction-lay consistency task: given an initial account state, a set of invariants (e.g., conservation of total funds, no negative balances), and short sequence of transfers, the model must select single bug type and pinpoint the first offending transaction. In the example, the model outputs NECATIVE the stated rules.
Together, these examples illustrate the input/output format of our synthetic tasks, the kind of struc- tured context provided to the model, and the expected concise targets (a specific line for code or (bugitype, Tx_id) pair for logs). We use similarly formatted instances throughout our evalua- tion.
B PROOFS FOR SECTION
Notation. For a fixed query qi logits are 24,5 = 2. T.k
attention weights ag = E. and
vda
0: E, Cigute We write #4 on
15

---

Task Description: Analyze this banking transaction log for bugs.
Initial state: (* ("account_A": 4000, "account_B": 4200, "total": 8200}
Rules: 1. Total money must remain constant (conservation)
2. No account go negative
3. All calculations must be mathematically correct
Transaction logs:
[TX001] : Transfer $107: A=4000 3993,
B=4200
4307
[TX002] : Transfer $204: A=3893
3689,
B=4307
4511
[TX003] : Transfer $780: A=3689 2909,
B=4511
5291
[TX004] : Transfer $2925:A=2909
-16,
B-5291
8216
[TX005] Transfer $699; B-6216
7517,
A=-16
683
Possible bug types (choose exactly one):
CALC_ERROR: Mathematical calculation is incorrect
NEGATIVE. BAL: Account balance becomes negative
LOST UPDATE: Concurrent update causes lost transaction
DUPLICATE_ Same transaction processed multiple times
Target: Please identify the bug type and location.
Model output: ("bug_type": NEGATIVE, BAL, lboatiocc""on": TX004)
Figure 7: An example of the log transactions synthetic task.
Proof of Lemma 2.3
(Score dilution). Let 5= (j j Zi.j 2 zij A} with [S] = m. Then
T
Σ
Fie 2 g=i +
Σ eas 2 e (1 + me m)-
t=1
JES
hence Oz.j* + oc.
E,
Is
If m el with 0 and A O(1), then ozj" - 0 as
Proof of Lemma 2.3 Ejjr 5 < (T
(Logarithmic margin requirement). Let y minyzy (zig- 24.)) Then
3) so
Obiij* 1 jd3j 6
1 (T 1)e-9
Rearranging 1+(T-1)e 1 yields y log (T-))(1=)))
Proof of Proposition 2.4 (Needle-signal bound). For any thinking token f,
Un ae.j" Ug- + at.z")
Σ āi.
āi,j
O4..j
1 Ci.j'
jet
For any Rdd, take inner products and upper bound the convex combination by its maximum term:
(u, S Origz (u, + (rr.g))
max vz).
ifj
Proof of Corollary 2.5 SSeecialization under small margin). By Definition 2.1 7t
log log(e/(1-e))
iff Ot,3" I E. Substitute Ot..'' I in Proposition 2.4 to obtain (u, 0t) I E(u, Uzz) ) + (1 E) max(u, Uz).
3$1
16

---

Moreover, Claim 2.3 implies at, 1 1/(1 + me me= 4) when least m distractors satisfy 21. 21,5" A. yiclding the bound with 1/(1 me )..
kt Proof of Claim 3.1 Directional query update). With 2i,4
vd.
T
(14))) = - log ai,j" = -2ij + log
e e
t=1
Differentiating w.r.t. qi and using
da:t 0q.
vdu"
T
ke
T
iie
Σ
-
ageeke -kg.)
(Mt kg.).
vdx
Ee
Vdk
Vdl
Vdlx
Em1
Thus a descent step moves qi toward ky. and away from 14g-
Proof of Lemma 3.2 (Monotone margin improvement). Define Mi(g:) (M,(g:) (x)()). For a step q." 9: V))())) first-order expansion gives
-l(()).
Then
M1gg7 = M;(g:) + (4))l + O(1²).
q: 2 = 1 1k.. - 11112. which strictly positive unless ky. = 14- V4 is L-Lipschitz, choosing (0, 1/ ensures M, 2 M.(g:) + (V(le)))))[((())))))
Using Claim 3.1
di
Remarks on multi-head attention.
All statements apply per head. Let superscript h index heads (h) (h))
and define per-head logits/weights {=ig Oi.. }. Claims on dilution and margin hold headwise; ag-
gregation across heads is via concatenation and an output projection, which preserves the directional
and imrrriieemtovveeentt arguments by linearity.
C FLOP DERIVATIONS FOR $3.3
We outline FLOP models for two inference-time modes and derive the equivalence summarized in Eq. equation 3.2
Consider dense Transformer with T. layers, hidden size d, MI. ratio (so dr rd), and long context length T. Let Tihink be the number of autoregressively generated "thinking" tokens, NorTr the number of query-only updates, and k the span size per update.
Cost coefficients. Ignoring lower-order terms (layer norms, biases), collect the dominant costs as
Gquad
2Ld (quadratic attention term),
Ciok Tad
(per-token projections/MI..P)
parallel forward over tokens (the prefill) costs
Eprenn (T')
Cqund 2 + Ciok T.
Case A (autoregressive "thinking"), After one prefill, generating Tinink tokens with a KV cache
costs
Tihink (Tinin
1) + Clok Thhink
Fgen (Tininki T) Cquad Tibink T +
2
so the total FA Fprenu (T') + Fgen (Tibink; T").
Case C (query-only TTT: query-only with cached K/V), With one prefill, each query-only
pass recomputes queries for l positions that attend to cached (K, V) and backpropagates only into [Wo). J. The per-pass cost is
Gpanial (k;; T) se 2(Cquned KT + (2+2r)Lkdd)
and the total Fc Epeenn (T)) + NGTTT Gpartial (k; T). (If the span also attends within itself, add 2quaa and + 2T.xd inside G Gpaniat, which dominated by KT when k kx T.)
17

---

Equivalence (A C). Cancelling the shared prefill and equating (Tennki T) Nggrr Gpartaal (A;; T) yields
Oquad TTnink T Thia (Tiik -1)
Cook Lihink
= 2NTTTT k (quad T + (2+2r)
For long contexts with 7> d and spans & & (hence Think < T in matched regimes), the dominant terms give
Think R 2 NNTTTT k
which is Eq. equation 3.2 First-order corrections are Tinek
from the
Toia (Tai term and O
2
from Cio--
Sanity check (numeric instantiation). Take L=32, d-4096, r=4 (a ~B dense model) and T=109. If the application budget allows decoding 1 thinking tokens after prefill, the matched query-only schedules include, e.g., (Narrr=10, k=400) since 2 10 400 s 8,000. This reallocation keeps the KV cache length fixed T and spends the same FLOPs to reshape queries against the existing {K, } instead of growing the cache with additional tokens.
D EXPERIMENTAL DETAILS
Models and tokenization. We evaluate Qwen3-[1.7B. 4B, 8B} with their native tokenizers and maximum supported context windows. All prompts use UTF-8, and inputs delimited with ex- plicit section headers (e.g., [CONTEXT], [QUESTION]), Unless otherwise noted, we evaluate on the official validation/dev splits and follow each benchmark's scoring script.
Decoding and "Thinking" budget. We adopt reoommeeco decoding parameters: Think- ing: temperature=0.6, top-p=0.95,, top-k=20; Non-thinking: temperature=0.7, top-p=0.8, 8, top-k=20. We cap total generation length that Thinking consumes exactly Titnk intermediate tokens plus the final answer; for compute matching, we use Tuhnk = 8192 unless otherwise stated. Self- consistency/bestoffrr are disabled by default keep FLOPs matched.
Query-only TTT (query-only TTT) hyperparameters. We update only Wo in all at- tention layers using AdamW (weight decay 0.01) with sweep over learning rates 38e-4,3c-5, lc-1, 6. le- 1c-6,3e-77 3c,3; we report the best per-dataset LR selected on a held-out portion of the validation set. Batch size is (long contexts). We perform NITr span updates of length k with single prefill/cached {K, VJ; unless stated otherwise, (k: NITT) (128, 32), compute- matched to Thinking via T think 2N ($3.). Spans are sampled uniformly over [1, T T-]: gradient clipping at 1.0; bf16 precision. Additionally, perform sensitivity analysis of TTTT across learning rates. Table [ shows the variation accuracy on our synthetic tasks across context lengths. We find that qTTT's not very sensitive the choice of LR: the performance is relatively consistent between [1e-5, le-6] and only falls on the extreme values LR.
Sensitivity to Learning Rate (n)). Performance of qTTT across varying learning rates. Extreme rates cause instability (high 77) or insufficient adaptation (low 17), with the optimal range typically between 1e-6 and 1e-5.
Table I:
Task / Context
le-4
3e-5
1e-5
3e-6
le-6
3e-7
Bank Transactions
512
4.2
26.5
28.0
27.2
26.8
15.5
2,536
1.5
13.8
14.4
14.0
12.5
6.2
5,120
0.8
10.0
9.2
8.5
7.8
3.5
9,560
0.0
7.8
8.4
7.9
7.0
1.2
OLMo Code Bugs
512
8.5
42.0
44.5
45.7
43.2
22.0
2,050
5.1
38.5
40.2
41.6
39.5
18.5
7,450
2.2
25.0
28.0
27.5
24.8
10.5
10,000
1.0
18.2
20.2
19.5
17.8
5.2
18

---

Evaluation metrics. We official scripts per subset: EM/FI dataset-specific accuracy for QA; (OOUGE-(1,2,L) or benchmark-provicee summary metrics for summarization, multiple-choce accuracy for QAuLITY. When subset defines both EM and Fl, report the primary metric specified by the benchmark.
Prompts and templates. Below we provide the base non-thinking and thinking templates used per task family, All runs share the same template within a family across methods; Thinking adds scratchpad section but the final answer must appear after a Final: tag.
Non-thinking (base)
[SYSTEM]
You are a careful assistant. Use only the provided context. the answer not supported, output "unknown".
[TASK]
(TASK_
# e.g., short answer QA sunmary MCQ
[CONTEXT]
CROTEXT_BLOCKK $ e.g., (DOCUMENTS) (DIALOGUE) (CODE) (TABLE)
[QUESTION INSTRUCTION]
prompt for the required output (QUESTION _R______________ INSTRUCTION; [CONSTRAINTS]
[ANSWER]
Thinking (base)
[SYSTEM]
Reason privately in [SCRATCHPAD],
then provide single final output after "Final:". not supported by the context, output "Final: unknown". [TASK]
(TASK_DESCRIPTINN;
[CONTEXT]
[CONTEXT_BLOC) BLOCKS}
[QUESTION INSTRUCTION)
CIRISTTTO_ORTNNSTRUUCE
[SCRATCHPAD]
# hidden chain-of- tokens (capped to T_think)
[FINAL]
Final:
Post-processing and extraction. For "thinking" runs, we extract the substring after Final: (trim, strip quotes). For MCQ, we regex-match [ABCD]: for extractive QA, we normalize punctua- tion/whitespace (SQuAD-style). For summarization, we truncate the requested budget (e.g.. 200 words) and use the benchmark scorer verbatim.
Compute matching and seeds. Unless otherwise specified, Thinking uses Think = 8192 and query-only TTT uses (k, Nrrr) (128, 32) that Thhink 2NrrTk. We fix the random seed for span sampling and decoding across methods per run; results are averaged over one run per configuration (low variance in setting).
E SCORE DILUTION EVIDENCE ON LONG CONTEXTS
Motivation. Long-context failures could be result of multitude of reasons and design choices. Past literature in long-context modeling has primarily focused on tuning positional encoding to improve long-context abilities. Here present some evidence supporting our claim that score dilution one of the primary reasons for long-come long-context failure. We show that as the context grows, attention mass the target collapses, and accuracy falls even when rotary position embeddings (RoPE) are present and the model not changed otherwise. We further show that qTTT counteracts this collapse suggesting that approach actually counteracts score dilution in practice.
19

---

Table 2:
Bank Transactions (Qwen3-4B): Accuracy (%) and attention mass vs, context length with and without RoPE, and with TTTT.
Context Tokens
Thinking (RoPE)
Thinking (No-RoPE)
qITT (Ours)
Acc
Mass
Acc
Mass
Ace
Mass
512
36.00
0.46 1 0.04
34.00
0.44 + 0.04
28.00
0.42 1 0.06
2,536
0.22
0.03
5.00
0.20 1 0.02
14.40
0.41 1 0.08
5,120
2.50
0.11 1 0.02
0.80
0.03 1 0.01
10.00
0.36 1 0.09
9,560
1.00
0.04 = 0.01
0.50
0.01 = 0.00
8.40
0.25 + 0.09
Table 3:
OLMo Code Bugs (Qwen3-4B): Accuracy (%) and attention mass vs. context length with and without RoPE, and with QTTT.
Context Tokens
Thinking (RoPE)
Thinking (No-RoPE)
qTTT (Ours)
Acc
Mass
Acc
Mass
Ace
Mass
512
50.00
0.64 + 0.05
47.40
0.61 = 0.05
45.70
0.58 + 0.06
2,050
21.60
0.38 0.07
16.20
0.29 0.04
41.60
0.51 1 0.08
7,450
17.20
0.26 + 0.06
10.60
0.14 + 0.02
28.00
0.42 1 0.09
10,000
10.00
0.12 + 0.03
3.00
0.04 + 0.01
20.20
0.35 + 0.09
Experimental setting (RoPE ablation). evaluate Qwen3-4B on two tasks (Bank Transactions; OLMo Code Bugs) under three test-time regimes: (1) Thinking-oni with fixed thinking budget (4k or 8k tokens). (2) qITT (ours) with a brief query-only adaptation while reusing the prefetched KV cache, and (3) No-RoPE ablation where we disable rotary phase application to Q (identity rotation), keeping all weights, prompts, and budgets unchanged and without any additional fine-tuning. This isolates the role of positional encoding while holding training and data fixed.
inference
Attention-mass metric. For each decode step t. layer e, and head h, let A(C-)) denote the softmax attention from the current query to context position т. Given a labeled set of target indices T, we define the attention mass at step & multi-token answers we average over their output steps. We report mean + std across multiple runs.
>EET A(th), then average over all layers and heads; for
Findings. Tables 2 and 3 show that thinking- accuracy and attention mass both decay sharply with context length. Disabling RoPE accelerates this collapse (lower mass and accuracy), but even with RoPE the decline is substantial. contrast, qTTT sustains markedly higher attention mass as context grows and correspondingly improves accuracy. These results support the view that score dilution, rather than training-data scarcity alone, the dominant failure mode in these settings.
F ZEROSCROLLS AND LONGBENCH-V22 ALL MODELS AND SUBSETS.
This appendix reports the complete breakdowns for benchmarks, models, and inference set- tings. We compare three modes vanilla in-context, chain-of-thougle "Thinking" and our test-time training method (qTTT)- for 1.wr331.7B/4BBBB across LongBench-v2 and ZeroScrolls. Unless otherwise noted, higher is better and bold indicates the best within each row/condition.
Figure g shows mathhe overview LongBench-v2 results across six domains. The detailed per-domain numbers that underlie this figure appear in Table 4 Figure summarizes the observed results on ZeroScrolls. The complete per-dataset numbers, including retrieval-heavy and summarization tasks, are provided in Table 5
G ADDITIONAL TEST- TIME SCALING BASELINES
Baselines. We compare Best-of-N (BoN) and Beam Search to our method under strict compute parity. BoN Self- SelffConssittecy (SC-N): N independent decodes, each with equal share of
20

---

LongBench-v2: Results across Models and Domains
Performance
0.5
0.3
0.2
0.0
Code Repositories
Long Dialogue History
Long Structured Data
Long In- Context
Multi Document QA
Single Document
QA
In-Context
w/ Thinking
w/ qTTT
Qwen3-1.7B
Qwen3 BBB
Qwen3 BBB
Figure 8: FLOP- matcheed comparison on LongBench-v2 [Bai al. 2023b) across six domains for Swee3-11.B/4BBS8 under vanilla in in-context only, with thinking (CoT), and with test-tin training (TTT). TTT consistently yields the best accuracy across domains and model sizes, with the largest gains on long-dialogue and document-QA tasks, and benefits growing with model size.
Table 4: Full LongBench-v22 results for
under In-context, Thinking, and qTTT. Scores follow benchmark-defined metrics; bold marks the best within each row/condition.
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
In-context
Thinking
qTTT
In-context
Thinking
ITTT
In-context
Thinking
qTTT
Code Repositories
26.0
18.0
26.0
25.0
28.0
32.0
30.0
44.0
52.0
Long Dialogue History
23.1
30.8
46.2
20.5
30.8
43.6
33.3
53.8
58.5
Long Structured Data
27.3
30,3
30,3
35,3
38.2
42.4
nn-Context
18.0
20.0
28.0
21.0
25.0
33.0
32.0
40.0
44.0
Multi-Docue molti-Doccument QA
26.0
26.0
42.0
30.0
40.0
46.0
32.0
34.0
50.0
Single-Document QA
32.0
34.0
38,0
35.0
42.0
48.0
32.0
44.0
46.0
Average
25.4
26.5
35.1
27.0
33.5
39.6
32.3
42.3
48.8
the extra reasoning budget, and select the final answer by majority vote (ties broken by sequence log- prob). Beam-k: we run left-to-right beam search of width k; to enforce parity with other test-time scaling, the total added "thinking" tokens across all beams fixed.
Design choices (strict matching). We match all methods to fixed extra budget corresponding to Think = 8192 tokens beyond the vanilla decode. SC-N allocates &s 8192/N tokens to each sample; Beam-k allocates z 8192/K tokens per beam. All results use the same prompt, output length (128 tokens); latencies are reported separately in { This protocol removes budget-induced confounders and isolates the effect of test-time scaling itself.
Conclusion. Across both LongBench-v2 and ZeroScrolls (Qwen3-4B), 4TTT is competitive with or better than strietly FLOP-matched BoN and Beam. SC-N helps when single-run accuracy is al- ready high Single Document QA, QUALITY), but often degrades when the per-sample accuracy below 50%. Beam-k provides only modest gains under equal budgets due correlated beams and imperfect ranking, and frequently trails gTTT.
H LATENCY AND COMPUTE-MATCHER MEASUREMENTS
Setup. All latency numbers were measured on a single NVIDIA A100 GPU in standard inference mode. We report wall-clock time in seconds (mean + std) for three different context lengths. For given model size and context length, we perform latency analysis based on the amount of FLOPs, FşTTT, it takes run NşITT 32 steps for k 128 single evaluation example. We report the following metrics:
N Number of thinking tokens that can be generated to match FTTT FLOPs.
NBox: Number of best-of-N trajectories that be generated match FrTT FLOPs.
21

---

ZeroScrolls: Results across Models and Domains
0.8
Performance
0.6
0.2
0.0
Gov-Report
MuSiQue
Narrative
Qasper
Quality
SQuality
QA
In-Context
w/ Thinking
w/ qTTT
Qwen3-1.77
Qwen3-4B
Qwen3-8B
Figure FLOP-matched comparison on the ZeroScrolls benchmark (Shaham et al. 2023) for Qwen3-1.7B/4B/BB under long contexts, with thinking (CoT), and with test rest-time training (TTT). TTT achieves highest scores on nearly all datasets- especially on retrieval-focused tasks, with general increase with model size.
Table 5: Full ZeroScrolls results across eight datasets for Qwen3- 1.7//4B/BB/ under In-context, Thinking, and qTTT. Datasets span retrieval and summarization; bold marks the best within each row/condition (higher better).
Qwen3-1.77
Qwen3-4B
Qwen3-8B
In-context
Thinking
4TTP
In-context
Thinking
qTTT
In-context
Thinking
4TTP
GovReport
22.5
21.8
26.0
24.9
20.2
33.5
22.0
17.8
29.8
MuSiQue
22.6
26.2
7.5
30.5
22.5
43,9
48.9
NarrativeQA
15.0
8.9
11.7
11.0
30.0
38.0
18.9
35.1
42.8
QASPER
25.7
21.4
31.1
23.2
24.7
34,0
19.6
21.1
26.1
QMSum
9.5
10.9
7.7
8.6
QUALITY
47.6
61.9
76.2
76,2
87.0
71.4
90.5
94.5
SQuALITY
9.2
14.6
18.0
9,9
16.8
18.7
18.1
15.3
18.3
SummSSrreenFFD
8.2
7.2
7.4
9,9
8.3
9.9
10.4
7.9
7.9
Average
18.3
20.7
25.8
18.4
23.9
32.5
24.1
30.0
34.6
tIcL:
Wall-clock time for a vanilla in-context pass on single example. This roughly corre- sponds to the prefill time.
lhink:: Wall-clock time to generate Nihink tokens, given a single example.
tBoN: The amount of time to compute best-of-N via self-consistency for NBoN trajectories given single example.
lurr: The amount of time perform NGTTT steps of qTTT steps with span length k 128 for a single example.
Tables 10 show the results of the measurements on Qwen3-1.7B, 4B, and 8B, respectively. We find that the wall-clock time for all three test-time compute Paategies-qTTY thinking, and best-of-N-1 quite similar. We also note that prefilling the KV cache, which is approximately equal tICL dominates most of the decoding time, especially for longer sequence lengths. This motivates the frozen K/V attention weights in our setup, without which the prefill would need to be recomputed with every training step.
22

---

Table 6: Qwen3-32B on LongBench-v2. Comparison of In-context, Thinking, and qTTT. These findings demonstrate that that the improvements with qTTT hold across model scales.
In-context
Thinking
qTTT
Code Repositories
36.00
61.00
74.00
Long In-Context
44.00
56.00
57.00
Long Structured Data
39.30
42.20
51.50
Long Dialogue History
47.10
77.90
75.50
Multi Document QA
35.00
41.00
56,00
Single Document QA
36.00
47.00
49.00
Table Qwen3-32B ZeroScrolls. Comparison of In-context, Thinking, and qTTT. These find- ings demonstrate that that the improvements with qTTT hold across model scales.
In-context
Thinking
qTTT
Gov Report
26.70
24.80
26.00
Musique
28.90
54.90
59.20
Narrative QA
27.70
42.40
49.60
Qasper
24.10
35.00
42.40
QMSum
11.90
9.90
10.80
Table LongBench-vi v2 (Qwen3-4B): Strict FLOP-matched test-time scaling, Numbers are accura- cies (%). SC-N uses /192/N tokens per sample; Beam-k uses 8192/K tokens per beam.
Task
Thinking
qTTT
SC-8
SC-16
SC-32
Beam-8
Beam-16
Beam-32
Code Repositories
28.0
32.0
30.5
18.4
5.2
27.5
15.1
4.8
Long In-Context
25.0
33.0
28.5
8.5
26.0
18.5
7.2
Long Structured Data
35.3
35.3
36.1
12.2
28.1
11.0
Long Dialogue History
30.8
43.6
34.2
29.5
25.2
12.0
Multi Document QA
40.0
46.0
44.5
41.2
25.8
39.8
35.5
22.1
Single Document QA
42.0
48.0
45.5
49.2
51.8
43.5
44.2
41.0
Avg.
33.5
39.7
36.6
31.7
19.8
33.5
27.8
16.4
Table ZeroScrolls (Qwen3-4B): Strict FLOP-matched test-time scaling. Numbers are accuracies (%). SC-N uses 8192/N tokens per sample; Beam-k uses 8192/K k tokens per beam.
Task
Thinking
TTTT
SC-8
SC-16
SC-32
Beam-8
Beam-16
Beam-32
gov report
20.2
33.5
24.5
15.2
2.1
22.8
12.5
1.8
musique
7.5
30.5
18.2
12.5
4.5
14.5
9.8
3.2
narrative qa
30.0
38.0
35.5
32.0
22.5
31.2
25.5
18.1
qasper
24.7
34.0
28.5
22.1
10.5
26.5
19.5
9.2
qmsum
7.7
8.6
9.2
5.1
0.8
8.5
4.5
0.7
quality
76.2
87.0
82.5
78.5
76.2
squality
16.8
18.7
17.5
19.2
20.5
17.1
17.5
17.8
summ screen fd
8.3
9.9
9.5
6.5
1.2
8.8
5.5
1.1
Avg.
23.9
32.5
28.2
24.7
18.3
26.0
21.4
15.3
Table 10: Latency and wall-clock time comparisons given a fixed FLOP budget for Qwen3-1.7B.
Context Length
faCL (s)
f
fBoN (s)
Nihink
NBa
8,000
8.73 = 0.35
16.92 + 0.68
16.93 + 0.68
16.05 + 0.64
1,434
11
32,000
34.93 + 1.40
43.12 + 1.72
43.11 + 1.72
40.78 + 1.63
358
128,000
139.70 + 5.59
147.89 5.92
147.93 + 5.92
139.70 + 5.59
90
23

---

Table 11: Lateney and wall-clock time comparisons given fixed FLOO budget for Qwen3-4B.
Context Length
fra.. (s)
iTrF (s)
fahink (8)
fRON (8)
Nihink
NBoN
8,000
14.61 + 0.58
28.27 + 1.13
28.26 + 1.13
27.41 + 1.10
1,365
11
32,000
58.45 = 2.34
72.11 + 2.88
72.09 + 2.88
68.69 + 2.75
341
3
128,000
233.81 9.35
247.47 9.90
247.41 9.90
233.81 + 9.35
85
Table 12: Latency and wall-clock time comparisons given fixed FLOP budget for Qwen3-8B.
Context Length
fICL (s)
brrr (s)
fahink (8)
fBoN (s)
Nihink
NBaN
8,000
22.13 + 0.89
42.61 + 1.70
42.62 + 1.70
41.33 + 1.65
1,229
10
32,000
88.53 + 3.54
109.01 = 4.36
109.00 + 4.36
97.07 + 3.88
307
2
128,000
354.13 14.17
374.61 + 14.98
374.67 + 14.99
354.13 14.17
77
24


Reviews:
[{"review_id": "cupbvDAkSx", "reviewer": "Reviewer_BjzK", "summary": "The author proposes a test-time learning method for long context handling with ICL examples.", "questions": "- What if training the whole model from scratch using this method? (Including encoder, meta learning approach)\n- Why do you need to update the query parameters? No need to finetune the MLP?\n- How can we serve different query weight parameters in a real-world serving framework, such as vLLM?\n  - This could be challenging due to the CUDA graph capturing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clever idea to learn only the test time \"decoder\", not the \"encoder\"\n- Extremely strong performance improvement", "weaknesses": "- Training required (during decoding)\n  - No detailed efficiency study has happened.\n- No large model is tested", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pmBbF1pQE1", "reviewer": "Reviewer_Yzb3", "summary": "This paper addresses a critical limitation of long-context Large Language Models (LLMs): while modern LLMs support context windows of millions of tokens, they often fail to reliably use information buried in long texts. Existing inference-time strategies (e.g., chain-of-thought \"thinking tokens\") show diminishing returns as context length grows, due to a phenomenon the authors term score dilution—static self-attention cannot sufficiently separate the \"target\" (relevant information) from \"distractor\" (irrelevant) tokens, leading to vanishing target probability.", "questions": "Does this paper provide a performance comparison with alternative test-time scaling methods, given an equivalent computational budget?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "*  It formally introduces \"score dilution\" to explain long-context LLM failures, turning vague issues (e.g., missed key info) into a quantifiable, solvable problem—filling a gap in prior research that lacked clear theoretical grounding for such limitations.\n\n* The proposed query-only Test-Time Training (qTTT) is innovative in its frugality: it reuses frozen KV caches and only updates query projections, avoiding the high compute of full-model fine-tuning or ineffective \"thinking tokens\" for long texts.\n\n* qTTT offers a low-overhead, drop-in fix for real-world long-context use cases (code analysis, EHR review), and its \"score dilution\" framework guides future research on improving long-context LLMs beyond incremental tweaks.", "weaknesses": "* While it highlights compute efficiency, it does not measure inference latency (critical for production) when qTTT is added—leaving unclear if its small compute overhead translates to acceptable delays for time-sensitive tasks (e.g., real-time code debugging).\n\n* It does not explore how qTTT performs with noisy or low-quality long texts (e.g., unstructured logs, messy code), where distractors are more prevalent—limiting understanding of its robustness beyond clean benchmark datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Puqk2I4d6I", "reviewer": "Reviewer_SSXV", "summary": "The paper argues that long-context failures in static self-attention stem from score dilution: as distractors accumulate, attention mass on the true “needle” vanishes unless the target–distractor logit gap grows with context length. The authors prove a logarithmic margin requirement and show that decoding-based inference-time scaling (e.g., “thinking” tokens) cannot reliably fix this. They propose query-only test-time training (qTTT): perform a single prefill to cache K/V, then run a few lightweight gradient steps updating only the query projections over short spans while reusing the KV cache.", "questions": "See the weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a compute-aware design. The proposed prefill-once, KV-cache reuse, and FLOP matching to thinking tokens make for a fair comparison and a practical recipe.\n\n2. The benchmarks in the paper are sufficient. It evaluates the proposed method across model sizes and multiple long-context benchmarks.\n\n3. The idea is really cute. The inference-time updating the parameters is novel in the community.", "weaknesses": "1. The theoretical analysis is too naive to capture the main motivation. Concretely, it cannot prove that the score dilution is the main reason for the poor performance. \n\n* For example, whether the poor performance comes from the small number of training samples. Usually, learning more complex abilities, i.e., solving problems with longer context, requires more training samples than learning the simple ability. The poor performance can simply come from the relatively small number of samples. \n\n* Even we assume that the poor performance comes from the score dilution. The current analysis is oversimplified. For example, Lemma 2.3 requires a $log T$ scaling of the logits. Such can be achieved by RoPE, which is designed for the long-range decay. In addition, adapting RoPE for the proper score decaying is the cornerstone of most long-context papers. The existing analysis just ignores the role of RoPE.\n\n2. The paper claims that the proposed method solves the score dilution problem. However, no evaluation or visualization of the attention scores are presented. \n\n3. Test-time scaling baseline is missing. The proposed method achieves better performance with more computation. However, no such baseline is included. For example, whether the beam search, BoN achieves the comparable performance with the same budget.\n\n4. Ablation is missing. The learning rate for the test-time training is important. Whether this depends on the context length, context semantic is not known.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)