In-Place Test-Time Training
Authors: Guhao Feng
Shengjie Luo
Kai Hua
Ge Zhang
Wenhao Huang
Di He
Tianle Cai
Venue: ICLR 2026 Oral (2026)
Keywords: Test-time Training
Large language model
LLM
Primary area: foundation or frontier models, including LLMs

Abstract: The static "train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce **In-Place Test-Time Training (In-Place TTT)**, a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling a ``drop-in" enhancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective explicitly aligned with the Next-Token-Prediction task governing autoregressive language modeling. This principled objective, combined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our framework's effectiveness: as an in-place enhancement, it enables a 4B-parameter model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.

Published as a conference paper at ICLR 2026
IN-PLACE TEST-TIME TRAINING
Guhao Feng 1.2+
Shengjüe Luo'
Kai Hua'
Ge Zhang'
Wenhao Huang! IT
Di He2t
Tianle Cai!
BByteDance Seed
SState Key Laboratory of General Artificial Intelligence, Peking University
ABSTRACT
The static "train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continu- ous streams new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility. computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place 77T), framework that seamlessly endows I.LMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling "drop-in" enbancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, dheoretically-groudde objective explicitly aligned with the Next-Token Prediction task governing autoregressive language modeling. This principled objective, com- bined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our in-place enhancement, it enables 4B -B-arammeeer
framework's efffectiveness: model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across range of complex tasks (Brown et al., 2020; Chowdhery & et al., 2022; Touvron et al., 2023; OpenAI, 2024). This success is largely built on static "train then deploy" paradigm, where a model first acquires knowledge from massive corpora and then kept fixed during inference. Yet this design imposes a fundamental limitation: once deployed, the model's weights cannot be updated, preventing dynamic adaptation to the specific context provided by streaming input tokens. Consequently, at test time, the model is constrained in its ability to process and reason over long-horizon, evolving tasks (Chan et al.. 2024; Starace et al., 2025), and to continuously leamn from unbounded streams of experience like humans (Silver& & Sutton, 2025).
In-context learning (Brown et al., 2020; Wei et al., 2023) offers way to mitigate this problem via maintaining all past input tokens in the context. However, its effectiveness is tethered the model's context window, restricted by the quadratic complexity of the de facto attention mechanism (Vaswani et al., 2017). This bottleneck has spurred a line of research into architectural solutions aimed at efficiently extending the context window (Beltagy et al., 2020; Peng al., 2023; Child et al., 2019; Dao et al., 2023). Differently, Test-Time Training (TTT) has emerged as a new paradigm (Sun et al., 2020; Wang et al., 2021; Sun et al., 2024; Behrouz et al., 2024; Yau et al., 2025). Instead of merely
"Equal contribution. |Correponnencee to: Di He <di he@pku. edu. en ent and Wenhao Huang <huang. wenhso@byttedacee.ccm.

---

Published as a conference paper at ICLR 2026
making static model more efficient, TTT enables the model to dynamically update the parameters and adapt to any specific context, directly targeting the aforementioned limitation. Specifically, TTT introduces small subset of model parameters, called fast weights (Schlag et al., 2021), which can be updated on the fly for each new input. By minimizing a self-supervised reconstruction objective, these fast weights compress and internalize contextual information, functioning as an expressive, online evolving state.
Despite its conceptual appeal, unleashing TTT's potential within the current LLM ecosystem is hindered by critical barriers: (i) Existing TTT methods often rely on specialized layers beyond standard Transformer blocks, which usually demand costly pretraining from scratch to achieve satisfactory performance. (Sun et al., 2020; Wang et al., 2021: Zhang et al., 2025; Sun et al., 2024); (ii) the canonical TTT mechanism is inherently sequential (Sun et al., 2020; 2024). While existing works explore chunk-wise acceleration (Sun et al., 2023; Behrouz et al., 2024; Irie & Gershman, 2025; Yau et al., 2025), TTT's role as the primary token mixer forces a reliance on small chunks to maintain performance, thereby bottlenecking the massive parallelism required to saturate modern accelerators; and (iii) the prevalent use of a generic reconstruction objective for TTT's fast weights updating is not explicitly tailored for the causal, Next Token Prediction task that governs autoregressiive LMs, potentially hindering their ultimate performance.
To bridge this gap, we introduce In-Place Test-Time Training (In-Place TTT), framework designed to seamlessly endow LLMSs with Test-TTime Training capabilities by directly addressing the afore- mentioned barriers. Our core insight is repurpose existing MLP blocks with an in-place design rather than introducing a new, specialized layer (tackling barrier i). Specifically, In-Place TTT treats the final projection matrix of MLP blocks as their fast weights, updating in-place during inference. This "drop-in" design requires no modifications to the model's architecture, preserving the integrity of pre-trained weights and enabling on- on-the-fly fly adaptation without costly retraining from scratch.
To tackle the computational inefficiency and objective misalignment, we further design a bespoke adaptation mechanism for language modeling. Following previous works (Sun et al., 2023: Behrouz et al., 2024; Irie & Gershman, 2025; Zhang et al., 2025; Yau et al., 2025), we replace the inefficient per-token updates with a scalable chunk-wis wise update rule (tackling barrier ii). Furthermore, our in-place design operates complementarily to the attention mechanism. This synergy obviates the need for small chunks required by standalone TTT layers, thereby ensuring high throughput on modern accelerators. Concurrently,
move beyond the generic reconstruction targets of prior work (Sun et al., 2024; Zhang et al., 2025) and introduce a novel objective explicitly aligned with the Next-Toker Prediction (NTP) goal (tackling barrier iii). Grounded in rigorous theoretical analysis, we show this NTP-aligned objective encourages the fast weights to store predictively useful information for autoregressive language modeling, leading highly effective and scalable algorithm.
Grounded in these principled design choices, our In-Place TTT provides a practical and effective framework for enhancing LLMs with dynamic, continual adaptation. We conduct extensive exper- iments on language modeling tasks of various compute scales, using them as practical proxy to probe the model's potential on hong-horizon. evolving tasks. Through relatively cheap continual training, our In-Place TTT enables Qwen3-4B -wen3-4B-Basee to achieve superior performance on tasks with contexts up to I 128k. Furthermore, we compare In-Place TTT with competitive TTT-related methods by conducting pretraining from scratch on up to 32k-length corpora, validating the architectural merit of our framework. Finally, ablation studies on state size, chunk size, and fast weight objectives provide deeper insights, confirming the critical role of each design choice. Collectively, our results establish In- Place TTT as a promising step towards a paradigm of continual learning in LLMs.
2 PRELIMINARY: TEST-TIME TRAINING
This section introduces Test- Time Training (TTT), paradigm that enables models to adapt dynami- cally to new data at inference time (Sun et al., 2020; 2024; Zhang et al., 2025). We will first elaborate on the TTT mechanism and then discuss the key desiderata for successfully applying TTT to LLMs, which directly motivates our framework.
The TTT mechanism. At its core, the TTT mechanism leverages fast weights (Ba al., 2016; Schlag et al., 2021), denoted by W. These weights constitute a small neural network fw(-): IRd + Rd which is rapidly updated at test time. Unlike standard model weights that are frozen after training, the
2

---

Published as a conference paper at ICLR 2026
fast weights W act dynamic memory, continuously storing and retrieving contextual information from the sequence.
To process an imput sequence x [21, -2. ....,........................... ÆNI. each token z, E R is typically projected to derive the necessary inputs for the TTT operations, such as a query (q;). a key (kz), and a value (v)). The TTT mechanism then operates through two core, sequential operations:
1. Update Operation: The fast weights W are updated to associate key k with its cor- responding value v,- This is framed as single optimization step that minimizes loss function ((:-)) (e.g.. Mean Squared Error), which measures the discrepancy this asso- ciation. Intuitively, this step encodes the information from the (ki, Us) pair into the neural memory fw. Given learning rate 7), the update rule is:
W f W - WWW.-1-7VWC(/H. WW (kk),
2. Apply Operation: The newly updated network fw, now parameterized by W, is used to process a query 4s. i.e., fw. (g:). This output o; is enriched with the contextual information from preceding key-value pairs, as that information is now encoded in W,,
While this two-step formulation describes the high-level mechanism of TTT, the specific imple- mentation details can vary significantly. Indeed, numerous recent studies have investigated rich design space, exploring different loss functions, more sophisticated optimizers, and alternative neural memory parameterizations improve performance and efficiency (Wang al., 2025; Behrouz et al., 2024; 2025b; Karami & Mirrokni, 2025). These design choices critically influence how effectively the fast weights can store, retrieve, and forget sequential information, positioning the TTT mechanism for different data modalities and tasks.
Desiderata for TTT within the LLM ecosystem. Despite its promise as a paradigm for dynamic adaptation, unleashing TTT's potential within the L.M ecosystem requires addressing several critical challenges. For TTT be a viable and effective component, must satisfy the following desiderata:
Architectural Compatibility. We call an architecture compatible with LLM if it can warm start from a pretrained checkpoint. However, current TTT mechanisms are often developed as standalone recurrent layers designed to replace attention, rather than complement it (Sun et al., 2020; Wang et al., 2021; Zhang et al., 2025; Sun al., 2024; Hu al., 2025). This necessitates costly pretraining from scratch, creating a significant barrier to adoption for the massive, billion-parameter models that dominate the L.LM ecosystem. Therefore, key desideratum is "drop-in" design that requires no fundamental architectural modifications.
Computational Efficiency. The mechanism must be efficient on modern parallel accel- erators. The canonical per-token update rule of TTT inherently sequential and, as a result, severely bottlenecks the parallel processing capabilities of GPUs and TPUs (Sun et al., 2020; 2024; Behrouz et al., 2024). This operational inefficiency makes fine-grained updates impractical for high-throughput language modeling. Consequently, implementation must move beyond per-token schemes and ensure scalability, for instance by adopting chunk- update mechanisms (Li al., 2025; Sun et al., 2023; Behrouz et al., 2024; Irie & Gershman, 2025).
efficient TTT
Tailored Learning Objective for Language Modeling, The predominant self- self-supervised objective in TTT is reconstruction, where the model learns associate (k,, pairs, and U; is typically derived from the input token v; itself (Sun et al., 2020; 2024; Zhang et al., 2025; Wang et al., 2021; Hu et al., 2025). While this generic objective enables the TTT mechanism to store information, its direct relevance to the ultimate goal of language the next token-is not guaranteed. The choice of the target value remains a critical, yet underexplored, design decision that may be suboptimal for capturing the complex causal dependencies required for LLMs.
3 IN-PLACE TEST-TIME TRAINING
To satisfy the desiderata outlined in Section 2, we introduce In-Place Test-Time Training (In-Place TTT), a framework designed to unlock TTT capabilities for LLMss. We first present our overall
3

---

Published as a conference paper at ICLR 2026
framework, which resolves architectural incompatibility via in-place design that repurposes existing MLP blocks, while ensuring computational efficiency with a chunk-wise update mechanism (Section 3.1). We then detail our novel LM-aligned objective, which is explicitly designed for LLMs by aligning with the Next-Token Prediction (NTP) goal (Section 3.2). Following this, we provide a theoretical analysis of our objective's 's superior properties (Section 3.3) and conclude with practical implementation details (Section 3.4).
3.1 OVERALL FRAMEWORK
Repurposing MLP Blocks for In-Place Adaptation. Previous TTT research has largely positioned as potential solution to replace the attention mechanism. However, these prior studies were typically conducted at moderate scales, a regime vastly different from that of modern, billion-paramete LLMs. learned properties critical to high-risk architectural modification. Moreover, introducing any new,
Consequently, replacing the core attention mechanism-whan an LLM's capabilities- randomly-initialized layer also creates conflict with the billions of trained parameters of LLMs, necessitating costly and often impractical retraining to resolve this imbalance.
Our core insight is sidestep these challenges entirely. Instead of replacing adding components, we repurpose a ubiquitous module-the Multi-Layer Perceptron (MLP) block-to also serve as the fast weights. Recalling the TTT formulations Section 2, there exist no constraints on the choice of fast weights, i.e., any parameters can serve as fast weights updated via the TTT mechanism. In particular, the MLP blocks in Transformers can also be viewed as a form of key-value memory (Geva et al., 2020), functioning therefore natural extension to leverage this same component to also function as the adaptive "fast weights", dynamically internalizing transient, in-context information at inference time.
"slow weights" for the vast, general knowledge acquired during pre-training.
Formally, we adapt the widely used gated MLP architecture (Grattafiori et al., 2024; Yang et al., 2025). Given the hidden representation H. the gated MLP computes its output representation O = (6HH gate)= @ (HW up)) W down. In our framework, we treat the input projections W Wap and W gate as frozen slow weights, while repurposing the final projection matrix, W dowa. as the adaptable fast weights. By exclusively updating W down in-place, we preserve the model's architectural integrity, transforming TTT from disruptive restructuring into lightweight, "drop-in" enhancement for
LLMs..
Efficient Adaptation with Chunk- Updates. Beyond architectural compatibility, our in-place design also unlocks significant computational efficiencies. Conventional TTT methods, by aiming to replace the attention mechanism, were bound to inefficient per-token updates enforce strict causality and perform fine-grained token mixing. Chunk-wise updating approaches have been explored by recent works achieve acceleration (Li al.. 2025; Sun al., 2023; Behrouz et al., 2024; Irie & Gershman, 2025). Our framework also follows to sidestep the trade-off entirely. Since we only adapt the MIP blocks and leave the attention layers intact, we are liberated from the per-token constraint, enabling far more efficient chunk-wise update strategy which further bypasses the small chunk constraints (our ablation study results (Section 4.3) also verify that our framework is naturally well-suited for chunk- and specifically large chunk- tonnk-wise-uppe updates achieving optimal performance with chunk sizes of 512 to 1024.
The process operates as follows. Given the intermediate activations Z = o HHW @ (HWT) up)
ggndd and corresponding value targets and outputs V,O IR xdmoded, we partition them into non- overlapping chunks of size C, denoted [+] Di++1(01) RCxd' for O {Z, V, O), i E [k] and d' being their corresponding dimension. Let (i
down be the fast weights state before processing W down- For each chunk E [M]] we perform two sequential operations:
chunk i and (i))) down
1. Apply Operation: The current state of the fast weights (i down are used to process chunk
Z[a. i.e., O18 Zig (i) (i)
2. Update Operation: The fast weight W down are updated using Zi as keys and V [3] as values, which is performed via one gradient descent step with loss function L. and learning rate V/: (((++) (
down - 1/ Ww [Zg (w) (W)) V
down
4

---

Published as a conference paper at ICLR 2026
This chunk-wise update strategy is designed for modern hardware, similar prior attempts (Li et al., 2025; Sun et al., 2023; Behrouz et al., 2024; Irie & Gershman, 2025). Moreover, due to our in-place MLP adaptation,
use large chunk size C' to process large blocks of tokens at once, thereby highly leveraging parallelism and utilizing the computational power of GPUs or TPUs.
3.2 LM-ALIGNED OBJECTIVE
With the efficient, in-place adaptation framework established, the performance of In-Place TTT now hinges on the design of its learning objective, In this subsection, we introduce our Language Aadellng-AAiigedd objective, which is explicitly tailored for LLMs.
Prior TTT approaches typically use reconstruction target, c.g., C(fw(k), where both k and are linear projection outputs of the same input token x (Sun et al., 2020; 2024; Zhang al., 2025), which encourages the model to simply memorize the current token's 's representation. We argue that this is suboptimal for language modeling tasks. Instead, we propose to align the objective with the Next-Tokee Prediction (NTP) goal governing LLMs.
To achieve this, we specify the target to include future token information. Formally, we derive our target V = ConvlD((x)) W target. where Xo E guXddindel denotes the token embedding. C-.v the ID Convolution operator and W Warget E [Remmoda is a trainable projection matrix. Under this formulation, the amount of future token information can be controlled in our target V. e.g., the Next-Token target can be achieved by parameterizing W target as an identity transformation and assigning ConviD(-) kernel weights to be 1 for the next token and 0 for other tokens.
With this aligned target, we use the widely used similarity measure to instantiate our loss function for simplicity, i.e., (()-)) = -(,-)p Under this loss function, the gradient with respect to the fast weights in our chunk-wise mechanism can directly derived:
w(() w(i-)) down
down
3.3 THEORETICAL ANALYSIS
Intuitively, our LM-Alligeed objective explicitly encourages the fast weights compress predictively useful information for future tokens, thereby enhancing the model's capacity for dynamic adaptation. In this subsection, we formalize this intuition by theoretically analyzing the benefits of our objective. We ground our analysis within the canonical induction head setting (Olsson et al., 2022; Elhage et al., 2021), mechanism understood be critical for in-context learning in LLMs.
Setup. Consider an input sequence where a key-value pair, (Ize, I1+41) ) = (k*,"*) v²), appears at an arbitrary position t". Subsequently, at a query position n t", the key k- reappears, such that In = k. The model must then correctly predict the associated value, In+1 = =."
Without loss of generality, we analyze a single Transformer block enhanced by our In In-Place TTT. Let Ze E Rdi be the intermediate activation of token da and Eg E [Rdmodall be the token embedding for Fw. In our framework, the fast weights update from prior context chunks is A A down V.Z, This update then change the output logit at the query position by Al, [w] n Exerrior chunks chunks
E. W down Zuz We compare two choices for the TTT target Ve:
w
Reconstruction Target: Vivec E,,,, the embedding of the current token.
* LLM-Aligned Target: VLM Em, i1 the embedding of the nex/ token.
Assumptions. Our analysis rests on two mild assumptions about the properties of the embeddings and intermediate activations, which are standard in theoretical analyses of Transformers:
1. Approximate Orthogonality of Embeddings: For any two distinct tokens w;, wy E V, their embeddings are nearly orthogonal: le ewy I e for small constant > 0. Additionally, embeddings have a non-trivial magnitude: ||ew. || 2 c > 0 for some constant Caomn-
2. Key-Query Alignment: The intermediate activations Z,, for the query token Fy = k is
aligned with Zy- of its corresponding key token zy-- = k= E. [2 Zn Catign > 0. For other positions i. o t", the tokens are unrelated to the query, i.e, E. (V Z, Zm] 0.
5

---

Published as a conference paper at ICLR 2026
MLP with In-Place TTT
Input Embedding
Willi
Generating Direction
Gated Linear Layer
Attention
Apply
Update
Split into
chunks
100 Vin!
SWS
Conv1D Projection
The overall framework of our In-Place Test-Time Training. The module operates sequen- tially input chunks. For each chunk, the current fast weights are first applied to the intermediate activations produce the output. Then, these weights are updated using the activations and value V derived from the token embeddings. This "apply-then-upoeee dynamically adapt to incoming context strictly causal manner.
Figure 1:
cycle allows the model to
With this setup, we present our main theoretical result:
Theorem 1 (Logit-wine wise Effect of LM-Aligned Target v.s. Reconstruction Target). Under the specified
setup and assumptions, for learning rute Åvr > O, the expected change in logits Aln after one
update step using the LM-Aligned target satisfies:
(Correct logit increases)
E Aen [uT] 2 A Cnam Callinn
(2)
(Other logits almost unchanged)
[ [w]]] S > C Calign:
Yw + 0²
(3)
In contrast, for the reconstruction target, the expected change in logits is negligible for the correct token: [B [e,,,, S. 7t,
Calign.
The proof is provided in Appendix In Theorem I, the LM-Aligned target guaranteed in expecta- tion to increase the logit of the correct next token v" and keep that of other tokens approximately unchanged, directly aiding the model's prediction task. In contrast, the reconstruction target pro- vides no such predictive benefit, failing to increase the logit of the correct token. In practice, our implementation extends this principle from single next token to learned, localized combination of future tokens, which also aligns with recent promising results of Multi-TToken Prediction in advanced LLMs (Liu et al., 2024) as an effective extension the NTP objective. This allows our In- In-Place TTT to capture richer predictive signal, thereby compressing useful contextual information more effectively than simple reconstruction.
3.4 IMPLEMENTATION DETAILS
Combining aforementioned designs, Figure illustrates our In-Place TTT framework. Here further elaborate on practical implementation details, which are engineered for high efficiency and scalability on modern hardware. In particular, our approach is fully compatible with Context Parallelism (CP), relying on a parallel scan algorithm to process sequence chunks simultaneously while preserving the strict causal semantics of an auto-regressive update. Additional discussions are further presented.
Efficient Implementation with Context Parallelism. The associative nature of our update rule in Equation (1) makes In-Place TTT amenable context-parallel implementation, which par- titions sequence along its length and processes the chunks simultaneously. The process un- folds into three stages: (i) for all chunks i E (I,..., T), we compute the intermediate acti-
vations Zi and the fast weight update (1) down (vm)) Zij in parallel; (ii) single prefix sum over [..., (i) (i+1)
down: down ...] is conducted to compute the aggregated updates for each chunk: AS, E AW,, which can be highly efficient modern accelerators; (iii) the ef-

---

Published as a conference paper at ICLR 2026
fective fast weights for each chunk, (1-1) ())
down + nAS; and the corresponding output, [ = [n) (W are computed in parallel.
down
dowa
Causality and Boundary Handling. To ensure that the update delta for chunk itself contains no future information, we apply causal padding to the ID convolution when generating the value. This isolates each delta calculation to its respective chunk, making the parallel scan mathematically equivalent to a sequential update. Moreover, at document boundaries, the fast weights are reset to their pre-trained state to prevent context leakage across independent sequences. The final context parallel algorithm is presented in Algorithm in Appendix C.
Discussion. summary, our implementation of In-Place TTT synergistically combines simple, computationally efficient update rule with parallel scan algorithm. This design choice makes method not only fast and scalable but also mathematically equivalent to a strictly causal sequential process, thanks careful boundary and padding management. The resulting module is CP-native, fully causal, and can be seamlessly integrated as a drop-in replacement for the MLP block in standard Transformer architectures. Lastly, is also noteworthy that the core principles of our framework orthogonal to the specific choice of loss functions and its optimizer, which have been widely studied in the broader TTT literature, exploration we leave promising direction for future work.
EXPERIMENTS
In this section, conduct series of experiments empirically validate the effectiveness of our In-Place TTT framework. Specifically, we aim to answer the following research questions:
Q1: How effectively can In-Place TTT enhance pre-trained LLMs in a "drop-in" manner?
Q2: When trained from scratch, how does In-Place TTT compare against prior TTT approaches?
Q3: What are the effects of key design choices in our In-Place TTT framework?
Using language modeling tasks of various scales as a practical proxy, we answer each question with carefully designed experiments in the following sub-sections, Due to space limits, we present detailed descriptions of experimental settings in Appendix D.
In-Domain Evaluation
Extrapolation
Model
4k
8k
16k
32k
64k
128k
256k
Mistral-7B (Jiang et al., 2023)
93.6
91.2
87.2
75.4
49.0
13.8
GLM3-6B (GLM, 2024)
87.8
83.4
78.6
69.9
56.0
42.0
Phi3-medium-144 (Abdin et al., 2024)
93.3
93.2
91.1
86.8
78.6
46.1
Llama3-8B (Pekelis et al., 2024)
92.8
90.3
85.7
79.9
76.3
69.5
Qwen3-4B (Instruct) (Yang et al., 2025)
95.1
93.6
91.0
87.8
77.8
66.0
Baseline
96.6
94.1
92.1
88.7
74.3
74.8
41.7
In-Place TTT
96.1
95.6
92.7
89.3
78.7
77.0
43.9
Table 1: Evaluation results on the RULER benchmark (Hsich et al., 2024). We report the average accuracy (%) scores, with the best results in bold.
4.1 IN-PLACE TTT DROP-IN ENHANCEMENT FOR PRE-TRAINED LLMS
To validate In-Place TTT as "drop-in" enhancement for existing, pre-trained LLMss, we start with the competitive open-sourced Qwen3-4B- model. Its original context window is 32k. thereby we can simulate the long-horizon, evolving tasks requiring Test-Time Training capabilities by language modeling tasks of varying context lengths. In particular, we compare the performance of (1) Qwen3- 4B Base (Baseline); (2) Qwen3-4B-BB 4B--aas-B In-Place TTT (In-Place TTT). Both models undergo the exact same continual training curriculum, ensuring a fair comparison where our In-Place TTT is the only variable.
7

---

Published as a conference paper at ICLR 2026
Table 2: Extension of In-Place TTT LLaMA-3.1-8B and Owen3-14B-Base on the RULER benchmark. We report the average accuracy (%) with the best results in bold.
Base Model
Method
4k
8k
16k
32k
64k
64k+YaRN
Baseline
93.9
92.1
92.5
91.1
81.6
LLaMA-3.1-8B
In-Place TTT
94.4
93.0
93.3
91.7
83.7
Baseline
96.8
95.0
94.6
90.7
67.9
81.3
Qwen3-14B
In-Place TTT
97.2
95.7
95.2
91.2
70.6
82.5
Training and Evaluation. The continual training curriculum is divided into two stages: initial phase of ~20 tokens with 32k context length, followed by a second phase of ~ 15B tokens with 128k context length. The detailed descriptions of training dataset can be found in Appendix D.1. To effectively manage these long sequences, we adapt the model's Rotary Position Embeddings using YaRN (Peng al., 2023). We evaluate the long-context performance of both models on the RULER benchmark (Hsieh et al., 2024) using the popular OpenCompass framework (Contributors, 2023), with context lengths ranging from 4k to 256k. The 256k setting specifically measures the models' ability to extrapolate beyond the 128k context length limit. Detailed descriptions of training details can be found in Appendix D.2.
The results, summarized in Table 1, demonstrate that In-Place TTT significantly boosts the long-contex proficiency of the pre-trained model. In particular, clear trend can be easily seen from the results: while both models are competitive at short contexts, Qwen3-4B- Base enhanced by our In-Place TTT establishes a consistent and widening advantage as the sequence length increases. It achieves substantial gains at the 64k and 128k context lengths. Crucially, this advantage is maintained when extrapolating to a 256k context, demonstrating superior generalization. These findings confirm that In-Place TTT can be seamlessly integrated into pre-trained LLM to boost its long-contex, proficiency. The model's 's strong performance at and beyond the context length validates method as practical and powerful tool for extending the capabilities of existing LLMs.
Results and Discussion.
Extension to More Models. To verify the generality our approach, we further apply In-Place TTT drop-in enhancement two additional models: LLaMA-3. 1-8B (Grattafiori et al., 2024) and Swen3-14B-BBB (Yang et al., 2025), following the same continual training protocol. As shown in Table In-Place TTT consistently improves the RULER scores across all context lengths on both models. The gains are particularly pronounced at longer contexts, c.g., +2. at 64k for LLaMA-3, 1-8B and +2.7 at 64k for Qwen3-14B-Basc. Moreover, the improvement is maintained when combined with YaRN (Peng et al., 2023) for position extrapolation, confirming that our method is orthogonal to RoPE extension techniques. These results, spanning different model families and scales (4B-14B), reinforce that In-Place TTT is broadly applicable drop-in enhancement for pre-trained LLMs.
4.2 PRE-TRAINING FROM SCRATCH: A COMPARATIVE ANALYSIS
(a) Sliding Window Perplexity of 500M Model
(b) Sliding Window Perplexity of 1.5B Model
12.0
In-Place
In-Place
SWA
SWA
11.5
GLA
9.5
GLA
Deltanet
Dellanet
Perplexity
11.0
LaCT
Perplexity
9.0
LaCT
10.5
100
8.0
9.5
2k
4k
8k
16k
32k
2k
4k
8k
16k
32k
Context Length
Context Length
Figure 2: Sliding Window Perplexity at varying context lengths on the Pile dataset for 500M (left) and 1.5B (right) parameter models. Our In In-Place TTT consistently achieves lower perplexity than all competitive baselines.
8

---

Published as a conference paper at ICLR 2026
Common Sense Reasoning
Long-ContexI Evaluation
Model Architecture
HellaSwag
ARC-E ARC-C MMLU PIQA
RULER-4K RULER-SK RULER-I6K
Full Attn.
55.67
64.52
33.19
36.43
72.63
45.77
38.09
6.58
Baselines
SWA
54.92
64.18
32.85
36.06
72.58
14.77
9.91
5.07
Full Attn.
55.85
64.98
32.34
37.42
73.29
49.98
43.82
19.99
I.P. TTT
SWA
55.24
64.60
33.70
36.48
72.03
28.33
26.80
7.57
Table 3: Evaluation results 4B models on common sense reasoning and long-context evaluation benchmarks. Best performance is in bold. "SWA" is BoddnngWWindww Attention, "Full Attn." is Full Attention, and "I.P. TTT is our In-Place TTT.
(a) State size
(b) Chunk size
(c) I.M-Aligned Objective
C-256
40
40
C-312
40
w/o Com
0.5x
C-1004
C-2048
wio Coms, Proj
20
RULER-4k
RULER-SK
RULER-I6K
RULER-4K RULER-8k RULER-I6k
RULER-K RULER-8K RULER-I6K
Figure 3: Ablation studies on the key design choices of the In-Place TTT framework, evaluated on the RULER benchmark with a 1.7B parameter model. The plots illustrate the impact of: (a) State size, showing that performance improves the state size scales; (b) Chunk size, demonstrating performance trade-off where intermediate sizes (e.g.. 512, 1024) are optimal; and (c) The LM-Aligned Value objective, confirming that both the convolution (w Conv) and the projection (w Proj) are crucial.
Having demonstrated In- TTT's effectiveness "drop-in" module, we further evaluate its performance and scalability when integrated into models pre-trained from scratch. Our analysis proceeds in two stages: we first establish its language modeling capabilities at the 500M and 1.5B scales, and then assess its scalability and impact on a larger 4B model.
Experimental Setup. Firstly, we benchmark our In-Place TTT against prior TTT-related approaches and efficient attention methods based on TogetherAl (2024) at 500M and 1.5B parameter scales. Various competitive baselines are compared: (1) standard Transformer with sliding window attention (SWA) (Child et al., 2019; Beltagy et al., 2020) (2) Gated Linear Attention (GLA) (Yang et al., 2024b); (3) DeltaNet (Schlag al., 2021; Yang et al., 2024d:a) (4) Large Chunk Test-Time Training (LaCT) (Zhang et al., 2025). For a fair comparison, both In-Place TTT and LaCT are built upon an SWA backbone. All models are trained on sequences with 32k context length.
Building on these results, we further scale up to 4B-parameter models to evaluate scalability of our In-Place TTT approach. In particular, we compare Transformers with Full Attention and Transformers with SWA against their counterparts enhanced by our In-Place TTT. These models are trained for 120B tokens with an 8k context length. Detailed descriptions of datasets, model configurations, and training procedures are available in Appendix D.I, D.3, and D.2.
Evaluation. For the 500M and 1.5B models, evaluate their long-context utilization using Sliding Window Perplexity on a validation set comprised of Pile (Gao et al., 2020) and Proof-Pile-2 (Paster al., 2023). This metric measures perplexity on a fixed final block of tokens when extending the preceding context, where decreasing perplexity trend indicates effective context usage. For the 4B models, we conduct a broader evaluation on a suite of downstream tasks, including common sense reasoning benchmarks (HellaSwar wag (Zellers et al., 2019), ARC (Clark et al., 2018), MMLU (Hendrycks et al., (0a),, PIQA (Bisk et al., 2019)) and the long-context RULER benchmark (Hsich et al., 2024).
Results and Discussion. In Figure 2, we plot the sliding window perplexity against context length for both 500M and 1.5B model. It can be easily seen that our In-Place TTT consistently achieves lower validation perplexity than all competitive baselines, with its performance steadily improving up to the full 32k context. This sustained improvement suggests its core mechanism successfully compresses and utilizes information from incoming context.

---

Published as a conference paper at ICLR 2026
(a) Throughput (SWA)
(b) Throughput (Full)
Memory (SWA)
(d) Memory (Full)
TPS (Kookmi))
20
Sautine
Stxx
Profil IPS (K tabomols
Bexciso
Polk Mensry (GB)
50
Budfon
Memory (CB))
Rodine
15-
40
40
10
-0
-6
32L
125L
0
33k
12%
8k
32% 128k
80
124k
Figure 4: Efficiency analysis of In-Place TTT. Both prefill throughput (a, b) and peak memory (c, d) metrics are presented for 4B models with Sliding-Window Window Attention (SWA) and Full Attention at various context lengths. Our In-Place TTT introduces negligible overhead in practical scenarios.
Moreover, the results in Table 3 further show that 11-parameter Transformers with both Full Attention and SWA are consistently improved across most common sense reasoning tasks. Furthermore, models with our In-Place TTT yield superior performance on the long-context evaluation, e.g., RULER- 166 score is improved from 6.58 to 19.99 for the Transformer with Full Attention and RULER-8K score is boosted from 9.91 to 26.80 for the SWA model. These substantial gains, particularly across models of various scales, establish our In-Place TTT as a highly effective and scalable approach.
4.3 ABLATION STUDIES: ON THE IMPACT KEY DESIGN CHOICES
Lastly, we conduct series of ablation studies on RULER with 1.7B-parameter model, providing deeper insights into our design choices. Detailed settings are presented in Appendix D.3 and D.2.
Impact of State Size. We first investigate how performance scales with the fast weights size, which can be controlled by varying the number of TTT-enabled layers. Figure 3 (a) shows a clear trend that the performance of our In-Place TTT consistently improves along with the state size scaling, This confirms that larger fast weights allow the model to more effectively adapt to contextual information, which further supports our repurposing approach leveraging the large amount of MLP states.
Impact of Chunk Size. The chunk size C' in Section 3.1 controls both the granularity of fast weights updating and parallelism, exposing tradeoff between efficiency and performance. By varying the chunk size, Figure 3 (b) shows that both C 512 and C 1024 competitively achieve better performance compared to other choices, while
1024 has better efficiency.
Impact of LM-Aligned Objective. Next, we delve deep into our tailored LM-Aligned objective, i.e., V = Conv ((X,)) W target in Section 3.2. In particular, the Conv1D operator is used to yield targets containing future token information, and the W Vtarget is a projection transformation. In Figure 3 (c), comprehensively ablate combinations of these components. The result shows that both of them are necessary for performance guarantee, while Conv 1D) plays an essential role on long context and W Wargett is erucial on short context. These results align with our theoretical analysis in Section 3.3, strongly supporting our motivation to derive a tailored objective for language modeling.
Efficiency Impact of In-Place TTT. We further study the computational overhead introduced by our In-Place TTT. In Figure 4, we compare both prefill throughput and memory consumptions of with and without our In-Place TTT. The results indeed verify the efficiency of our practical implementations.
5 CONCLUSION
We introduced In-Place Test-Time Training, a practical framework that resolves the critical barriers of TTT for LLMs. Principled design choices are proposed including an in-place mechanism that repurposes existing MLP blocks, efficient chunk-wise update rule, and theoretically-groune objective aligned with language modeling. Extensive experiments validate that our approach not only serves as a powerful "drop-in" enhancement for pre-trained LLMs but also outperforms strong baselines when trained from scratch. By providing a scalable solution for on- on-the-fly fly adaptation, our work makes a promising step towards new paradigm of more dynamic, continual learning for LLMss.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work introduces a foundational model architecture and does not present any direct real-world applications with immediate ethical concerns. We acknowledge the broader societal risks associated with Large Language Models, such as potential biases inherited from training data, and advocate for further research into their responsible development and deployment.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we provide comprehensive details all experimen- tal settings, model configurations, and training hyperparameters in Appendix D. The theoretical claims supported by complete proof in Appendix B, and the implementation is detailed in the pseudocode in Appendix C. We will release source code and model checkpoints to facilitate further research.
ACKNOWLEDGEMENT
DH is supported by National Science Foundation of China (NSFC62376007), National Science Foun- dation of China (under Key Project No. 92570203), Beijing Natural Science Foundation (Z250001) and Beijing Major Science and Technology Project under Contract no. Z251 1000084250044.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, al. Phi-3 technical report: highly capable language model locally on your phone, 2024. URL https: : / /aaaiiiiiiv.////i///aaaa//aaaavv org/ abs/ /404. 14219.
Jimmy Lei Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances Neural Information Processing Systems, 2016. URL https:tt////o. //ar/aavvvvv/////a///.aa///aaavv org/ wrgg www.a//11..... 06258.
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time, arXiv preprint arXiv:2501.0066 2024.
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization, 2025a. URL
https:: /tttps://axivve http///axvvvvvv org wtwpg////////////eee//g/e//saae/ abs///b////44 13173.
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504.13173, 2025b.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint wtw.:::22000055530000 2020. URL htttt:t///a/xiiiii wwwp:xwwv..rrrroeoooooooo org/ abs harg//bb//000/ /2044. 05150.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019, URL https: http//aaxvvvggg..a.aa//// /1911 11641.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.0m0
2024.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904. 10509, 2019.

---

Published as a conference paper at ICLR 2026
Aakanksha Chowdhery and et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL
org/ abs/ 2204.2204.000 22211.211.....
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge.
arXiv: 1rXiv:/803.05557v1, 2018.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: github.......//. htwettth.ooooooeeecoooooottotttt com/ open compass/ /pppppppss/p.cccppssss.... 2023.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov: Transformer-XX XL: Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Association for Computational Linguistics, 2019. URL https:
ac/ http:///aaaaatttoooo.ooooo wtt.hhhwwooy..oooo-1/e119-
Karan Dalal, Daniel Koceja, Jiarui Xu, Yue Zhao, Shihao Han, Ka Chun Cheung, Jan Kautz, Yejin Choi, Yu Sun, and Xiaolong Wang. One-minute video generation with test-time training. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17702-17711, 2025.
Tri Dao and Albert Gu. Transformers ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https:/ / /arxiv..vvvv org/ abs/2405. 21060.
Tri Dao, Albert Gu, et al. Hungry Hungry Hippos: Towards language modeling with state space
models. arXiv preprint arXiv:2312.00752, 2023.
Sri Harsha Dumpala, Chandramouli Sastry, and Sageev Oore. Test-time training for speech, 2023.
URL httpt:htt/aaaae /aapiv//aaiivvv/aa org/abs/2309 10930.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Brown, and et al.
mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https://
transforree-circukee http://eee..aaaaaaaaaaa/a/aa///a
html.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2 101i:2201.0002... 2020. URL attpss://arxt orgo abs////////// /atss///10 2101 00027.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: www.oooooo.//..ooooooooeeooeeeee
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. arXiv preprint 2...:20022.149133 2020.
Team GLM. Chatglm: A family of large language models from glm-130h to glm-4 all tools, 2024. URL hwtpt:ttt////..... /ttp///axvvvvvv/vv .rrg abs 6 12793.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. In ICML, 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021b. URL https: /tte://///vivvieii wttp:/iwwwwgg///////////////
03300..
12

---

Published as a conference paper at ICLR 2026
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's 's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024. URL https: wtw.v.iiiiiv.//ii./...aaaaaaaeva http:/awwww...gaaa////a/ /2404 06654.
Jinwu Hu, Zhitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanging Li, and Mingkui Tan. Test-time learning for large language models. arXiv preprint urXiv:2505.20633 2025. URL https::///. / a arw.a.vvvooommm wrgg
http://oowooooooa////////// / 2.0.. 20633. Accepted at ICML 2025.
Kazuki Irie and Samuel Gershman. Fast weight programming and linear transformers: from machine learning to neurobiology. 2025. URL pottps://aet http///axvvvv///////////..a org/ abs 2508 08435.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ama Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //rxiv.. org/abs/ 231 06825.
Mahdi Karami and Vahab Mirrokni. Lattice: Learning to efficiently compress the memory. arXiv preprint urXiv:2504.056-h 2025.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rons: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2020. URL https: arxivv. org/ abs/ 6236.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. augmenee generation for knowledge- intensive nlp tasks. NeurIPS, 2020.
Zeman Li, Ali Behrouz, Yuan Deng. Peilin Zhong, Praneeth Kacham, Mahdi Karami, Meisam Razaviyayn, and Vahab Mirrokni. Tnt: Improving chunkwise training for test-time memorization. arXiv preprint arXiv:2511.0734 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda La, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXhv:2412.19437, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac HatfieldDDodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, arXiv preprint arXiv:2209. 1895, 2022. URL httpt::://// https///axxxvv.ooo abs 2200 11895.
OpenAl. Gpt-4 technical report. arXiv preprint urXiv:2303.08777 2024.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. Llama gra- dient: A series of long context models, 2024. URL https::/grradient.te http://ww.....ccccccccc..eeeooar
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007|, 2023.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pp. 9355-9366. PMLR, 2021.
David Silver and Richard S Sutton. Welcome to the of experience. Google AI, 1, 2025.
13

---

Published as a conference paper at ICLR 2026
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 9229-92244 PMLR, 2020. URL https: ptttp::/pproceedingg. mlr press/ v119 //s/uun000 0b. html.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Runs with expressive hidden states, arXiv preprint arXiv:2407.04620 2024.
URL https:tpp/// /tt///aaxx /tta/aaivvvorr.. abs 2407 04620.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, arXiv preprint arXiv:2307.08621 2023.
TogetherAI. Long data collections database, 2024.
Hugo Touvron, Théo Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Ke Alexander Wang, Jiaxin Shi, and Emily Fox. Test-time regression: a unifying framework for designing sequence models with associative memory. arXiv preprint arXiv:2501. 12352, 2025.
Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang. Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang. and Julian McAuley. Memoryllm: Towards self-bpdatable large language models, 2024. URL https::///// aariiv org/ abs /22//2404 22022 04624.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi. Quoe Le, and Denny Zhou, Chain-of-thouuht prompting elicits reasoning in large language models, 2023. URL https: wwwaiivvvvoo........ abs/ 11903.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 1:XXi:2505509388 2025.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06633, 2023. URL
https://arxe
org/ als 2312. 06635.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.066464 2024a.
Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efffcicen training. In International Conference on Machine Learning. pp. 56501-56523, PMLR, 2024b.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv.2406.06488, 2024c. URL https: // rxiv. wrg.ass.r/abba..../ooo.oo..o 6 0...6.664555 6684.
14

---

Published as a conference paper at ICLR 2026
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. The Thirty- Annual Conference on Neural Information Processing Systems, 2024d.
Morris Yau, Sharut Gupta, Valerie Engelmayer, Kazuki Irie, Stefanie Jegelka, and Jacob Andreas. Sequential-parallel duality in prefix scannable models, 2025, URL /tttps://arxiv.... org/ abs/2506. 10918.
Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping long-context Ilm with multi-conv rl-based memory agent, 2025. URL httph:ttp///..m httpt::/t//aaaee /ttps://arxvvvoi wtwpaia/.rraaa//a.eeeaoaa..oaaaa httt::://axxiv.orr/aab/////// /2507. 02259.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL
2ttp://wwww.....////...22.2..... 11089.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019. URL aattps://arte http://awwwww.gaaa///////a//a//a 07830.
Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T. Freeman, and Hao Tan. Test-time training done right. arXiv preprint wrXi:22505.2884, 2025. URL https:://// / arxiv. www.vv.rgg/aboo. abs/aabs...5.../// 23884.
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Test-Time Training (TTT). Test-Time Training (TTT) is a paradigm that enables a model to adapt dynamically in response to continuous streams of data at inference by updating a small subset of its parameters, known as fast weights (Ba et al., 2016). Initially demonstrating success in computer vision (Sun et al., 2020; Wang et al., 2021), TTT has since been extended to numerous other modali- ties ties-including language (Sun et al., 2024). video (Dalal et al., 2025), and audio (Dumpala et al., 202d)_underscoriing its broad applicability. Research in this area has largely focused two avenues for improving TTT's effectiveness: the design of more sophisticated test-time optimizers (Behrouz et al., 2025a) and the formulation of novel, self-supervised online learning objectives (Behrouz et al., 2024; Karami & Mirrokni, 2025). However, the computational efficiency of TTT remained a critical bottleneck due to its inherently sequential, per-token update process. The chunk wise Training framework was the first to directly address this challenge by introducing chunk- wwie update mechanism to better leverage parallel hardware (Behrouz et al., 2024; Irie& & Gershman, 2025; Zhang et al., 2025; Sun et al., 2023; Yau et al., 2025; Li et al., 2025). Despite these advances, TTT's function as the primary token mixer necessitates reliance on small chunks to preserve performance-thie in turn creates bottleneck that limits the massive parallelism needed to fully utilize modern accelerators. Furthermore, prior work has not addressed how to seamlessly integrate TTT into large, pre-trained models, nor developed learning objectives specifically tailored for the autoregressive nature of LLMss, which are gaps our work directly addresses.
Efficient Long-Context Architectures. A parallel line of research seeks to extend the effective context window of LLMs by mitigating the quadratic complexity of the standard attention mechanism.
Major approaches include: 1) Sparse attention methods, which restrict the range of token-to-token interactions via fixed patterns like sliding or strided windows (Child et al., 2019; Beltagy et al., 2020; Yuan et al., 2025); 2) linear-tmme variants. which approximate the attention mechanism or replace with efficient recurrent or gated formulations, such as linear attention (Katharopoulos et al., 2020; Schlag et al., 2021) Gated Linear Attention (GLA) (Yang et al., 2023); and 3) State-Space Models (SSMs). which compress sequence history into a compact latent state, enabling processing with linear complexity (Dao et al., 2023; Dao & Gu, 2024). Recently, the delta rule has emerged as a popular design choice for linear attention and SSMs, enabling better experessivity and highly parallelizable implementations (Yang et al., 2024c). These architectural advances are complementary to our framework. While they focus efficiently processing long contexts, TTT provides mechanism for online adaptation to the information within that context. Our In- In-Place TTT can be also naturally integrated with these efficient backbones, as they also have MLP blocks. And we leave these as the future work.
Memory Design and Augmentation. A related domain of research involves augmenting neural architectures with explicit memory modules to enhance their reasoning and contextual understanding capabilities. These approaches can be broadly distinguished by their function: some are designed to store persistent, task-agnostic knowledge in an external memory bank, while others focus on capturing transient, data-dependent information from the immediate context (Khandelwal et al., 2020; Guu et al., 2020; Lewis et al., 2020; Wang et al., 2024; Yu et al., 2025). The latter, contextual memories, have been implemented using various mechanisms, including recurrent state transitions, attention- based context aggregation in Transformers (Dai et al., 2019), and rapid, gradient- based updates to fast weights (Yang et al., 2024c). Test-Time Training (TTT) represents a powerful instance of this latter category, which conceptually extends the notion of hidden state found in Recurrent Neural Networks (RNNs). Rather than compressing contextual history into a fixed-size activation vector, TTT designates subset of the model's 's own parameters- the fast weights-to function hagh-capacity, dynamic memory (Ba et al., 2016; Schlag et al., 2021). These weights are updated on the fly at inference time, allowing the model to continuously internalize evolving contextual information and thereby function as an expressive, online evolving state.
B PROOF OF THEOREM 1
For completeness, we first restate the theorem with the precise bounds derived from the assumptions. Theorem (Logit-wise Effect of LM-Aligeed Target v.s. Reconstruction Target (Restated)). Under the specified setting and assumptions, for learning rate > > 0, the expected change in logits Al..
16

---

Published as a conference paper at ICLR 2026
after update step using the NTP-aligned target satisfies:
(Correct logit increases)
IE [ Xir Grom Caligns
(4)
(Other logits almost unchanged)
E [ & X Calign:
Vw # v²
(5)
In contrast, for the reconstruction target, the expected change in logits is negligible for the correct token:
E R '0 II S Air E Calign.
(6)
Proof. We begin from the setup defined in Section 3.3. The change the fast weights W down from all prior context tokens is given by A down down = Arr Eteprior V.Z, where we use D to denote the
learning rate for consistency with the theorem statement. The resulting change in the logit for an arbitrary token at the query position is:
[n [w] e (AW Lown)2n.
Air
Σ
(Vi2, )zn.
téprior
Since e, and z, Zn scalars, rearrange the terms get:
Alr [w] Avr
Σ vz) 2n).
(8)
DEprior
To analyze the expected change, we take the expectation over the representations. Applying the linearity of expectation, have:
B[Aen[w)] [w]] = Avr
Σ
Bl( 2n)]
tEprior
Per our setup, the target vectors V, (e.g., ex, or @ are treated as determined. Thus, we can factor them out of the expectation:
E B[A,//w]] = Ahr Σ E.v,2, z,
(10)
rEprior
Now, we invoke Assumption 2. It states that for the unique key position 1", we have Elz,, zn] = Calign and for all other prior positions t * t", the updates provide no information gain, which implies single term corresponding to the key-value pair
El,,z, z,] O. This simplifies the summation
(k* w² at position !t:
E [ [w]] = Air E [[c] v..).
(z,,. 2..]
(11)
We now analyze this simplified expression for the two target choices.
Case 1: NTP-Aligned Target (V-- = ex
First, we consider the logit of the correct token, = v². Substituting the target and the token into Equation (I1) yields:
[A [v*]] = Air E [(eT.. ew-) (z zm.]
(12)
By Assumption 1, token embeddings have toivial magnitude, ||e 112 2 This gives us the lower bound in Equation (4):
E [ 2 A Calign C
(13)
Next, for any incorrect token w 3 v², the expected change is E [Ae, [w]] E (e.Te (z.T. zn) ] Taking the absolute value and applying Assumption 1, which states that distinct embeddings are nearly orthogonal (fe],, <), obtain the bound in Equation (5):
IE Ae, [W][W]]]
Air Calign
(14)
Case 2: Reconstruction Target (V:- Oz.))
17

---

Published as a conference paper at ICLR 2026
Here, we analyze the effect on the correct logit o². The expected change is:
E [v*] EE [(.) (z 2n]]
(15)
In induction task, the key k* is distinct from the value v". We again invoke Assumption 1 for these distinct tokens. Taking the absolute value gives the bound in Equation (6):
[[
S Air
Calign
(16)
This confirms that the reconstruction target has negligible expected effect on the logit of the correct answer o*.
The results from these cases establish the claims in Theorem 1. providing clear theoretical basis for the superiority of the NTP-aligned objective in the context of in-context learning. This completes the proof.
C CONTEXT PARALLEL ALGORITHM FOR IN IN-PLACE TTT
For more clarity, list the pseudocode of the context parallel implementation of our In-Place TTT here in Algorithm 1.
Algorithm In-Place TTT with Context Parallelism (Single Layer)
Require: Pre-trained weights (incl. Waps W Wgater Wr(0 down); Conv ID kernel K; projection W Wargeti
learning rate 7).
1: Input: Sequence chunks {x(0)
Sequence partitioned for Context Parallelism (CP). D Step Compute update deltas. Standard attention, no changes required.
2: for all E (I....:77 7) in parallel do
3:
H4, < AttentionBlock( (x()))
4:
Us, G. H.W up H.W gate
5:
Zi < ((G)) @ U,
6:
Vi + Conv1Dx (x( W
Compute NTP-aligned target with causal padding. Compute gradient for the fast weight update.
7:
AW, + V,T Zi
8: end for
9: (S.TT + CUMSUM((AW.)T
Aggregate deltas associatively. >> Step 3: Apply updates and compute outputs. > Effective weight for chunk uses updates from chunks < i.
> Step 2:
T in parallel do 11: w(--)) + ((0))
10: for all i {I....,
down + inS, down
12: O (Wx(()))
13: end for
At document boundaries: Reset fast weights W Wo(o) down
D EXPERIMENT DETAILS
This appendix provides all details of the experimental settings, datasets, model configurations, and training hyperparameters used for the results presented in Section 4. The following subsections detail the setups for our three primary sets of experiments: the continual pre-training of Qwen3-4B 1e-nn----B-ar 4B-Base, the from-seratch pre-training of models multiple scales (500M. 1.5B, and 4B), and the targeted ablation studies. Our goal is to provide sufficient detail to ensure the reproducibility of our findings.
D.1 DETAILS OF DATASETS
For the large scale pretraining, continual pretraining, and ablation study, we use the dataset collected by ourselves, give the details of these datasets as follows.
From Scratch Pretraining Dataset. The pretraining dataset mainly includes general English and Chinese text, along with high knowledge- reasoning-density data, code, mathematics data, and
18

---

Published as a conference paper at ICLR 2026
multilingual text, forming balanced mixture of linguistic diversity, knowledge and reasoning-rich content, programming material, and mathematical reasoning.
Continual Pretraining Dataset. The continual pretraining dataset is designed to enhance long- context modeling: its short-document portion follows a distribution similar to Pretrain Data, while the long-document portion combines natural data such books and repository-level code with synthetic data including retrieval-augmenned and long-context-Q4 style constructions, ensuring both consistency with pretraining and coverage of challenging long-context scenarios. The data organized into subsets with maximum sequence lengths of 32k and 128k for our two-stage training curriculum. During the continual pretraining experiments, we set the target value origin from the hidden states of the current layer, instead of the input embeddings. The convolution and projection are maintained process the hidden states to get the target value.
D.2 DETAILS OF TRAINING AND EVALUATION
Training Details. All models are trained on Nvidia H800 GPUs, with the detailed training hyperpa- rameters listed in Tables 4 through 6.
Table 4: Training hyperparameters for 500M and 1.5B models.
Hyperparameter
500M Model
1.5B Model
Optimizer
AdamW
AdamW
Learning Rate
5e-4
3e-4
Batch Size
2M tokens
4M tokens
Weight Decay
0.1
0.1
Gradient Clipping
1.0
1.0
Warmup Steps
1024
1024
Sequence Length
32,768
32,768
Tokens Trained
20B
60B
Sliding Window Size
2,048
4,096
Table 5: Training hyperparameters for 1.7B models and 4B models pretraining
Hyperparameter
value
Optimizer
AdamW
Learning Rate
3e-4
Batch Size
8M tokens
Weight Decay
0.1
Gradient Clipping
1.0
Warm-up Tokens
1.6B
Sequence Length
8,192
Tokens Trained
120B
Table 6: Hyperparameters for two-stage continual pre-training.
Hyperparameter
Stage (32k Context)
Stage 2 (128k Context)
Base Model
Qwen3-4B-Ban
Qwen3-4B-Ban
Optimizer Learning Rate
AdamW
AdamW
5c-6
5e-6
Weight Decay
0.1
0.1
Sequence Length
32,768
131,072
Tokens Trained
20B
15B
RoPE Extension
None
YaRN
Cony Size
19

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for continual pre-training of LLaMA-3.1-881 and Qwen3-14B-Ban
Hyperparameter
L.LMA-3.1-8B
Qwen3-14B-Base
Optimizer
AdamW
AdamW
Learning Rate
5e-6
5e-6
Weight Decay
0.1
0.1
Sequence Length
32,768
32,768
Tokens Trained
~0B
~20
RoPE Extension
None
None
Conv Size
5
Evaluation Details. We employ the evaluation framework im-evaluation-harte harness (Gao al., 2024) to evaluate the models on the common sense reasoning benchmarks and employ the evaluation framework opencompass (Contributors, 2023) evaluate models on the long context benchmarks. All evaluation are conducted on Nvidia H800 GPUs.
In the evaluation our continual pretrained Qwen3-4B model, we apply a clipping mechanism at inference time to ensure stable fast-weight updates. Specifically, if the Frobenius norm of update delta | (
down |r exceeds a predefined threshold т, the delta matrix rescaled have norm before
downllp. lp. This prevents the accumulated updates from growing unboundedly as the sequence length increases, thereby maintaining numerical stability for long-context inference. For all reported evaluations of the Qwen3-4B model, this threshold was set to r = 1e-5.
being applied, i.c., A Aw(() down
A W(() down/ (Awlo down
To evaluate the efficiency of our In-Place TTT. evaluate the prefill throughput and peak memory for sequence length ranging from 8k to 128k. We run the inference for our continual pretrained checkpoints based Qwen3-4B-Ban attention mechanism of these pretrained checkpoints to sliding window of 1024 tokens manually. We run the inference Nvidia H800 GPUs with batch size of 1.
model. For the setting of sliding window, set change the
D.3 DETAILS MODEL CONFIGURATION
This section details the architectural configurations of the models used in our experiments. All models are decoder-only Transformer architectures featuring standard components, including SwiGLU acti- vations and Rotary Position Embeddings (RoPE) (Su et al., 2023). The key architectural parameters for models trained from scratch are summarized Table
Table 8: Model architectural configurations for 500M and 1.5B Model.
Parameter
500M
1,5B
Parameters (Approx.)
500M
1.5B
Hidden Size (dmodet)
1024
2048
Num Layers
24
24
Num Attention Heads
16
FFN Hidden Size (da)
3072
6144
Window Size
2048
4096
Vocabulary Size
32,000
32,000
Rope Base
le6
The models trained from scratch employ different attention mechanisms based on their experimental purpose. The 500M, 1.5B utilize sliding-window window attention and we list the model configuration in Table The 4B-scale experiments and ablation study evaluate two variants: one with full attention and another with sliding- wiiddng-window attention. The backbone architectures for the 4B models and 1.7B models are identical to the Qwen3 --------aas -B-a model and the Qwen3- 1..7--Bass model.
For the continual pre-training experiments described in Section 4.1, start directly from publicly available pre-traine trairrined models- Qwen3-4B- Ba-e (Yang et al., 2025), LLaMA-3.. 1-BB (Grattafiori et al., 2024), and Qwen3- 14B- BB-Base (Yang et al., 2025) 2025)-inheriting their architectures without modification.
20

---

Published as a conference paper at ICLR 2026
In experiments featuring method, the In-Place TTT module is integrated into the MLP blocks and applied to every sixth layer. For the ablation studies, this frequency is varied as described the main paper. The training hyperparameters for Qwen3-4B-Bin Base are listed in Table 6, and those for LLaMA-3.1-88 and Qwen3-14hh 1---BBse are listed in Table 7.
Initialization of In-Place TTT Modules. When integrating In-Place TTT into pre-trained model for continual training. careful initialization is essential to preserve the model's ' pre-trained capabiliiiess the start of training. Specifically, we initialize the newly introduced TTT components-the Conv ID operator and the projection matrix W Wugert-uch that the TTT update A AW down is negligible initialization, ensuring the model begins from its original pre-trained behavior. Concretely, the depthwise ConvlD (kernel size 5, causal padding, no bias) is vero-initialized so the target V is zero at initialization. The projection matrix W uugeet E IR Rudde Xdadtal is initialized as a sparse diagonal matrix, where all off- off-diagonal entries are zero and the diagonal entries are drawn from N (O, o²) with being the model's standard initializer range. This near-zero initialization of both components guarantees that the initial fast- weight update 17 V | O, and consequently the effective W W down remains identical to its pre-trained value. As training progresses, the ConvID and projection gradually learn to produce meaningful NTP-aligned targets, allowing the TTT mechanism smoothly emerge without disrupting the pre-trained model.
E
USAGE OF LLMS
During the preparation of this manuscript, LLMs was used to check grammar and improve readability. All authors have reviewed, edited, and take full responsibility for the paper's final version.
21


Reviews:
[{"review_id": "ny9sdofC3a", "reviewer": "Reviewer_TXQK", "summary": "The paper introduces a novel in place test time training framework that enables LLMs to dynamically adapt a set of ‘fast’ weights during inference time with new information. The authors provide a way to repurpose the existing final projection matrix of the MLP blocks of the decoder transformers and avoid costly retraining. They also provide an efficient chunk wise update rule and an objective that aligns with the next token prediction task. Lastly they also provide a context parallel algorithm. Experiments on a 4B qwen show that the approach achieves improvements over baseline. Good practical paper, just a few questions below.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method is practical and drop in design that is novel and straightforward to integrate in practical deployments\n2. The authors have made the learning objective aligned with next token prediction from a generic reconstruction objective and show ablations to prove their point\n3. The authors also additionally provide context parallel algorithms for maximal computation efficiency", "weaknesses": "1. The experimental section is based on a single family of model qwen. What if selecting that specific layer is suboptimal in some other family of models like llama. What will the authors do with a mixture of experts paradigm?\n\n2. The authors provide limited justification for selecting the MLP block’s final projection matrix. There are other MLP blocks with large matrices inside the decoder transformer as well, perhaps the attention layers output projection matrix etc. This seems like a heuristic and an ablation could make it clear which layer is the optimal layer to choose for this purpose. What if a combination of layers provides the best performance. \n\n3. On that note isn't it possible to do a modified Lora TTT[1]? For example, instead of changing the weights of the model itself directly as fast weights, we keep a set of lora weights that we keep updating at inference time, and in lieu of its small flops footprint, instead of changing the full matrix of only the projection matrix, one can now change a few different matrices across the transformer block in a flop matched regime. Isn't that a reasonable baseline to consider?\n\nReferences: \n1. Kojima, Yuto, Jiarui Xu, Xueyan Zou, and Xiaolong Wang. \"Lora-ttt: Low-rank test-time training for vision-language models.\" arXiv preprint arXiv:2502.02069 (2025).\n\n\n3. Some figures like figure 3 are missing legends.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jkvUvNTu7P", "reviewer": "Reviewer_LqGp", "summary": "This paper introduces a Test-Time Training framework for LLMs which does not require any additional layers / parameters and can be introduced in existing pre-trained LLMS with some training as well as when a model is being trained from scratch. The main idea is to repurpose the down projection component of a GLU styled MLP into fast weights. Compared to previous fast-weight and TTT implementations, they also introduce a separate objective for the fast-weights which is more in tune with the overall LLM task of next-token prediction and is also theoretically motivated. Based on these two core ideas, the authors perform extensive evaluation of their technique, called In-Place TTT, on pretrained LLMs using continual learning as well as training LLMs of different sizes from scratch. They compare IPTTT to full attention, sliding window based attention methods and demonstrate the effectiveness of their scheme.", "questions": "The major confusion I have about the methodology is understanding how it works during inference. In regular fast weights, the goal is to make sure the fast weights compress the context (the reconstruction objective). But here, it is to have them predict info about the future tokens using V. However, V for future tokens is not available during inference, so how do the fast weights even updated?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, the core strength of this paper lies in the idea of repurposing the existing weight matrices in a neural network and using them as fast weights. This means their technique can be used to improve context management in existing LLMs as well as train large scale models using TTT. Moreover, the technique uses TTT not at the attention level but at the MLP block level which allows for efficient parallelization of the IPTTT framework using prefix sums and chunking sequences into token blocks. The theoretical analysis seems grounded and correct although I did not verify it fully. The experimental section is also done well and I don’t have any major concerns. The performance difference between their technique and the baseline, especially on long context tasks shows the promise of this technique as well. I believe this work will be useful to the community to build upon.", "weaknesses": "The main issue I have with the paper is certain sections confused me a little bit. There were some language issues, typos, and a few instances of confusion caused by notation. Here are some typos / language improvements I would suggest.\n\nLine 161 -  also viewed  → also be viewed\n\nLine 192 -  onece → once\n\nLine 175 - we adapt only the MLP blocks → we only adapt the MLP blocks \n\nAlso, in the paragraphs on Efficient Adaptation with Chunk-Wise Updates, the target is referred to as V but then later it becomes V^{hat}. Is there a reason for that?\n\nAlso, I believe the authors have made bit of an overstatement in the explanation of  barrier (ii) \"the canonical per-token update mechanism of TTT is inherently sequential, severely bottlenecking the parallel processing capabilities of modern accelerators. \nAnd also here : \"This chunk-wise update strategy is designed for modern hardware. By processing large blocks of\ntokens at onece, it highly leverages parallelism and utilizes the computational power of GPUs or\nTPUs, thus resolving the efficiency bottleneck that hinders prior research.\"\n\nWhile it is true that Fast-weight like implementations are inherently sequential, chunk-wise training is not a new approach and was already explored / discussed in several fast-weight like papers e.g [1], [2]. I suggest the authors adjust for this.\n\n\n[1] Titans: Learning to Memorize at Test Time \n\n[2] : Fast weight programming and linear transformers: from machine learning to neurobiology", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wc2PQpeX87", "reviewer": "Reviewer_cw7y", "summary": "In this paper, the authors propose a new method that allows small LLMs to adapt and learn from new information while they are being used, rather than only during initial training. The approach, called In-Place Test-Time Training, updates specific internal components of the model efficiently without needing full retraining. Concretely, it uses the last layer of MLPs in transformer blocks. Further, it also introduces a new learning goal aligned with the model’s main prediction task and uses an optimized update process for speed and scalability. The authors show some experiments in small to moderate size LLMs showing that this method improves performance on long and complex tasks while remaining computationally efficient. Finally, they present a set of nice ablation studies.", "questions": "It would be nice if the authors are able to show those experiments. I would increase my score in such a case.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a very good paper. In particular, I liked the following aspects of the paper:\n\n1) The idea of the paper makes perfect sense, and the problem it tackles is important. Having LLMs that can be adapted at test time, improving their performance, is definitely a step in the right direction towards building strong models.\n\n2) The authors solve the method using two simple key ideas: (1) using only the last linear layers in MLPs of Transformer blocks, (2) using a novel objective that somehow aligns with the task of next-token predictions. In both cases, these idea make complete sense, and are shown to be effective, despite being quite simple.\n\n3) Because, the authors are using only a small subset of parameters, I assume that the method is quite computationally efficient.\n\n4) As shown in several experiments, the method seems to improve over the baseline.\n\n5) The paper is excellently written. I had a good time reading and understanding the paper.", "weaknesses": "1) The main weakness of the paper is that it has been evaluated in a relatively limited setting. Qwen3-4B, while a strong baseline, is still a very small LLM for modern standards, thus it is unclear if the method shows the same effectiveness when used with larger, or even significantly larger LLMs, which tend to be state-of-the-art.\n\n2) Furthermore, it would have been cool if the authors would have checked the effectiveness of their method in other families of LLMs (even if having the same size), for example LLama, Mistral, Intern etc.\n\n3) Finally, would the results hold if going in the domain of VLMs. Such results be it on Qwen3-VL or some version of InternVL would make the contribution of the paper even more solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)