Title: Test-Time Training Done Right
Venue: NeurIPS 2025
Accepted: no
Keywords: Fast weight

long context models

visual memory

video generation

multimodal

test-time-training
Abstract: Test-Time Training (TTT) models context dependencies by adapting part of the model's weights (often referred to as fast weights) at inference time. This adapted fast weight, similar to recurrent states in RNNs, stores temporary memories of past tokens in the current sequence. Existing TTT methods have struggled to demonstrate effectiveness in handling long-sequence data, due to their computational inefficiency on modern GPUs. The TTT layers in
many of these approaches operate with extremely low FLOPs utilization (often below 5\%) because they deliberately apply small online mini-batch sizes (e.g., updating fast weights every 16 or 64 tokens). Moreover, a small mini-batch implies fine-grained block-wise causal dependencies in the data, making them unsuitable for data beyond 1D ordered sequences, like sets or N-dimensional grids such as images or videos.
In contrast, we pursue the opposite direction by proposing an extremely large chunk update, ranging from 2K to 1M tokens across tasks of varying modalities, which we refer to as Large Chunk Test-Time Training (LaCT). 
This approach improves hardware utilization by orders of magnitude, and more importantly, facilitates scaling of nonlinear state size (up to 40\% of model parameter size), hence
substantially improving state capacity, all without requiring cumbersome and error-prone custom kernel implementations.
It also allows easy integration of sophisticated optimizers like Muon for online memory updates.
We validate our approach across diverse data modalities and tasks, including novel view synthesis from image sets, language models, and auto-regressive video diffusion models. 
Our approach can scale up to 14-billion-parameter auto-regressive video diffusion models handling sequences of up to 56K tokens. 
In our longest sequence experiment, we perform novel view synthesis with more than one million context length. Our results highlight the computational and performance benefits of large-chunk test-time training, paving the way for more efficient and scalable long-context sequence modeling. We hope that this work will inspire and accelerate new research in the field of long-context modeling and test-time training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTest time training has the goal of increasing model efficiency and performance by adapting a subset of network parameters during inference. This paper proposes a new approach to this problem by introducing algorithmic and architectural innovations that allow for the use of large chunk sizes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The proposed approach is general and can be utilized and adapted to a variety of different contexts and applications.\n- It can be easily integrated with current deep learning libraries.\n- The performance was demonstrated in multiple challenging tasks. \n\nWeaknesses:\n- Clarity of presentation can be improved as the paper currently requires a very good familiarity with TTT.\n- The maximum number of TTT parameters tested is about 40% of the total number of model parameters. The authors should provide experiments to show the effect of further increases in the number of parameters or discuss the potential of this change.\n\nQUESTIONS:\n- The maximum number of TTT parameters tested is about 40% of the total number of model parameters. Can the authors provide experiments to show the effect of further increases in the number of parameters or discuss the potential of this change.\n\nLIMITATIONS:\nSuggestions to increase clarity:\n-Fig. 3 is not clear. I agree that the decoupling between the update and apply functions allows for interesting combinations but what do the blue squares indicate and how are they linked to the architectures shown above?  \n- Fig. 3: the arrow from update to apply should be labeled f(w) to avoid confusion. \n- In the video diffusion experiment, the frozen weights are used for denoising. Please clarify the role played by the TTT wights. \n- The L2 norm equation for weight update should be introduced earlier in 2.1\n- Mention the GPU resources used", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to scale up the existing \"test-time training\" (TTT) paradigm from [1] by extending this methodology with large batch sizes across many inputs in parallel. The central idea is to combat the poor observed FLOP utilization of TTT with small batches without custom kernels. The resulting pipeline, named LaCT (large chunk TTT), is tested primarily on novel view synthesis and also on other different applications with transformers, as language and video modeling.,\n\n[1] https://arxiv.org/abs/2310.13807\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n\n1. The paper targets three relevant empirical domains to validate their scaled-up version of TTT. The most significant of the results appears to be for the view synthesis application, which shows TTT can achieve only slightly inferior performance to full-attention transformers.\n2. The paper introduces the prior work of the TTT paradigm and its efficiency challenges when considering 'naive' Python implementations, as well as the traditional need for specialized kernels, both succinctly and clearly (Section 2).\n3. Some of the figures are explanations (e.g., ordering in of apply and shift operations to simulate a block causal mask in Figure 3) are intuitive and would again provide a nice explanation for some implementation decisions needed for applying TTT to different domains,\n\nWeaknesses and areas for improvement.\n\n1. The core contribution appears as minor hyperparameter changes, such as substituting the 2-layer MLP in (2) with a SwiGLU 2-layer MLP, and the slight change in the fast weight loss, removing a |f_W|^2 term from the L2.\n2. Claims in the checklist appear to be poorly justified or even completely false. For instance, the authors answered [YES] to the questions regarding code sharing and reproducibility. Yet, they did not provide any code with the submission, with this claim being \"justified\" by a promise of sharing code conditioned on acceptance.\n3. Related to the point above, as I believe that the main paper's advancement is an efficient implementation, rather than a new method, algorithm, or idea, its contribution largely hinges on the author's promise of sharing the code upon acceptance.\n\n\n4. The central claim that the proposed method improves \"improves hardware utilization by orders of magnitude\" in the Appendix (e.g., lines 15-16) seems to be entirely dependent on the specific hardware (an A100 GPU in this case). This claim should be properly contextualized to avoid potentially misleading a non-familiar audience. Other claims in the paper, such as \"Language Modeling. Our model achieves competitive performance compared to SoTA methods such as DeltaNet,\" (lines 62-63) also seem potentially misleading. DeltaNet is a model that has been partially validated mostly on very small-scale language modeling and synthetic tasks like MAD [2].\n\n5. Many critical details are deferred to the Appendix. However, references to these details are never specific. E.g., lines 240, 252, ... \"See Appendix for details.\" At least the authors should have provided Section numbers to facilitate parsing the text. Moreover, I could not even find many of these details anywhere e.g., lines 51-53 \"LaCT significantly enhances parallelism, leading to substantially improved GPU utilization with just a few dozen lines of pure PyTorch code (see the Appendix).\" There seems to be no lines of Python code in the appendix.\n\n6. While most of the contributions appear to be changes in architectural and other small hyperparameter choices (e.g., SwiGLU vs 2-layer MLP, new loss function etc.) There is a lack of proper validation with, for instance, target ablations and other kinds of small-scale analysis.\n\n[1] https://arxiv.org/abs/2310.13807\n[2] https://arxiv.org/abs/2403.17844\n\nQUESTIONS:\n1. In contrast to the paper's initial claim, the original TTT implementation from [1] already seems to use JAX compilation for kernel fusion and a few updates (up to 4) with a very large batch size in their pixel imagenet experiment. In fact, they show almost equivalent cost of their TTT layer with standard linear attention. Am I missing something?\n\n[1] https://arxiv.org/abs/2310.13807\n\nOverall, while I believe the paper shows some potentially interesting results applying TTT on novel view synthesis, it seems to lack several components to support its very specific contribution (e.g., ablations and analysis of the novel design decisions), and seems to have numerous areas of clear improvement as outlined above. Thus, I do not believe the current submission is ready for acceptance.\n\nLIMITATIONS:\nMany of the above-mentioned limitations, such as the lack of proper ablations and analysis, are never explicitly discussed. Instead, the authors mention aspects that do not even seem as limitations in comparison e.g.: lines 566-567 \"We conduct our experiments on three tasks. Although the tasks are diverse and cover different  modalities, the effectiveness of our method would request of more tasks.\"\n\nMoreover, the authors never discuss societal implications. Their justification seems weak \"This is a general method paper and it does not have specific concerns regarding 807 societal impacts comparing with other papers.\"", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the computational inefficiency and scalability limitations of existing Test-Time Training (TTT) methods for long-sequence modeling. Current TTT approaches often use small mini-batches for online updates of \"fast weights,\" leading to poor hardware utilization on modern GPUs. The authors propose Large Chunk Test-Time Training (LaCT), a method that updates fast weights using extremely large chunks of tokens, ranging from thousands to a million. To handle local dependencies within these large chunks, LaCT is combined with a window attention mechanism. The paper validates this approach across diverse modalities, including novel view synthesis, language modeling, and autoregressive video diffusion.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n- The LaCT approach demonstrates enhanced GPU utilization for TTT\n- The authors evaluate their method on three different domains including novel view synthesis, language modeling, and autoregressive video diffusion\n- Several interesting ablations (unfortunately buried in the appendix)\n\n**Weaknesses:**\nThe paper's primary weaknesses lie in its empirical evaluation, unclear attribution of performance gains, and overall structure.\n- **Insufficient Empirical Evaluation:** The validation of the proposed method is unconvincing across multiple domains.\n   - For language modeling, the authors rely on validation loss and a simple retrieval task (S-NIAH). This is insufficient; standard practice, as seen in baselines like DeltaNet [10] that is used by the authors, involves evaluation on a suite of downstream tasks (e.g., ARC, Hellaswag) to properly assess a model's capabilities beyond simple long-context recall.\n   - Similarly, for autoregressive video diffusion, performance is measured solely by denoising loss on a validation set. This metric is notoriously uncorrelated with perceptual quality. The evaluation lacks standard video generation metrics or qualitative examples, making it impossible to judge the model's actual performance in terms of temporal consistency or frame quality. The authors acknowledge this limitation in the appendix, but it remains a significant flaw in the experimental design. See the work by Yin, Tianwei, et al. [1] for relevant benchmarks and evaluation measures used in the field (e.g. temporal quality, frame quality, text alignment).\n\n[1] Yin, Tianwei, et al. \"From slow bidirectional to fast autoregressive video diffusion models.\" _Proceedings of the Computer Vision and Pattern Recognition Conference_. 2025.\n\n- **Unclear Source of Performance Gains:** The paper fails to clearly disentangle the benefits of the core LaCT architecture from the optimizer used. In the language modeling experiments, the performance improvement is almost entirely dependent on the *Muon optimizer*; the variant using a momentum optimizer performs on par with the DeltaNet baseline (Figure 5). This suggests that the enhanced results may stem more from the choice of optimizer than from the proposed large-chunk training strategy itself. The authors argue that LaCT's efficiency enables the use of such sophisticated optimizers, but they do not provide evidence that other TTT methods could not also benefit from Muon, which is a critical missing ablation. Furthermore, the paper lacks direct comparisons against other relevant baselines that combine local attention with recurrence, such as InfiniAttention or MEGA, which are only briefly mentioned in the appendix's related work section.\n\n- **Poor Paper Organization and Clarity:** The structure of the paper hinders a clear understanding of its contributions and context. A discussion of related work is relegated to the appendix, which is a significant omission for a main conference paper, especially given that Test-Time Training is not a widely studied field. This section is crucial for positioning the work and should be in the main body (related work concerning novel view synthesis and video generation can be left for the appendix). Conversely, key ablation studies that are vital to understanding the method's design choices—such as the impact of state size and different optimizers are also in the appendix. The main paper's space could have been better utilized by moving some training/implementation/dataset details to the appendix and featuring these more critical analyses instead.\n\n\nMinor points:\n- References 10 and 15 are the same. Only keep 15 which is a NeurIPS citation rather than arxiv.\n- Figure 5: there is a typo in the word validation\n- Typo in \"Muon\" optimizer mixed with \"Moun\" in several places the paper\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nLimitations are discussed in the Appendix only.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Large Chunk Test-Time Training (LaCT), a new framework for test-time training (TTT) that significantly improves hardware efficiency and scalability for long-context modeling. Traditional TTT approaches rely on small mini-batch updates (e.g., every 16 or 64 tokens), leading to poor FLOPs utilization (<5%) and limited fast-weight state capacity. LaCT instead proposes using large chunks of data—ranging from 2K to over 1M tokens—for fast-weight updates, enabling 10–15× higher utilization (up to 70% on A100 GPUs), and scaling state size up to 40% of total model parameters.\n\nLaCT is evaluated on three diverse tasks: (1) novel view synthesis with up to 1 million token context from 128 input views; (2) long-context language modeling (up to 32K context length) with 760M and 3B parameter models; and (3) autoregressive video diffusion using a 14B parameter model. Across all domains, LaCT achieves comparable or superior performance to state-of-the-art baselines (e.g., DeltaNet, Mamba, 3D Gaussian Splatting), while maintaining high throughput and implementation simplicity using native PyTorch code.\n\nSTRENGTHS AND WEAKNESSES:\n### strength\n\n1.Hardware-efficient design through large chunk updates: \nThe key idea of large-chunk updates (e.g., 2K–1M tokens) allows significant FLOPs utilization improvements. For example, on novel view synthesis with 196K input tokens, LaCT achieves 1.4s prefill time and 38.7 FPS rendering, while Perceiver and full attention baselines take over 16s prefill and <35 FPS (Table 2).\n\n2.Effective across modalities with flexible architecture: \nLaCT is applied to image sets (e.g., Google Scanned Objects and DL3DV datasets), text (Long-Data-Collections), and video (autoregressive denoising in Wan 2.1). In language modeling, a 3B LaCT model with Muon optimizer achieves lower per-token loss and higher retrieval accuracy (S-NIAH-2) than GLA and DeltaNet baselines (Figure 5c–d).\n\n3.Nonlinear fast weights and advanced update rules: \nThe paper shows that using SwiGLU MLPs for fast weights, combined with the Muon update rule, yields strong empirical gains. For instance, in LM tasks, the Muon variant consistently outperforms Momentum and vanilla gradient descent (Figure 7b).\n\n4.Modular, scalable, and easily implemented: \nLaCT’s design—chunked memory updates + local window attention—enables native PyTorch implementation in “a few dozen lines of code” while supporting distributed context-parallelism. On large-scale video generation (14B params), it autoregressively generates sequences of up to 56K tokens with stable training and clean decoding (Figure 6c).\n\n### weakness\n1. The paper uses denoising loss at multiple timesteps as its primary metric for autoregressive video. However, no perceptual or user-based evaluation (e.g., FVD, CLIP-similarity, human rating) is provided, making it hard to assess real-world generative quality.\n2. Why the model learning has a big performance gap when using momentum or moun based optimizer for LM (figure 5)\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-07"}