TTOM: Test-Time Optimization and Memorization for Compositional Video Generation
Authors: Leigang Qu
Ziyang Wang
Na Zheng
Wenjie Wang
Liqiang Nie
Tat-Seng Chua
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-Video Generation
Test-Time Optimization
Memory
Primary area: generative models
TL;DR: We propose a latent memory–based test-time optimization framework that improves compositional text-to-video generation by dynamically adapting to unseen prompts during inference.

Abstract: Video Foundation Models (VFMs) exhibit remarkable visual generation performance, but struggle in compositional scenarios (\eg, motion, numeracy, and spatial relation). 
In this work, we introduce **Test-Time Optimization and Memorization (TTOM)**, a training-free framework that aligns VFM outputs with spatiotemporal layouts during inference for better text-image alignment.
Rather than direct intervention to latents or attention per-sample in existing work, we integrate and optimize new parameters guided by a general layout-attention objective. 
Furthermore, we formulate
video generation within a streaming setting, and maintain historical optimization contexts with a parametric memory mechanism that supports flexible operations, such as insert, read, update, and delete. 
Notably, we found that TTOM disentangles compositional world knowledge, showing powerful transferability and generalization. 
Experimental results on the T2V-CompBench and Vbench benchmarks establish TTOM as an effective, practical, scalable, and efficient framework to achieve cross-modal alignment for compositional video generation on the fly.

Published as a conference paper at ICLR 2026
TTOM: TEST-TIME OPTIMIZATION AND MEMORIZA- TION FOR COMPOSITIONAL VIDEO GENERATION
Leigang Qu
Ziyang Wang", Tat-Seng Chua
Na Zheng'
Wenjie WangSt
Liqiang Nie
National University Singapore University of Science Technology of China
HHarbin Institute of Technology (Shenzhen)
(leigangqu, wenjinwn,,,,, zhengnagrape, nieliqiang/ gma il. com wangziyang@u. edu
dcsctsênus.eduu
https:/ttom--2v.iihubbbio
ABSTRACT
Video Foundation Models (VFMs) exhibit remarkable visual generation perfor- mance, but struggle in compositional scenarios (e.g., motion, numeracy, and spatial relation). In this work, we introduce Test-Time Optimization and Memorization (TTOM), model-agnostic framework that aligns VFM outputs with spatiotem- poral layouts during inference for better text-image alignment. Rather than direct intervention to latents or attention per-sample in existing work, we integrate and op- timize new parameters guided by general layout-attention objective. Furthermore, we formulate video generation within a streaming setting, and maintain historical optimization contexts with parametric memory mechanism that supports flexible operations, such as insert, read, update, and delete. Notably, we found that TTOM disentangles compositional world knowledge, showing powerful transferability and generalization. Experimental results on the 12V-ComBBench and Vbench bench- marks establish TTOM effective, practical, scalable, and efficient framework to achieve cross-modal alignment for compositional video generation on the fly.
Prompt
Wan2. 1-14B (Baseline)
Ours
robot and sorcerer are stealthily approaching each other.
An elderly man walking to right in sunny park.
blue bird flies upwards sunny garden.
Four pandas munch bamboo in the garden.
Figure 1: Current video generative models (Wan et al., 2025) still suffer from text-video misalignment problems in compositional scenarios. We introduce test-time optimization and memorization method that substantially enhances alignment while maintaining high visual fidelity.
INTRODUCTION
Recent years have witnessed a rapid stride in video generation (Ho al., 2022; Polyak et al., 2024) and world simulation (Brooks et al., 2024). Benefiting from the significant advance in flow matching (Lipman et al., 2022; Esser et al., 2024) and diffusion transformers (DiTs) (Peebles Xie,
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2026
2023), current text-to- video generation (T2V) models (Yang al., 2024b; Kong et al., 2024) are able to synthesize realistic and vivid videos. However, even state-of-the-an models still suffer from text-video misalignment in compositional scenarios, i.e., composing multiple objects, attributes, and relations into a complex scene (Sun et al., 2025; Huang et al., 2024).
Remarkable efforts have been made to improve text-video alignment by introducing explicit spa- tiotemporal layout guidance (Lian et al., 2023b; He et al., 2025) as a bridge between text prompts and videos. In this paradigm, layout represented by bounding box (bbox) sequences-each corre- sponding to an object-is first generated by a large language model (LLM), and subsequently used to optimize intermediate representations (e.g., latents and attention maps) by enforcing alignment with the layout constraints. Despite notable progress, this line of work faces several limitations: 1) direct intervention on intermediate representations may disrupt feature distributions, leading to degraded video quality (e.g., inconsistency, flickering artifacts, or even collapse); 2) the per-sample control paradigm treats each case independently, thereby neglecting the historical context of previously generated videos; and 3) these methods do not enhance the intrinsic capability of generative models, since interventions on one sample fail to generalize to others.
In real-world scenarios, models are not presented with set of independent test cases but with continuous stream of user prompts, as illustrated in Fig. 2. Previous successful generations can serve as valuable references for future cases. In light of this, core idea is to formulate compositional video generation in a streaming setting, leveraging history test-time optimization as context for future inference. However, modeling history contexts is non-trivial, it requires addressing the fundamental challenge of how to represent, store, reuse, and update past information.
In this work, we introduce Test-Time Optimization and Memorization (TTOM) for compositional T2V. For the first test case (cold start), a large language model (LLM) first derives a spatiotemporal layout from the text prompt, serving as a controllable condition. Conditioned on this layout, we perform Test-Time Optimization (TTO) by instantiating and updating sample-specific parameters, thereby steering video generation toward adherence the prescribed layout. Through this process, the compositional patterns embedded in the layout, such as motion, numeracy, and multi-object interactions, are attained and saved into the new parameters. After optimization, the parameters are unloaded and stored in memory, using the extracted layout-related keywords from the prompt keys. For subsequent test cases, when matched parameters are retrieved, they can be integrated into the foundation model and further optimized for improved adaptation, directly applied for efficient inference. If no match is found, new parameters are initialized, optimized, and subsequently recorded in memory expand the memory. In practice, the memory is assigned fixed capacity; when full, items are removed according to predefined policies (e.g., least frequently used). This mechanism enables the flexible reuse of informative optimization results by preserving historical context, thereby enhancing both efficiency and scalability. In summary, our main contributions are as follows.
We propose a test-time optimization framework without any supervision for compositional T2V. With spatiotemporal layout as guidance, incorporate and optimize lightweight parameters for each data sample.
We present a parametric memory mechanism to maintain historical optimization context, which naturally supports lifelong learning. A series of operations (e.g.. insert, read, update, and delete) for memory ensures flexibility, scalability, and efficiency.
Extensive experimental on two T2V benchmarks (Sun et al., 2025: Huang et al., 2024) demon- strate the effectiveness and superiority of the proposed method. Notably, it achieves relative improvements of 34% and 14% over Cog CogVideoX-5B (Yang et al., 2024b) and 1-12 (Wan al., 2025) on 12V-CompBench (Sun et al., 2025), respectively.
RELATED WORK
Compositional Visual Generation. Despite the thrilling progress in foundation models (Yang et al., 2024b; Kong et al., 2024; Polyak et al., 2024; Seawead et al., 2025), compositional generation (Chefer et al., 2023; Feng et al., 2022; Qu et al., 2024b; 2025b) remains an open challenge and has attracted substantial research interest. To enhance text-image alignment in compositional scenes, a rich line of studies (Qu et al., 2023; Feng et al., 2023) explores incentivizing the layout planning ability of I.LMs (Brown al., 2020; Ouyang et al., 2022) enable layout-guided controllable generation (Tian
2

---

Published as a conference paper at ICLR 2026
LLM Layout Planning
Test-Time Optimization
denolsing
danaising
boy and girl runting
Text
Video?
towards each other.
Video
DIT
Prompt
query
unload
Stream
Parametric Memory
iiprrt
You are an intiligent spatitemporal ayout
planner for videcs Generate bounding
Abstraction of C (CC)
Layouu Relevancy Probe
booes (Rioos) for visual otyects.
Book rules: promos following.
Output formatt Frame i: 2'd" kints, "name" <odject namer, boxt witth, heightf) On context examp les)
consictens noy
Cross-Aft.
Alt.
Two soljects lowards other.
Layout Goneration
Consistency Check
1
Y,
M (= ICEC)
7, (o(Co
bey
A B)
Gradiont
girl
boy
Spabotemporal Layout
Attention-to-Layout Alignment
Cross-Attention Maps
Figure 2: Overview of the TTOM framework for compositional text-to-video generation. A stream of text prompts is first fed into LLMs for spatial-tempozae layout planning. Meanwhile, a denoising sampling process of video foundation models is performed, in which cross-attention maps extracted, followed by test-time optimization for alignment. Historical optimization context is maintained by the parametric memory.
et al., 2024; Lin et al., 2023; Feng et al., 2025) through attention control (Chen et al., 2024; Xie et al., 2023; He al., 2023; Wang al., 2024; He et al., 2025) latent modification (Qi et al., 2024; Wu et al., 2024; Lian et al., 2023a; Yang et al., 2024a; Wang et al., 2025c; Lian et al., 2023b), typically in training-free manner. However, direct intervention on attention maps latents may compromise visual quality. Moreover, the per-sample paradigm treats test samples in isolation, disregarding informative historical context.
Test- Time Optimization. TTO adapts a trained model to test instance(s) by performing optimization during inference, which has been widely applied in domain adaptation (Li et al., 2016; Sun al., 2020; Shu et al., 2022) and alignment (Kim et al., 2025; Li et al., 2025). Existing approaches can be categorized into three groups: 1) optimizing intermediate representations (Kim al., 2025; et al., 2025), 2) prompt tuning (Shu al., 2022), and 3) optimizing model parameters (Schneider et al., 2020; Liang al., 2020; Shu et al., 2021; Zhang et al., 2022; Gandelsman et al., 2022). In visual generation, recent work (Lian et al., 2023b; Wu et al., 2024) optimizes latents guided by layouts, yielding significant gains. However, these methods perform optimization on a per-sample basis and subsequently discard the results, thereby neglecting historical context.
Memory Mechanism. To track and model historical contexts, existing Test-Time Training (TTT) (Sun et al., 2024) work designs memory mechanisms instantiated with inserted layers, i.e., TTT layers. The memory can be updated linearly (Sun et al., 2020) or a local temporal window (Wang al., 2025b), by optimizing TTT layers via self-supervised learning loss (He et al., 2022). Despite the remarkable advance, they often consider intra-sample context (e.g., frames in video) (Wang et al., 2025b; Dalal et al., 2025) and assume the independence among samples. Besides, the memory has fixed size and does not support flexible operations.
3 METHODOLOGY
In this section, introduce the proposed TTOM framework for compositional T2V, shown Fig. 2. Given text prompt describing a compositional scene (e.g., including multiple objects, attributes, numeracy, and relationships), first conduct spatial-temporal layout (STL.) planning
3

---

Published as a conference paper at ICLR 2026
driven by LLMs, and obtain the bbox sequence for each object (Sec. 3.1). Subsequently, perform test-time optimization to achieve attention-to-layorr alignment, making the generated video follow the explicit layout guidance (Sec. 3.2). To take advantage of history context, we record previous optimization results with a parametric memory structure, and keep updating and reusing it as the sequential inference goes, Le., a ceaseless stream of user prompts is fed into the model (Sec. 3.3).
3.1 LLM-DRIVEN SPATTAL-TEMPORAL LAYOUT PLANNING
Layout Representation. We represent a video-level STL with a collection of object-level layouts (Yo..., Yx) where N denotes the number of objects. Each layout Y; is a tuple (cs, B,), consisting of an object phrase Gj and its associated hounding box sequence B; [b....... ber The object phrase e, is extracted from the user prompt C (e.g. the underlined text in "A vibrant red balloon drifts right to left above a grand statue."). ") Each b is 4-dimensional coordinate vector, while 5; and e, denote the start and end frame indices of the object's 's temporal occurrence, respectively.
Text-to-Layoue Generation. Prior work has shown that LLMs possess strong
dynamics understanding abilities (Qu et al., 2021; 2023; Feng et al., 2023; Qu et al., 2024a), and capture physical properties (Lian et al., 2023b), such gravity, elasticity, and perspective projection, which align closely with real-world physics. Inspired by it, we prompt LLMs to generate SPLs given user prompts via in-context learning (Dong et al., 2022; Qu et al., 2025a). To enhance scene comprehension, the LLM is first instructed to produce descriptions of object motions and camera behaviors, followed by the corresponding layout generation. Finally, a verification step ensures spatial and temporal consistency of object phrases and bounding box sequences, with corrections applied when discrepancies are detected.
3.2 TEST-TIME OPTIMIZATION FOR LAYOUT-TO- VIDEO GENERATION
In this section, we first discuss the relation between intermediate attention maps of diffusion models and generated videos, and conduct a probe experiment to verify the relevance. On the basis of the relevance, we propose model-agnostic controllable generation strategy by optimizing attention-to- layout alignment during inference, i.e., Test- Time Optimization (TTO).
Attention-Layout Relevance Probe in Diffusion Trans- formers. Prior work (Hertz et al., 2022) has demonstrated the relevance between cross-attention maps and spatial lay- outs, which is exploited for video editing (Qi et al., 2023) and controllable generation (Xie al., 2023). However, with the architecture shift from UNets (Ronneberger et al., 2015) to Diffusion Transformers (DiTs) (Peebles Xie, 2023) due its stronger scalability, the recent study (Wang et al., 2025c) pointed out their difference, i.e., the cross-attention maps in DiTs distribute more evenly than those in UNets, which may cause the ineffectiveness previous methods due to the weak attention-layout relevance.
0.10
Wan2.1-14B CogVideoX-58 5B
0.08
mool
0.06-
0.04
0.02
10
Layer
Motivated by it, we propose a probe strategy to assess attention- -layoi relevance across different layers: 1) prepar- ing a set of prompts describing a common scene in which an object performs motion. 2) generating videos given the prompts, and meanwhile, extracting text-video cross-attention maps. Specifically, we first extract the cross-attention map' A, E IR xx from each layer for the object phrase c; from a text prompt C, where Ic; denotes the token number of G. T, ht and refer the frame number, height, and width of the video latent, respectively. Afterwards, we perform average pooling along the text dimension and obtain A e Txxxxx This process is formulated as:
Figure 3: Attention-layout overlap (evaluated by mloU (Everingham et al., 2010) over 200 prompts) be- tween cross-modal attention maps ex- tracted from each layer of founda- tion models and segmentation maps detected from generated videos by GroundingDINO (Liu et al., 2024) + SAM (Ravi et al., 2024).
A = fe (C, C, 24, t),
(I)
Forr foundation models (Yang al., 2024b) without cross-attention layers, we extract those parts in self- attention maps where text-vision interaction happens.
4

---

Published as a conference paper at ICLR 2026
where 2 and represent the latent and timestep, respectively. Q denotes the parameters of all layers before the attention map in a DiT. 3) applying GroundingDINO (Liu et al., 2024) to detect the related bboxes and SAM 2 (Ravi et al., 2024) for segmentation. And 4) computing the overlap between segmentation maps and attention maps A; in different layers to quantify relevance. The result shown in Fig. indicates remarkable variance of attention-layout relevance among different layers.
Attention-to-Layour Alignment for Controllable Generation. As discussed above, attention maps in certain layers of DiTs reflect strong relevance the layout of the final generated videos. Motivated by this observation, we propose a test-time optimization approach to align the high- relevance attention maps with the layout generated by LLMs for controllable generation. Specifically, we first smooth each spatial layout b, in B, with a Gaussian kernel and transform it into soft mask b;, and then calculate the Sonsen-Shannee divergence (JSD) between attention maps and soft masks of N objects as the loss function:
Lalign
(A,
We insert new parameters & into VFMs and get θ 0 U Φ calculate the gradient of the loss function w.r.t.hee newly introduced parameters, i.e., SLalian and update them with optimizer and get ø' 86 finally. Compared with prior work (Lian al., 2023b) that optimizes the latent 2 denoising timestep by 3z. we have the following strengths: 1) optimizing ø for alignment avoids possible distribution collapse caused by direct intervention to the latents. And 2) the updated şi memorizes layout patterns for specific compositional scenes, holding a potential for future re-usage. For example, if a similar text prompt comes in the future, a simple and efficient strategy to directly load ø the VFM, and perform generation without any optimization, which motivates to propose memory mechanism as follows.
Hhation
3.3 PARAMETRIC MEMORY
To maintain historical contexts of test-time optimization and support future reuse, we present the parametric memory mechanism.
Memory Structure. We define memory structure as collection of key-value pairs:
M {g(C)) &c e CH):
(3)
where Cи is set of history prompts. The function g(-) consists of scene abstraction, text embedding.
and indexing precedes. Specifically, for prompt "A vibrant red balloon drifts right to left above grand statue.", we first abstract it into "<object A> drists right to left above <object B> .", and then extract text feature (Radford et al., 2021) and index
Memory Operations. Based on the memory structure, we define the following basic operations to perform interaction between the video foundation model and the memory.
Insert. For new text prompt Cj. we first retrieve from the memory. If no matched item, we perform test-time optimization and insert a new item (g(C;), oz,) to M.
Read. If the matched item(s) exist, we read and load the corresponding parameters into the foundation model. Next, we have two options: 1) directly generate video without optimization, and 2) continue optimization with the loaded parameters as initialization.
Update. After the loaded parameters are further optimized, we unload them and update the corresponding matched items in the memory.
Delete. If the total number of items in M exceeds its maximum capacity, we delete those items that are least frequently used.
Compared with prior methods (Lian et al., 2023b; He et al., 2025) treating each sample independently, the advantages of the proposed memory mechanism lie in: 1) Superiority. The history optimization maintained in the memory provides abundant seene knowledge, serving practice, we can maintain a user-specific memory for each user, track their historical requests, and model the intention for better personalized
Discussion.
as good initialization for TTO. 2) Personalization.
5

---

Published as a conference paper at ICLR 2026
Table Evaluation results of compositional text-to-video generation on T2V-CompBench (Sun et al., 2025), reported over 7 categories and the overall average (Avg.).
Model
Avg.
Motion
Num
Spatial
Con-atir
Dyn-alir
Action
Interact
Commerciaa
Pika-1.0
0.3752
0.2234
0.3870
0.4650
0.5536
0.0128
0.4250
0.5198
Gen-3
0.4094
0.2754
0,2306
0.5194
0.5980
0.0687
0.5233
0.5906
Dreamina 1.2
0.4689
0.2361
0.4380
0.5773
0.6913
0.0051
0.5924
0.6824
Kling-1.0
0.4630
0.2562
0.4413
0.5690
0.6931
0.0098
0.5787
0.7128
Diffusion Unet based
ModelScope
0.3468
0.2408
0.1986
0.4118
0.5148
0.0161
0.3639
0.4613
LVD
0.3912
0.2457
0.2008
0.5405
0.5439
0.0171
0.3802
0.4502
Show-1
0.3676
0.2291
0.3086
0.4544
0,5670
0.0115
0.3881
0.6244
VideoTeetris
0.4097
0.2249
0.3467
0.4832
0.6211
0.0104
0.4839
0.6578
T2V-Turbo-V2
0.4317
0.2556
0.3261
0.5025
0.6723
0.0127
0.6087
0.6439
DiT based
Open-Sora 1.2
0.3851
0.2468
0.3719
0.5063
0.5639
0.0189
0,4839
0.5039
Open-Sora-Peee Plan v1.3.0
0.3670
0.2377
0,2952
0.5162
0.6076
0.0119
0.4524
0.4483
CogVideoX
0.4189
0.2658
0.3706
0.5172
0.6164
0.0219
0.5333
0.6069
DyST-XL
0.5081
0,2712
0.3969
0.6110
0.8696
0.0221
0.7321
0.6536
+ -VDD
0.4739
0.3291
0.3825
0.5274
0.7534
0.0219
0.6826
0.6204
Ours
0.5632
0.4351
0,5081
0.6173
0.8782
0,0341
0.7191
0.7502
%IIpprove.
Wan2.1-14B
0.5314
0.2696
0.5113
0.5709
0.8369
0.0570
0.7504
0.7239
+VV
0.5439
0.2864
0.4707
0,5753
0.8610
0.0829
0.8107
0.7201
Ours
0.6155
0.4922
0.5881
0.6275
0.8982
0.1182
0.8152
0.7691
%Improve.
+15.83
+82.57
+15.02
+9.91
+7.32
+107337
+8.64
+6.24
generation (Wang et al., 2025a; Zhao et al., 2025; 2026). 3) Efficiency. For those prompts similar historical ones, we can skip optimization and directly read and load items into the foundation model for inference. 4) Scalability. We can scale the memory up in two dimensions: increasing the item-level capacity (i.e,, enlarging the history context window) could memorize more historical optimization information; and increasing the number of per-item parameters o may capture more scene patterns each TTO.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Foundation Models. We integrate method into two representative T2V models: Cog 5B (Yang et al., 2024b) and Wan2.1-14B (Wan et al., 2025). For CogVideoX-5B, each clip contains 49 frames at 8 FPS, whereas for Wan2.1-14B, each clip contains 81 frames at 16 FPS. In the Spatial-Temporal Layout Planning stage, we leverage the OpenAl GPT-4o model.
Streaming Video Generation Setting. To align with practical video creation scenarios where prompts arrive sequentially, we propose modeling historical streaming contexts. For a fair comparison with prior methods that treat test samples independently, we introduce a test-time independence setting. Specifically, we generate 200 prompts covering common compositional scenes as pseuto-training data, via GPT-4o, and subsequently generate videos from these prompts while applying TTOM to construct memory. During inference, entries can be retrieved and loaded from the memory; however, inserting or updating the memory with optimization results from test samples is prohibited.
Implementation Details. We employ the lightweight LoRA (Low-Rank Adaptation) (Hu et al., 2022) method to introduce new parameters enable test-time optimization for controllable generation. The optimized LoRA parameters are inserted into or used to update the parametric memory. Specifically, LoRA weights are injected into the Query, Key, Value, and Output projection layers of each cross- attention block in DiT architectures, with the LoRA rank set to 32. This optimization is applied only to the first five denoising steps of the diffusion process, using the AdamW optimizer with learning rate of 1e-4. Unless otherwise noted, all other hyperparameters follow the default settings of the respective backbone models. For fair comparison, we also re- immmlemmnnt the LVD method (Lian et al., 2023b) the same Dis-b architecture. The layouts and selected layers used for LVD exactly

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results on the semantic categories of VBench.
Model
Obj. Class
Multi- Obj.
Human Act.
Color
Spatial Rel.
Scene
Appear. Style
Temp. Style
Overall
Cons.
CogVideo--BB
0.8342
0.6728
0.9760
0.8840
0.7943
0.5328
0.2468
0.2542
0.2742
LVD
0.8661
0.6782
0.9740
0.8802
0.8014
0.5406
0.2472
0.2509
0.2609
Ouus
0.9486
0.7952
0.9820
0.9246
0,8215
0.5682
0.2550
0.2576
0.2678
Wan2.1-
0.8628
0.6958
0,9540
0.8859
0.7539
0.4575
0,2264
0.2319
0,2591
+ -VVD
0,9083
0.7286
0,9620
0.8832
0.7547
0.5135
0.2280
0.2407
0,2507
+ Ours
0.9921
0.8216
0,9800
0.9289
0.8074
0.5286
0.2305
0.2444
0.2542
match those used for our TTO version, while all other training hyperparameters follow the original LVD paper.
Benchmarks and Evaluation Metrics. We evaluate our approach on two large-scale public bench- marks, T2V-Compencch (Sun et al., 2025) and VBench (Huang et al., 2024). T2V C-ComBhnnch targets compositional T2V with about 1,400 prompts across seven categories, including consistent and dynamic attribute binding, spatial and motion relations, action binding, object interactions, and generative numeracy. Its evaluation combines MLLM-based scoring, detection-buseed measures, and tracking-basee based indicators to provide fine-graines assessment of compositional abilities. VBench offers broad evaluation suite, decomposing video generation quality into 16 dimensions, including subject and background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic and imaging quality, as well as object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style, and overall consistency. Each dimension contains roughly 100 text prompts with automatic evaluation scripts and human preference annotations. We follow the official toolkits of both benchmarks and report corresponding scores.
Baselines. We conduct a comprehensive comparison of our approach on T2V-CompBench against a diverse set of baselines across three categories: commercial solutions (Pika-1.0 (Pika, 2024), Gen-3 (Runway, 2024), Dreamina-1.2 (CapCut, 2024), Kling-1.0 ,. (Kuaishou, 2024)); diffusion U- Net-based methods (ModelScope (Wang et al., 2023), Show-1 (Zhang et al., 2024), VideoTeetris (Tian et al., 2024), T2V-Turbo-V2 (Li et al., 2024), LVD (Lian et al., 2023a)); and DiT-based models (Open- Sora-1.2 (HPC-AI Tech, 2024), Open-Sora-Plaan Plan-llllllliii 1.3.0 (Lin et al., 2024), Cog Vog (Yang et al., (Wan et al., 2025) with LVD), Except
2024b) with DyST-XL (He et al., 2025) and LVD, Wan2. for X with LVD, Wan2.1-14B and Wan2.1-14B with LVD. for which reproduce the results in our experiments, the scores of all remaining baselines are taken directly from the official T2V-CompBench paper (Sun et al., 2025).
4.2 PERFORMANCE COMPARISON
Evaluation on Compositionality. Table presents the compositional evaluation results of our method compared to wide range of baselines on 12V-CompBench. Across all metrics, our approach consistently surpasses the baseline models, demonstrating effectiveness and superiority. Notably, our method improves Cog VideoX-5B by 34.45 % and 1an2.1-14B by 15.83 % in terms of overall average performance. The most pronounced gains are observed in Motion and Numeracy, with relative improvements of up to 63.69% and 37.10% on Cog VideoX-5B, and 82.57 % and 15.02% on Wan2.1-143 respectively. These categories are known to be particularly challenging due to the need for precise modeling of dynamic object trajectories, object counts, and spatial-temporal coordination.
Evaluation on Semantic Consistency. Beyond compositionality. we further evaluate semantic consistency on VBench (Table 2). Here, our method also delivers clear improvements across multiple dimensions, including object classification accuracy, multi-object handling, and color/spatial relation fidelity. These aspects are crucial for ensuring that generated videos do not only satisfy individual attributes but also maintain coherent semantic integrity across frames. The gains in semantic consistency complement the compositionality improvements observed on T2V-CompBench, providing strong evidence that method reinforces both the local correctness of attributes (e.g.. colors, actions, object categories) and the global contextual structure (e.g., multi -ojeet interactions and spatial layouts).
7

---

Published as a conference paper at ICLR 2026
4.3 IN-DEPTH ANALYSIS
To explore the efficacy of TTOM, we conduct extensive ablation studies and hyperparameter analyses. We first investigate the TTO and Memorization, followed by an in-depth study of key components, including continual TTO, loss function for attention-to ditention-to-layou to-layout alignment, guidance timesteps, and optimization iteration.
Test-Time Optimization and Memorization. We conduct comparative experiments to evaluate the effectiveness of TTO and the memory mechanism, reported in Tab. 3. The results show that TTO substantially improves motion quality in generated videos by 60.27% %, guided by LLM- planned spatiotemporal layouts. Moreover, incorporating historical context from memory yields an additional 13.91% % improvement.
Table 3: Ablation study for TTO and
Memory on the motion category of 12V-CompBench (Sun et al., 2025).
TTO
Memory
Motion
X
0.2696
X
0.4321
0.4922
Continual Test-Time Optimization. The parametric mem- ory enables the model to exploit historical context for future inference. In practice, however, memory capacity limited and cannot fully encompass world knowledge (e.g., object motion patterns). Consequently, imperfect matching may arise between the current prompt and the retrieved items, leading to suboptimal parameter initialization. Therefore, continual optimization can help balance historical context and the current sample. To examine its effect, we conduct experiments Tab. 4.
The results indicate that: 1) The memory mechanism is ef- fective, as evidenced by substantial gains even without per- sample optimization. 2) With memory- based initialization, continual TTO further improves motion performance at the cost of additional test-time computation. And 3) increas- ing the number of memory entries as initialization provides richer context but may also introduce noise or irrelevant in- formation, suggesting that more sophisticated fusion strate- gies are required for better leveraging context the future. In summary. the proposed TTOM offers flexible trade-off between video quality and efficiency, making well-suiter for complex practical scenarios.
Table 4: Ablation study for continual test-time optimization on the motion cat- egory of 12V-CompBench (Sun al., 2025). Motion score and latency are used to evaluate motion quality and effi- ciency, respectively. Init.: initialization from memory with the average fusion of Top-k matched entries. TTO: whether to perform test-time optimization.
Init.
TTO
Top-k
Motiont Latency (s)
Loss Function for Attention-to-LLyquu Alignment. To drive controllable generation guided by spatiotemporal layout, we propose a JSD loss to perform attention-to- layout alignment optimization during inference. To vali- date its effectiveness, we conduct experiments to compare it with two other variants: BCE loss (Sella et al., 2025) and Center-of-Mass (CoM) loss (Lian et al., 2023b). Results in Tab. 5 show that the proposed ISD loss achieves the best performance motion and numeracy, demonstrating its effectiveness and universality.
x
0,2696
425
x
0.4754
427
d
5
0.4846
627
x
10
0.4437
427
10
0,4705
627
Pseudo-training Data Scale. To simulate the stream- ing generation scenario and make a strictly fair com- parison with prior methods, we first generate user prompts via GPT-4o to initialize the memory. Results Tab. 6 show the influence of different numbers of pseudo-training data on the performance. We can see that more data induces better performance, due to the more abundant compositional patterns captured and saved into the memory as context.
Table 5: Comparison among different loss functions for attention-to- layyouu alignment, evaluated on the motion and numeracy cate- gories of T2V-CompBench.
Loss Function
Motion
Numeracy
CE Loss
0.2912
0.5218
CoM Loss
0.3626
0.4697
Guidance Timesteps and Optimization Iteration. Existing work (Choi et al., 2022; Pan et al., 2024) has shown that early denoising steps in diffusion models primarily define structure, while later steps progressively refine details. Motivated by this observation, apply TTO only in the initial denoising steps and compare performance across different numbers
JSD Loss
0.4321
0.5881
8

---

Published as a conference paper at ICLR 2026
Table Impact of pseudo- training data scale, evaluated on the motion category of T2V- CompBench.
Table 7: Impact of number of timesteps for optimization dur- ing the early stage of denoising sampling.
Table 8: Impact the number of test-time optimization iter- ations per denoising sampling timestep.
#Pseudo iPssudo-training Data
Motion
#Timestep
Motion
IIteration
Motion
0.3519
0.2730
0.3789
100
0.4344
0.3692
0.4321
150
0.4751
0.4321
12
0,4130
200
0.4922
0.4296
16
0.3547
Prompt
Wan2.1 (Baseline)
TTOM (Ours)
Memory
cat slinking the left side of cozy living room.
fresh oronge rolls left across the countertop.
gentle swan glides left the lake.
Figure 4: Qualitative results of motion pattern transfer with memory. Solid arrows indicate insert update operations, while dotted arrows represent reading and loading parameters from memory into foundation models for inference.
of steps, as presented in Tab. 7. Within each step, we further examine the impact of varying the number of optimization iterations, reported in Tab. The results reveal clear saturation points for both timesteps and optimization iterations, beyond which performance degrades. This degradation may arise because attention maps encode not only layout but also entangled information, and excessive alignment optimization may disrupt other essential signals.
Qualitative Results. Fig. shows the qualitative comparison between Wan2.1-14B and our method based on which. To further delve into the impact of the proposed memory mechanism, show the qualitative results of motion pattern transfer from memory to foundation models in Fig. More qualitative results can be found in the Appendix.
CONCLUSION
In this work, tackle the compositional limitations of Video Foundation Models and introduce TTOM, a training-free framework for aligning video generation with spatiotemporal layouts at inference. By integrating layout-guided optimization with parametric memory mechanism, TTOM enables consistent streaming generation and supports flexible operations on memory, such as insertion, update, and retrieval of historical contexts. Our experimental analysis shows that TTOM not only improves alignment in compositional scenarios, but also disentangles compositional world knowledge, leading to strong transferability and generalization. Extensive experiments on T2V- Compechhe and VBench confirm its effectiveness, scalability, and practicality, establishing TTOM as a versatile solution for enhancing compositional video generation on the fly.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work compositional video generation holds significant promise for democratizing creative tools, enhancing accessibility, streamlining media production, and advancing intuitive human-Al collaboration. At the same time, it raises critical concerns, including potential misuse for misinforma- tion or manipulation, biases in generated content, job displacement in creative industries, and the environmental costs of intensive computation. Addressing these challenges requires careful dataset curation, robust privacy safeguards, systematic bias mitigation, responsible deployment strategies, and sustained engagement with diverse stakcholders.
REPRODUCIBILITY STATEMENT
have taken several steps to ensure the reproducibility our work. link (https://tton t2v. github. ic/ /) to the source code is provided, enabling replication of our implementation. The main text and appendix together provide comprehensive descriptions the model design, optimization procedure, and evaluation protocol. These resources collectively ensure that readers reproduce and validate our experimental results.
REFERENCES
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL /tttt::://pppaaiiii com/ research/ video-genersn geweratiennnnnnn -oneeraatooommmmlssssss models-as as-world simulators.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
CapCut. Dreamina. https: /ttp:s//ddraamaaaa.mmm wwwpct.ppc..oooooc cww/coo/aaaacoooooo ai http:/aittooo// tttllooooooeeee 2024. Accessed: 2024.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4):1-10, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross- cooss-attention guidance. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5343-5353, 2024.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffuusinn models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11472-11481, 2022.
Karan Dalal, Daniel Koceja, Jiarui Xu, Yue Zhao, Shihao Han, Ka Chun Cheung, Jan Kautz, Yejin Choi, Yu Sun, and Xiaolong Wang. One-minute video generation with test-time training. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17702-17711, 2025.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning, arXiv preprint arXiv:2301.0000 000344 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-firs international conference on machine learning, 2024.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International jornnaatioaaouuraa of computer vision, 88(2): 303-338, 2010.
10

---

Published as a conference paper at ICLR 2026
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05503 2022.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances Neural Information Processing Systems, 16:18225-18250 2023.
Weixi Feng, Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, and Weili Nie. Blobgen-vit -iid: Compositional text-to-video generation with blob video representations, Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12989-12998, 2025.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoen- coders. Advances in Neural Information Processing Systems, 02229744-23889 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Weijie He, Mushui Liu, Yunlong Yu, Zhao Wang, and Chao Wu. Dyst-xl: Dynamic layout planning and content control for compositional text-to-video generation. arXiv preprint arXiv:2504. 15032, 2025.
Yutong He, Ruslan Salakhutdinov, and J Zico Kolter. Localized text-to- text-to-image generation for free via cross attention control. arXiv preprint arXiv:2306. 14636, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint TrXiv:2208.01620 2022.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633-8644 2022.
HPC-AI Tech. Open-sora: Democratizing efficient video production for all. https: com/ homm//pcaitech/Opeeeee -ooppennn-rnaan.e---eaaeaaaaaaar 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICIR, 1(2):3, 2022.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818. 2024.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment diffusion models without reward over-optimization, arXiv preprint arXiv:2501.05800 2025.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603,, 2024.
Kuaishou. Kling 1.0. httph:tt//// //kingg kuaishou. com/ 2024. Accessed: 2024.
Jiachen Li, Qian Long, Jian Zheng. Xiaofeng Gao. Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v:: Enhancing video generation model post-training through data, reward, and conditional guidance design. 2024. URL pettts://areee 05677.
/240. httt::::/axxiv..rr//bbs//// wttp:/wwwwww/w////////a////
Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li. and Yu Cheng. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. arXiv preprint arXiv:2501.12895, 2025.
Yanghao Li. Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation, arXiv preprint arXiv:1603.04777 2016.
11

---

Published as a conference paper at ICLR 2026
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023a.
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Lim Lim-grounden video diffusion models. arXiv preprint 2rXiv:2309.1744/ 2023b.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pp. 6028-6039. PMLR, 2020.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. 2024. URL https::////.......... /awxiv..vvv..
wwrrii.oorre www.worg/aaaa. abs// 2412 00131.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309./5091, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. Attentive moment retrieval in videos. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 15-24, 2018.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren. Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching, arXiv preprint arXiv:2402. 14167, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 4195-4205, 2023.
Pika. Pika 1.0. https:/ www.:pik...ar...... 2024. Accessed: 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.137200 2024.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15932-15942, 2023.
Zipeng Qi, Guoxi Huang, Chenyang Liu, and Fei Ye. Layered rendering diffusion model for controllable zero-shot image synthesis. In European Conference on Computer Vision, pp. 426-443. Springer, 2024.
Leigang Qu Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1104-1113, 2021.
Leigang Qu, Shenggiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutllm-t2. Eliciting lay- out guidance from IIm for text-to-image generation. In Proceedings of the 31 ACM International Conference Multimedia, pp. 643-654, 2023.
12

---

Published as a conference paper at ICLR 2026
Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, and Tat-Seng Chua. Tiger: Unifying text-to-image generation and retrieval with large multimodal models. arXiv preprint arXiv:2406.0588 05814, 2024a.
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng Chua. Dis- criminative probing and tuning for text-to- text-to-image generation. arXiv preprint arXiv:2403.03321, 2024b.
Leigang Qu, Feng Cheng, Ziyan Yang, Qi Zhao, Shanchuan Lin, Yichun Shi, Yicong Li, Wenjie Wang, Tat-Seng Chua, and Lu Jiang. Vincie: Unlocking in-context image editing from video. In The Fourteenth International Conference on Learning Representations, 2025a.
Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, and Tat-Seng Chua. Silmm: Self-improving large multimodal models for compositional text-to-image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18497-18508, 2025b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 007 14, 2024.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Runway. Introducing gen-3 alpha: new frontier for video generation. h/tpss///ruwayml.... 3-apgeennn-aaaaaaaaaaaaaaa 2024. Accessed: 2024.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33: 1539-11111 2020.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, al. Seaweed-Th 7b: Cost-effective training of video generation foundation model. arXiv preprint 250.::2500000888.. 2025.
Etai Sella, Yanir Kleiman, and Hadar Averbuch-Elor. Instancegen: Image generation with instance- level instructions. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1-10, 2025.
Manli Shu, Zuxuan Wu, Micah Goldblum, and Tom Goldstein. Encoding robustness to image style via adversarial feature perturbations. Advances in Neural Information Processing Systems, 34: -8042, 2021.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. Advances in Neural Information Processing Systems, 35:14274- 35:14974-14228, 2022.
Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehensive benchmark for compositional text-to-video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8406 8416, 2025.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9249 PMLR, 2020.
Yu Sun, Xinhao Li. Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, et al. Learning to (lear at test time): Rnns with expressive hidden states. arXiv preprint wxXiv:247..04620 2024.
13

---

Published as a conference paper at ICLR 2026
Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di ZHANG, et al. Videotetris: Towards compositional text-to- video generation. Advances in Neural Information Processing Systems, 37:29489- -9513,, 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXhv:2503.20314 2025.
Chengbing Wang. Yang Zhang, Wenjie Wang, Xiaoyan Zhao, Fuli Feng, Xiangnan He, and Tat-Seng Chua. Thin--while-generatingg On-the-fly reasoning for personalized long-form generation. arXiv preprint arXiv:25/2.0666 2025a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- clscope text-to-video technical report. 2023. URL https: /atp///aavvv.v/.. orgg/ba//20000000 /2308. 06571.
Renhao Wang, Yu Sun, Arnuv Tandon, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-time training on video streams. Journal of Machine Learning Research, 26(9):1-29, 2025b.
Shulei Wang, Wang Lin, Hai Huang, Hanting Wang. Sihang Cai, WenKang Han, Tao Jin, Jingyuan Chen, Jiacheng Sun, Jieming Zhu, et al. Towards transformer-base aligned generation with self- coherence guidance. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18455-18464, 2025c.
Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang. Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. arXiv preprint arXiv:2401. 2024.
Haokun Wen, Xuemeng Song. Jianhua Yin, Jianlong Wu, Weili Guan, and Liqiang Nie. Self-training boosted multi-facte matching network for composed image retrieval. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(5):3665- 3678, 2024.
Tsung-Har Han Wu, Long Lian, Joseph Gonzalez, Boyi Li, and Trevor Darrel.. Self-correcting Ilm- controlled diffusion models. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6327-6336, 2024.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
Ling Yang. Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. In Forty-first International Conference Machine Learning, 2024a.
Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.0661 06072, 2024b.
David Junhao Zhang, Jay Zhangjie Wu, Nea Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision (IJCV), pp. 1-15, 2024.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642, 2022.
Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. Nextquill: Causal preference modeling for enhancing I1m personalization. arXiv preprint arXiv:2506.02366 2025.
Ziyi Zhao, Chongming Gao, Yang Zhang, Haoyan Liu, Weinan Gan, Huifeng Guo, Yong Liu, and Fuli Feng. Don't start over: A cost-effective framework for migrating personalized prompts between 1ms. arXiv preprint arXiv:2601. 12034, 2026.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We declare that we used large language models (LLMs) to assist in refining this manuscript, specifi- cally for grammar checking, language polishing, and enhancing textual clarity and fluency. We also employed LLMs in a limited capacity for minor debugging and syntactic corrections of code snippets.
B ADDITIONAL EXPERIMENTAL RESULTS
B. OVERALL AND QUALITY EVALUATION ON VBENCII
Tables and 10 report the overall and quality dimension results on VBench. Our method achieves consistently higher scores than both the original backbones and the LVD-enhanced variants, con- firming effectiveness improving perceptual quality. the quality dimensions (Table 10), our method matches or surpasses the strongest baseline on five dimensions, generating visually sharper and temporally smoother videos. These findings underline that our method yields better balance between perceptual fidelity and text adherence than all compared basclines.
Table 9: Overall evaluation results VBench.
Model
Total
Quality
Semantic
VodVioeX-SBB
0.8201
0.8272
0.7917
+ L.D
0.7992
0,8010
0.7921
+ Ours
0.8318
0.8314
0.8332
Wan2.1-14B 0.8369
0.8559
0.7611
LVD
0.8106
0.8186
0.7788
+ Ours
0,8492
0.8573
0.8166
Table 10: Quality dimension results VBench.
Model
Sub.
B.g. Cons.
Temp. Flick.
Motion Smooth
Dyn.
Aesth.
Imag.
Cons.
Deg.
Qual.
Qual.
CogVideoX 5B
0.9656
0.9681
0.9853
0.9815
0.5616
0.6207
0.6534
+ LVD
0.9532
0.9602
0.9823
0.9830
0.4672
0.6008
0.5782
+ Ours
0.9682
0,9773
0.9860
0.9855
0.5426
0.6273
0.6527
Wan2.1-145
0.9752
0.9809
0.9946
0,9830
0.6546
0.6607
0.6943
LVD
0.9531
0.9719
0.9942
0.9835
0.5476
0.6437
0.5602
Ours
0.9577
0.9901
0.9952
0.9895
0.6602
0.6619
0.6837
B.2 ADDITIONAL QUALITATIVE RESULTS
15

---

Published as a conference paper at ICLR 2026
Motion Binding
Spatial Relationship
Wan2.1-14B
Ours
Promps
A petal flonts left right by the flowing stream.
A gurilla sitting on the right side vending machine in forest.
Numeracy
Consistent Attribute
Wan2.1-14B
Ours
Prompt
Five elephants splash water from river.
A girl wearing a colorful scarf gigeles with boy in cute hat they play with a the park.
Interaction
Action
Wan2.1-14B
LYD
Ours
Prompt
Boy shares cream dog.
A panguin slides snowy slope claps.
Planned layout position correctly generated.
X Generated incorrectl..
Planned layout this position but not generated here.
Figure 5: Qualitative comparison between the foundation, the baseline, and our method on T2V. CompBench.
16

---

Published as a conference paper at ICLR 2026
Wan2.1-14B
LVD
Ours
Wan2.1-14B
LVD
Ours
A balloon drifts right to left above statue in a city square.
robot vacuum is sweeping floor from left right.
basketball thrown from right the left.
A child climbs down the slide.
bright lantern floats left down the river, its warm glow reflecting off the gentle ripples.
A car is lifted upward on a platform, wheels spinning as bright showroom lights showcase glossy detail.
Figure 6: More Qualitative comparison between the foundation, the baseline, and our method on T2V- Be.hhhheench
17

---

Published as a conference paper at ICLR 2026
Wan2.1-14B
TTO w/o Memory
TTO Memory
Prompe
A fish swims downward frame.
A tractor drives across the frame from left to right.
(5)
dynamic performance. Stand constrained by the layout- if the LLM generates a layout with small motion amplitudes, the movement direction may not be clearly visible. However, with memory mitialization, the motion dynamics in the frames are significantly enhanced.
examples show adding memory initialization
Wan2.1-14B
Memory w/o TTO
Memory TTO
Prompe
deer stands left and flies right left.
walks the right vehicle drives to the left.
(b) These two examples show adding TTO on top of memory initialization yields better constraint behavior, Some prompts are relatively in the memory, and in extreme may match perfectly aligned motion pattern for example, the two subjects performing different actions in these examples. Applying TTO such cases lends to much stronger consistency.
Figure 7: Qualitative comparison to the impact of memory and TTO.
Matched Pattern
Video
TTOM, Rank-1
<object> <verb> from right to left <environment>,
TTOM, Rank-3
<bjject> <verb>> from the right side senvironment>.
TTOM, Rank-5
<object> <verb> the left side.
Wan2.1-14B
Prompt
A sailboat sails from right to left across the blue waters.
Figure 8: Qualitative comparison when loading items with different retrieval ranks in memory.
18

---

Published as a conference paper at ICLR 2026
Wam2.1-14B
Ours
Layout
Prompt
In sunset sky, eagle flies toward the right side of the frame while drone flies loward the left.
Bird's s-eye view, A cat moves upward from the left side frame, then heads toward the lower right,
Figure 9: Qualitative results on complex cases involving multiple objects or non- non-linear motion.
Wan2.1-14B
TTOM
Prompt
(a). Eight oranges roll across table.
(b), A eyclist pedals from left side to the right side strect.
Figure 10: Failure of TTOM.
19

---

Published as a conference paper at ICLR 2026
C PROMPT DESIGN FOR LLM PLANNING
This appendix details the system prompts used the first stage (LLM Planning) of our pipeline. The planning process involves prompt enrichment, metadata extraction, layout generation, and dedicated verification stage to ensure consistency.
C.I PROMPT ENRICHMENT
The following system prompt used to expand short user inputs into detailed video descriptions.
You are part of team of bots that creates videos. You work with assistant bot that will draw anything you say square brackets.
For example, outputting "a beautiful morning in the weods with the sun peaking through the trees" will trigger your partner bot to output video of forest morning, people looking to create detailed, amazing videos. The way to accomplish this is to take their short prompts and make them extremely detailed and descriptive.
described. You will prompted by
There are few rules follow:
You will only ever output a single video description per user request. When modifications are requested, you should not simply make the description longer. You should refactor the entire description to integrate the suggestions.
Other times the user will not want modifications, but instead want new image. In this case, you should ignore your previous
conversation with the user.
Video descriptions must have the same num of words examples below. Extra words will be ignored.
Always preserve the original prompt's s core content. For example, if the original prompt describes a motion such as "to the left side", please explicitly inciude both the starting Location and the say "from the
destination. For instance, zevise the description right side the left side". Integrate all given details faithfully into your expanded description.
C.2 METADATA EXTRACTION
This prompt extracts structured object information from the enriched description.
You are an intelligent assistant that extracts e*object metadata** from descriptive paragraphs.
You will receive a descriptive paragraph. Your job is to:
1. Identify all distinct **bbjccs*** the paragraph. Objects must be exconcrete physical entities*x that play a vsvisible or active rolex*.
2. Do **not** extract parts attributes (e.g. "wings", "eyes")
3. Only extract objects that literally appear in the original
4. For each object:
Assign a unique "object_id".
Provide the "object" canonical name.
Under "object_phrases", list quoted phrases (min words). Under "object_words" list single words correspending to phrases.
5. If multiple objects with the same name appear, keep the name the same, but assign different ids.
Respond in the specified JSON format.
Guidelines:
Exclude abstract ideas, emotions, locations, and background el elements. include synonyms unless literally mentioned.
20

---

Published as a conference paper at ICLR 2026
Strictly preserve original and formatting.
C.3 LAYOUT GENERATION
This prompt generates bounding box sequences based on the prompt and metadata.
You are intell igent bounding ng box generator for vi deos.
You will receive:
1. original pronpt (core motion intent)
2. converted pronetted_prompt (visual details)
3. object metadata (entities list)
Your task: generate realistic bounding boxes (720x480) per frame (1-6) for main objects.
Bounding box rules:
Follow metion path from aeiginal_prongg exactly.
Format: ('id': <int>, 'name' <object name>, 'box': [x, width, heigat]
**Mumeaacy*:: If muitiple objects the same type exist, assign unique IDs and generate separate boxes. Match box size to camera framing (Close-ups: 5-50x000)
Avoid overlap unless implied.
Output format:
Reasoning paragraph.
Frames 1 to 6: list of bounding bex dicts.
Background keyword,
C.4
VERIFICATION
To ensure the robustness of the generated plan, we employ distinct verification steps: one for textual metadata and one for spatiotemporal layout.
C.4.1 METADATA VERIFICATION PROMPT
This step corrects format errors and ensures linguistic guidelines are followed.
Please check if the extracted object metadata strictly follows the guidelines.
If there are any issues (e.g., inclusion background elements, abstract parts, possessive pronouns like "its", or mismatches between "object_phrasee and "object_words",, correct them.
Return ONLY the corrected "objects" field. If no problems, return unchanged.
Rules:
.*Do net include possessive рreпсuпs** (e.g., "its", "his") in
"object_pphrase"
bobjecctwordds must only contain base object words** (e.g., use "car" instead of "car's"),
Ensure robject_worde are strictly nouns representing whole entities, not attributes.
C.4.2 LAYOUT VERIFICATION PROMPT
This step critiques the generated bounding boxes, focusing on **numerical accuracy** and **spa- tiotemporal consistency**
21

---

Published as a conference paper at ICLR 2026
You are a Quality Assurance expert for video Layout generation.
You will receive:
1. The Object Metadata (containing the list of objects to be detected).
2. The Generated Layout (Frames 1-6).
Your task to verify and correct the layout based on the following strict criteria:
1. **Nueracy Check (Critical) **;
Verify that the number of bounding boxes for each object category matches the count specifled in the metadata.
Example: If the metadata lists "cat" with ID 0 and "cat" with ID I (total 2 cats), every frame MUST contain exactly 2 boxes for "cat". If specific number menticned (e.g., "5 ris"), are distinct IDs.
there
2. ** Temporal Consistency=*:
Motion must be smooth. Eliminate random jitter sudden jumps in position unless the prompt describes erratic movement. Ensure ID consistency: Object ID in Frame must correspond to the same entity in Frane 6.
3. **Size Bounds**:
Objects should not randomly shrink/grow without perspective justification.
Ensure all boxes are within valid coordinates [0, 0, 720, 480).
errors are found, regenerate the "frames" section with corrected coordi nates and counts.
Return the output in the exact same JSON format as the input.
D THEORETICAL ANALYSIS OF THE MEMORY MECHANISM
In this section, we provide a formal theoretical analysis of the proposed memory mechanism. We frame the mechanism merely as caching strategy, but as a Manifold-Aware Initialization scheme that accelerates convergence by leveraging the local smoothness of the task muk-parameter mapping. We further analyze the stability of the update rule to demonstrate its robustness against catastrophic forgetting.
D.1 PROBLEM FORMULATION: TTO AS TRAJECTORY OPTIMIZATION
Let T denote the manifold of compositional video generation tasks (user prompts), and let & Rd represent the parameter space of the trainable modules (e.g., LoRA weights). We define mapping F such that for any task C' T, there exists an optimal parameter set & that minimizes the alignment objective Laligni
dC arg min Latign (d): C*)
DE
The standard Test-Time Optimization (TTO) process seeks to find &e via Gradient Descent starting from generic initialization do. The memory M is defined discrete set of tuples {(k, e) J. where k; = g(C,) is the semantic representation (key) of a historical task, and v; = Øc, is the converged parameter state (value).
D.2 CONVERGENCE ANALYSIS: THE INITIALIZATION ADVANTAGE
The efficiency of our method stems from the bounding of the optimization trajectory. We rely on the Task-Parania Parameter Smoothness Assumption: assume that if two tasks C, and Cý are semantically close in the embedding space (i.c., (lg(C,) g(C,)l || < <), their corresponding optimal parameters lie within local neighborhood on the parameter manifold: || &C, |l 8.
22

---

Published as a conference paper at ICLR 2026
Under the assumption that the loss landscape Lalign is locally ju-strongly convex and L-smooth around the optimum (a standard assumption in tin-tuning analysis), the convergence rate Gradient Descent at step t is bounded by the initial distance to the solution:
1 2 1 S (1 m/4) (I &c.l
(5)
where y) is the learning rate.
In proposed mechanism, the retrieval (Liu al., 2018; Wen al., 2024) function R(M, C) selects an initialization Omem such that:
Omem
if (k, u) E .M s.t. sim(g(C), k.) >
Onnit
(6)
Orandom
otherwise
When a similar structural layout retrieved, ||@ - 22. < Prandom &c Consequently, the number of steps 7 required to reach an error tolerance is significantly reduced. This theoretical bound formally explains the reduction in inference latency and the enhanced motion scores observed in our empirical results.
D.3 STABILITY AND ORTHOGONAL INFORMATION RETENTION
A critical challenge in online learning is Catastrophic Forgetting, where updates for a new task degrade performance previous tasks. We prove that our mechanism immune this phenomenon via its discrete storage topology.
Unlike continual learning approaches that update shared parameter set 0 Loow. our memory update mechanism is additive and orthogonal. The update rule is formally defined as:
Me+1 + Me U (g((nnew) &new))
Because each optimized parameter set d² is stored as discrete entry effectively isolated in the memory bank, the gradient updates for task Gnew do not alter the values en associated with historical keys kg.
Forgetting Dynamies. The only information loss in the system is governed by the cache replace- ment policy (e.g., Least Frequently Used) which occurs only when the memory capacity [M] exceeds the maximum limit Kmax- This ensures that the system prioritizes the retention of high-probability compositional structures, effectively approximating the optimal storage policy for the distribution of user prompts.
23


Reviews:
[{"review_id": "xi0yl6WzGi", "reviewer": "Reviewer_f1wz", "summary": "This paper introduces **TTOM**, a training-free framework that enhances compositional text-to-video (T2V) generation through *test-time optimization (TTO)* and a *parametric memory* mechanism. TTOM aligns cross-attention maps of video foundation models (VFMs) with LLM-generated spatiotemporal layouts at inference time, improving text–video compositional alignment. Its parametric memory stores optimized parameters from previous prompts, enabling reuse, continual learning, and efficient streaming inference. Experiments on **T2V-CompBench** and **VBench** demonstrate significant improvements in compositional reasoning, semantic consistency, and visual quality over baselines such as CogVideoX‑5B and Wan2.1‑14B.", "questions": "see the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposes a novel test-time optimization framework with memory-based reuse, extending inference-time adaptation for video generation in a principled way.  \n- Seamlessly integrates LLM-driven spatiotemporal layout planning for controllable, interpretable video synthesis.  \n- Achieves consistent quantitative and qualitative gains on major T2V benchmarks, including clear improvements in motion and numeracy.  \n- The design supports streaming inference and user-specific memory, aligning well with practical workflows.  \n- Comprehensive ablations, detailed comparisons, and visual evaluations substantiate the method’s effectiveness.", "weaknesses": "1. The method is largely empirical; there is a lack of theoretical explanation or analysis regarding optimization stability and memory dynamics over time.  \n2. Evaluation scenarios mainly focus on compositional prompts; testing under more complex, long-horizon, or open-domain conditions would strengthen generalization claims.  \n3. Dependence on large LLMs (e.g., GPT‑4o) for layout planning introduces computational and cost overheads that are not thoroughly discussed.  \n4. Runtime performance and latency analysis are insufficient; the paper should quantify the cost of test-time optimization steps and memory operations.  \n5. Some ablations, while numerically helpful, could benefit from visual examples or error diagnostics to clarify how memory and optimization complement each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgIYLezevV", "reviewer": "Reviewer_9VTH", "summary": "This paper proposes a framework named TTOM (Test-Time Optimization and Memorization) for improving compositional text-to-video (T2V) generation. During inference, TTOM aligns video generation results with spatiotemporal layouts and introduces lightweight, optimizable parameters for adaptive per-sample updates. It further incorporates a parameterized memory mechanism that supports insertion, retrieval, update, and deletion operations, enabling the reuse of historical optimization context. Experimental results on T2V-CompBench and VBench demonstrate substantial gains, especially in motion, quantity, and spatial relation metrics.\n\nHowever, the paper suffers from a serious conceptual confusion. Although TTOM is repeatedly described as a “training-free” framework, the core algorithm explicitly performs gradient-based parameter updates (e.g., optimizing LoRA weights) during inference. Such optimization is, by definition, a form of test-time training or test-time adaptation, not a training-free operation. Therefore, labeling the method as “training-free” is conceptually incorrect and misleading. The authors must clarify this terminology to ensure methodological and scientific accuracy.", "questions": "**1. Clarification of “training-free” Definition**\n\nThe authors must explicitly state what “training-free” means in this context. Since the method involves gradient updates, how does it fundamentally differ from test-time training or test-time adaptation?\n\n**2. Mathematical Modeling and Stability of Memory**\n\nCould the memory insertion, update, and deletion processes be formalized (e.g., via state transition equations) and analyzed for stability or boundedness?\n\n**3. Memory Capacity and Replacement Policies**\n\nWhen memory reaches capacity, how do different replacement policies (LRU, LFU, FIFO) affect performance, latency, and storage footprint?\n\n**4. Impact of Layout Errors on TTO**\n\nHow sensitive is TTOM to LLM-induced layout errors? How is the “verification step” implemented and evaluated quantitatively?\n\n**5. Fault Tolerance and Error Accumulation**\n\nHow does TTOM prevent error propagation when mismatched memory entries are loaded or when the retrieved parameters are irrelevant?\n\n**6. Fine-Grained Resource Analysis**\n\nCould the authors provide detailed per-sample latency, GPU-hour costs, and LoRA parameter size during streaming?\n\n**7. Hyperparameter Sensitivity**\n\nWhy were LoRA rank=32 and five denoising steps chosen? Are these settings robust across different backbones?\n\n**8. Reproducibility and Randomness Control**\n\nPlease specify random seeds, GPT-4o prompt templates, and evaluation scripts for reproducibility.\n\n**9. Failure Cases and Limitations**\n\nWhat are the main failure modes (e.g., occlusion, excessive object count, ambiguous prompts)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an innovative test-time compositional generation framework equipped with a streaming memory mechanism, distinguishing it from conventional per-sample optimization approaches. This enables continual adaptation and historical context reuse during inference.\n\n2. By unifying per-sample test-time optimization with a parametric memory module, the method achieves continual adaptation and efficient context reuse across inference streams, representing a novel direction in test-time generative modeling.\n\n3. Extensive experiments provide strong empirical evidence of its effectiveness, showing performance improvements of +34.45% and +15.83% over CogVideoX-5B and Wan2.1-14B on T2V-CompBench (Table 1), together with notable gains in semantic consistency on VBench.\n\n4. While the idea of “training-free test-time optimization” is conceptually appealing, the misuse of terminology undermines the paper’s scientific rigor. Any process involving gradient updates or parameter optimization cannot be described as training-free. The term must be corrected in the abstract, introduction, and method sections to maintain terminological precision.", "weaknesses": "**1. Conceptual ambiguity in the “training-free” claim:**\n\nThe paper repeatedly claims that TTOM is “training-free,” yet the method explicitly introduces learnable parameters and updates them via gradient descent during inference. Regardless of whether this occurs at test time, such an optimization process constitutes training—albeit without external training data. Hence, “training-free” and “test-time optimization” are not interchangeable concepts.\nThis is a critical conceptual flaw. The authors should clearly distinguish among:\n\n- training-free: no gradient updates at all;\n\n- fine-tuning-free: no updates to the backbone parameters, but auxiliary modules may be trained;\n\n- test-time optimization: gradient-based updates performed at inference for adaptation.\nSince TTOM belongs to the third category, it should be explicitly defined as a test-time optimization–based framework, not a training-free one.\n\n**2. Lack of theoretical analysis of the memory mechanism:**\n\nThe insertion and update strategies in Equation (3) lack mathematical formalization and convergence analysis. The paper provides only an implementation-level description, without proving or empirically testing stability, information retention, or forgetting dynamics under continual updates.\n\n**3. Dependence on LLM-generated layouts:**\n\nSection 3.1 heavily relies on GPT-4o for layout generation but lacks robustness evaluation. The paper does not quantify how common layout errors (e.g., bounding-box misalignment, temporal inconsistencies) influence test-time optimization or how the “verification step” mitigates error propagation.\n\n**4. Insufficient evaluation of computational and storage overhead:**\n\nThe latency and storage implications of the memory system (especially when full) are not systematically reported. The paper lacks detailed measurements of optimization time, memory growth (LoRA parameter accumulation), and replacement cost under streaming conditions.\n\n**5. Lack of justification for hyperparameter choices:**\n\nThe paper sets LoRA rank=32 and optimizes only the first five denoising steps without explaining why these configurations are near-optimal. No sensitivity analysis is provided for LoRA rank, learning rate, or optimization steps, limiting generalization insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6hH5E1sEl8", "reviewer": "Reviewer_6jPR", "summary": "TTOM is a training-free framework for compositional text-to-video generation. It aligns model attention with LLM-generated spatiotemporal layouts via test-time optimization and stores optimized parameters in a parametric memory for reuse. This design enables continual adaptation across prompts, yielding significant improvements in motion, spatial alignment, and semantic consistency.", "questions": "1. How strongly does TTOM depend on the LLM’s prompt design, model choice, or layout verification? And what happens when the LLM produces noisy or uncertain spatial temporal layouts, can TTOM still perform reliably?\n2. Has the system been tested for cases where outdated or irrelevant memories build up and start to hurt performance? Are there situations where a faulty memory causes negative transfer? i hope the author can provide some comparison between successful and failure cases.\n3. as shown in the above weakness part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear motivation. the paper addresses a key limitation of video foundation models by improving their compositional understanding of objects, relations, attributes, and temporal dynamics within a realistic streaming generation setting.\n2. Parametric memory flexibly manages prompt-level optimization results, enabling efficient transfer and adaptive refinement across prompts.", "weaknesses": "1. Line54 has two \"in this paradigm”, typo error.\n2. The cases shown in Figure 1, when compared with the baseline, reveal that the baseline also produces outputs that are partially semantically correct and sometimes exhibit higher visual quality — for instance, in the case of “An elderly man walking to the right in a sunny park.”\n3. TTOM relies heavily on the layout quality produced by the LLM, assuming that the generated spatiotemporal layouts are accurate and consistent. However, layout hallucinations or spatial inaccuracies in the LLM outputs can directly cause alignment errors, misleading the optimization process and leading to degraded results.\n4. There exists a structural trade-off between efficiency and quality.  Continuous TTO introduces significant latency, while memory mechanisms can reduce computational overhead but require a warm-up phase to accumulate sufficient samples.  Consequently, the deployment barrier remains high for online or low-latency applications such as real-time interactive editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5mRQiwUqHE", "reviewer": "Reviewer_HTdp", "summary": "The paper proposes a method to enhance the alignment capability of compositional text-to-video generation during the inference phase through test-time optimization and a memory mechanism.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces the TTOM framework, which combines test-time optimization with a parametric memory mechanism, enabling dynamic adjustment of model parameters at inference time and retaining historical optimization results for reuse with similar future prompts.\n2. This approach not only improves text-video alignment in compositional scenarios but also supports continuous learning and personalized generation, demonstrating strong practicality and scalability. On mainstream benchmarks such as T2V-CompBench and VBench, the TTOM method significantly outperforms existing approaches across multiple key metrics (e.g., motion, numeracy, spatial relations), achieving overall improvements of up to 34% and 15% on large models like CogVideoX-5B and Wan2.1-14B, respectively, thereby validating its effectiveness and generalization capability.", "weaknesses": "1. The effectiveness of the method heavily relies on the quality of the spatiotemporal layouts generated by the LLM. If the bounding box sequences produced by the LLM are inaccurate, subsequent attention alignment optimization may fail or even introduce misleading guidance, compromising the quality of the generated content. \n\n2. Test-time optimization requires extra gradient computations and parameter updates, which significantly increase inference latency, especially during continuous optimization and memory retrieval. Although the paper mentions that optimization is applied to the first five denoising steps of the diffusion process to partially mitigate this issue, its acceptability in real-world streaming generation scenarios still requires further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)