TTOM: Test-Time Optimization and Memorization for Compositional Video Generation

## Abstract

Abstract Video Foundation Models (VFMs) exhibit remarkable visual generation performance, but struggle in compositional scenarios ( e.g. , motion, numeracy, and spatial relation).
In this work, we introduce Test-Time Optimization and Memorization (TTOM) , a training-free framework that aligns VFM outputs with spatiotemporal layouts during inference for better text-image alignment.
Rather than direct intervention to latents or attention per-sample in existing work, we integrate and optimize new parameters guided by a general layout-attention objective.
Furthermore, we formulate
video generation within a streaming setting, and maintain historical optimization contexts with a parametric memory mechanism that supports flexible operations, such as insert, read, update, and delete.
Notably, we found that TTOM disentangles compositional world knowledge, showing powerful transferability and generalization.
Experimental results on the T2V-CompBench and Vbench benchmarks establish TTOM as an effective, practical, scalable, and efficient framework to achieve cross-modal alignment for compositional video generation on the fly.

## 1 Introduction

Recent years have witnessed a rapid stride in video generation (Ho et al., 2022; Polyak et al., 2024) and world simulation (Brooks et al., 2024).
Benefiting from the significant advance in flow matching (Lipman et al., 2022; Esser et al., 2024) and diffusion transformers (DiTs) (Peebles & Xie, 2023), current text-to-video generation (T2V) models (Yang et al., 2024b; Kong et al., 2024) are able to synthesize realistic and vivid videos. However, even state-of-the-art models still suffer from *text-video misalignment* in compositional scenarios, *i.e.*, composing multiple objects, attributes, and relations into a complex scene (Sun et al., 2025; Huang et al., 2024).

Remarkable efforts have been made to improve text-video alignment by introducing explicit spatiotemporal layout guidance (Lian et al., 2023b; He et al., 2025) as a bridge between text prompts and videos.
In this paradigm,
In this paradigm, a layout represented by bounding box (bbox) sequences—each corresponding to an object—is first generated by a large language model (LLM), and subsequently used to optimize intermediate representations (*e.g.*, latents and attention maps) by enforcing alignment with the layout constraints.
Despite notable progress, this line of work faces several limitations:
1) direct intervention on intermediate representations may disrupt feature distributions, leading to degraded video quality (*e.g.*, inconsistency, flickering artifacts, or even collapse);
2) the per-sample control paradigm treats each case independently, thereby neglecting the historical context of previously generated videos;
and 3) these methods do not enhance the intrinsic capability of generative models, since interventions on one sample fail to generalize to others.

In real-world scenarios, models are not presented with a set of independent test cases but with a continuous stream of user prompts, as illustrated in Fig. [2](https://arxiv.org/html/2510.07940v1#S2.F2).
Previous successful generations can serve as valuable references for future cases.
In light of this, our core idea is to formulate compositional video generation in a streaming setting, leveraging history test-time optimization as context for future inference.
However, modeling history contexts is non-trivial, as it requires addressing the fundamental challenge of how to represent, store, reuse, and update past information.

In this work, we introduce Test-Time Optimization and Memorization (TTOM) for compositional T2V.
For the first test case (cold start), a large language model (LLM) first derives a spatiotemporal layout from the text prompt, serving as a controllable condition.
Conditioned on this layout, we perform Test-Time Optimization (TTO) by instantiating and updating sample-specific parameters, thereby steering video generation toward adherence to the prescribed layout.
Through this process, the compositional patterns embedded in the layout, such as motion, numeracy, and multi-object interactions, are attained and saved into the new parameters.
After optimization, the parameters are unloaded and stored in memory, using the extracted layout-related keywords from the prompt as keys.
For subsequent test cases, when matched parameters are retrieved, they can be integrated into the foundation model and further optimized for improved adaptation, or directly applied for efficient inference.
If no match is found, new parameters are initialized, optimized, and subsequently recorded in memory to expand the memory.
In practice, the memory is assigned a fixed capacity; when full, items are removed according to predefined policies (*e.g.*, least frequently used).
This mechanism enables the flexible reuse of informative optimization results by preserving historical context, thereby enhancing both efficiency and scalability.
In summary, our main contributions are as follows.

- •
We propose a test-time optimization framework without any supervision for compositional T2V. With spatiotemporal layout as guidance, we incorporate and optimize lightweight parameters for each data sample.
- •
We present a parametric memory mechanism to maintain historical optimization context, which naturally supports lifelong learning. A series of operations (*e.g.*, insert, read, update, and delete) for memory ensures flexibility, scalability, and efficiency.
- •
Extensive experimental on two T2V benchmarks (Sun et al., 2025; Huang et al., 2024) demonstrate the effectiveness and superiority of the proposed method. Notably, it achieves relative improvements of 34% and 14% over CogVideoX-5B (Yang et al., 2024b) and Wan2.1-14B (Wan et al., 2025) on T2V-CompBench (Sun et al., 2025), respectively.

## 2 Related Work

Compositional Visual Generation.
Despite the thrilling progress in foundation models (Yang et al., 2024b; Kong et al., 2024; Polyak et al., 2024; Seawead et al., 2025), compositional generation (Chefer et al., 2023; Feng et al., 2022) remains an open challenge and has attracted substantial research interest.
To enhance text-image alignment in compositional scenes, a rich line of studies (Qu et al., 2023; Feng et al., 2023) explores incentivizing the layout planning ability of LLMs (Brown et al., 2020; Ouyang et al., 2022) to enable layout-guided controllable generation (Tian et al., 2024; Lin et al., 2023; Feng et al., 2025) through attention control (Chen et al., 2024; Xie et al., 2023; He et al., 2023; Wang et al., 2024; He et al., 2025) or latent modification (Qi et al., 2024; Wu et al., 2024; Lian et al., 2023a; Yang et al., 2024a; Wang et al., 2025b; Lian et al., 2023b), typically in a training-free manner.
However, direct intervention on attention maps or latents may compromise visual quality. Moreover, the per-sample paradigm treats test samples in isolation, disregarding informative historical
context.

Test-Time Optimization.
TTO adapts a trained model to test instance(s) by performing optimization during inference, which has been widely applied in domain adaptation (Li et al., 2016; Sun et al., 2020; Shu et al., 2022) and alignment (Kim et al., 2025; Li et al., 2025).
Existing approaches can be categorized into three groups: 1) optimizing intermediate representations (Kim et al., 2025; Li et al., 2025), 2) prompt tuning (Shu et al., 2022), and 3) optimizing model parameters (Schneider et al., 2020; Liang et al., 2020; Shu et al., 2021; Zhang et al., 2022; Gandelsman et al., 2022).
In visual generation, recent work (Lian et al., 2023b; Wu et al., 2024) optimizes latents guided by layouts, yielding significant gains. However, these methods perform optimization on a per-sample basis and subsequently discard the results, thereby neglecting historical context.

Memory Mechanism.
To track and model historical contexts, existing Test-Time Training (TTT) (Sun et al., 2024) work designs memory mechanisms instantiated with inserted layers, *i.e.*, TTT layers. The memory can be updated linearly (Sun et al., 2020) or in a local temporal window (Wang et al., 2025a), by optimizing TTT layers via a self-supervised learning loss (He et al., 2022).
Despite the remarkable advance, they often consider intra-sample context (*e.g.*, frames in video) (Wang et al., 2025a; Dalal et al., 2025) and assume the independence among samples. Besides, the memory has a fixed size and does not support flexible operations.

Figure: Figure 2: Overview of the TTOM framework for compositional text-to-video generation. A stream of text prompts is first fed into LLMs for spatial-temporal layout planning. Meanwhile, a denoising sampling process of video foundation models is performed, in which cross-attention maps are extracted, followed by test-time optimization for alignment. Historical optimization context is maintained by the parametric memory.
Refer to caption: x2.png

## 3 Methodology

In this section, we introduce the proposed TTOM framework for compositional T2V, as shown in Fig. [2](https://arxiv.org/html/2510.07940v1#S2.F2).
Given a text prompt describing a compositional scene (*e.g.*, including multiple objects, attributes, numeracy, and relationships), we first conduct spatial-temporal layout (STL) planning driven by LLMs, and obtain the bbox sequence for each object (Sec. [3.1](https://arxiv.org/html/2510.07940v1#S3.SS1)).
Subsequently, we perform test-time optimization to achieve attention-to-layout alignment, making the generated video follow the explicit layout guidance (Sec. [3.2](https://arxiv.org/html/2510.07940v1#S3.SS2)).
To take advantage of history context, we record previous optimization results with a parametric memory structure, and keep updating and reusing it as the sequential inference goes, *i.e.*, a ceaseless stream of user prompts is fed into the model (Sec. [3.3](https://arxiv.org/html/2510.07940v1#S3.SS3)).

### 3.1 LLM-driven Spatial-Temporal Layout Planning

Layout Representation.
We represent a video-level STL with a collection of object-level layouts $\{Y_{0},...,Y_{N}\}$ where $N$ denotes the number of objects.
Each layout $Y_{i}$ is a tuple $(c_{i},B_{i})$, consisting of an object phrase $c_{i}$ and its associated bounding box sequence $B_{i}=[\mathbf{b}_{s_{i}},\ldots,\mathbf{b}_{e_{i}}]$.
The object phrase $c_{i}$ is extracted from the user prompt $C$ (*e.g.*, the underlined text in “A vibrant red balloon drifts right to left above a grand statue.”).
Each $\mathbf{b}$ is a 4-dimensional coordinate vector, while $s_{i}$ and $e_{i}$ denote the start and end frame indices of the object’s temporal occurrence, respectively.

Text-to-Layout Generation.
Prior work has shown that LLMs possess strong spatial-temporal dynamics understanding abilities (Qu et al., 2023; Feng et al., 2023),
and capture physical properties (Lian et al., 2023b), such as gravity, elasticity, and perspective projection, which align closely with real-wrold physics.
Inspired by it, we prompt LLMs to generate SPLs given user prompts via in-context learning (Dong et al., 2022). To enhance scene comprehension, the LLM is first instructed to produce descriptions of object motions and
camera behaviors, followed by the corresponding layout generation.
Finally, a verification step ensures spatial and temporal
consistency of object phrases and bounding box sequences, with corrections applied when discrepancies are detected.

### 3.2 Test-Time Optimization for Layout-to-Video Generation

In this section, we first discuss the relation between intermediate attention maps of diffusion models and generated videos, and conduct a probe experiment to verify the relevance. On the basis of the relevance, we propose a training-free controllable generation strategy by optimizing attention-to-layout alignment during inference, *i.e.*, Test-Time Optimization (TTO).

Figure: Figure 3: Attention-layout overlap (evaluated by mIoU (Everingham et al., 2010) over 200 prompts) between cross-modal attention maps extracted from each layer of foundation models and segmentation maps detected from generated videos by GroundingDINO (Liu et al., 2024) + SAM 2 (Ravi et al., 2024).
Refer to caption: x3.png

Attention-Layout Relevance Probe in Diffusion Transformers.
Prior work (Hertz et al., 2022) has demonstrated the relevance between cross-attention maps and spatial layouts, which is exploited for video editing (Qi et al., 2023) and controllable generation (Xie et al., 2023).
However, with the architecture shift from UNets (Ronneberger et al., 2015) to Diffusion Transformers (DiTs) (Peebles & Xie, 2023) due to its stronger scalability, the recent study (Wang et al., 2025b) pointed out their difference, *i.e.*, the cross-attention maps in DiTs distribute more evenly than those in UNets, which may cause the ineffectiveness of previous methods due to the weak attention-layout relevance.

Motivated by it, we propose a probe strategy to assess attention-layout relevance across different layers:
1) preparing a set of prompts describing a common scene in which an object performs a motion. 2) generating videos given the prompts, and meanwhile, extracting text-video cross-attention maps.
Specifically, we first extract the cross-attention map(^†^†For foundation models (Yang et al., 2024b) without cross-attention layers, we extract those parts in self-attention maps where text-vision interaction happens.) $A_{i}\in\mathbb{R}^{\tau\times h\times w\times|c_{i}|}$ from each layer for the object phrase $c_{i}$ from a text prompt $C$, where $|c_{i}|$ denotes the token number of $c_{i}$. $\tau$, $h$, and $w$ refer to the frame number, height, and width of the video latent, respectively.
Afterwards, we perform average pooling along the text dimension and obtain $\bar{A}_{i}\in\mathbb{R}^{\tau\times h\times w}$. This process is formulated as:

$$ $\bar{A}_{i}=f_{\Theta}(C,c_{i},z_{t},t),$ (1) $$

where $z_{t}$ and $t$ represent the latent and timestep, respectively. $\Theta$ denotes the parameters of all layers before the attention map in a DiT.
3) applying GroundingDINO (Liu et al., 2024) to detect the related bboxes and SAM 2 (Ravi et al., 2024) for segmentation. And 4) computing the overlap between segmentation maps and attention maps $\bar{A}_{i}$ in different layers to quantify relevance. The result shown in Fig. [3](https://arxiv.org/html/2510.07940v1#S3.F3) indicates a remarkable variance of attention-layout relevance among different layers.

Attention-to-Layout Alignment for Controllable Generation.
As discussed above, attention maps in certain layers of DiTs reflect a strong relevance to the layout of the final generated videos.
Motivated by this observation, we propose a test-time optimization approach to align the high-relevance attention maps with the layout generated by LLMs for controllable generation.
Specifically, we first smooth each spatial layout $\mathbf{b}_{i}$ in $B_{i}$ with a Gaussian kernel and transform it into a soft mask $\bar{\mathbf{b}}_{i}$, and then calculate the Jensen–Shannon divergence (JSD) between attention maps and soft masks of $N$ objects as the loss function:

$$ $L_{align}=\frac{1}{N}\sum_{i=1}^{N}JSD(\bar{A}_{i}\|\bar{B}_{i}),$ (2) $$

We insert new parameters $\phi$ into VFMs and get $\Theta=\theta\cup\phi$, calculate the gradient of the loss function *w.r.t.*the newly introduced parameters, *i.e.*, $\frac{\partial L_{align}}{\partial\phi}$, and update them with an optimizer and get $\phi^{*}$ finally.
Compared with prior work (Lian et al., 2023b) that optimizes the latent $z_{t}$ at denoising timestep $t$ by $\frac{\partial L_{align}}{\partial z_{t}}$, we have the following strengths: 1) optimizing $\phi$ for alignment avoids possible distribution collapse caused by direct intervention to the latents. And 2) the updated $\phi^{*}$ memorizes layout patterns for specific compositional scenes, holding a potential for future re-usage.
For example, if a similar text prompt comes in the future, a simple and efficient strategy is to directly load $\phi^{*}$ to the VFM, and perform generation without any optimization, which motivates us to propose a memory mechanism as follows.

### 3.3 Parametric Memory

To maintain historical contexts of test-time optimization and support future reuse, we present the parametric memory mechanism.

Memory Structure.
We define a memory structure as a collection of key-value pairs:

$$ $\mathcal{M}=\{g(C):\phi^{*}_{C}|C\in\mathcal{C}_{H}\},$ (3) $$

where $\mathcal{C}_{H}$ is a set of history prompts. The function $g(\cdot)$ consists of scene abstraction, text embedding, and indexing precedes. Specifically, for a prompt “A vibrant red balloon drifts right to left above a grand statue.”, we first abstract it into “<object A> drifts right to left above <object B>.”, and then extract its text feature (Radford et al., 2021) and index it.

Memory Operations.
Based on the memory structure, we define the following basic operations to perform interaction between the video foundation model and the memory.

$\bullet$ Insert. For a new text prompt $C_{j}$, we first retrieve from the memory. If no matched item, we perform test-time optimization and insert a new item $(g(C_{j}),\phi^{*}_{C_{j}})$ to $\mathcal{M}$.

$\bullet$ Read. If the matched item(s) exist, we read and load the corresponding parameters into the foundation model.
Next, we have two options: 1) directly generate a video without optimization, and 2) continue optimization with the loaded parameters as initialization.

$\bullet$ Update. After the loaded parameters are further optimized, we unload them and update the corresponding matched items in the memory.

$\bullet$ Delete. If the total number of items in $\mathcal{M}$ exceeds its maximum capacity, we delete those items that are least frequently used.

Discussion.
Compared with prior methods (Lian et al., 2023b; He et al., 2025) treating each sample independently, the advantages of the proposed memory mechanism lie in:
1) Superiority. The history optimization maintained in the memory provides abundant scene knowledge, serving as good initialization for TTO.
2) Personalization. In practice, we can maintain a user-specific memory for each user, track their historical requests, and model the intention for better personalized generation.
3) Efficiency. For those prompts similar to historical ones, we can skip optimization and directly read and load items into the foundation model for inference.
4) Scalability. We can scale the memory up in two dimensions: increasing the item-level capacity (*i.e.*, enlarging the history context window) could memorize more historical optimization information; and increasing the number of per-item parameters $\phi^{*}$ may capture more scene patterns in each TTO.

**Table 1: Evaluation results of compositional text-to-video generation on T2V-CompBench (Sun et al., 2025), reported over 7 categories and the overall average (Avg.).**
| Model | Avg. | Motion | Num | Spatial | Con-attr | Dyn-attr | Action | Interact |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Commercial |  |  |  |  |  |  |  |  |
| Pika-1.0 | 0.3752 | 0.2234 | 0.3870 | 0.4650 | 0.5536 | 0.0128 | 0.4250 | 0.5198 |
| Gen-3 | 0.4094 | 0.2754 | 0.2306 | 0.5194 | 0.5980 | 0.0687 | 0.5233 | 0.5906 |
| Dreamina 1.2 | 0.4689 | 0.2361 | 0.4380 | 0.5773 | 0.6913 | 0.0051 | 0.5924 | 0.6824 |
| Kling-1.0 | 0.4630 | 0.2562 | 0.4413 | 0.5690 | 0.6931 | 0.0098 | 0.5787 | 0.7128 |
| Diffusion Unet based |  |  |  |  |  |  |  |  |
| ModelScope | 0.3468 | 0.2408 | 0.1986 | 0.4118 | 0.5148 | 0.0161 | 0.3639 | 0.4613 |
| + LVD | 0.3912 | 0.2457 | 0.2008 | 0.5405 | 0.5439 | 0.0171 | 0.3802 | 0.4502 |
| Show-1 | 0.3676 | 0.2291 | 0.3086 | 0.4544 | 0.5670 | 0.0115 | 0.3881 | 0.6244 |
| VideoTetris | 0.4097 | 0.2249 | 0.3467 | 0.4832 | 0.6211 | 0.0104 | 0.4839 | 0.6578 |
| T2V-Turbo-V2 | 0.4317 | 0.2556 | 0.3261 | 0.5025 | 0.6723 | 0.0127 | 0.6087 | 0.6439 |
| DiT based |  |  |  |  |  |  |  |  |
| Open-Sora 1.2 | 0.3851 | 0.2468 | 0.3719 | 0.5063 | 0.5639 | 0.0189 | 0.4839 | 0.5039 |
| Open-Sora-Plan v1.3.0 | 0.3670 | 0.2377 | 0.2952 | 0.5162 | 0.6076 | 0.0119 | 0.4524 | 0.4483 |
| CogVideoX-5B | 0.4189 | 0.2658 | 0.3706 | 0.5172 | 0.6164 | 0.0219 | 0.5333 | 0.6069 |
| + DyST-XL | 0.5081 | 0.2712 | 0.3969 | 0.6110 | 0.8696 | 0.0221 | 0.7321 | 0.6536 |
| + LVD | 0.4739 | 0.3291 | 0.3825 | 0.5274 | 0.7534 | 0.0219 | 0.6826 | 0.6204 |
| + Ours | 0.5632 | 0.4351 | 0.5081 | 0.6173 | 0.8782 | 0.0341 | 0.7191 | 0.7502 |
| %Improve. | +34.45 | +63.69 | +37.10 | +19.35 | +42.47 | +55.71 | +34.84 | +23.61 |
| Wan2.1-14B | 0.5314 | 0.2696 | 0.5113 | 0.5709 | 0.8369 | 0.0570 | 0.7504 | 0.7239 |
| + LVD | 0.5439 | 0.2864 | 0.4707 | 0.5753 | 0.8610 | 0.0829 | 0.8107 | 0.7201 |
| + Ours | 0.6155 | 0.4922 | 0.5881 | 0.6275 | 0.8982 | 0.1182 | 0.8152 | 0.7691 |
| %Improve. | +15.83 | +82.57 | +15.02 | +9.91 | +7.32 | +107.37 | +8.64 | +6.24 |

## 4 Experiments

### 4.1 Experimental Setup

Foundation Models.
We integrate our method into two representative T2V models: CogVideoX-5B (Yang et al., 2024b) and Wan2.1-14B (Wan et al., 2025).
For CogVideoX-5B, each clip contains 49 frames at 8 FPS, whereas for Wan2.1-14B, each clip contains 81 frames at 16 FPS.
In the Spatial–Temporal Layout Planning stage, we leverage the OpenAI GPT-4o model.

Streaming Video Generation Setting.
To align with the practical video creation scenarios in which streaming user prompts come one by one, we propose to model historical streaming contexts. To strictly keep a fair comparison with existing methods that view test samples independently, we have a test-time independence setting: we first generate 200 user prompts using GPT-4o to cover common compositional scenes, and then generate videos with these prompts, at the same time, perform TTOM to construct a memory. During inference, we can read and load entries from the memory, but forbid inserting or updating the memory with the optimization results of test samples.

Streaming Video Generation Setting.
To align with practical video creation scenarios where user prompts arrive sequentially, we propose modeling historical streaming contexts. For a fair comparison with prior methods that treat test samples independently, we introduce a *test-time independence* setting. Specifically, we generate 200 prompts covering common compositional scenes as *pseudo-training data*, via GPT-4o, and subsequently generate videos from these prompts while applying TTOM to construct a memory. During inference, entries can be retrieved and loaded from the memory; however, inserting or updating the memory with optimization results from test samples is prohibited.

Implementation Details.
We employ the lightweight LoRA (Low-Rank Adaptation) (Hu et al., 2022) method to introduce new parameters to enable test-time optimization for controllable generation. The optimized LoRA parameters are inserted into or used to update the parametric memory. Specifically, LoRA weights are injected into the Query, Key, Value, and Output projection layers of each cross-attention block in DiT architectures, with the LoRA rank set to 32. This optimization is applied only to the first five denoising steps of the diffusion process, using the AdamW optimizer with a learning rate of 1e-4. Unless otherwise noted, all other hyperparameters follow the default settings of the respective backbone models.
For fair comparison, we also re-implement the LVD method (Lian et al., 2023b) on the same DiT-based architecture. The layouts and selected layers used for LVD exactly match those used for our TTO version, while all other training hyperparameters follow the original LVD paper.

Benchmarks and Evaluation Metrics.
We evaluate our approach on two large-scale public benchmarks, T2V-CompBench (Sun et al., 2025) and VBench (Huang et al., 2024). T2V-CompBench targets compositional T2V with about 1,400 prompts across seven categories, including consistent and dynamic attribute binding, spatial and motion relations, action binding, object interactions, and generative numeracy. Its evaluation combines MLLM-based scoring, detection-based measures, and tracking-based indicators to provide fine-grained assessment of compositional abilities.
VBench offers a broad evaluation suite, decomposing video generation quality into 16 dimensions, including subject and background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic and imaging quality, as well as object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style, and overall consistency. Each dimension contains roughly 100 text prompts with automatic evaluation scripts and human preference annotations. We follow the official toolkits of both benchmarks and report corresponding scores.

Baselines.
We conduct a comprehensive comparison of our approach on T2V-CompBench against a diverse set of baselines across three categories: commercial solutions (Pika-1.0 (Pika, 2024), Gen-3 (Runway, 2024), Dreamina-1.2 (CapCut, 2024), Kling-1.0 (Kuaishou, 2024)); diffusion U-Net–based methods (ModelScope (Wang et al., 2023), Show-1 (Zhang et al., 2024), VideoTetris (Tian et al., 2024), T2V-Turbo-V2 (Li et al., 2024), LVD (Lian et al., 2023a)); and DiT-based models (Open-Sora-1.2 (HPC-AI Tech, 2024), Open-Sora-Plan-v1.3.0 (Lin et al., 2024), CogVideoX-5B (Yang et al., 2024b) with DyST-XL (He et al., 2025) and LVD, Wan2.1-14B (Wan et al., 2025) with LVD). Except for CogVideoX-5B with LVD, Wan2.1-14B, and Wan2.1-14B with LVD, for which we reproduce the results in our experiments, the scores of all remaining baselines are taken directly from the official T2V-CompBench paper (Sun et al., 2025).

### 4.2 Performance Comparison

Evaluation on Compositionality.
Table [1](https://arxiv.org/html/2510.07940v1#S3.T1) presents the compositional evaluation results of our method compared to a wide range of baselines on T2V-CompBench. Across all metrics, our approach consistently surpasses the baseline models, demonstrating effectiveness and superiority. Notably, our method improves CogVideoX-5B by 34.45% and Wan2.1-14B by 15.83% in terms of overall average performance. The most pronounced gains are observed in Motion and Numeracy, with relative improvements of up to 63.69% and 37.10% on CogVideoX-5B, and 82.57% and 15.02% on Wan2.1-14B, respectively. These categories are known to be particularly challenging due to the need for precise modeling of dynamic object trajectories, object counts, and spatial-temporal coordination.

**Table 2: Evaluation results on the semantic categories of VBench.**
| Model | Obj.<br>Class | Multi-<br>Obj. | Human<br>Act. | Color | Spatial<br>Rel. | Scene | Appear.<br>Style | Temp.<br>Style | Overall<br>Cons. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CogVideoX-5B | 0.8342 | 0.6728 | 0.9760 | 0.8840 | 0.7943 | 0.5328 | 0.2468 | 0.2542 | 0.2742 |
| + LVD | 0.8661 | 0.6782 | 0.9740 | 0.8802 | 0.8014 | 0.5406 | 0.2472 | 0.2509 | 0.2609 |
| + Ours | 0.9486 | 0.7952 | 0.9820 | 0.9246 | 0.8215 | 0.5682 | 0.2550 | 0.2576 | 0.2678 |
| Wan2.1-14B | 0.8628 | 0.6958 | 0.9540 | 0.8859 | 0.7539 | 0.4575 | 0.2264 | 0.2319 | 0.2591 |
| + LVD | 0.9083 | 0.7286 | 0.9620 | 0.8832 | 0.7547 | 0.5135 | 0.2280 | 0.2407 | 0.2507 |
| + Ours | 0.9921 | 0.8216 | 0.9800 | 0.9289 | 0.8074 | 0.5286 | 0.2305 | 0.2444 | 0.2542 |

Evaluation on Semantic Consistency.
Beyond compositionality, we further evaluate semantic consistency on VBench (Table [2](https://arxiv.org/html/2510.07940v1#S4.T2)). Here, our method also delivers clear improvements across multiple dimensions, including object classification accuracy, multi-object handling, and color/spatial relation fidelity. These aspects are crucial for ensuring that generated videos do not only satisfy individual attributes but also maintain coherent semantic integrity across frames. The gains in semantic consistency complement the compositionality improvements observed on T2V-CompBench, providing strong evidence that our method reinforces both the local correctness of attributes (*e.g.*, colors, actions, object categories) and the global contextual structure (*e.g.*, multi-object interactions and spatial layouts).

### 4.3 In-depth Analysis

To explore the efficacy of TTOM, we conduct extensive ablation studies and hyperparameter analyses. We first investigate the TTO and Memorization, followed by an in-depth study of key components, including continual TTO, loss function for attention-to-layout alignment, guidance timesteps, and optimization iteration.

**Table 3: Ablation study for TTO and Memory on the motion category of T2V-CompBench (Sun et al., 2025).**
| TTO | Memory | Motion |
| --- | --- | --- |
| ✗ | ✗ | 0.2696 |
| ✓ | ✗ | 0.4321 |
| ✓ | ✓ | 0.4922 |

Test-Time Optimization and Memorization.
We conduct comparative experiments to evaluate the effectiveness of TTO and the memory mechanism, as reported in Tab. [3](https://arxiv.org/html/2510.07940v1#S4.T3).
The results show that TTO substantially improves motion quality in generated videos by 60.27%, guided by LLM-planned spatiotemporal layouts.
Moreover, incorporating historical context from memory yields an additional 13.91% improvement.

**Table 4: Ablation study for continual test-time optimization on the motion category of T2V-CompBench (Sun et al., 2025). Motion score and latency are used to evaluate motion quality and efficiency, respectively. Init.: initialization from memory with the average fusion of Top-$k$ matched entries. TTO: whether to perform test-time optimization.**
| Init. | TTO | Top-$k$ | Motion$\uparrow$ | Latency (s)$\downarrow$ |
| --- | --- | --- | --- | --- |
| ✗ | ✗ | – | 0.2696 | 425 |
| ✓ | ✗ | 5 | 0.4754 | 427 |
| ✓ | ✓ | 5 | 0.4846 | 627 |
| ✓ | ✗ | 10 | 0.4437 | 427 |
| ✓ | ✓ | 10 | 0.4705 | 627 |

Continual Test-Time Optimization.
The parametric memory enables the model to exploit historical context for future inference. In practice, however, memory capacity is limited and cannot fully encompass world knowledge (*e.g.*, object motion patterns). Consequently, imperfect matching may arise between the current prompt and the retrieved items, leading to suboptimal parameter initialization.
Therefore, continual optimization can help balance historical context and the current sample. To examine its effect, we conduct experiments in Tab. [4](https://arxiv.org/html/2510.07940v1#S4.T4).

The results indicate that:
1) The memory mechanism is effective, as evidenced by substantial gains even without per-sample optimization.
2) With memory-based initialization, continual TTO further improves motion performance at the cost of additional test-time computation.
And 3) increasing the number of memory entries as initialization provides richer context but may also introduce noise or irrelevant information, suggesting that more sophisticated fusion strategies are required for better leveraging context in the future.

**Table 5: Comparison among different loss functions for attention-to-layout alignment, evaluated on the motion and numeracy categories of T2V-CompBench.**
| Loss Function | Motion | Numeracy |
| --- | --- | --- |
| CE Loss | 0.2912 | 0.5218 |
| CoM Loss | 0.3626 | 0.4697 |
| JSD Loss | 0.4321 | 0.5881 |

In summary, the proposed TTOM offers a flexible trade-off between video quality and efficiency, making it well-suited for complex practical scenarios.

Loss Function for Attention-to-Layout Alignment.
To drive controllable generation guided by spatiotemporal layout, we propose a JSD loss to perform attention-to-layout alignment optimization during inference. To validate its effectiveness, we conduct experiments to compare it with two other variants: BCE loss (Sella et al., 2025) and Center-of-Mass (CoM) loss (Lian et al., 2023b).
Results in Tab. [5](https://arxiv.org/html/2510.07940v1#S4.T5) show that the proposed JSD loss achieves the best performance on motion and numeracy, demonstrating its effectiveness and universality.

Pseudo-training Data Scale.
To simulate the streaming generation scenario and make a strictly fair comparison with prior methods, we first generate user prompts via GPT-4o to initialize the memory. Results in Tab. [8](https://arxiv.org/html/2510.07940v1#S4.T8) show the influence of different numbers of pseudo-training data on the performance. We can see that more data induces better performance, due to the more abundant compositional patterns captured and saved into the memory as context.

**Table 6: Impact of pseudo-training data scale, evaluated on the motion category of T2V-CompBench.**
| Table 6: Impact of pseudo-training data scale, evaluated on the motion category of T2V-CompBench.<br>#Pseudo-training Data<br>Motion<br>50<br>0.3519<br>100<br>0.4344<br>150<br>0.4751<br>200<br>0.4922<br>Table 7: Impact of number of timesteps for optimization during the early stage of denoising sampling.<br>#Timestep<br>Motion<br>1<br>0.2730<br>3<br>0.3692<br>5<br>0.4321<br>7<br>0.4296<br>Table 8: Impact of the number of test-time optimization iterations per denoising sampling timestep.<br>#Iteration<br>Motion<br>4<br>0.3789<br>8<br>0.4321<br>12<br>0.4130<br>16<br>0.3547 |
| --- |

Guidance Timesteps and Optimization Iteration.
Existing work (Choi et al., 2022; Pan et al., 2024) has shown that early denoising steps in diffusion models primarily define structure, while later steps progressively refine details.
Motivated by this observation, we apply TTO only in the initial denoising steps and compare performance across different numbers of steps, as presented in Tab. [8](https://arxiv.org/html/2510.07940v1#S4.T8).
Within each step, we further examine the impact of varying the number of optimization iterations, reported in Tab. [8](https://arxiv.org/html/2510.07940v1#S4.T8).
The results reveal clear saturation points for both timesteps and optimization iterations, beyond which performance degrades.
This degradation may arise because attention maps encode not only layout but also entangled information, and excessive alignment optimization may disrupt other essential signals.

Figure: Figure 4: Qualitative results of motion pattern transfer with memory. Solid arrows indicate insert or update operations, while dotted arrows represent reading and loading parameters from memory into foundation models for inference.
Refer to caption: x4.png

Qualitative Results.
Fig. [1](https://arxiv.org/html/2510.07940v1#S0.F1) shows the qualitative comparison between Wan2.1-14B and our method based on which. To further delve into the impact of the proposed memory mechanism, we show the qualitative results of motion pattern transfer from memory to foundation models in Fig. [4](https://arxiv.org/html/2510.07940v1#S4.F4). More qualitative results can be found in the Appendix.

## 5 Conclusion

In this work, we tackle the compositional limitations of Video Foundation Models and introduce TTOM, a training-free framework for aligning video generation with spatiotemporal layouts at inference. By integrating layout-guided optimization with a parametric memory mechanism, TTOM enables consistent streaming generation and supports flexible operations on memory, such as insertion, update, and retrieval of historical contexts. Our experimental analysis shows that TTOM not only improves alignment in compositional scenarios, but also disentangles compositional world knowledge, leading to strong transferability and generalization. Extensive experiments on T2V-CompBench and VBench confirm its effectiveness, scalability, and practicality, establishing TTOM as a versatile solution for enhancing compositional video generation on the fly.

## Ethics Statement

Our work on compositional video generation holds significant promise for democratizing creative tools, enhancing accessibility, streamlining media production, and advancing intuitive human–AI collaboration. At the same time, it raises critical concerns, including potential misuse for misinformation or manipulation, biases in generated content, job displacement in creative industries, and the environmental costs of intensive computation. Addressing these challenges requires careful dataset curation, robust privacy safeguards, systematic bias mitigation, responsible deployment strategies, and sustained engagement with diverse stakeholders.

## References

- Brooks et al. (2024)
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.
Video generation models as world simulators.
2024.
URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators).
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- CapCut (2024)
CapCut.
Dreamina.
[https://dreamina.capcut.com/ai-tool/home](https://dreamina.capcut.com/ai-tool/home), 2024.
Accessed: 2024.
- Chefer et al. (2023)
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.
Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.
*ACM transactions on Graphics (TOG)*, 42(4):1–10, 2023.
- Chen et al. (2024)
Minghao Chen, Iro Laina, and Andrea Vedaldi.
Training-free layout control with cross-attention guidance.
In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pp. 5343–5353, 2024.
- Choi et al. (2022)
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon.
Perception prioritized training of diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11472–11481, 2022.
- Dalal et al. (2025)
Karan Dalal, Daniel Koceja, Jiarui Xu, Yue Zhao, Shihao Han, Ka Chun Cheung, Jan Kautz, Yejin Choi, Yu Sun, and Xiaolong Wang.
One-minute video generation with test-time training.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 17702–17711, 2025.
- Dong et al. (2022)
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al.
A survey on in-context learning.
*arXiv preprint arXiv:2301.00234*, 2022.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first international conference on machine learning*, 2024.
- Everingham et al. (2010)
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.
The pascal visual object classes (voc) challenge.
*International journal of computer vision*, 88(2):303–338, 2010.
- Feng et al. (2022)
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.
Training-free structured diffusion guidance for compositional text-to-image synthesis.
*arXiv preprint arXiv:2212.05032*, 2022.
- Feng et al. (2023)
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang.
Layoutgpt: Compositional visual planning and generation with large language models.
*Advances in Neural Information Processing Systems*, 36:18225–18250, 2023.
- Feng et al. (2025)
Weixi Feng, Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, and Weili Nie.
Blobgen-vid: Compositional text-to-video generation with blob video representations.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 12989–12998, 2025.
- Gandelsman et al. (2022)
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros.
Test-time training with masked autoencoders.
*Advances in Neural Information Processing Systems*, 35:29374–29385, 2022.
- He et al. (2022)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 16000–16009, 2022.
- He et al. (2025)
Weijie He, Mushui Liu, Yunlong Yu, Zhao Wang, and Chao Wu.
Dyst-xl: Dynamic layout planning and content control for compositional text-to-video generation.
*arXiv preprint arXiv:2504.15032*, 2025.
- He et al. (2023)
Yutong He, Ruslan Salakhutdinov, and J Zico Kolter.
Localized text-to-image generation for free via cross attention control.
*arXiv preprint arXiv:2306.14636*, 2023.
- Hertz et al. (2022)
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
Prompt-to-prompt image editing with cross attention control.
*arXiv preprint arXiv:2208.01626*, 2022.
- Ho et al. (2022)
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.
Video diffusion models.
*Advances in neural information processing systems*, 35:8633–8646, 2022.
- HPC-AI Tech (2024)
HPC-AI Tech.
Open-sora: Democratizing efficient video production for all.
[https://github.com/hpcaitech/Open-Sora](https://github.com/hpcaitech/Open-Sora), 2024.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Huang et al. (2024)
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al.
Vbench: Comprehensive benchmark suite for video generative models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 21807–21818, 2024.
- Kim et al. (2025)
Sunwoo Kim, Minkyu Kim, and Dongmin Park.
Test-time alignment of diffusion models without reward over-optimization.
*arXiv preprint arXiv:2501.05803*, 2025.
- Kong et al. (2024)
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al.
Hunyuanvideo: A systematic framework for large video generative models.
*arXiv preprint arXiv:2412.03603*, 2024.
- Kuaishou (2024)
Kuaishou.
Kling 1.0.
[https://kling.kuaishou.com/](https://kling.kuaishou.com/), 2024.
Accessed: 2024.
- Li et al. (2024)
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang.
T2v-turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.
2024.
URL [https://arxiv.org/abs/2410.05677](https://arxiv.org/abs/2410.05677).
- Li et al. (2025)
Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng.
Test-time preference optimization: On-the-fly alignment via iterative textual feedback.
*arXiv preprint arXiv:2501.12895*, 2025.
- Li et al. (2016)
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou.
Revisiting batch normalization for practical domain adaptation.
*arXiv preprint arXiv:1603.04779*, 2016.
- Lian et al. (2023a)
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.
Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.
*arXiv preprint arXiv:2305.13655*, 2023a.
- Lian et al. (2023b)
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li.
Llm-grounded video diffusion models.
*arXiv preprint arXiv:2309.17444*, 2023b.
- Liang et al. (2020)
Jian Liang, Dapeng Hu, and Jiashi Feng.
Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.
In *International conference on machine learning*, pp. 6028–6039. PMLR, 2020.
- Lin et al. (2024)
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al.
Open-sora plan: Open-source large video generation model.
2024.
URL [https://arxiv.org/abs/2412.00131](https://arxiv.org/abs/2412.00131).
- Lin et al. (2023)
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal.
Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.
*arXiv preprint arXiv:2309.15091*, 2023.
- Lipman et al. (2022)
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.
Flow matching for generative modeling.
*arXiv preprint arXiv:2210.02747*, 2022.
- Liu et al. (2024)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al.
Grounding dino: Marrying dino with grounded pre-training for open-set object detection.
In *European conference on computer vision*, pp. 38–55. Springer, 2024.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Pan et al. (2024)
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar.
T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching.
*arXiv preprint arXiv:2402.14167*, 2024.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 4195–4205, 2023.
- Pika (2024)
Pika.
Pika 1.0.
[https://www.pika.art](https://www.pika.art), 2024.
Accessed: 2024.
- Polyak et al. (2024)
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al.
Movie gen: A cast of media foundation models.
*arXiv preprint arXiv:2410.13720*, 2024.
- Qi et al. (2023)
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.
Fatezero: Fusing attentions for zero-shot text-based video editing.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 15932–15942, 2023.
- Qi et al. (2024)
Zipeng Qi, Guoxi Huang, Chenyang Liu, and Fei Ye.
Layered rendering diffusion model for controllable zero-shot image synthesis.
In *European Conference on Computer Vision*, pp. 426–443. Springer, 2024.
- Qu et al. (2023)
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua.
Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation.
In *Proceedings of the 31st ACM International Conference on Multimedia*, pp. 643–654, 2023.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021.
- Ravi et al. (2024)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al.
Sam 2: Segment anything in images and videos.
*arXiv preprint arXiv:2408.00714*, 2024.
- Ronneberger et al. (2015)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *International Conference on Medical image computing and computer-assisted intervention*, pp. 234–241. Springer, 2015.
- Runway (2024)
Runway.
Introducing gen-3 alpha: A new frontier for video generation.
[https://runwayml.com/blog/introducing-gen-3-alpha/](https://runwayml.com/blog/introducing-gen-3-alpha/), 2024.
Accessed: 2024.
- Schneider et al. (2020)
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.
Improving robustness against common corruptions by covariate shift adaptation.
*Advances in neural information processing systems*, 33:11539–11551, 2020.
- Seawead et al. (2025)
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al.
Seaweed-7b: Cost-effective training of video generation foundation model.
*arXiv preprint arXiv:2504.08685*, 2025.
- Sella et al. (2025)
Etai Sella, Yanir Kleiman, and Hadar Averbuch-Elor.
Instancegen: Image generation with instance-level instructions.
In *Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers*, pp. 1–10, 2025.
- Shu et al. (2021)
Manli Shu, Zuxuan Wu, Micah Goldblum, and Tom Goldstein.
Encoding robustness to image style via adversarial feature perturbations.
*Advances in Neural Information Processing Systems*, 34:28042–28053, 2021.
- Shu et al. (2022)
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao.
Test-time prompt tuning for zero-shot generalization in vision-language models.
*Advances in Neural Information Processing Systems*, 35:14274–14289, 2022.
- Sun et al. (2025)
Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu.
T2v-compbench: A comprehensive benchmark for compositional text-to-video generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 8406–8416, 2025.
- Sun et al. (2020)
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.
Test-time training with self-supervision for generalization under distribution shifts.
In *International conference on machine learning*, pp. 9229–9248. PMLR, 2020.
- Sun et al. (2024)
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al.
Learning to (learn at test time): Rnns with expressive hidden states.
*arXiv preprint arXiv:2407.04620*, 2024.
- Tian et al. (2024)
Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di ZHANG, et al.
Videotetris: Towards compositional text-to-video generation.
*Advances in Neural Information Processing Systems*, 37:29489–29513, 2024.
- Wan et al. (2025)
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al.
Wan: Open and advanced large-scale video generative models.
*arXiv preprint arXiv:2503.20314*, 2025.
- Wang et al. (2023)
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.
Modelscope text-to-video technical report.
2023.
URL [https://arxiv.org/abs/2308.06571](https://arxiv.org/abs/2308.06571).
- Wang et al. (2025a)
Renhao Wang, Yu Sun, Arnuv Tandon, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang.
Test-time training on video streams.
*Journal of Machine Learning Research*, 26(9):1–29, 2025a.
- Wang et al. (2025b)
Shulei Wang, Wang Lin, Hai Huang, Hanting Wang, Sihang Cai, WenKang Han, Tao Jin, Jingyuan Chen, Jiacheng Sun, Jieming Zhu, et al.
Towards transformer-based aligned generation with self-coherence guidance.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 18455–18464, 2025b.
- Wang et al. (2024)
Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li.
Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation.
*arXiv preprint arXiv:2401.15688*, 2024.
- Wu et al. (2024)
Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell.
Self-correcting llm-controlled diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 6327–6336, 2024.
- Xie et al. (2023)
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.
Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 7452–7461, 2023.
- Yang et al. (2024a)
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui.
Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms.
In *Forty-first International Conference on Machine Learning*, 2024a.
- Yang et al. (2024b)
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
*arXiv preprint arXiv:2408.06072*, 2024b.
- Zhang et al. (2024)
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou.
Show-1: Marrying pixel and latent diffusion models for text-to-video generation.
*International Journal of Computer Vision (IJCV)*, pp. 1–15, 2024.
- Zhang et al. (2022)
Marvin Zhang, Sergey Levine, and Chelsea Finn.
Memo: Test time robustness via adaptation and augmentation.
*Advances in neural information processing systems*, 35:38629–38642, 2022.

## Appendix A The Use of Large Language Models

We declare that we used large language models (LLMs) to assist in refining
this manuscript, specifically for grammar checking, language polishing, and
enhancing textual clarity and fluency. We also employed LLMs in a limited
capacity for minor debugging and syntactic corrections of code snippets.

## Appendix B Additional Experimental Results

### B.1 Overall and Quality Evaluation on VBench

Tables [9](https://arxiv.org/html/2510.07940v1#A2.T9) and [10](https://arxiv.org/html/2510.07940v1#A2.T10) report the overall and quality dimension results on VBench. Our method achieves consistently higher scores than both the original backbones and the LVD-enhanced variants, confirming its effectiveness in improving perceptual quality. In the quality dimensions (Table [10](https://arxiv.org/html/2510.07940v1#A2.T10)), our method matches or surpasses the strongest baseline on five dimensions, generating visually sharper and temporally smoother videos. These findings underline that our method yields a better balance between perceptual fidelity and text adherence than all compared baselines.

**Table 9: Overall evaluation results on VBench.**
| Model | Total | Quality | Semantic |
| --- | --- | --- | --- |
| CogVideoX-5B | 0.8201 | 0.8272 | 0.7917 |
| + LVD | 0.7992 | 0.8010 | 0.7921 |
| + Ours | 0.8318 | 0.8314 | 0.8332 |
| Wan2.1-14B | 0.8369 | 0.8559 | 0.7611 |
| + LVD | 0.8106 | 0.8186 | 0.7788 |
| + Ours | 0.8492 | 0.8573 | 0.8166 |

**Table 10: Quality dimension results on VBench.**
| Model | Sub.<br>Cons. | B.g.<br>Cons. | Temp.<br>Flick. | Motion<br>Smooth | Dyn.<br>Deg. | Aesth.<br>Qual. | Imag.<br>Qual. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CogVideoX-5B | 0.9656 | 0.9681 | 0.9853 | 0.9815 | 0.5616 | 0.6207 | 0.6534 |
| + LVD | 0.9532 | 0.9602 | 0.9823 | 0.9830 | 0.4672 | 0.6008 | 0.5782 |
| + Ours | 0.9682 | 0.9773 | 0.9860 | 0.9855 | 0.5426 | 0.6273 | 0.6527 |
| Wan2.1-14B | 0.9752 | 0.9809 | 0.9946 | 0.9830 | 0.6546 | 0.6607 | 0.6943 |
| + LVD | 0.9531 | 0.9719 | 0.9942 | 0.9835 | 0.5476 | 0.6437 | 0.5602 |
| + Ours | 0.9577 | 0.9901 | 0.9952 | 0.9895 | 0.6602 | 0.6619 | 0.6837 |

### B.2 Additional Qualitative Results

Figure: Figure 5: Qualitative comparison between the foundation, the baseline, and our method on T2V-CompBench.
Refer to caption: x5.png

Figure: Figure 6: More Qualitative comparison between the foundation, the baseline, and our method on T2V-CompBench.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: see the weakness
soundness: 3
strengths: - Proposes a novel test-time optimization framework with memory-based reuse, extending inference-time adaptation for video generation in a principled way.  
- Seamlessly integrates LLM-driven spatiotemporal layout planning for controllable, interpretable video synthesis.  
- Achieves consistent quantitative and qualitative gains on major T2V benchmarks, including clear improvements in motion and numeracy.  
- The design supports streaming inference and user-specific memory, aligning well with practical workflows.  
- Comprehensive ablations, detailed comparisons, and visual evaluations substantiate the method’s effectiveness.
summary: This paper introduces **TTOM**, a training-free framework that enhances compositional text-to-video (T2V) generation through *test-time optimization (TTO)* and a *parametric memory* mechanism. TTOM aligns cross-attention maps of video foundation models (VFMs) with LLM-generated spatiotemporal layouts at inference time, improving text–video compositional alignment. Its parametric memory stores optimized parameters from previous prompts, enabling reuse, continual learning, and efficient streaming inference. Experiments on **T2V-CompBench** and **VBench** demonstrate significant improvements in compositional reasoning, semantic consistency, and visual quality over baselines such as CogVideoX‑5B and Wan2.1‑14B.
weaknesses: 1. The method is largely empirical; there is a lack of theoretical explanation or analysis regarding optimization stability and memory dynamics over time.  
2. Evaluation scenarios mainly focus on compositional prompts; testing under more complex, long-horizon, or open-domain conditions would strengthen generalization claims.  
3. Dependence on large LLMs (e.g., GPT‑4o) for layout planning introduces computational and cost overheads that are not thoroughly discussed.  
4. Runtime performance and latency analysis are insufficient; the paper should quantify the cost of test-time optimization steps and memory operations.  
5. Some ablations, while numerically helpful, could benefit from visual examples or error diagnostics to clarify how memory and optimization complement each other.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: **1. Clarification of “training-free” Definition**

The authors must explicitly state what “training-free” means in this context. Since the method involves gradient updates, how does it fundamentally differ from test-time training or test-time adaptation?

**2. Mathematical Modeling and Stability of Memory**

Could the memory insertion, update, and deletion processes be formalized (e.g., via state transition equations) and analyzed for stability or boundedness?

**3. Memory Capacity and Replacement Policies**

When memory reaches capacity, how do different replacement policies (LRU, LFU, FIFO) affect performance, latency, and storage footprint?

**4. Impact of Layout Errors on TTO**

How sensitive is TTOM to LLM-induced layout errors? How is the “verification step” implemented and evaluated quantitatively?

**5. Fault Tolerance and Error Accumulation**

How does TTOM prevent error propagation when mismatched memory entries are loaded or when the retrieved parameters are irrelevant?

**6. Fine-Grained Resource Analysis**

Could the authors provide detailed per-sample latency, GPU-hour costs, and LoRA parameter size during streaming?

**7. Hyperparameter Sensitivity**

Why were LoRA rank=32 and five denoising steps chosen? Are these settings robust across different backbones?

**8. Reproducibility and Randomness Control**

Please specify random seeds, GPT-4o prompt templates, and evaluation scripts for reproducibility.

**9. Failure Cases and Limitations**

What are the main failure modes (e.g., occlusion, excessive object count, ambiguous prompts)?
soundness: 3
strengths: 1. The paper presents an innovative test-time compositional generation framework equipped with a streaming memory mechanism, distinguishing it from conventional per-sample optimization approaches. This enables continual adaptation and historical context reuse during inference.

2. By unifying per-sample test-time optimization with a parametric memory module, the method achieves continual adaptation and efficient context reuse across inference streams, representing a novel direction in test-time generative modeling.

3. Extensive experiments provide strong empirical evidence of its effectiveness, showing performance improvements of +34.45% and +15.83% over CogVideoX-5B and Wan2.1-14B on T2V-CompBench (Table 1), together with notable gains in semantic consistency on VBench.

4. While the idea of “training-free test-time optimization” is conceptually appealing, the misuse of terminology undermines the paper’s scientific rigor. Any process involving gradient updates or parameter optimization cannot be described as training-free. The term must be corrected in the abstract, introduction, and method sections to maintain terminological precision.
summary: This paper proposes a framework named TTOM (Test-Time Optimization and Memorization) for improving compositional text-to-video (T2V) generation. During inference, TTOM aligns video generation results with spatiotemporal layouts and introduces lightweight, optimizable parameters for adaptive per-sample updates. It further incorporates a parameterized memory mechanism that supports insertion, retrieval, update, and deletion operations, enabling the reuse of historical optimization context. Experimental results on T2V-CompBench and VBench demonstrate substantial gains, especially in motion, quantity, and spatial relation metrics.

However, the paper suffers from a serious conceptual confusion. Although TTOM is repeatedly described as a “training-free” framework, the core algorithm explicitly performs gradient-based parameter updates (e.g., optimizing LoRA weights) during inference. Such optimization is, by definition, a form of test-time training or test-time adaptation, not a training-free operation. Therefore, labeling the method as “training-free” is conceptually incorrect and misleading. The authors must clarify this terminology to ensure methodological and scientific accuracy.
weaknesses: **1. Conceptual ambiguity in the “training-free” claim:**

The paper repeatedly claims that TTOM is “training-free,” yet the method explicitly introduces learnable parameters and updates them via gradient descent during inference. Regardless of whether this occurs at test time, such an optimization process constitutes training—albeit without external training data. Hence, “training-free” and “test-time optimization” are not interchangeable concepts.
This is a critical conceptual flaw. The authors should clearly distinguish among:

- training-free: no gradient updates at all;

- fine-tuning-free: no updates to the backbone parameters, but auxiliary modules may be trained;

- test-time optimization: gradient-based updates performed at inference for adaptation.
Since TTOM belongs to the third category, it should be explicitly defined as a test-time optimization–based framework, not a training-free one.

**2. Lack of theoretical analysis of the memory mechanism:**

The insertion and update strategies in Equation (3) lack mathematical formalization and convergence analysis. The paper provides only an implementation-level description, without proving or empirically testing stability, information retention, or forgetting dynamics under continual updates.

**3. Dependence on LLM-generated layouts:**

Section 3.1 heavily relies on GPT-4o for layout generation but lacks robustness evaluation. The paper does not quantify how common layout errors (e.g., bounding-box misalignment, temporal inconsistencies) influence test-time optimization or how the “verification step” mitigates error propagation.

**4. Insufficient evaluation of computational and storage overhead:**

The latency and storage implications of the memory system (especially when full) are not systematically reported. The paper lacks detailed measurements of optimization time, memory growth (LoRA parameter accumulation), and replacement cost under streaming conditions.

**5. Lack of justification for hyperparameter choices:**

The paper sets LoRA rank=32 and optimizes only the first five denoising steps without explaining why these configurations are near-optimal. No sensitivity analysis is provided for LoRA rank, learning rate, or optimization steps, limiting generalization insight.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How strongly does TTOM depend on the LLM’s prompt design, model choice, or layout verification? And what happens when the LLM produces noisy or uncertain spatial temporal layouts, can TTOM still perform reliably?
2. Has the system been tested for cases where outdated or irrelevant memories build up and start to hurt performance? Are there situations where a faulty memory causes negative transfer? i hope the author can provide some comparison between successful and failure cases.
3. as shown in the above weakness part.
soundness: 2
strengths: 1. Clear motivation. the paper addresses a key limitation of video foundation models by improving their compositional understanding of objects, relations, attributes, and temporal dynamics within a realistic streaming generation setting.
2. Parametric memory flexibly manages prompt-level optimization results, enabling efficient transfer and adaptive refinement across prompts.
summary: TTOM is a training-free framework for compositional text-to-video generation. It aligns model attention with LLM-generated spatiotemporal layouts via test-time optimization and stores optimized parameters in a parametric memory for reuse. This design enables continual adaptation across prompts, yielding significant improvements in motion, spatial alignment, and semantic consistency.
weaknesses: 1. Line54 has two "in this paradigm”, typo error.
2. The cases shown in Figure 1, when compared with the baseline, reveal that the baseline also produces outputs that are partially semantically correct and sometimes exhibit higher visual quality — for instance, in the case of “An elderly man walking to the right in a sunny park.”
3. TTOM relies heavily on the layout quality produced by the LLM, assuming that the generated spatiotemporal layouts are accurate and consistent. However, layout hallucinations or spatial inaccuracies in the LLM outputs can directly cause alignment errors, misleading the optimization process and leading to degraded results.
4. There exists a structural trade-off between efficiency and quality.  Continuous TTO introduces significant latency, while memory mechanisms can reduce computational overhead but require a warm-up phase to accumulate sufficient samples.  Consequently, the deployment barrier remains high for online or low-latency applications such as real-time interactive editing.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Please refer to the weaknesses.
soundness: 3
strengths: 1. This paper introduces the TTOM framework, which combines test-time optimization with a parametric memory mechanism, enabling dynamic adjustment of model parameters at inference time and retaining historical optimization results for reuse with similar future prompts.
2. This approach not only improves text-video alignment in compositional scenarios but also supports continuous learning and personalized generation, demonstrating strong practicality and scalability. On mainstream benchmarks such as T2V-CompBench and VBench, the TTOM method significantly outperforms existing approaches across multiple key metrics (e.g., motion, numeracy, spatial relations), achieving overall improvements of up to 34% and 15% on large models like CogVideoX-5B and Wan2.1-14B, respectively, thereby validating its effectiveness and generalization capability.
summary: The paper proposes a method to enhance the alignment capability of compositional text-to-video generation during the inference phase through test-time optimization and a memory mechanism.
weaknesses: 1. The effectiveness of the method heavily relies on the quality of the spatiotemporal layouts generated by the LLM. If the bounding box sequences produced by the LLM are inaccurate, subsequent attention alignment optimization may fail or even introduce misleading guidance, compromising the quality of the generated content. 

2. Test-time optimization requires extra gradient computations and parameter updates, which significantly increase inference latency, especially during continuous optimization and memory retrieval. Although the paper mentions that optimization is applied to the first five denoising steps of the diffusion process to partially mitigate this issue, its acceptability in real-world streaming generation scenarios still requires further validation.


Decision: Accept (Poster)