VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis
Authors: Yumeng Li
William H. Beluch
Margret Keuper
Dan Zhang
Anna Khoreva
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-Video
Diffusion Models
Temporal Attention
Primary area: generative models
TL;DR: VSTAR enables pretrained text-to-video models to generate longer videos with dynamic visual evolution in a single pass, without finetuning needed.

Abstract: Despite tremendous progress in the field of text-to-video (T2V) synthesis, open-sourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to synthesize quasi-static videos, ignoring the necessary visual change-over-time implied in the text prompt. Meanwhile, scaling these models to enable longer, more dynamic video synthesis often remains computationally intractable. To tackle this challenge, we introduce the concept of Generative Temporal Nursing (GTN), where we adjust the generative process on the fly during inference to improve control over the temporal dynamics and enable generation of longer videos. We propose a method for GTN, dubbed VSTAR, which consists of two key ingredients: Video Synopsis Prompting (VSP) and Temporal Attention Regularization (TAR), the latter being our core contribution. Based on a systematic analysis, we discover that the temporal units in pretrained T2V models are crucial to control the video dynamics. Upon this finding, we propose a novel regularization technique to refine the temporal attention, enabling training-free longer video synthesis in a single inference pass. For prompts involving visual progression, we leverage LLMs to generate video synopsis - description of key visual states - based on the original prompt to provide better guidance along the temporal axis. We experimentally showcase the superiority of our method in synthesizing longer, visually appealing videos over open-sourced T2V models.

Published as a conference paper at ICLR 2025
VSTAR: GENERATIVE TEMPORAL NURSING FOR
LONGER DYNAMIC VIDEO SYNTHESIS
Yumeng Li William Beluch² Margret Keuper Dan Zhang? Anna Khoreva
"Amazon Bosch Center for Artificial Intelligence 3 Universityy of Mannheim
MMax Planck Institute for Informatics 7alando
yumengl 16 amazon com, (william beluch, dan. zhang2 @de. basee.bocchcoom com keuper @uni suun-mannheimm de, anna www.kkoorrvaaaaa ando. de
Project page: https://ummeglil0..gtuubb..e ..t..
ABSTRACT
Despite tremendous progress in the field of text-to-video (T2V) synthesis, open- sourced T2V diffusson models struggle generate longer videos with dynamically varying and evolving content. They tend to synthesize quasi-static videos, ignoring the necessary visual change-over-time implied in the text prompt. Meanwhile, scaling these models to enable longer, more dynamic video synthesis often remains computationally intractable. To tackle this challenge, we introduce the concept Generative Temporal Nursing (GTN), where adjust the generative process the fly during inference to improve control over the temporal dynamics and enable generation of longer videos. We propose a method for GTN, dubbed VSTAR, which consists of two key ingredients: Video Synopsis Prompting (VSP) and Temporal Attention Regularization (TAR), the latter being our core contribution. Based a systematic analysis, we discover that the temporal units in pretrained T2V models are crucial control the video dynamics. Upon this finding. we propose a novel regularization technique to refine the temporal attention, enabling training-fres free longer video synthesis in a single inference pass. For prompts involving visual progression, we leverage LLMs to generate video synopsis description of key visual states based on the original prompt to provide better guidance along the temporal axis. We experimentally showcase the superiority of our method in synthesizing longer, visually appealing videos over spurcse T2V models.
"A corgi walking the green grass at sunrise"
TAR VideoCratter2
"A spectacular fireworks display over Sydney Harbour"
VideoCrafee2
YAR
"The process of a lava eruption. from smoke emission lava cooling"
VSTAR VadeoCrafter2
Figure 1: Our Temporal Attention Regularization (TAR) enables longer video generation, e.g., 128 frames, in single pass. When the initial prompt involves progression with multiple stages (3rd row), enabled by Video Synopsis Prompting (VSP), VSTAR can synthesize dynamic visual evolution, i.c. from lava eruption, flowing, cooling. The first column is GIF, best viewed in PDF Reader.
Work done Bosch Center for Artificial Intelligence

---

Published as a conference paper at ICLR 2025
INTRODUCTION
In recent years, rapid advances in the research and open-source community have led to significant progress in text-to-image synthesis, as well as its natural extension to text-to-vided synthesis. Having transformed the idea of content creation, they are now widespread as both a research topic and an industry application. In the realm of text-to-video (T2V) synthesis specifically, recent advancements in video diffusion models (Blattmann et al., 2023; Wang et al., 2023c;b; Guo et al., 2024; Chen et al., 2024a; OpenAI, 2024) have offered promising possibilities for creating novel video content from textual deseriptions.
However, despite these advancements, we observe two common issues in current open-source T2V models (Wang et al., 2023b;e; Guo et al., 2024; Chen et al., 2024a): limited visual changes within the video, and poor ability to generate longer videos with coherent temporal dynamics. More specifically, the synthesized scenes often exhibit a high degree of similarity between frames (see Figs. 1, and S.7), frequently resembling a static image with minor variations as opposed to video with varying and evolving content. Additionally, these models do not generalize well to generate videos with more than the typical 16 frames in one pass (see Fig. 7). While several recent works attempt to generate long videos in sliding window fashion (Wang et al., 2023a; Qiu et al., 2024), these methods not only introduce considerable overhead due to requiring multiple inference passes, but also face the new challenge of preserving temporal coherence throughout the passes.
To mitigate the aforementioned issues, and remain computationally affordable, we propose the "(GTN) - which aims to enhance the temporal
training-free free concept- "Generative Temporal dynamics of longer video synthesis in a single pass, on the fly during inference, without the need for retraining T2V models. As a form of GTN, we propose VSTAR, consisting of Video Synopsis Prompting (VSP) and Temporal Attention Regularization (TAR).
Most soureeourred T2V models, such ModelScope (Wang al., 2023b), Li (Wang et al., 2023c), and VideoCrafter (Chen et al., 2023; 2024a), are built upon T2I models, and process all frames within one batch. The single text prompt is conditioned via cross-attention in the spatial transformer of the UNet and shared by all frames. Different from T2I models with only spatial modules, the newly incorporated temporal attention units in T2V models serve critical component in driving the dynamic aspects of video synthesis, thus has significant impact on the outcome. Therefore, we systematically investigate these temporal units, which are based on temporal transformers consisting of self-attende atffntttenion layers. The visual gap between real videos and synthesized ones leads us to compare their temporal attention maps (see Fig. 3). We discover that real videos have a -aad---trix- like structure, with strong correlation between adjacent frames and weaker correlation distance increases. Intriguingly, the attention maps of the synthesized ones are less structured, explaining their inferior temporal dynamics.
Inspired by this observation, we propose a simple yet effective Temporal Attention Regularization (TAR) technique, which improves the video dynamics of generated videos and enables generation of longer videos (see Figs. and 5). More specifically, we design a symmetric Toeplitz matrix with values along the off-diagonal direction following Gaussian distribution. The standard deviation of this distribution can control the regularization strength, i.c., the visual variation along the temporal dimension. Adding it the existing temporal attention maps strengthens the temporal correlation between adjacent frames, while reducing it between more distant frames. Notably, TAR is readily applicable to pre-trained T2V models and requires no optimization, thus introducing no extra inference overhead. Enabled by TAR, we additionally demonstrate a user-friendly way to transfer dynamics from given reference video to the generated one, by employing its inter-frame similarity as the temporal regularization matrix (see Fig. 8).
When the user input describes scenarios that involve multi-stage changes or high dynamics, as shown in the 3rd row of Fig. 1, T2V models struggle to transform the semantics from single prompt into the required visual change across frames. Prior T2I works (Betker et al., 2023; Yang et al., 2024a) have utilized large language models (LL.Ms) to decompose a single prompt containing multiple entities into several subprompts for spatial compositional generation. Analogously, for dynamic video synthesis faithful to the input prompt, the generation could benefit from synopsis that describes the main events of the video, with explicit deseriptions about the desired visual development over time, As a component of GTN, our Video Synopsis Prompting (VSP) leverages LLMs to generate key visual states, thus providing the T2V model more accurate guidance from the spatial perspective, i.c., via cross attention. To ensure temporal coherency, we propose to interpolate between textual embeddings
2

---

Published as a conference paper at ICLR 2025
of key stages, ensuring smooth conditioning without abrupt changes between frames. Equipped with both strategies, our VSTAR can produce long videos with appealing visual changes in one single inference pass (see Figs. and S.5).
Finally, in addition to improving longer video synthesis, we analyze the temporal attention mecha- nisms of different T2V models, establishing valuable connections between their capability to generate longer videos and their architectures. Following the analysis, we offer several training suggestions for enhancing the generalization ability of future models. In summary, our contributions include:
We introduce a novel concept of "Generative Temporal Nursing", aiming to improve tem- poral dynamics, especially for long videos, without training needed or introducing high computing overhead at inference time.
We propose VSTAR, method for Generative Temporal Nursing, consisting of two simple yet effective strategies: Video Synopsis Prompting and Temporal Attention Regularization, which enable long video generation a single inference pass with improved video dynamics.
We are the first provide an analysis of temporal attention within video diffusion models, and unleash its potential for controlling the video dynamics. Based on the analysis, we provide insights on how to improve the training of the next generation of T2V models.
2 RELATED WORK
Text-to- Video Diffusion Models. Remarkable progress in T2V has been made by industrial research labs (OpenAI, 2024; Kaishou, 2024; AI, 2024; Runway, 2024), however, their models are not publicly accessible. Therefore, we focus on investigating open-sourced T2V diffusion models (Blattmann et al., 2023; Wang et al., 2023b;c; Chen et al., 2023; Guo et al., 2024; Chen et al., 2024a), which are commonly built upon large-scale pretrained T2I model, e.g.. Stable Diffusion (Rombach et al., 2022). Such methods generally introduce a temporal dimension to the T21 model and include temporal transformer for temporal modeling with finetuning on a video dataset. A few recent works, e.g., Open-Sora-Plan (Lab & etc., 2024) and Cog VideoX (Yang et al., 2024b) have attempted to reproduce Sora using 3D full attention directly, without explicitly modeling the temporal dimension (i.e., without using temporal attention modules), which falls outside the scope of our approach.
Since long video generation is especially difficult, there are also works (Qiu et al., 2024; Wang al., 2023a) focusing specifically this application. FreeNoise (Qiu et al., 2024) proposed noise rescheduling combined with local window based attention fusion. Gen-L- -ideeo (Wang et al., 2023a) casts the problem as fusing multiple short video clips with temporal overlapping. However, they necessitate several passes for generation, significantly raising the inference overhead. Some recent works (Kondratyuk et al., 2023: Long et al., 2024; Zhuang et al., 2024) incorporated large-scale training and aim at multi-scent storytelling, allowing significant scene discontinuity, with the final video being a concatenation of multiple runs. Different from these methods, our VSTAR targets coherent long video generation with pretrained T2V model in one single pass.
Attention Manipulation. In the realm T2I models, many works (Cao et al., 2023; Chefer et al., 2023; Li et al., 2023; Hertz al., 2023; Chen et al., 2024b) have identified the attention layers as potential targets to manipulate for improving synthesis. Chefer et al. (2023); Li et al. (2023) employs inference time latent optimization based on the cross-attention maps, enhance faithfulness to the input prompt, e.g., encourage object presence and proper attribute binding. However, such optimization increases the computation cost at inference time. There are also methods (Hertz et al., 2023; Feng et al., 2023) which directly modify or reweight the attention maps to enable text-controlled image editing or improve attribute binding and compositionality. Nonetheless, for T2V diffusion models, there is still lack of comprehensive understanding of the temporal attention mechanism. Our work is the first to investigate this aspect and unleash its manipulation potential for improving video generation of pretrained T2V models without extra optimization overhead during inference.
METHOD
Our concept of Generative Temporal Nursing (GTN) aims at improving the video dynamics of pre- trained T2V diffusion models. Besides the text prompt, identify in Section 3.1 that the temporal attention layer is another key component of T2V models responsible for determining video dynamics. In Section 3.2, we conduct a systematic temporal attention analysis real videos, which leads to
3

---

Published as a conference paper at ICLR 2025
A boy is getting old
Temporal (batch)
Video Synopsis Prompting
lively
ehoorful
man, with subile signa aging
An eidetly man with
21
2
CLIP
CLIP
CLIP
Spanial Conv Spatial Tran.
Temporal dimension
Temporal Attention Regularization
Tomporal Thuu..
Synthesized VSTAR
Figure 2: Method overview. VSTAR consists of two strategies: Video Synopsis Prompting (left) and Temporal Attention Regularization (right), where TAR is the key enabler for long video synthesis. We used color coding to indicate textual embedding interpolation for individual frames.
simple yet effective Temporal Attention Regularization (Section 3.3), encouraging the temporal attention of synthetic videos mimic the attention of real videos. We describe Video Synopsis Prompting in Section 3.4, which expands the initial text prompt for the whole video into several detailed descriptions that control the video progression respectively on different frames.
3.1 PRELIMINARY: TEXT-TO-VIDESS DIFFUSION MODEL
Most open-sourced text-to-video (T2V) diffusion models (Wang et al., 2023b;c; Chen et al., 2023; 2024a) share similar high-level design, even if training strategies and specific implementations vary. Based on the text-to-image (T2I) latent diffusion model, e.g., Stable Diffusion (SD) (Rombach al., 2022), two main changes introduced for video diffusion models: inflating the 2D UNet to a 3D UNet and adding temporal transformers to capture the requisite temporal relationship found between video frames. With the addition of temporal axis to the 2D convolutional kernels of SD, the resulting pseudo-3D convolutional layers can handle the input video latent 2 E xCxx xw where is the number frames and C, H, W represent the channel and spatial dimension of each frame in the latent space, respectively. To generate video of N frames given text prompt, these T2V methods (Wang et al., 2023b;e; Chen et al., 2023; 2024a: Guo et al., 2024) process all N frames within one batch, and simply repeat the same prompt embedding for all frames. Inherently, the provided text prompt is conditioned via cross-attention of the spatial transformer in the UNet. The temporal transformer consists of several self-attention layers that operate along the temporal axis. More specifically, the spatial dimension of the intermediate features is merged into the batch dimension, resulting in shape of (B x h w, N). Since the spatial layers inherited from SD can only handle each frame independently, the temporal attention layers thus play a crucial role for modeling the video dynamies.
3.2 TEMPORAL ATTENTION ANALYSIS
To properly synthesize videos that capture the dynamics conveyed in the input prompt, we delve into the synthesis model itself. An examination of the components new to T2V models, beyond the common building blocks already used in T2I models, leads naturally to the temporal attention layers. These new modules are crucial for facilitating proper video synthesis, i.e., generating sequential frames with dynamic yet consistent content that reflect the input text information. We hypothesize that the ineffectiveness of current T2V models arises from unstructured interactions among frames in the same video within the temporal attention layers. To verify our hypothesis, we conduct a systematic analysis comparing the attention maps of real and synthetic videos. Specifically, the attention map A expressed as:
(QKT Vd
A = Softmar (6 (Q, K)) -Sffmma
ERNNN
(1)
4

---

Published as a conference paper at ICLR 2025
Real
Synthetic
TAR
Real
Synthetic
TAR
(A) fames
(B) 48 fames
Figure 3: Temporal attention visualization of real and synthetic videos and 48 frames using VideoCrafter2. Attention of real videos exhibits band-matrix like structure, indicating high correla- tion with adjacent frames. Synthetic videos exhibit less structured attention maps, especially for 48 frames, which explains the low quality of long video generation. The proposed Temporal Attention Regularization (TAR) can make the attention more structured, resembling the attention of real videos.
Res. 64
Res.
Res. 32
Res. 16
Res.
static dyynami
Original
Figure 4: Per-layer temporal attention analysis. We replace the temporal attention maps different resolutions with diagonal matrix (Ist row) and an all-ones matrix (2nd row), which leads to more dynamic or more static video, respectively. We observe that high resolution attention has larger impact on the video dynamics. Note that this is GIF, best viewed in PDF Reader.
where Q and K represent the query and key of the self-attention layer, and is the latent dimension. This attention matrix essentially depicts the pairwise correlation between the N frames of one video. For real videos, their attention maps can be obtained by adding noise their clean latent and extracting the attention during the denoising process. For synthetic videos, we can read out their attention maps directly during their synthesis passes.
For our analysis, we use VideoCrafter2 (Chen et al., 2024a) along with videos from the DAVIS dataset (Perazzi al., 2016) and additional videos collected from the web. As shown in Fig. for both 16 and 48 frame real videos, the attention matrix forms hand-matrix-like structure, with closer frames showing higher correlation to maintain temporal coherence. Compared to real videos, attention matrix of the synthetic ones is less structured, especially for 48 frames. That explains why the model generalizes even worse to longer videos. High correlation is spread across a wide range of frames, resulting in harmonized sequence with similar appearances.
To understand the effect temporal attention layers at different resolutions, we further conducted a per-resolution ablation shown in Fig. 4. We replace the attention map each individual resolution. i.e. 64, 32, 16, and 8, while keeping the other resolution untouched. We experiment with two extreme cases: using the Identity matrix (IN) and the all-ones matrix (Jx). The former regularizes the frames to be mutually independent, while the latter oppositely requires full correlation, i.c., static sequence. The observations from Fig. are highly consistent. When utilizing In to encourage independence among frames, the temporal coherence of the synthesized frames is indeed compromised. Conversely, employing J. can significantly diminish the video dynamics, leading quasi- static video. This controlled experiment clearly demonstrates how the temporal attention layer impacts the dynamics of the video synthesis model.
Finally, we investigate the effect of the interplay between attention and resolution on the content dynamics of videos. As also shown in Fig. 4, the replacement at the higher resolutions 64 and 32 has more evident effect than at lower resolutions. Applying the changes jointly both resolutions, 64 & 32, further amplifies the effect. In contrast, the videos are much less responsive to the attention replacement resolution 8. Likely, the low resolution features encode high-level semanties, while with higher resolution features there is more capacity for representing varying local details in the scene; such details necessary for reflecting coherent change frames.

---

Published as a conference paper at ICLR 2025
"A polar bear playing drum kit in NYC Times Square"
VideoCrafter2
TAR
VideoCrafter2
TAR
VideoCrafter2
TAR
N 48
N
Figure 5: TAR enables faithful longer video generation without training. VideoCrafter2 exhibits degradation in generating longer sequences, beyond 16 frames. Employing TAR leads to appealing results with improved dynamics. All results generated with single prompt. 1st row is GIF,
Based on these controlled experiments, we can conclude that manipulating temporal attention allows us to alter the video dynamics. i.e., making the visual process either more static or more dynamic. In particular, adjustments higher resolutions, e.g. 64 & 32, are more effective.
3.3 TEMPORAL ATTENTION REGULARIZATION (TAR)
The experiments above highlight the importance of temporal attention layers in determining video dynamics. Naturally, the attention matrices of synthetic videos should be similar to that real videos. Therefore, propose simple regularization technique applied on the temporal attention layers for pretrained T2V model. Note that, our proposal is directly applied to pretrained T2V models without requiring re-training, and incurs no additional optimization costs during inference.
As illustrated in Fig. 3, the attention correlation of the real video resembles a band matrix, with high correlation between neighboring frames and lower correlation the larger the frame offset. To approximate such a structure, we design a symmetric Toeplitz matrix as the regularization matrix AA, with its values along the off- oif-diago direction following the Gaussian distribution:
where i.j E (1,..., N} represent the entry index of the attention regularization map, and a is the standard deviation of the normal distribution. As indicated in Fig. 9, the standard deviation a can control the regularization strength, i.e. larger leading to less visual variations along the temporal dimension. This regularization matrix is then added to the original attention matrix in (I), i.e.
A' + Softmaz (K(Q,K)+ + max (Q, K)] AA).
To balance both terms, we additionally introduce max[o(Q, K)], which weights AA based on maximum the attention matrix &(Q, K). ). As illustrated in Fig. 2, the regularized attention map A' will be inserted back for further processing. After employing TAR, the regularized attention becomes more structured, resembling the attention of real videos as presented Fig. 3.
TAR effectively supports temporal nursing for longer video generation using pretrained T2V models using a single text prompt, as demonstrated in Fig. while also introducing no optimization overhead. VideoCrafter2 (Chen et al., 2024a) by default can generate 16- 16-frame clips, with minor variations, and visual degradation occurs for longer videos. Using the same prompt, TAR can produce longer videos with improved dynamics, closely adhering to the prompt - all without the need for additional training. We find temporal attention analysis to be a powerful tool for understanding the temporal modeling of video diffusion models and leverage it to analyze other T2V models in Section 5. We establish valuable connections to their architecture designs, and provide guidance for the future training of T2V models for long video generation.

---

Published as a conference paper at ICLR 2025
3.4 VIDEO SYNOPSIS PROMPTING (VSP)
When desired video involves gradual visual changes (see 3rd row Fig. and Fig. 7), a single text is often not descriptive enough. Recent T21 works (Betker et al., 2023; Yang et al., 2024a; Wang et al., 2024a) leverage the in-context learn- ing ability of LLMs for region-wise prompting and compositional generation. We similarly explore LLMs for Video Synopsis Prompting (VSP), to provide key visual states to decom- pose the transition along the temporal axis, as shown in Fig. 6. More implementation details can be found in Appendix C. It is sufficient to generate text descriptions for the main event changes in a video, rather than for each frame. text encoder e.g., CLIP text decoder (Radford al., 2021), is then applied to extract the text embeddings of these descriptions, which are then interpolated to guide each frame's 's synthesis via cross attention as illustrated in Fig. 2. This process yields more accurate guidance for transitioning visual stages while ensuring smooth conditioning without abrupt changes between frames. VSP is simple solution that can introduce more variation along the temporal axis, yet it alone cannot tackle the challenge of longer dynamic video synthesis, as seen in Fig. 7. Combined with TAR, which rectifies the temporal attention, our VSTAR is capable of synthesizing long sequences with appealing visual evolutions.
have prompt "A landscape transitioning from winter to spring". Can you split and describe visual states separately?
Winter's End: The landscape is covered in thick layer of snow, with bare trees.
2. Thaw's 's First Touch: Snow begins to melt, revealing
patches of ground and the bases of trees.
Spring's Prelude: Early signs of greenery emerge as plants start to sprout and trees bud.
Spring's Arrival: Flowers bloom and trees are adomedd with fresh leaves, adding color the landscape.
Spring's Full Bloom: The landscape is fully transdormeed
vibrant with lush greenery and blooming florn.
Figure 6: An illustration example of VSP. With LLMs, we can obtain more detailed descriptions for key stages.
4 EXPERIMENTS
Experimental setting. To demonstrate the effectiveness of VSTAR in creating more dynamic videos, we run experiments and ablations on ChronoMagic-Bench--155 (Yuan et al., 2024) and prompts generated by ChatGPT (OpenAI, 2022) describing various visual transitions. These prompts are provided in the Supp. Material. We use CLIP-VL score for measuring alignment with the text prompt. Additionally, we employed the recent Metamorphic Score (MTScore) (Yuan et al., 2024) to measure the metamorphic amplitude of videos, where higher MTSccree indicates better video dynamics. More details can be found in Appendix A.3. By default, we employ the T2V model VideoCrafter2 (Chen al., 2024a) with x 12 resolution our base model, which is combined with the proposed video synopsis prompting (VSP) and temporal attention regularization (TAR). We refer this combination as our method or VSTAR throughout the experiments.
open-sourced
4.1 MAIN RESULTS
Comparison with other T2V methods. In Table 1, we provided quantitative comparison between our VSTAR and SoTA T2V methods including ModelScope (Wang et al., 2023b), La (Wang et al., 2023c), VideoCrafter2 (Chen et al.. 2024a), Open-Sora-Pee
V1.2 (Lab & etc., 2024) and Vi 5B (Yang et al., 2024b), which requires a single inference pass, alongside FreeBloom (Huang et al., 2023) and FreeNoise (Qiu et al., 2024), which utilize multiple passes, i.e., fusion of multiple inference runs. For fair comparison, we adapted the first three models to support multi-prompt generation via VSP However, this is not feasible with Open-Sora-Pinn and VideoX, as they do not explicitly model the temporal dimension. For multi-pass methods, different prompts from VSP are naturally used in different inference runs. It can be seen that, our VSTAR achieves the best alignment with the text prompt, and more importantly, outperforms the others on MTScore by large margin, demonstrating our advantage in generating dynamic visual evolution. Notably, enabled by TAR, our VSTAR can generate long videos in a single inference pass, without facing the challenges of maintaining the coherency of different passes.
We provide qualitative comparisons in Figs. S.7 and S.8 for visual evolution synthesis. Although all methods generate meaningful results for 16-frame videos (see Figs. S.7 and S.8), the videos created by the other T2V models cannot properly reflect the visual content specified by the input prompts, i... lacking dynamic progression throughout the video. When generating 32 frames in one pass, as shown in Fig. 7, our method exhibits even greater advantages. The comparison methods yet again fail to produce the desired content, but this time to the extent that the visual quality of the individual frames
We used the official implementation of FreeNoise which only supports two subprompts.
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison with SoTA T2V methods. For fair comparison, we adapted other methods to use Video Synopsis Prompting (VSP) when compatible. VSTAR outperforms the others, showing its strength in generating dynamic visual evolution, notably in single inference pass.
ModelScope LaVie V.Crafter2 Open- Sora-Sora-iee Plan CogVideoX FreeBloom FreeNoise VSTAR
VSP
X
Single-Pass
/
/
/
/
X
X
CLIP-VIL VLT
0.221 0.322
0.221
0.220
0.220
0.221
0.219
0.224
0.233
MT-Scoref
0.306
0.401
0.239
0.312
0.361
0.379
0.448
"A landscape transitioning from winter spring"
LaVie Mode'Scope
V.Calter22 AnimateDin
VSTAR FroeNoive
Figure 7: Comparison with other T2V methods on frames generation, which is double the length of the default option. Our VSTAR can generate long videos with desired dynamics, while the others struggle to synthesize faithful results with visual progression using the same multi-prompt of VSP.
is also greatly compromised. In contrast, our VSTAR is able to generate longer videos with dynamic visual evolution, which is aligned with the quantitative evaluation Table 1. More visual examples of VSTAR are presented in Appendix A.4, and qualitative results of Open-Sora-Plam and Cog VideoX are provided the Supp. Material. We also conducted a user study, detailed in Appendix A.2, where VSTAR clearly emerged as the preferred option.
Transferring dynamics from real videos via TAR Instead of using the predefined matrix formulated in Eq. (2), we explore transferring dynamics from reference video. by employing DreamSim (Fu et al., 2023) based similarity matrix as AA for TAR. DreamSim is a recently proposed visual similarity metric that has been demonstrated align closely with human judgment. In the 1st row of Fig. the inter-frame DreamSim similarity of the real video closely mirrors the temporal attention maps of the real videos, indicating that it well-suited for TAR. In the 2nd and 3rd rows, we compare the results before and after applying DreamSim of the real video provided in the 1st row for TAR, using the same text prompt and random seed. As shown, after employing TAR with the DreamSim matrix, video dynamics of the rainbow. Mcanwhile, the inter-frame similarity matrix of the synthesized video, shown on the right, aligns more closely with that of the real video. This demonstrates that dynamics from reference video can be effectively transferred a user-friendly manner to improve the generated output. Additionally, use DreamSim for quantitative comparison T2V models in Appendix A.
noticeably enhanced, resembling the reference video. i.e., the gradual appearance
4.2 ABLATION STUDY
Ablation on the effect of TAR and VSP. We investigate the effects of the proposed Tem- poral Attention Regularization and Video Synopsis Prompting quantitatively in Table 2 us- ing ChronoMagic-Benchh 150. Employing VSP alone improves the CLIP-VL score slightly,
8

---

Published as a conference paper at ICLR 2025
"Rainbow starts to appear after the rainy day"
DreamSim
Reference
Buseline
DreamSim
Figure 8: Transferring dynamics from real reference video (Ist row) via employing its inter- DreamSim matrix TAR enhances the output's dynamics as desired (3rd row).
"A peony starts bloom, in the field"
Nonr
4
0b4 1
Ogg
03g 1
Figure 9: Ablation of attention regularization matrix AA. Smaller induces stronger regularization effect, leading to increasing temporal dynamics. When applying regularization at both 64 32, the video becomes more dynamic, i.e., the peony is fully bloomed. 1st row is GIF,
Table 2: Ablation on VSP and TAR.
but brings little help on introducing more visual changes, ie., minor difference MTScore. This is aligned with our qualitative results in Appendix A.5. Despite VSP providing more descriptive summary of different vi- sual states, without TAR, the temporal attention remains strongly correlated, yielding limited visual dynamics. When combining both strategies, our VSTAR can effectively synthesize the desired visual content, exhibiting improved dynamics with a more appealing time-lapse effect. For prompts that do require gradual visual appearance changes, VSP is not necessary. However, TAR still plays crucial role in enabling high quality longer video generation, as shown Figs. and 5.
CLIP-VLI MT-Scoret
Baseline VSP
0.214
0.397
0.220
0.401
VSP & TAR
0.233
0.448
Ablation regularization matrix. We further ablate by investigating the effect of using different standard deviation a in the regularization matrix AA, shown in Fig. 9. We start from applying regularization at the highest temporal resolution i.e., 64, since high-resolution temporal attention more greatly influences the video dynamics, as demonstrated in the temporal attention analysis in Section 3.2. The results show that decreasing o results in a stronger regularization effect, inducing more pronounced visual changes throughout the video (c.g. compare row 2 to row 4, and notice the extent of the blooming of the flower). Going one step further, applying regularization also at resolution of 32 results in the peony reaching its fullest bloom. However, when equally strong regularization is applied at both a resolution of and 32, i.e., des = 022 = 1, the visual changes can be too excessive, leaving the impression of poor temporal coherency across frames. Empirically, find that applying σ&4 = strikes good balance between dynamic changes and temporal coherency, which is our default option.

---

Published as a conference paper at ICLR 2025
ModelScope
LaVie
AnimateDDiff
16 32
16 N 32 = 16 32
Figure 10: Temporal attention visualization of other T2V Models for the default 16 frame and longer 32 frame videos. ModelScope faces similar issues VideoCrafter2 (see Fig. 3), with high correlatio spread across many frames, especially for N = 32. LaVie and AnimateDiff, using frame index positional encoding, struggle to generalize beyond the 16-frame limit they were trained on.
DISCUSSION
VSTAR offers simple yet effective vaining-frree solution for longer dynamic video synthesis. Yet, certain limitations of the pretrained T2V models, e.g., poor understanding of long-range motion, can be further addressed during training. Therefore, we next analyze the temporal attention units of various T2V models to offer insights for improving architecture design and training strategies.
In Fig. 10, we visualize the temporal attention layers of ModelScope (Wang al., 2023b), LaVie (Wang et al., 2023c) and AnimateDiff (Guo et al., 2024). It can be seen that ModelScope ex- hibits similar attention behavior VideoCrafter2 (see Fig. 3), with temporal correlation deteriorating in longer videos, even at 32 frames, i.e., twice the standard length. This aligns with the qualitative comparison in Fig. In Appendix A.6, we show that VSTAR can also improve ModelScope's 's long video generation. AnimateDiff and La Vie demonstrate different temporal attention behavior, due to the incorporation of Rotary Positional Encoding (Touvron et al., 2023) in the former and Sinusoidal Positional Encoding in the latter. With the positional encoding, the models learn better temporal correlation among neighboring frames for 16 frames, showing a band-matrix structure more closely resembles that of real videos. However, when generating videos longer than its training capacity, the model faces considerable difficulty in preserving the desired temporal dynamics, resulting in inferior synthesis quality, depicted in Fig. 7. The Rotary Positional Encoding employed in LaVie is form of relative positional encoding, i.e., depends on the relative offsets of frames, which could explain the periodic pattern seen in the attention maps. While the Sinusoidal Positional Encoding used in AnimateDiff' is based on the absolute frame index, leading to the model failing completely for indices unseen during training (past 16). Transformer-based models often rely on positional encodings, which largely constrain their generalization ability for longer video synthesis. These observations concerning T2V models are interestingly aligned with prior studies regarding Positional Encoding on length generalization in Transformers (Kazemnejad et al., 2023) in the context of LLMss.
This comparison offers valuable insights into improving the training of the next generation of T2V models. For instance, omitting positional encoding can improve generalization capability, and incorporating a regularization loss on the temporal attention maps can help to enforce the desired temporal dynamics. Alternatively, one can employ a better combination of data format and positional encodings, as explored in the recent work (Zhou et al., 2024), which achieves improved length generalization. For instance, Randomized Positional Encoding (Ruoss et al., 2023) can help avoid overfitting on the position indices, and mixing up subsampled video sequences can further strengthen local correlations. Combining such techniques with our VSP and TAR may further improve long video generation with better long-range understanding.
CONCLUSION
In this paper, we propose a training-free Generative Temporal Nursing method VSTAR that enables the generation of longer (e.g. 32 ~ 128 frames) and temporally coherent appealing videos in a single inference pass. As our core contribution, Temporal Attention Regularization can effectively improve the temporal modeling of pretrained T2V models, especially for longer video synthesis. Additionally, TAR offers flexible control over video dynamics. Besides motivating TAR, our analysis of temporal correlation in real videos offers valuable insights into improving the design and training of the next generation of T2V models. For example, some form of positional encoding appears to hampering generalization capability, while the incorporation of a regularization loss on temporal attention maps can help to enforce temporal dynamics. While VSTAR is readily applied to pretrained T2V models, future work may incorporate it during training for improved procedural dynamics, such as complex activities on respective data.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We have carefully read the ICLR 2025 code of ethics and confirm that we adhere to it. The proposed method training-free and built upon pretrained T2V models, thereby no further privacy concerns should arise. Nevertheless, given the imbalanced nature of large-scale datasets used to train the original T2V models, these pretrained models may inherit certain data biases, inaccurately representing the diversity of the overall population. These biases can potentially reinforce existing societal stereotypes and inequalities. Therefore, it is advisable to undertake proactive steps to identify and mitigate such biases, which may include the involvement of human reviewers in sensitive contexts.
REPRODUCIBILITY STATEMENT
Regarding reproducibility, implementation and experiments based on publicly available models Wang et al. (2023c;b); Guo et al. (2024); Chen et al. (2024a); Yang et al. (2024b); Lab etc. (2024). Details on the experimental settings are given the beginning of Section and Appendix A. We plan to release the code upon acceptance.
REFERENCES
Luma AI. Dreammachine. https:://// / umalabs attesaaaa///eerr het//deea-machhine 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cd.. openai. com/papers/dall-e33 pdf. 2023. 7
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 2,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. 22
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465. 2023. 3
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-Excee Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023. 3, 23
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.195 2023. 2, 3,4,
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024a. 2, 3, 4, 5, 6, 7, 11, 14, 20
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV, 2024b.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. ICLR, 2023.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023. 8, 14, 15
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR. 2024. 2, 3, 4, 10, 11
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-or. Prompt- to-prompt image editing with attsetttt control. In ICLR, 2023. 3
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah Smith, and Mari Ostendorf. In-context learning for few-shot dialogue state tracking. In EMNLP, 2022. 22
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. NeurIPS, 2023.
Kaishou. Kling. https: / /igii/iiiigg.g//./ kua shou. www.coo////o//oo en, 2024. 3
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan, Payel Das, and Siva Reddy. The impact of positional encoding length generalization in transformers. In NewrIPS, 2023. 10
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint 2312 2023. 3
etc. Open-sora-plan. PKU-Yuan Lab and Tuzhan Al
https://githin
PKU- YuanGruorrooppeen
2024. 3, 7, 11
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. In BMVC, 2023. 3, 23
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In CVPR, 2024. 23
Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Videodrafter: Content-consisten multi-scene video generation with Ilm. arXiv preprint arXiv:2401.0/256, 2024. 3
Ze Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang. Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffusion. arXiv preprint 093866, 2024. 22
OpenAl. Introducing ChatGPT. httpps://openaal.c//////////////
2022. 7,22
OpenAl. Sora: Video generation models as world simulators. hitpps://openal com/ senearchhviideo-geeeeaae wwweiiooooooosssaoeeeeeeeee modelss-ssworrle www.dw world-similators, 2024. 2,3
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkine-Horuung. benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016. 5, 14, 23
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.77 23
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. ICLR, 2024. 2, 3,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021. 7, 23
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 3,
Runway. Gen-3 alpha. https: htatps:/rrunwayml. ht/////wwwwllmm/// wwtp:llccccaiiiiaaeeaaooee /genl////gnnneeeee 3- a pha/, 2024. 3
12

---

Published as a conference paper at ICLR 2025
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint wrXi::2305.68845 2023. 10
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.3399 2023. 10
Fu-Yun Yun Wang. Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint (tt.:2305.18644 2023a. 3
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:3308.0557,, 2023b. 3, 4, 7, 10, 11,
20
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffu- sion: Instance-level control for image generation. In CVPR, 2024a. 7
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu. Peiqing Yang, al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023c. 2, 3, 4, 7, 10, 11
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, al. Internvide02: Scaling video foundation models for multimodal video understanding, arXiv preprint 2033 2024b. 17
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. arXiv preprint arXiv:2401. /708, 2024a. 7
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408. 06072, 2024b. 3, 7,
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic- A benchmark for metamorphic evaluation of sox-to-time-lapse video generation. arXiv preprint (xXXv:2406.18222 2024. 16,
17
Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Trans- formers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09371, 2024.
10
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang. Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream a vlog. In CVPR, 2024. 3
13

---

Published as a conference paper at ICLR 2025
VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis Supplementary Material
This supplementary material to the main paper is structured as follows:
In Appendix A, we provide more experimental results, including quantitative comparisons, user study, additional visual results, ablation the effect of TAR and VSP, and the combination of VSTAR and ModelScope. We also include more details on the MTScore evaluation.
In Appendix B, we discuss our attempts and insights into optimization- temporal generative nursing, which might spark interest for subsequent studies.
In Appendix C, we elaborate further on Video Synopsis Prompting.. e.g., how to instruct LLMs to generate the video synopsis.
In Appendix D, we provide additional visualizations of the regularization matrix and samples of collected real dynamic videos.
In Appendix E, we discuss limitations of the proposed method.
In wotp.o/oo/
Supp, we provide more visuals including:
Qualitative results SOTA T2V models, i.e., Open-Sora-Plan- V1.2 and Cog
GIF version of our VSTAR's qualitative results.
Effect of TAR on longer video generation.
Qualitative comparison with multi-pass methods.
Additional ablation results of attention regularization matrix.
Samples of real videos.
A MORE EXPERIMENTAL RESULTS
A.1 QUANTITATIVE COMPARISON
We quantitatively compute the similarity of two frames certain interval apart using the recent perceptual similarity metric DreamSim (Fu et al., 2023). In Fig. S.1, we plot the similarity matrices of real videos from DAVIS (Perazzi et al., 2016) and web sources, alongside those synthesized by VideoCrafter2 and our VSTAR. The values in the matrix are normalized across all methods. VideoCrafter2 has overall high similarity across all frames, which is aligned with the visual outcomes containing limited visual variations. In contrast, our VSTAR mirrors the perceptual similarity patterns of real videos, demonstrating the effectiveness of our approach in enhancing video dynamics. The similarity distribution of is presented in Fig. S.2. For desired video dynamics, the similarity should decrease as the interval between frames increases, signaling a steady visual evolution. Meanwhile, frames that are closer together should exhibit higher similarity compared to those further apart, indicating preserved temporal coherence. The distribution exhibited by VideoCrafter2 highly concentrated at the high similarity region, even with large intervals. This can be explained by the fact that it often generates videos with limited visual variation over time, which aligned with the qualitative results and analysis in Sec. 4.1 of main paper. In contrast, with increasing interval between frames, the distribution for both our VSTAR and real dynamic videos slightly shifts towards region of lower similarity, indicating that more visual variation has been introduced within the video. Our distribution extensively overlaps with that of real videos, suggesting that our results not only exhibit improved temporal dynamics, but also maintain the continuity.
A.2 USER STUDY
For further evaluation, we conducted user study to compare our VSTAR with the SOTA T2V model VideoCrafter2 (Chen et al., 2024a). 110 individuals with diverse backgrounds participated in the user
14

---

Published as a conference paper at ICLR 2025
Real
VideoCrafter2
Ours
105
0775
Inter- frame perceptual similarity matrix based on DreamSim (Fu et al., 2023), where values are normalized across all methods. VideoCrafter2 has high similarity across nearly all frames, which is aligned with the visual results lacking variation. In contrast, our synthesized videos highly resemble the real ones, indicating desired dynamics.
Figure S.1:
Sinilarit Setware franci andi4)
Sinilarit Issween and
Siniltmit bettean
Sintlarity betwars and
0,40
0.30
0.25
0,30
0.20
0.30
0.20
0.10
0.10
0.00
0.00
0.00
0.00
Similanit
Similarity
Similarity
Simiknity
Figure S.2: Comparison of DreamSim Similarity Distribution between real videos, VideoCrafter and our VSTAR. For preferred video dynamics, the similarity should decrease as the interval between frames increases, signaling a steady visual evolution. Mcanwhile frames that are closer together should exhibit higher similarity compared to those further apart, indicating sustained temporal coherence. For VideoCrafter2, the majority the interval, indicating that there is limited visual variation within the video. The distribution of ours overlaps significantly with that of real videos, suggesting improved video dynamics without compromising coherency.
located in the high similarity region regardless of
study, working in fields such computer vision, reinforcement learning, natural language processing, art design, medical engineering, mechanical engineering, and administrative management, among others. We assess the videos across four dimensions: text alignment, video dynamics, visual quality and temporal coherency. Text alignment concerns whether the synthesized results properly reflect the input text prompt. Video dynamics examines the dynamic visual changes within the progression of the video. A higher visual quality indicates fewer artifacts and distortions, leading to a more visually pleasing result. Temporal coherency evaluates if the result is temporally smooth, i.e., there abrupt or unexplained changes that could disrupt the viewing experience. For the first three aspects, participants are presented with paired results evaluate, selecting one the other deeming them equivalent. Regarding temporal coherency, we pose simple yes-or-no question, asking whether the participants perceive the video being temporally smooth.
The outcome is summarized in Fig. S.3. Our VSTAR emerges as the preferred choice across various frame lengths from all aspects, with its advantages becoming more pronounced in the generation of longer videos with N = 32 ~ 64. Importantly, our method not only enhances video dynamics but also preserves temporal coherency. majority of participants confirmed that our results exhibit smooth temporal transitions, with 87.6% for standard-length videos and 79.1% 1% for longer videos agreeing this assessment. This favorable reception surpasses the baseline VideoCrafter2, possibly result of its less engaging content.
Additionally, we included pairs of videos, both generated by VSTAR, to verify the consistency of our method's improvement, making it challenging for users to make clear choice. As shown in Fig. S.4, participants indeed often found it difficult to differentiate, with 52.7%. 40.3% and 50.9% of them rating both videos as equal in terms of text alignment, visual dynamics, and visual quality, respectively. The remaining participants were divided their preference between the two videos. This indicates that our synthesis results consistent and display narrow gap between them.
15

---

Published as a conference paper at ICLR 2025
16
VideoCrafier2
VSTAR (Ouns)
90.9%
52.4% Yes
56.55
87 ..6%%
Yes
12.4% No
47.6%
4.1%
6.6%
Text Alignment
Video Dynamics
Visual Quality
Temporal Coherency
(Ours)
N = 32 64
VideoCrafter2
VSTAR (Ouns
77.2%
40.4% Yes
79. 15 Yes
20.9% No
59
18.6%
0.7%
1.7%
5.7%
Alignmen
Quality
Temporal Coherency
VideoCrafter2
VSTAR (Ours)
Same
Figure S.3: User study on both standard 16 frames and longer videos with 32 64 frames. For the first three aspects, participants review pairs of videos, choosing between them or rating them as the same. For temporal coherency, the numbers are the absolute probability that a participant perceives the video from the respective method having smooth temporal progression.
52.7%
50.9%
4003%
Text Alignment
Video Dynamics
Visual Quality
Figure S.4: User study on paired of videos, both generated by our VSTAR, to verify the consistency of our method's improvement, making it challenging for users to make clear choice. Indeed, large number of participants perceived both videos as identical across all three aspects. The rest had diverse preferences between the two videos. This demonstrates the consistency of our synthesis results and their closely matched quality.
A.3 MTSCORE EVALUATION
Metamorphic Score (MTScore) (Yuan al., 2024) a newly proposed evaluation metric designed to assess the metamorphic amplitude within generated videos, specifically measuring the extent and degree of visual transformations over time. Previous metrics such as CLIP related scores focus on the vision-language alignment and temporal coherence, however, they overlook the crucial aspect of visual evolution within generated videos. For instance, prior methods often tend to generate quasi-static content with minimal visual change across frames, which cannot be properly reflected CLIP scores. MTScore, therefore, addresses this gap by directly measuring the degree visual metamorphosis leveraging the video retrieval model.
16

---

Published as a conference paper at ICLR 2025
"Superman flying in the sky, sunny day becomes a dark rainy day". N 48
"A young boy becomes an old man, and turns into a young girl"
"A day the beach from dawn till dusk, bird's-eye view" N 64
"The seasonal cycle of lake from frozen winter to autumn" N = 641
Figure S.5: Qualitative results videos with 48 and frames synthesized by VSTAR. Images are sub-sampled from the sequence. GIF version is provided in G.2.
More specifically, Yuan et al. (2024) designed ten retrieval texts to differentiate between progressing and normal videos, e.g., "A normal video, not time-lapse video" and "A time-condensed video, not a conventional video". Using multiple retrieval sentences enhances the evaluation robustness and accuracy. These text prompts, alongside the videos, fed into video retrieval model, i.e., Intern Video2 (Wang et al., 2024b), which yields the probabilities for being categorized into metamorphic or general videos. Therefore, a higher MTScore indicates the video demonstrates more pronounced and dynamic transformation over time, effectively capturing visual evolution. As shown in Table 1, our VSTAR outperforms the other SOTA T2V models on MT'Score by a large margin, highlighting its strength in synthesizing dynamic videos.
A.4 MORE VISUAL RESULTS
More qualitative visual results are provided in Figs. S.5 and S.6, in which the length of videos varies from the default 16 frames longer ones with 64 frames. Intriguingly, all videos generated in
17

---

Published as a conference paper at ICLR 2025
"A mural being painted city wall""
"The process woman losing 50kg of weight"
"A caterpillar transforming into a butterfly"
"A female person's 's hairstyle changing through the years"
"A pizza is being made"
Figure S.6: Qualitative results of videos with 16 frames synthesized by VSTAR. Images are sub- sampled from the sequence. GIF version is provided in G.2.
"A Ferrari driving on the road, starts to snow"
ModatSoope
LaVie
AnimareDiff
Ours V.Crafte22
Figure S.7: Comparison with other T2V models on 16 frames generation. Our VSTAR can synthesize desired visual development from a clear day snowy scene, while the others tend to generate the final visual state, i.e., snowy day. GIF version is provided G.4.
single pass using VSTAR. Additional results the comparison with other T2V methods be found in Fig. S.8. can be seen that our VSTAR consistently outperforms the other T2V models, demonstrating better dynamics with more visual changes time complying with the text prompt.
18

---

Published as a conference paper at ICLR 2025
"A flower starts to bloom"
ModelSeape
*
LaVie
Animate
V.Cafler2
MIL
Ours
"A young girl is aging"
Model iSoppe
tucck
sck
ack
tock
LaVi
AnimateDiff
V.Crafter2
Ours
Figure S.8: Comparison with other T2V models on 16 frames generation. Our VSTAR consistently demonstrates improved video dynamics, resulting more visually appealing content compared to the other methods. GIF version provided in G.4.
A.5 ABLATION ON THE EFFECT OF TAR AND VSP
We investigate the effects of the proposed Temporal Attention Regularization and Video Synopsis Prompting individually for dynamic visual evolution generation Fig. S.9, where we generate videos of 48 frames in pass based on the prompt "Spiderman the beach from morning to evening", using the same initial noise. The synthesized video clips are presented in the first column as GIFs; the other images subsampled from the full sequence. The baseline model VideoCrafter2 struggles to synthesize a video faithful to the input prompt, generating a sequence highly similar frames, with stride-like texture in the background, that fail to depict the time-lapse video. When employing the
19

---

Published as a conference paper at ICLR 2025
"Spiderman on the beach from morning to evening"
Baseline
+TAR
VSP
Ours
Figure S.9: Ablation on the effect of Video Synopsis Prompting (VSP) and Temporal Attention Regularization (TAR). Subsampled from 48 frames. Combination of TAR and VSP effectively enables long video generation with desired visual evolution. While individual strategy improves upon the baseline, there still lacks of desired dynamics.
Baseline
+ VSP
+ VSTAR
Figure S.10: Comparison of temporal attention maps between using and without using TAR. It can be seen that using VSP alone does little to improve the temporal attention structure while employing TAR is more effective.
TAR, the model generates more realistic sequence, however without the desired visual evolution; the single plain prompt is insufficient to describe the scene changes. Interestingly, while VSP provides more descriptive summary of different visual states, without TAR, the temporal attention remains strongly correlated. The model then attempts to depict the provided textual description, however with limited visual variation. When combining both strategies, our VSTAR can effectively synthesize the desired visual content, exhibiting improved dynamics with more appealing time-lapse effect. In Fig. S.10. we compare the temporal attention maps when using different strategies. It can be seen that using VSP alone does little improve the temporal attention structure. In contrast, when combined with TAR, the temporal attention becomes more structured, and the synthesized results align with the desired content. This highlights the crucial role of TAR in generating long dynamic videos.
For the input text prompts that do not require gradual visual appearance changes within the video, VSP is not necessary. However, TAR still plays crucial role enabling high quality longer video generation. In Fig. 5 and G.3, we provide visual comparison without and with TAR for longer video generation, employing a single text prompt without employing VSP.
A.6 COMBINATION OF VSTAR AND MODELSCOPE
In the main paper, we by default apply proposed VSTAR with state-of-the- open-sourced T2V model VideoCrafter2 (Chen et al., 2024a). Nonetheless, VSTAR can also be combined with other pretrained T2V models to enhance their video dynamics. In Fig. S.11, we showcase that VSTAR can boost the long video generation ability of pretrained ModelScope (Wang et al., 2023b), resulting in
20

---

Published as a conference paper at ICLR 2025
"The flower starts to bloom"
ModelSeape
+VSTAR
"A landscape transitioning from winter to spring"
+VSTAR ModelSoppe
Figure S.11: Combination of ModelScope with VSTAR on 32 frames generation, which is double the length of the default option. The same random seed is used. ModelScope cannot generalize well to unseen frames, as discussed in Sec. 4.1. Applying our VSTAR can significantly boost its generalization ability without fine-tuning required.
"A day from sunrise sunset on the beach"
Before
Afler
"The flower is blooming"
Before
After
Figure S.12: Comparison before and after applying initial noise optimization for 16 frames generation using the same single prompt and random seed, After optimization, the video dynamics has been enhanced, i.e., more visual variation has been introduced over time. Note that the first column is GIF, best viewed in Acrobat Reader.
better visual quality and video dynamics. However, due to the constrained capacity of the base model ModelScope, the overall synthesis results underperform those achieved by combining VSTAR with VideoCrafter2 shown in the main paper.
21

---

Published as a conference paper at ICLR 2025
B OPTIMIZATION-BASER GENERATIVE TEMPORAL NURSING
As a method of generative temporal nursing (GTN), our VSTAR completely training- and optimization-free and can be readily applied to frozen pretrained T2V models without introducing inference time overhead. Additionally, we explored optimization- reference video is available to guide the learning of desired dynamics. Inspired by the temporal attention analysis detailed in Sec. 3.3, we attempt to optimize the initial noise latents at inference time align the attention maps of the given real video and the synthesized one, as outlined below.
GTN, assuming that a real
Following Ma et al. (2024), we parameterize the initial video latents of frames with a Multivariate Gaussian distribution (1, (v, 7))), where EN (3, 7) denotes the covariance matrix:
8
y²
N-1
7
8
?
N-2
Σn(r)
72
Y
B
N-3
(5)
N
7
B
Given a real reference video, we can add noise to its clean latent and extract temporal attention maps ATo/ from its denoising process at the timestep f. Then, we perform initial noise optimization to match the temporal attention maps A, during the synthesis process with that of the reference ones:
A A.
L
Furthermore, to prevent the initial noise from deviating significantly from the Gaussian Distribution, we minimize the Kullback-I.ee eeibler divergence between the optimized latents and the standard Gaussian Distribution:
LKL KL(N( EIIA
The joint optimization loss is weighted sum of both loss terms:
min Lijoint min Lalt min Latta + MLL
(8)
M.8.7 p.3.7
where À is weighting factor.
As shown in Fig. S.12, after applying the initial noise optimization, the temporal dynamics of synthesized results from the same single prompt have noticeably improved, with more visual changes occurring throughout the video's progression. However, this optimization-based technique increases the inference time and demands more memory, making it challenging to scale for longer videos. In this regard, VSTAR stands out more scalable and efficient, demonstrating its capability for facilitating long video generation. Overall, we can see both approaches highlight that regularizing the temporal attention is effective solution, suggesting that further exploration in this area could present an intriguing direction for future research.
C MORE DETAILS ON VIDEO SYNOPSIS PROMPTING
Leveraging the in-context learning capability (Brown al., 2020; Hu al., 2022) of LLMs, we can guide them to perform the video synopsis prompting task automatically through prompting with single concrete example. For instance, we can instruct ChatGPT (OpenAI, 2022) with the following prompt:
I have prompt "A landscape transitioning from winter spring" for video generation. Can you split the process and describe the states separately? Each state described in only one sentence and please consider the coherency between sub-prompts. Please be straightforward and do not use a narrative style. For example, for prompt "a boy is getting old", can be divided into two states, e.g., "a young boy" and "an old man".
Based on this example, can you provide the description? The number states not limited to two.
22

---

Published as a conference paper at ICLR 2025
0=
Figure S.13: Visualization of regularization matrix AA with different standard deviation a. A Smaller can enhance the effect of regularization.
Subsequently, ChatGPT can provide a detailed video synopsis that includes multiple visual states. Once the LLM has learned such task, then simply prompt it execute the task without reiterating the examples:
I have a prompt "A peony starts to bloom, in the field". Can you split the process and describe the states separately?
Original prompts and the ChatGPT generated video synopsis are available in the prompt_list.jee file included in the Supp. Material.
is sufficient to generate text descriptions for the main event changes in a video rather than for each frame. A text encoder e.g.. CLIP text decoder (Radford et al., 2021), then applied extract the text embeddings of these descriptions, which are then interpolated to guide each frame's 's synthesis via cross attention illustrated in Fig. 1. This process yields more accurate guidance for transitioning visual stages, while ensuring smooth conditioning without abrupt changes between frames. All textual embeddings are fed the model in single inference pass.
D OTHER VISUALIZATION
D.1 VISUALIZATION ATTENTION REGULARIZATION MATRIX
The regularization matrix AAA is designed symmetric Toeplitz matrix with values along the off-diagonal direction following Gaussian distribution. In Fig. S.13, we visualize AA with different standard deviations a. We can see that decreases, the correlation increasingly concentrates on adjacent frames, thereby amplifying the regularization effect.
D.2 EXAMPLES OF REAL VIDEOS
We provide some examples of real dynamic videos G.7. They are collected from the web, DAVIS dataset (Perazzi et al., 2016), showcasing diverse content. The selected videos contain ample visual changes over time, as opposed static clips, and they are all captured using a single-camera setup.
E LIMITATIONS
VSTAR offers simple yet effective solution for improving pretrained T2V models, however, there are fundamental issues of pretrained models that may not be completely resolved via generative nursing at inference time only. Although our VSTAR has eased the process of reasoning prompts that involve dynamic evolution, the model can still struggle with responding to the decomposed open-world prompts, resulting in visuals that are not aligned with the prompt, potentially due to limited capability of the text encoder (Podell et al., 2024; Liu et al., 2024). Nevertheless, several recent works (Chefer et al., 2023; Li et al., 2023) have employed on-the-fly fly latent optimization to improve the textual alignment of frozen T2I model. One may explore the combination of VSTAR with such techniques for further improvement. Due to the limited capability of pretrained T2V models in motion generation, the results may occasionally appear less natural. Nevertheless,
23

---

Published as a conference paper at ICLR 2025
optimistic that the community will continue develop and open-source better and larger models trained on more extensive datasets, further enhancing motion quality.
24


Reviews:
[{"review_id": "Yjiz8YFHvS", "reviewer": "Reviewer_Apsy", "summary": "The paper proposes a new concept of \"Generative Temporal Nursing\" and aims at enhancing the temporal dynamics of long video synthesis without requiring additional training or introducing significant computational overhead during inference. Specifically, this paper first find out that the temporal correlation varies a lot betwen frames, and introduce TAR module to adjust the temporal correlation with regularization. Then this paper propose VSP module to provide key visual states to decompose the transition along the temporal axis with LLM.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of temporal correlation is meaningful and interesting.\n\n2. TAR module is well-motivated, effective and readily applicable to pre-trained T2V models.\n\n3. The paper is well-structured and clearly presents the methodology, experiments, and results.", "weaknesses": "1. The side effect of TAR: In the Fig.5, the introduction of TAR in generation models may reduce the amplitude of motion.\n\n2. Further ablation on TAR. Does VSP module is used in Fig.9? How do the TAR module and attention map change with the introduction of VSP module?\n\n3. Temporal consistency. Does the introduction of VSP lead to a decrease in temporal consistency? Can authors provide some video demos?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZFGzfl2tL", "reviewer": "Reviewer_4ewj", "summary": "The authors systematically analyze the role of temporal attention in Video Generation and contribute a training-free technique called “Generative Temporal Nursing” to create more temporal dynamics for long video generation.  Besides, LLM-based video synopsis prompting is explored to facilitate gradual visual changes. Experimental results show the effectiveness of the proposed methods in terms of enhancing video dynamics.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is simple and easy to follow.\n2. The motivation of the method is reasonable and strong.\n3. The analysis of temporal attention in T2V model may benefit other research.\n4. The paper is well-written and well-organized.", "weaknesses": "1. The introduction to the VSP is short and some of the details are not clear: do all frames share one interpolated text embedding or does each group share a different embedding?\n2. Introducing a Toeplitz matrix to temporal attention could help to improve temporal dynamics. What I am concerned about is that this kind of hard modification may break the original motion, since I find that the motions of Superman and Spiderman are wired.\n3. I wonder about the proportion of the \"less structured\" temporal attention maps in video diffusion models because only some cases and layers do not satisfy the band-matrix pattern according to my personal experience.\n4. It seems that MTScore from ChronoMagic-Bench is adopted to evaluate the video dynamics. I want to see other metrics in the same benchmark like Coherence Score (CHScore). I am happy to raise my rate if more quantitative results are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "39FDeOIp7V", "reviewer": "Reviewer_dcFR", "summary": "The paper proposes an attention regularization technique for long video generation. In addition to that, the paper also proposes to use existing LLMs to generate better video text descriptions. The main contribution of this work lies in the temporal attention regularization technique, which has been presented with good motivations and was able to regularize the temporal attention to mimic the ones in real videos. The other merit of this approach is that it can be directly applied to certain T2V models without retraining.", "questions": "There are a bunch of questions raised in the weakness section. I would appreciate it if the authors can address these questions, especially these related to the TAR.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of this paper is the proposed Temporal Attention Regularization (TAR), which has been inspired by comparing temporal attention maps between the real and the generated videos. The motivation for this idea is clear and strong, and we can see the differences in attention maps. This helps motivate the use of the proposed TAR approach. In particular, the demonstration of attention to visual patterns offers convincing reasons why the regularization should be performed. \n\nAnother good thing about this TAR is that it could be applied to existing T2V models without re-training as long as the existing models satisfy certain conditions. The paper also performs extensive experiments to justify such a design, pertaining to the impact of resolutions and number of frames.", "weaknesses": "The per-layer temporal attention analysis part is not very clear in Sec. 3.2. Are the resolutions corresponding to the layer dimensions in the UNET? Does that occur in both the encoder and decoder parts of UNET? \n\nThe use of additional matrix \"max\" in Eqn. (3) needs to be further validated. The Toeplitz matrix should ensure that the values for distance frames will decrease. The multiplication of delta_A with the max function will only scale the values but does not change the ordering. I am just curious about the impact of this \"max\" function and how will affect the regularization performance. It appears that the paper does not offer a more detailed analysis of this part, including experimental validations. \n\nThe use of DreamSim in Sec. 4.1 could transfer the dynamics from real videos to the target one based on this delta_A. By looking at the results presented in Figure 8, it appears that the dynamics transfer also includes the style transfer. Intuitively, the attention map was used to regularize the interactions among frames to ensure that the evolving content was consistent with the real videos. With that being said, should we assume that these \"dynamics\" also model the video motions? If that is the case, the style transfer appears to be confusing here.  Is this more due to the use of DreamSim to calculate the similarity matrix? Please also clarify what are the captured dynamics by using the default delta_A. \n\nBased on the discussions from Sec.5, there are limitations to applying the TAR to existing T2V models, depending on how the temporal attention was designed. Though this was not specifically mentioned, was this mainly due to the use of a Transformer backbone instead of the UNET? The Transformer does use frequently with the position embeddings. Most existing video diffusion models, however, are based on the Transformer architecture, such as W.A.L.T, Latte, and VDT. Please offer more comments in this regard by looking at these Transformer-based architecture and whether the proposed approach can be applied to them. Some of them are also open-sourced. The paper have already provided some analysis. But it will be appreciated with further analysis.  \n\nThe use of VSP in Sec 4.3 is straightforward. After applying the LLM, does the new generated prompt considered as the whole prompt to be used in cross-condition? Or does the paper treat the generated prompts as sub-segments and sequentially input that into the model? Please also clarify that. Compared to StoryDiffusion, which also models long-range video generation, what will be the advantage of the VSP? In general, what is the VSP+TAR compared to StoryDiffusion in particular in terms of long-range video generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtgRhFrZjY", "reviewer": "Reviewer_JuSf", "summary": "In this paper, the authors first analyze the behavior of temporal attention when generating longer videos. Then based on the observation, they propose VSTAR, a tuning-free method for longer video generation. The framework supports the transition of mult-prompt through the planning of pre-trained LLMs.", "questions": "Most concerns are listed in weakness. Out of curiosity, FreeLong also mentions a similar phenomenon in temporal attention. Can you briefly explain the pros and cons of the two designs? Of course, it is not necessary to discuss the concurrent work according to the Reviewer Guide.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of temporal attention has some inspiration.\n2. The design of TAR enables longer video generation in a one-pass behavior.\n3. Benefiting from one-pass generation, the framework naturally supervises the coherence of results described by VSP, making it easier to achieve the transition of mult-prompt than the previous methods.\n4. Although the quantitative results in this paper are still to be completed, the qualitative results are relatively sufficient, fully explaining the role of each module of VSTAR and other methods compared.", "weaknesses": "1. This paper does not evaluate some classic metrics, such as FVD. Alternatively, evaluating on a comprehensive benchmark like VBench will make the comparison more convincing.\n2. Table 2 lacks a setting of Baseline+TAR. Purely Baseline+VSP is supposed to bring obvious content mutation while gaining better scores compared to Baseline, further illustrating that the two indicators in the paper are not enough to fully evaluate the quality of video generation.\n3. When comparing VSTAR with FreeNoise in a mult-prompt setting, noticeable flickering in FreeNoise exhibits the implementation is not aligned with the original paper. (Although this does not affect my recognition of the effectiveness of VSTAR in the mult-prompt setting.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)