UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control
Authors: Tian Xia
Xuweiyi Chen
Sihan Xu
Venue: Accepted by TMLR ()

Abstract: Video Diffusion Models have been developed for video generation, usually integrating text and image conditioning to enhance control over the generated content. Despite the progress, ensuring consistency across frames remains a challenge, particularly when using text prompts as control conditions. To address this problem, we introduce UniCtrl, a novel, plug-and-play method that is universally applicable to improve the spatiotemporal consistency and motion diversity of videos generated by text-to-video models without additional training. UniCtrl ensures semantic consistency across different frames through cross-frame self-attention control, and meanwhile, enhances the motion quality and spatiotemporal consistency through motion injection and spatiotemporal synchronization. Our experimental results demonstrate UniCtrl's efficacy in enhancing various text-to-video models, confirming its effectiveness and universality.

Published in Transactions on Machine Learning Research (11/2024)
UniCtrl: Improving the Spatiotemporal Consistency of Text- to-Video Diffusion Models via Training-Free Unified Atten- tion Control
Tian Xia University of Michigan
Kanzio@umichee edu
Xuweiyi Chen * University of Virginia
wwweic@email.viigiinn edu
Sihan Xu
sihanonu@huicch..oo edu
University of Michigan
Reviewed on OpenReview: https: /// openreview. net/ forum? id= =2uFJ7900
Abstract
Vidco Diffusion Models have been developed for video gencration, usually integrating text and image conditioning enhance control over the generated content. Despite the progress, ensuring consistency across frames remains challenge, particularly when using text prompts as control conditions. To address this problem, we introduce UniCtrl, a novel, plug- and-play method that is universally applicable to improve the spatiotemporal consistency and motion diversity of videos generated by text-to-video models without additional train- ing. UniCtrl ensures semantic consistency across different frames through cross-frame self- attention control, and meanwhile, enhances the motion quality and spatiotemporal consis- tency through motion injection and spatiotemporal synchronization. Our experimental sults demonstrate UniCtrl's 's efficacy in enhancing various text-to-video models, confirming its effectiveness and universality.
1 Introduction
Diffusion Models (DMs) have excelled in image generation, offering enhanced stability and quality over
methods like GANs Goodfellow et al. 2020 Karras et al. 2019 2020) and VAEs (Kingına & Welling 2014
Van Den Oord et al.
2017 Ramesh et al. 2021). The superior image ion capability of diffusion models
stems from the critical role of the attention mechanism Rombach et al. 2022a Peebles Xie
2023 Dhariwal
& Nichol 2021) Foundational studics (Sohl-Dickstein et al. 2015 Ho al. 2020 Song ct al.) 2020a 2023
Luo al. 2023 Karras et al.
2022) established the groundwort for DMs capabilities in elliciently scaling
up with diverse data. Recent advancements (Rombach et al. 2022a Ramesh et al. 2022 Nichol et all
2022 Saharia et al.] 2022 Xu et al. 2023b Zhang et al. 2023b Mou et al. 2023 have further improved
controllability and user interaction, enabling the creation of images that better reflect user intentions.
Recently, Video Diffusion Models (VDMs) (Ho et al. 2022b have been proposed for utilizing DMs for video generation tasks. VDMs are capable of generating videos with a wide variety of motions in text-to-video
*Authors contributed equally to this work. *Corresponding author and project lead. Work completed Pixta.rt.

---

Published in Transactions on Machine Learning Research (11/2024)
AnimateDiff
UniCtrl+ UniCtrl+AnimateDDe Diff
polaroid photo, night photo, photo 24 y.o beautiful woman, pale skin, bokeh
AnimateDiff
UniCtrl+ Diff
professional movie, movie of autumn landscape, dramatic lighting, gloomy, cloud weather, tree
Figure 1: UniCtrl for Video Generation. we propose UniCtrl, a concise yet effective method to significantly improve the temporal consistency of videos generated by diffusson models yet also preserve the motion. UniCtrl requires no additional training and introduces no learnable parameters, and can be treated plug-and-play module al inference lime.
synthesis lasks, supported by the integration of text encoders RRdford el al. 2021 b, self attention, cross
attention and temporal attention, as shown in (Guo et al. 2023b Zhang et al. 2023a Hu et al. 2022 Ho
ett al. 2022a; Blattmann et al. 2023a b). Many open-source text-to-video models have been introduced,
including MooeeSSooe((aarg
al., 2023). AnimateDiff(Guo et al. 2023b OideoCratterCheer
al., 2023
and on. These models typically require a pre-trained image generation model, e.g., Stable Diffusion [Rombach al. 2022b). and introduce additional temporal or motion modules. However, unlike images that contain rich semantic information, text conditions are more difficult to ensure consistency between different frames of the generated video. At the same time, some work also uses image conditions to achieve image-to-
video generation with improved spatial semantic consistency (Guo et al. 2023a Hu & Xu 2023 Blattmann et al. 2023a Some works have proposed the paradigm image conditions alone cannot effectively control the motion of videos. Combining text and image conditions leads enhanced spatiotemporal consistency in rext-and-image-to-iiin workflow (Zhang et al. 2023c
tatto-iinagg-l-vviee Girdhar el al., 2023), but
Chen et al. 2023 2024
Xing et al. 2023 Gu et al. 2023b). but these methods require additional training.
To this end, our research goal in this work is to develop an effective plug-and-play method that is training- free, and can be applied to various text-to-video models to improve the performance of generated videos. To solve this problem, we first attempt to ensure that the semantic information between each frame of the video consistent principle. As the attention mechanism plays a significant role, this principle draws inspiration from previous rescarch attention-bascl control (Hertz et al., 2023; Tumanyan et al.] 2023; Cao et al. 2023 Xu al. 2023a)). These works have demonstrated in DMs that the queries in attention layers determine the spatial information of the generated image, and correspondingly, values determine the semantic information. We observe that this finding also holds in VDMs and propose the cross-frame self- attention control method. We thus apply the keys and values of the first frame in self-attention layers to each frame and achieve satisfying consistency in the generated video.
Secondly, we observe that as the vidco's consistency improves, the motions within vidcos tend become less pronounced. To solve this problem, we propose the motion injection mechanism. Based on the assumption that queries control spatial information (Cao al. 2023), divide the sampling process into two branches:
2

---

Published in Transactions on Machine Learning Research (11/2024)
output branch for cross-frame self-attention control and a motion branch without any attention control. We reserve the queries in the motion branch as motion queries, and use the corresponding motion queries in the output branch. Through the cross-frame self-attention control and motion injection, we ensure that the semantic information between cach video frame is consistent, while the motion is preserved.
Lastly, we note that motion queries cannot guarantee the spatiotemporal consistency of video. Observing that the output of the output branch has better spatiotemporal consisteney, we further propose spatiotemporal synchronization, that is, before each sampling step, the latent of the output branch is copied the initial value of the latent of the motion branch. Our UniCtrl framework combines the above three methods into plug-in-and-play method that can improve the quality of spatiotemporal consistency and motion quality of the generated videos, while ensuring the consistency of the semantic information of each frame of the video. A recent survey (Melnik et al., 2024) also emphasizes the importance of enhancing spatiotemporal coherence and motion consistency to advance video diffusion models. In the other domains related to video gencration, various methods attempt to address similar challenges (Huang et al., 2023 Yang et al.) 2023) Through experiments, several text-lo-video models have been improved after applying the UniCtel method, proving the effectiveness and universality of UniCtrl. As illustrated in Figure I UniCtrl plays a significant role in improving spatiotemporal consistency and proserving the motion dynamics of generated frames. This method can be readily applied during inference without the need for parameter tuning.
2 Related Work
Video Generation
Many previous efforts have explored the task of video generation, e.g., GAN-based
models (korookhodov et al.
2022
Tian et al.
2021
Brooks et al. 2022) and transformer- based models
Hong
et al.
2022 Villegas et al.
2022
Wu et al.
2021
2022 ). Recently, ly, following that Diffusion models
(MMs)
(Sohl- Dickstin et al. 2015 Ho et al.) 2020:
Song al. 2020a 2023 Luo et al. 2023 Karras et al.
2022
have achieved remarkable results in image generation (Rombach al. 2022a Ramesh al. 2022
Nichol
et al.
2022 Saharia et al.
2022 Nichol & Dhariwal 2021), video diffusion models (VDMs) (Ho et al.
b022b
has also demonstrated their capabilitios in video generation (Blattmann et al.
2023b Girdhar et al.
2023
Gu et al. 2023b Ho et al. 2022a Hu et al. 2022: Singer et al. 2022; Zhang et al. 2023a Guo et al.
2023b
Chen et al. 2023 2024 Xing et al. 2023 Wang el al. 2023). At present, VDMs are mainly implemented
by adding additional temporary layers to 2D UNet, which leads to a lack of cross-frasd frame constraints in the
training process of the 2D UNet model. Some methods (Wu et al.] 2023: Qiu et al.,
2023 Gu ct al., (023a))
tried to use a training-free method to make the generated videos more smooth. However, how to maintain the cross- framm consistency in videos generated by VDMs remains unresolved.
Attention Control in Diffusion Models Different from models that require training Zhang et al. 2023b; Xu et al.) 2023b Mou et al.] 2023), attention control (Hertz et al.) 2023 Tumanyan et al. 2023 Cao et al. 2023 Xu et al. 2023a Ge et al. 2023b) is training free method which has been widely applied
in the task of image editing. Previous work has found that Attention Control can be used to ensure both
semantic (Cao et al. 2023) and spatial consistency (Hertz et al., 2023 Tumanyan et al.j 2023 Ge et al. 2023b) in image editing. InfEdit (uu el al.] 2023a) unified the control of semantic consistency and spatial consistency for the first time, proposing unified attention control (UAC). Text2Video (Khachatryan
et al. 2023) applics frame-level self-attention on tcxt-to-image synthesis methods and enables text-to-video through manipulating motion dynamncc in latent codes. Some work has introduced atlention control to VDMs for video editing (Liu et al. 2023 Geyer et al. 2023 Khandelwal 2023), but no one has improved the consistency of generated videos through video diffuusin by attention control. Is that possible to introduce UAC into VDMs to ensure cross-framd frame semantic consistency and spatial consistency throughout the video is an interesting and worthwhile question to explore.
3 Preliminary
3.1 Diffusion Models (DMs)
Diffusion Models (DMs) (Song et al. 2020a Ho et al. 2020 Song al.
2023
Karras al. 2022) are a
type of generative model trained via score matching (Hyvärinen & Dayan
2005 Lyu 2012
Song & Ermon
3

---

Published in Transactions on Machine Learning Research (11/2024)
2019 Song al. 2020b). The forward process gradually adds noise to data to make it follow the Gaussian distribution: b (=x;; 20 (1 V. where are samples from the data distribution and .1, are from variance schedule. In Ho et al. (2020), this process is re- re-parameterized into the following form:
2t ar + vI ate; E NO(,
(1)
The training objective of DMs is predict the added noise via a neural network Eø, to reconstruct the original input from the noisy sample by minimizing the distance d(-,)):
min Exo, s, [d =0(2,,
(2)
The sampling process of DMs (Ho et al. 2020 Song al. 2020a) is iterative and can be represented in the form with different noise schedule dz:
vI (n
2t-1 = ar-1
(predicted zo)
Vān
V aa- - of c0(
(3)
(direction to zt)
+ 0ge where 8 ~ I)
(random noise)
Latent Diffusion Models (LDMs) Rombach et al. 2022a encodes samples into latents using encoder E, such that 2g = E(zo). Also, the output is reconstructed via a decoder D, represented by D(2). This approach has led to improvements in stability and efficiency in the training and generation process.
Video Diffusion Models(VDMSs (Ho et al. 2022b) extended the application of DMs the domain of video gencration, adapting the framework to handle 4D vidco tensors in the form of frames height width channels, which we can use zf to describe the frame f+ 1 at the timestep
3.2 Attention Control
We follow the notation from (Hertz al. 2023: Xu et al., 2023a). In the fundamental unit of the diffusion UNet model, there two main components: cross-atlention and self-attention blocks. The process begins with the linear projection of spatial features to generate queries (Q). In the self-attention block, both keys (K) and values (V) are derived from the spatial features through lincar projection. Conversely, for the cross-attention part, text feat ures undergo a linear transformation to form keys (K) and values (V). The attention mechanism (Vaswani et al. 2017) can be described as:
QKT Vd.
ArTENTION(Q, MV softmax
V
(4)
Mutual Self-Attenion Control (MasaCtrl) is proposed by (Cao et al. 2023). and find that replacing the Qs in attention layers while keeping the Ks and Vs same, can change the spatial information of generated images but keeping the semantic information preserved. This technique can help diffusion in spatial-level editing, e.g. sitting dog to a running dog. Here we use (-)are to represent the tensor obtained from the source image and (o) for the larget output we want, we can use the following formula. to define the MasaCtrl algorithm.
out = ATTENTION(Q"* K /
Cross-Attenion Control (P2P) is method mentioned in (Hertz al. 2023 which is for semantic-level image editing (e.g. dog to cat). This work observed that different Vs from different text prompts decide the semantic information of generated images. If attention maps (M) in cross-attention layers have been reserved, but use different Vs for the allention calculation, most of the spatial information will be preserved. We here use (.)are to represent the tensor obtained from the source image and prompt and (-)tst for the target output with target prompt, this algorithm can be described as following as same Qaee, Kere lead same a
out CROSS ATTSSAATTNTO Kere V(St)

---

Published in Transactions on Machine Learning Research (11/2024)
AnimateDiff
+SAC
+MI
+SS
Aston Martin is moving
The first demonstrates the original frames generated with the baseline model, in which the vehicle and the road are inconsistent across the frames. The second row shows frames generated with baseline model augmented with cross-frame Belf-Attentinn Control (SAC). While it maintains incredible spatiotemporal consistency, it exhibits little motion. The third row explains frames augmented with SAC and Motion Injection (MI). Although MI injects more motion in addition to SAC, the results demonstrate that it falls short on spatiotemporal consistency again. The fourth row contains frames further augmented with Spatiolemporal Symchronization (SS) in addition to SAC and MI, which improves spatiotemporal consistency over the results from the third row and achieves balance between motion and spatiotemporal consistency, both in-frame and fram-.
Figure 2:
4 UniCtrl: Cross-Frame Unified Attention Control
In the text-to-video task, it is difficult to ensure the consistency between different frames of the generated video due to the lack of semantic level conditions and the constraint of different frames in the 2D UNet layers. Based on the previous work DMs(Hertz et al., 2023 Tumanyan et al. 2023 Xu et al. 2023a Cao et al. 2023). is found that the queries in the attention layers determine the spatial information of the generated images, and correspondingly, the values determine the semantic information. We assume these properties still exist in VDMs.
We first analyzed the role of keys and values in the self-attention layers of VDMs, and then analyzed the role of queries in all the attention layers. Inspired by InfEdit (Xu et al. 2023a),, we then propose the cross-frame unified attention control to achieve both semantic level consistency and better spatiotemporal consistency. Lastly, we apply Spatiotemporal Synchronization (SS) by replacing the latent of the motion branch with the latent from the output branch cach sampling step. We demonstate the effectiveness of cach module of UniCtrl in Figure
4.1
Fross-Frame Seff-Attention Control
2023 Tumanyan et al. 2023 Xu et al. 2023a has observed that queries in the allenioon mechanism form the layout and semantic information of generated images (Cao et al. 2023 Xu et al. 2023a
Previous work (Cao et al.
2023
Hertz et al.
Tumanyan et al. 2023). while values contribute to the semantic information (eaimattio(Herrt (Hertz ct al. 2023 Xu et al. 2023ah. Therefore, we hypothesize that using the same values the attention of different frames can ensure cross-frame consistency. Additionally, we observed that the mismatch between keys and values will degrade the quality of generated videos through experiment and provide
5

---

Published in Transactions on Machine Learning Research (11/2024)
Decoder
Self- Attention
Cross- Attention
Temp- Attention
Output Branch
k
vo
*+
Copy
Text Prompt
Copy
CreesFrame
Qm
K
Qm
Self-Aitn Ctrl
Motion Branch
Self-
Cross- Attention
Temp- Attention
Motion
Attention
Injection
Spatiotemporal
Synchromization
mLT
m,++1
Figure 3: In our framework, use key and value from the first. frame represented by K and v in the self-attention block. We also use another branch to keep the motion query Qm for motion control. At the beginning of every sampling step, we let the motion latent equal to the sampling latent, to avoid spatial- temporal inconsistency. Note that in the actual workflow, Q replacement occurs only in cross-altention, as the Q in the self-attention blocks of both branches are always the same. We explain details our framework in Algorithm [ and Algorithm
Consequently, in our cross-frame Self-Attention Control (SAC), we inject both keys and values from the first frame, as detailed in Algorithm [ to every other frame at cach self-attention layer during denoising. We showcase one example of the effectiveness of SAC by comparing the first and the second row in Figure? 2
qualitative example Section 5.4
4.2 Motion Injection
In our experiments, detailed in Section
5 we iden-
Algorithm 1 Cross-Franc Self-Attention Control
tify a significant. limitation associated with cross- frume self-attention control: it tends to produce overly similar consecutive frames, leading to mini- mal motion within the video sequence. Again, as described in (Cao et al.) 2023 Xu et al.) 2023a), the queries in self-attention and cross-attention de- termine the video's spatial information, which cor- responds to motion. Therefore, by preserving the original queries, we can maintain the motion effect.
Input: Hidden State
2: bs, f,h, = z.shape()
3: 2 = 2||,;, 0, :,. j.unqueezze()) # Get the 1st frame.
4: dz = = 1,
5: Q = to_q(2)
6: K to k(2) # Get the Key of the first frame.
7: 0 vo_v(2") # Get the Value of the first frame.
8: out SELFATTENTION(Q, K", Vo)
9: Output: out
As F we have divided the inference process into two branches: the output branch, which undergoes cross-frame self- att-nnttentoon control, and the motion branch, which does not involve attention control. We retain the queries in the motion branch motion queries and use the corresponding motion queries in the output branch. Here, we denote motion queries Qm The method for motion injection can be expressed by the following formula: out = ArTENTION(Qm; where ATTENTION refers to both of self- and cross-attention layers. This method is designed to fully re- tain the motion of the original vidco. However, in practice, a trade-off between motion preservation and spatiotemporal coherence remains evident. Consequently, we propose an additional refinement: selectively preserving motion at specific steps throughout the sampling process enhance control. To this end, we introduce a technique that modulates motion through the integration of motion injection degree, c, which is defined within the interval 0 < 1. This approach is further detailed and formalized presented in A Algorithm 2 and the following outlines this approach in detail
out ATTENTION(Qm:". AUTENTION(Q,::)
12 (1 - c) xT
<< -1 x

---

Published in Transactions on Machine Learning Research (11/2024)
4.3 Spatiotemporal Synchronization
Algorithm 2 Motion Injection
Upon closer inspection of the results, notably ex- emplified by the third row of Figure 2 it becomes evident that simply injecting spatial information from the original video is insufficient for guarantee- ing spatiotemporal consistency. Considcring that output branch yields more spatiotemporally con- sistent results, we further propose Spatiotemporal Synchromizztiin (SS): the latent of the output branch is copied as the initial value of the latent for the motion branch before each sampling step. By doing so, our method can simultancously ensure the semantic consistency of the generated video and im- prove the quality of spatiotemporal consistency and preserve the degree of motion diversity. We present qualitative example to demonstrate the effective- ness of SS by comparing the results depicted in the third and fourth rows of Figure 2
1: Input:
Video Diffusion Model VDM
Sequence of timesteps TO T1 TN
Text Condition c
Timestep condition for Motion Control
2: To = 2Th ~ ~N(0 I)
3: for n = 0 to N - do
4:
2m. 2T
5:
Qm VDM (m,
6:
if t 1 nn then
7:
Tr VDM Tn) + Qm)
8:
else
9:
2T VDM C, Tr)
10: end if
11: end for
12: Output: 2TN
4.4 Cross-Frame Unified Attention Control
As illustrated in Figure 8. we integrate cross-frame self-attention control, motion injection and spatiotemporal synchronization into a cohesive framework termed Cross- Frame Unified Allention Control(Unittr)), In the output branch, we employ the key K and value 0 derived from the initial frame to maintain cross-frame semantic consistency.
scparate branch is utilized to preserve the query specifically for motion control, replacing the output branch's 's query with Qm from the motion control branch. To prevent spatiotemporal inconsistencies, synchronize the latent representation of the motion branch with the output branch's 's preceding latent state before each sampling step.
5
Experiments
In this section, evaluate the effectiveness of UniCtrl. We discuss metrics, backbones and baseline in Section Then we include both qualitative comparisons in Section 5.2 and quantitative comparisons in Section to showcase the effectiveness of UniCtrl in terms spatiotemporal consistency and motion preservation. Additionally, we explore the contribution of each component within UniCtrl, the motion injection degree, and the specific design choice of swapping Key and Value together in the SAC procedure, as detailed in Scction 5.4
5.1
5.3
5.1 Experimental Setup
To evaluate the effectiveness of our model, we collect prompts from two datasets UCF-101 (Soomro et al. 2012) and MSR-VTT (Xu et al.) 2016) for generating vidcos. Following Ge et GlfGe Ge et al. 2023a), we use the same UCF-101 prompls for our experiments. We also randomly selected 100 unique prompls from the MSR-V VTT dataset for our evaluation. Those two parts of data consist of our dataset for evaluation. To mitigate the stochasticity inherent in diffusion models, all experiments were conducted using multiple random seeds. The scores reported in the tables represent the average results across these runs, and we provide the corresponding standard deviations in the Appendix. Next, we briefly introduce evaluation metrics and we also provide details in the Appendex.
Metric To quantitatively evaluate our results, we consider standard metrics following Singer et al. 2022 Wu et al. 2023
DINO: To evaluate the spatiotemporal consistency the generated video, we employ DINO et al. 2024) to compare the cosine similarity between the initial frame and subsequent frames. In our experiments, utilize the 10000vits66(Caree et al. 2021) model to compute the DINO cosine similarity.
7

---

Published in Transactions on Machine Learning Research (11/2024)
RAFT:: To compare the magnitude of motion in the videos, utilize RAFT Teed & Deng 2020 to estimate the optical flow, thereby inferring the degree of motion. We utilize the off-the-shelf RAFT model from torchvision (maintainers & contributors 2016).
FVD: To assess video quality, particularly the quality of individual frames during realistic motions, we employ the Dontent-Debiased FVD metric (Ge et al. 20244) to evaluate the generated videos. Tn our experi- ments, we use the official library to compute the FVD.
FVMD: To evaluate the quality of motion consistency in video generation, use the Fréchet Video Motion Distance (FVMD) metric (Liu et al., 2024) to assess temporal coherence based on velocity and acceleration patterns. In our experiments, we employ the official library to calculate the FVMD with the ground truth videos from our prompt dataset.
Backbones Since our method plug-and-play, we decide to evaluate our methods on few popular basclines:
AnimateDiff /Guo et al. 2023b) introduces practical framework for adding motion dynamics to person- alized text-to-inage models, such those created by Stable Diffusion, without the need for model-specific adjustments. Central to AnimateDiff is a motion module, trainable once and universally applicable across personalized lext-lo-image models derived from the same base model, leveraging transferable motion priors from woaldworld videos for animation.
VideoCrafter (Chen et al.] 2023 introduces two novel diffusion models for video generation: T2V, which synthesizes high-quality videos from text inputs, achieving cinematic-uuliity resolutions up to 1024x576, and 12V, the first open-source model that transforms images into vidcos while preserving content, structure, and style. Both models represent significant advancements in open-source video generation technology, offering new tools for researchers and engineers. We use the T2V version in our experiments.
CM (Wang et al. 2024) introduces a novel framework for efficient and high-quality video gener- ation by leveraging consistency learning. builds upon the principles of the Consistency Model (CM) and Latent Consistency Model (LCM) from image diffusion models, accelerating video generation with minimal steps. AnimatcLCCM employs a decoupled consistency learning strategy, scparating the learning of image generation priors and motion generation priors.
Baseline We select Freelnit (Wu et al. 2023 as our baseline since Freelnit is a training-free method that attempts to improve the subject appearance and temporal consistency of generated videos through iteratively refining the spatial-temporal low-frequenee components of the initial latent during inference. However, given that our method UniCtrl operates on the attention mechanism and FreeTnit adjusts the frequency domain of the latent space, UniCtrl and Freelnit are orthogonal approaches. Both are training-free methods capable of enhancing the spatiotemporal consistency of generated videos via diffusion models. We demonstrate the integration of UniCtrl and Freelnit both in 5.2 and 5.3
5.2 Qualitative Comparisons
Qualitative comparisons, depicted in Figure | reveal that our UniCtrl method markedly improves spatiotem- poral consistency and maintains motion diversity. For example, with the text prompt "walking with a dog", Freelnit produces inconsistent appearances for both the lady and the dog, whereas UniCtrl ensures consistent representations of both entities. Furthermore, when processing the prompt "A young woman walks through flashing lights", UniCtrl maintains the detailed features of the young woman's 's dress while ensuring her walk- ing motion remains natural, in contrast to the vanilla AnimateDiff model. Additionally, we demonstrate UniCtrl's flawless integration with FreeInit, consistently preserving the young woman's appearance.
We show additional qualitative results in Figure and Figure G to demonstrate the efficacy of UniCtrl to significantly improving spatiotemporal consistency and preserve motion dynamics for diverse set of prompls.
5.3 Quantitative Comparisons
For quantitative comparisons, the quantitative results on UCF-101 and MSR-VTT are reported in Table [ We compare the backbones and backbones augmented by UniCtrl and FreeInit respectively. According to
8

---

Published in Transactions on Machine Learning Research (11/2024)
AnimateDiff
Freelnit+ AnimateDiff
UniCtl-AnimatteDiff
walking with dog
AnimateDiff
FreeInit AnimateDiff
cute cat walking on the city streets
AnimateDiff
Aremattt-A
UniCtrl+ AnimateDiff
A young woman walks through flashing lights
UniCtrI+ Freelnit
AnimateDiff
Figure 4: Qualitative Comparisons. We demonstrate UniCtrl's adaptability to diverse prompts, enhanc- ing temporal consistency and preserving motion diversity. Comparative inference results with Freelnit are presented for context. Additionally, we demonstrate UniCtrl's seamless integration with FreeInit.

---

Published in Transactions on Machine Learning Research (11/2024)
VideoCrafter
UniCtrl + VideoCrafter
Turtle swimmiig in ocean
VideoCrafter
UniCtrl+VideoCrafee
mopping floor
AnimateL.CM
UniCtel-AnimateL.MM
1girl, solo, cherry blossoms, walking
AnimateLCM
ajaazz musician playing the saxophone in smoky club
We provide additional qualitative examples across various backbones to demonstrate UniCtrl's capability in enhancing spatiotemporal consistency while effectively preserving motion dynamics.
Figure 5:
the motrics, UniCtrl significantly improves the spatiotemporal consistency in the generated videos across all backbones on both prompt sets from 2.12 to 2.44. While FreeInit achieves remarkable improvements over spatiotemporal consistency, found that UniCtrl outperforms FreeInit the strength of motions on both AnimateDiff and VideoCrafter by large margin from 11.67 to 17.65. Note that purposely chose the motion injection degree c= 1 to demonstrate how UniCtrl can preserve the motion compared with FreeInit. However, there still exists trade-off between spatiolemporal consistency and motion diversity. In section 5.4 we show spatiotemporal consistency can further improved with a smaller motion injection degree. Thus, we recommend motion injection degree 0.4 for real-world applications. Lastly, we showcase how UniCtrl and FreeInit can improve AnimateDiff together and we found this integration can improve spatiotemporal consistency together. We will introduce the details of how we integrate UniCtrl and FreeTnit
10

---

Published in Transactions on Machine Learning Research (11/2024)
AnimateDiff
AnimateDiff + UniCtrl
drumming in lighting
AnimateDift
AnimateDiff UniCtrl
A fancy car is moving
Animate Diff
AnimateDiff + UniCtl
Firework in Disney castle
We present more qualitative examples to show UniCtrl's ability to improve spatiotemporal consis- tency and significantly preserve motion dynamics.
Figure 6:
Table 1: Quantitative Comparisons on UCF-101 and MSR-VTT. UniCtrl significantly improves the temporal
consistency while keeping the motion in the generated videos. indicates the motion injection degree. I indicates the number of iterations for FreeInit.
Method
DINO (t)
RAFT (t)
DnimateDiff Guo et al. 2023b
94.26
25.44
FreeInit + AnimateDiff (I /=3 =3
96.04
11.62
UniCtrl + AnimateDiff (c = 1)
96.38 (+00.34)
23.29 (+11.67)
VidcoCrafter (Chen et al. 2023
93.53
29.20
FreeInit + VideoCraller (1 /= 3)
96.75
7.46
UniCtrl + VideoCrafter (c = 1)
95.55 (-01.20)
25.11 11 (+17.65)
AnimateI.CM (Wagg al. 2024
92.96
20.80
FreeInit + AnimateLMM (I 3)
94.90
14.64
UniCtrl + Animatel.CM (c=1)
95.40 (+00.50)
17.53 (+2.89)
UniCtrl Freelnit AnimateDiff (1 (4=3, 1)
96.85
9.82
in the Appendix. Note that we obtained different scores for FrocInit because we randomly sampled different set of prompts from MSR-VTT and we used our own implementations for FreeInit on VidcoCrafter since we cannot find official implementation.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Furthermore, regarding video quality and mo- tion consistency, we present the quality results on UCF-101 in Table 2 We compare AnimateDiff with FreeInit augmented and Unictrl augmented. The metrics show that. UniCtrl significantly en- hances the quality of individual video frames, im- proving the score from 1069.90 to 819.74. Addi- tionally, observe substantial improvement in FVMD, decreasing from 27,124.17 to 8,864.07, which indicates a marked enhancement in motion con-
Table 2: Quality results on UCF-101, which show a sig-
nificant improvement in both video quality and motion consistency in video generation by UniCtrl.
Method
FVD (4)
FVMD (4)
AnimateDiff (Guo et al. 2023b
1069.90
27124.17
FreeInit + AnimateDiff
958.97
25078.05
UniCtrl + AnimateDiff
819.74
8864.07
sistency. These results further demonstrate our model's capability to produce higher-quality videos with improved motion coherence, highlighting the significant effectiveness of our approach in advancing lext-lo- video models.
5.4 Ablation Study
In this section, we assess the impact of each UniCtrl module and the efficacy of the motion injection degree. We further corroborate our design choice of swapping Key and Value together in the SAC procedure through the subsoquent ablation studies.
The Impact of SAC, MI, and SS To assess the contribution of the SAC, we conducted periments on both datasets using AnimateDiff as the baseline, incorporating our pipeline but dis- abling SAC, For motion injection, we set the mo- tion injection degree to be The findings reveal our method without SAC works the same on the backbone because the motion branch is exactly the same the output branch. This observation underscores the critical role of SAC in enhane- ing spatiotemporal consistency, as evidenced by the comparisons with the scores from the vanilla. AnimateDiff.
Table 3: Ablation results on UCF-101 and MSR-VTI VTT. We ablate each module of UniCtrl and show the effectiveness of them each. c iidicates the motion injection degree.
Method
DINO (f)
RAFT (t)
AnimateDiff (Guo et al. 2023b
94.26
25.44
UniCtrl w/o SAC AmmateDDiff
94.26
25.42
UniCtrl w/o MI + AnimateDiff
98.08
4.12
UniCtrl w/o SS + AnimateDiff
94.26
21.90
only SAC + AnimateDiff
98.08
4.12
only MI + AnimateDiff"
94.26
25.42
only SS + AnimateDiff
94.26
25.42
UniCtrl (c = 0) + AnimateDiff
98.08
4.12
UniCtrl {c 0..2) AnimateDiff
97.41
9.04
In exploring the significance of Motion Injection (MT), additional tests were performed on both
UniCtrl (c = 0.4) + AnimateDiff
96.69
15.56
UniCtrl (c 0.6) + AnimateDiff
96.46
20.16
UniCtrl {c 0.8) AnimateDiff
96.37
22.50
datasets with AnimateDiff serving as the base-
UniCtrl (c (c= 1.0) + AuimateDiff
96.38
23.29
line, this time with MI deactivated. The results indicated a notable consistency with the baseline, yet with a substantial reduction in motion diversity. This was quantitatively supported by a docrease in the RAFT score from 31.81 to 4.19. Such a marked disparity highlights MT's vital contribution to maintaining the motion dynamics.
Finally, the necessity of the Spatiotemporal Synchronization (SS) was examined by excluding SS from our pipeline and conducting experiments across both datasets, again using AnimateDiff the reference point and setting molion injection degree to be 1. The outcomes showed diminished spatiotemporal consistency in comparison to the baseline, while motion diversity was not significantly impacted. These results emphasize the importance of integrating SS into our pipeline, as corroborated by the UniCtrl's findings presented in Table D illustrating the essential role of SS in achieving the desired balance of spatiotemporal consistency and motion diversity.
Additionally, we conduct experiments on each individual component to provide a deeper understanding of their respective roles and interactions, as shown in Table B The results align with previous analysis: when only SAC is present, we observe significant increase in the DINO score, confirming the critical role of SAC in enhancing spatiotemporal consistency. In contrast, when only motion injection is applied without SAC and the consistency SAC brings, motion injection alone proves ineffective. Similarly, SS alone shows no impact; only when combined with SAC and MI does it effectively achieve spatiotemporal consistency.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Key and Value Mismatch
Key and Value Match
A panda standing on a surfboard in the ocean under moonlight Figure 7: Each row displays a sequence of vidco frames generated from the identical prompt: "A panda standing on a surfboard in the ocean under moonlight." " The section labeled K and V mismatch illustrates the frames produced when there is a discrepancy between key and value pairs. Conversely, the section titled K and match showcases frames generated when key and value pairs are in alignment.
The Impact of Motion Injection Degree To assess the impact of varying degrees of motion injection, we conducted experiments across both dalasets using UniCtrl with motion injection degree set al c = O, c= 0.2, c = 0.4, c = 0.6, c 0.8, and c = The effects of different motion injection levels are depicted quantitatively in Table 3 As the degree of motion injection escalates, we observed that the DINO score consistently outperforms baseline metrics, and the RAFT score progressively increases. This trend indicates amplification in motion diversity. We showcase qualitative examples of the influence of motion injection degree in the supplementary material.
The Impact of Swapping Key and Value We aim to provide a qualitative example to underscore the importance of simultancously modifying both key and value, highlighted in our discussion on the impact of key and value mismatches in Section 4.1 Initially, alter only the value while maintaining the same key within the cross-frame Self-Attention Control (SAC) performing the UniCtrl pipeline. As depicted in the first row of Figure 7 the panda begins to fade the subsequent frames, indicating a substantial decrease in the quality of the generated videos with respect to spatiotemporal consistency. This decline particularly evident when compared to the approach of modifying both the key and value concurrently using the same UniCtrl pipeline.
6 Conclusion
We introduce UniCtrl to address the challenges maintaining cross-frame consistency and preserving mo- tions for Video Diffusion Models. By incorporating UniCtrl, we have notably improved the spatiotemporal consistency across frames of generated videos. Our approach stands out as it requires no additional train- ing, making it adaptable to various underlying models. The efficacy of UniCtrl has been rigorously tested, demonstrating its potential to be widely applied to text-to-video generative models. We discuss the primary limitations of UniCtrl in Section 6.1 and detail our ethics statement in Section 6.2
6.1 Limitations
Our method needs to operate the attention mechanism, which limits the application of our method con-attention--asee models. Also, since we ensure the same value for each frame, changing colors within the video is not possible, which limits the model's ability to generate videos. Additionally, we have not yet guaranteed that spatial information is completely consistent across frames; this might be addressed in future work by controlling the temporal attention block. Furthermore, during the inference process, we need additional computation to preserve the motion query, which affects the inference speed. Our method can still be improved by addressing the above issues.
6.2 Ethics Statement
While UniCtrl offers significant advancements in video generation, it is imperative to consider its broader ethical, legal, and societal implications.
13

---

Published in Transactions on Machine Learning Research (11/2024)
6.2.1 Copyright Infringement.
As an advanced video generation tool, UniCtrl could be utilized to modify and repurpose original video works, raising concerns over copyright infringement. It. crucial for users to respect the rights of content creators and uphold the integrity of the creative industry by adhering copyright and licensing laws.
6.2.2 Deceptive Misuse.
Given its ability to generate high-quality, consistent video content, there a risk that UniCtrl could be exploited for deceptive purposes, such as creating misleading or fraudulent content. This underscores the need for responsible usage guidelines and robust socurity measures to prevent such malicious applications and protect against security Chreats.
6.2.3 Bias and Fairness.
UniCtrl relies on underlying diffusion models that may harbor inherent biases, potentially leading to fairness issues in the generated content. Although our method is algorithmic and not directly trained on large-scale datasets, it is essential acknowledge and address any biases present in these foundational models ensure equitable and ethical utilization.
By proactively addressing these ethical considerations, can responsibly leverage the capabilities of UniCtrl, ensuring its application aligns with legal standards and societal welfare. Emphasizing ethical practices, legal compliance, and the well-being of society is paramount advancing video generation technology while maintaining public trust and upholding community values.
Acknowledgment
We thank Raven, Tom, Martin, Yichi Zhang, Shengyi Qian for their helpful feedback and advice.
14

---

Published in Transactions on Machine Learning Research (11/2024)
References
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable vidco diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation, pp. 22563-22577 2023b.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yo Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769-31781 2022.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 22560-22570, October 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation, 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan, Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL https:
7/arxiv. ahaxxiv.ogg/bbee abrg/abs/2105... 052233300552333
05233
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yr Yu Liu, and Yogesh Balaji. Preserve your own correlation: noise prior for video diffusion models. 2023a.
Songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. IEEE International Conference on Computer Vision (ICCV), 2023b.
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Yan Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance, 2024. URL http::////axxvvooo///bbbe/e/ .23991
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.1087 2023.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXio preprint XxXi:2311.10700 2023.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Wardo-Farloy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communicattins of the ACM, 63(11): 139 144, 2020.
Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing Zhang, Zaxuan Wu, Songcen Xu, Wei Zhang, Yu- Gang Jiang, and Hang Xu. Reuse and diffuse:: Iterative denoising for text-to-video generation. arXiv preprinl Xiv: 03549, 2023a.
Xianfan Gu, Chuan Wen, Jiaming Song, and Yang Gao. Secr: Language instructed video prediction with latent diffusion models. aiXiv preprint ar orXi::2303.1 14897, 2023b.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Yuwei Guo, Ceyuan Yang, Anyi Rao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint or arxiv:2111 10:2211.110 1.19933, 2023a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatedift: Animate your personalized text-to-image diffusion models without specific tuning. arxie preprint (xXx::330.044223 2023b.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapaa,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Pi- cus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):337 362, September 2020. doi: 10.1038//441556-000 2649-2. URL https: 7/doi. 10. 0038/ 2649-2.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://openrevienn http://wwwwweeettwm//eteeaaeee
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Aduances in neural information processing systems, 13:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint wxxi:2210.003000 2022a.
Jonathan Ho, Tim Salimans, Alexcy Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint 220. 15868, 2022.
Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: controllable image-to-video generation with text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18219-18228, 2022.
Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307.14073, 2023.
Nisha Huang, Yuxin Zhang, and Weiming Dong. Style-a-video: Agile diffusion for arbitrary text-based video style transfer, 2023. URL
05464
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6(4), 2005.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
Tero Karras, Mila Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero Text-to-image diffusion models are zero-shot video generators. Xiv preprint or Xiv:2333. 13439, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Anant Khandelwal. Infusion: Inject and attention fusion for multi concept zero-shot text-based video editing. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3017-3026, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. Fry 'eceet video motion distance: A metric for evaluating motion consistency in videos. arXiv preprint arXiv:2407.11122 2024.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya. Jía. Video-p2p: Video editing with cross- attention control. arXiv preprint or Xii:23033044766 2023.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. Xiviv preprint arXiv:2310.0439 2023.
Siwei Lyu. Interpretation and generalization of score matching. arXiv preprint arXix:1205. 2629, 2012.
TorchVision maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https: //ap::://githut
com/pytorcchviison 2016.
Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, and Helge Ritter. Video diffusion models: A survey, 2024. URL https://arxivv wwwpsiii.orrr.eeooooeee ab0/
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for lext-lo-image diffusion models. arXiv preprint (tt::2302.084555 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020. URL httpss///eeeeloper nvidia. nvidia.com/cudae www.c//mccaaoooiitii.i
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. Dinov2: Learning robust visual features without supervision. 2024.
William Poobles and Saining Xic. Scalable diffusion models with transformers, 2023. URL htttp://aarx::
htg/abb//211.0997 09748
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiu preprint arXim:2910.1516) 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821 8831. PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint or Xix::2204..0 0022200440622,, (2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: fexteoo-video generation without text-video data, arXiv preprint Xrv::22209.1 14792, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3636-3636, 2022.
Jascha Soik-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020a.
Yang Song and Stelano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXi preprint arXiv:2011.13456, 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. arXiv:2308.01463 2023.
arXiv preprint
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. 2012.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. 2020.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov, A good image generator is what you need for high-resolution video synthesis. arXiv preprint 1.Xiv:2104,110000 2021.
Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusson features for tcxt-driven image-lo-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pp. 1921-1930, June 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Ruben Villegas, Mohammad Babacizadeh, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXic:2210.088 02399,, 2022.
Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Keqiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. Animatclem: Accelcrating the animation of personalized diffusion models and adapters with decoupled consistency learning, 2024. URL https://arxix: org/ wttp/bb///0../00/////
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modclscope text-to-video technical report, 2023.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.1480 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on compuler vision, pp. 720-736. Springer, 2022.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiw:2312.07537 2023.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien- Tsin Wong, and Ying Shan. Dy- namicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310. 12190, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. June 2016.
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. arXiv preprint arXiv:2312 arXiv:2312.4495,, 2023a.
Sihan Xu, Ziqiao Ma, Yidong Huang, Honglak Lee, and Joyce Chai. Cyclenet: Rethinking cycle consistent in text-guided diffusion for image manipulation. In Advances in Neural Information Processing Systems. 2023b.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text-guided video- http://awxxiv.og/bbbb//////
to-video translation, 2023. URL
07954
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. anXiv preprinl ar 2rv: 15818, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to texl-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2023b.
Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. Xixiv preprint ar Xiv:23312 13964, 2023c.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
Here we present some of the sections left for discussion in the main paper. This supplementary report includes discussions on Details Metrics, Implementation Details, and more Qualitative Results on the effectiveness of the motion injection degree, ablations of each module UniCtrl, and how UniCtrl and FreeInil. (Wu al. 2023 can work together. These results further demonstrate UniCtrl's efficacy in improving spatiotemporal consistency and preserving motion dynamics.
A Details of Metrics
We consider standard metries: DINO and RAFT following (Sineer et al. 2022 Wu et al. 2023 and we present details of our evaluation metrics:
DINO: To evaluate the spatiotemporal consistency in the generated video, we employ DINO (Oquab et al. 2024 to compare the cosine similarity between the initial frame and subsequent frames. The average DINO score across all consecutive frames is then used as the video's overall score. In our experiments, we utilize the DINO-vits 1(Crron al.) 2021) model compute the DINO cosine similarity. We present the details of our implementation in Algorithm 3
RAFT: To compare the magnitude of motion in the videos, we utilize RAFT (Teed & Deng. 2020) to estimate the optical flow, thereby inferring the degree of motion. To quantify the motion intensity between every two consecutive frames, we employ the RAFT model to estimate the optical flow for each pixel. We calculate the magnitude of all pixel optical flow using the norm, and finally, we obtain the average of all magnitude as the score for every consccutive frame in the video. In this process, we utilize the RAFT model from torchvision maintainers & contributors 2016). We present the details of the RAFT implementation in Algorithm
Algorithm 3 DINO Score
Input: Generated Video
2: frames = video to frames(v) # Obtain frames from generated video
3: features = DINO((rames) # Obtain features for each frame using DINO
4: for n = 1 to N do
5:
scores CosineSimilarityy(ettusss features(i))
end for
DINO SCORE (SRRAG)(SCores)
Output: DINO SCORE DINO_SCORE
Algorithm 4 RAFT Score
Input: Generated Video frames video__ to frames(r)
for 1 to N do
4: flow vee_oeetors = RAFFT(faaeee 1], frames(n])
5: scores = AVERAGE (oooc///ooooeeetossssssllloo # Calculate average flow magnitude
6: end for
RAFT SCORE = AVERAGE(SCores)) # Average score from pairs of frames
Output: RAFT SCORE
Implementation Details
B.1 Backbones
Two open-sourced text-to-videg video models are used as the base models for FreeInit evaluation. For VideoCrafter Chen et al. 2023), adopt the TideoCraffeerhhhss--004--- For AnimateDiff Guo
20

---

Published in Transactions on Machine Learning Research (11/2024)
al. 2023b); use the mm-sd-v14 motion module with the Realistic- Vision-VV.. LoRA model for eval- uation. For AnimateI.CM Waan et al. (2024), we use the Animatel.CM--sdI155t.. motion module and also Realistic- Vision-V5.. V5.1 LoRA model for evaluation.
B.2 Inference Details
Experiments VideoCrafter is conducted on 512 x 320 spatial scale and 16 frames, while experiments AnimateDiff is conducted on video size of 512 x 512, 16 frames. AnimateLCM is also conducted on 512 x 512, 16 frames. During the inference process, we use classifier-free guidance for all experiments including the comparisons and ablation studies, with constant guidance weight 7.5, All experiments are conducted on Nvidia A10G GPU. We conduct experiments across different GPUs such Nvidia A10G, Nvidia A40, and Nvidia A100 and we observe different inference results using the same seed while the CUDA (NVIDIA) et al.
2020) random algorithm is deterministic. We resolve this behavior by initializing latent using NumPy (Harris et al. 2020). We hope this change will further enhance the reproducibility of our code across different CUDA architectures.
C Detailed Quantitative Results Tables
Since our experiments are conducted with multiple seeds, we report the mean values and standard deviations in the following table as supplement to the previous tables
Table 4: Detailed Table for Table
Method
DINO (1)
RAFT (t)
AnfmateDiff (Guo et al.
2023b
94.26 + 0.63
25.44 + 5.81
FreeInit + AnimateDif" (7 /33)
96.04 0.22
11.62 + 1.72
UniCtrl + AuimateDiff (c 1)
96.38 0.21
23.29 + 2.09
VideoCrafter (Chen et al. 2023
93.53 = 0.27
29.20 + 1.16
Freelnit + VideoCralter = 3)
96.75 + 0.07
7.46 + 0.59
UniCtrl + VidcoCrafter (c (c=11 1)
95.55 + 0.17
25.11 + 1.98
AnimateLCM /Wang et al. 2024
92.96 + 0.18
20.80 + 5.15
FreeInit + Animatel.CM =3)
94.90 + 0.89
14.64 + 4.59
UniCtrl + AnimateLCM (c=
95.40 + 0.27
17.53 + 3.47
UniCtrl + FreeInit AnimateDiff (I = 3, c=
96.85 + 0.19
9.82 + 1.13
Table 5: Detailed Table for Table
Method
FVD (L)
FVMD (L)
DuimateDiff (Guo et al. 2023b
1069.90 + 57.89
27124.17 + 3234.76
FreeInit + AniateDiif
958.97 - 40.82
25078.05 + 5490.52
UniCtrl + AnimateDiff
819.74 + 29.42
8864.07 + 577.43
21

---

Published in Transactions on Machine Learning Research (11/2024)
Table Detailed Table for Table
Method
DINO (1)
RAFT (t)
AnimateDiff (uuo et al. 2023b)
94.26 + 0.63
25.44
5.81
UniCtrl w/o SAC AnimateDiff
94.26 - 0.62
25.42
5.86
UniCtrl w/o MI + AnimateDiff
98.08
0.47
4.12
2.33
UniCtrl w/o SS + AnimateDiff
94.26 + 1.16
21.90 + 2.08
only SAC + AnimateDiff
98.08 0.47
+
only MI + AnimateDiff
94.26 0.62
25.42
5.86
only SS + AnimateDiff
94.26 + 0.62
25.42 + 5.86
UniCtrl (c = 0) + AnimateDiff
98.08 0.47
4.12 + 2.33
UniCtrl (c 0.2) AnimateDiff 97.41 + 0.10
9.04
+ 1.37
UniCtrl c 0.4) AnimateDiff
96.69 0.14
15.56 = 1.44
UniCtrl [c 0.6)
AnimateDiff
96.46 0.17
20.16
1.73
UniCtrl c 0.8) AnimateDiff
96.37 + 0.19
22.50
+ 1.90
UniCtrl (c L.0)
AnimateDiff
96.38 0.21
23.29
2.09
D More Qualitative Results
D.1 Motion Injection Degree
In the main paper, quantitatively demonstrate that higher degrees of motion injection results in videos with more pronounced motion. Here, we provide a qualitative example in Fig. X The motion of the Corgi is significantly more pronounced when comparing vidcos generated with higher motion injection degree to those with a lower degree, while the appearance of the Corgi remains consistent across frames in each video.
MI=1
MI=0.25
MI=0
MI=0.1
A Corgi is surfing
Figure 8: We present 4 sequences of images inferenced from the same prompt but with different Motion Injection Degree. With larger motion injection degree, corgi in each generated video presents more motion while preserve its appearance.
D.2 Qualitative Ablation Results
In the main paper, we conduct quantitative ablation studies of each module and we would like emphasize the indispensable nature of every component within UniCtrl again by presenting Fig. This figure qual- itatively illustrates the ablations through single set of results. By comparing the first and second rows, it becomes evident that the results in the second-row display inconsistencies across frames, both in terms of the car and the road. Intuitively, the motion branch and output branch keep deviating while denoising, SS effectively bridges the difference between the motion branch and the output branch and significantly improves spatiotemporal consistency as a result. By comparing the first and third rows, we observe the third row's result shows much more inconsistencies which clearly explains the necessity of incorporating SAC in UniCtrl. Lastly, we show the fourth row's result contains little motion compared with the first row's result. Thus, we prove that MI is also of the indispensable modules of UniCtrl.
22

---

Published in Transactions on Machine Learning Research (11/2024)
UniCtrl
SS
-SAC
-MI
Aston Martin is moving Figure 9: The first row demonstrates results generated with UniCtrl, which the vehicle and the road are both consistent across the frames. The second row shows frames generated with SAC and MI. The third row explains frames augmented with SS and MI. The fourth row contains frames shows results with SS and SAC. These comparisons serve as qualitative examples for ablation for each module in UniCtrl.
D.3 Unification between UniCtrl and Freelnit
We state that UniCtrl and FreeTnit explore orthogonal directions to improve the spatiotemporal consistency of video diffusion models. Now we want to first explain how to unify UniCtrl and Freelnit. In the framework of FreeInit, it requires N FreeInit iterations, N = in our implementation, and we apply UniCtrl during the first iteration. We find the unification between UniCtrl and FreeInit improves the quality of generated videos and we showcase additional qualitative examples Fig. 10
Original
UniCtrl- Freelnit
jet ski on the water
Original
FreeIntt
person plays a video game Figure 10: We present more qualitative examples in order to show the unification between UniCtrl and FreeInit can improve spatiotemporary consistency and preserve motion dynamics, which further demonstrate that Uni and FreeInit are orthogonal works that both can improve spatiotemporary consistency.
23


Reviews:
[{"review_id": "cklTVLdqjk", "reviewer": "Reviewer_63d9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written.\n2. The motivations of the proposed components are intuitive and reasonable.\n3. The evaluation and ablation study are convincing.\n\nWeaknesses:\n1. To my understanding, in motion injection, when the input latent is copied, the Qs of self-attention in the output branch and the motion branch should be the same. If so, the only operation would be the Q replacement in cross attention. The authors should clarify this point.\n2. From Table 1, integrating both UniCtrl and FreeInit into AnimateDiff yields lower RAFT compared to only using UniCtrl. Can the authors provide the intuition behind this observation?\n3. In Table 2, the authors only report the results of the method without each component. I would like to see the results of the method with each component, which may better show each component’s effect.\n4. I believe that substantial constraints, such as using KV from the first frame and copying the latent, would significantly limit the diversity of the generated videos, even though motion injection may alleviate this issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a plug-and-play method to enhance spatiotemporal consistency and motion diversity in T2V generation. It comprises three components: cross-frame self-attention control, motion injection, and spatiotemporal synchronization. The method requires no additional training and is applicable to various T2V models. Extensive experiments demonstrate its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ICcybcpFPZ", "reviewer": "Reviewer_ndyH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and the motivation is straightforward.\n* The proposed method is training-free and can be employed in various attention-based video diffusion models.\n\nWeakness:\n* My largest concern is that the proposed attention control method somewhat mitigates temporal inconsistency, but it may result in limited or near-static motion. This trade-off suggests that while the method enhances stability, it may reduce the diversity and realism of movement, especially in dynamic scenes.\n* The experiments only report DINO and RAFT metrics, omitting widely-used metrics such as FVD, KVD, and CLIP in the evaluation.\n* All visual examples are based on the AnimateDiff model, with no results presented for VideoCrafter.\n* The improvement of the proposed method appears to be limited and trade-offs. According to Table 1, the DINO score is lower than that of FreeInit+baseline. The improvement is seen only in the RAFT metric, possibly due to the generation of “small motion” results. Therefore, further evaluation using additional metrics is necessary to fully verify its effectiveness.\n* The proposed method is only compared to FreeInit. Additional comparisons with other competing methods are needed.\n* The paper only selected a few text-to-video models as baselines. Since the proposed method relies on K and V features from the first frame, it may be more suited for image-to-video models. Additional experiments are needed to further support the claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a training-free method to enhance the spatiotemporal consistency and motion diversity of videos generated by existing text-to-video models. The approach incorporates cross-frame self-attention control to ensure semantic consistency across frames, and motion injection to improve motion quality. Experiments are conducted using widely-used text-to-video models, such as AnimateDiff and VideoCrafter, to verify the method’s effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OSn1fAa9yb", "reviewer": "Reviewer_ykrr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1) The ablation study is extensive from the number of setups point of view.\n2) The writing is clear and easy to follow.\n3) MI seems to be a promising module which enhances the motion in the generated video while decreasing the consistency between frames, which is understandable.\n4) The limitation have been included.\n\nWeakness:\n\n1) [Major] Given the stochastic nature of diffusion models the experiment section should image multiple runs with different random seeds and also report the standard deviation in the score.\n2) [Major] From the ablation study MI seems to be the most impactful module and other modules seem to have little to no effect. Given the results of the ablation study it is hard to conclude the usefulness of SAC and SS. Given this the writing claim with respect to SAC and SS need to be significantly revised. There is a clear mismatch between claims and evidence.\n3) [Minor] More methods need to be compared in the baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is motivated by the problem that the Diffusion video generation framework frames are not in accordance with the other frames in the video. It presents 3 submethods that work on any transformer-based video generation model, namely, SAC, MI, and SS.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) I would like to see UniCtrl compared with MI only. Given the ablation numbers, I suspect that UniCtrl can be reduced to Motion Injection (MI).\n2) All experiments done with multiple seeds and the standard error should be reported.", "broader_impact_concerns": "Have been addressed in text", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fcbvb7Hswo", "reviewer": "Reviewer_SM4o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n1. The writing is clear and easy to follow. The video samples in the supplementary materials are straightforward to compare, effectively validating the proposed method's efficacy.  \n2. The proposed methods are simple and easy to implement for video diffusion models (VDMs) using attention modules.  \n\nWeaknesses:  \n1. It would be beneficial to provide more quantitative results using dedicated metrics for video motion consistency, such as FVMD [1], especially to highlight the effectiveness of the motion injection module. I don't think RAFT alone is sufficient, as a larger magnitude of motion (which can be captured by RAFT) does not necessarily indicate better video quality. A poor video generative model may produce inconsistent motion and still result in a large amount of motion/RAFT value. Simply put, a larger amount of motion is not naturally equivalent to better video quality or diversity. The evaluation of the motion injection module needs to take this into account.\n\n2. More ablation studies on the effect of the motion injection parameter $c$ are needed. Currently, the early denoising stages use motion queries in the output branch to enhance motion diversity, while the later denoising steps do not employ this technique. The cutoff is applied at a single timestep. Given the importance of this parameter, as highlighted in the experimental sections, a more detailed discussion on the design of the switch schedule would be valuable. For instance, does implementing a linear interpolation (between two queries) over a range of transition timesteps instead of a single cutoff work? Additional engineering details would assist researchers in the community to more easily adapt this method.  \n\n3. In the introduction, it is better to emphasize that attention modules are widely used and explain why they are crucial components in modern text-to-video generation pipelines. It seems somewhat taken for granted that VDMs heavily rely on attention modules from the text.\n\n4. Some recent works on video-to-video editing share similar features, particularly the focus on spatiotemporal consistency, but are not mentioned in this paper (for example, [2][3]). The authors are encouraged to refer to recent surveys, such as [4], to discuss related works in this field with a similar focus. Although the specific methods differ, the common goal of enhancing spatiotemporal and motion consistency is evident in recent studies.  \n\n\nReferences:  \\\n[1] Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos, ICML 2024 CVG Workshop. \\\n[2] Style-a-video: Agile diffusion for arbitrary text-based video style transfer, IEEE Signal Processing Letters (2024). \\\n[3] Rerender a video: Zero-shot text-guided video-to-video translation. SIGGRAPH Asia 2023 Conference Papers. 2023. \\\n[4] Video Diffusion Models: A Survey, preprint.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a couple of useful techniques to improve the spatiotemporal consistency of pre-trained text-to-video diffusion models without the need for GPU resources to fine-tune the model. The core of the approach lies in the smart use and rearrangement of key components in the attention layer. The key methodologies involve: 1) cross-frame self-attention control, 2) motion injection, and 3) spatiotemporal synchronization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This paper focuses heavily on engineering implementation and empirical performance rather than theoretical aspects. The proposed methods are simple and appear effective in addressing some of the problems in current video generation pipelines. However, to make the contributions more solid and the claims clearly supported, I would like the authors to address the above concerns (mostly related to the experimental side) to improve the paper's quality.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "37A4qFvrs2", "reviewer": "Reviewer_hjXg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is a plug-and-play module that does not require any finetuning.\n2. The method shows enhanced spatial consistency of the objects in the video compared to baseline methods.\n\nWeaknesses:\n1. While the cross-frame self-attention control module improves spatial consistency and reduces the chance of object appearance changing in the generated video, it also heavily affects the motion magnitude of the video. Based on the examples the author shared, even after applying motion injection and spatiotemporal injection, the generated video with UniCtrl still suffers from low motion magnitude or near static video scenes. I'm wondering if there is a way to control the strength of UniCtrl affecting the video content such that it can strike a balance between consistency and motion degree. Can the author provide a more detailed analysis regarding this issue?\n2. The proposed motion injection is based on cross-attention control. It is interesting that the author did not consider the temporal attention layers, which are often considered as the key module for modelling temporal information in T2V models. Can the author provide more insights on the design considerations of the motion injection mechanism?\n3. The experiments only considered two metrics and did not include any video quality metrics such as FVD and video inception score (see appendix G in [1]). It is unknown whether UniCtrl generates better quality videos compared to baseline methods and FreeInit. Also, since the method is designed to enhance text-to-video generation, it would be interesting if the author could provide more analysis on video-text metrics such as CLIPSIM, to indicate if there are differences in the alignment with text prompts before and after applying UniCtrl.\n4. The paper could benefit from including qualitative examples with more diverse frames and higher motion magnitudes. For example, currently in all 4 examples in Figure 2, it is hard to recognize whether the car is actually moving forward or not.\n5. While the paper claims that UniCtrl can be applied to any T2V models, only AnimateDiff examples are shown in the paper and there is no VideoCrafter1 examples. From the experimental results, UniCtrl + VideoCrafter1 also suffers from a decreased DINO score compared to FreeInit. The author may want to provide more qualitative and quantitative results to support their claim on the universality of UniCtrl.\n\n[1] Blattmann, Andreas, et al. \"Align your latents: High-resolution video synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a systematic framework for enhancing the spatiotemporal consistency of text-to-video diffusion models. The proposed method, UniCtrl, leverages cross-frame self attention control to maintain spatial consistency, motion injection to preserve motion magnitude and spatiotemporal synchronization to further enhance consistency. The proposed method is tested on pretrained T2V backbones such as AnimateDiff and VideoCrafter and showed improved results on metrics such as DINO and RAFT.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In summary, the paper is well-executed and provides a meaningful contribution in video generation refinement. All reviewers are leaning towards acceptance. However, the reviewers suggest selecting better examples to better showcase the results. Thus, minor revisions are required, primarily to improve the presentation of qualitative results.
Decision: Accept (poster)