Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Model

## Abstract

Abstract Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models.
Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.
See our project page: https://frame-guidance-video.github.io/ .

## 1 Introduction

The rapid advancement of diffusion models [19, 40, 25] has led to the development of powerful video generation models. Recent large-scale video diffusion models (VDMs) have made significant progress in high-quality text-to-video (T2V) and image-to-video (I2V) generation, which are capable of generating diverse and realistic video content [3, 4, 15, 49, 42]. With ongoing advancements, there is a growing interest in enabling more fine-grained control over the generation process.

Recent progress underscores the need for a practical approach to controllable video generation.
Hence, we identify two major desiderata: (1) a model-agnostic, training-free framework, and (2) a general-purpose guidance method.
Existing methods [5, 16, 23] typically fine-tune large-scale VDMs [4, 31, 49, 42] for each specific control task, which is increasingly impractical due to high computational cost and the burden of retraining with every new model release.
This highlights the need for training-free guidance methods that work across models.
Moreover, end users prefer simple, generalizable frameworks that support diverse tasks and inputs, such as reference images, depth maps, or sketches, rather than task-specific models [20, 44] that are restricted to a fixed input type.

Existing methods fall short of satisfying both desiderata simultaneously: training-free approaches [22, 24, 20] are often task-specific and lack generalizability, while general-purpose methods [5, 23, 21] require fine-tuning and need substantial training resources. Many existing methods [44, 46, 1] are both task-specific and training-dependent, making them difficult to adapt to new models or tasks.

In this work, we propose Frame Guidance, a novel guidance method for VDMs that is model-agnostic, training-free, and supports a wide range of controllable video generation tasks using frame-level signals. As illustrated in [Figure 2](https://arxiv.org/html/2506.07177v1#S0.F2), Frame Guidance steers the video generation process by applying guidance to selected frames based on frame-level signals, which produce temporally coherent video.

We present two core components for efficient and flexible frame-level guidance.
First, we introduce *latent slicing*, a simple latent decoding technique that enables efficient training-free guidance for large-scale VDMs.
Based on temporally local patterns of video encoding, we propose to decode only the short temporal slices of the video latent for computing the guidance loss.
Furthermore, we present *video latent optimization* (VLO), a novel latent update strategy designed for precise control of the video diffusion process.
Observing that the overall layout of the frames is largely determined in the first few inference steps, we apply deterministic optimization at the early stages for globally coherent layout, and employ stochastic optimization until the mid-stage for refining the details.

Frame Guidance is applicable to general frame-level control tasks, as shown in [Figure 1](https://arxiv.org/html/2506.07177v1#S0.F1), including keyframe-guided generation, stylized video generation, and looped video generation.
In particular, Frame Guidance supports general input conditions, such as depth maps, sketches, and color blocks as shown in [Figure 1](https://arxiv.org/html/2506.07177v1#S0.F1)(d,e,f).
We demonstrate that Frame Guidance consistently produces superior results on frame-level control tasks across various VDMs [49, 15, 42].

## 2 Related work

#### Training-required controllable video generation

Advances in T2V and I2V generation have opened up new opportunities for fine-grained user control.
These include conditioning on keyframes [52, 44], using style reference images for stylized generation [26, 43], and incorporating trajectory-based signals such as camera movement [53, 1] or motion trajectory [47, 29, 14] for dynamic scene generation.
However, existing methods often require extensive training and model-specific data preparation, such as fixed resolution or frame counts, making fine-tuning increasingly impractical for general users as model sizes and resource requirements continue to grow.

#### Training-free controllable video generation

To reduce the burden of training large models, several approaches have explored training-free controllable video generation [22, 24, 20]. For example, CamTrol [20] enables camera control using external 3D point clouds, while MotionClone [24] performs motion cloning based on temporal attention maps extracted from a reference video. However, these methods are tailored to specific tasks and are thereby ill-suited for more general scenarios requiring different types or even multiple input signals.
In this work, we propose a training-free guidance method that generalizes to a wide range of video generation tasks using frame-level signals.

## 3 Preliminaries

#### Video diffusion models (VDMs)

Recent video diffusion models [4, 49, 15, 42] learn to generate video by reversing the noising process in the latent space.
The high-dimensional video $x_{0}$ is encoded into a lower-dimensional latent $z_{0}=\mathcal{E}(x_{0})$.
The forward noising process corrupts the latent $z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$, where $\epsilon\sim\mathcal{N}(0,I)$ and $\{\bar{\alpha}_{t}\}_{t\in[0,T]}$ is a pre-defined noise schedule.
The reverse denoising process is learned through predicting a time-dependent velocity $v_{t}=\sqrt{\bar{\alpha}_{t}}\epsilon-\sqrt{1-\bar{\alpha}_{t}}z_{0}$, which represents the direction from a noisy sample toward the clean sample [36].
For each time step $t$, the clean sample $z_{0|t}$ can be computed from the noisy sample $z_{t}$ using Tweedie’s formula [11]:

$$ $z_{0|t}\coloneqq\mathbb{E}[z_{0}|z_{t}]=\sqrt{\bar{\alpha}_{t}}z_{t}-\sqrt{1- \bar{\alpha}_{t}}\cdot v_{\theta}(z_{t},t),$ (1) $$

where $v_{\theta}$ is the predicted velocity.
Latents $z_{0}$ are decoded into videos with the decoder $\hat{x}_{0}=\mathcal{D}(z_{0})$.

Recent large-scale VDMs [42, 49] commonly employ spatio-temporal VAEs to encode high-dimensional video data. A notable example is the CausalVAE [51, 4], which enforces temporal causality in the latent space by allowing only past frames to influence future ones. While this design encourages temporally coherent video generation, it also introduces temporal dependencies within the latent sequence, requiring the entire sequence to be decoded even to reconstruct a single frame.

#### Training-free guidance

Training-free guidance [2, 50, 17, 37] uses pre-trained diffusion models to generate samples that satisfy a specific condition, without additional training. At each denoising step $t$, it estimates a clean image $x_{0|t}=\mathcal{D}(z_{0|t})$ from the current latent $z_{t}$, and computes a guidance loss $\mathcal{L}_{e}(\mathcal{D}(z_{0|t}),c)$ that measures alignment with the target control $c$.
The latent $z_{t}$ is then updated using the gradient $\nabla_{z_{t}}\mathcal{L}_{e}$ during inference. One such strategy is the time-travel trick [2, 50, 17], which alternates between denoising and renoising steps to correct accumulated sampling errors.

Figure: Figure 3: Frame Guidance for keyframe-guided video generation task. (Left) Illustration of our method with *latent slicing* and spatial down-sampling ([Section 4.1](https://arxiv.org/html/2506.07177v1#S4.SS1)), and gradient propagation with L2 loss (blue arrows) [Section 4.3](https://arxiv.org/html/2506.07177v1#S4.SS3)). (Right) Visualization of the *video latent optimization* (VLO; [Section 4.2](https://arxiv.org/html/2506.07177v1#S4.SS2)) and the generated video frames during inference with guidance.
Refer to caption: x3.png

## 4 Method

We present Frame Guidance, a simple yet effective training-free framework for controllable video generation using frame-level signals, designed to be compatible with modern large-scale VDMs.
Our approach guides the generation process of pre-trained VDMs by optimizing video latents to minimize frame-level guidance loss applied to selected frames.
In this section, we introduce two key components that enable efficient and flexible frame-level guidance for large-scale VDMs.

### 4.1 Latent slicing

The main challenge of training-free guidance on video generation is the computational constraint. To compute the guidance loss for latent optimization, we should keep track of the gradient chain passing through the whole network ([Figure 3](https://arxiv.org/html/2506.07177v1#S3.F3)). In [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(a), we analyze the memory usage and find that it exceeds 650GB even with gradient checkpointing [8], mostly due to CausalVAE [51, 4].
This overhead arises from the design of CausalVAE, which requires decoding the entire latent sequence even to reconstruct a single frame.
To tackle this, we first analyze the latent space of CausalVAE.

#### Analysis of CausalVAE’s latent space

While CausalVAE is designed to enforce temporal causality in the video latent sequence, we observe that such causality is absent in practice.
To validate this, we conduct a simple experiment: replace a single frame in a real video with a black image (all pixels set to zero), and measure the difference between the latents of the original video and the modified video. As shown in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(b), the perturbation affects only a few consecutive latents rather than the entire sequence. This behavior consistently appears across various VDMs [49, 42, 15]. We refer to this property as *temporal locality*, a key observation for our efficient decoding method.

#### Efficient decoding with sliced latent

We introduce *latent slicing*, a simple technique that significantly reduces the cost of gradient computation on CausalVAE by reconstructing only a few frames from the sliced latents.
When reconstructing the $i$-th frame $x^{i}$, we decode a small window of 3 latents, starting from the latent $z^{j}$, where the latent index $j$ is determined by $i$ and the temporal compression rate of its CausalVAE. Thanks to temporal locality, it is sufficient to decode only the corresponding latents to reconstruct a single video frame. As shown in [Figure 19](https://arxiv.org/html/2506.07177v1#A3.F19), the reconstructed frames are nearly identical to those from full-sequence decoding.
As highlighted in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(a), this latent slicing reduces memory usage by up to $15\times$ compared to using the entire latent sequence.

In parallel with latent slicing, we can further reduce the memory usage by spatially down-sampling the latents before decoding. Despite the lower resolution, the guidance loss from the down-sampled latents still provides meaningful signals to guide the generation. As shown in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(a), applying $2\times$ spatial down-sampling combined with latent slicing reduces memory usage by up to $60\times$, enabling gradient computation to be maintained on a single GPU even for large VDMs [42].

Figure: Figure 4: (a) GPU memory required for guidance when using full latent sequence, sliced latents, and latent slicing with spatial down-sampling. (b) Temporal locality of CausalVAEs from various models. Each latent (y-axis) is primarily affected by a small subset of temporally local video frames. (c) Video generation phase. We measure the L2 distance in the low-frequency region through the inference steps. Layouts are mostly determined during the first few steps. (d) Guidance influence through the inference steps. Yellow box indicates the latent corresponding to the guided frame.
Refer to caption: x4.png

### 4.2 Video latent optimization strategy (VLO)

We introduce an effective video latent update strategy for generating globally coherent video that applies to any type of video diffusion model.

#### Empirical analysis of layout formation

We empirically observe that the generation process in VDMs involves two distinct stages: an early *layout stage*, where the global spatio-temporal structure is established, followed by a *detail stage* for refining fine-grained details.
This behavior is represented in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(c), which plots the difference between the final output video and the downsampled reconstructions at each denoising step.
We observe that the global layout of the video is primarily established within the first five inference steps, which we refer to as the layout stage.

#### Hybrid video latent optimization strategy

Prior training-free guidance methods for image generation [2, 50, 37] use time-travel trick ([Algorithm 2](https://arxiv.org/html/2506.07177v1#alg2)), which optimizes the latent $z_{t}$ via three steps: (1) denoising to $z_{t-1}$, (2) updating $z_{t-1}$ with a gradient step, and (3) renoising updated latent back to $z_{t}$.

However, we find that renoising the updated latent during the early stage yields an adverse effect on the guidance. Due to the high noise scale in the early stage, the renoising step significantly weakens the influence of guidance, and the layouts are established regardless of the guidance.
Furthermore, as shown in the gradient propagation map in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(d), gradients initially propagate globally across the latent sequence but gradually become localized around the guided frames.
Consequently, the absence of guidance during the early stage impairs the temporal coherence of the generated video, making it difficult to correct in later stages.

To address this limitation, we introduce *video latent optimization* (VLO), a simple hybrid strategy, which applies different update rules to video latents depending on the generation stage.
Specifically, at each denoising step $t$ in the layout stage, the latent $z_{t}$ is updated as follows:

$$ $z_{t}\leftarrow z_{t}-\eta\nabla_{z_{t}}\mathcal{L}_{e}(x_{0|t}^{\mathcal{I}}, c_{\texttt{frames}}),$ (2) $$

where $\eta$ is the guidance step size, $x_{0|t}^{\mathcal{I}}$ is the predicted clean frames where we apply guidance, and $\mathcal{L}_{e}$ is a guidance loss with frame-level controls $c_{\texttt{frames}}$.
Since this update does not introduce noise in the early inference steps, it effectively steers the latent to minimize the guidance loss. This *deterministic* update results in a temporally aligned global layout.

In the detail stage, we switch to time-travel to improve the fine details. Updating the latent with renoising in this stage helps reduce accumulated errors during the sampling while preserving the established layout in the earlier stage. After the detail stage, our inference proceeds without guidance.

### 4.3 Frame guidance

Figure: Algorithm 1 Frame Guidance

In [Algorithm 1](https://arxiv.org/html/2506.07177v1#alg1), we provide the overall procedure of our Frame Guidance, which incorporates both the latent slicing and VLO.
Given a set of frame-level controls $c_{\texttt{frames}}$ and selected frame indices $\mathcal{I}\subseteq\{i_{1},\cdots\}$ where we apply the guidance, we first compute their corresponding latent indices $\mathcal{J}\subseteq\{j_{1},\cdots\}$ (see [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(b)). For pre-defined generation phases $t_{L}$ and $t_{D}$, we optimize the video latents in the following manner: At each denoising step $t>t_{D}$, we extract the sliced latents $z_{0|t}^{\mathcal{J}}$ from the latent indices $\mathcal{J}$ ( Line 7) and compute the guidance loss $g_{t}=\nabla_{z_{t}}\mathcal{L}_{e}(x^{\mathcal{I}}_{0|t},c_{\texttt{frames}})$ (Lines 8-9).
With VLO strategy, we deterministically optimize the latent $z_{t}$ ( Line 11) in the layout stage ($t>t_{L}$), and optionally switch to time-travel trick ([Algorithm 2](https://arxiv.org/html/2506.07177v1#alg2)) during the detail stage ($t_{L}\geq t>t_{D}$).
After performing latent optimization $M$ times, we proceed to the next denoising step via DDIM [39].
Our Frame Guidance algorithm for flow matching based models, such as Wan [42], is provided in Appendix [C.3](https://arxiv.org/html/2506.07177v1#A3.SS3).

#### Gradient propagation after slicing

Even without processing the full latent sequence, guidance applied to sliced latents produces temporally coherent video.
This temporal coherence arises from the denoising network $v_{\theta}$, which enables the gradient of the guidance loss to propagate through the entire video latents.
We visualize the gradient propagation in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(d), where the guidance applied to the middle frame influences all other frames during the layout stage.
Applying guidance to target frames causes non-targeted frames to become harmonized with the guided ones, as visualized in [Figure 3](https://arxiv.org/html/2506.07177v1#S3.F3) right.
In Appendix [C.4](https://arxiv.org/html/2506.07177v1#A3.SS4), we demonstrate that the temporal structure of the generated video is primarily determined by the denoising network, whereas the contribution of CausalVAE is minimal.

### 4.4 Loss design for various tasks

Frame Guidance is readily applicable to a wide range of frame-conditioned video generation tasks, with appropriately designed guidance loss. Here, we provide simple loss designs for representative frame-conditioned video generation tasks and general user inputs.

Keyframe-guided video generation aims to synthesize videos that transition smoothly between multiple user-specified keyframes, without enforcing strict pixel-level reconstruction. Given an initial image as the input to the I2V model, we minimize a simple *L2 loss* defined as $\mathcal{L}_{e}=\sum_{i\in\mathcal{I}}\|x_{*}^{i}-x_{0|t}^{i}\|_{2}^{2}$, where $x_{*}^{\mathcal{I}}$ denotes the target keyframes and $x_{0|t}^{i}$ is the predicted clean $i$-th frame. The similarity to each keyframe can be controlled by adjusting the guidance strength, such as the number of repeat steps $M$ or step size $\eta$. Unlike training-based approaches [52, 44] that are limited to fixed keyframe positions (e.g., the last frame), our method supports arbitrary keyframe placements.

Stylized video generation aims to synthesize videos in the style of a given reference image using a T2V model. We employ a differentiable style encoder $\Psi$ to compute the *style loss* defined as $\mathcal{L}_{e}=-\sum_{i\in\mathcal{I}}\cos(\Psi(x_{\text{style}}),\Psi(x_{0|t
}^{i}))$, where $x_{style}$ is the style reference image.
We use the Contrastive Style Descriptor (CSD) [38] for $\Psi(\cdot)$, and find that guiding only a few selected (or randomly chosen) frames is sufficient to propagate the desired style across the entire video.

Looped video generation aims to synthesize videos where the first and last frames match, producing a seamless loop using a T2V model. We define the loss as $\mathcal{L}_{e}=\|\operatorname{sg}(x_{0|t}^{1})-x_{0|t}^{L}\|_{2}^{2}$, where $\operatorname{sg}(\cdot)$ denotes the *stop-gradient* operator. This design prevents over-saturation of the generated frames by forcing the last frame to be updated the most to match the first frame.

General input guidance aims to synthesize videos conditioned on general user-specified conditions beyond RGB images, for example, depth maps or sketches.
We use a differentiable encoder $\Psi$, such as a depth estimator [48] or an edge predictor [6], to extract structural features from the estimated clean image. We minimize an encoder-aligned L2 loss defined as $\mathcal{L}_{e}=\sum_{i\in\mathcal{I}}\|\Psi(x_{*}^{i})-\Psi(x_{0|t}^{i})\|_{2
}^{2}$, where $\Psi(x_{*}^{i})$ denotes the encoded target conditions.

## 5 Experiments

### 5.1 Keyframe-guided video generation

We evaluate Frame Guidance on keyframe-guided video generation tasks, which aim to synthesize videos that transition smoothly between multiple user-specified keyframes. As opposed to video interpolation tasks [12, 44] that require exact frame matching, keyframe-guided generation only requires the visual similarity to the keyframes, and addresses the generation of longer videos.

Datasets   We select 40 clips with more than 81 frames from DAVIS [32] dataset and 30 real-world videos from Pexels(^1^11https://huggingface.co/datasets/jovianzm/Pexels-400k (Accessed: 2025-05-07)) dataset. Pexels features more dynamic and human-centric videos than DAVIS, making it more difficult for video generation.
We provide more details on the dataset in Appendix [B.2](https://arxiv.org/html/2506.07177v1#A2.SS2).

Baselines
We compare Frame Guidance against frame interpolation methods, including TRF [12], SVD-Interp [44], and CogX-Interp. TRF is a training-free approach for Stable Video Diffusion (SVD) [3], SVD-Interp uses a fine-tuned reversed-motion SVD, and CogX-Interp(^2^22https://github.com/feizc/CogvideX-Interpolation (Accessed: 2025-05-07)) fine-tunes CogX with first and last frame conditioning. We also compare with basic I2V baselines (CogX [49] and Wan [42]).
For our method, we apply Frame Guidance on CogX and Wan models using the L2 loss defined in [Section 4.4](https://arxiv.org/html/2506.07177v1#S4.SS4) with the final frame given, and further report results additionally using the middle frame. We also report results of applying Frame Guidance to CogX-Interp.

Figure: Figure 5: Qualitative comparison on keyframe-guided video generation tasks. Yellow box indicates the keyframe condition. Orange box in (a) shows a disconnection in SVD-Interp. Red box in (d) visualizes a failure case for the CogX-Interp baseline for dynamic human motion.
Refer to caption: x7.png

Figure: Figure 6: Keyframe-guided generation results. (Left) Human evaluation. (Right) Quantitative results. $I$, $M$, and $F$ denote initial, middle, and final frames, respectively. “Train-free” indicates whether the backbone VDM is fine-tuned for the frame interpolation task, not a base I2V model.
Refer to caption: x8.png

Qualitative comparison   As shown in [Figure 5](https://arxiv.org/html/2506.07177v1#S5.F5), our approach generates videos with natural transitions, where the selected frames closely resemble the keyframes. For example, [Figure 5](https://arxiv.org/html/2506.07177v1#S5.F5)(c) visualizes well-aligned frames, with the paraglider appearing in a consistent position. In contrast, CogX-Interp often struggles with challenging motion. Applying Frame Guidance to CogX-Interp ([Figure 5](https://arxiv.org/html/2506.07177v1#S5.F5)(e)) or to a stronger VDM backbone ([Figure 5](https://arxiv.org/html/2506.07177v1#S5.F5)(f)) results in notably improved output quality.

Human evaluation   We conduct human evaluations to assess the quality of generated videos, focusing on (1) video quality and (2) similarity to the keyframes. As shown in [Figure 6](https://arxiv.org/html/2506.07177v1#S5.F6) left, applying Frame Guidance to Wan yields the highest video quality, surpassing the trained model CogX-Interp.
Applying guidance to CogX-Interp produces high-quality videos with guided frames nearly identical to the keyframes.
Further details are provided in Appendix [B.2](https://arxiv.org/html/2506.07177v1#A2.SS2.SSS0.Px2).

Quantitative results   We measure FID [18] and FVD [13] to assess the quality of the generated videos.
As shown in [Figure 6](https://arxiv.org/html/2506.07177v1#S5.F6) right, Frame Guidance applied to pre-trained I2V models significantly outperforms all other training-free methods. Moreover, Frame Guidance applied to CogX-Interp outperforms all the training-required baselines.
These results, combined with the human evaluation, demonstrate that our method effectively guides video generation without additional training.
We discuss further details regarding the quantitative results in Appendix [B.2](https://arxiv.org/html/2506.07177v1#A2.SS2.SSS0.Px3).

### 5.2 Stylized video generation

We also validate Frame Guidance on stylized video generation tasks, which aim to synthesize videos in the style of a given reference image, using a T2V model.

Dataset   We use a subset of the stylized video dataset introduced in StyleCrafter [26], which consists of 6 challenging style reference images, each paired with an aligned style prompt and 9 distinct content prompts.
We provide further details about the dataset in Appendix [B.3](https://arxiv.org/html/2506.07177v1#A2.SS3).

Baselines   We compare our method with three baselines. CogX-T2V is a pre-trained T2V model. VideoComposer [43] is a training-based method supporting multiple conditions, such as style image and depth maps.
StyleCrafter [26] is also a training-based method that solely trains a style adapter on top of VideoCrafter [7].
For our method, we apply Frame Guidance to the CogX-T2V [49] model using the style loss defined in [Section 4.4](https://arxiv.org/html/2506.07177v1#S4.SS4). We provide more details of our method in Appendix [B.3](https://arxiv.org/html/2506.07177v1#A2.SS3).

Qualitative comparison
Examples in [Figure 7](https://arxiv.org/html/2506.07177v1#S5.F7) show that our method can generate balanced stylized videos in terms of both text alignment and style conformity, with diverse motion.
In contrast, VideoComposer fails to disentangle content and style in the reference images, while StyleCrafter produces videos with minimal motion that are poorly aligned to the reference style. CogX-T2V struggles to capture detailed textures or patterns, for example, geometric shapes or sunflowers.

Human evaluation   We conduct human evaluation to assess the quality of stylized videos, evaluating three criteria (1) style alignment, (2) text alignment, and (3) motion dynamics. As shown in [Figure 8](https://arxiv.org/html/2506.07177v1#S5.F8) left, our method achieves the best results across all criteria, significantly outperforming the training-based baselines.
These results show that Frame Guidance successfully guides video generation to follow the reference style without any additional training.
Further details and the results on overall preference are provided in Appendix [B.3](https://arxiv.org/html/2506.07177v1#A2.SS3.SSS0.Px2).

Figure: Figure 7: Qualitative comparison on stylized video generation. Ours generates high-quality videos that follow the reference style, whereas baselines fail to produce motion or show poor alignment.
Refer to caption: x9.png

Figure: Figure 8: Stylized video generation results. (Left) Human evaluation. (Right) Quantitative results.
Refer to caption: x10.png

Quantitative results   We evaluate text alignment using *CLIP-T* [33] and *ViCLIP-T* [45], and assess style alignment using *CLIP-S* [33] and *ViCLIP-S* [45] on the generated videos. As shown in [Figure 8](https://arxiv.org/html/2506.07177v1#S5.F8) right, our method achieves the best scores on all metrics, except for CLIP-S, where it matches the performance of StyleCrafter. While VideoComposer achieves the highest style alignment scores, this is largely due to replicating the style image without adhering to the text prompt, as in [Figure 7](https://arxiv.org/html/2506.07177v1#S5.F7) left.

### 5.3 Looped video generation

We further apply Frame Guidance on the looped video generation task, which aims to synthesize videos where the first and last frames match, producing a seamless loop.
We use the loop loss defined in Section [4.4](https://arxiv.org/html/2506.07177v1#S4.SS4) to steer the last frame to match the first. Guidance is applied to the generated video *without requiring any external conditions*, using only text prompts as input.
As shown in [Figure 1](https://arxiv.org/html/2506.07177v1#S0.F1)(c) and [Figure 16](https://arxiv.org/html/2506.07177v1#A2.F16), Frame Guidance generates high-quality looped videos featuring dynamic motions that are well-aligned with the input text prompt.

### 5.4 Other applications

Using color block drawing
During keyframe-guided generation, keyframe similarity can be flexibly controlled by adjusting the guidance strength. This allows new forms of user-provided control signals that are easy to create, such as coarse sketches or color blocks.
In particular, we introduce a novel application that allows users to guide video generation using edited frames, where simple visual edits via color blocks indicate changes in color or detail.
As illustrated in [Figure 1](https://arxiv.org/html/2506.07177v1#S0.F1)(d), the generated video depicts the mountain changing color and texture in three distinct ways, which is difficult to achieve using text prompts alone.
For Frame Guidance, color blocks act as rough visual hints that allow natural scene transitions while preserving the contents.
We provide more examples in [Figure 17](https://arxiv.org/html/2506.07177v1#A2.F17).

Masked region guidance
While our previously described methods apply guidance to the entire frame, we demonstrate that the guidance can be effectively restricted to specific regions, by using L2 loss with a binary mask.
In [Figure 9](https://arxiv.org/html/2506.07177v1#S5.F9)(a), we present an example of generating a video with object motion, guided by a cropped image and its segmentation mask. By applying guidance solely to the object region, the background remains unchanged while the object shows smooth movement.

Depth map / Sketch guidance
Furthermore, Frame Guidance supports general types of frame-level signals, such as depth maps and sketches, which offer more user-friendly conditioning compared to RGB images as input.
Using the general input guidance defined in [Section 4.4](https://arxiv.org/html/2506.07177v1#S4.SS4), Frame Guidance is capable of generating high-quality guided videos as shown in [Figure 1](https://arxiv.org/html/2506.07177v1#S0.F1)(e) and (f).

Video style transfer
We extend Frame Guidance to video editing tasks. Taking a video as input, we apply Frame Guidance to generate an edited video that follows a reference style.
Video style transfer can be achieved by applying a simple V2V SDEdit [27] in the detail stage.
This results in preserving the original motion and layout while successfully transferring the reference style, as in [Figure 9](https://arxiv.org/html/2506.07177v1#S5.F9)(b).

Multi condition guidance
Frame Guidance can integrate multiple input types by combining losses. As shown in [Figure 9](https://arxiv.org/html/2506.07177v1#S5.F9)(c), we apply guidance to intermediate frames, combining the depth map loss and sketch loss for the CogX-Interp model. The generated video demonstrates smooth motion that follows the input signals, showing the flexibility of Frame Guidance in handling complex scenarios.

### 5.5 Ablation studies

#### Necessity of VLO

**Table 1: Ablation study on latent optimization strategy.**
| Method | FID $\downarrow$ | FVD $\downarrow$ |
| --- | --- | --- |
| Time-travel | 57.37 | 778.4 |
| Deterministic | 56.61 | 637.3 |
| VLO (Ours) | 55.60 | 577.1 |

To validate the importance of VLO in Frame Guidance, we compare it against two variants: one that uses only the time-travel trick and another that applies only the deterministic update from [Equation 2](https://arxiv.org/html/2506.07177v1#S4.E2) during the guidance process.
[Table 1](https://arxiv.org/html/2506.07177v1#S5.T1) shows that using only the time-travel trick yields higher FVD scores due to difficulty in forming coherent layouts, while the deterministic update alone produces over-saturated or temporally disconnected videos.

Figure: Figure 9: Examples of other applications. (a) Object movement guided by masked region. (b) Video style transfer with SDEdit [27]. (c) Guidance using multiple types of inputs: depth map and sketch.
Refer to caption: x11.png

#### Model agnostic

As shown in [Figure 6](https://arxiv.org/html/2506.07177v1#S5.F6), our method is compatible with a variety of VDMs, including CogVideoX [49], its fine-tuned variant CogVideoX-Interpolation, and Wan-14B [42], a flow-matching-based model. To further demonstrate its generality, we also apply our approach to two additional models: SVD [3], a U-Net-based [34] diffusion model, and LTX-2B [15] which supports sequences up to 161 frames. As illustrated in [Figure 18](https://arxiv.org/html/2506.07177v1#A2.F18), our method consistently performs well across all these VDMs.

## 6 Conclusion

In this work, we present Frame Guidance, a novel training-free framework for diverse control tasks using frame-level signals. By applying guidance to selected frames, our method enables natural control throughout the video. To achieve this, we partially decode sliced latents during guidance computation and introduce a latent optimization strategy designed for video. Our approach supports a wide range of tasks without training, including special cases such as color block guidance and looped video generation. We discuss the limitations and societal impacts of our method in Appendix [D](https://arxiv.org/html/2506.07177v1#A4).

## References

- Bai et al. [2025]
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al.
Recammaster: Camera-controlled generative rendering from a single video.
*arXiv preprint arXiv:2503.11647*, 2025.
- Bansal et al. [2024]
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Roni Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
Universal guidance for diffusion models.
In *International Conference on Learning Representations*, 2024.
- Blattmann et al. [2023]
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.
Stable video diffusion: Scaling latent video diffusion models to large datasets.
*arXiv preprint arXiv:2311.15127*, 2023.
- Brooks et al. [2024]
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.
Video generation models as world simulators.
2024.
URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators).
- Burgert et al. [2025]
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, et al.
Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise.
*arXiv preprint arXiv:2501.08331*, 2025.
- Chan et al. [2022]
Caroline Chan, Frédo Durand, and Phillip Isola.
Learning to generate line drawings that convey geometry and semantics.
In *Conference on Computer Vision and Pattern Recognition*, 2022.
- Chen et al. [2023]
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al.
Videocrafter1: Open diffusion models for high-quality video generation.
*arXiv preprint arXiv:2310.19512*, 2023.
- Chen et al. [2016]
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.
Training deep nets with sublinear memory cost.
*arXiv preprint arXiv:1604.06174*, 2016.
- Chung et al. [2023]
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye.
Diffusion posterior sampling for general noisy inverse problems.
In *International Conference on Learning Representations*, 2023.
- Dhariwal and Nichol [2021]
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans on image synthesis.
*Advances in neural information processing systems*, 2021.
- Efron [2011]
Bradley Efron.
Tweedie’s formula and selection bias.
*Journal of the American Statistical Association*, 106(496):1602–1614, 2011.
- Feng et al. [2024]
Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael J Black, and Xuaner Zhang.
Explorative inbetweening of time and space.
In *European Conference on Computer Vision*, 2024.
- Ge et al. [2024]
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang.
On the content bias in fréchet video distance.
In *Conference on Computer Vision and Pattern Recognition*, 2024.
- Geng et al. [2024]
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al.
Motion prompting: Controlling video generation with motion trajectories.
*arXiv preprint arXiv:2412.02700*, 2024.
- HaCohen et al. [2024]
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al.
Ltx-video: Realtime video latent diffusion.
*arXiv preprint arXiv:2501.00103*, 2024.
- He et al. [2025]
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang.
Cameractrl: Enabling camera control for video diffusion models.
In *International Conference on Learning Representations*, 2025.
- He et al. [2024]
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon.
Manifold preserving guided diffusion.
In *International Conference on Learning Representations*, 2024.
- Heusel et al. [2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 2017.
- Ho et al. [2020]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 2020.
- Hou et al. [2024]
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen.
Training-free camera control for video generation.
*arXiv preprint arXiv:2406.10126*, 2024.
- Jiang et al. [2025]
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu.
Vace: All-in-one video creation and editing.
*arXiv preprint arXiv:2503.07598*, 2025.
- Li et al. [2025a]
Jialu Li, Shoubin Yu, Han Lin, Jaemin Cho, Jaehong Yoon, and Mohit Bansal.
Training-free guidance in text-to-video generation via multimodal planning and structured noise initialization.
*arXiv preprint arXiv:2504.08641*, 2025a.
- Li et al. [2025b]
Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu.
Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance.
*arXiv preprint arXiv:2503.16421*, 2025b.
- Ling et al. [2025]
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin.
Motionclone: Training-free motion cloning for controllable video generation.
In *International Conference on Learning Representations*, 2025.
- Lipman et al. [2022]
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.
Flow matching for generative modeling.
*arXiv preprint arXiv:2210.02747*, 2022.
- Liu et al. [2023]
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Yibo Wang, Xintao Wang, Yujiu Yang, and Ying Shan.
Stylecrafter: Enhancing stylized text-to-video generation with style adapter.
*arXiv preprint arXiv:2312.00330*, 2023.
- Meng et al. [2022]
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
SDEdit: Guided image synthesis and editing with stochastic differential equations.
In *International Conference on Learning Representations*, 2022.
- Nair and Patel [2024]
Nithin Gopalakrishnan Nair and Vishal M Patel.
Dreamguider: Improved training free diffusion-based conditional generation.
*arXiv preprint arXiv:2406.02549*, 2024.
- Namekata et al. [2025]
Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B. Lindell.
SG-i2v: Self-guided trajectory control in image-to-video generation.
In *International Conference on Learning Representations*, 2025.
- OpenAI [2024]
OpenAI.
Gpt-4o system card.
*Technical report*, 2024.
- Polyak et al. [2025]
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas
Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du.
Movie gen: A cast of media foundation models.
*arXiv preprint arXiv:2410.13720*, 2025.
- Pont-Tuset et al. [2017]
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.
The 2017 davis challenge on video object segmentation.
*arXiv preprint arXiv:1704.00675*, 2017.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *International Conference on Machine Learning*, 2021.
- Ronneberger et al. [2015]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, 2015.
- Rout et al. [2025]
Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu.
RB-modulation: Training-free stylization using reference-based modulation.
In *International Conference on Learning Representations*, 2025.
- Salimans and Ho [2022]
Tim Salimans and Jonathan Ho.
Progressive distillation for fast sampling of diffusion models.
In *International Conference on Learning Representations*, 2022.
- Shen et al. [2024]
Yifei Shen, Xinyang Jiang, Yifan Yang, Yezhen Wang, Dongqi Han, and Dongsheng Li.
Understanding and improving training-free loss-based diffusion guidance.
In *Advances in Neural Information Processing Systems*, 2024.
- Somepalli et al. [2024]
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein.
Measuring style similarity in diffusion models.
*arXiv preprint arXiv:2404.01292*, 2024.
- Song et al. [2020]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv preprint arXiv:2010.02502*, 2020.
- Song et al. [2021]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
- von Platen et al. [2022]
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf.
Diffusers: State-of-the-art diffusion models.
[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers), 2022.
- Wang et al. [2025a]
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al.
Wan: Open and advanced large-scale video generative models.
*arXiv preprint arXiv:2503.20314*, 2025a.
- Wang et al. [2023a]
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.
Videocomposer: Compositional video synthesis with motion controllability.
*Advances in Neural Information Processing Systems*, 2023a.
- Wang et al. [2025b]
Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher-Shlizerman, Aleksander Holynski, and Steve Seitz.
Generative inbetweening: Adapting image-to-video models for keyframe interpolation.
In *International Conference on Learning Representations*, 2025b.
- Wang et al. [2023b]
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al.
Internvid: A large-scale video-text dataset for multimodal understanding and generation.
*arXiv preprint arXiv:2307.06942*, 2023b.
- Wang et al. [2024]
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.
Motionctrl: A unified and flexible motion controller for video generation.
In *ACM SIGGRAPH*, 2024.
- Wu et al. [2024]
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang.
Draganything: Motion control for anything using entity representation.
In *European Conference on Computer Vision*, 2024.
- Yang et al. [2024]
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao.
Depth anything v2.
*Advances in Neural Information Processing Systems*, 2024.
- Yang et al. [2025]
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan.Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang.
Cogvideox: Text-to-video diffusion models with an expert transformer.
In *International Conference on Learning Representations*, 2025.
- Yu et al. [2023]
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.
Freedom: Training-free energy-guided conditional diffusion model.
In *International Conference on Computer Vision*, 2023.
- Yu et al. [2024]
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang.
Language model beats diffusion - tokenizer is key to visual generation.
In *International Conference on Learning Representations*, 2024.
- Zeng et al. [2024]
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li.
Make pixels dance: High-dynamic video generation.
In *Conference on Computer Vision and Pattern Recognition*, 2024.
- Zheng et al. [2024]
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li.
Cami2v: Camera-controlled image-to-video diffusion model.
*arXiv preprint arXiv:2410.15957*, 2024.

#### Organization

The Appendix is organized as follows: In Section [A](https://arxiv.org/html/2506.07177v1#A1), we provide the additional backgrounds of our work. We describe the details of the experiments and our framework in Section [B](https://arxiv.org/html/2506.07177v1#A2), and further discussion in Section [C](https://arxiv.org/html/2506.07177v1#A3). Lastly, in Section [D](https://arxiv.org/html/2506.07177v1#A4), we discuss the limitations of our work.

## Appendix A Backgrounds

### A.1 Training-free diffusion guidance

Recent works [40, 10, 50, 9, 2, 17, 37] have explored conditional generation by injecting external conditions into pre-trained diffusion models. Among them, training-free guidance methods [50, 9, 2, 17, 37] achieve controllable generation without additional training by optimizing the noisy latent during the reverse process. This optimization is guided by a loss function that measures the alignment between intermediate latents and the target condition at each denoising step. FreeDom [50] and UniversalGuidance [2] leverage off-the-shelf models to compute the various guidance losses, achieving a wide range of controllable image generation tasks. Later works [17, 28, 35] bypass the denoising module for computing the guidance loss, enabling more efficient training-free diffusion guidance.

### A.2 Flow matching

Flow matching [25] belongs to the family of flow-based generative models, which are known for faster sampling compared to diffusion models [19]. Let $t\in[0,1]$ be the time, $x\in^{d}$ be a data, and $q$ be a unknown target distribution. The goal of flow matching [25] is to estimate a time-dependent transformation $z_{t}:[0,1]\times^{d}\to^{d}$ (referred to as flow) that maps a prior distribution $p_{0}$ (e.g., Gaussian) to a distribution $p_{1}\approx q$. Instead of directly estimating the flow, Lipman et al. [25] proposes to regress a generating vector field $v_{t}(\cdot,t):[0,1]\times^{d}\to^{d}$ that induces the flow $z_{t}$ via the following ordinary differential equation (ODE):

$$ $\frac{dz_{t}(x)}{dt}=v_{t}(z_{t}(x))\quad\text{and}\quad z_{0}(x)=x.$ (3) $$

It is common practice to design this flow $\phi_{t}$ along an optimal transport (OT) trajectory that connects a prior sample to a target sample with a straight interpolation: $z_{t}:=(1-t)x_{0}+tx_{1}$, where $x_{0}\sim p_{0}$ and $x_{1}\sim q$. In this case, the target $v_{t}$ is computed as a constant: $v_{t}(x,t)=x_{1}-x_{0}$ for all $t\in[0,1]$. With a neural network $v_{\theta}$ that estimates $v_{t}$, we can generate a data $x_{1}$ by numerically solving the ODE in [Equation 3](https://arxiv.org/html/2506.07177v1#A1.E3) (e.g., Euler method). Similar to Tweedie’s formula [11], we can approximate a cleaned sample at each time $t$ by

$$ $z_{1|t}:=z_{t}+\frac{1}{1-t}v_{\theta}(z_{t},t).$ (4) $$

Throughout this paper, we interchangeably reverse the direction of time by parameterizing it as $s(t)=T(1-t),\leavevmode\nobreak\ t\in[0,1]$ to align with the convention of the diffusion models where the generative process proceeds from $T$ to 0 0.

## Appendix B Experimental details

### B.1 Implementation details

All our experiments are conducted on a single H100 GPU. Hyperparameters related to guidance, such as step size $\eta$ and repetition $M$, are adjustable depending on the task and model characteristics.
For example, in keyframe-guided video generation using diffusion-based CogVideoX [49], we define the layout stage within the first 5 steps, set $M=10$, and use a step size of $\eta=3.0$. For the time-travel trick, $M$ is linearly decreased over 15 steps. At each step, gradients are L2-normalized before being scaled by $\eta$ for the update.

Since Wan-14B [42] employs flow matching as its generative modeling, its inference is fully deterministic, and the layout is mostly established within 2 steps. Therefore, we set the layout stage to the first 2 inference steps, and apply the same $M,\eta$, and time-travel configuration. Moreover, since our implementation introduces more stochasticity (see Appendix [C.3](https://arxiv.org/html/2506.07177v1#A3.SS3)), we slightly reduce the number of time-travel steps.
To maintain practicality, we empirically limit the number of guidance steps such that the overall runtime does not exceed $4\times$ the base model’s inference time.

To reduce GPU memory usage, we apply gradient checkpointing [8] to the denoising network using the Diffusers [41] library. For the CausalVAE, gradient checkpointing is applied only in CogVideoX [49], as Wan-14B [42] implementation does not currently support it. We do not apply spatial downsampling in CogVideoX, since it runs on a single GPU without it. In contrast, we apply $2\times$ spatial downsampling in experiments with Wan-14B.

### B.2 Keyframe-guided video generation

#### Dataset

For evaluation, we use videos from the DAVIS [32] dataset and Pexels. From DAVIS, we select 40 videos with at least 81 frames, matching the maximum frame length supported by Wan-14B [42]. The resolution of each video is resized and center-cropped according to the requirements of each pre-trained model. To ensure fair comparisons across models, the same initial and final frames are used. Based on this setup, the reference set for each model is configured with slightly different FPS settings. For example, for an 81-frame video, CogVideoX [49] supports only 49 frames, so we temporally downsample the video accordingly. The Pexels dataset contains more real-world videos with challenging motions and frequent camera view changes. We randomly select a subset of 30 videos, which features more dynamic and human-centric content compared to DAVIS.

For pre-trained models that accept text prompts as input, except for Stable Video Diffusion [3](SVD)-based methods [12, 44], we used prompts derived from the original videos. Specifically, we concatenated three frames from each original video and generated a caption using GPT-4o [30]. The same prompt was applied consistently across all baseline models.

Figure: Figure 10: A screenshot of questionnaires from our human evaluation on keyframe-guided generation.
Refer to caption: x12.png

Figure: Figure 11: Human evaluation results on keyframe-guided generation including Wan-I2V.
Refer to caption: x13.png

#### Human evaluation

We conduct human evaluation for keyframe-guided video generation task to evaluate two main aspects: (1) video quality and (2) similarity to the keyframes. Both metrics are rated on an absolute scale from 1 to 5. As shown in [Figure 10](https://arxiv.org/html/2506.07177v1#A2.F10), participants evaluated all videos generated from the same keyframes side by side. We collected responses from 20 participants, evaluating 5 types of videos across 5 different methods. The full human evaluation results, including Wan-I2V, are provided in [Figure 11](https://arxiv.org/html/2506.07177v1#A2.F11).

#### Evaluation metric

For evaluation metric, we employ FID [18] and content-debiased FVD [13] between generated videos and real videos. Both metrics quantify the distributional distance between generated videos and real videos from the dataset. FID is computed by extracting all frames from the video and treating them as individual images. FVD is measured against reference videos adjusted to match each model’s resolution and FPS. Therefore, cross-model comparisons are not strictly valid.

As shown in [Figure 6](https://arxiv.org/html/2506.07177v1#S5.F6) right, our method with Wan slightly outperforms Wan I2V in these quantitative metrics. However, human evaluations in [Figure 6](https://arxiv.org/html/2506.07177v1#S5.F6) left suggest a more noticeable improvement, which may not be fully captured by such metrics. Notably, the overall FID and FVD scores are relatively high, as our setting involves longer and more dynamic videos compared to related tasks such as video interpolation, making the dataset more challenging.

We provide more qualitative examples in [Figure 14](https://arxiv.org/html/2506.07177v1#A2.F14).

### B.3 Stylized video generation

Based on our analysis of layout formation in [Section 4.2](https://arxiv.org/html/2506.07177v1#S4.SS2), we apply VLO with a different schedule for stylized video generation compared to keyframe-guided video generation. Specifically, we start applying the deterministic latent update ([Equation 2](https://arxiv.org/html/2506.07177v1#S4.E2)) at step 3 before entering the detail stage (step 5), and then switch to time travel during steps 15 - 20. This design helps shape the geometric patterns and structure of the style reference image during the layout stage. After that, we proceed the inference without guidance. We set the guidance step size $\eta=3$ and the number of repetition $M=5$. We compute the style guidance loss on 4 evenly spaced frames from the entire video.

Figure: Figure 12: A screenshot of questionnaires from our human evaluation on stylized video generation task.
Refer to caption: x14.png

Figure: Figure 13: Human evaluation results on stylized video generation including overall preference.
Refer to caption: x15.png

#### Dataset

We use a subset of the test dataset introduced in StyleCrafter [26], which consists of 9 content prompts and 6 style reference images with corresponding style descriptions. In [Table 3](https://arxiv.org/html/2506.07177v1#A2.T3) and [Table 3](https://arxiv.org/html/2506.07177v1#A2.T3), we detail our test dataset. The content prompts describe an entire video content using a simple sentence, while the style prompts describe the styles of the video. The style prompts are generated by GPT-4o [30]. We concatenate each content prompt with each style prompt, resulting in a total of 54 full prompts for stylized video generation.

#### Human evaluation

In [Figure 12](https://arxiv.org/html/2506.07177v1#A2.F12), we provide screenshots of the questionnaires and labeling instructions. 20 participants are asked to evaluate four metrics: (1) style alignment, (2) text alignment, (3) motion dynamics, and (4) overall video preference of five stylized videos generated by four models. All metrics were rated on an absolute scale from 1 to 5. The complete evaluation results, including overall preference, are provided in [Figure 13](https://arxiv.org/html/2506.07177v1#A2.F13).

#### Evaluation metric

We employ CLIP-Text and ViCLIP-Text to access the text alignment of the generated videos. We also compute CLIP-Style and ViCLIP-Style to access the style conformity of the generated videos. Specifically, CLIP-Text and CLIP-Style are computed by using the CLIP [33] text and image encoders, respectively:

$$ $\frac{1}{L}\sum_{l=1}^{L}\frac{f_{I}(x_{l})\cdot f_{T}(p)}{\|f_{I}(x_{l})\|_{2 }\|f_{T}(p)\|_{2}}\quad\text{and}\quad\frac{1}{L}\sum_{l=1}^{L}\frac{f_{I}(x_{ l})\cdot f_{I}(x_{\text{style}})}{\|f_{I}(x_{l})\|\|_{2}f_{I}(x_{style})\|_{2}},$ (5) $$

where $x_{l}$ is the $l$-th frame, $p$ is the text prompt, $x_{\text{style}}$ is the style reference image, and $f_{I}(\cdot)$ and $f_{T}(\cdot)$ are the CLIP [33] image and text encoders, respectively.

Similarly, ViCLIP-Text and ViCLIP-Style are both computed by using Video CLIP model [45]:

$$ $\frac{f_{V}(x)\cdot f_{T}(p)}{\|f_{V}(x)\|_{2}\|f_{T}(p)\|_{2}}\quad\text{and} \quad\frac{f_{V}(x)\cdot f_{T}(p_{\text{style}})}{\|f_{V}(x)\|_{2}\|f_{T}(p_{ \text{style}})\|_{2}},$ (6) $$

where $x$ is the video, $p$ and $p_{\text{style}}$ are the full and style prompts, and $f_{V}(\cdot)$ and $f_{T}(\cdot)$ are the ViCLIP video and text encoders, respectively.

We provide more qualitative examples in [Figure 15](https://arxiv.org/html/2506.07177v1#A2.F15).

**Table 2: Text prompts [26] used for stylized video generation.**
| Content prompt | Content prompt |
| --- | --- |
| A street performer playing the guitar. | A wolf walking stealthily through the forest. |
| A chef preparing meals in kitchen. | A hot air balloon floating in the sky. |
| A student walking to school with backpack. | A wooden sailboat docked in a harbor. |
| A bear catching fish in a river. | A field of sunflowers on a sunny day. |
| A knight riding a horse through a field. |  |

### B.4 Loop video generation

We use the similar guidance schedule with keyframe-guided video generation task, but reduce the early guidance strength to avoid producing over-saturated examples. We provide more qualitative examples in [Figure 16](https://arxiv.org/html/2506.07177v1#A2.F16).

### B.5 Additional generated examples

We provide more examples on Frame Guidance with color block image in [Figure 17](https://arxiv.org/html/2506.07177v1#A2.F17). We show examples generated by other models, SVD [3] and LTX-2B [15], are shown in [Figure 18](https://arxiv.org/html/2506.07177v1#A2.F18).

Figure: Figure 14: Qualitative comparison of keyframe-guided video generation. Orange arrows indicate temporally disconnected frames, and red boxes highlight poor keyframe similarity. Our method generates temporally coherent videos while maintaining semantic similarity to the keyframes.
Refer to caption: x16.png

Figure: Figure 15: Stylized video generated by Frame Guidance using style loss. These videos are generated by CogVideoX-T2V.
Refer to caption: x17.png

Figure: Figure 16: Loop video generated by Frame Guidance using loop loss. These videos are generated by Wan-14B T2V.
Refer to caption: x18.png

Figure: Figure 17: Frame Guidance with a color block image allows the generation of a video with a complex scene. These videos are generated by CogVideoX-I2V.
Refer to caption: x19.png

Figure: Figure 18: Frame Guidance is model-agnostic. It is compatible with both SVD [3] and LTX-2B [15]. For SVD, since it does not use a temporally compressed VAE, we skip latent slicing. Some saturation observed in the LTX-2B results occasionally occurs due to the model itself.
Refer to caption: x20.png

## Appendix C More discussions

Figure: Figure 19: Video reconstruction with temporally sliced latent. (a) Decoding the full latent sequence successfully reconstructs the original video. (b)–(c) Using 4 or 3-length latent around the target latent (frame) is sufficient for accurate reconstruction. (d) With only 2-length latent, there is slight degradation, therefore, we adpot 3-length latent for the main experiments.
Refer to caption: x21.png

### C.1 Video reconstruction with sliced latent

As shown in [Figure 19](https://arxiv.org/html/2506.07177v1#A3.F19), we can reconstruct nearly identical frames even with temporally sliced latents. After encoding the dynamic original video using CogVideoX’s CausalVAE, decoding with only a 3-length latent sequence yields frames that closely match the original ([Figure 19](https://arxiv.org/html/2506.07177v1#A3.F19)(c)).

### C.2 Time-travel trick in layout stage

**Table 4: Forward process coefficients in early inference steps.**
| Step ($\cdot$/50) | $\sqrt{\beta_{t}}$ | $\sqrt{1-\beta_{t}}$ |
| --- | --- | --- |
| 1 | 0.00 | 1.00 |
| 2 | 0.48 | 0.88 |
| 3 | 0.64 | 0.77 |

As discussed in Section [4.2](https://arxiv.org/html/2506.07177v1#S4.SS2.SSS0.Px2), directly applying the time-travel trick [37, 50, 2] to video diffusion models struggles due to excessive stochasticity. The time-travel trick in [Algorithm 2](https://arxiv.org/html/2506.07177v1#alg2) includes a single-step forward process, but in practice, the added noise is extremely large, and the coefficient multiplied with the latent is very small, as shown in [Table 4](https://arxiv.org/html/2506.07177v1#A3.T4). In fact, in the very first inference step, the coefficient $\sqrt{\beta_{t}}$ becomes 0, resulting in no guidance effect at all. Therefore, since the effect of guidance is absent during the early stages when the layout is largely established, the model fails to produce a layout that aligns with the given condition. Even when guidance is applied later, as discussed in [Figure 4](https://arxiv.org/html/2506.07177v1#S4.F4)(d), only the guided frame is updated, and it cannot correct the overall layout. Our proposed VLO addresses this issue by applying a deterministic latent optimization in the layout stage.

Figure: Algorithm 2 Time Travel

### C.3 Video latent optimization (VLO) for flow matching

As noted in Section [A.2](https://arxiv.org/html/2506.07177v1#A1.SS2), we follow the time convention of diffusion models by reversing the flow matching time axis, aligning $t=0$ with clean data and $t=T$ with pure noise.

In [Algorithm 4](https://arxiv.org/html/2506.07177v1#alg4), we extend our Frame Guidance to video generation models, which employ the flow matching [25] for their generative modeling (e.g., Wan [42] and LTX [15]). Similar to the diffusion case in [Equation 2](https://arxiv.org/html/2506.07177v1#S4.E2), we apply the latent slicing (Lines 7) and optimize the current latent $z_{t}$ through the guidance loss $g_{t}$ (Lines 9-11). Specifically, we predict the clean sample $z_{0|t}$ by based on the tweedie-like formula in [Equation 4](https://arxiv.org/html/2506.07177v1#A1.E4).

#### Time-travel for flow matching

However, directly applying the time-travel trick to flow matching is non-trivial, as a single forward step (Line 5 in [Algorithm 2](https://arxiv.org/html/2506.07177v1#alg2)) is not explicitly defined in the context of flow matching. While renoising in time travel is effective for mitigating accumulated sampling errors, it cannot be directly utilized here.
Our deterministic optimization excludes renoising entirely and can be applied as is, but performing it fully during inference, as in diffusion, can result in over-saturated samples or temporally disconnected videos.

To address this, we adopt a simple alternative: instead of stepping from $t$ to $t{-}1$, we move directly from $t$ to 0 (i.e., the estimated clean latent), apply guidance there, and then simulate a forward step from 0 back to $t$. Although a single forward step is not defined in flow matching, it is still possible to apply the forward process for time $t$ from clean data. While this process introduces higher stochasticity than a single diffusion step, applying it in the later stages of VLO, after the layout has already been established, does not significantly disrupt the structure. This makes it a viable option. Empirically, this approach enables the application of VLO to flow matching-based models as well.

Figure: Figure 20: Shortcut-based approaches [17, 35, 28] lead to temporal disconnects in video generation.
Refer to caption: x22.png

### C.4 Importance of gradient propagation via denoising network

In training-free guidance for image generation, a "shortcut" [35, 17] method has been proposed that utilizes a proximal gradient approach to bypass back-propagation through the denoising network. This strategy significantly reduces memory usage and enables efficient sampling for gradient-based optimization. While effective for static images, directly applying this method to video generation poses challenges due to the temporal characteristic of video data.

Specifically, when guidance is applied to only a few frames, the resulting video often becomes temporally inconsistent. As illustrated in [Figure 20](https://arxiv.org/html/2506.07177v1#A3.F20), the latents corresponding to the guided frames are updated to resemble the target frames, and adjacent frames may also partially align. However, earlier frames remain disconnected, and the guided frames themselves may exhibit unnatural artifacts. This is because temporal priors, crucial for maintaining coherence across frames, are primarily encoded in the denoising network. Consequently, for video generation tasks where temporal consistency is critical, gradient propagation through the denoising network is essential.

## Appendix D Limitations and societal impacts

#### Limitations

Although Frame Guidance is training-free and supports various applications, it has some limitations:

(1) The computational cost of guidance sampling is higher than that of training-based methods. Since it requires back-propagation and multiple predictions, the inference speed is approximately up two to four times slower than that of the base model, depending on the task. This issue is particularly significant in video generation, which is computationally intensive. We leave addressing this inefficiency to future work.

(2) While our method is model-agnostic, it is heavily dependent on the performance of the base model. Since our approach samples videos that align with given conditions within the generation distribution of the base model, it struggles to generate videos that are either too dynamic or contain fine-detailed objects the model has not encountered during training.

#### Societal impacts

The proposed method has potential benefits in areas such as animation, education, and creative content generation. However, there is also a risk of misuse, particularly in the creation of deceptive or harmful content such as deepfakes. To address these concerns, it is important to incorporate safeguards such as watermarking and to advance detection methods that can identify manipulated or synthetically generated videos.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How does the temporal locality of CausalVAE generalize to other VAE architectures or non-causal latent spaces?
2. Could the gradient propagation analysis (Fig. 4c) be extended to quantify temporal coherence, e.g., via optical flow consistency?
3. For multi-condition guidance, how are conflicting losses (e.g., style vs. depth) balanced? Is there a risk of gradient domination?
4. Have you explored adaptive strategies for selecting guided frames (e.g., based on motion complexity) to further reduce computational cost?
soundness: 3
strengths: Among the strong points of the paper I would focus on the following ones:
1.  Introduces a general-purpose, training-free framework for frame-level control, addressing a gap between task-specific training-free methods and general-purpose fine-tuning approaches.
2. Rigorous experiments across multiple VDMs (e.g., CogVideoX, Wan, SVD) and tasks, supported by human evaluations and metrics (FID, FVD, CLIP scores).
3. Latent slicing reduces GPU memory usage by up to 60×, enabling application to large-scale models.
4. Detailed algorithms, hyperparameters, and ablation studies are provided, including adaptations for flow-matching models.

The method’s generality across models (CogVideoX, Wan, SVD) and tasks (keyframing, stylization, looping) demonstrates its broad applicability. Human evaluations and quantitative metrics show superior or competitive performance against training-based baselines. The memory efficiency achieved via latent slicing is critical for scaling to modern VDMs. The hybrid VLO strategy effectively balances layout coherence and detail preservation, addressing limitations of prior training-free methods. These contributions are valuable for the community, enabling accessible and flexible video control without costly fine-tuning.
summary: This paper proposes Frame Guidance - a novel training-free guidance framework for controllable video generation using frame-level signals such as keyframes, style references, sketches, or depth maps. The authors claim three primary contributions: (1) a model-agnostic, training-free approach compatible with diverse video diffusion models (VDMs); (2) "latent slicing," a memory-efficient technique for partial decoding of video latents based on temporal locality in CausalVAEs; and (3) "video latent optimization (VLO)," a hybrid strategy combining deterministic updates in early denoising steps for global coherence and stochastic updates later for detail refinement. The method is demonstrated across tasks like keyframe-guided generation, stylization, and looping, achieving competitive results without fine-tuning.
weaknesses: The weak points are:
1. Guidance increases inference time by 2–4×, limiting real-time applicability.
2. Performance is constrained by the base VDM’s capabilities, especially for dynamic or fine-grained content.
3. While latent slicing is justified via experiments on CausalVAE, broader validation across architectures is limited.
4. Although multi-condition guidance is shown, combining losses for complex controls (e.g., motion + style) is not deeply explored.
5. As you know, one of the main problems in video generation is the object state change, like melting ice, etc. There is a number of training-free solutions for such tasks, e.g., "State & Image Guidance: Teaching Old Text-to-Video Diffusion Models New Tricks" (https://openreview.net/forum?id=zkGxROm7D3). Deeper comparison would be a benefit of the proposed paper.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Are consistent seeds used in Fig. 6 and 7? In Fig. 7, can the method improve style alignment without altering content? Was the same seed used? 
2. What is the runtime overhead of the time-travel strategy? 
3. How does the method handle large motion or occlusion? Is the slicing strategy sensitive to motion scale and object scale?
4. How does the method prevent generating identical frames in looped videos?
5. What does “N-length latent” mean in Figure 19? Which frames are anchored for each latent segment? How are occluded or missing objects decoded reliably?
soundness: 3
strengths: 1. Originality: While the paper builds on existing concepts like latent optimization, its integration into a training-free video generation framework with frame-level control is novel and practical.
2. Quality: The method is well-engineered and demonstrates compatibility with multiple video diffusion backbones. The results show improved coherence and controllability across tasks.
3. Clarity: The paper is generally well-written and structured. The methodology is explained clearly, and the figures help illustrate the effects of guidance.
4. Significance: Training-free methods are increasingly important for accessibility and scalability. Frame Guidance contributes meaningfully to this direction by enabling flexible control without retraining, which is valuable for both research and real-world applications.
summary: The paper introduces Frame Guidance, a training-free framework for controllable video generation using video diffusion models. It enables frame-level control through latent optimization and guidance, supporting tasks like stylization, looping, and keyframe-based generation. By operating in the latent space, the method reduces memory usage while enhancing global coherence across frames. Its plug-and-play compatibility with existing video diffusion models allows versatile application across tasks without retraining.
weaknesses: 1. Limited Novelty: The core techniques (e.g., latent optimization and latent sliding) are adaptations of existing methods. The novelty lies more in the integration and application than in the underlying algorithms. The proposed did show improved performance but with basis from its underlying video diffusion model.
2. The latent slicing strategy, while memory-efficient, may be sensitive to frame rate and motion magnitude. In cases of large motion or occlusion, it may fail to maintain coherence. No quantitative evaluation is provided to assess this. The meaning of “N-length latent” in Fig.19 is unclear, and it is not specified which frames are anchored for each latent segment. Some frames show object disappearance or occlusion, raising questions about how they are decoded reliably for other cases.
Compared to more robust video compression models like LTX-Video use 8 temporal compression and 32 spatial compression, the slicing strategy feels more like a heuristic trick than a principled solution.
3. The looping task lacks detail on how the method avoids generating identical frames, which is critical for realistic looping.
4. The time-travel strategy lacks runtime analysis, and the pseudo-code appears incorrect or incomplete.
5. The paper does not include a discussion of failure cases.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to above weaknesses.
soundness: 3
strengths: 1. This paper proposes a training-free guidance method that eliminates the need for fine-tuning large-scale video models for controllable generation. 
2. This paper introduces a memory-efficient latent processing technique and optimization strategy that enables practical application on large-scale models while ensuring temporal coherence. 
3. This paper demonstrates versatility across diverse control tasks including keyframes, stylization, and looping, with compatibility across any video generation model.
summary: This passage introduces Frame Guidance, a training-free method for controllable video generation that uses frame-level signals like keyframes, style references, or depth maps applied to selected frames to guide entire videos while maintaining temporal coherence. To enable this approach on large-scale models, the authors propose a memory-efficient latent processing method and a novel optimization strategy for globally coherent generation. The method works across diverse tasks including keyframe guidance, stylization, and looping, is compatible with any video model, and produces high-quality controlled videos without requiring training.
weaknesses: 1. The authors mention in the limitations that "The computational cost of guidance sampling is higher than that of training-based methods." Please provide timing comparisons in the rebuttal to better benchmark the method against alternatives.
2. This paper focuses on training-free controllable video generation, but the related works section lacks discussion of previous relevant methods, such as Tune-A-Video[1], Text2Video-Zero[2], and ControlVideo[3].
3. Can the proposed method be extended to control video generation using canny edges, depth maps, and other modalities?

[1] "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." In ICCV 2023
[2] "Text2video-zero: Text-to-image diffusion models are zero-shot video generators." In ICCV 2023
[3] "ControlVideo: Training-free Controllable Text-to-Video Generation." In ICLR 2024.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to weakness.
soundness: 3
strengths: 1.  Training-Free and Model-Agnostic Control: The paper proposes Frame Guidance, a novel training-free approach for controlling video generation in pre-trained Video Diffusion Models (VDMs). This is a significant advantage as it avoids the need for costly fine-tuning for specific tasks or models, making it broadly applicable to various existing and future VDMs. The model-agnostic nature is well-demonstrated across different architectures.

2.  General-Purpose Frame-Level Guidance: The method offers a unified framework for diverse frame-level control tasks. It supports a wide range of input signals beyond simple keyframes, including style reference images, depth maps, sketches, and color blocks, showcasing its versatility for different creative applications (e.g., keyframe guidance, stylization, looping, structure guidance).

3.  Efficient Guidance Strategy for Large Models: The paper introduces two key technical contributions, Latent Slicing and Video Latent Optimization (VLO), to enable efficient training-free guidance on large-scale VDMs. Latent Slicing cleverly exploits the observed temporal locality in CausalVAE latents to dramatically reduce memory consumption during gradient computation. VLO provides a tailored optimization strategy for videos, balancing global layout coherence (early deterministic updates) with detail refinement (later stochastic updates).
summary: This paper introduces Frame Guidance, a novel training-free and model-agnostic framework for controlling video generation in pre-trained Video Diffusion Models (VDMs). The method aims to provide general-purpose, frame-level control using diverse input signals like keyframes, style images, depth maps, or sketches. By applying guidance to only selected frames, it steers the entire video generation process towards temporally coherent results. Key technical contributions include "Latent Slicing," which leverages observed temporal locality in VDM latents (specifically CausalVAE) to reduce memory usage during gradient computation, and "Video Latent Optimization" (VLO), a hybrid optimization strategy balancing early deterministic updates for global layout and later stochastic updates for detail refinement.

However, I think the evaluation seems a little bit out of dated. More recent Vbench and Vbench2.0 evaluation results are needed to further verify the effectiveness of the proposed method.
weaknesses: The evaluation metrics used (FID, FVD) are insufficient for this task. While I am not an expert in training-free video editing, relying solely on FID/FVD seems questionable for evaluating frame-level control. These metrics may not adequately capture temporal consistency, fine-grained adherence to control signals, or motion quality. The paper should ideally include results from more comprehensive video generation benchmarks like VBench to provide a more convincing assessment. (If the authors can supplement the evaluation with VBench results, I will raise my score based on discussions with other reviewers.)


Decision: Accept (Poster)