Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models
Authors: Sangwon Jang
Taekyung Ki
Jaehyeong Jo
Jaehong Yoon
Soo Ye Kim
Zhe Lin
Sung Ju Hwang
Venue: ICLR 2026 Poster (2026)
Keywords: controllable video generation
training-free guidance
video diffusion models
Primary area: generative models
TL;DR: We present Frame Guidance, a training-free framework that supports diverse control tasks using frame-level signals.

Abstract: Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. By applying guidance to only a few selected frames, Frame Guidance can steer the generation of the entire video, resulting in a temporally coherent controlled video. To enable training-free guidance on large-scale video models, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, and is compatible with any models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.

Published as a conference paper at ICLR 2026
FRAME GUIDANCE: TRAINING-FREE GUIDANCE FOR FRAME-LEVEL CONTROL IN VIDEO DIFFUSION MODELS
Sangwon Jang
Taekyung Ki
Jaehyeong Jo
Jachong Yoon? Soo Ye Kim
Zhe Lin
Sung Ju Hwang!
KAIST NTU Singapore" Adobe Research DeepAuto.ai"
sangwon, jang, taekyung. ki,
sungju. hwang } @kaist.ac.ki
First
Time
Time i
Keyframes
"A grey SUV driving along winding mountain road through a forested landscape...
(a) Key video generation
Style
Style
"A New York City a man dog watercolor painting style."
"A bustling Paris cale waiters dog cubist aesthetic style."
(b) Stylized video generation
First
Last "Three green mountains transformine forming into different seasons..."
"A red monster toy jumping on a trampoline slow motion..."
(c) Loop video generation
(d) Color block guidance
"A hand delicately places a thinly sliced radish on top ..."
"A solitary hiker.. clad in sun hat and mountain trail."
(c) Depth map guidance
(f) Sketch guidance
Figure I: Frame Guidance enables training-free controllable video generation using flexible frame- level inputs. It supports diverse applications, including keyframe eeyframe-guided generation, stylization, and looping, using general frame-level inputs such as depth maps, sketches, and color blocks.
ABSTRACT
Advancements in diffusion models have significantly improved video quality, directing attention to fine- mne-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. By applying guidance only few selected frames, Frame Guidance can steer the generation of the entire video, resulting in temporally coherent controlled video. To enable training-free guidance large-scale video models, we propose simple latent processing method that dramatically reduces memory usage, and apply novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, and is compatible with any models. Experimental results show that Frame Guidance can produce high-que gugh-qualtty controlled videos for wide range of tasks and input signals. See our project page: https: // frame- gui dance- vi video. g thub. io/.
Equal contribution.

---

Published as a conference paper at ICLR 2026
Lateni
Controls
Other frames also adjust naturally
RGB
Depch
Sketch
Style
Any
Color Block
Frame-level Conditions
Controllable video generation
Figure 2: Frame Guidance steers the video generation process of a VDM by applying gradient- based guidance selected frames, resulting in temporally coherent controlled video. Our method is training-free, agnnss-g,,,tic, and supports a wide range of frame-level conditions.
INTRODUCTION
The rapid advancement of diffusion models (Ho et al., 2020; Song al., 2021; Lipman al., 2022) has led to the development of powerful video generation models. Recent large-scale video diffusion models (VDMs) have made significant progress in high-quality text-to-video (T2V) and image-to- video (12V) generation, which are capable of generating diverse and realistic video content (Brooks al., 2024; Polyak al., 2025; Yang al., 2025; Wang al., 2025a). With ongoing advancements, there is growing interest in enabling more fine-grained control over the generation process.
Recent progress underscores the need for a practical approach controllable video generation. Hence, we identify two major desiderata: (1) model-agnostic,, training-fres framework, and (2) general-purposs guidance method. Existing methods (Burgert al., 2025; He et al., 2025; et al., 2025b) typically fine-tune large-scale VDMs (Yang et al., 2025; Wang al., 2025a) for each specific control task, which is increasingly impractical due to high computational cost and the burden of retraining with every new model release. This highlights the need for training-free guidance methods that work across models. Moreover, end users prefer simple, generalizable frameworks that support diverse tasks and inputs, such reference images, depth maps, sketches, rather than task-specific models (Hou et al., 2024; Wang et al., 2025b) that are restricted to a fixed input type.
Existing methods fall short of satisfying both desiderata simultaneously: training-free frree ap- proaches (Ling et al., 2025; Hou et al., 2024) are often task-specific and lack generalizability. while general-purpose methods (Li et al., 2025b; Jiang et al., 2025) require fine-tuning and need substantial training resources. Many existing methods (Wang et al., 2025b; 2024; Bai et al., 2025) are both sask-specific and training-dependenm making them difficult to adapt to new models or tasks.
In this work, we propose Frame Guidance, a novel guidance method for VDMs that is model-agnostic, training-free, and supports a wide range of controllable video generation tasks using frame-level signals. As illustrated in Figure 2, Frame Guidance steers the video generation process by applying guidance to selected frames based on frame-level signals, which produce temporally coherent videos.
We present two core components for effective and flexible frame-level guidance. First, we introduce latent slicing, a simple latent decoding technique that enables efficient training-free guidance for large-scale VDMs. Based on temporally local patterns of video encoding, we propose to decode only the short temporal slices of the video latent for computing the guidance loss, Furthermore, we present video latent optimization (VLO), novel latent update strategy designed for precise control of the video diffusion process. As the overall layout of the frames is largely determined the first few inference steps (Wu al., 2024a), we apply deterministic optimization the early stages for globally coherent layout, and employ stochastic optimization until the mid-stage for refining the details.
Frame Guidance is applicable to general frame-level control tasks, shown in Figure 1, including keyframe-gunt guideddg generation, stylized video generation, and looped video generation. In particular, Frame Guidance supports general input conditions, such depth maps, sketches, and color blocks. We demonstrate that Frame Guidance consistently produces superior results on frame-level control tasks across various VDMs (Yang et al., 2025; HaCohen et al., 2024; Wang et al., 2025a).
2 RELATED WORK
training-required controllable video generation Advances in T2V and I2V generation have opened up new opportunities for fine- -ee-ggained user control. These include conditioning on keyframes (Zeng al., 2024; Wang et al., 2025b), using style reference images for stylized genera- tion (Liu et al., 2023; Wang et al., 2023a), and incorporating trajectory-basee signals such as camera movement (Zheng al., 2024; Bai et al., 2025) or motion trajectory (Wu al., 2024b; Namekata
2

---

Published as a conference paper at ICLR 2026
al., 2025) for dynamic scene generation. However, existing methods often require extensive training and model-specific data preparation, such as fixed resolution or frame counts, making fine- mue-tuning increasingly impractical for general users as model sizes and resource requirements continue to grow.
Training-free controllable video generation To reduce the burden of training large models, several approaches have explored training-free free controllable video generation (Li et al., 2025a; Ling et al., 2025; Hou et al., 2024; Wu et al., 2023; Zhang et al., 2024; Khachatryan et al., 2023; Geyer et al., 2024). For example, CamTrol (Hou et al., 2024) enables camera control using external 3D point clouds, while MotionClone (Ling et al., 2025) performs motion cloning based on temporal attention maps extracted from a reference video, and Tune-AAVideo (Wu et al., 2023) enables video editing with image diffusion models. However, these methods are tailored to specific tasks and are thereby ill-suited for more general scenarios requiring different types or even multiple input signals. In this work, propose training-free free guidance method that generalizes to wide range of video generation tasks using frame-level signals.
3 PRELIMINARIES
Video diffusion models (VDMs) Recent video diffusion models (Brooks et al., 2024; Yang et al., 2025; HaCohen et al., 2024; Wang et al., 2025a) learn to generate video by reversing the noising process in the latent space. The high-dimensional video zo is encoded into lower-dimensional latent Fu = E (xx). The forward noising process corrupts the latent & = a /1 age, where e~ NO, I) and {āe [ is pre- noise schedule. The reverse denoising process is learned through predicting dimendependent velocity e, vage vi 20 which represents the direction from noisy sample toward the clean sample (Salimans and Ho, 2022). For each time step t, the clean sample zoje can be computed from the noisy sample 2, using Tweedie's 's formula (Efron, 201 I):
Fola E[ v Vā v1 a. (=( t),
(1)
where up is the predicted velocity. Latents 20 are decoded into videos with the decoder dn = D(zo).
Recent large-scale VDMs (Wang et al., 2025a; Yang et al., 2025) commonly employ spatio-temporal VAEs to encode high- tigh-dimensional video data. A notable example is the CausalVAE (Yu al., 2024; Brooks et al., 2024), which enforces temporal causality in the latent space by allowing only past frames to influence future ones. While this design encourages temporally coherent video generation, it also introduces temporal dependencies within the latent sequence, requiring the entire sequence to be decoded even to reconstruct a single frame.
Training-free guidance Training-free guidance (Bansal al., 2024; Yu al., 2023; Rout et al., 2025; Shen et al., 2024) uses pre-trained diffusion models to generate samples that satisfy a specific condition, without additional training. At each denoising step it estimates clean image Zoje D(20je) from the current latent 24, and computes guidance loss Le (D(()))) c) that measures aligmment with the target control c. The latent 2, is then updated using the gradient V L, during inference. One such strategy is the time-travel trick (Bansal et al., 2024; Yu et al., 2023; He et al., 2024), which alternates between denoising and renoising steps to correct accumulated errors.
4
METHOD
We present Frame Guidance, simple yet effective training free framework for controllable video generation using frame-level signals, designed to be compatible with modern large-scale VDMs. Our approach guides the generation process of pre-trained VDMs by optimizing video latents to minimize frame-level guidance loss applied to selected frames. In this section, we introduce two key components that enable efficient and flexible frame-level guidance for large-scale VDMs.
4.1 LATENT SLICING
The main challenge of training- faining-ffree guidance on video generation is the computational constraint. To compute the guidance loss for latent optimization, we should keep track of the gradient chain passing through the whole network (Figure 3). In Figure 4(a), we analyze the memory usage and find that
3

---

Published as a conference paper at ICLR 2026
zt
Early Steps
Dataminiesa
Update (VI.O)
Vg
T 5150
0
Later Steps
Selected Frames
Sampling
Slicing
T
T: 20/50 (
Olt
VAF
VAE
Guidance
50/50
Target
Guidance
Target
Forward Back-prop.
frames
(L2 loss)
frumes
Figure 3: Frame Guidance for keyframe-guided video generation task. (Left) Illustration of our method with latent slicing and spatial down-sampling (Section 4.1), and gradient propagation with L.2 loss (red arrows; Section 4.3). (Right) Visualization of the video latent optimization (VLO; Section 4.2), showing the generated video frames at each guided inference step.
exceeds 650GB even with gradient checkpointing (Chen al., 2016), mostly due to CausalVAE (Yu et al., 2024; Brooks et al., 2024). This overhead arises from the design of CausalVAE, which requires decoding the entire latent sequence even to reconstruct a single frame. To tackle this, we first analyze the latent space of CausalVAF.
Analysis of CausalVAE's latent space While Causal VAE is designed to enforce temporal causality in the video latent sequence, we observe that such causality is absent in practice. To validate this, we conduct a simple experiment: replace single frame in real video with a black image (all pixels set to zero), and measure the difference between the latents of the original video and the modified video. As shown in Figure 4(b), the perturbation affects only few consecutive latents rather than the entire sequence. This behavior consistently appears across various VDMs (Yang et al., 2025; Wang et al., 2025a; HaCohen et al., 2024). We refer to this property as temporal locality, key observation for our efficient decoding method.
Decoding with sliced latent We introduce latent slicing, an essential decoding method for training- free guidance that significantly reduces the cost of gradient computation CausalVAE. Instead of reconstructing the entire sequence, we decode only a few frames from the selected sliced latents. To be specific, when reconstructing the i-th frame x², decode small window of 3 latents, starting from the latent 2², where the latent index j is determined by and the temporal compression rate of its Causal Thanks to the temporal locality, is sufficient to decode only the corresponding latents to reconstruct a single video frame. As shown in Figure 23, the reconstructed frames are nearly identical to those from full-sequence decoding. As highlighted in Figure 4(a), this latent slicing reduces memory usage by to 15x compared to using the entire latent sequence.
In parallel with latent slicing, we can further reduce the memory usage by spatially down-sampling the latents before decoding. Despite the lower resolution, the guidance loss from the down-sampled latents still provides sufficient signals to guide the generation. Moreover, reduced spatial detail can emphasize semantic structure, yielding more effective guidance. As shown in Figure 4(a), applying x spatial down-sampling combined with latent slicing reduces memory usage by up to 60x, enabling gradient computation to be maintained on single GPU even for large VDMs (Wang et al., 2025a).
4.2 VIDEO LATENT OPTIMIZATION (VLO)
Previous training-fio guidance methods for images (Bansal et al., 2024; Yu et al., 2023; Shen et al., 2024) typically reintroduce noise after a gradient update. However, in the video domain, we observe that this strategy often has adverse effects on guidance. The overall layout of the frames largely determined during the early denoising steps (Wu et al., 2024a). Similarly, the influence of guidance is most significant on the overall layout in these stages. As shown in Figure 4(c) top, applying guidance to single frame (yellow arrow) has higher influence (dark green) on neighboring latents early on, with the effect diminishing later. This confirms that carly-stage guidance is critical for temporal coherence. Yet, the noising scale the early stage is often too large, washing out the guidance signal.
4

---

Published as a conference paper at ICLR 2026
(a) GPU memory usage
(b) Temporal Locality
(c) Guidance Influence
60G
Lateot Slicing
CausalVAE
Score Nutwork
15
Modil weights
CogVideoX
Lolent Index
Latent index
200-
Dowax2
OOMI
Ours
10
GPC
Wan2.1
Wan2.11
VidVideoX (5
21
Guidance dose not affect other frumss
(14B)
14
Shortcut base
LTX-Video
(Grad. w/o DIT).
20
Full
Slice
Down
Full
Slice
Down
Video Frame Index (replaced (rame)
Inference step
Figure 4: (a) GPU memory for guidance when using full latent sequence, sliced latents, and latent slicing with spatial down-sampling, (b) Temporal locality of CausalVAES. Each latent (y-axis) is primarily affected by small subset of temporally local video frames. (c) Guidance influence during the denoising steps. Yellow arrows indicate the location for the guidance frame.
To address this limitation, we propose video latent optimization (VLO), hybrid strategy that applies different update rules to video latents depending on the denoising stage. Specifically, at each denoising step \ in the early stage, we update the latent 2n with guidance in a deterministic manner:
Zt 24 - -----1V V/V Cérames),
(2)
where 7) the guidance step size, oof is the predicted clean frames where we apply guidance. and C. a guidance loss with frame-level controls Crrames. This deterministic update results in temporally aligned global layout. In the later steps, we update the latent 2, in a stochastic manner by reintroducing noise in order to reduce accumulated errors during guidance, similar to the time-travel trick (Yu et al., 2023; Shen et al., 2024). This stage-aware procedure is illustrated in Figure 3 right. We show in Figure 24 that stochastic updates in the early steps fail to capture the desired layout, whereas our VLO successfully reflects the layout changes specified by the guidance frames.
4.3 FRAME GUIDANCE
In Algorithm 1, we provide the overall procedure of our Frame Guidance, which incorporates both the latent slicing and VLO. Given a set of frame- level controls Cérames and selected frame indices I S. {i.,...} to apply the guidance, we first compute their corresponding latent indices J S {.,,..} (see Figure 4(b)). For pre-defined genera- tion phases tg and t, (we provide details on deter- mining their values in Appendix C.5), we optimize the video latents in the following manner: At each denoising step > ti, extract the sliced latents _JT from the latent indices I (Line 7) and com- *Oje
Algorithm 1 Frame Guidance
Require: I, tg, tL, repeat step M, step size 17,
guidance loss Las model vo(::))
1: 2T I)
2: J t- Frx---ldx-to- Loai--to-Laeet--ddx(/(
3: for f. - T, .... do
4:
if > tz then (Guidance step)
5:
for m ..., . do
6:
201: - Vonz v1 Cn (2, 1)
7:
..
Slient-Slicing(20)e>>
8:
2or + D(=0())
I
pute the guidance loss 9r = Ce, Crzames)
9:
= V L Crrames)
(Lines 8-9). We optimize the latent 2, using VLO (Line 11) where 2, is updated deterministically in
10:
if l > LE then (Farly steps)
11:
2x < 2t 119h
the early denoising steps (t > (.)) and stochasti-
12:
else (Later steps]
cally (Algorithm 2) during the later steps (tæ 2 t tz). After M times of latent optimization, we
13:
21 < Time Travel T-me-TTrael(eee
Zofe: 9t)
end if end for
proceed to the next denoising step via DDIM Song
15:
16:
end if
et al. (2020). We provide detailed time-travel al- gorithm and Frame Guidance algorithm for flow
17:
21-1 + DDIM 2ojr)
18:
end for
matching based models, such Wan (Wang al., 2025a), in Appendix C.3.
19:
return 20
Gradient propagation after slicing Without processing the full latent sequence, guidance applied to sliced latents can control the entire video, resulting in temporally coherent outputs. This coherence arises from the denoising network Up. which propagates the gradient of the guidance loss across the entire video latents. We show in Figure 4(c) bottom that excluding the denoising network when
5

---

Published as a conference paper at ICLR 2026
computing the gradient, i.e., shortcut-based update (He al., 2024; Rout et al., 2025; Nair and Patel, 2024), restricts the gradients to the guided frame only (bottom), leading to a temporally disconnected video. On the other hand, using the denoising network propagates the gradients across all frames (top), allowing guidance on target frames to harmonize with other frames, as illustrated in Figure (right). Therefore, guidance on a few frames where the gradient through the denoising network can control the whole video, which enables tasks such as stylized video generation. In Appendix C.4, we further demonstrate that the temporal coherence is primarily determined by the denoising network, whereas the contribution of CausalVAE is minimal.
4.4 Loss DESIGN FOR VARIOUS TASKS
Frame Guidance is readily applicable to wide range of with appropriately designed guidance loss. Here, we provide simple loss designs for representative frame-conditioned video generation tasks and general user inputs.
video generation tasks,
Keyframe-guided video generation aims to synthesize videos that transition smoothly between multiple user-specified keyframes, without enforcing strict pixel-level reconstruction. Given initial image as the input to the 12V model, we minimize a simple L.2 loss, L. = Σier ||=; - = 11.. where
xf denotes the target keyframes and = is the predicted clean i-th frame. The similarity to each keyframe can controlled by adjusting the guidance strength, such as the number of repeat steps M or step size 1). Unlike training-based approaches (Zeng et al., 2024; Wang et al., 2025b) that are limited to fixed positions (e.g., the last frame), our method supports arbitrary keyframe placements.
Stylized video generation aims to synthesize videos in the style of given reference image using T2V model. We employ a differentiable style encoder V/ to compute the style loss defined as L. = Eiez c (V (z V (z))e),, where sstyle is the style reference image. We use the Contrastive Style Descriptor (CSD) (Somepalli et al., 2024) for (.). and find that guiding only a few selected (or randomly chosen) frames is sufficient to propagate the desired style across the entire video.
Looped video generation aims to synthesize videos where the first and last frames match, producing seamless loop using T2V model. We define the loss L. ||= = l1.. which encourages the first and last frames to become naturally aligned, enabling smooth looping.
General input guidance aims to synthesize videos conditioned on general user-specified conditions beyond RGB images, for example, depth maps or sketches. We use differentiable encoder V, such as a depth estimator (Yang et al., 2024) or an edge predictor (Chan et al., 2022), to extract structural features from the estimated clean image. We minimize encoder-aligned L2 loss defined as La Eier (x*)) - V (xrje) I12. where (xx) denotes the encoded target conditions.
EXPERIMENTS
5.1 KEYPRAME-GUIDED VIDEO GENERATION
We evaluate Frame Guidance on keyframe eeyframe-guiedd video generation tasks, which aim to synthesize videos that smoothly follow multiple user-specified keyframes. Unlike frame interpolation tasks (Feng al., 2024; Wang al., 2025b) that require exact frame matching, keyframe- -uided generation only requires the visual similarity to the keyframes, and addresses the generation of longer videos.
Datasets We select 40 clips with more than 81 frames from DAVIS (Pont-Tuuet et al., 2017) and 30 real-world videos from Pexels! dataset. Pexels features more dynamic and human-centric videos, making it more difficult for video generation. We provide more details on the dataset in Appendix B.2.
Baselines We compare Frame Guidance against frame interpolation methods, including TRF (Feng et al., 2024), SVD-Interp (Wang et al., 2025b), and CogX-Interp. TRF is a training-free approach for Stable Video Diffusion (SVD) (Blattmann et al., 2023), SVD-Interp uses a fine-tuned reversed-motic SVD, and CogX-Interp fine-tunes CogX with first and last frame conditioning. We also compare with basic 12V baselines (CogX (Yang et al., 2025) and Wan (Wang et al., 2025a)). For our method,
http:://hggnnggecccoaaaaae//ooee (Accessed: 2026-02-18) hhtps///sthaiicccmmmmrooooooolee (Accessed: 2026-02-18)

---

Published as a conference paper at ICLR 2026
(a) SVD-Interp.
(d) Interrp
Cnterp
(c) Ours (CogX-Interpp)
(c) Ours (CogX-12V)
Similar
Ours (Wan-12V)
Similar
Figure Qualitative comparison on keyframe-guided video generation tasks. Yellow box indicates the keyframe condition. Orange box in (a) shows disconnection in SVD-Interp. Red box (d) visualizes failure case for the CogX-Interp baseline for dynamic human motion.
Input frames
Train free
DAVIS
Pexels
FID
FVD
FID.
FVD |
CogX-12V
V
60.36
890.1
74.98
1122.6
Wan-14B-12V TRF
59.04
772.8
73.03
1033.3
I,F
62.07
923.1
79.03
1106.2
Ours (CogX)
L.F
57.62
613.4
68.54
1027.3
Ours (CogX)
L.M. F
V
55.60
577.1
68.97
989.3
Keyframe-sim Quality
Ours (Wan- (aan-14B
I,M.F
57.68
761.1
71.63
904.8
SVD-Interp.
L.F
X
63.89
800.3
75.37
1210.7
Wan CogX-Int.
Ours
CogX-Interp.
L.F
X
46.59
506.0
58.73
1081.5
CogX CogX-Int.
Wan
Ours (CogX-Interp.)
I,M,F
X
37.95
420.3
47.86
723.26
Figure 6: Keyframe-guided generation results. (Left) Human evaluation. (Right) Quantitative results. M,, and F denote initial, middle, and final frames, respectively. "Train-free" indicates whether the backbone VDM is base 12V model or fine-tuned for the frame interpolation task.
we apply Frame Guidance on CogX and Wan models using the L2 loss defined in Section 4.4 with the final frame given, and restrict the number of guidance steps so that the total runtime does not exceed 4x the base model's 's inference time (details in Appendix B.1). We further report results that additionally use the middle frame. We also report results applying Frame Guidance to CogX-Interpp
Qualitative comparison As shown in Figure 5, our approach generates videos with natural transitions, where the selected frames closely resemble the keyframes. For example, Figure 5(c) visualizes well-aligned frames, with the paraglider appearing in consistent position. In contrast, IngX-Interp often struggles with challenging motion. Applying Frame Guidance to TogX-Interp (Figure 5(e)) to stronger VDM backbone (Figure 5(f)) results in notably improved output quality.
Human evaluation We conduct human evaluations to assess the quality of generated videos, focusing (1) video quality and (2) similarity to the keyframes. As shown in Figure 6 left, applying Frame Guidance to Wan yields the highest video quality, surpassing the trained model CogX-Interp. Applying guidance CogX-Interp produces high-quality videos with guided frames nearly identical to the keyframes. Further details are provided in Appendix B.2.
Quantitative results We measure FID (Heusel al., 2017) and FVD (Ge et al., 2024) to assess the quality of the generated videos. As shown in Figure 6 right, Frame Guidance applied to pre-trained 12V models significantly outperforms all other training-fre free methods. Moreover, Frame Guidance applied to CogX-Interp outperforms all the training training-required baselines. These results, combined with the human evaluation, demonstrate that our method effectively guides video generation without additional training. We discuss further details regarding the quantitative results in Appendix B.2.
5.2 STYLIZED VIDEO GENERATION
We also validate Frame Guidance stylized video generation tasks, which aim to synthesize videos in the style of given reference image, using a T2V model.
subset of the stylized video dataset introduced in StyleCrafter (Liu et al., 2023), which consists of 6 challenging style reference images, each paired with an aligned style prompt and 9 distinct content prompts. We provide further details about the dataset in Appendix B.3.
Dataset We
7

---

Published as a conference paper at ICLR 2026
Reference
Referasce
4 sarel performur playing the guilar. Love Puly Digial 4tt, goumatic shapes, vilrunt colors, flas teanree shop gradiens shading, modern graphic
field of sanflowee da: Mange Style, Macc and white dightal high contrast desaled for work crunwhetching shadowr, clean, rolor
Eempp
Comp
Figure 7: Qualitative comparison on stylized video generation. Ours generates high-quality videos that follow the reference style, whereas baselines fail to produce motion or show poor alignment.
Train free
Text-Alignment
Style-Aligmment
CLIP-T
ViCLIP-T'
CLIP-S
ViCLIP-S
Vid.Comp.
X
0.211
0.137
0.869
0.219
Style
StyleCrafter
X
0.207
0.273
0.635
0.157
CogX-T2V
0,220
0.259
0.588
0.139
Mation
Ours
0.224
0.285
0.624
0.185
Cogx
Vid.Comp StyleCrafter
Ours
Figure 8: Stylized video generation results. (Left) Human evaluation. (Right) Quantitative results.
Baselines We compare our method with three baselines. CogX-T2V is pre-trained T2V model. VideoComposer (Wang et al., 2023a) is a training- based method supporting multiple conditions, such as style image and depth maps. StyleCrafter (Liu et al., 2023) is also training- based method that solely trains a style adapter on top of VideoCrafter (Chen et al., 2023). For our method, we apply Frame Guidance to CogX-T2V (Yang et al., 2025) model using the style loss defined in Section 4.4. We provide more details of our method in Appendix B.3.
Qualitative comparison Figure show that method can generate balanced stylized videos in terms of both text alignment and style conformity, with diverse motion. In contrast, VideoComposer fails to disentangle content and style in the reference images, while StyleCrafter produces videos with minimal motion that are poorly aligned to the reference style. CogX-T2V struggles to capture detailed textures or patterns, for example, geometric shapes or sunflowers.
Human evaluation We conduct human evaluation to assess the quality of stylized videos, evaluating three criteria (1) style alignment, (2) text alignment, and (3) motion dynamics. As shown in Figure 8 left, our method achieves the best results across all criteria, significantly outperforming the training- based baselines. These results show that Frame Guidance successfully guides video generation to follow the reference style without any additional training. Further details and the results on overall preference are provided in Appendix B.3.
We evaluate the generated videos for text aligmment and style alignment using CLIP-T, ViCLIP-T, CLIP-S, and VICLIP-S (Radford et al., 2021; Wang al., 2023b). As shown in Figure 7 and Figure 8, our method achieves the best scores on all metrics, except for CLIP-S, where it matches the performance of StyleCrafter. While VideoComposer achieves the highest style alignment scores, this is largely due to replicating the style image without adhering to the text prompt.
Quantitative results
5.3 LOOPED VIDEO GENERATION
We further apply Frame Guidance on the looped video generation task, which aims to synthesize videos where the first and last frames match, producing a seamless loop. We use the loop loss defined in Section 4.4 to steer the last frame to match the first. Guidance is applied to the generated video without requiring any external conditions, using only text prompts as input. As shown in Figure 1(c) and Figure 18, Frame Guidance generates high-quality looped videos featuring dynamic motions that are well-aligned with the input text prompt.
5.4 OTHER APPLICATIONS
Using color bluck drawing During ssyfram--guide generation, keyframe similarity can be flexibly controlled by adjusting the guidance strength. This allows new forms of user-provided control signals
8

---

Published as a conference paper at ICLR 2026
First
Last
Mask
(a) Masked region guidance
<Syyle>
Original Video
(b) Video style transfer
Depth
Sketch
CogX-Interp. input
Style
Loop
(c) Multi condition guidance
Figure 9: Examples of other applications. (a) Object movement guided by masked region. (b) Video style transfer with SDEdit (Meng et al., 2022). (c) Guidance using multiple types inputs. Top: depth map and sketch, Bottom: style and looping.
that are easy to create, such as coarse sketches or color blocks. In particular, we introduce a novel application that allows users guide video generation using edited frames, where simple visual edits via color blocks indicate changes in color or detail. As illustrated Figure ((d), the generated video depicts the mountain changing color and texture in three distinet ways, which difficult achieve using text prompts alone. For Frame Guidance, color blocks act as rough visual hints that allow natural scene transitions while preserving the contents. We provide more examples in Figure 19.
Masked region guidance While our previously described methods apply guidance to the whole area of frame, we demonstrate that the guidance can be effectively restricted specific regions by using L2 loss with a binary mask. In Figure 9a), we present an example of generating a video with object motion, guided by cropped image and its segmentation mask. By applying guidance solely to the object region, the background remains unchanged while the object shows smooth movement.
Depth map/ Sketch guidance Furthermore, Frame Guidance supports general types of frame-level signals, such as depth maps and sketches, which offer more user-frient beer-friendly conditioning compared to RGB images as input. Using the general input guidance defined Section 4.4, Frame Guidance capable of generating high-quality guided videos shown in Figure ((e) and (I).
Video style transfer We extend Frame Guidance to video editing tasks. Taking a video as input, we apply Frame Guidance to generate an edited video that follows a reference style. It can be achieved by applying simple SDEdit (Meng et al., 2022) with a small noise. This results in preserving the original motion and layout while successfully transferring the reference style, in Figure 9(b).
Multi condition guidance Frame Guidance can integrate multiple imput types by combining losses. As shown in Figure 9c) top, we apply guidance intermediate frames, combining the depth map loss and sketch loss for the CogX-Interp model. The generated video demonstrates smooth motion that follows the input signals, showing the flexibility of Frame Guidance in handling complex scenarios. As shown in Figure 9() bottom, combining style and loop losses enables the generation of stylized looping videos. We provide additional examples on multi condition guidance in Figure 21.
5.5 ABLATION STUDIES
Table 1: Ablation study on la- tent optimization strategy.
Necessity VLO To validate the importance of VLO in Frame Guidance, we compare it against two variants: one that uses only the time-travel trick and another that applies only the deterministic update from Equation 2 during the guidance process. Table 1 shows that using only the time-travel trick yields higher FVD scores due to difficulty in forming coherent layouts, while the deterministic update
Method
FID
FVD
Time-travel
57.37
778.4
Deterministic
56.61
637.3
VL.O (Ours)
$5.60
577.1

---

Published as a conference paper at ICLR 2026
Style
Style
(a) Unseen (OOD) style generation
(b) Failure with edge map (weak guidance)
Figure 10: Failure cases of Frame Guidance.
alone produces over-saturated temporally disconnected videos. We provide additional ablation study on tg which determines when to apply deterministic update in Appendix C.5.
Model agnostic As shown in Figure our method compatible with a variety of VDMs, including Cog CogVideoX (Yang et al., 2025), its fine-tuned variant VegeeoX-Interrpa al., 2025a), flow-m www-matchingg-ere based model. To further demonstrate generality, we also apply our approach to two additional models: SVD (Blattmann et al., 2023), U-Net-based (Ronneberger et al., 2015) diffusion model, and LTX-2B (HaCohen et al., 2024), which supports sequences up 161 frames. As illustrated in Figure 22, our method consistently performs well across all these VDMs.
and Wan-14B (Wang
GPU Memory Usage We further analyze which factors affect GPU memory usage, including the number of frames used for guidance and the guidance type. We observe that peak mem- ory is primarily determined by the number of frames decoded through the VAU Different guidance types (c.g.. RGB pixel vs. depth) introduce additional overhead, but this increase remains small compared to the memory cost of the VAE decoder and score network, as auxiliary modules are computationally lightweight. As shown in Table 2, incremental memory overhead scales roughly linearly with the number of guided frames and remains limited even when using Depth-Anything- Antth-nn-tttnn-n-eaa V2-Large (Yang et al., 2024).
Table 2: Peak GPU memory (GB) under different numbers of guided frames [Z] and guidance types, measured on Cog CogVideoX-12V.
Guided Frames [Z]
Loss
#1
#2 (+1)
#4 (+3)
RGB
24.75
+ 1,448
+ 4,46
Depth 26.00
+ 2.05
+ 6.15
6 LIMITATIONS
Although Frame Guidance is training-free free and supports various applications, it has some limitations:
(1) The computational cost of guidance sampling is higher than that of training- based methods. Since it requires back-propage propagation and multiple predictions, the inference speed is approximately up two to four times slower than that of the base model, depending the task. This issue is particularly significant in video generation. We leave addressing this inefficiency to future work.
(2) While our method is model-agnostic, its performance depends on the base model. Since it samples videos within the base model's 's generation distribution, it struggles with highly dynamic scenes or fine-grained objects unseen during training. For example, as shown in Figure 10(a), it often fails to generate unseen (OOD) styles, such as 3D animation character.
(3) As discussed FreeDom (Yu et al., 2023), loss-based guidance struggles to control fine-grained structural features. For example, in Figure 10(b), applying Frame Guidance with Sobel edge maps often leads to weak or unstable control, whereas it performs well with RGB keyframes. In such cases, training- based methods (Jiang et al., 2025; Li et al., 2025b) provide a more reliable alternative.
7 CONCLUSION
In this work, we present Frame Guidance, a novel training-free framework for diverse control tasks using frame-level signals. By applying guidance selected frames, our method enables natural control throughout the video. To achieve this, we partially decode sliced latents during guidance computation and introduce latent optimization strategy designed for video. Our approach supports wide range of tasks without training, including special cases such as color block guidance and looped video generation.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGMENT
This work was supported by Institute for Information communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) (RS-2019- Artificial Intelligence Graduate School Program (KAIST)) and the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research (No. RS-2024-00469482 & RS-2024-00359927)) and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00256255), and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.
REPRODUCIBILITY STATEMENT
To ensure reliable and reproducible results, we have provided the source code at https::///// sectps://gieee //ghthub. com / htmm//gwoo//fffe frameeeooorreeeoeooo gui dance, and detailed experiment settings in Appendix B.
REFERENCES
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from single video. arXiv preprint arXiv:2503.11647 2025.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Roni Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In International Conference on Learning Representations, 2024.
Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:231..15127 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https::/ htapps:///ppaa.... com/ research/ video-generate iittiionn mooonmmooelle s-alsssaooooos 1d- simulators.
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li. Mohsen Mousavi, et al. Go-with-the- Motion-controlladt video diffusion models using real-time warped noise. arXiv preprint 25Xi:2551.0083/ 2025.
Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Conference on Computer Vision and Puttern Recognition, 2022.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19517 2023.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 0rXiv:1604.000
2016.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference Learning Representations, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496): 2011.
Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya. Michael J Black, and Xuaner Zhang. Explorative inbetweening of time and space. In European Conference on Computer Vision, 2024.
11

---

Published as a conference paper at ICLR 2026
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun- Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In Conference on Computer Vision and Pattern Recognition, 2024.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In International Conference on Learning Representations, 2024.
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00100 2024.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for video diffusion models. In International Conference on Learning Representations, 2025.
Yutong He, Naoki Murata, Chieh- Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. International Conference on Learning Representations, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Chen Hou, Guoqiang Wei, Yan Zeng and Zhibo Chen. Training-free camera control for video generation. arXiv preprint arXiv: 10126, 2024.
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598 2025.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-7ero: Text-to-image diffusion models are zero-shot video generators. In International Conference on Computer Vision, 2023.
Jialu Li, Shoubin Yu, Han Lin, Jaemin Cho, Jaehong Yoon, and Mohit Bansal. Training- free guidance in text-to-video generation via multimodal planning and structured noise initialization. arXiv preprint arXiv:2504.0mm 08641, 2025a
Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance. arXiv preprint arXiv:2503. 16421, 2025b.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free free motion cloning for controllable video generation. In International Conference on Learning Representations, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Yibo Wang, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter. arXiv preprint 332002231..00330, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Nithin Gopalakrishnan Nair and Vishal Patel. Dreamguider: Improved training free diffusion-based conditional generation. arXiv preprint arXiv:2406.02544 024.44246 2024.
Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B. Lindell. SG-i2v: Self-guided trajectory control in image-to-vide generation. In International Conference Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-4o system card. Technical report, 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang. Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Ramb- hatla, Sam Tsai, Samanch Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei- Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang. John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720 2025.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1004.00655. 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical Image Computing and Computer- Assisted Intervention, 2015.
Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. RB-modulation. Training-fres free stylization using reference-based modulation. In International Conference on Learning Representations, 2025.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Yifei Shen, Xinyang Jiang, Yifan Yang, Yezhen Wang, Dongqi Han, and Dongsheng Li. Understand- ing and improving training-free loss-based diffusion guidance. Advances in Neural Information Processing Systems, 2024.
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein. Mcasuring style similarity in diffusion models. arXiv preprint arXiv:2404.01292, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:/ //tt:////ttbbbb cwm/cooo////eeeeee facee////////////ooo diffusers, 2022.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint 2003 2025a.
13

---

Published as a conference paper at ICLR 2026
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 2023a.
Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher--Shizemeeen ski, and Steve Seitz. Generative inbetweening: Adapting image-to-video models for keyframe interpolation. In International Conference on Learning Representations, 2025b.
Aleksander Holyn-
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.066 3.7.006442 2023b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPHI 2024.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In International Conference on Computer Vision, 2023.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initializa- tion gap in video diffusion models. In European Conference on Computer Vision, 2024a
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, 2024b.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances Neural Information Processing Systems, 2024.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference Learning Representations, 2025.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training-free -ugdded conditional diffusion model. In International Conference on Computer Vision, 2023.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In International Conference on Learning Representations, 2024.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Conference on Computer Vision and Pattern Recognition, 2024.
Yabo Zhang. Yuxiang Wei, Dongsheng Jiang, XIAOPENG ZHANG, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-fres free controllable text-to-video generation. In International Conference Learning Representations, 2024.
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2y: Camera-conttooled image-to-video diffusion model. arXiv preprint arXiv:2410.1599 15957, 2024.
14

---

Published as a conference paper at ICLR 2026
Appendix
Organization The Appendix organized as follows: In Section A, provide the additional backgrounds of our work. We describe the details of the experiments and our framework in Section B, and further discussion in Section C. Lastly, in Section 6, discuss the limitations of our work.
A BACKGROUNDS
A.1 TRAINING-FREE DIFFUSION GUIDANCE
Recent works Song et al. (2021); Dhariwal and Nichol (2021); Yu et al. (2023); Chung et al. (2023); Bansal al. (2024); He et al. (2024); Shen al. (2024) have explored conditional generation by injecting external conditions into pre-trained diffusion models. Among them, training-free guidance methods (Yu et al., 2023; Chung al., 2023; Bansal et al., 2024: He et al.. 2024; Shen al., 2024) achieve controllable generation without additional training by optimizing the noisy latent during the reverse process. This optimization is guided by loss function that measures the alignment between intermediate latents and the target condition at each denoising step. FreeDom (Yu et al., 2023) and UniversalGuidance (Bansal et al., 2024) leverage off-the off-the-shelf models to compute the various guidance losses, achieving a wide range of controllable image generation tasks. Later works (He al., 2024; Nair and Patel, 2024; Rout al., 2025) bypass the denoising module for computing the guidance loss, enabling more efficient training-free free diffusion guidance.
A.2 FLOW MATCHING
Flow matching (Lipman et al., 2022) belongs to the family flow- bow-based generative models, which are known for faster sampling compared diffusion models (Ho al., 2020). Let / E [0, 1] be the time, & E I2d be a data, and be a unknown target distribution. The goal of flow matching Lipman et al. (2022) is estimate time-dependent transformation =, [0, Rd - Rd (referred to flow) that maps a prior distribution (c.g., Gaussian) to distribution p1 z q. Instead of directly estimating the flow, Lipman al. (2022) proposes regress generating vector field ve(-, t) [0, 1] x 12 R that induces the flow 24 via the following ordinary differential equation (ODE):
dz{(2) v=(2((z)) and 2o(x)
(3)
dt
is common practice to design this flow de along an optimal transport (OT) trajectory that connects a prior sample to target sample with straight interpolation: : (1 - t) + tæ where and Vi ~ q. In this case, the target o, is computed as a constant: vy 4) = æ1 zo for all E [o. 1]. With neural network vy that estimates 1/, we can generate data .r by numerically solving the ODE in Equation (e.g., Euler method). Similar to Tweedie's formula Efron (2011), we can approximate a cleaned sample at each time t by
Po
11p 24 + 1-t vo(2,, t).
Throughout this paper, we interchangeably reverse the direction of time by parameterizing as s(t) = 7(1 t), t E [0, 1] to align with the convention of the diffusion models where the generative process proceeds from I O.
B EXPERIMENTAL DETAILS
B. IMPLEMENTATION DETAILS
All our experiments are conducted on single H100 GPU. Hyperparameters related to guidance, such as step size 1) and repetition M, are adjustable depending on the task and model characteristics. For video generation using diffusion- Cog VideoX (Yang et al., 2025),
example, in keyframe-guide we define the layout stage within the first 5 steps, set M = 10, and use a step size of 7) = 3.0. For the travel trick, M is linearly decreased over 15 steps. At each step, gradients are L2 22-nommalized before being scaled by for the update. All comparisons in our paper were conducted using the same random seed.
15

---

Published as a conference paper at ICLR 2026
Vaboc und hore to basee the folower
The oide sholld perornt decat the fimet the hrrmes
tho Wiaal sen an free the trowes
sewekk und werhive
Trura Snalutt
0
0
Creal
Figure 11: screenshot of questionnaires from our human evaluation on keyframe soiinee-gddedddd generation.
5-
3-
2
Keyframe-sim
1-
Quality
Wan-12V
CogX-Interp.
Ours CogX-12V
Ours CogX-Interp. Interp.
Ours Wan-12V
Figure 12: Human evaluation results on keyframe-guided generation including Wan-12V.
Since Wan- 14B (Wang al., 2025a) employs flow matching its generative modeling, its inference fully deterministic, and the layout is mostly established within steps. Therefore, we set the layout stage to the first 2 inference steps, and apply the same M, 7). and time-travel configuration. Moreover, since our implementation introduces more stochasticity (see Appendix C.3), we slightly reduce the number of time-travel steps. To maintain practicality, empirically limit the number of guidance steps such that the overall runtime does not exceed 4x the base model's 's inference time.
To reduce GPU memory usage, we apply gradient checkpointing (Chen et al., 2016) the denois- ing network using the Diffusers (von Platen et al., 2022) library. For the VAU, gradient checkpointing is applied only in VogVideoX (Yang al., 2025), as Wan-14F 14B (Wang et al., 2025a) implementation does not currently support it. We do not apply spatial downsampling in VogVideoX, since it runs on a single GPU without In contrast, we apply 2x spatial downsampling experi- ments with Wan-14B.
B.2 KEYFRAME-GUINE GUIDED VIDEO GENERATION
Dataset For evaluation, we use videos from the DAVIS (Pont-Tuset et al., 2017) dataset and Pexels. From DAVIS, we select 40 videos with least 81 frames, matching the maximum frame length supported by Wan-14B (Wang et al., 2025a). The resolution of each video is resized and center-cropped according to the requirements of each pre-trained model. To ensure fair comparisons across models, the same initial and final frames are used. Based on this setup, the reference set for each model is configured with slightly different FPS settings. For example, for an 81 11-frame video, Cog VideoX (Yang et al., 2025) supports only 49 frames, video accordingly. The Pexels dataset contains more real-world videos with challenging motions and frequent camera view changes. We randomly select a subset of 30 videos, which features more dynamic and human-centti content compared to DAVIS.
temporally downsample the
For pre-trained models that accept text prompts input, except for Stable Video Diffusion (Blattmann et al., (SD3)) based methods (Feng et al., 2024; Wang et al., 2025b), we used prompts derived from the original videos. Specifically, concatenated three frames from each original video and generated a caption using GPT-4o (OpenAI, 2024). The same prompt was applied consistently across all baseline models.
16

---

Published as a conference paper at ICLR 2026
Vuder Guslt Feet
Sydtach
adcond The
enteret herig pollo bnteooo paotier
Fyneeree
Overal
0
begnn Roceet des
Tru
0
Figure 13: A screenshot of questionnaires from our human evaluation on stylized video generation.
Style
Text
Motion
Overall
Cogx-T2V
Vid.Comp
StyleCrafter
Ours
Figure 14: Human evaluation results on stylized video generation including overall preference.
Human evaluation We conduct human evaluation for leyframe-guided video generation task to evaluate two main aspects: (1) video quality and (2) similarity to the keyframes. Both metrics are rated absolute scale from 1 to 5. As shown in Figure 11, participants evaluated all videos generated from the same keyframes side by side. We collected responses from 20 participants, evaluating 5 types of videos across 5 different methods. The full human evaluation results, including Wan-12V, are provided in Figure 12.
Evaluation metric For evaluation metric, we employ FID (Heusel et al., 2017) and content- debiased FVD (Ge al., 2024) between generated videos and real videos. Both metrics quantify the distributional distance between generated videos and real videos from the dataset. FID is computed by extracting all frames from the video and treating them as individual images. FVD is measured against reference videos adjusted to match each model's 's resolution and FPS. Therefore, cross-model comparisons are not strictly valid.
As shown in Figure 6 right, our method with Wan slightly outperforms Wan 12V in these quantitative metrics. However, human evaluations in Figure 6 left suggest more noticeable improvement, which may not be fully captured by such metrics. Notably, the overall FID and FVD scores are relatively high, as our setting involves longer and more dynamic videos compared to related tasks such as video interpolation, making the dataset more challenging.
We provide more qualitative examples in Figure 15.
B.3 STYLIZED VIDEO GENERATION
Based on our analysis of layout formation in Section 4.2, we apply VLO with different schedule for stylized video generation compared to keyframe-guided video generation. Specifically, we start applying the deterministic latent update (Equation 2) at step 3 before entering the detail stage (step 5), and then switch to time travel during steps 20. This design helps shape the geometric patterns and structure of the style reference image during the layout stage. After that, proceed the inference without guidance. We set the guidance step size 1) = 3 and the number repetition M = We compute the style guidance loss 4 evenly spaced frames from the entire video.
17

---

Published as a conference paper at ICLR 2026
Table Text prompts Liu et al. (2023) used for stylized video generation.
Content prompl
Content prompt
A street performer playing the guitar.
A wolf walking stealthily through the forest.
A chef' preparing meals kitchen.
A hot air balloon Bloating in the sky.
A student walking school with backpack.
A wooden sailboat docked in harbor.
A bear catching fish in river.
A field of sunflowers sunny day.
A knight riding a horse through field.
Table 4:
Style references and style prompts (Liu et al., 2023) used for stylized video generation.
Style image
Style prompt
Style image
Style prompe
Manga Style, black white digital inking, high contrasi, detailed line work. cross-hatching for shadows, clean, no color.
Ink wateroolor on paper, urban sketching style. detailed line work, washed colors, realistic shading, and vintage feel.
Manga-inspinn digital art, dynamic composition, exaggerated proportions, sharp lines, cel-shading, high-contrast colors with a focus on sepin tones and blues.
Low Poly Digital Art, genmetric shapes, vibrant colors, texture, sharp odges, gradient shading, modern graphic style.
Pixel art illustration, digital medium, de- lailed sprite work, vibrant color puletle, smooth shading, and a nostalgic, retro video game aesibetic.
Wartercolor Paining. fluid brushstrokes. transparent washes, color blending, visi- ble paper texture, impressionistic style.
Dataset We use subset of the test dataset introduced in StyleCrafter (Liu et al., 2023), which consists of 9 content prompts and 6 style reference images with corresponding style descriptions. In Table 3 and Table 4, we detail our test dataset. The content prompts describe an entire video content using simple sentence, while the style prompts describe the styles of the video. The style prompts are generated by GPT-4o (OpenAI, 2024). We concatenate each content prompt with each style prompt, resulting total of 54 full prompts for stylized video generation.
Human evaluation In Figure 13, provide screenshots of the questionnaires and labeling instructions. 20 participants are asked to evaluate four metrics: (1) style alignment, (2) text alignment, (3) motion dynamics, and (4) overall video preference of five stylized videos generated by four models. All metrics were rated on an absolute scale from to 5. The complete evaluation results, including overall preference, provided in Figure 14.
Evaluation metric We employ CLIP-Text and ViCLIP-TexI access the text alignment of the generated videos. We also compute CLIP-Style and ViCLIP-Style to access the style conformity the generated videos. Specifically, CLIP-Text and CLIP-Style are computed by using the CLIP (Radford et al., 2021) 1) text and image encoders, respectively:
I T.
f((z)) (p) [a(a)aallTT
1 T, 1=1 [[l(a
Σ
fi( fI((ss)e)
and
(5)
where 2y is the 1-th frame, p is the text prompt, Tayle is the style reference image, and Sr(-)) and fr(+)
are the CLIP (Radford et al., 2021) image and text encoders, respectively.
Similarly, ViCLIP-Text and Viyl are both computed by using Video CLIP model (Wang al., 2023b):
fv(x) fr(p) and I (x)allal()))
(v) fr (P
(6)
[[vv (rlllll(((
where x is the video, p and Payle the full and style prompts, and fv(.) and fr(:) are the ViCLIP video and text encoders, respectively.
We provide more qualitative examples in Figure 16 and Figure 17.
18

---

Published as a conference paper at ICLR 2026
(a)
(b) Ours (CogX-interp)
(c) Ours (CogX-I2V I2V)
(d) Wan-I2V
(e) Ours (Wan)
Figure 15: Qualitative comparison of keyframe-guides video generation. Orange arrows indicate temporally disconnected frames, and red boxes highlight poor keyframe similarity. Our method generates temporally coherent videos while maintaining semantic similarity to the keyframes.
B.4 LOOP VIDEO GENERATION
We use the similar guidance schedule with keyframe-guidee video generation task, but reduce the early guidance strength to avoid producing over-saturated examples. We provide more qualitative examples in Figure 18.
B.5 ADDITIONAL GENERATED EXAMPLES
We provide more examples on Frame Guidance with color block image in Figure 19, multi condition (style and loop loss) in Figure 21. We show examples generated by other models, SVD (Blattmann et al., 2023) and LTX-2B (HaCohen et al., 2024), are shown in Figure 22.
19

---

Published as a conference paper at ICLR 2026
"A New York City street scene with man and woman walking down the street, dog running after them, and bicyclist passing by in oil painting style."
"A New York City street scene with and woman walking down the street, dog rumning after them, and bicyclist passing by in Ukiyo- e style.."
"A bustling Paris café in the morning, waiters serving coffee, people chatting at tables, and dog lying under a chair Impasto painting style with vibrant colors."
Figure 16: Stylized video generated by Frame Guidance using style loss. These videos are generated by Cog VodeoxX--....
"A New York City street scene with man and woman walking down the street, dog running after them, bicyclist passing by, watercolor painting style." "
(Base T2V)
Base T2V
New York City street scene with man and woman walking down the street, dog running after them, and a bicyclist passing by, in watercolor painting style."
(b) Frame Guidanee using style loss
Figure 17: Stylized video generated by Frame Guidance using style loss with the same random seed. While their content remains similar, the style is primarily altered. These videos are generated by Cog C
20

---

Published as a conference paper at ICLR 2026
Start
Last
"A red monster toy jumping trampoline in slow motion, landing and bouncing back up endlessly, playful loop"
squirrel runs along branch, carrying its mouth, it stops midway, sniff's bark, then hides the in small crevice, it turms back and runs same path again toward another nut on ground"
Figure 18: Loop video generated by Frame Guidance using loop loss. These videos are generated by Wan- 14B T2V.
"A of colorful liquids As.As meet the glass, the yellow liquid, being denser, creating boo-iaett seguraation. The interface between the twa liquids is sharp and clear.
(a) Base CogVideoX -IIV
(b) Frame guidance with color block
Figure 19: Frame Guidance with a color block image allows the generation of a video with a complex scene. These videos are generated by Cog VideoX-12V.
"A mature turtle glides through clear blue waters above coral reef, its flippers moving gracffully. Sunlight filters through, casting tranquil glow on the turtle and screne surroundings."
(a) Base Cog Vodeoxxx
(b) Frame guidance with edge map (Sobel filter)
Figure 20: Frame Guidance with edge map. Canny edges are intractable, so we replaced it with Sobel filter. While this approach works some extent, it struggles capture fine details and fails under large scene changes, which we discuss in our limitation Section 6. These videos are generated by CogVideoX-2V.
21

---

Published as a conference paper at ICLR 2026
"A cat sits stone wall, turns to watch falling leaves, then turns its head forward again, Impasto oil painting style with vibrant colors."
Start
Last
"A raccoon dips its paw into a shallow stream, ripples spreading out, then looks up before dipping its paw again, in watercolor painting style."
Start
Last
Figure 21: Frame Guidance with style and loop loss. Simply summing the two losses enables effective composition of both guidance signals. These videos generated by Vo
(a) LTX-2B
(a) Stable Video Diffusion
compatible with both SVD (Blattmann al., 2023) and LTX-2B (HaCohen et al., 2024). For SVD, since it does not use a temporally compressed VAE, we skip latent slicing. Some saturation observed in the LTX-2B results occasionally occurs due to the model itself.
Figure 22: Frame Guidance is model-agnostic.
22

---

Published as a conference paper at ICLR 2026
Original full video (49 frames)
Frames #25 28
Frames #45 48
(a) Full latent sequence reconstruction (13- lengthhthh latent)
(b) 4-length latent
(c) 3-length latent (we use this in latent slicing)
(d) 2-length latent
Figure 23: Video reconstruction with temporally sliced latent. (a) Decoding the full latent sequence successfully reconstructs the original video. (b)-(c) Using 4 or 3-1ength latent around the target latent (frame) is sufficient for accurate reconstruction. (d) With only 2-length latent, there is slight degradation, therefore, we adpot 3-length latent for the main experiments.
C MORE DISCUSSIONS
C.I VIDEO RECONSTRUCTION WITH SLICED LATENT
As shown in Figure 23, we can reconstruct nearly identical frames even with temporally sliced latents. When the 49-mre real video at the top is encoded by CogVideoX's CausalVAE, each frame is mapped to latent 24 E Rex x with a temporal latent length of f = 13. The four reconstructed frames the right side of panels (a-d) correspond to the last four frames of the original video.
(a), we fully decode the entire 13-1ength latent 2; obtain the 49-frame reconstructed video and visualize the last four frames.
In (b), we decode only the last four temporal slices (i.e., adl: -4:)), which we refer to as 4-length latent. From this partial latent, the model produces 13-frame reconstructed video and visualize the last four frames.
These qualitative results indicates that even for fast-motion videos, 3-length latent around the target frame is sufficient for accurate reconstruction (Figure 23(c)), while a 2-length latent shows minor degradation but remains close to the full-latent result (Figure 23(d)).
C.2 TIME-TRAVEL TRICK LAYOUT STAGE
As discussed in Section 4.2, directly applying the time-travel trick (Shen al., 2024; Yu al., 2023; Bansal et al., 2024) to video diffusion models struggles due to excessive stochasticity. The time- travel trick in Algorithm includes single-step forward process, but in practice, the added noise is extremely large, and the coefficient multiplied with the latent is very small, as shown in Table 5. In fact, the very first inference step, the coefficient v3, becomes resulting in no guidance effect at all.
23

---

Published as a conference paper at ICLR 2026
Therefore, since the effect of guidance is absent during the early stages when the layout largely established, the model fails to produce layout that aligns with the given condition. Even when guidance is applied later, as discussed in Figure 4(d), only the guided frame is updated, and it cannot correct the overall layout. Our proposed VLO addresses this issue by applying deterministic latent optimization in the early stage.
Table 5: Forward process coefficients in early inference steps.
Step (-50)
vBi
v1
Bg
0.00
1.00
0.48
0.88
0.64
0.77
Additionally, we provide visual comparison Figure 24.
The time-travel trick offers almost no guidance effect in the early steps, which are crucial for layout formation. Since fails establish a proper layout early on, later steps cannot correct this deficiency. As shown in Figure 24(a), it generates static camera view, similar to ordinary 12V generation. In contrast, our VL.O provides sufficiently effective guidance through deterministic updates the early steps, enabling the model to establish proper left-to-right moving layout, which in turn allows later guidance to take meaningful effect, as illustrated in Figure 24(b).
Algorithm 2 Time Travel (diffusion model)
Algorithm 3 Time Travel-F (flow matching)
Require: 2t, Zojes t, St
Require: 2z, Zojes t,
I: - No0. I)
1: +- //( 1)
2: 2t-1 + +DD 201()
2 f- 0g6 + (1 - oz)= (Renoising)
3: At-1 + --2-1-
St
3: 2t 17 It
4: Bt + //ta/ar ar/ar-1
return 2t
5: 2, +- VBia VI Bie (Renoising)
6: return 2t
Algorithm 4 Frame Guidance (Diffusion, full)
Algorithm Frame Guidance (flow matching)
Require: I, tg. tL. repeat step M, step size f), guidance loss Le, model va(-,-)
Require: I, tg. tL. repeat step M, step size n)
guidance loss Le, model vo(:,-)
1: 2T ~ (N(0
1: 2T ~ NN00 I)
2: J +- Frame-Iax-to-LLatnn----n
2: T -ramo--dds-t--LLteen---aaaa
3: for t T, T,..., do
3: for T,.., 1 do
4: if > 1 then (Guidance step)
4: if > t then |Guidance step)
5:
for m = I,....M M - do
5: for m 1, ...., M - 1 do
6:
20j: vā,z /I a. (2(2)
Znit f 2 o, Uo (Zt, t)
7:
F01t + Latent- Sieent-Slicing(20p1
7:
.J < Latent Allent--Sicing(aan J)
8:
Of + D(20())
*Ot + D(=0(t)
9:
9: 2x, Crrames)
9:
9: Ve,Co(17 Cerames)
ift f te then (Early steps)
10:
if l > ig then (Farly steps)
21 < 21 1796
11:
f- 21 177 17.
12:
else [Later steps)
12:
else [Later steps)
13:
t 10,
13:
2: - Time-TTavel--fee
14:
4-1 < DDIM(2 20j2)
14:
end if end for
15:
4-1 t 21-1 17.
15:
16:
3. - a1/au-1
17: 21-1 +- + (0,-1 o,) v0(2s, t)
16:
end if
2, < VB
B.c
18:
end for
18:
end
19:
return 20
end for
20: end if
21: 21-1 -DDIM(2t, 201z)
22:
end for
23:
return 2g
24

---

Published as a conference paper at ICLR 2026
First frame condition (I2V input)
frame condition (Frame Guidance)
(n) Time-travel in early steps
(b) Deterministic update in early steps (VI.O)
Figure 24: Importance of VL.O in early steps. (a) The time-travel method fails to produce a proper
layout. (b) VI.O successfully generates the video, capturing the view transition from left to right.
Each top row shows early inference steps, and the bottom row shows the final generated results. Red boxes are drawn the same fixed location across all frames.
Time
20/
Latent
Disconnect
(a) Shortout method with video diffusion
(b) Visual izationzation of generated video
Figure 25: Shortcut-based approaches (He et al., 2024; Rout et al., 2025; Nair and Patel, 2024) lead to temporal disconnects in video generation.
C.3 VIDEO LATENT OPTIMIZATION (VLO) FOR FLOW MATCHING
As noted in Section A.2, we follow the time convention of diffusion models by reversing the flow matching time axis, aligning = 0 with clean data and t = T with pure noise.
In Algorithm 5, we extend our Frame Guidance to video generation models, which employ the flow matching (Lipman et al., 2022) for their generative modeling (e.g.. Wan (Wang et al., 2025a) and LTX (HaCohen et al., 2024)). Similar the diffusion case in Equation 2, we apply the latent slicing (Lines 7) and optimize the current latent 2; through the guidance loss Sr (Lines 9-11). Specifically, we predict the clean sample Zuje by based on the tweedie-like formula in Equation 4.
Time-travel for flow matching However, directly applying the time-travel trick to flow matching is non-trivial, single forward step (Line in Algorithm 2) is not explicitly defined in the context of flow matching. While renoising in time travel is effective for mitigating accumulated sampling errors, cannot be directly utilized here. Our deterministic optimization excludes renoising entirely and can be applied as is, but performing it fully during inference, as in diffusion, can result in over-saturated samples or temporally disconnected videos.
To address this, we adopt simple alternative: instead of stepping from to f-1, we move directly from to (i.c., the estimated clean latent), apply guidance there, and then simulate a forward step from 0 back to f. Although single forward step is not defined in flow matching. it is still possible to apply the forward process for time from clean data. While this process introduces higher stochasticity than single diffusion step. applying it in the later stages of VLO, after the layout has already been established, does not significantly disrupt the structure. This makes it viable option. Empirically, this approach enables the application of VLO flow matching- based models well.
25

---

Published as a conference paper at ICLR 2026
Low res. diff.
Video generation phase
Early stage
t step (-/50)
FVD ())
FID (L)
0.
0 (Time-travel)
725.30
59.03
Later stage
2
730.38
59.18
10
20
48
D0
Guidance influence
4
569.02
57.74
17
6
514.47
58.03
Latent index
8
572.55
56.18
10
617.42
59.12
12
662.37
58.17
14
642.01
58.36
10
20
48
50
16 (Rz Deter.)
662.63
58.80
Inference step
Figure 26: (Left) Video generation phases and the corresponding guidance influence maps. (Right)
Ablation study on tg.
C.4 IMPORTANCE OF GRADIENT PROPAGATION VIA DENOISING NETWORK
In training-free free guidance for image generation, "shorteut" (Rout et al., 2025; He et al., 2024; Nair and Patel, 2024) method has been proposed that utilizes a proximal gradient approach to bypass nack-propagation through the denoising network. This strategy significantly reduces memory usage and enables efficient sampling for gradient- -baieett-baeddddde optimization. While effective for static images, directly applying this method to video generation poses challenges due the temporal characteristic of video data.
Specifically, when guidance is applied to only few frames, the resulting video often becomes temporally inconsistent. As illustrated in Figure 25, the latents corresponding to the guided frames are updated resemble the target frames, and adjacent frames may also partially align. However, earlier frames remain disconnected, and the guided frames themselves may exhibit unnatural artifacts. This because temporal priors, crucial for maintaining coherence across frames, are primarily encoded in the denoising network. Consequently, for video generation tasks where temporal consistency is critical, gradient propagation through the denoising network is essential.
C.5 THE CHOICE OF THE TIMESTEP RANGE FOR STAGES (tg, tz)
As discussed in Section 4.2, VLO employs a hybrid strategy that applies different update rules depending on the generation stage. We define the early stage, where deterministic updates are applied, as complete once the low-frequency structure of the video stabilizes. Concretely, this is when the difference from the final layout falls below 20% of the difference from the initial step, shown in Figure 26 left top. To quantify this, we measure the L2 distance in the low-frequenc, region across inference steps, which confirms that video layouts are largely determined within the first few steps. Based on this stabilization criterion, we set ig automatically rather than tuning it manually according downstream video quality.
We further conduct an ablation study on tg using the keyframe-guidee generation task across 20 DAVIS videos (Figure 26 right). The results show that best performance occurs at L = which closely matches our stabilization-basel criterion. Notably, performance remains robust over a range of nearby values, indicating that the method is relatively insensitive to the precise choice of tg. Furthermore, shown in Figure 26 left, the gradient propagation map reveals that gradients become increasingly localized around the guided frame. This trend mirrors the behavior the video generation process itself.
Regarding tr, which specifies how long guidance is applied, it correlates most directly with inference time. This reveals trade-off between the strength of guidance and the additional NFEs. In practice, we set , such that the overall runtime does not exceed 4x that of the base model's inference time.
26

---

Published as a conference paper at ICLR 2026
C.6 Is TEMPORAL LOCALITY LIMITED ON RAPID MOTION VIDEO?
We conduct the same experiment in Figure 4(b) on a rapid motion video. Specifically, we replace single frame with a black image and measured the differ- ence between the latents of the original video and the modified video. To simulate rapid motion, we sparsely sample the frames from video at of 16 times the original frame rate, which results in large differences between adjacent frames.
(a) CogVideoX CausalVAE
Original Video (x1)
26
As shown in Figure 27, we still observe the same pattern in Figure 4(b), with activations remaining localized around the modified frame. Notably, this behavior persists even when we extremely increase the motion speed by up to 16x, indicating that the same localized pattern consistently holds. This result confirms that temporal locality is largely independent of motion speed, as reflects how latent frames are mapped video frames during latent decoding. Tem- poral locality stems from the design of CausalVAE, from the video content itself.
Motion speed x4
26
48
Motion speed x16
Video Frame Index (replaced frame) 48
ais
Figure 27: Temporal locality persists even under rapid motion.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this paper, we used large language models (LLMs) to assist with writing refinement, such as checking for grammatical errors.
27


Reviews:
[{"review_id": "uUMJdgKkaQ", "reviewer": "Reviewer_3A8P", "summary": "This paper proposes Frame Guidance - a novel training-free guidance framework for controllable video generation using frame-level signals such as keyframes, style references, sketches, or depth maps. The authors claim three primary contributions: (1) a model-agnostic, training-free approach compatible with diverse video diffusion models (VDMs); (2) \"latent slicing,\" a memory-efficient technique for partial decoding of video latents based on temporal locality in CausalVAEs; and (3) \"video latent optimization (VLO),\" a hybrid strategy combining deterministic updates in early denoising steps for global coherence and stochastic updates later for detail refinement. The method is demonstrated across tasks like keyframe-guided generation, stylization, and looping, achieving competitive results without fine-tuning.", "questions": "1. How does the temporal locality of CausalVAE generalize to other VAE architectures or non-causal latent spaces?\n2. Could the gradient propagation analysis (Fig. 4c) be extended to quantify temporal coherence, e.g., via optical flow consistency?\n3. For multi-condition guidance, how are conflicting losses (e.g., style vs. depth) balanced? Is there a risk of gradient domination?\n4. Have you explored adaptive strategies for selecting guided frames (e.g., based on motion complexity) to further reduce computational cost?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Among the strong points of the paper I would focus on the following ones:\n1.  Introduces a general-purpose, training-free framework for frame-level control, addressing a gap between task-specific training-free methods and general-purpose fine-tuning approaches.\n2. Rigorous experiments across multiple VDMs (e.g., CogVideoX, Wan, SVD) and tasks, supported by human evaluations and metrics (FID, FVD, CLIP scores).\n3. Latent slicing reduces GPU memory usage by up to 60×, enabling application to large-scale models.\n4. Detailed algorithms, hyperparameters, and ablation studies are provided, including adaptations for flow-matching models.\n\nThe method’s generality across models (CogVideoX, Wan, SVD) and tasks (keyframing, stylization, looping) demonstrates its broad applicability. Human evaluations and quantitative metrics show superior or competitive performance against training-based baselines. The memory efficiency achieved via latent slicing is critical for scaling to modern VDMs. The hybrid VLO strategy effectively balances layout coherence and detail preservation, addressing limitations of prior training-free methods. These contributions are valuable for the community, enabling accessible and flexible video control without costly fine-tuning.", "weaknesses": "The weak points are:\n1. Guidance increases inference time by 2–4×, limiting real-time applicability.\n2. Performance is constrained by the base VDM’s capabilities, especially for dynamic or fine-grained content.\n3. While latent slicing is justified via experiments on CausalVAE, broader validation across architectures is limited.\n4. Although multi-condition guidance is shown, combining losses for complex controls (e.g., motion + style) is not deeply explored.\n5. As you know, one of the main problems in video generation is the object state change, like melting ice, etc. There is a number of training-free solutions for such tasks, e.g., \"State & Image Guidance: Teaching Old Text-to-Video Diffusion Models New Tricks\" (https://openreview.net/forum?id=zkGxROm7D3). Deeper comparison would be a benefit of the proposed paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9GTcAG59M", "reviewer": "Reviewer_qnqQ", "summary": "The paper introduces Frame Guidance, a training-free framework for controllable video generation using video diffusion models. It enables frame-level control through latent optimization and guidance, supporting tasks like stylization, looping, and keyframe-based generation. By operating in the latent space, the method reduces memory usage while enhancing global coherence across frames. Its plug-and-play compatibility with existing video diffusion models allows versatile application across tasks without retraining.", "questions": "1. Are consistent seeds used in Fig. 6 and 7? In Fig. 7, can the method improve style alignment without altering content? Was the same seed used? \n2. What is the runtime overhead of the time-travel strategy? \n3. How does the method handle large motion or occlusion? Is the slicing strategy sensitive to motion scale and object scale?\n4. How does the method prevent generating identical frames in looped videos?\n5. What does “N-length latent” mean in Figure 19? Which frames are anchored for each latent segment? How are occluded or missing objects decoded reliably?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: While the paper builds on existing concepts like latent optimization, its integration into a training-free video generation framework with frame-level control is novel and practical.\n2. Quality: The method is well-engineered and demonstrates compatibility with multiple video diffusion backbones. The results show improved coherence and controllability across tasks.\n3. Clarity: The paper is generally well-written and structured. The methodology is explained clearly, and the figures help illustrate the effects of guidance.\n4. Significance: Training-free methods are increasingly important for accessibility and scalability. Frame Guidance contributes meaningfully to this direction by enabling flexible control without retraining, which is valuable for both research and real-world applications.", "weaknesses": "1. Limited Novelty: The core techniques (e.g., latent optimization and latent sliding) are adaptations of existing methods. The novelty lies more in the integration and application than in the underlying algorithms. The proposed did show improved performance but with basis from its underlying video diffusion model.\n2. The latent slicing strategy, while memory-efficient, may be sensitive to frame rate and motion magnitude. In cases of large motion or occlusion, it may fail to maintain coherence. No quantitative evaluation is provided to assess this. The meaning of “N-length latent” in Fig.19 is unclear, and it is not specified which frames are anchored for each latent segment. Some frames show object disappearance or occlusion, raising questions about how they are decoded reliably for other cases.\nCompared to more robust video compression models like LTX-Video use 8 temporal compression and 32 spatial compression, the slicing strategy feels more like a heuristic trick than a principled solution.\n3. The looping task lacks detail on how the method avoids generating identical frames, which is critical for realistic looping.\n4. The time-travel strategy lacks runtime analysis, and the pseudo-code appears incorrect or incomplete.\n5. The paper does not include a discussion of failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wTVgHIBDI", "reviewer": "Reviewer_TPuU", "summary": "This passage introduces Frame Guidance, a training-free method for controllable video generation that uses frame-level signals like keyframes, style references, or depth maps applied to selected frames to guide entire videos while maintaining temporal coherence. To enable this approach on large-scale models, the authors propose a memory-efficient latent processing method and a novel optimization strategy for globally coherent generation. The method works across diverse tasks including keyframe guidance, stylization, and looping, is compatible with any video model, and produces high-quality controlled videos without requiring training.", "questions": "Please refer to above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a training-free guidance method that eliminates the need for fine-tuning large-scale video models for controllable generation. \n2. This paper introduces a memory-efficient latent processing technique and optimization strategy that enables practical application on large-scale models while ensuring temporal coherence. \n3. This paper demonstrates versatility across diverse control tasks including keyframes, stylization, and looping, with compatibility across any video generation model.", "weaknesses": "1. The authors mention in the limitations that \"The computational cost of guidance sampling is higher than that of training-based methods.\" Please provide timing comparisons in the rebuttal to better benchmark the method against alternatives.\n2. This paper focuses on training-free controllable video generation, but the related works section lacks discussion of previous relevant methods, such as Tune-A-Video[1], Text2Video-Zero[2], and ControlVideo[3].\n3. Can the proposed method be extended to control video generation using canny edges, depth maps, and other modalities?\n\n[1] \"Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.\" In ICCV 2023\n[2] \"Text2video-zero: Text-to-image diffusion models are zero-shot video generators.\" In ICCV 2023\n[3] \"ControlVideo: Training-free Controllable Text-to-Video Generation.\" In ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ng8qvDtoi5", "reviewer": "Reviewer_esr1", "summary": "This paper introduces Frame Guidance, a novel training-free and model-agnostic framework for controlling video generation in pre-trained Video Diffusion Models (VDMs). The method aims to provide general-purpose, frame-level control using diverse input signals like keyframes, style images, depth maps, or sketches. By applying guidance to only selected frames, it steers the entire video generation process towards temporally coherent results. Key technical contributions include \"Latent Slicing,\" which leverages observed temporal locality in VDM latents (specifically CausalVAE) to reduce memory usage during gradient computation, and \"Video Latent Optimization\" (VLO), a hybrid optimization strategy balancing early deterministic updates for global layout and later stochastic updates for detail refinement.\n\nHowever, I think the evaluation seems a little bit out of dated. More recent Vbench and Vbench2.0 evaluation results are needed to further verify the effectiveness of the proposed method.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Training-Free and Model-Agnostic Control: The paper proposes Frame Guidance, a novel training-free approach for controlling video generation in pre-trained Video Diffusion Models (VDMs). This is a significant advantage as it avoids the need for costly fine-tuning for specific tasks or models, making it broadly applicable to various existing and future VDMs. The model-agnostic nature is well-demonstrated across different architectures.\n\n2.  General-Purpose Frame-Level Guidance: The method offers a unified framework for diverse frame-level control tasks. It supports a wide range of input signals beyond simple keyframes, including style reference images, depth maps, sketches, and color blocks, showcasing its versatility for different creative applications (e.g., keyframe guidance, stylization, looping, structure guidance).\n\n3.  Efficient Guidance Strategy for Large Models: The paper introduces two key technical contributions, Latent Slicing and Video Latent Optimization (VLO), to enable efficient training-free guidance on large-scale VDMs. Latent Slicing cleverly exploits the observed temporal locality in CausalVAE latents to dramatically reduce memory consumption during gradient computation. VLO provides a tailored optimization strategy for videos, balancing global layout coherence (early deterministic updates) with detail refinement (later stochastic updates).", "weaknesses": "The evaluation metrics used (FID, FVD) are insufficient for this task. While I am not an expert in training-free video editing, relying solely on FID/FVD seems questionable for evaluating frame-level control. These metrics may not adequately capture temporal consistency, fine-grained adherence to control signals, or motion quality. The paper should ideally include results from more comprehensive video generation benchmarks like VBench to provide a more convincing assessment. (If the authors can supplement the evaluation with VBench results, I will raise my score based on discussions with other reviewers.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)