SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Authors: Koichi Namekata
Sherwin Bahmani
Ziyi Wu
Yash Kant
Igor Gilitschenski
David B. Lindell
Venue: ICLR 2025 Poster (2025)
Keywords: zero-shot
tuning-free
self-guided
image-to-video diffusion
trajectory control
Primary area: generative models
TL;DR: We enable zero-shot trajectory control in image-to-video generation by relying solely on the knowledge present in a pretrained image-to-video diffusion model.

Abstract: Methods for image-to-video generation have achieved impressive, photo-realistic quality. 
However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., involving re-generating videos with different random seeds. 
Recent techniques address this issue by fine-tuning a pre-trained model to follow conditioning signals, such as bounding boxes or point trajectories. 
Yet, this fine-tuning procedure can be computationally expensive, and it requires datasets with annotated object motion, which can be difficult to procure. 
In this work, we introduce SG-I2V, a framework for controllable image-to-video generation that is self-guided—offering zero-shot control by relying solely on the knowledge present in a pre-trained image-to-video diffusion model without the need for fine-tuning or external knowledge. 
Our zero-shot method outperforms unsupervised baselines while significantly narrowing down the performance gap with supervised models in terms of visual quality and motion fidelity.
Additional details and video results are available on our project page: https://kmcode1.github.io/Projects/SG-I2V

Published as a conference paper at ICLR 2025
SG-12V: SELF-GUIDED TRAJECTORY CONTROL IN IMAGE-TO- VIDEO GENERATION
Koichi Namekata', Sherwin Bahmani 12 Ziyi Wu Yash Kant Igor Gilitschenski David B. Lindell
Universiit of Toronto, Vector Institute
Input
Generated Vidco Frames
Input
Generated Video Frames
Figure Image-to-video generation based on self-guided trajectory control. Given a set of bounding boxes with associated trajectories, we achieve object and camera motion control in image- to-video generation by leveraging the knowledge present pre-trained image-to-video diffusion model. Our method is self-guided, offering zero-shot trajectory control without fine-tuning rely- ing on external knowledge.
ABSTRACT
Methods for image-to-video generation have achieved impressive, photo-realistic quality. However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., in- volving re-generating videos with different random seeds. Recent techniques ad- dress this issue by fine-tuning a pre-trained model follow conditioning signals, such as bounding boxes or point trajectories. Yet, this fine-tuning procedure can be computationally expensive, and requires datasets with annotated object motion, which can be difficult to procure. In this work, we introduce SG-I2V, framework for controllable image-to-vee to-video generation that is self-guided -offeringgggggg zero-shot control by relying solely the knowledge present in a pre- tre-trained image-to-vin video diffusion model without the need for fine-tuning or external knowledge. Our zero- shot method outperforms unsupervised baselines while significantly narrowing

---

Published as a conference paper at ICLR 2025
down the performance gap with supervised models in terms of visual quality and motion fidelity. Additional details and video results are available on our project page: https: htop:://kmcoder github.io/Pro jects//G--
INTRODUCTION
Recent advances in video diffusion models demonstrate significant improvements in visual and mo- tion quality (Ho et al.] 2022b; Blattmann et al.] [2033a bJ He et al. (2022). These models typically take text prompt (Ho et al. [022b] Blattmann al. [023a Ho a 2022a) or image (Chen al.) 2023 2024a Guo et al. 2024 Xing et al. (024) as input and generate video frames of a photoreal- istic, animated scene. Current methods can generate videos that are largely consistent with input text description or image; however, fine-grained adjustment of specific video elements (c.g. object motion or camera movement) is conventionally tedious process that requires re-running the model with different text prompts or random seeds (Wu et al. 2024b Qiu et al. 2024).
Approaches for controllable video generation aim to eliminate this process of trial- -nd--rrorr through direct manipulation of generated video elements, such as object motion (Wu et al. (202cc) Yin et al.,
2023 Wang et al. (0024)), pose (Hul 2024] Xu et al. (024b), and camera movement (Wang et al.
2024c Li et a 2024 He et al., 2024a Hu et al., (024). One line of work fine-tunes pre-trained
video generators to incorporate control signals such as bounding boxes or point trajectories (Wu et al. 2024c Wang et al. 2024c). One of the primary challenges with these supervised methods is the expensive training cost, and thus, previous methods usually incorporate trajectory control by fine-tuning at lower resolution than the original model Wu et al. (2024c); Yin al. (203)). More recently, several methods for zero-shot, controllable text-to-video generation have been de- veloped (Ma al. 2023] Qiu et al., 2024 Jain al., 2024). They control object trajectories by modulating the cross tross-attention maps between features within a bounding box and an object-related text token. Still, it is not always possible to associate a desired edit with the input text prompt (con- sider, e.g., motion of object parts). Moreover, these methods cannot be directly applied to animate existing images, as they are only conditioned on text.
In this work, we propose SG-I2V, a new method for controllable image-to-vides generation. Our approach is selff-guded. in that it offers zero-shot control by relying solely on knowledge present in a pre-trained video diffusion model. Concretely, given an input image, a user specifies a set of bound- ing boxes and associated trajectories. Then, our framework alters the generation process to control the motion of target scene elements. It is essential to manipulate the structure of the generated video to achieve precise control over element positions, which is mainly decided by early denoising steps (Balaji et al. 2022 Wang & Vastola 2023). In image diffusion modcls, it known that feature maps extracted from the output of upsampling blocks are semantically aligned, i.e., pixels belonging to the same object share similar feature vectors on the feature map and thus can be used to control the spatial layout of generated images (Tang et al. 2023 Shi et al. 2024 Namekata et al. 2024 Tu- manyan al. (023).. However, our analysis reveals that feature maps extracted from the upsampling blocks of video diffusion models are only weakly aligned across frames (see Fig. 2 This misalign- ment poses challenges, as directly manipulating these feature maps fails to give useful guidance signals for layout control. Instead, we find that feature maps extracted from the self-attention layers can be semantically aligned by replacing the key and value tokens for each frame with those of the first frame (see bottom row of Fig. 2. After that, we can control the motion of generated videos by optimizing the latent (the input to the denoising network) with a loss that encourages similarity between the aligned features within each bounding box along the input trajectory. Finally, we apply a post-processing step to enhance output quality by ensuring that our optimization does not disrupt the distribution of high- fiequeeyy noise expected by the diffusion model.
In summary, our work makes the following contributions:
We conduct first-of-its-kind analysis of semantic feature alignment in a pre-trained image- to-video diffusion model and identify important differences from image diffusion models.
Building on this analysis, we propose SG-I2V, zero-shot, self-guided approach for con- trollable smage-to-video generation. Our method can control object motion and camera dynamics for arbitrary input images and any number of objects or regions of scene.
2

---

Published as a conference paper at ICLR 2025
Example I
Example 2
Example
Generated Video
Up-Block
Oulput
Self-attn.
Oulput
Modified
Self-attn.
Figure 2: Semantic correspondences in video diffusion models. We analyze feature maps in the image-to-videss diffusion model SVD (Blattmann et al. (023a)) for three generated video sequences (row 1). We use PCA to visualize the features at diflusion timestep 30 (out of 50) the output of an upsampling block (row 2), a self-attention layer (row 3), and the same self-attention layer after aligmment procedure (row 4). Although output feature maps of upsampling blocks in image diffusion models are known to encode semantic information (Tang et al.) (023). we only observe weak semantic correspondences across frames in SVD. Thus, we focus the self-attention layer and modify it to produce feature maps that are semantically aligned across frames.
We conduct extensive experiments to show superior performance over zero-shot baselines while significantly narrowing down the performance gap with supervised baselines in visual and motion quality.
2 RELATED WORK
Diffusion-based image-to-video generation.
With the recent advances in diffusion models (Shhl-
Dickstein et al. 2015 Ho et al. 2020),
image animation has achieved tremendous progress Waan
al. 2023 Chen al. 2023
2024a; Guo et al. 2024). Early methods inflate pre-trained text-
to-image models (Rombach et al.
2022) to add motions to an image (Wu et al. 2023 Khacha-
tryan al. 2023 Singer et al. 2023). A notable example is AnimateDiff (Guo al.
2024)
which learns low-rank adapters Hu et al. (202)) for different motions. Later works seek to in-
ject conditioning frame into pre-trained text-to-video model (Ho et al. 2022b He al. 2022 VideoCrafterl (Chen et al. 2023) leverages a dual cross-attention layer to condition on features of both the image and the text prompt. DynamicCrafter (Xing et al. 2024) further improves it by concatenating the input image with noisy latent. Stable Video Diffusion (SVD) (Blattmann et al., 2023a) instead works in image-only manner, which replaces the CLIP (Radford al. [022] text embedding of the text prompt with the CLIP image embedding of the conditioning frame. However, none of these models support direct trajectory control of scene elements, and rather require multiple attempts to obtain desired result. In this work, we aim to enable intuitive motion control in ani- mating a pre-existing image. Since SVD only takes an image without any text prompt, we utilize it the base model following prior work Yin et al., 2023 Wu et al. 2024c).
Spatial control in image diffusion models. One common way to incorporate spatial control into the image generation process is fine-tune pre- tre-trained models to incorporate conditioning depth
maps or bounding boxes (Zhang et al. 2023 Ye et al. 2023 Avrahami et al. 2023 Li et al. 2023 Goel al. 2024 Wang al.] (024b). While these methods demonstrate high fidelity, they require excessive computing resources and labor-intensive data annotations. Therefore, several tuning- free
approaches have been proposed (Cao
2023 Chen et al. 2024b; Feng et al.. 2023 Hertz
et al., (023). Self- Soff-Guidance (Epstein et al. 2023) Attend-and Exxiite (Chefer et al. 2023 and
TraDiffusion (Wu et al., 2024a) control the image layout by manipulating intermediate attention maps. They first estimate the generated objects' positions using the attention maps produced from text-image cross-attention layers and then optimize the latent to increase the attention values at specific positions. However, these approaches require associating the control target with a specific token in the text prompt and thus cannot be directly applied to image-only editing. Closest to ours are methods that alter image layouts without text input (Pan et al. 2023 Mou et al. 2024a b: Ling al.
2023
Liu et al. 2024 Zhang et al.) 2024c Cui et 20.24 Hou et al. 2024b] Zhao et al. 2024|jm Shhi
al. 2024).
They enforce similarity between semantically correlated feature maps extracted from
3

---

Published as a conference paper at ICLR 2025
Input latent X4
semantically aligned feature maps (See 3.3)
High feegulireeencccs preserved
poot-pooessing (Sec 3.5)
Optinized latent
PPT
Compute loes
Book propagali
X4
IFFT
Enforce feature similarity along trajectory (Sec 3:4)
Updated latent &
FFTC
2
Input condition
Figure 3: Overview of the controllable image-to-video generation framework. To control trajec- tories of scene elements, we optimize the latent 2, specific denoising timesteps of a pre- trained video diffusion model. First, we extract semantically aligned feature maps from the denoising U-Net estimate the video layout. Next, we enforce cross-frame feature similarity along the bounding box trajectory to drive the motion of each region. To preserve the visual quality of the generated video, a frequency-basee post-processing method is applied retain high-frequency noise of the original latent 2, The updated latent ž, is input to the next denoising step.
the upsampling blocks a denoising U-Net (Tang et al. 2023: Namekata et al. 2024 Hedlin et al, 2024 Zhang al. 2024a; Luo et al. 2023) In contrast, show that the cross cross- frame semantic correspondence of upsampling feature maps in image-to-video diffusion models (Blattmann et al.) 2023a is weak. Thus, optimization directly based on these feature maps leads to sub-optimal results.
Motion control in video diffusion models.
Several recent works have studied camera pose control
in video diffusion models (He et al. 2024a Xu et al.
2024a
Kuang et al. 2024 Hu et al 2024
Xiao et 2024b Hou et 2024a Bahmani al.
2024
Li et al. 2024 Zhang al. 2024bb.
The representative work MotionCtrl (Wang et al.) 2024c) fine-tunes pre-trained video generators to
follow camera trajectory input. However, video datasets with accurate camera pose annotations are
limited (Zhou et al. 2018), and fine-tuning video models requires high computation costs. Another line of work focuses on object trajectory control (Wang al. [023) Yin al. 2023 Wu et 2024c
Zhou et al.) 2024) Wang et al.] [2024a] /Zhang et al. 2024d). among which our work is particularly
related to the tuming-fire free variants (Qiu et al. 2024, Ma et al. 2023 Yang et al. 2024 Jain et al.,
2024 Yu et al. 20244). Yet, all of these methods focus text-based generation. In contrast, method uses an image-to-viiee model and thus can turn existing, real-world images into controllable control camera motion by specifying trajectories of background regions.
videos. Moreover,
METHOD
In this section, we describe our method for the trajectory control task in image-to-video generation (Sec. Our framework, SG-12V, builds on the publicly available image-to-video diffusion model Stable Video Diffusion (SVD) (Blattmann et al. 2023a). and consists of two main steps. First, we extract and semantically align the feature maps from specific layer of SVD during the early steps of the diffusion process (Sec. 3.2); we show that such feature maps are especially effective at influencing motion in the output video. Second, we optimize the noisy latent (i.e., the input to the denoising network) to enforce similarity between features within the bounding box trajectories find that naive optimization of latent is prone to overfitting and often
(Sec. 3.3). However, results in low-quality generation. Thus, we employ frequency-batt post cost-processing to retain an in-distribution noisy latent (Sec. 3.4. Our entire pipeline is summarized in Fig.
4

---

Published as a conference paper at ICLR 2025
3.1 TRAJECTORY CONTROL IN IMAGE- TO- VIDEO GENERATION
The goal of our work is to build zero-shot framework that takes an imput image I E " (with height H, width W) and generates a video with N frames, where elements in the input image move in user-specified fashion. Inspired by DragAnything (Wu et al. 2024c),
further assume that a user provides a set of B input bounding box trajectories { b= each parameterized by a height hy and width Wh, as well as center point coordinates for each output video frame: Cb,n R², 22,13 n S N. For simplicity, we denote the b-th bounding box in the n-th frame as B = (ho, Wh Co.n)-
L
assume bounding boxes cannot extend outside the image, and
Our aim is to constrain regions of the imput image falling within bounding box to follow the trajec- tory of the same bounding box in the output video. Thus, the motion of dynamic foreground objects can be controlled by placing bounding boxes around them and specifying the desired trajectory. On the other hand, camera motion can be specified by placing bounding boxes on static background regions and specifying trajectory opposite to the desired camera movement. Further, we can also set the bounding box trajectory to the zero vector to keep the region static. Overall, this formu- lation provides intuitive and unified control over object and camera motion, which are sometimes treated separately in previous controllable image-to-video frameworks (Wang et al. 2024c Yang et al. 2024.1 Li al. 2024).
3.2 EXTRACTING SEMANTIC VIDEO LAYOUT
Preliminaries: Stable Video Diffusion. Video diffusion models (Ho et al. 2022b) learn a data distribution Ep ~ (Z() by gradually denoising video corrupted by Gaussian noise. The output denoised video is thus drawn from the distribution (=() = J (a)T) dæi,T, where 27D E Nx is clean video, and #1:T are intermediate noisy samples. For simplicity, we omit the channel dimension throughout the paper. To reduce computation, Stable Video Diffusion (SVD) [Blattmann et al.) (023a)) performs the diffusion process in a latent space, where a variational autoencoder (Kingma & Welling 2013) maps raw video Zg latent zn E IR xhxw
Since this work aims to animate an existing image, we utilize the image-to- videe variant of SVD, which concatenates conditioning frame with noisy latent (z²) and runs 3D U-Net (Ronneberger ett al. 2015) to predict the noise. The 3D U-Net contains a downsampling and an upsampling path. Specifically, the upsampling path consists of three stages operating different resolutions, where each stage contains three blocks with interleaved residual blocks (He et 2016). spatial, and tem- poral attention layers (Vaswani et al. 2017). We will call these three stages bottom, middle, and top from lower to higher resolution. For more details, we refer readers to the original paper of SVD Blattmann et al. 2023a).
SVD feature map analysis. In image diffusion models, prior work has shown that output fea- ture maps of upsampling blocks in the middle stage of the denoising U-Net are semantically aligned (Tang et al.] [202]] Namekata et al.] 2024, Hedlin al. 2024 Zhang et al.] 2024a Luo let al. 2023). i.e., regions belonging to the same object tend to have similar feature vectors. Such semantically aligned feature maps are useful in estimating the layout of generated images, enabling spatial control of objects (Shi et al. 2024; Mou et al., 2024a). Therefore, we first examine whether SVD feature maps are also semantically correlated across both spatial and temporal dimension. Fig. 2 visualizes the principal components of feature maps extracted from the upsampling block and spatial attention layers. We observe that SVD feature maps exhibit weak semantic correspondence across frames at early denoising steps, leading to inaccurate object trajectory estimation. Yet, we want to operate at early steps as they decide the structure of generated videos (Materzynska et al.) 2023). This dilemma prompts us to align these features before applying optimization.
Feature alignment with modified self-attention. SVD leverages separate spatial and temporal self-attention to model the entire video. Since spatial self-attention is only applied per frame, it does not produce cross-fraid frame aligned features. While temporal attention communicates across frames, only attends to the same pixel position on the feature map, which may be inadequate for capturing semantic information spatially. To address this issue, inspired by (Wu et al.] 2023). we modify the spatial self-attention on each frame to directly attend to the first frame. Concretely, for the n-th frame, the original spatial self-attention works as F, = Softmax( Q.. Vn. where F, is the outputs of self-attention, Q, K, V are the query, key, and value tokens, respectively, and
yD
5

---

Published as a conference paper at ICLR 2025
D is the dimensionality of the key and query tokens Vaswani al. (2017). Instead, we replace the key K, and value Vn of each frame with K1 and Vn2 from the first frame, leading to a new operation Fn SoftMax( 2 SG(V)). )-V(V)) We apply stop gradient SG(-) K and Vi to stabilize the subsequent optimization process. Now, all the modified feature maps F, are weighted combinations of V1, exhibiting a stronger cross-frame correspondence while still maintaining the object layout of each frame, as shown in the bottom row Fig. [ Notably, this modification occurs during the loss computation only and does not affect the denoising steps.
VD
3.3 TRAJECTORY CONTROL WITH LATENT OPTIMIZATION
So far, we have obtained spatio-temporally aligned feature maps (xt) E Rxxxx at each frame given the noisy latent 2, as input (for simplicity, we resize F, to the same resolution as the noisy latent). Recall that our goal is to control the output video frames so that the bounding boxes B; identified in the first frame move along the associated trajectories. Inspired by prior brag-basedd control methods (Shi et al. 2024) Pan et al., 2023). we optimize the noisy latent 2 to enforce cross- frame similarity between features within bounding boxes. The optimization objective is as follows:
= arg min St
Σ 6E(1..)
|G; @ (its@(Fatee (F.(z) - SG (z.)
(1)
where @ is Hadamard product, and (B))) E Reaxwsxd is feature maps cropped by the bounding box B. Following DragAnything (Wu et al. 2024c). we weight the feature difference using a Gaussian heatmap G. E h This focuses on optimizing pixels closer to the bounding box center, as pixels near the edge may be background pixels that we do not want to move.
Selective latent optimization. We optimize Eq. (I) on a subset of denoising timesteps and self- attention layers. Concretely, we only select early denoising timesteps as the coarse structure of output frames determined at these timesteps (Wang Vastola 2023 Materzynska et al.) 2023). In addition, consistent with previous works in image diffusion models (Shi et al. [224]] Mou et al., 2024a), we observe that feature maps extracted from the middle stage the denoising U-Net are more semantically correlated, and thus we use them for optimization.
3.4
FREH--RREUENCY PRESERVED
Although the presented pipeline already enables trajectory control in video generation, we notice quality degradation in generated videos. We attribute this degradation to the deviation of 2 from the sampling distribution of the diffusion process after optimization. A recent work FreeInit (Wul ctt al. 2024b) observed that motions of generated videos are mostly encoded in the low- owwfirequency component of noisy latent. Inspired by this, we propose to discard the high-frequency component of the optimized latent 2; and replace it with the high-frequency component of the original latent z,- Formally, obtain the new latent ž, as follows:
1S IFFT2D (FFT)D(2;) @ H. + FFF22((2) の (1 - -H,)),
(2)
where FFT2D is the Fast Fourier transformation applied to each frame, and IFFT2D the corre- sponding inverse operation. We set H,, as the frequency response of 2D low-pass filter (we follow FreeTraa and use a Butterworth filter) with cut-off frequency 7. This post- post-processing step retains the target motion signals encoded in the low- oow-freuency component of 2, while eliminating undesirable high-frequency disruptions.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation details. In all experiments, we leverage the image-to-video variant of Stable Video Diffusion (Blattmann et al. 2023a to generate videos with 14 frames and 576 x 1024 reso- lution. The default discrete Euler scheduler (Karras et al., 2022) is applied with = 50 sampling steps. We extract feature maps from the last two self-attention layers from the middle stage in the denoising U-Net. We optimize Eq. (T) at the early denoising timesteps E [45, 44..... 30) for iterations per timestep. We use the A optimizer (Loshchilov & Hutter 2019) with learning

---

Published as a conference paper at ICLR 2025
Drag- NUWA
Ours
Drag-
Anything
Ours
Figure 4: Failure cases in supervised baselines. We observe that DragNUWA tends to distort objects rather than move them, and DragAnything is weak at part-level control as it is designed for entity-level control. In contrast, our method can generate videos with natural motion for diverse object and camera trajectories. Please see our project page for additional comparisons.
rate of 0.21. All these design choices carefully ablated and analyzed in Sec. 4.4 During loss calculation, Gaussian heatmap G, is constructed following (Wu et al.j (202c),, where a heatmap for bounding box of size (hs, wx) is created by Gaussian distribution with standard deviation = (0.2hy, 0.2wg). For the low-pass filter Hy,, we set the cut-off frequency 7 to 0.5.
Baselines. We compare with methods that enable motion control on existing images and have publicly available code. Specifically, we compare with supervised baselines DragNUWA (Yin al. 2023) and DragAnything (Wu et al., 2024c). which both add motion adapters to SVD. We also compare with Image Conductor (Li et 2024), which based on AnimateDiff (Guo al. 2024). Since all supervised baselines are tine-tuned to generate videos at lower resolution, we resize the generated videos from all the methods to 320 x 576 for fair comparison. Since no previous methods exist for zero-shot crajeoloyycootrollled image-to-video generation, adopt techniques from text-to-video methods to create new baselines. Specifically, FreeTraaj incorporates the noise initialization technique from (Qiu et al. (2024) by copy-pasting the initial noise the first frame to other frames along the trajectories. Drffuiifusion' is inspired by the image editing method in (Shi et al. 2024), which utilizes feature maps extracted from the outputs of upsampling blocks to guide the generation process without feature alignment. MOF71 instead utilizes feature maps derived from Content Correlation Removal proposed by Xiao al. (2024a) known encode motion information.
Datasets and evaluation metrics. Following prior works (Wu et al. 2024c Zhou et 2024), we evaluate our method on the validation set of the VIPSeg dataset (Miao et al. (022).. We test on the same control regions and target trajectories as DragAnything, where the size of our bounding boxes is the same as the diameter of the circles in their work. For quantitative metrics, we report Frechet Inception Distance (FID) (Heusel et al., (017) and Frechet Video Distance (FVD) (Un-) terthiner al. 2018) to measure the visual quality, and ObjMC (Wu al. 2024c) measure the motion fidelity. ObjMC computes the average distance between generated and target trajectories, where Co- Tracker (Karaev et al. 2024) is used to estimate the trajectory of generated videos.
4.2
QUALITATIVE RESULTS
Fig. presents the versatile control ability of our method. We can control foreground objects to perform rigid motions, such trains moving, and non-rigid motions, such the movement of human hairs. In addition, we can control non-physical entities such as smoke and fire. The moved scene elements naturally adapt the new location while preserving their original identity. Thanks to our general formulation of trajectories, camera motion control is also supported. We highly encourage the reader view ourr project page for additional results.
4.3 QUANTITATIVE ASSESSMENT
Comparison with supervised baselines. We provide quantitative comparisons to baselines in Tab. Despite being trained on large-scale datasets, Image Conductor underperforms method

---

Published as a conference paper at ICLR 2025
Method
FID (4)
FVD (4)
ObjMC (L)
Zero-shot
Resolution
Backbone
Image Conductor
48,81
463.21
21.07
256 384
DiifateDiff v3
DragNUWA v1.5
30.73
253.57
10.84
320 576
SVD
Drythng
30.81
268.47
11.64
320 x 576
SVD
SVD (No Control)
30.50
340.52
39.59
576 x 1024
SVD
FreeTrajj
46.61
394.14
36.43
576 1024
SVD
MOFT
30.76
402.09
33.58
576 x 1024
SVD
DragDiffusion'
30.93
458.29
31.49
576 x 1024
SVD
SG-12V
28.87
298.10
14.43
576 1024
SVD
indicates methods adapted to our image-to-video setting.
Table 1: Quantitative comparison on the VIPSeg dataset. Despite being a zero-shot method, achieve small gaps in motion fidelity (ObjMC) supervised baselines without degrading video quality (FID, FVD). Furthermore, our approach outperforms zero-shot baselines across all metrics.
FID across Pestures (c)
acroes Festures (1)
ObjMC across Fantures
34
30-
460-
27-
40t-
360-
0p 192 28 TV 502 BA S0u
Up 122 TR. TV
10
36
3V
Crace
0p TQ IX TV 20 502 5K 3V 20700m
Upe
Upesmpling Block Output
TK: Temporai Key
TO:
Tempora Outgut
Sputisl Key
50: Spotisl Outpui
TQ: Tempoma Query
TV: Temporal Value
Spartal Query
SV: Spatial Valce
Oune Aliined Spatial Outget
Figure 5: Performance across U-Net feature maps used to compute loss in Eq. 11. For all metries, lower values are better. Temporal and spatial refer to the temporal and spatial self-attention layers. We find that features extracted from self-attention layers generally perform better than those from upsampling blocks and temporal attention layers. In addition, using the feature maps of our modified self-attention layer achieves the best results, since they are semantically aligned across frames. Corresponding qualitative visuals are presented in Fig. 13 and our project page.
by large margin, mainly because of the limited capacity of the base model AnimateDiff.. Com- pared to methods that also build upon SVD, we achieve competitive performance in visual quality, with slightly worse motion fidelity. Yet, these methods are trained to generate low-resolution (i.e., 320 576) videos due to the high cost of fine-tuning high resolution. At the same time, tuning-free approach can maintain the original resolution SVD (i.e., 576 x 1024). Fig. iillus- trates comparison in failure cases of supervised baselines. Similar to observations in (Wu et al., 2024c), DragNUWA tends to distort objects rather than naturally move them, while our method can generate more natural movements. DragAnything is weak at part-level motion control (e.g., closing a cat's mouth) as it is only trained on datasets annotated with entity-level control, such as object segmentation masks. contrast, our approach can handle different granularities of control regions.
Comparison with adapted zero-shot baselines. The noise initialization technique in FreeTraii improves motion control slightly compared to original videos but significantly degrades visual qual- ity. This indicates that motion prior can be easily incorporated into initial noises of SVD by hand-crafted algorithm. DragDiffusion' and MOFT* leverages feature maps that are not semanti- cally corresponding across frames. As a result, the motion fidelity much lower than ours. Overall, SG-I2V significantly outperforms zero-shot baselines across all metrics. This highlights that zero- shot techniques applied text-to-video diffusion models do not necessarily transfer to image-to- video models.
4.4 ABLATION STUDIES
Feature map selection. Fig. 5 we analyze the effect of the choice of U-Net feature map in the optimization of Eq. [I] DIFT (Tang et al. 2023) pointed out that outputs from the U-Net upsampling block in image diffusion models have strong semantic correspondence across spatial di- mensions, which is used in image editing methods (Shi et al. 2024). However, find them to have inferior correspondence across the temporal dimension, and optimizing them leads to inaccurate object trajectories, indicated by the high ObjMC value. Next, examine features in temporal
8

---

Published as a conference paper at ICLR 2025
FTD across Layers (1)
FVD across Layers (4)
Obj MC across Layers (L)
60
600
40
400
20
10
13M53
B1:
Bottom
B3: Bottom
M2: Mid
T1: Top
13: Top
B2:
Mid
M3:
Top 2
M42-3: Mid 2-3 (Ours)
Figure 6: Performance across U-Net layers used to extract feature maps. Lower is better for all metrics. Bottom, mid, and top indicate the three resolution levels the U-Net's upsampling path, each containing three self-attention layers numbered 1, 2, and 3. for example "M2-3" means applying the loss to features from both mid-resolution layers and 3. We observe that mid-resolution feature maps perform best for trajectory guidance. In addition, using features from both M2 and M3 leads to the best result. See our project page for visualizations.
Input
w/o post-processing
w/ post-processng
Figure Effect of high-frequency preservation in post-processing Videos without post- processing tend to demonstrate oversmoothing and have artifacts. In contrast, our post-processing technique retains videos with sharp details and eliminates most of the artifacts. See our project page for more examples.
(1) FFT cutoff (x)
(1) FFT rutoff (7)
ObjM (1) va cutoff (x)
35
30
80
Figure 8: Study of the cut-off frequency in post-processing. Lower is better for all metries. The value y indicates the cut-off frequency. Fully keeping the optimized latent (y = 1) results in degraded video quality, shown high FID and FVD values. On the other hand, replacing many frequency components diminishes motion control, as indicated by the increasing ObjMC.
self-attention layers. However, using these features also leads inferior motion guidance, as indi- cated by the high ObjMC errors. This may be because the temporal layers in SVD always attend to the same spatial location, thus focusing less on each frame's 's spatial layout. Finally, we study each component in spatial self-attention layers. As discussed in Sec. 3.2. the lack of cross-frame corre- spondence in the original self-attention is problematic-in Fig. [ we see that this results in worse ObjMC and FVD scores. Overall, optimizing outputs from our modified self-attention operation achieves the best motion fidelity, showing the importance of feature alignment.
Cut-off frequency in post-processing We study the cut-off frequency in our high-frequency-
preserving post-process proteprocessing step in Fig. g the visual quality drastically, shown by the higher FID and FVD. Yet. discarding part of the high- frequency component has negligible impact on motion control while eliminating most artifacts. We thus choose 7 0.5 sweet spot for effective video quality restoration.
Naively keeping the optimized latent (y = 1) degrades

---

Published as a conference paper at ICLR 2025
FID va Learning Rate (4)
FVD vs Learning Rate (1)
OhjMC vs Learning Rate (4)
35
350
16
325
14
30
022
0.00
0.00
0.00
Figure 9: Ablation on optimization learning rates. Larger learning rates lead to video quality degradation (i.e., higher FID and FVD), while smaller learning rates result in lower motion fidelity (i.e., higher ObjMC). We choose the learning rate considering this tradeoff.
Denoising Timestop
vs Denoising Timestep (1)
OhiMC vs Denoising Timestop (
35
S500
25
1500
200
25
00
35
30
22
10
30
30
20
10
30
45
40
35
30
25
10
Figure 10: Effect of optimizing latent at individual denoising timesteps. For all metrics, lower values are better. Here, we optimize Eq. ( on a single denoising timestep (7 = 50 corresponds standard Gaussian noise), and we find middle timesteps (e.g. 30) achieve the best motion fidelity while maintaining visual quality. More results on optimizing the latent at multiple timesteps can be found in Fig. 16 See Fig. 15 and our project page for qualitative comparisons.
U-Net layer. Fig. alblates from which layer extract the feature maps. The upsampling path of SVD's U-Net contains three stages (bottom, mid, top), each with three spatial self-attention layers (indexed 1, 2, 3). We observe that blocks the middle-resolution level capture most semantically meaningful information, which aligns with prior work in image diffusion models (Shi et al., 2024] Tang ct al. 2023). We also try joint optimization of several layers, which gives the best performance.
Learning rate. Fig. examines the effect of learning rate under a fixed number of optimization steps. We observe clear trade-off between visual quality and motion fidelity of generated videos. This is because a large learning rate quickly leads to noisy latents that are out of distribution.
Denoising timesteps. Fig. 10 summarizes the effects of optimizing features across different de- noising timesteps
corresponds to standard Gaussian noise). Lower timesteps (e.g., f = 20, 1) significantly impair visual quality, motivating us optimize only on earlier denoising steps. Con- versely, timesteps for / 45 degrade motion fidelity due to the lack of detailed semantic information at extremely high noise levels. Notably, ObjMC improves at t [20, 1]. but due to the Co-tracker tracking moving artifacts (e.g., see project webpage) Based on these observations, we optimize feature maps extracted between timesteps 30-45 to balance visual quality and motion fidelity.
5 CONCLUSION
Our work introduces the first framework for zero-shot trajectory control in image-to-video gen- cration. Our thorough analysis of diffusion features reveals that the knowledge acquired in the pre-trained image-to-video diffusion models guide them to generate videos with desired mo- tions. Quantitative and qualitative results demonstrate the effectiveness of our approach, both on synthetic and real-world images. We hope findings can shed light the inner mechanism of image-to-video diffusion models and inspire better architecture designs in the future.
Limitations and future work. Since our pipeline works in a zero-shot and self-guided manner, the base video diffusion model bounds the quality of generated videos, e-g., for subjects with large motion or complex physical interactions (Blattmann et al. 2023a). Another area for improvement the out-of-distribution issue when optimizing the latent code. Although we have incorporated frequency-basee post-processing to mitigate it, we sometimes still observe artifacts when set the learning rate higher. How to alter the denoising process while keeping in-distribution latents is an open problem itself (He et al. 2024b; Garibi et al.. 2024). With the recent rapid progress in video generators (Brooks et al. 2024 Gupta et al. 2023). ) we also expect our framework could be extended newly released models to achieve improved generation quality.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
DBL acknowledges support from NSERC under the RGPIN and Alliance programs, the Canada Foundation for Innovation, and the Ontario Research Fund.
ETHICS STATEMENT
Currently, our synthesized videos are not yet indistinguishable from real footage. Still, we an- ticipate that future iterations large video models will achieve even greater quality, potentially producing videos that look photoreal. This prospect brings significant ethical and safety consider- ations, as cutting-edge generative models can be misused to ill effect. We advocate against misuse of generative models to malign misinform, and we encourage safe and responsible use of such models (Google 2023).
REFERENCES
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Yin. SpaText: Spatio-textual representation for controllable image generation. In CVPR, 2023.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407. 127811 2024.
Yogesh Balaji. Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Mïika Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023b.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video gen- eration models world simulators. OpenAl technical reports, 2024. UtLhhttps:///oen com/ http://rseeaaccc.ceeeeeeeeraaaaa gemeratinnnnnnnnne models walksswwollee world- world-simulatorr
Mingdeng Cao, Xintao Wang. Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. MasaC- trl: funing-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attention-bass semantic guidance for text-to-image diffusion models. ACM TOG, 2023.
Attend-and-excite:
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512. 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024a.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-fre free layout control with cross-attention guidance. In WACV, 2024b.
11

---

Published as a conference paper at ICLR 2025
Yutao Cui, Xiaotong Zhao, Guozhen Zhang, Shengming Cao, Kai Ma, and Limin Wang. Stabledrag: Stable dragging for point-base image editing. arXiv preprint arXiv:2403.04437, 2024.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. NeurlPS, 2023.
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fres structured diffusion guidance for compositional text-to-image synthesis. In ICLR, 2023.
Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Eloe and Daniel Cohen-Or. Renoise:
Real image inversion through iterative noising. In ECCV, 2024.
Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. PAIR-Diffusion: Object-level image editing with structure-and -ede--anaaappaaneeeee paired diffusion models. In CVPR, 2024.
Google.
Safety & fairness considerations for generative models, 2023. URL https://
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang. and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXhv:2404.02111, 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.132 2022.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In ICLR, 2024b.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. NeurlPS, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NewriPS, 2022b.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-fre camera control for video generation. arXiv preprint arXiv:2406. 10126, 2024a.
12

---

Published as a conference paper at ICLR 2025
Xingzhong Hou, Boxiao Liu, Yi Zhang, Jihao Liu, Yu Liu, and Haihang You. Easydrag: Efficient point-based manipulation on diffusion models. In CVPR, 2024b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
Li Hu. Animate Anyone: Consistent and controllable image-to-vine video synthesis for character anima- tion. In CVPR, 2024.
Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-fre free camera motion transfer for video generation. arXiv preprint arXiv:2404. 150,:22944.1799, 2024.
Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. In CVPR, 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. In ECCV, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurlPS, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models zero-shot video generators. In CVPR, 2023.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint wwx.iv24055.74111 2024.
Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. Image conductor: Precision control for interactive video synthesis. arXiv preprint arXiv:2406. 243.139900..333, 2024.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-set grounded text-to-imape generation. In CVPR, 2023.
Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, and Yi Jin. Freedrag: Point tracking is not you need for interactive point-based image editing, arXiv preprint arXiv:2307.04683 2023.
Haofeng Liu, Chenshu Xu, Yifei Yang. Lihua Zeng, and Shengfeng He. Drag your noise: Interactive point-based editing via diffusion semantic propagation. In CVPR, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In NeurlPS, 2023.
Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. arXiv preprint arXiv:2401.00883 2023.
Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Rus- sell. Customizing motion in text-to-video diffusion models. anXiv preprint arXiv:2312.04900 2023.
Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In CVPR, 2022.
Chong Mou, Xintao Wang, Jicchong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation diffusion models. In ICLR, 2024a.
13

---

Published as a conference paper at ICLR 2025
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting accu- racy and flexibility on diffusion-based image editing. In CVPR, 2024b.
Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models. In ICLR, 2024.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image mani- fold. ACM TOG, 2023.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang. Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. arXiv preprint arXiv:2466 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In CVPR, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-vident Text-to-video generation without text-video data. In ICLR, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In (CML,20155
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurlPS, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint
2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
Binxu Wang and John J Vastola. Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv preprint 23X1:22303.02490 2023.
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566. 2024a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang. Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. NeurIPS, 2023.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. InstanceDif- fusion: Instance-level control for image generation. In CVPR, 2024b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In ACM SIGGRAPH PH 2024 Conference Proceedings, 2024c.
14

---

Published as a conference paper at ICLR 2025
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-o-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023.
Mingrui Wu, Oucheng Huang, Jiayi Ji, Jiale Li, Xinyue Cai, Huafeng Kuang, Jianzhuang Liu, Xi- aoshuai Sun, and Rongrong Ji. Tradiffusion: Trajectory-based training-free image generation, 2024a.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initial- ization gap in video diffusion models. In ECCV, 2024b.
Wejia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In ECCV, 2024c.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint vrxiv22455.14864 2024a.
Zeqi Xiao. Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free free motion interpreter and controller. arXiv preprint arXiv:2405. 2024b.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. DynamiCrafter: Animating open-domain images with video diffusion priors. In ECCV, 2024.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-vides generation. arxiv preprint arXiv:240.002509 2024a.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. MagicAnimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024b.
Shiyuan Yang. Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Proceedings, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 06721, 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint ar Xrx:22230..0 08089, 2023.
Shoubin Yu, Jacob Zhiyuan Fang, Jian Zheng, Gunnar A Sigurdsson, Vicente Ordonez, Robinson Piramuthu, and Mohit Bansal. Zero-shot controllable image-to-video animation via motion de- composition. In ACM MM, 2024.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurlPS, 2024a.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. ICCV, 2023.
Wan Zhang, Sheng Tang, Jiawei Wei, Ruize Zhang, and Juan Cao. Dragentity.trajectorr guided video generation using entity and positional relationships. In ACM Multimedia 2024, 2024b.
Zewei Zhang, Huan Liu, Jun Chen, and Xiangyu Xu. Gooddrag: Towards good practices for drag editing with diffusion models. arXiv preprint arXiv:2404.00 072066, 2024c.
Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory- oriented diffusion transformer for video generation. arXiv preprint 10Xiv:2477.21705, 2024d.
15

---

Published as a conference paper at ICLR 2025
Xuanjia Zhao, Jian Guan, Congyi Fan, Dongli Xu, Youtian Lin, Haiwei Pan, and Pengming Feng. Fastdrag: Manipulate anything in one step. arXiv preprint arXiv:240S.15769 2024.
Haitao Zhou, Chuang Wang, Rui Nie, Jinxiao Lin, Dongdong Yu, Qian Yu, and Changhu Wang. Trackgo: A flexible and efficient method for controllable video generation. arXiv preprint arXiv:2408. 11475, 2024.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM TOG, 2018.
16

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP DETAILS
Details of FVD. As mentioned Sec. we resize generated videos to 320 x 576 before feeding them into the evaluation scripts to ensure fair comparison. For FVD computation, we further resize all videos to 256 x 256, following DragAnything (Wu et al. 2024c).
Details of ObjMC. Following DragAnything (Wu et al. 2024c), we compute ObjMC to evaluate motion fidelity. ObjMC is defined the framewise average distance between the generated and ground truth trajectories, where the ground truth trajectories are reused from DragAnything, and the generated trajectories estimated using Co-Tracker v2 (Karaev et al. 2024). Since the VIPSeg dataset contains videos with varying resolutions (e.g.. 480 x 800, 1440 x 2560), we resize all videos uniform resolution (as specified in Tab. [ before feeding them into the models.
Unlike DragAnything, which resizes generated videos back to their original resolutions for ObjMC computation, resize all videos ground truth trajectory points that fall outside the image space due to objects moving out of frame. We also omit short videos with fewer than 14 frames from the evaluation.
fixed resolution of 320 576. Additionally, exclude
Details of zero-shot adopted methods. Due to the unavailability of zero-shot trajectory-controlled image-to- video generation methods, we adopt techniques from text-to-image and text-to-video methods to establish new baselines in Tab. [
FreeTraft incorporates the noise initialization technique (called trajectory injection) from (Qiu al, 2024). This work has shown that introducing motion bias into the initial boise-by copy-pasting the noise from the first frame to other frames along the trajectory-can influence motion in the generated videos of text-to-vider diffusion models. However, we did not observe similar effects in image-to- video diffusion models. Instead, we found that this approach degraded visual quality, likely because the modified input latents fell out of distribution.
DragDiffusion' is inspired by the image editing technique in (Shi et al. 2024), which utilizes feature maps extracted from the outputs of upsampling blocks to guide the generation process. Specifically, we optimize on the feature maps output by the second and third upsampling blocks at the middle resolution level. Although the original DragDiffusion does not include high-frequency preserved post-processing, we incorporate it in our baseline to enhance visual quality. We use the same hyper- parameters as in our method.
MOFT is similar to DragDiffision' but optimizes feature maps derived from Content Correlation Removal, as proposed by Xiao et al. (2024a). Specifically, we optimize on the outputs of the up- sampling layers subtracted by their mean features averaged across frames. These feature maps are well-suited for reference-guide guided motion generation because they are motion-consister (i.e., pix- els with similar motion exhibit similar feature vectors), enabling direct comparison with reference motion feature maps. However, they are less sensitive to semantic information and, therefore, inef- fective for our optimization, where reference motion is unavailable.
B ADDITIONAL QUALITATIVE RESULTS
We strongly encourage readers to refer to our project website: htttps://kmcode: . io/Pro Soectt//G--a T2V for more visualizations in video format, where we have released qualita- tive results with various trajectories, guallitatve results on VIPSeg dataset, qualitative comparisons with supervised and zero-shot baselines, and qualitative analysis for ablation study.
C ADDITIONAL RESULTS FOR ABLATION STUDY
In this subsection, we provide full results for our ablation analysis.
Inference time. Our method generates 14 frames videos at resolution of 576 x 1024 with T = 50 sampling steps. Optimization is performed from the timestep 45 to the timestep 30 with 5 iterations per timestep, totaling 75 optimization iterations. The runtime depends the number of trajectory conditions, with an average runtime of 305 seconds on the VIPSeg dataset with A6000 48GB. Due to the need of backpropagation, the peak GPU memory usage amounts around 30 GB.
17

---

Published as a conference paper at ICLR 2025
Mask shape
FID (1)
FVD (J)
ObjMC (L)
Gaussian weighting
28.87
298.10
14.43
Identity weighting
28.88
300.20
14.72
Table 2: Ablation on Gaussian weighting on the VIPSeg dataset. Using Gaussian heatmap in loss computation consistently improves the results across all metrics.
PCA visualization. Fig. 11 and Fig. 12 present the visual results of PCA analysis on feature maps extracted from various layers of SVD across different timesteps (corresponding to Sec. 3.2) We can confirm that our modified self-attention consistently produces semantically aligned feature maps throughout the denoising process,
Feature map selection. Fig. presents the visual results of ablating feature maps for optimization. Performing optimization with feature maps naively extracted from original self/temporal attention or upsampling blocks fails follow the input trajectory due to the weak semantic alignment across frames. In contrast, performing optimization with our modified self-attention features successfully produces videos consistent with the input trajectory. This highlights the importance of extracting semantically aligned feature maps for our optimization.
has qualitatively confirmed that optimizing with feature maps extracted from the middle layers of the upward path produces plausible videos consistent with the input trajectory.
U-Net layer. Consistent with the quantitative results (Fig. 6 Fig. 14
Denoising timesteps. Continuing from the main paper, further analyze the effect of extracting feature maps from different timesteps. Fig. [5 qualitatively demonstrates the effect of optimizing latent at individual denoising timesteps. Consistent with the quantitative results demonstrated in Fig. T0 performing optimization at later stage denoising steps (i.e. t = 10, 20) severely intro- duces artifacts in the generated videos. Next, examine the effects of performing optimization for multiple timesteps by fixing the total number of iterations for optimization. As summarized in Fig. I6 observe similar trends as optimizing individual timesteps, where performing optimiza- tion at the later stage of the denoising process significantly degrades visual quality, while performing optimization up to timestep is not enough for motion control. Overall, running optimization for multiple timesteps performs better than running optimization only at a single timestep, and hence we adapt continuous timesteps for our experiments.
Gaussian weighting. Following Dryhhni, we weigh the feature difference using a Gaussian heatmap during the loss computation Sec.B.3) This accounts for potential errors in placing bound- ing boxes, where the bounding box may include background pixels around the object that are not intended to be moved. As shown in our ablation on the VIPSeg dataset (Tab, 2}. Gaussian weighting results in small but consistent improvement across all metries.
18

---

Published as a conference paper at ICLR 2025
Time- 50
Up-Block
Self-attm
Modified
self-attn
Time 40
Upsample (DIFT)
Self-attn
Modified
Self-attn
Time 30
Upsample (DIFT)
Self-atmn
Modified
Self-attn
Upsample (DIFT)
Time-22 20
Self-atim
Modified
Self-uttn
Time=10 10
Upsample (DIFT)
4
Self attn
Modified Self-attn
Upsample (DIFT)
Time=1
Selt-attn
Modified
Self-attn
Figure 11: Semantic correspondences in video diffusion models across timesteps. Output feature maps of upsampling blocks have limited semantic correspondences across frames. In contrast, our modified self-attention layers produce semantically aligned feature maps across all the timesteps.
19

---

Published as a conference paper at ICLR 2025
Upsample (DIFT)
Temp-artu Query
Temp-atta Key
Temp-aita Outpur
Temp-sttn Outpur
Self-attn.
Query
Self-attn.
Key
Self-wita.
Valce
Self-attu
Output
Modified
Self-atin
Figure 12: Semantic correspondences of different features in video diffusion models. We find features from self-attention layers to be more semantically aligned than that of temporal attention layers and upsampling layers, while our modified self-attention layer produces the aligned results due to its explicit formulation to attend to the first frame
20

---

Published as a conference paper at ICLR 2025
Modified Self-attn.
Self atin. Query
Self-attn, Key
Self-attn.
Value
Self-attn.
Output
Temp-attn.
Query
Temp atin, Key
Temp-attn. Value
Temp-atta.
Output
Upsample (DIFT)
Figure 13: Ablation on U-Net feature maps. Applying loss on feature maps extracted from original self/hemporal-attenle layers upsampling blocks fails to follow the trajectory due the semantic misalignment across frames. In contrast, performing optimization with our modified self-attention layers can produce videos consistent with the input trajectory, indicating the importance of using semantically aligned feature maps. Please see our project page for more qualitative results.
21

---

Published as a conference paper at ICLR 2025
Botton
Botion 2
Mid
Mid 3
3p
213
Figure 14: Ablation U-Net layer to extract feature maps. Consistent with the quantitative results in Fig- 6 feature maps extracted from middle resolution level are most useful for trajec- tory guidance. Optimizing other feature maps may generate unrealistic videos with low motion fidelity.
Input
Timestep 50
Timestep 40
Timestop 30
Timestop = 20
Timestop 10
Figure 15: Visual comparison of different denoising timesteps. Here we show the last frame of generated video. Optimizing latent at later denoising process leads to severe artifacts.
22

---

Published as a conference paper at ICLR 2025
Timesteps Range (1)
Timesteps Range (4)
ObjMC Timestepe Range (1)
25
75
1000
20
50
500
18
25
40 80
50
1- 50
50
45
80
50
50
50
45
10
50
30
20
20
10
40
30
30
20
10
Figure 16: Effect of optimizing latent multiple denoising timesteps. Here we perform opti-
mization on multiple denoising timesteps (t 50 corresponds to standard Gaussian noise). Similar to performing individual timestep Fig. TO performing optimization up to middle timesteps (e.g. 50 30) achieves the best motion fidelity while maintaining visual quality.
23


Reviews:
[{"review_id": "x7zaXbHFg5", "reviewer": "Reviewer_StsS", "summary": "The paper introduces SG-I2V, a self-guided method designed to achieve zero-shot controllable image-to-video (I2V) generation. Technically, SG-I2V first extracts feature maps from the self-attention module during the video denoising process. To ensure that the generated video’s motion aligns with the given trajectory, the input latent code is optimized by enforcing cross-frame feature consistency along the bounding box trajectory. Furthermore, to preserve the quality of the generated video, the authors propose a post-processing method that retains the high-frequency components of the original latent code. The evaluation on the VIPSeg dataset demonstrates the effectiveness of the proposed method.", "questions": "Q1: The paper proposes optimizing the input latent code and applying high-frequency preserved post-processing between steps 30 and 45 during the denoising process. The inference time required to generate a 14-frame video should be reported.\n\nQ2: I am curious why the modified spatial self-attention mentioned in Section 3.2 shows better alignment with the initial frame. A more detailed explanation would be helpful. Additionally, after modifying the self-attention in SVD, there is a gap between the modified inference pattern and the original learnt pattern, could this affect the quality of video generation? It would be beneficial to evaluate the change in FVD for I2V generation before and after modifying the self-attention in SVD. \n\nQ3: I hope the authors can address the concerns raised in the \"Weaknesses\" section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1: The paper analyzes the intrinsic relationship between the motion in the generated video and the UNet feature maps. \n\nS2: Building on this analysis, a self-supervised framework is designed to achieve zero-shot controllable image-to-video (I2V) generation.", "weaknesses": "W1: The performance values reported in Table 1 on the VIPSeg dataset differ from those in the DragAnything paper, which reports ObjMC, FVD, and FID values of 305.7, 494.8, and 33.5, respectively. Additionally, Table 1 shows DragNUWA outperforming DragAnything, which contradicts the findings in the DragAnything paper. Could the authors clarify the specific versions of DragAnything and DragNUWA used in their experiments, any differences in their evaluation setup compared to the original papers, and whether they used the same evaluation code and metrics? This information would help pinpoint the source of these discrepancies and ensure a fair comparison.\n\nW2: Generally, in the field of video generation, the FVD and FID metrics often exhibit similar trends. I’m curious why SG-I2V performs worse than DragNUWA in FVD (by approximately 45 points) while showing better performance in FID (by about 2 points). Additionally, there seems to be a notable gap between SG-I2V and the best supervised method, with differences of around 45 in FVD and 3.6 in ObjMC. This disparity is quite significant, yet the paper describes the results as \"competitive,\" which appears to be an overstatement. \n\nW3: There is a discrepancy between Table 1 and Figure 4. According to the results in Table 1, SG-I2V performs worse than DragNUWA, while Figure 4 suggests that SG-I2V performs better, leading to contradictory information between the two experiments. Additionally, since DragAnything focuses on entity control, it may be more objective to include a comparison of entity control performance. Furthermore, the primary focus of the comparisons in this paper seems to be on zero-shot methods. Therefore, it would be beneficial for Figure 4 to include a comparison of visual quality with those zero-shot approaches as well. \n\nW4: In the zero-shot baselines, the methods FreeTraj and DragDiffusion are not complete versions of the original models, therefore, more implementation details should be provided. Additionally, the original DragDiffusion model supports drag-based image editing, making it necessary to directly present the model’s performance results in video generation (by dividing the trajectory into 15 segments and running DragDiffusion on each segment individually). \nFurthermore, since DragDiffusion requires an edit region as input, how should this input be provided appropriately to ensure a fair comparison? \n\nW5: Some modules may not appear particularly novel from a technical perspective. For instance, the method of optimizing the input latent code to control the layout of video generation is a technique that has been widely utilized in various zero-shot controllable generation papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PFoytsiqug", "reviewer": "Reviewer_AD9e", "summary": "This paper proposes a training-free, self-guided trajectory control framework for image-to-video generation, named SG-I2V. The authors also design operations for semantically aligned feature map extraction, trajectory enforcement, and high-frequency preservation in post-processing, which together constitute the SG-I2V framework. Additionally, comprehensive experiments are conducted to demonstrate the effectiveness of SG-I2V and the impact of specific parameters in SG-I2V.", "questions": "1.How long does it take for SG-I2V to infer a single example?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper exhibits several strengths:\n1.The motivation behind the research is clear and well-founded.\n2.The use of latent optimization for trajectory control appears both intuitive and innovative.\n3.The experimental results validate the effectiveness of the proposed method on the SVD model. The findings are convincing, and the ablation study is thorough.", "weaknesses": "This paper exhibits several Weaknesses:\n1.There is a lack of generalization verification across various types of video diffusion models, such as VideoCrafter, EmuVideo (based on the U-Net structure), and CogVideoX-I2V (based on Diffusion Transformer).\n2.The issue of inconsistent features across frames, as discussed in the introduction, may not be universally applicable. This problem might not be evident in some of the latest 3D full attention-based video diffusion models (e.g., OpenSoraPlan 1.2, EasyAnimate V4, and CogVideoX) or when using VideoVAE.\n3.There are errors in the formula expressions:\n1)The symbols in Formula 1 are confusing. Does Fn represent an operation or a feature map? How does Bb,n (denoting [h, w, x, y]) perform matrix operations with feature maps (shape of [h, w, d])? What does “[Bb,n]” signify?\n2)There are two instances of zt* in formula 2.\n4.In line 194, the author emphasizes \"Yet, all of these methods focus on text-based generation.\" However, MOFT[1] is also training-free and can achieve image-to-video motion control. In addition, MOFT incorporates the concept of optimizing denoising latents. Can the authors elaborate on the similarities and differences compared to MOFT in terms of functionality and methodology?\n\n[1] Video Diffusion Models are Training-free Motion Interpreter and Controller.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKqEmjUYup", "reviewer": "Reviewer_g4dj", "summary": "The paper presents a zero-shot image-to-video diffusion framework for controllable motion generation. Specifically, the proposed SG-I2V exploits the self-guidance in the cross-frame attention feature maps to regulate object trajectory. To further improve the video quality, a post-processing with high-frequency noise injection is designed to ensure that the latent distribution is not disrupted in online latent optimization. Experiments conducted on the VIPSeg dataset verify the proposed approach on both object motion control and camera control.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed approach employs the feature map correspondences to regulate video latent updating and achieves object trajectory control in a zero-shot manner.\n2.\tBoth of the quantitative and qualitative experimental results demonstrate the effectiveness of the proposed approach.  \n3.\tThe paper is well-written and the technical figures are very clear.\n4.\tThe investigation and ablation studies (including the visualization of feature maps) on the feature map selection is comprehensive for proposal validation.", "weaknesses": "1.\tThe major concern for this paper is about the technical contribution. Even though the weakness of the unsatisfactory feature alignment in the up-sampling blocks (the generalization problem in video domain) has been revealed, the latent optimization is still borrowed from the drag-based control methods, e.g., DragDiffusion. The feature map selection and validation are valuable in my personal opinion. Nevertheless, there is no concept new in the whole architecture design. Therefore, it can be treated as another DragDiffusion of the video diffusion version. Moreover, the high-frequency noise initialization is also proposed in FreeTraj, which should not the key technical contribution in this paper.\n2.\tThe motivation of involving Gaussian heatmap $\\textbf{G}_{b}$ in the Eq. (1) is not clear.\n3.\tIt is interesting that the performance trend of ObjMC is flickering as shown in the Figure 10. The reason behind this could be discussed and analyzed.\n4.\tIs there any additional computational (time and memory) cost for the latent optimization? Including some analysis of them will make the experimental section stronger. \n5.\tAre there any visualized comparisons on the VIPSeg dataset? All the presented cases seem be derived from the web images with high quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMwbs7Rb2C", "reviewer": "Reviewer_F4s1", "summary": "In this paper, the authors first analyze the semantic correspondences among frames in different latent. Then based on the observation, they propose SG-I2V, a training-free framework for trajectory control through aligning the self-attention with given bounding boxes based on pre-trained I2V model SVD. Additionally, SG-I2V supports arbitrary input images and any number of objects or regions of a scene.", "questions": "Has the author tried this method on other I2V frameworks? Specifically, what is the generalizability of this method? Can it be adapted to other UNet models, such as DynamiCrafter? What about the DiT architecture?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors use PCA to visualize feature maps of SVD and demonstrate inspiring motivation of method design.\n2. This method is training-free and thus does not require additional data to tune the modules. Through optimizing the latent, SG-I2V succeeds in generating trajectory-controllable videos with high fidelity.\n3. Inspired by previous work, they apply a high-frequency resampling post-processing to preserve the visual quality. The ablation study demonstrates the effectiveness of this module.\n4. Exhaustive ablation experiments demonstrate the rationality of the designed modules and parameters.", "weaknesses": "1. MotionCtrl is a well-known method for trajectory control and also provides the official code and pre-trained model for SVD (same backbone). SG-I2V is supposed to compare with MotionCtrl.\n2. Since each inference requires individual optimization calculations, authors are supposed to discuss it. If the additional time is too much, is there any way to reduce it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pACxVAefFi", "reviewer": "Reviewer_JD8W", "summary": "The paper introduces a novel framework for controllable image-to-video generation. The core innovation of SG-I2V lies in its ability to achieve zero-shot control over object motion and camera movement within generated videos, leveraging a pre-trained video diffusion model. The method does not require task-specific fine-tuning or additional external knowledge. The authors propose an approach that analyzes feature maps from the upsampling blocks of video diffusion models to identify weak semantic alignments across frames. To address this, they introduce a modified self-attention mechanism that aligns features across frames, enabling precise control over element positions in the generated video. The framework optimizes the latent space to enforce similarity between aligned features within bounding box trajectories, and a post-processing step is applied to maintain the expected distribution of high-frequency noise expected by the diffusion model. The paper demonstrates that SG-I2V outperforms unsupervised baselines and is competitive with supervised models in terms of visual quality and motion fidelity.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors have been inspired by tuning-free text-to-video generation methods and have proposed a similarly tuning-free approach for image-to-video tasks. This eliminates the need for computationally expensive fine-tuning procedures, making the framework more accessible and efficient.\n2. The paper provides a thorough analysis of semantic feature alignment within a pre-trained image-to-video diffusion model. \n3. The authors have provided a project page that offers a more intuitive presentation of their work.", "weaknesses": "1. While the paper claims the ability to control an arbitrary number of objects, it lacks experimental evidence to support this claim. The examples provided for controlling object motion are limited to simple scenarios with only 1-2 objects, which may not demonstrate the method's capability across more complex scenarios with multiple objects.\n2. The method's robustness is questionable due to the fine-tuning of parameters specifically for the VIPSeg dataset. The learning rate, for instance, is set to a specific value (0.21), raising concerns about whether these parameters would be universally effective across various image-to-video examples without further adjustments.\n3. It is unclear whether the proposed method could be adapted to other base video generation models beyond the one used in the paper. The transferability of the method to different models is an important consideration for its broader adoption.\n4. The current method appears to support only linear trajectories, which may not be sufficient for more complex motion control. There is a need to explore whether the method can handle other shapes and longer, more complex trajectories.\n5. Lacking of reference or analyze the relationship with related work on trajectory-based image or video generation, including TraDiffusion[1] and DragEntity[2].\n\n[1] TraDiffusion: Trajectory-Based Training-Free Image Generation.\n\n[2] DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)