SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation

## Abstract

Abstract Methods for image-to-video generation have achieved impressive, photo-realistic quality.
However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., involving re-generating videos with different random seeds.
Recent techniques address this issue by fine-tuning a pre-trained model to follow conditioning signals, such as bounding boxes or point trajectories.
Yet, this fine-tuning procedure can be computationally expensive, and it requires datasets with annotated object motion, which can be difficult to procure.
In this work, we introduce SG-I2V, a framework for controllable image-to-video generation that is self-guided—offering zero-shot control by relying solely on the knowledge present in a pre-trained image-to-video diffusion model without the need for fine-tuning or external knowledge.
Our zero-shot method outperforms unsupervised baselines while being competitive with supervised models in terms of visual quality and motion fidelity.
Additional details and video results are available on our project page: https://kmcode1.github.io/Projects/SG-I2V .

## 1 Introduction

Recent advances in video diffusion models demonstrate significant improvements in visual and motion quality  (Ho et al., ; Blattmann et al., ; ; He et al., ).
These models typically take a text prompt (Ho et al., ; Blattmann et al., ; Ho et al., ) or image (Chen et al., ; ; Guo et al., ; Xing et al., ) as input and generate video frames of a photorealistic, animated scene.
Current methods can generate videos that are largely consistent with an input text description or image; however, fine-grained adjustment of specific video elements (e.g., object motion or camera movement) is conventionally a tedious process that requires re-running the model with different text prompts or random seeds (Wu et al., ; Qiu et al., ).

Approaches for controllable video generation aim to eliminate this process of trial-and-error through direct manipulation of generated video elements, such as object motion (Wu et al., ; Yin et al., ; Wang et al., ), pose (Hu, ; Xu et al., ), and camera movement (Wang et al., ; Li et al., ; He et al., ; Hu et al., ).
One line of work fine-tunes pre-trained video generators to incorporate control signals such as bounding boxes or point trajectories (Wu et al., ; Wang et al., ).
One of the primary challenges with these supervised methods is the expensive training cost, and thus, previous methods usually incorporate trajectory control by fine-tuning at a lower resolution than the original model Wu et al. (); Yin et al. ().
More recently, several methods for zero-shot, controllable text-to-video generation have been developed (Ma et al., ; Qiu et al., ; Jain et al., ).
They control object trajectories by modulating the cross-attention maps between features within a bounding box and an object-related text token.
Still, it is not always possible to associate a desired edit with the input text prompt (consider, e.g., motion of object parts).
Moreover, these methods cannot be directly applied to animate existing images, as they are only conditioned on text.

In this work, we propose SG-I2V, a new method for controllable image-to-video generation.
Our approach is self-guided, in that it offers zero-shot control by relying solely on knowledge present in a pre-trained video diffusion model.
Concretely, given an input image, a user specifies a set of bounding boxes and associated trajectories.
Then, our framework alters the generation process to control the motion of target scene elements.
It is essential to manipulate the structure of the
generated video to achieve precise control over element positions, which is mainly decided by early denoising steps (Balaji et al., ; Wang & Vastola, ).
In image diffusion models, it is known that feature maps extracted from the output of upsampling blocks are semantically aligned, i.e., pixels belonging to the same object share similar feature vectors on the feature map and thus can be used to control the spatial layout of generated images (Tang et al., ; Shi et al., ; Namekata et al., ; Tumanyan et al., ).
However, our analysis reveals that feature maps extracted from the upsampling blocks of video diffusion models are only weakly aligned across frames (see Fig. 2).
This misalignment poses challenges, as directly manipulating these feature maps fails to give useful guidance signals for layout control. Instead, we find that feature maps extracted from the self-attention layers can be semantically aligned by replacing the key and value tokens for each frame with those of the first frame (see bottom row of Fig. 2).
After that, we can control the motion of generated videos by optimizing the latent (the input to the denoising network) with a loss that encourages similarity between the aligned features within each bounding box along the input trajectory.
Finally, we apply a post-processing step to enhance output quality by ensuring that our optimization does not disrupt the distribution of high-frequency noise expected by the diffusion model.

In summary, our work makes the following contributions:

- •
We conduct a first-of-its-kind analysis of semantic feature alignment in a pre-trained image-to-video diffusion model and identify important differences from image diffusion models.
- •
Building on this analysis, we propose SG-I2V, a zero-shot, self-guided approach for controllable image-to-video generation.
Our method can control object motion and camera dynamics for arbitrary input images and any number of objects or regions of a scene.
- •
We conduct extensive experiments to show superior performance over zero-shot baselines while being competitive with supervised baselines in visual and motion quality.

Figure: Figure 2: Semantic correspondences in video diffusion models. We analyze feature maps in the image-to-video diffusion model SVD (Blattmann et al., ) for three generated video sequences (row 1). We use PCA to visualize the features at diffusion timestep 30 (out of 50) at the output of an upsampling block (row 2), a self-attention layer (row 3), and the same self-attention layer after our alignment procedure (row 4). Although output feature maps of upsampling blocks in image diffusion models are known to encode semantic information (Tang et al., ), we only observe weak semantic correspondences across frames in SVD. Thus, we focus on the self-attention layer and modify it to produce feature maps that are semantically aligned across frames.
Refer to caption: extracted/5985497/imgs/semantic_correspondence_labeled.jpg

## 2 Related Work

#### Diffusion-based image-to-video generation.

With the recent advances in diffusion models (Sohl-Dickstein et al., ; Ho et al., ), image animation has achieved tremendous progress (Wang et al., ; Chen et al., ; ; Guo et al., ).
Early methods inflate pre-trained text-to-image models (Rombach et al., ) to add motions to an image (Wu et al., ; Khachatryan et al., ; Singer et al., ).
A notable example is AnimateDiff (Guo et al., ), which learns low-rank adapters (Hu et al., ) for different motions.
Later works seek to inject a conditioning frame into a pre-trained text-to-video model (Ho et al., ; ; He et al., ).
VideoCrafter1 (Chen et al., ) leverages a dual cross-attention layer to condition on features of both the image and the text prompt.
DynamicCrafter (Xing et al., ) further improves it by concatenating the input image with noisy latent.
Stable Video Diffusion (SVD) (Blattmann et al., ) instead works in an image-only manner, which replaces the CLIP (Radford et al., ) text embedding of the text prompt with the CLIP image embedding of the conditioning frame.
However, none of these models support direct trajectory control of scene elements, and rather require multiple attempts to obtain a desired result.
In this work, we aim to enable intuitive motion control in animating a pre-existing image.
Since SVD only takes in an image without any text prompt, we utilize it as the base model following prior work (Yin et al., ; Wu et al., ).

#### Spatial control in image diffusion models.

One common way to incorporate spatial control into the image generation process is to fine-tune pre-trained models to incorporate conditioning on depth maps or bounding boxes (Zhang et al., ; Ye et al., ; Avrahami et al., ; Li et al., ; Goel et al., ; Wang et al., ).
While these methods demonstrate high fidelity, they require excessive computing resources and labor-intensive data annotations.
Therefore, several tuning-free approaches have been proposed (Cao et al., ; Chen et al., ; Feng et al., ; Hertz et al., ).
Self-Guidance (Epstein et al., ) and Attend-and-Excite (Chefer et al., ) control the image layout by manipulating intermediate attention maps. They first estimate the generated objects’ positions using the attention maps produced from text–image cross-attention layers and then optimize the latent to increase the attention values at specific positions.
However, these approaches require associating the control target with a specific token in the text prompt and thus cannot be directly applied to image-only editing.
Closest to ours are methods that alter image layouts without text input (Pan et al., ; Mou et al., ; ; Ling et al., ; Liu et al., ; Zhang et al., ; Cui et al., ; Hou et al., ; Zhao et al., ; Shi et al., ).
They enforce similarity between semantically correlated feature maps extracted from the upsampling blocks of a denoising U-Net (Tang et al., ; Namekata et al., ; Hedlin et al., ; Zhang et al., ; Luo et al., ).
In contrast, we show that the cross-frame semantic correspondence of upsampling feature maps in image-to-video diffusion models (Blattmann et al., ) is weak. Thus, optimization directly based on these feature maps leads to sub-optimal results.

#### Motion control in video diffusion models.

Several recent works have studied camera pose control in video diffusion models (He et al., ; Xu et al., ; Kuang et al., ; Hu et al., ; Xiao et al., ; Hou et al., ; Bahmani et al., ; Li et al., ).
The representative work MotionCtrl (Wang et al., ) fine-tunes pre-trained video generators to follow camera trajectory input.
However, video datasets with accurate camera pose annotations are limited (Zhou et al., ), and fine-tuning video models requires high computation costs.
Another line of work focuses on object trajectory control (Wang et al., ; Yin et al., ; Wu et al., ; Zhou et al., ; Wang et al., ; Zhang et al., ), among which our work is particularly related to the tuning-free variants (Qiu et al., ; Ma et al., ; Yang et al., ; Jain et al., ; Yu et al., ).
Yet, all of these methods focus on text-based generation. In contrast, our method uses an image-to-video model and thus can turn existing, real-world images into controllable videos.
Moreover, we can control camera motion by specifying trajectories of background regions.

Figure: Figure 3: Overview of the controllable image-to-video generation framework. To control trajectories of scene elements, we optimize the latent $\bm{z}_{t}$ at specific denoising timesteps $t$ of a pre-trained video diffusion model. First, we extract semantically aligned feature maps from the denoising U-Net to estimate the video layout. Next, we enforce cross-frame feature similarity along the bounding box trajectory to drive the motion of each region. To preserve the visual quality of the generated video, a frequency-based post-processing method is applied to retain high-frequency noise of the original latent $\bm{z}_{t}$. The updated latent $\tilde{\bm{z}}_{t}$ is input to the next denoising step.
Refer to caption: x1.png

## 3 Method

In this section, we describe our method for the trajectory control task in image-to-video generation (Sec. 3.1).
Our framework, SG-I2V, builds on the publicly available image-to-video diffusion model Stable Video Diffusion (SVD) (Blattmann et al., ),
and consists of two main steps.
First, we extract and semantically align the feature maps from a specific layer of SVD during the early steps of the diffusion process (Sec. 3.2); we show that such feature maps are especially effective at influencing motion in the output video.
Second, we optimize the noisy latent (i.e., the input to the denoising network) to enforce similarity between features within the bounding box trajectories (Sec. 3.3).
However, we find that naive optimization of latent is prone to overfitting and often results in low-quality generation.
Thus, we employ frequency-based post-processing to retain an in-distribution noisy latent (Sec. 3.4).
Our entire pipeline is summarized in Fig. 3.

### 3.1 Trajectory Control in Image-to-Video Generation

The goal of our work is to build a zero-shot framework that takes an input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ (with height $H$, width $W$) and generates a video with $N$ frames, where elements in the input image move in a user-specified fashion.
Inspired by DragAnything (Wu et al., ), we further assume that a user provides a set of $B$ input bounding box trajectories $\{\mathcal{B}_{b}\}_{b=1}^{B}$, each parameterized by a height $h_{b}$ and width $w_{b}$, as well as center point coordinates for each output video frame: $\mathbf{c}_{b,n}\in\mathbb{R}^{2},1\leq n\leq N$.
For simplicity, we assume bounding boxes cannot extend outside the image, and we denote the $b$-th bounding box in the $n$-th frame as $\mathcal{B}_{b,n}=\{h_{b},w_{b},\mathbf{c}_{b,n}\}$.

Our aim is to constrain regions of the input image falling within a bounding box to follow the trajectory of the same bounding box in the output video.
Thus, the motion of dynamic foreground objects can be controlled by placing bounding boxes around them and specifying the desired trajectory.
On the other hand, camera motion can be specified by placing bounding boxes on static background regions and specifying a trajectory opposite to the desired camera movement. Further, we can also set the bounding box trajectory to the zero vector to keep the region static.
Overall, this formulation provides intuitive and unified control over object and camera motion, which are sometimes treated separately in previous controllable image-to-video frameworks (Wang et al., ; Yang et al., ; Li et al., ).

### 3.2 Extracting Semantic Video Layout

#### Preliminaries: Stable Video Diffusion.

Video diffusion models (Ho et al., ) learn a data distribution $\bm{x}_{0}\sim p_{\theta}(\bm{x}_{0})$ by gradually denoising a video corrupted by Gaussian noise.
The output denoised video is thus drawn from the distribution $p_{\theta}(\bm{x}_{0})=\int p_{\theta}(\bm{x}_{0:T})\ d\bm{x}_{1:T}$, where $\bm{x}_{0}\in\mathbb{R}^{N\times H\times W}$ is a clean video, and $\bm{x}_{1:T}$ are intermediate noisy samples.
For simplicity, we omit the channel dimension throughout the paper.
To reduce computation, Stable Video Diffusion (SVD) (Blattmann et al., ) performs the diffusion process in a latent space, where a variational autoencoder (Kingma & Welling, ) maps a raw video $\bm{x}_{0}$ to a latent $\bm{z}_{0}\in\mathbb{R}^{N\times h\times w}$.
Since this work aims to animate an existing image, we utilize the image-to-video variant of SVD, which concatenates a conditioning frame with noisy latent ($\bm{z}_{t}$) and runs a 3D U-Net (Ronneberger et al., ) to predict the noise.
The 3D U-Net contains a downsampling and an upsampling path.
Specifically, the upsampling path consists of three stages operating at different resolutions, where each stage contains three blocks with interleaved residual blocks (He et al., ), spatial, and temporal attention layers (Vaswani et al., ).
We will call these three stages bottom, middle, and top from lower to higher resolution.
For more details, we refer readers to the original paper of SVD (Blattmann et al., ).

#### SVD feature map analysis.

In image diffusion models, prior work has shown that output feature maps of upsampling blocks in the middle stage of the denoising U-Net are semantically aligned (Tang et al., ; Namekata et al., ; Hedlin et al., ; Zhang et al., ; Luo et al., ), i.e., regions belonging to the same object tend to have similar feature vectors. Such semantically aligned feature maps are useful in estimating the layout of generated images, enabling spatial control of objects (Shi et al., ; Mou et al., ).
Therefore, we first examine whether SVD feature maps are also semantically correlated across both spatial and temporal dimension. Fig. 2 visualizes the principal components of feature maps extracted from the upsampling block and spatial attention layers.
We observe that SVD feature maps exhibit weak semantic correspondence across frames at early denoising steps, leading to inaccurate object trajectory estimation.
Yet, we want to operate at early steps as they decide the structure of generated videos (Materzynska et al., ).
This dilemma prompts us to align these features before applying optimization.

Feature alignment with modified self-attention.
SVD leverages separate spatial and temporal self-attention to model the entire video.
Since spatial self-attention is only applied per frame, it does not produce cross-frame aligned features.
While temporal attention communicates across frames, it only attends to the same pixel position on the feature map, which may be inadequate for capturing semantic information spatially. To address this issue, inspired by (Wu et al., ), we modify the spatial self-attention on each frame to directly attend to the first frame.
Concretely, for the $n$-th frame, the original spatial self-attention works as $\bm{F}_{n}=\mathrm{Softmax}(\frac{\bm{Q}_{n}\cdot\bm{K}_{n}^{T}}{\sqrt{D}})
\cdot\bm{V}_{n}$, where $\bm{F}_{n}$ is the outputs of self-attention, $\bm{Q},\bm{K},\bm{V}$ are the query, key, and value tokens, respectively, and $D$ is the dimensionality of the key and query tokens Vaswani et al. ().
Instead, we replace the key $\bm{K}_{n}$ and value $\bm{V}_{n}$ of each frame with $\bm{K}_{1}$ and $\bm{V}_{1}$ from the first frame, leading to a new operation $\tilde{\bm{F}}_{n}=\mathrm{SoftMax}(\frac{\bm{Q}_{n}\cdot\mathrm{SG}(\bm{K}_{1
})^{T}}{\sqrt{D}})\cdot\mathrm{SG}(\bm{V}_{1})$.
Notably, we apply a stop gradient $\mathrm{SG}(\cdot)$ on $\bm{K}_{1}$ and $\bm{V}_{1}$ to stabilize the subsequent optimization process.
Now, all the modified feature maps $\tilde{\bm{F}}_{n}$ are weighted combinations of $\bm{V}_{1}$, exhibiting a stronger cross-frame correspondence while still maintaining the object layout of each frame, as shown in the bottom row of Fig. 2.

### 3.3 Trajectory Control with Latent Optimization

So far, we have obtained spatio-temporally aligned feature maps $\tilde{\bm{F}}_{n}\in\mathbb{R}^{h\times w\times d}$ at each frame.
For simplicity, we resize $\tilde{\bm{F}}_{n}$ extracted from different U-Net stages to the same resolution as the noisy latent $\bm{z}_{t}$.
Recall that our goal is to control the output video frames so that the bounding boxes $\mathcal{B}_{b}$ identified in the first frame move along the associated trajectories.
Inspired by prior drag-based control methods (Shi et al., ; Pan et al., ), we optimize the noisy latent $\bm{z}_{t}$ to enforce cross-frame similarity between features within bounding boxes.
The optimization objective is as follows:

$$ $\bm{z}_{t}^{*}=\operatorname*{arg\,min}_{\bm{z}_{t}}\sum_{b\in[1,B],n\in[2,N]} \lVert\bm{G}_{b}\odot\left(\tilde{\bm{F}}_{n}(\bm{z}_{t})[\mathcal{B}_{b,n}]- \mathrm{SG}(\tilde{\bm{F}}_{1}(\bm{z}_{t})[\mathcal{B}_{b,1}])\right)\rVert_{2},$ (1) $$

where $\odot$ indicates element-wise multiplication, and we optimize the latent code $\bm{z}_{t}$ at the denoising timestep $t$.
Following DragAnything (Wu et al., ), we weigh the feature difference using a Gaussian heatmap $\bm{G}_{b}\in\mathbb{R}^{h\times w}$, which applies a stronger weight to the bounding box center.

#### Selective latent optimization.

We optimize Eq. 1 on a subset of denoising timesteps and self-attention layers.
Concretely, we only select early denoising timesteps as the coarse structure of output frames is determined at these timesteps (Wang & Vastola, ; Materzynska et al., ).
In addition, consistent with previous works in image diffusion models (Shi et al., ; Mou et al., ), we observe that feature maps extracted from the middle stage of the denoising U-Net are more semantically correlated, and thus we use them for optimization.

### 3.4 High-Frequency Preserved Post-Processing

Although the presented pipeline already enables trajectory control in video generation, we notice a quality degradation in generated videos.
We attribute this degradation to the deviation of $\bm{z}_{t}^{*}$ from the sampling distribution of the diffusion process after optimization.
A recent work FreeInit (Wu et al., ) observed that motions of generated videos are mostly encoded in the low-frequency component of noisy latent.
Inspired by this, we propose to discard the high-frequency component of the optimized latent $\bm{z}_{t}^{*}$ and replace it with the high-frequency component of the original latent $\bm{z}_{t}$.
Formally, we obtain the new latent $\tilde{\bm{z}}_{t}$ as follows:

$$ $\displaystyle\tilde{\bm{z}}_{t}=\textsc{IFFT}_{\mathrm{2D}}\left(\textsc{FFT}_ {\mathrm{2D}}(\bm{z}_{t}^{*})\odot\mathbf{H}_{\gamma}+\textsc{FFT}_{\mathrm{2D }}(\bm{z}_{t}^{*})\odot(1-\mathbf{H}_{\gamma})\right),$ (2) $$

where $\textsc{FFT}_{\mathrm{2D}}$ is the Fast Fourier transformation applied to each frame, and $\textsc{IFFT}_{\mathrm{2D}}$ is the corresponding inverse operation.
We set $\mathbf{H}_{\gamma}$ as the frequency response of a 2D low-pass filter (we follow FreeTraj and use a Butterworth filter) with cut-off frequency $\gamma$.
This post-processing step retains the target motion signals encoded in the low-frequency component of $\bm{z}^{*}_{t}$ while eliminating undesirable high-frequency disruptions.

## 4 Experiments

### 4.1 Experimental Setup

#### Implementation details.

In all experiments, we leverage the image-to-video variant of Stable Video Diffusion (Blattmann et al., ) to generate videos with 14 frames and $576\times 1024$ resolution.
The default discrete Euler scheduler (Karras et al., ) is applied with $T=50$ sampling steps.
We extract feature maps from the last two self-attention layers from the middle stage in the denoising U-Net. We optimize Eq. 1 at the early denoising timesteps $t\in[45,44,...,30]$ for $5$ iterations per timestep.
We use the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $0.21$.
All these design choices are carefully ablated and analyzed in Sec. 4.4.
During loss calculation, Gaussian heatmap $\bm{G}_{b}$ is constructed following (Wu et al., ), where a heatmap for a bounding box of size $(h_{b},w_{b})$ is created by Gaussian distribution with standard deviation $\sigma=(0.2h_{b},0.2w_{b})$.
For the low-pass filter $\mathbf{H}_{\gamma}$, we set the cut-off frequency $\gamma$ to $0.5$.

#### Baselines.

We compare with methods that enable motion control on existing images and have publicly available code.
Specifically, we compare with supervised baselines DragNUWA (Yin et al., ) and DragAnything (Wu et al., ), which both add motion adapters to SVD.
We also compare with Image Conductor (Li et al., ), which is based on AnimateDiff (Guo et al., ).
Since all supervised baselines are fine-tuned to generate videos at a lower resolution, we resize the generated videos from all the methods to $320\times 576$ for a fair comparison. Since no previous methods exist for zero-shot trajectory-controlled image-to-video generation, we adapt techniques from text-to-video methods to create new baselines.
Specifically, FreeTraj^† incorporates the noise initialization technique from (Qiu et al., ) by copy-pasting the initial noise on the first frame to other frames along the trajectories.
We cannot apply the attention manipulation technique in their work as it is primarily designed for single-object control. DragDiffusion^† is inspired by the image editing method in (Shi et al., ), which utilizes feature maps extracted from the output of upsampling blocks to guide the generation process without feature alignment.

#### Datasets and evaluation metrics.

Following prior works (Wu et al., ; Zhou et al., ), we evaluate our method on the validation set of the VIPSeg dataset (Miao et al., ).
We test on the same control regions and target trajectories as DragAnything, where the size of our bounding boxes is the same as the diameter of the circles in their work. For quantitative metrics, we report Frechet Inception Distance (FID) (Heusel et al., ) and Frechet Video Distance (FVD) (Unterthiner et al., ) to measure the visual quality, and ObjMC (Wu et al., ) to measure the motion fidelity. ObjMC computes the average distance between generated and target trajectories, where Co-Tracker (Karaev et al., ) is used to estimate the trajectory of generated videos.

### 4.2 Qualitative Results

Fig. 1 presents the versatile control ability of our method.
We can control foreground objects to perform rigid motions, such as trains moving, and non-rigid motions, such as the movement of human hairs.
In addition, we can control non-physical entities such as smoke and fire.
The moved scene elements naturally adapt to the new location while preserving their original identity.
Thanks to our general formulation of trajectories, camera motion control is also supported.
We highly encourage the readers to view [our project page](https://kmcode1.github.io/Projects/SG-I2V#obj-motion-results) for additional results.

Figure: Figure 4: Qualitative comparison with supervised baselines. We observe that DragNUWA tends to distort objects rather than move them, and DragAnything is weak at part-level control as it is designed for entity-level control. In contrast, our method can generate videos with natural motion for diverse object and camera trajectories. Please see [our project page](https://kmcode1.github.io/Projects/SG-I2V#baseline-comparison) for more comparisons.
Refer to caption: extracted/5985497/imgs/figure_baseline_labeled.jpg

**Table 1: Quantitative comparison on the VIPSeg dataset. Despite being a zero-shot method, we show competitive motion fidelity (ObjMC) to supervised baselines without degrading video quality (FID, FVD). Furthermore, our approach outperforms other zero-shot baselines across all metrics.**
| Method | FID ($\downarrow$) | FVD ($\downarrow$) | ObjMC ($\downarrow$) | Zero-shot | Resolution | Backbone |
| --- | --- | --- | --- | --- | --- | --- |
| Image Conductor | 48.81 | 463.21 | 21.07 |  | $256\times 384$ | AnimateDiff v3 |
| DragNUWA | 30.73 | 253.57 | 10.84 |  | $320\times 576$ | SVD |
| DragAnything | 30.81 | 268.47 | 11.64 |  | $320\times 576$ | SVD |
| SVD (No Control) | 30.50 | 340.52 | 39.59 | ✓ | $576\times 1024$ | SVD |
| FreeTraj^† | 46.61 | 394.14 | 36.43 | ✓ | $576\times 1024$ | SVD |
| DragDiffusion^† | 30.93 | 458.29 | 31.49 | ✓ | $576\times 1024$ | SVD |
| SG-I2V | 28.87 | 298.10 | 14.43 | ✓ | $576\times 1024$ | SVD |

### 4.3 Quantitative Assessment

#### Comparison with supervised baselines.

We provide quantitative comparisons to baselines in Tab. 1.
Despite being trained on large-scale datasets, Image Conductor underperforms our method by a large margin, mainly because of the limited capacity of the base model AnimateDiff.
Compared to methods that also build upon SVD, we achieve competitive performance in visual quality, with slightly worse motion fidelity.
Yet, these methods are trained to generate low-resolution (i.e., $320\times 576$) videos due to the high cost of fine-tuning at high resolution. At the same time, our tuning-free approach can maintain the original resolution of SVD (i.e., $576\times 1024$).
We show qualitative comparisons in Fig. 4.
Similar to observations in (Wu et al., ), DragNUWA tends to distort objects rather than naturally move them, while our method can generate more natural movements.
DragAnything is weak at part-level motion control (e.g., closing a cat’s mouth) as it is only trained on datasets annotated with entity-level control, such as object segmentation masks.
In contrast, our approach can handle different granularities of control regions.

#### Comparison with adapted zero-shot baselines.

The noise initialization technique in FreeTraj^† improves motion control slightly compared to original videos but significantly degrades visual quality. This indicates that motion prior can not be easily incorporated into initial noises of SVD by a hand-crafted algorithm.
DragDiffusion^† leverages feature maps extracted from the upsampling block, which, unlike SG-I2V, are not semantically aligned across frames.
As a result, the motion fidelity is much lower than ours.
Overall, SG-I2V significantly outperforms zero-shot baselines across all metrics.
This highlights that zero-shot techniques applied in text-to-video diffusion models do not necessarily transfer to image-to-video models.

### 4.4 Ablation Studies

Figure: Figure 5: Performance across U-Net feature maps used to compute loss in Eq. 1. For all metrics, lower values are better. Temporal and spatial refer to the temporal and spatial self-attention layers. We find that features extracted from self-attention layers generally perform better than those from upsampling blocks and temporal attention layers. In addition, using the feature maps of our modified self-attention layer achieves the best results, since they are semantically aligned across frames. Corresponding qualitative visuals are presented in Fig. 13 and [our project page.](https://kmcode1.github.io/Projects/SG-I2V#ablation-unet-featuremap)
Refer to caption: x2.png

#### Feature map selection.

In Fig. 5, we analyze the effect of the choice of U-Net feature map in the optimization of Eq. 1.
DIFT (Tang et al., ) pointed out that outputs from the U-Net upsampling block in image diffusion models have strong semantic correspondence across spatial dimensions, which is used in image editing methods (Shi et al., ). However, we find them to have inferior correspondence across the temporal dimension, and optimizing them leads to inaccurate object trajectories, as indicated by the high ObjMC value.
Next, we examine features in temporal self-attention layers.
However, using these features also leads to inferior motion guidance, as indicated by the high ObjMC errors.
This may be because the temporal layers in SVD always attend to the same spatial location, thus focusing less on each frame’s spatial layout.
Finally, we study each component in spatial self-attention layers.
As discussed in Sec. 3.2, the lack of cross-frame correspondence in the original self-attention is problematic—in Fig. 5, we see that this results in worse ObjMC and FVD scores. Overall, optimizing outputs from our modified self-attention operation achieves the best motion fidelity, showing the importance of feature alignment.

Figure: Figure 6: Performance across U-Net layers used to extract feature maps. Lower is better for all metrics. Bottom, mid, and top indicate the three resolution levels in the U-Net’s upsampling path, each containing three self-attention layers numbered 1, 2, and 3. for example “M2-3” means applying the loss to features from both mid-resolution layers 2 and 3. We observe that mid-resolution feature maps perform best for trajectory guidance. In addition, using features from both M2 and M3 leads to the best result. See [our project page](https://kmcode1.github.io/Projects/SG-I2V#ablation-unet-layer) for visualizations.
Refer to caption: x3.png

#### Cut-off frequency in post-processing.

We study the cut-off frequency in our high-frequency-preserving post-processing step in Fig. 8.
Naively keeping the optimized latent ($\gamma=1$) degrades the visual quality drastically, as shown by the higher FID and FVD.
Yet, discarding part of the high-frequency component has negligible impact on motion control while eliminating most artifacts.
We thus choose $\gamma=0.5$ as a sweet spot for effective video quality restoration.

Figure: Figure 7: Effect of high-frequency preservation in post-processing. Videos without post-processing tend to demonstrate oversmoothing and have artifacts. In contrast, our post-processing technique retains videos with sharp details and eliminates most of the artifacts. See [our project page](https://kmcode1.github.io/Projects/SG-I2V#ablation-post-processing) for more examples.
Refer to caption: extracted/5985497/imgs/figure_fft_labeled.jpg

Figure: Figure 8: Study of the cut-off frequency in post-processing. Lower is better for all metrics. The value $\gamma$ indicates the cut-off frequency. Fully keeping the optimized latent ($\gamma=1$) results in degraded video quality, as shown by high FID and FVD values. On the other hand, replacing too many frequency components diminishes motion control, as indicated by the increasing ObjMC.
Refer to caption: x4.png

#### U-Net layer.

Fig. 6 ablates from which layer to extract the feature maps.
The upsampling path of SVD’s U-Net contains three stages (bottom, mid, top), each with three spatial self-attention layers (indexed 1, 2, 3).
We observe that blocks at the middle-resolution level capture most semantically meaningful information, which aligns with prior work in image diffusion models (Shi et al., ; Tang et al., ).
We also try joint optimization of several layers, which gives the best performance.

Figure: Figure 9: Ablation on optimization learning rates. Larger learning rates lead to video quality degradation (i.e., higher FID and FVD), while smaller learning rates result in lower motion fidelity (i.e., higher ObjMC). We choose the learning rate considering this tradeoff.
Refer to caption: x5.png

#### Learning rate.

Fig. 9 examines the effect of learning rate under a fixed number of optimization steps.
We observe a clear trade-off between visual quality and motion fidelity of generated videos.
This is because a large learning rate quickly leads to noisy latents that are out of distribution.

Figure: Figure 10: Effect of optimizing latent at individual denoising timesteps. For all metrics, lower values are better. Here, we optimize Eq. 1 on a single denoising timestep ($t=50$ corresponds to standard Gaussian noise), and we find middle timesteps (e.g. $t=30$) achieve the best motion fidelity while maintaining visual quality. More results on optimizing the latent at multiple timesteps can be found in Fig. 16. See Fig. 15 and [our project page](https://kmcode1.github.io/Projects/SG-I2V#ablation-denoising-timestep) for qualitative comparisons.
Refer to caption: x6.png

#### Denoising timesteps.

Fig. 10 summarizes the effects of optimizing features from different denoising timesteps ($t=50$ corresponds to standard Gaussian noise).
We first study the effect of individual timesteps.
Timesteps lower than $t=30$ give significantly worse visual quality, supporting our hypothesis that only early denoising steps determine the layout of generated videos.
Timesteps greater than $t=45$ also lead to bad motion fidelity, likely because detailed semantic information has yet to be generated at these extremely high noise levels.
Based on these observations, we optimize feature maps extracted between timesteps 30 and 45, balancing visual quality and motion fidelity.

## 5 Conclusion

Our work introduces the first framework for zero-shot trajectory control in image-to-video generation.
Our thorough analysis of diffusion features reveals that the knowledge acquired in the pre-trained image-to-video diffusion models can guide them to generate videos with desired motions. Quantitative and qualitative results demonstrate the effectiveness of our approach, both on synthetic and real-world images.
We hope our findings can shed light on the inner mechanism of image-to-video diffusion models and inspire better architecture designs in the future.

#### Limitation and future works.

Since our pipeline works in a zero-shot and self-guided manner, the base video diffusion model bounds the quality of generated videos, e.g., for subjects with large motion or complex physical interactions (Blattmann et al., ).
Another area for improvement is the out-of-distribution issue when optimizing the latent code. Although we have incorporated frequency-based post-processing to mitigate it, we sometimes still observe artifacts when we set the learning rate higher.
How to alter the denoising process while keeping in-distribution latents is an open problem itself (He et al., ; Garibi et al., ).
With the recent rapid progress in video generators (Brooks et al., ; Gupta et al., ), we also expect our framework could be extended to newly released models to achieve improved generation quality.

## Ethics Statement

Currently, our synthesized videos are not yet indistinguishable from real footage. Still, we anticipate that future iterations of large video models will achieve even greater quality, potentially producing videos that look photoreal. This prospect brings significant ethical and safety considerations, as cutting-edge generative models can be misused to ill effect. We advocate against misuse of generative models to malign or misinform, and we encourage safe and responsible use of such models Google ().

## Appendix A Additional Qualitative Results

We strongly encourage readers to refer to our project website: [https://kmcode1.github.io/Projects/SG-I2V](https://kmcode1.github.io/Projects/SG-I2V) for more visualizations in video format, where we have released qualitative results with various trajectories, comparisons with baselines, and qualitative analysis for ablation study.

## Appendix B Additional Results for Ablation Study

In this subsection, we provide full results for our ablation analysis.

PCA visualization. Fig. 11 and Fig. 12 present the visual results of our PCA analysis on feature maps extracted from various layers of SVD across different timesteps (corresponding to Sec. 3.2).
We can confirm that our modified self-attention consistently produces semantically aligned feature maps throughout the denoising process.

Feature map selection. Fig. 13 presents the visual results of ablating feature maps for optimization.
Performing optimization with feature maps naively extracted from original self/temporal attention or upsampling blocks fails to follow the input trajectory due to the weak semantic alignment across frames.
In contrast, performing optimization with our modified self-attention features successfully produces videos consistent with the input trajectory.
This highlights the importance of extracting semantically aligned feature maps for our optimization.

U-Net layer.
Consistent with the quantitative results
(Fig. 6), Fig. 14 has qualitatively confirmed that optimizing with feature maps extracted from the middle layers of the upward path produces plausible videos consistent with the input trajectory.

Denoising timesteps.
Continuing from the main paper, we further analyze the effect of extracting feature maps from different timesteps.
Fig. 15 qualitatively demonstrates the effect of optimizing latent at individual denoising timesteps.
Consistent with the quantitative results demonstrated in Fig. 10, performing optimization at later stage of denoising steps (i.e. $t=10,20$) severely introduces artifacts in the generated videos.
Next, we examine the effects of performing optimization for multiple timesteps by fixing the total number of iterations for optimization.
As summarized in Fig. 16, we observe similar trends as optimizing individual timesteps, where performing optimization at the later stage of the denoising process significantly degrades visual quality, while performing optimization up to timestep $40$ is not enough for motion control.
Overall, running optimization for multiple timesteps performs better than running optimization only at a single timestep, and hence we adapt continuous timesteps for our experiments.

Figure: Figure 11: Semantic correspondences in video diffusion models across timesteps. Output feature maps of upsampling blocks have limited semantic correspondences across frames. In contrast, our modified self-attention layers produce semantically aligned feature maps across all the timesteps.
Refer to caption: extracted/5985497/imgs/correspondence_full_labeled.jpg

Figure: Figure 12: Semantic correspondences of different features in video diffusion models. We find features from self-attention layers to be more semantically aligned than that of temporal attention layers and upsampling layers, while our modified self-attention layer produces the most aligned results due to its explicit formulation to attend to the first frame.
Refer to caption: extracted/5985497/imgs/correspondence_full_feature_labeled.jpg

Figure: Figure 13: Ablation on U-Net feature maps. Applying loss on feature maps extracted from original self/temporal-attention layers or upsampling blocks fails to follow the trajectory due to the semantic misalignment across frames. In contrast, performing optimization with our modified self-attention layers can produce videos consistent with the input trajectory, indicating the importance of using semantically aligned feature maps. Please see [our project page](https://kmcode1.github.io/Projects/SG-I2V#ablation-unet-featuremap) for more qualitative results.
Refer to caption: extracted/5985497/imgs/figure_fet_full_labeled.jpg

Figure: Figure 14: Ablation on U-Net layer to extract feature maps. Consistent with the quantitative results in Fig. 6, feature maps extracted from the middle resolution level are most useful for trajectory guidance. Optimizing on other feature maps may generate unrealistic videos with low motion fidelity.
Refer to caption: extracted/5985497/imgs/figure_layer_labeled.jpg

Figure: Figure 15: Visual comparison of different denoising timesteps. Here we show the *last* frame of the generated video. Optimizing latent at later denoising process leads to severe artifacts.
Refer to caption: extracted/5985497/imgs/figure_time_labeled.jpg

Figure: Figure 16: Effect of optimizing latent at multiple denoising timesteps. Here we perform optimization on multiple denoising timesteps ($t=50$ corresponds to standard Gaussian noise). Similar to performing individual timestep Fig. 10, performing optimization up to middle timesteps (e.g. $50-30$) achieves the best motion fidelity while maintaining visual quality.
Refer to caption: x7.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Q1: The paper proposes optimizing the input latent code and applying high-frequency preserved post-processing between steps 30 and 45 during the denoising process. The inference time required to generate a 14-frame video should be reported.

Q2: I am curious why the modified spatial self-attention mentioned in Section 3.2 shows better alignment with the initial frame. A more detailed explanation would be helpful. Additionally, after modifying the self-attention in SVD, there is a gap between the modified inference pattern and the original learnt pattern, could this affect the quality of video generation? It would be beneficial to evaluate the change in FVD for I2V generation before and after modifying the self-attention in SVD. 

Q3: I hope the authors can address the concerns raised in the "Weaknesses" section.
soundness: 2
strengths: S1: The paper analyzes the intrinsic relationship between the motion in the generated video and the UNet feature maps. 

S2: Building on this analysis, a self-supervised framework is designed to achieve zero-shot controllable image-to-video (I2V) generation.
summary: The paper introduces SG-I2V, a self-guided method designed to achieve zero-shot controllable image-to-video (I2V) generation. Technically, SG-I2V first extracts feature maps from the self-attention module during the video denoising process. To ensure that the generated video’s motion aligns with the given trajectory, the input latent code is optimized by enforcing cross-frame feature consistency along the bounding box trajectory. Furthermore, to preserve the quality of the generated video, the authors propose a post-processing method that retains the high-frequency components of the original latent code. The evaluation on the VIPSeg dataset demonstrates the effectiveness of the proposed method.
weaknesses: W1: The performance values reported in Table 1 on the VIPSeg dataset differ from those in the DragAnything paper, which reports ObjMC, FVD, and FID values of 305.7, 494.8, and 33.5, respectively. Additionally, Table 1 shows DragNUWA outperforming DragAnything, which contradicts the findings in the DragAnything paper. Could the authors clarify the specific versions of DragAnything and DragNUWA used in their experiments, any differences in their evaluation setup compared to the original papers, and whether they used the same evaluation code and metrics? This information would help pinpoint the source of these discrepancies and ensure a fair comparison.

W2: Generally, in the field of video generation, the FVD and FID metrics often exhibit similar trends. I’m curious why SG-I2V performs worse than DragNUWA in FVD (by approximately 45 points) while showing better performance in FID (by about 2 points). Additionally, there seems to be a notable gap between SG-I2V and the best supervised method, with differences of around 45 in FVD and 3.6 in ObjMC. This disparity is quite significant, yet the paper describes the results as "competitive," which appears to be an overstatement. 

W3: There is a discrepancy between Table 1 and Figure 4. According to the results in Table 1, SG-I2V performs worse than DragNUWA, while Figure 4 suggests that SG-I2V performs better, leading to contradictory information between the two experiments. Additionally, since DragAnything focuses on entity control, it may be more objective to include a comparison of entity control performance. Furthermore, the primary focus of the comparisons in this paper seems to be on zero-shot methods. Therefore, it would be beneficial for Figure 4 to include a comparison of visual quality with those zero-shot approaches as well. 

W4: In the zero-shot baselines, the methods FreeTraj and DragDiffusion are not complete versions of the original models, therefore, more implementation details should be provided. Additionally, the original DragDiffusion model supports drag-based image editing, making it necessary to directly present the model’s performance results in video generation (by dividing the trajectory into 15 segments and running DragDiffusion on each segment individually). 
Furthermore, since DragDiffusion requires an edit region as input, how should this input be provided appropriately to ensure a fair comparison? 

W5: Some modules may not appear particularly novel from a technical perspective. For instance, the method of optimizing the input latent code to control the layout of video generation is a technique that has been widely utilized in various zero-shot controllable generation papers.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.How long does it take for SG-I2V to infer a single example?
soundness: 3
strengths: This paper exhibits several strengths:
1.The motivation behind the research is clear and well-founded.
2.The use of latent optimization for trajectory control appears both intuitive and innovative.
3.The experimental results validate the effectiveness of the proposed method on the SVD model. The findings are convincing, and the ablation study is thorough.
summary: This paper proposes a training-free, self-guided trajectory control framework for image-to-video generation, named SG-I2V. The authors also design operations for semantically aligned feature map extraction, trajectory enforcement, and high-frequency preservation in post-processing, which together constitute the SG-I2V framework. Additionally, comprehensive experiments are conducted to demonstrate the effectiveness of SG-I2V and the impact of specific parameters in SG-I2V.
weaknesses: This paper exhibits several Weaknesses:
1.There is a lack of generalization verification across various types of video diffusion models, such as VideoCrafter, EmuVideo (based on the U-Net structure), and CogVideoX-I2V (based on Diffusion Transformer).
2.The issue of inconsistent features across frames, as discussed in the introduction, may not be universally applicable. This problem might not be evident in some of the latest 3D full attention-based video diffusion models (e.g., OpenSoraPlan 1.2, EasyAnimate V4, and CogVideoX) or when using VideoVAE.
3.There are errors in the formula expressions:
1)The symbols in Formula 1 are confusing. Does Fn represent an operation or a feature map? How does Bb,n (denoting [h, w, x, y]) perform matrix operations with feature maps (shape of [h, w, d])? What does “[Bb,n]” signify?
2)There are two instances of zt* in formula 2.
4.In line 194, the author emphasizes "Yet, all of these methods focus on text-based generation." However, MOFT[1] is also training-free and can achieve image-to-video motion control. In addition, MOFT incorporates the concept of optimizing denoising latents. Can the authors elaborate on the similarities and differences compared to MOFT in terms of functionality and methodology?

[1] Video Diffusion Models are Training-free Motion Interpreter and Controller.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 3
strengths: 1.	The proposed approach employs the feature map correspondences to regulate video latent updating and achieves object trajectory control in a zero-shot manner.
2.	Both of the quantitative and qualitative experimental results demonstrate the effectiveness of the proposed approach.  
3.	The paper is well-written and the technical figures are very clear.
4.	The investigation and ablation studies (including the visualization of feature maps) on the feature map selection is comprehensive for proposal validation.
summary: The paper presents a zero-shot image-to-video diffusion framework for controllable motion generation. Specifically, the proposed SG-I2V exploits the self-guidance in the cross-frame attention feature maps to regulate object trajectory. To further improve the video quality, a post-processing with high-frequency noise injection is designed to ensure that the latent distribution is not disrupted in online latent optimization. Experiments conducted on the VIPSeg dataset verify the proposed approach on both object motion control and camera control.
weaknesses: 1.	The major concern for this paper is about the technical contribution. Even though the weakness of the unsatisfactory feature alignment in the up-sampling blocks (the generalization problem in video domain) has been revealed, the latent optimization is still borrowed from the drag-based control methods, e.g., DragDiffusion. The feature map selection and validation are valuable in my personal opinion. Nevertheless, there is no concept new in the whole architecture design. Therefore, it can be treated as another DragDiffusion of the video diffusion version. Moreover, the high-frequency noise initialization is also proposed in FreeTraj, which should not the key technical contribution in this paper.
2.	The motivation of involving Gaussian heatmap $\textbf{G}_{b}$ in the Eq. (1) is not clear.
3.	It is interesting that the performance trend of ObjMC is flickering as shown in the Figure 10. The reason behind this could be discussed and analyzed.
4.	Is there any additional computational (time and memory) cost for the latent optimization? Including some analysis of them will make the experimental section stronger. 
5.	Are there any visualized comparisons on the VIPSeg dataset? All the presented cases seem be derived from the web images with high quality.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Has the author tried this method on other I2V frameworks? Specifically, what is the generalizability of this method? Can it be adapted to other UNet models, such as DynamiCrafter? What about the DiT architecture?
soundness: 3
strengths: 1. Authors use PCA to visualize feature maps of SVD and demonstrate inspiring motivation of method design.
2. This method is training-free and thus does not require additional data to tune the modules. Through optimizing the latent, SG-I2V succeeds in generating trajectory-controllable videos with high fidelity.
3. Inspired by previous work, they apply a high-frequency resampling post-processing to preserve the visual quality. The ablation study demonstrates the effectiveness of this module.
4. Exhaustive ablation experiments demonstrate the rationality of the designed modules and parameters.
summary: In this paper, the authors first analyze the semantic correspondences among frames in different latent. Then based on the observation, they propose SG-I2V, a training-free framework for trajectory control through aligning the self-attention with given bounding boxes based on pre-trained I2V model SVD. Additionally, SG-I2V supports arbitrary input images and any number of objects or regions of a scene.
weaknesses: 1. MotionCtrl is a well-known method for trajectory control and also provides the official code and pre-trained model for SVD (same backbone). SG-I2V is supposed to compare with MotionCtrl.
2. Since each inference requires individual optimization calculations, authors are supposed to discuss it. If the additional time is too much, is there any way to reduce it?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. The authors have been inspired by tuning-free text-to-video generation methods and have proposed a similarly tuning-free approach for image-to-video tasks. This eliminates the need for computationally expensive fine-tuning procedures, making the framework more accessible and efficient.
2. The paper provides a thorough analysis of semantic feature alignment within a pre-trained image-to-video diffusion model. 
3. The authors have provided a project page that offers a more intuitive presentation of their work.
summary: The paper introduces a novel framework for controllable image-to-video generation. The core innovation of SG-I2V lies in its ability to achieve zero-shot control over object motion and camera movement within generated videos, leveraging a pre-trained video diffusion model. The method does not require task-specific fine-tuning or additional external knowledge. The authors propose an approach that analyzes feature maps from the upsampling blocks of video diffusion models to identify weak semantic alignments across frames. To address this, they introduce a modified self-attention mechanism that aligns features across frames, enabling precise control over element positions in the generated video. The framework optimizes the latent space to enforce similarity between aligned features within bounding box trajectories, and a post-processing step is applied to maintain the expected distribution of high-frequency noise expected by the diffusion model. The paper demonstrates that SG-I2V outperforms unsupervised baselines and is competitive with supervised models in terms of visual quality and motion fidelity.
weaknesses: 1. While the paper claims the ability to control an arbitrary number of objects, it lacks experimental evidence to support this claim. The examples provided for controlling object motion are limited to simple scenarios with only 1-2 objects, which may not demonstrate the method's capability across more complex scenarios with multiple objects.
2. The method's robustness is questionable due to the fine-tuning of parameters specifically for the VIPSeg dataset. The learning rate, for instance, is set to a specific value (0.21), raising concerns about whether these parameters would be universally effective across various image-to-video examples without further adjustments.
3. It is unclear whether the proposed method could be adapted to other base video generation models beyond the one used in the paper. The transferability of the method to different models is an important consideration for its broader adoption.
4. The current method appears to support only linear trajectories, which may not be sufficient for more complex motion control. There is a need to explore whether the method can handle other shapes and longer, more complex trajectories.
5. Lacking of reference or analyze the relationship with related work on trajectory-based image or video generation, including TraDiffusion[1] and DragEntity[2].

[1] TraDiffusion: Trajectory-Based Training-Free Image Generation.

[2] DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships.

Meta-review: This work receives mixed reviews. While reviewers initially raised concerns regarding performance discrepancies, limited evaluation, technical novelty, and computational cost, the authors' detailed response and clarification have largely addressed these concerns. The paper's contributions, particularly in analyzing and leveraging semantic feature alignment for controllable video generation, are considered to be novel and effective. Therefore, it is recommended to be accepted. However, the authors are encouraged to address the remaining concerns regarding performance discrepancies, generalizability, and computational cost in their future work.

Decision: Accept (Poster)