FreeTraj: Tuning-Free Trajectory Control via Noise Guided Video Diffusion

Authors: Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, Ziwei Liu
Primary area: generative models
Keywords: Diffusion Model, Video Diffusion, Trajectory Control, Motion Control

Abstract:
Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomena and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models. Generated video samples are available at the anonymous website: https://FreeTraj.github.io.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729515316991

thread_type:
official_review

<thread_object>
creator: reviewer_2sgj

<content>
summary:
This paper introduces FreeTraj, a tuning-free framework for trajectory-controllable video generation using diffusion models, eliminating the need for additional training. It controls motion trajectories by guiding noise sampling and attention mechanisms. FreeTraj supports both manual and LLM-generated trajectories and enables longer video generation with precise trajectory control. Extensive experiments validate its effectiveness in enhancing trajectory controllability in video diffusion models.

strengths:
1. The method is training free unlike most existing works.
2. Extensive quantitative evaluation.
3. Methods are intuitive and the paper is easy to follow.
4. Superiority over baselines compared.
5. Providing observation before they build their method sounds convincing and well-structured.
6. Applications of Longer / Larger video generation.

weaknesses:
1. By "LLM trajectory planner", do the authors mean like LLM-grounded video diffusion [1]. Plus, I think LLM-grounded video diffusion would serve as a good baseline to be compared (ignoring the llm planning part). Also, if the LLM trajectory planner is not implemented or showcased, the authors should not sell this point.

2. Does the method increase computation burden? Providing and comparing information on the memory / time consumption would increase value of the paper.

3. How long and big are the "long" and "large" video generation in the results of appendix?

4. I think the intuition behind High-Frequency Noise Resampling is well explored in both image/video diffusion works. Not just FreeInit, but also in Diffusion-Motion-Transfer [2] (their initial noise preparation stage). And the novelty is limited.

5. It's hard to grasp what attention isolation is easily. Providing visualizations in relevant sections would help reviewers understand the issue and how they overcome it.

6. Is Cross Attention Guidance method not explored in diffusion-based image generation/editing literature?

7. Isn't TrackDiffusion [3] a relevant work? If so, how does the model perform compared to the work? 

8. Quantitative ablations studies would add value of the paper.

[1] LLM-grounded Video Diffusion Models, ICLR 2024

[2] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR 2024

[3] TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models, Arxiv 2023

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730117746802

thread_type:
official_review

<thread_object>
creator: reviewer_zfvl

<content>
summary:
The paper presents a tuning-free video diffusion framework for object trajectory control. The authors in this paper first analyze the influence of noise construction for video motion generation and then introduce the FreeTraj. The proposed framework modifies the noise sampling and involves the local trajectory injection in the noise initialization. Besides, the object mask is also integrated into the attention to emphasize the content generation across frames. Experiments on both controllable short- and long-video generation tasks verify the proposed approach.

strengths:
1.	The exploration on the noise initialization for object motion control is interesting and demonstrates the importance of noise structure for content generation in video diffusion models. 
2.	Both of the qualitative and quantitative experimental results demonstrate the effectiveness of the proposed approach for tuning-free motion control.  
3.	The potential of complex motion control is verified by the model that is combined with FreeNoise under the setting of long video generation.

weaknesses:
1.	My major concern is about the technical contribution which could be limited. The investigation of noise construction is more like an intuitive engineering work and there is no rationale behind. The influence of high-frequency noise part has been also explored in FreeNoise. The trajectory injection seems reasonable but more details should be included. Meanwhile, the authors argue that Peekaboo exploits the hard mask in attention. The motivation of the proposed soft Gaussian mask is not clear either. The hyper-parameter of such Gaussian kernel is not mentioned in the paper.
2.	In the experimental section, the comparison and discussion (line 465 to 470) between FreeTraj and MotionCtrl should be detailed. It is very difficult for readers to judge the technical differences or improvements from these descriptions. Why dose MotionCtrl only roughly control the object trajectory but not align the trajectory? The reason behind these results is not provided or discussed. 
3.	One suggestion is about ablation studies. There could be some quantitative analysis for different variants rather than only showing the visual cases. The quantitative evaluation results can reflect the performance from a global view.
4.	Some of the definitions should be aligned. For instance, the $F_{z_T}^{low}$ and $F_{z_T}^{L}$ in Eq. (7).

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537490505

thread_type:
official_review

<thread_object>
creator: reviewer_ssw8

<content>
summary:
This paper proposes a diffusion-based video generation method for controlling the trajectory of moving objects in a zero-shot manner and investigates the impact of initial noise on the trajectories of moving objects. By guiding the generated target in noise and attention, the proposed method can generate controlled trajectory of the target. Experimental results demonstrate the effectiveness of this method.

strengths:
- The method described in this paper does not require any training, which can significantly reduce computational overhead.
- This method analyzes the high and low frequencies of the initial noise and Attention Isolation, and proposes corresponding utilization methods and solutions.
- The provided generated video looks good in controlling.

weaknesses:
- In the paper, the author uses a box to control the motion of objects, but in the experiments, the objects are not actually inside the box; they merely maintain a trajectory consistent with that of the box.
- The method performs significantly worse than the Direct approach on the FVD and KVD metrics, and the author does not analyze this. This could potentially have negative effects on the clarity of the generated videos and on maintaining the identity of the objects.
- This method shows some weaknesses in generating complex and random trajectories. However, it seems feasible to achieve better results by increasing the control strength, according to the method. The author does not further analyze this aspect. The degree of control could potentially impact video quality. It would be beneficial for the author to include an ablation study on the intensity of control to better understand its effects.

questions:
- The author mentions that this method can be used for LLM trajectory planners, but how this can be implemented? This lack of detail could leave readers uncertain about the practical application of the method in such contexts.
- Since the method does not require training, it could be applicable to any diffusion-based video generation method, such as Stable Video Diffusion or OpenSora. Could you provide more experiments with different video-diffusion baselines? This universality could potentially make it a versatile tool in the field of video synthesis, especially for applications requiring rapid deployment without the need for extensive training data.
- As in zero-shot setting, Motion-Zero[1] has some similar features, modifying initial noise, and guidance in attention. in video generation. What's the difference between the proposed method and [1]?

[1] C. Chen et al. Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation. Arxiv 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730775068738

thread_type:
official_review

<thread_object>
creator: reviewer_3hqh

<content>
summary:
This paper analyze the trajectory-controllable video generation and introduce Free_Traj, a training-free motion control method.

strengths:
- It is intuitive and reasonable to use noise and motion guidance.

- Noise resampling is an interesting idea.

- The analysis is comprehensive, and the results seem good to me.

weaknesses:
- Is mIoU a good metric for trajectory control? 


- Can this method unify trajectory control and motion control? For example, a man waves his hand. In terms of motion, it is a "wave" action; regarding trajectory, the hand follows its specific path. Can we unify them to achieve more realistic motion control?

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732528160792

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We sincerely thank all reviewers for their constructive suggestions and recognition of our work. We are encouraged that reviewers find that our observation and analysis are **“convincing and well-structured.”** (Reviewer 3hqH, sSw8, zfvL, 2SGJ); our proposed method achieves **good traject control** (Reviewer 3hqH, sSw8, zfvL, 2SGJ); and tuning-free strategy is **valuable and meaningful** (Reviewer sSw8, 2SGJ). We have separate responses for each reviewer and also updated our submission to include the following changes according to reviewers' feedback. Note that the main revisions in the main paper and appendix are highlighted in blue:

1. We have added several missing relevant papers in the Related Work.
2. According to the advice from reviewers, we have added four parts to the Appendix: Quantitative Ablation (Section C.3), Control Intensity (Section C.4), Method Compatibility (Section C.5), and LLM-Planed Generation (Section C.6).

Please do not hesitate to let us know if you have any additional comments or there are more clarifications that we can offer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732992257902

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Contribution Clarification

comment:
Thanks to the constructive suggestions and active discussions from the reviewers, we have addressed most reviewers’ concerns through sufficient additional experiments and detailed explanations. The remaining concern is mainly about the contributions thus we make this clarification.

>**Our core contributions include:**

**1. Observations and analysis of the relations between initial noise and trajectory.** We first propose the **Noise Flow**, a naive way to prove that initial noise can guide the trajectory of generated videos. However, this rude manipulation will cause obvious artifacts. Therefore, we use **High-Frequency Noise Resampling** to preserve the video quality while still maintaining the ability of trajectory control. Finally, we inject the noise locally to separate the control of foreground and background.

**2. Trajectory Injection.** Based on our observations and analysis, we propose trajectory injection, which **directly** injects the target trajectory into the low-frequency part of the initial noise. The difficulties are how to construct a trajectory directly **without any iterative process** and how to preserve the quality after manipulating the initial noise. 

Since all motivation, design logic, and details are carefully introduced in our observations and analysis part (Section 3.2), we have omitted the repetitive and complicated instructions and only simply introduced the final operation in Section 3.3.1. That should be the reason why the novelty of this part is ignored. We apologize for the confusion and invite reviewers to consider the design logic and final solution of noise injection as a whole when evaluating the novelty of the article. We sincerely thank reviewer 2SGJ for recognizing that our effective initial noise initialization strategy will bring enlightenment value to the video gen community.

Additionally, our other two contributions consist of (1) improving the masked attention based on previous work [2,3,4], and 	
(2) extending the control mechanism to achieve longer and larger video generation with a controllable trajectory. These two contributions are effective but belong to engineering improvement and technical application. We agree that they are not very novel and the novelty discussion can focus on our core contributions. 

>**Difference with FreeInit [1]:**
  
FreeTraj only utilizes the idea of **High-Frequency Noise Resampling** from FreeInit and the two methods are obviously different in terms of motivation, pipeline, and dynamics:
1. **Motivation.** FreeInit targets to improve the generated quality by reducing the gap between training and inference. FreeTraj targets to control the trajectory of generated videos.
2. **Pipeline.** FreeInit needs to find a better initial noise through an iterative process, which is time-consuming. However, FreeTraj directly constructs the target initial noise without any iterative process.
3. **Dynamics.** Compared to the direct inference of the base model, videos generated by FreeInit tend to have fewer dynamics, while videos generated by FreeTraj have significantly larger dynamics.

>**Difference with Motion-Zero [5]:**
  
Motion-Zero is limited to replicating object motions derived from existing videos. It attains the original initialized noise by **reversing the given video using DDIM**, a process that is both time-consuming and constrained in terms of control diversity. In comparison, FreeTraj **directly** injects the target trajectory into the low-frequency part of the initial noise, enabling more versatile and efficient object motion control. In other words, Motion-Zero realizes that initial noise can guide the trajectory of generated videos but has no idea about constructing initial noise directly with preserved video quality and trajectory control ability.

[1] FreeInit: Bridging Initialization Gap in Video Diffusion Models   
[2] Peekaboo: Interactive video generation via masked-diffusion   
[3] Trailblazer: Trajectory control for diffusion-based video generation   
[4] Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion    
[5] Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission propose a training-free approach for object-trajectory-controllable in video generation. Specifically, the authors (1) analyze how the initial noise affects generated videos including the motion and output quality, (2) propose to adjust the initial noise and noise resampling for high frequency signals, and (3) introduce a soft-masking mechanism based on object locations.

Strength
* The idea of directly manipulating initial noise to influence object trajectories is interesting, intuitive, and requires no additional training.
* The paper offers a comprehensive analysis of how noise and attention contribute to video generation.
* The proposed method can produce promising results in terms of controlling object trajectories in generated videos.

Weakness
* The chosen metrics for trajectory control evaluation are not fully justified.
* The controllability is not accurate. Objects may not precisely follow the inpu bounding boxes.
* The generated videos show much worse quality metrics (FVD, KVD). The authors’ rebuttal attributes this to large motion within the video, but there's no analysis for how control may affect video quality and the implication of this observation.
* The technical contribution is limited. The proposed noise and attention manipulations has been studied in the same context.
* The paper lacks thorough analysis and detailed comparisons with closely related baseline methods, leaving the novelty and advantages of the approach unclear.
* While the authors mention using an LLM-based trajectory planner, the claim lacks justification.

The reviewers raised concerns about the technical novelty of the submission, as the proposed noise and attention manipulations have been previously explored in similar contexts. The lack of detailed analysis and comprehensive comparisons with relevant baseline methods further diminishes the clarity and significance of the contributions. Additionally, there are concerns about the output quality, including inaccuracies in trajectory control and poor performance on video quality metrics such as FVD and KVD. While the authors attributed the lower metrics to large motion dynamics in the videos, this explanation does not sufficiently address broader concerns regarding the practical effectiveness of the proposed method in generating high-quality, controllable video outputs.