3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation

Authors: Xiao FU, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di ZHANG, Dahua Lin
Primary area: generative models
Keywords: Controllable Video Generation, 3D Motion Control, Multi-Entity Motion

Abstract:
This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: http://fuxiao0719.github.io/projects/3dtrajmaster

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730234672027

thread_type:
official_review

<thread_object>
creator: reviewer_95hu

<content>
summary:
Paper tackles the problem of multi-entity 3D control signals for video generation

Approach works by constructing an Unreal Engine dataset of assets with GPT4V generated trajectories and various camera angles, and then LoRAing it to do domain adaptation to prevent it from looking too much like the UE assets.

Extensive qualitative results indicate the method clearly works, and quantitative results indicate value of components + favorable results compared to several baselines.

strengths:
The method clearly works. The supplemental provides extensive videos paired with prompts showcasing multiple walking agents. While generations are imperfect, I was shocked at the quality -- the assets seem to properly interact with light in the scenes, including full long shadows, as well as the terrain below, such as water. 

The method used to achieve this adaptation seems straightfoward, and the recipe seems generally replicable and applicable to other video generation models despite the fact that the generation model used is proprietary.

weaknesses:
- Pose evals are human only; however, I think this is well motivated given the stated lack of general video pose predictors.

Quite honestly, I am not an expert in this domain and I lack the background to provide meaningful criticism. The results look good, the actors clearly follow the given trajectories, and the recipe given to achieve this generally makes sense and feels general enough to be useful for arbitrary video generation models.

questions:
- Why does the alligator gait look so much worse than the other animals? Its legs seem to be almost static in some scenes.
 - It seems that the proprietary base video generation model is extremely strong (congrats!). Do you think this recipe will work with less strong video generation models, such as current open source models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730400661563

thread_type:
official_review

<thread_object>
creator: reviewer_2hag

<content>
summary:
1. This paper aims to control the entities’ motion with 3D control signals in video generation.  3D control signals are a more natural representation compared to 2D signals as the motion is in 3D space. 
2. This paper proposed a plug-and-play module to integrate the entities with their respective 3D trajectories into a pre-trained video generative model to control the 3D motion. 
3. To avoid video quality degradation during the fine-tuning, they use a Lora-like domain adaptor and an annealed sampling strategy.
4. They construct a synthetic dataset collecting dynamic 3D human and animal assets with ground-truth 3D motion for training.
5. Experiments show that the proposed methods can achieve state-of-the-art performance in 3D motion control.

strengths:
1. The proposed method is the first to control entities’ motion with 3D trajectories in video generation. The task is novel and reasonable as 3D control signals can fully express the inherent 3D nature of motion and offer better controllability in video generation compared to 2D control signals.
2. The method design is clear and reasonable.
3. The paper constructs a new synthetic dataset for this task. The dataset potentially benefits the following video generation with 3D entity control. 
4. The experiments are thorough and solid. Plenty of visualization results as well as the videos in the supplementary demonstrate the effectiveness and generability of the proposed method.
5. The paper is well-written and easy to follow.

weaknesses:
1. The dataset lacks diversity in terms of background and motion types. The setting is restricted to a "City" environment (as noted in the paper's Limitations section), and the actions are primarily limited to walking. Consequently, models trained on this dataset are also constrained in their generalizability.
2. Foot skating/floating issues are prevalent in the dataset. This appears to result from inconsistencies between the relative motion and global motion of the dynamic entities, which could negatively impact model training by introducing artifacts.

Minor:

1. The explanation of the "ControlNet-like architecture" is vague and lacks clarity. The paper references this term in Lines 20 and 115, suggesting it pertains to the Object Injector, whose initial weights stem from the 2D spatial self-attention layer in the video generative model. However, this does not align with a true ControlNet-like module, which would typically function as a parallel module with zero-initialized layers designed to adjust the original features. Instead, the Object Injector is positioned after the 2D spatial self-attention layer, differing fundamentally from ControlNet-like behavior.
2. It would have been better to elaborate on Line 334: “This phenomenon reflects the model’s ability to learn 3D motion representations”? The reasoning behind this statement isn’t entirely clear.

questions:
In Table 2, could you clarify the distinction between “Multiple Entities” and “All Entities”? Does “All Entities” include both “single” and “multiple” cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677700147

thread_type:
official_review

<thread_object>
creator: reviewer_wqpa

<content>
summary:
3DTrajMaster" introduces a method for controlling multi-entity 3D motion in video generation using 6DoF pose sequences. The authors propose a novel plug-and-play 3D-motion grounded object injector that fuses entity descriptions with corresponding 3D trajectories using a gated self-attention mechanism. They address the lack of suitable training data by constructing a 360°-Motion Dataset, combining collected 3D assets with GPT-generated trajectories. The method is tested against prior 2D motion control approaches and shows state-of-the-art performance in both motion control accuracy and generalization.

strengths:
The proposed 3D-motion grounded object injector, combining 6DoF pose sequences with entity descriptions, is an innovative contribution that extends beyond 2D control limitations.

**Dataset Creation**: The construction of the 360°-Motion Dataset addresses a notable gap in available training data, particularly for multi-entity scenarios, using an innovative combination of GPT and UE.

**Flexibility**: The plug-and-play nature of the proposed object injector facilitates broader applicability across different generative models, with the gated self-attention mechanism ensuring entity-specific trajectory adherence.

weaknesses:
**Dataset Limitation**: The reliance on synthetic data and a limited number of assets may hinder real-world generalization. The "city" setting constraint for the dataset also limits the diversity of possible outputs.

**Generalizability**: The model's performance for generalized 3D scenes beyond those captured in the MatrixCity platform remains unclear. More evaluation of real-world, diverse datasets would strengthen the contributions.

**Evaluation Scope**: While evaluation metrics like FVD and CLIP Similarity are used, the lack of real-world evaluations or qualitative feedback from human users makes it hard to gauge practical effectiveness fully. Also, the author could consider comparing with recent 4D generation methods such as TC4D which also can control trajectory.

I will also check other reviewers's feedback.

questions:
Will the dataset and code will be public after acceptance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725279155

thread_type:
official_review

<thread_object>
creator: reviewer_zy5x

<content>
summary:
The paper proposes a 3D-trajectory-conditioned video generation method, fusing prior from pre-trained video diffusion models and from a proposed motion dataset.

strengths:
The paper addresses the lack of 6-DoF controllability of existing video generation methods. The method is well-motivated and method designs are clearly explained. The advantage of 6-DoF control over 2D motion control is clearly demonstrated in experiments.

weaknesses:
* The section on related works discusses prior methods on motion control and motion synthesis tasks, but could also include discussions on techniques for injecting controls to video foundation models, including ControlNet [1] and methods that allow 2D image editing by manipulation attention maps. In particular, ControlNet [1] is currently mentioned but not cited in the paper. 
* The proposed dataset is restricted to human and animal categories, and locations remain to be in cities. Whether it's feasible to scale this method to generic object categories and generic scenes remains an open question.  

[1] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models.

questions:
* Evaluation of multi-entity input sequence sets $N=2$, i.e., 2 entities, based on the qualitative examples. Is the method restricted to a small number of entities, and if so, does the restriction come from training data? If it's tied to training data distribution, it remains even more unclear if the method can potentially apply to generic settings where >>2 objects are moving, which is fairly common in dynamic scenes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731486214672

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Conclusion of Discussion Period, Paper Revision, and Dataset&Code Release (for AC's Meta Review Convenience)

comment:
> **1. Conclusion of Discussion Period**

Our work is `the first to control 3D multi-entity motion in video generation`. In contrast to the previous 2D control signals, we propose to manipulate dynamics in 3D space, which can better express the 3D nature of object motions, with entity-specific 3D pose sequences (6 DoF) as additional inputs. We are pleased to see that the reviewers acknowledge the following aspects of our work.

1. **The first to control multi-entity motion with entity-specific 3D trajectories in video generation.** (Reviewer zy5x, WQpa, 2Hag)
2. **Clear advantage of 6-DoF control over 2D motion control** (Reviewer zy5x, WQpa, 2Hag, 95Hu)
3. **The method is reasonably designed, straight-forward and flexible** (Reviewer zy5x, WQpa, 2Hag, 95Hu)
4. **The contribution of 360-Motion Dataset** (Reviewer WQpa, 2Hag)
5. **Solid and convincing experiments** (Reviewer zy5x, 2Hag, 95Hu)
6. **Impressive generalizable results** (Reviewer 95Hu)

We make our anonymous website live at https://3dtrajmaster.github.io/. It includes all the videos in our paper, benchmark results, samples from the 360-Motion Dataset, and additional videos requested by the reviewers (we kindly invite all reviewers to **request extra video samples** with specific entity prompts, trajectory templates, and locations, and generate those videos and make them available on the anonymous website)

Regarding reviewers' concerns, we also have made efforts to address the reviewers' concerns in the relevant sections and reflect them in the Paper Revision.

_________________

> **2. Paper Revision**

**(1) More Diverse Training Dataset (Reviewer zy5x, WQpa, 2Hag)**: This is the primary limitation pointed out nearly by all reviewers. To address it, we extend the pure city dataset to more diverse 3D UE scenes such as desert, forest, Asian town, HDRIs (projected into 3D), as well as more complex 3D trajectory templates. Please see the dataset samples on our [anonymous website](https://3dtrajmaster.github.io/). Below is a summary of the differences.

| Dataset      | 3D Scenes | Samples     |  3D Trajectory Templates     |
| --- | --- | :---: |:---: |
| Rebuttal Version   | 9 (city, desert, forest, Asian town, 5 HDRIs)       | 54,000     | 96  |
| Submission Version     | 1 (city)       | 12,000   | 48  |

**(2) More Diverse Evaluation Benchmark (Reviewer zy5x, WQpa)**: Based on reviewers' doubts and Revision (1), we have expanded our evaluation to include a wider range of non-human and location prompts. Below is a summary of the differences. For more details, please refer to Tables R4-R7.

| Eval. Benchmark | Num. of Human Prompts  | Num. of Non-Human Prompts |  Num. of Location Prompts  |  
| --- | :---: | :---: |:---: |
|  Rebuttal Version   | 20    | 52 (animal, car, robot)    |  32 |
| Submission Version      | 20     | 40 (animal)  |  1 (city) |

**(3) Further Revisions based on Reviewer 2Hag**: 

1. Explanation of the"ControlNet-like architecture" (See ‘’W3-Minor Explanation of the 'ControlNet-like architecture' ")
2. Elaboration on Line 334 (see “W4-Minor Elaboration on Line 334”)
3. Elaboration on the “Limitation” section (see “2. Motion Type”)

**(4) Further Revisions based on Reviewer zy5x**: 

1. Update missing related work (See "W1 Missing Related Work" and Sec. B in the paper)
2. Update "Clarification of the Limited Entity Number" in Sec. D of the paper
3. Clarify the multi-entity evaluation in Sec. 4.4

**(5) Further Revisions based on Reviewer WQpa**: 

1. Update "Qualitative Feedback from Human Users" in Sec. F.2.3
2. Update "Additional Applications" in Sec. C

**(6) More SOTA baseline**: We additionally incorporate Tora [1], which is released after the ICLR submission DDL, in our comparison for more comprehensive results.

**(7) More Visually-Appealing Figures based on Revisions (1) and (2)**: We have redesigned the main paper figures to be more visually pleasing and adhere to our latest model requested by reviewers.

_________________

> **3. Dataset & Code Release**

Reviewer WQpa raises a question that we believe will be a concern for all reviewers. Below is our response.

**(1) Dataset Release**: Yes. We plan to release our dataset to support the research community once the paper is accepted, or possibly even earlier if the review scores are favorable. To demonstrate our commitment, we have provided a subset of smaller datasets on our anonymous website (including 5 scenes, and 480 videos).

**(2) Code Release**:  Yes. The release of the code based on our internal model will go through our internal license review process, but we commit to releasing a version based on the publicly available CogVideoX/Mochi 1 once the paper is accepted, or potentially even earlier if the review scores are favorable. 

_________________

[1] Tora: Trajectory-oriented Diffusion Transformer for Video Generation. ARXIV, 2024
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel method for controlling 3D multi-entity motion in video generation. The proposed feed-forward approach allows for the manipulation of dynamics in 3D space using entity-specific 3D pose sequences (6 DOF) as additional inputs rather than relying on 2D control signals. Another significant contribution of this study is the creation of the 360-Motion Dataset, which the authors have committed to releasing along with the code.

The paper's main weaknesses include a lack of comparison with some state-of-the-art baselines, a need for a more diverse training dataset plus a diverse evaluation benchmark, and limitations on the number of entities for multi-entity input. The authors have addressed these concerns by providing extended data on different scenes, trajectory templates, and prompt types. The authors also conducted a user study and provided further explanations of weaknesses that they could not currently resolve.

The paper received ratings of (5, 6, 8, 8) from reviewers. Most expressed positive views regarding the paper's contributions, quality, and the significance of the experimental results. The area chair concurs with the reviewers' suggestions and recommends accepting the paper.