3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation
Authors: Xiao FU
Xian Liu
Xintao Wang
Sida Peng
Menghan Xia
Xiaoyu Shi
Ziyang Yuan
Pengfei Wan
Di ZHANG
Dahua Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Controllable Video Generation
3D Motion Control
Multi-Entity Motion
Primary area: generative models
TL;DR: 3DTrajMaster masters multiple entity motions in 3D space for text-to-video (T2V) generation by leveraging entity-specific 6DoF pose sequences as additional inputs.

Abstract: This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: http://fuxiao0719.github.io/projects/3dtrajmaster

Published as a conference paper at ICLR 2025
3DTRAJMASTER: MASTERING 3D TRAJECTORY FOR MULTI- MOTION IN VIDEO GENERATION
Xiao Fult Xian Liu'
Xintao Wang 25t Sida Peng" Menghan Xia² Xiaoyu Shi?
Ziyang Yuan² Pengfei Wan? Di Zhang² Dahua Lin
The Chinese University of Hong Kong KKuishhu Technology 3Zhejiang University
ABSTRACT
This paper aims to manipulate multi-entity 3D motions in video generation. Pre- vious methods on controllable video generation primarily leverage 21) control sig- nals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce Masrajaaster, robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our ap- proach a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video qual- ity degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable train- ing data, we construct a 360%-Motio Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Exten- sive experiments show that 3DTrajMaster sets a new state-of-the-a in both ac- curacy and generalization for controlling multi-entity 3D motions. Project page: httt:://fuxia07701 githeb. htt.:b/////orooooe.ssooo /ddt 3dt ra jmaster)
INTRODUCTION
Controllable video generation (Brooks et al. 2024; Guo et al.] 2023b] Chen et al. 2023) aims to synthesize high-fidelity vidcos that are controlled by user inputs, such as text prompts, sketches, or bounding boxes. A critical objective in controllable video generation is the precise manipulation of object motions within videos, which is essential for simulating the dynamic world and potentially aids video generative models in understanding the underlying physics of the world. In addition, it can unleash many applications of video generative models, such virtual cinematography for the film industry, acting as interactive games, and providing world models for embodied AI systems.
Recently, there has been some methods attempting to manipulate object motions in video generation by introducing 2D control signals, such as 2D sketches (Wang et al. 2024b] Guo et al. 2023a), bounding boxes (Yang al., 2024, Wang et al., 2024a). and points (Wang et al. [2024c] Zhang et al. 2024). These methods offer convenient user interactions and have delivered impressive video generation results. However, we argue that 2D control signals cannot fully express the inherent 3D nature of motion, which limits the control capability of object motions. As real-world objects move in 3D space, some motion properties can only be described through 3D representations. For example, the rotation of an object be succinctly described using three parameters in 3D, and occlusions between objects can be simply represented using bubuffering. In contrast, it is quite difficult for 2D control signals represent these concepts.
In this paper, we focus on the problem of controlling multi-object 3D motions in video genera- tive models, aiming to simulate the authentic dynamics of objects in 3D space. This setting is
t: Work done during an internship KwaiVGT, Kuaishou Technology. 25, Corresponding Authors.

---

olly Published as a conference paper at ICLR 2025
Entily a flamingo with pink feathers, long slender legs, curved neck, and a feathers on back
Entity gazalle with a body covered sleek tan fur, long lege, and elagant curved horna man with medium langth straight hair, tall slendar, gray orew-neek t-shirt, beige trousers
Location glaoler
1
Ensty a body oovered soft fur, quiok hops, a playful demeanor
Location seaside gentle breeze with soft tendrils of pale blue mist resembling flowing fabrio,
fabrio,delioate delioate streaks white vapor
Entity arocodie with body eovered in sealy green skin, powerful fail, sharp teeth Location snouy wolf with thick ailver gray fur, alert golden eyes, lean yet strong body, exuding confidence and boldness
10
Entity man with spiky blonde hair, alim build, black french coat, blue jeans
Location savanna
@
bear with dark brown fur, small but fieroe black eyes, and broad and muscular build, radiating power parrot with bright red, blue, and yellow feathers, ourved beak, and sharp intelligent eyes
Entity vintage convartible siiilleihh body covered shiny red paint, chroma bumpars
Location cave
man with short black wavy hair, lean Agurre, green and yellow plaid shirt, dark brown pants panda with body covered fluffy blaok-and-wwhie fur, round face, and gentle eyes, radiating warmth
Entity an exploration rover robot with solar panels, durable wheels, and advanced sensors storm enfitt with dark suirling clouds as body, streaks of electrio blue lightning shooting across it a man with buzz-out blonde hair, stooky build, a gray zip-up sweater, black shorts, and red baskstball shoes
Location park
Entity man with short black wavy hair, lean figure, a green and yellow plaid shirt gazelle with body covered tan fur, long lege, and elegant ourved horns
Location beach
Modify with short black wavy hair, lean figure, grean and yellow plaid shirt robotio gazelle with sturdy aluminum frame, an agile build, artioulated legs and ourved, metalioo horma
Location sunset beach
Figure 1: 3DTrajMaster controls one or multiple entity motions in 3D space with input entity- specific 3D trajectories for text-to-video (T2V) generation. It allows diverse entity categories (human, animal, car, robot, natural force, etc) and flexible edits on entity descriptions (see more in Fig. S11.. The text prompt is "[Entity !)..., and (Entity N is/are moving in the {Location)". (We kindly urge readers to check more generalizable results (2200) our website)
more aligned with the requirements of downstream applications, such as emulating realistic human motions in movies exploring 3D virtual scenes in games. However, this problem is extremely challenging. There are three core questions we need to answer: 1) How to precisely represent the 3D motions of objects; 2) How to correlate multiple object descriptions with their respective motion
2

---

Published as a conference paper at ICLR 2025
sequences in video generative models; 3) How to maintain the generalization capability of video models after injecting 3D motion information.
To address these, we propose novel approach, Master, which is able to manipulate multi- entity motions in 3D space for video generation by leveraging entity-specific 6DoF pose sequences as additional inputs. The core of our model is plug-and-play 3D-motion grounded object injector, which associates each entity with their corresponding pose sequences, and then injects these condi- tions into the foundation model, to control the entity motion. Specifically, the entities and trajectories are projected into latent embeddings via frozen text encoder and a learnable pose encoder, respec- tively. These two modality embeddings are then entity- added to form correspondences, which are further fed into a gated self-attention layer for motion fusion. This plug-and- and-play architecture preserves the video model's prior and can generalize on more diverse entities and 3D trajectories.
However, another challenge in training our model lies in data availability. Existing video datasets face two key limitations: 1) Low entity diversity: Datasets with paired entities and 3D trajectories are mostly limited to humans and autonomous vehicles, with inconsistent spatial distributions and overcrowded entities. 2) Inaccurate/Falles pose estimation; Current 6D pose estimation methods focus on rigid objects, while non-rigid objects, such as animals, are underrepresented, with only human poses studied using SMPL (Loper et al.) 2023). To this end, we choose to construct a cus- tom dataset. termed 3602-Motion Dataset, with unified trajectory distribution using advanced UE rendering techniques. We start by collecting 3D assets of humans and animals and rescaling them to unified cubic space. GPT (Achiam et al.) 2023) is then employed to generate 3D trajectory tem- plates for these assets. Various entities and trajectory templates are arranged and combined to create diverse motions. These globally animated assets are captured using 12 evenly positioned cameras within the collected 3D scenes, including city (MatrixCity (Li et al. 2023a)) ) desert, forest, and HDRIs (projected into 3D space P To prevent video domain shift in our constructed dataset, we in- troduce two key components: 1) A video domain adaptor, which is trained to fit data distribution and slightly reduced during inference. 2) An annealed sampling strategy, where trajectories are injected to guide general motion in the early steps and drop out in the later stages.
We evaluate our 3aTrejMaster in the curated novel pose sequences with GPT-generated entity prompts, obtaining significant lead over current SOTAs. In summary, our contributions are:
1) We are the first to customize 6 degrees of freedom (DoF) multi-entity motion in 3D space for controllable video generation, establishing a new benchmark for fine-grained motion control.
2) We propose 3D-motion grounded video diffusion model that controls multi-entity motions using pose sequences as motion representations. Our flexible object injector enforces entity- corre- spondence between objects and their motions and preserves the video diffusion prior.
3) We introduce scalable 4D motion dataset construction mechanism, and techniques like the video domain adaptor and annealed sampling to enhance video quality while maintaining motion accuracy.
4) 3DTrajMaster achieves state-of-the the-r-rrtttttttoo art accuracy in controlling 3D entity motions and allows fine- grained entity input customization such as changing human hair, clothing, gender, and figure size.
2 RELATED WORK
Customizing Video Motion with 2D Guidance. Previous methods predominantly perform motion control on 2D spaces, as this aligns more easily with the input video format. A straightforward path is direct videos based on motion patterns from reference videos (Zhao et al. [0023] Jeong et al. 2024 Ling et al.) 2024). However, they require users to provide reference video templates. While training-free paradigms Yang et al., 2024 Xiao et al. 2024). utilizing attention mechanisms to edit spatial-temporal layouts, can mitigate this issue, they exhibit poor generalization in real- world scenarios and rely heavily on trdal-and-error. Further advancements utilize more high-level representations, such sketches&depphs (dense or sparse) (Wang et al. 2024b; Guo et al. (023a) pose skeletons (Feng et al. 2023]] Xu et al. 2024 [Chen et al. (2024). bounding boxes (Wang et al. 2024a), and 2D trajectories (Wang et al.] [202cc] Zhang al. [024 Yin et al. [023] Yang al. 2024], to enable more flexible motion generation. Although these methods can model camera, object, or joint movements, the lack of 3D awareness limits precise 3D motion control.
Poly Haven: http://polyhaven.com/
3

---

Published as a conference paper at ICLR 2025
Learning 3D-aware Motion Synthesis. Considering that video is sequence of images projected from 3D world, manipulating vidco in 3D space is both more crucial and impactful. A key aspect of this manipulation is camera movement. MotionCtrl (Wang al., 2024c) is the first to regu- late video using camera poses (rotation and translation) in 3D space, while CameraCtrl (He et al.) (2024) and VD3D (Bahmani et al. 2024b) further enhance camera representation with plücker em- beddings (Sizzmann et al. (021)) SynCamMaster (Bai et al. (024) extends single-camera control to multi-camera synchronization. GameGen-X (Che al. (024) can generate game videos with novel 'WASD' keyboard inputs. Other approaches (Hou et al., 2024 Hu et al., (024a) also explore training-fres free paradigms. However, none address the customization of object motion in 3D space. Manipulation on 2D maps (Wang et al., [2024c] Zhang et al. 2024) often fails in multi-object scenar- ios, particularly with 1) aligning each entity and corresponding motion, 2) handling 3D occlusion. In contrast, 3asrajMaster is the first to overcome them and simulate plausible 3D motions.
3 3DTRAIMASTER
Our goal is to master entity motions in 3D space for text-to-video (T2V) generation by leverag- ing entity-ppecific 3D trajectories as additional inputs. To this end, we introduce 3DTrajMaster (see Fig. 2). 3D-motion grounded video diffusion model trained in two stages. First, we describe the video diffusion model and the task formulation (Sec. [3.I]. Then, we present our proposed model, whose core is to train and--and 3D grounded object injector to integrate multiple detailed en- tity descriptions and the respective pose sequences (Sec. (.2). We further incorporate a domain adaptor to mitigate video domain shifts introduced by our constructed training data (Sec. 3.3 Fi- nally, we detail the inference process using annealed sampling to enhance video quality (Sec.
Trooue
Modified Video Latent B. (X(WWW))
Elemento-wise Addition
R (
'Entity Ii and (Entily N) (P moving (Location).
Gated Self-Attn. (Trainable Copy)
D
3D VAE
Text Encoder
g
Video Latent 2 (x(BxW) (DD
Concatenation
Remaining Layers
Entities fex L.
N. Aena i
expansion
expansion
R NNFFD
bokon padding
Entittywwise Addition
12 i. D
Object Injector
paired
Text Encoder
Pose Encoder
Trajectories (P.) y
Spatial Self-Attn.
"a woman with blonde hair,
DIT Blockx.N
"an alpasa with soft woo,
Ky
& (BxWW)
Entities () J
Trajectories RXXFX12
(a) 3D-Motion Grounded Video Diffusion
(b) Object Injector
Figure 2: 3DTrajMaster Framework. Given a text prompt consisting of N entities {e,, IN 3aTrerMMsster (a) is able to generate the desired video with entity motions that conform to the input entity-wise pose sequences [P,) Specifically, it involves two training phases. First, it utilizes a domain adaptor mitigate the negative impact of training videos. Then, an object injector module is inserted after the 2D spatial self-attention layer to integrate paired entity prompts and 3D trajectories. (b) Details of the object injection process. The entities are projected into latent embeddings through the text encoder. The paired pose sequences are projected using a learnable pose encoder and then fused with entity embeddings to form entity-trajectory correspondences. This condition embedding is concatenated with the video latent and fed into gated self-attention layer for motion fusion. Finally, the modified latent gets back to the remaining layers in the DiT block.
3.1
PRELIMINARIES ON 3D-ENTITY: AWARE VIDEO DISTRIBUTION
Video Diffusion Models. Latent text- to- video diffusion model (Ho et al. 2022a by Brooks et al.,
2024
Chen et al.) 2023 Blattmann et al. 2023) learns the conditional distribution p(x c) of encoded
video data x (x = E(X), ), E(.) is VAE encoder) given text description e in latent space. In the forward
progress, it progressively transits the clean data Xn the desired Gaussian distribution in Markov
4

---

Published as a conference paper at ICLR 2025
chain: fx,, (1,T) X =axi + + -
To iteratively recover the data Xy from the noise 6 ~ (0, it learns a denoising model ég with the objective function: R éo (x4: 1, c). With the preconditioning strategy (Karras et al. 2022 Salimans & Ho 2022), it optimizes the neural
network Fg by parameterizing the ég as: Eg - Conr(ar) Lo (Cin( C, o4) + Cskip (ax)
Task Formulation. Given an input text prompt e consisting of entities (e,, ) and their paired 3D trajectories {P PP where Pl = [R; T E _2x4 for f-th frame and object orientation and translation are represented by R E RR3 and T IR, respectively. our goal is to generate plausible video X E Fx that accords with each entity description and the respective trajectory P. The overall generative formulation f(.) is
:())cce D (en E Eyun ,P, E R 1n=1 + XEERXHXW
(1)
where X (xxn) (D(.) is the VAE decoder), = ((x)) [ Po (R,-1 &r C, (e,, P Pa)
Y' is the alphabet, and 7. is the token length. Our primary challenge lies in modeling the distribution Pe or specifically éạ to generate realistic vidcos that accurately correspond to the given multiple 3D entity conditions. Here we structure éø (x; c, or, (e,, P., pw) as transformer architecture (Peebles & Xie 2023) for its superior scalability and performance over U-Net (Ronneberger et al. 2015).
3.2 PLUGGAND-PPAAY 3D-MOTION GROUNDED OBJECT INJECTOR
Matching Entity-Trajecton Pair. The entity prompts {e, J/V are projected into latent embed- N
dings { n. n/ using a frozen text encoder ( : C22 E y) + Z E IR. lun xD, where each em- bedding Z is zero-padded to maximum token length Imax- Correspondingly, the pose sequences [Pr)a are also projected into latent embeddings (Z ) through the trainable pose encoder (p)): P. E IF
Z E IR FXD, The pose encoder Ep consists of a linear layer and downsampler along the temporal dimension, resembling the causal encoding applied to video in- put x in 3D VAE, where the mapping function is Ex(-): X E Fx + E Fx H xW Here the downsampler refers to interval sampling of tensors, where we also tried several sequen- tial one-dimensional convolution layers but achieved similar results. Then, the paired entity and trajectory embeddings are expanded and combined through entity-wise addition to form a bonded untity-motion correspondence ZPo
R FxNx Tames x
Gated Self-Attention for Motion Fusion. Inspired by (Li et al., 2023b), we employ a gated self- attention layer to handle multiple entity-trajecton pairs n (with varying dimensional embeddings) as input, while further refining the correlated features. Specifically, we replicate the weight of the 2D spatial self-attention layer in each DIT block as initialization to enable grounding. The input video tokens x, and Ze are passed through this trainable copy via truncated self-attention. The output can be expressed in a residue-connection form:
X: X, + B- Te(Att( q, k, v))
Q-Q-10 Q-Q.T.k- K T V T,T = x, Pe
where 3 is a trainable scale, Te(-) is the truncation operation to preserve x,, tokens, Att(-) is softmax attention, Q, K and V are query, key and value embedding matrices, and @ denotes con- catenation. In this stage, train the 0₂ including the pose encoder and the gated self-attention parameters as follow.
[(0))= = Excee-Wo.cc [||e ćo, (xx, c, (Cn, DW)n), 1)
(3)
3.3 ALLEVIATING VIDEO DOMAIN SHIFT FROM CONSTRUCTED TRAINING DATA
460--Motion Dataset. High-quality training data is vital for learning generalizable 3D motion control. A straightforward preparation is to extract paired entity descriptions and 6DoF poses from common video datasets. However, is hard due twofold: 1) Low diversiry/quality entiry: Datasets with paired entities and 3D trajectories are mostly limited to humans (Jiag et al.) 2022 Araújo al. 2023) and autonomous vehicles (Geiger et al., 20122 [Sun al. (0200). where the spatial distributions vary between datasets and the entities may be overcrowded. In video datasets like Artgrid, Pixabay, and Pexels" human category occupies a relatively large proportion in 3D/4D asset
Arigrid: https://argrrd..oo. Pixahay: http://wwww.vdeev..cc. Pexels: http://wwwwppxxlsscomm
5

---

Published as a conference paper at ICLR 2025
single оbj (25%)
two obss (65%)
three abjs (10%)
(a) Collected 3D Assets
(b) GPT- Trajectories (Bird's Eye View)
(c) Collected 3D UE Scenes
(d) 360 Surround Camera Shooting of Object Motion
Figure 3: Dataset Construction Illustration. We correlate (a) collected 3D assets with (b) GPT- generated 3D trajectories on (c) diverse 3D UE platforms, positioning (d) 12 evenly distributed surrounding cameras to capture the object motions video format.
objectives (refer to Sec. E.2). limiting model generalization to other categories like animals and vehicles. Issues like watermarks in WebVid (Bain et al., 202T) further increase the cost of filtering- 2) Low-accuracyFaalle pose estimation: Most 6D pose estimation methods exclusively focus on rigid objects, and rely on CAD models (Labbé et al. 2022 Wen et al.] (024) or posed multi-view images (Liu et al. 2022 Sun et al.) 2022). For non-rigid animated objects, only human poses have been widely studied via methods like SMPL (Loper et al. 2023), limiting the estimation for general 4D objects, such animals. models (Hu et al. 2024b, Ke et al., 2024; Fu et al.) 2025). However, there exist errors segmenting the foreground entities from the background and can not generate consistent video metric depth.
simpler alternative is represent only 3D locations via depth
To circumvent the aforementioned challenges, we opt to construct a synthetic dataset, named 360-- Motion, through Unreal Engine (UE) with advanced rendering technologies (see Fig. (3) We begin by collecting 70 animated 3D assets across two categories: human and animal. Humans are dif- ferentiated by attributes such as gender, clothing, body shape, and hairstyle. GPT-4V (OpenAl, 2023 is then used to generate text descriptions E yyta ( Ln, S 20) for each rendered asset image (Fig. (a)). For posed object trajectory templates (Fig, 3 (b)), we follow TC4D (Bahmani et al.) 2024a) by leveraging GPT to generate 3D spline (location T) and additional orientation R via the gradient calculation on spline. This process yields approximately 96 templates in canonical space, each associated with one to three assets. We additionally reduce the size of the animals by ratio of 0.6 to prevent collisions with other assets. The paired assets and their motion templates are then placed within 5x5 square meter range in one of the 3D platforms, including city (MatrixCity (Li et al. 2023a)), ) dessert, forest, and HDRIs (projected into 3D). We position 12 sets of cameras evenly around the scene to capture 360-degree views, producing 100 frames per video clip at 384x x 672 res- olution for each camera. This process produces a total of 54,000 videos by arranging and combining various objects and trajectories. (see Sec. E. and Supp. video samples for illustration)
Video Domain Adaptor. Training video diffusion models on this relatively small set of constructed video clips lead to undesirable UE style, limiting the generalization ability. To prevent learning this variation in quality and retain the knowledge of the base T2V, we train LoRA mod- ules (Hu et al. 202T) that serve video domain adaptor. Specifically, we integrate LoRA into self- attenntnnio cross attention, and linear layers of the base T2V model, as shown in Fig. 2 The attention/linear projection matrices (Wn) are associated with additional trainable lower rank matrices (AW,, n a A,, B J==1. where a is the scaler that can be adjusted to control the adaptor influence. During inference, we set
small value to mitigate the negative impact of synthetic video data. We optimize 02 (AW, nWWni ) with the training objective:
((22) = - [|e (le-6o, (xe, c, a)
(4)
Note that the domain adaptor 0₂ is frozen when training the object injector 01.

---

Published as a conference paper at ICLR 2025
3.4 INFERENCE PROCEDURE
We initialize the video latent Xr as standard Gaussian noise, and progressively denoise with the guidance of desired entity-trajectory pairs (e,, P. Pr) following the same schedule the previous two training stages. We apply classifier-free guidance (Ho & Salimans (022) and use DDIM (Song et al. 2020) for re-spaced sampling for acceleration. To further enhance the video quality, we employ an annealed sampling strategy (Algorithm 1): During inference in the former steps, trajectories are inserted into the model to define the general object motions, while in the latter stage, they are dropped out, transitioning to the standard T2V generation process. We also observe that setting negative 3D trajectories static motions {(P,) P. Pu. Vn} can further improve pose accuracy. This phenomenon reflects the model's ability to learn 3D motion representations: Since do not randomly drop out motion sequences during training like text, the model implicitly learns static motion modeling from videos where entities are primarily in motion. Thus when setting static motion "negative motion prompt", leading to improved pose accuracy during evaluation. However, we do not adopt it as it sometimes results in video quality decline (refer to Sec. F.2.2).
can amplify the magnitude of entity movement,
Algorithm Annealed conditional sampling with classifier-free guidance (CFG)
Require: w: guidance strength, Te: annealed timestep, LoRA modulator, O: frozen base T2V model, 0:: object injector, 02: domain adaptor, c: text condition, (c,P); P): entity-trajectory pairs 1: X1 N' (0, II
2: for 1..... do
3:
if S T. then
4:
(1 + w) (2,, c, (en: Pa) wē 0, (8)
5:
else
6:
à = (1+ + w)eg (2,, c) - weg (St)
7:
end if
8:
2 = (&t o,ēt) /0n
9: Re+1 ~ ( (Er, &t), ,0+1111) if < T else 81+1 2..
10: end for
11: return X,+1
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
unified template: "[Entity I}...., and Entity N} are moving in the [Location). )." Here we set "[Location]" based on the respective 3D UE platform. We train 3DTraajMastrr based
For input text prompts,
internal video diffusion model for research purposes (see Sec. A for more details), which contains 1B parameters. The clipped training video and inference video are to 384 672 resolutions. Each video segment is seconds long. We utilize the Adam optimizer and train on a cluster of 8 NVIDIA H800 GPUs, with a learning rate of 5 x 10 and a batch size of The training process consisted of 50,000 steps for the domain adaptor and an additional 36,000 steps for the object injector. During inference, we set the DDIM steps as 50 and the CFG as 12.5.
4.2 BASELINES
We compare Master with existing SOTA methods that are capable of customizing object motions: MotionCtrl (Wang al. 2024c), Direct-a-Video YYang et al. 2024) and Tora (Zhang al.) 2024). We configure these baseline models using their best performance settings, based on their official open-sourced codebases.
4.3 EVALUATION METRIC
1) Trajectory accuracy: Due to the absence of a pose estimator for open-world 4D objects, limit our evaluation to only human objectives. Specifically, we utilize GVHMR (Shen et al. (024) estimate human poses (RRat T and compare them with the input pose sequences

---

Published as a conference paper at ICLR 2025
{RR." T We align the two trajectories at the first frame location. We follow CameraC- trl (He et al. 2024) to estimate the rotation angle error RotErr and translation scale error TransErr, but take the average rather than the sum. 2) Video quality: We leverage standard metrics such as Frechét Video Distance (FVD) (Unterthiner et al. 2018), Frechét Image Distance (FID) (Seitzer 2020), and CLIP Similarity (CLIPSIM) (Wu et al.
2021 to assess the video appearance.
4.4 EVALUATION DATASET
object motions. 2) Entity Description: we use GPT to generate 20 novel human, 52 novel non-human descriptions, and 32 novel locations (refer to Sec, [.33) which randomly assigned poses to form 100 pairs (12 single-entity, 72 two-entity, and 16 three-entity each pair has one human entity).
1) Pose Sequence: We collect 44 novel pose templates, each comprising
4.5 COMPARISON
Granularity Level. As shown in Table 3DTrajMaster can customize object location and orien- tation in 3D space. In contrast, 2D motion representations such points (MotionCtri/Tora) and bounding box (Direct-a- Video), lack awareness of the z dimension. This ambiguity becomes more problematic when handling 3D occlusion. Besides, MotionCtul and Tora integrate multiple entities into single 2D feature, lacking the capability to correlate individual entities with their respec- tive trajectories (see failure case in Fig. (6). When tested on multi-entity input, Direct-a-Viden (a training-free frriingg-eeeeeeee paradigm) shows particularly weak results. Furthermore, 3DsrajMaster allows for di- verse entities and backgrounds (see Fig. and detailed control of entity inputs (see Fig.
Table 1: Fine Control Comparison with Multi-Entity Input.
Location
Orientation
Entity-Trap Corresp.
Learning- based??
Direct-a- Video
(2DD)
MotionCtrl/Tora
V(D)
X
X
/ (not decoupled)
3DTrajMaster (Ours)
(3D)
v (decoupled)
humgon
snimal
natural foree
Location
auiumn forest
coaptal harbor
glacier
forest
Figure 4: Diversity Entity and Background. 3DTrajMaster control versatile entities (hu- man, animal, car, robot, and even abstract natural force), while also generating diverse locations.
Input human entity 'a with short black hair, medium figuregray striped sweater, black Jeans, light brown leather shoes"
result
"shorl black har" "bald head"
"blcck "Hlack_jaam" Jeanu"
"red pants"
1/" add "with backpook"
Figure 5: Fine-grained Editing on Human Entity Input. 3DTrajMaster supports modifications in attributes such as hair, clothing, figure size, and so on. (Please check more in Fig. SIT
Quantative & Qualitative Results. To align with the input requirement of MotionCtrl and Direct- V-didoo project the 3D pose trajectories onto 2D space. For baselines, simplify the entity
8

---

Published as a conference paper at ICLR 2025
Entity a dog with fluffy coat, wagging tall and warm brown for
Entity woman with long straight blaok hair, a blue denim jaoket
Loc.
sunset beach
olty
Projection
Tra].
Traj.
frame
freme
Direct-a- a-Vida
Ours
Direct-a-Videe
man with measy blaok hair, tall frame, a red shirt
woman long wany blonde hair, petite figure, wolf with thlok sllver-gray für, alert golden eyes
Entity
zebra with black and white stripes, shurdy lege
20 Projection
glacler
Traj.
Traj.
frame
Frame
Tora
Direct-a-Videa
Ours
Tora
Direct-a-Video
Ours
Figure 6: Qualitative Comparison on Single/Multiple Entity Motion. 3DTrajMaster outperforms all 2D baselines by modeling 6 DoF entity motion, which can better express the inherent 3D nature of motion. In the last figure, Tora mistakenly regards the background entity as the girl entity.
description, such as changing "a man with messy black hair, tall frame, a red shirt" to "a man" or "a man in red". Otherwise, they may fail to generate videos with detailed descriptions. As shown in Fig. 6. in single entity settings, 3DTrajMaster generates precise entity motion, such as 180 back and continuous inward 90º turn-around. In contrast, Tora and Direct-a-Video produce simpler motions, merely shifting objects from left to right or top-right. In the multi-entity benchmark, Master successfully handles 3D occlusions, such man walking in front of

---

Published as a conference paper at ICLR 2025
Table 2:
Quantative Comparison on Single/Multiple Entity Motion.
MasterMaster performs
better on multiple entity input since the single entity trajectory more complex.
Single Entiyy
Muliiple Entities
Emiites
Methods
TransEer (m) L
RotEr (deg)
TransErr (m) 4 RodEr (deg)
TransEm (m)
RotErr (deg)
Base 1.946
1.799
1.586
1.208
1.629
1.279
MotionCtrl
1.752
2.134
1.682
1.613
1.690
1.675
Tora
1.707
1.158
1.867
1.514
1.848
1,471
Mirect-a-Video
1.632
1.902
1.391
0.942
1.420
1.057
3DstrjMMaster
0.456
0.319
0.390
0.272
0.398
0.277
zebra. Direct-a- Video, however, fails in overlapping regions with mixed man and zebra. We report metric results in T It is not surprising that ours significantly outperforms all baselines.
4.6
ABLATION STUDY
Table 3: Ablation Study on Full Testest and Base T2V Videos (As Reference Video).
Video Quality
3D Trajectory Accuracy
Ablation Setting
FVD
FID 4
CLIPSIM
TransErr (m) RotEr (deg) |
w/ Cross-Attn. Fusion
1673.24
102.13
32.87
0,453
0.341
w/ 3D Self-Attn.
1597.51
98.74
33.15
0,427
0.296
w/o Domain Adaptor
2379.89
157.51
30.50
0,415
0.301
w/o Annealed Sampl.
1841.64
112.57
32,26
0.407
0.265
Full Model
1546.15
96.75
33.77
0.398
0.277
base T2V model
LoRA Scalar a
Annealed Timestep Tg
Figure 7: Ablation Results on Domain Adaptor (upper) and Annealed Sampling (the bottom). We provide more experiments in Sec. F.2. I to choose suitable a and Te to improve video quality.
Improving Video Quality. As illustrated in Fig. and Table 3 without the video domain adaptor, the video quality deteriorates significantly, reverting the training set. Likewise, omitting the annealed sampling strategy results in a decline in video quality (see the beard of the lion and overall scene style). While the rotation accuracy drops slightly (0.277-00.2655, this is acceptable since there exist errors in evaluating open-world human poses.
purely UE-style appearance similar to
Motion Fusion Design. As shown in Table 3 replacing gated self-attention with cross-attention fusion (w/ Cross-Attn. Fusion, here we use the entity-motion bonded feature Pe the query) or placing the object injector after the 3D self-attention layer (w/ 3D Self-Attm.) results in slight decline in both video quality and pose sequence accuracy.
CONCLUSION
In this work, we introduce 3DTrajMaster, a unified framework for controlling multi-entity motions in 3D space, with motion representation as 6DoF location and rotation sequences. Our flexible object injector establishes entity-wise correspondence and allows flexible editing of entity descriptions.
Generalizable entities, like animals, cannot be edited with the same level of granularity as humans. This limitation can be addressed by constructing more diverse and detailed 3D assets of the same category. Currently, the model is constrained to global motion patterns; however, fine- grained local motions (e.g., human dancing or waving hands) and interactions between different entities (e.g.,
Limitation.
picking up dog) also be modeled similarly to our 6 DoF motions with structured motion patterns. At present, our model can only generate limited entities (<3) at a time, but this can be improved with more powerful video foundation models and paired datasets.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jinwen Cao, Yisong Guo, Haowen Ji, Jichao Wang, and Yi Wang from Kuaishou Technol- ogy for their help in constructing our 360--Motion Dataset. As for the fruitful discussion, we thank Yuzhou Huang, Qinghe Wang, Runsen Xu, Zeqi Xiao, and Zhouxia Wang.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774, 2023.
Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
21211-21221, 2023.
Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, al. Te4d: Trajectorycoondttinee text- to-4d generation. arXiv preprint arXiv:2403.17920 2024a.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint 0rXiv:2407.1278/ 2024b.
Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, and Di Zhang. Syncammaster: Synchronizing multi-camera video generation from diverse viewpoints. arXiv preprint, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 0022308..2299 12966, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738. 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. technique report, 2024. URL htttts:// wor ld- ad-siine sit-ssimmltorss
Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open- world game video generation. arXiv preprint arXiv:241 1000669, 2024.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprini 230.v:2310.195/11 2023.
Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, and Yueqi Duan. Dreamcin- ema: Cinematic transfer with free camera and 3d character. arXiv preprint arXiv:2408. 12601, 2024.
Mengyang Feng, Jinlin Liu, Kai Yu, Yuan Yao, Zheng Hui, Xiefan Guo, Xianhui Lin, Haolan Xue, Chen Shi, Xiaowen Li, et al. Dreamoving: human video generation framework based on diffusion models. arXiv e-prints, pp. arXiv-2312, 2023.

---

Published as a conference paper at ICLR 2025
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference Computer Vision, pp. 241-258. Springer, 2025.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 13661. IEEE, 2012.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrk: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933 2023a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint vrXiv:2307.04725, 2023b.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint 1rXiv:2312.06662 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:240..22001, 2024
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:220..12598, 2022.
arXiv preprint
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210002303: 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation. arXiv preprint arXiv:2406. 10126, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2021.
Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-free camera motion transfer for video generation. arXiv preprint arXiv:2404. (tt.:2404..5789, 2024a.
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-worke world videos. arXiv preprint arXiv:2409.0200 2024b.
Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9212-9221, 2024.
Nan Jiang, Zhiyuan Zhang. Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 1737-1747, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565 266677,,6677
2022.
12

---

Published as a conference paper at ICLR 2025
Bingxin Ke, Dominik Narnhofer, Shengyu Huang, Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and Konrad Schindler. Video depth without video models. arXiv preprint arXiv:2411.19189 2024.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Trem- blay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. Megapose: 6d pose estima- tion of novel objects via render & compare. arXiv preprint arXiv:2212.06870, 2022.
Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, and Bo Dai. Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3205-3215, 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1151112252,, 2023b.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-fres motion cloning for controllable video generation. arXiv preprint 2xXv:2406.053338 2024.
Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang. Genod: Generalizable model-free 6-dof object pose estimation from rgb images. In European Conference on Computer Vision, pp. 298-315. Springer, 2022.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851 866, 2023.
Xin Ma, Yaohui Wang, Gengyun Jía, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXhv:2401.03048 2024.
OpenAl. Gpt-4v(ision) system card. OpenAI, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4195-4205, 2023.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. arXiv preprint 21XX::224061666.6 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. part III 18, pp. 234-241, Springer, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 02t.2222220011,, 2022.
Maximilian Seitzer. fitorch-fid: Fid score for pytorch. hutps://githb..com/ fieitzerpyyorrch-fddd 2020.
Zehong Shen, Huajjin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounder human motion recovery via gravity-view coordinates. In SIGGRAPH Asia Conference Proceedings, 2024.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19325, 2021.
Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2Xiv:2010002502 2020.
Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. Onepose: One-shot object pose estimation without cad models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6834,6634, 2022.
13

---

Published as a conference paper at ICLR 2025
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566. 2024a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. Advances in Neural Information Processing Systems, 36, 2024b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024c.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17868-17877 2024.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806 2021.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405.14886 2024.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1481-1490, 2024.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma. Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine- Fine-grained grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.080 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory- oriented diffusion transformer for video generation. arXiv preprint 20X.v:240..21705, 2024.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08440 2023.
14


Reviews:
[{"review_id": "Ut5u502Fcr", "reviewer": "Reviewer_zy5x", "summary": "The paper proposes a 3D-trajectory-conditioned video generation method, fusing prior from pre-trained video diffusion models and from a proposed motion dataset.", "questions": "* Evaluation of multi-entity input sequence sets $N=2$, i.e., 2 entities, based on the qualitative examples. Is the method restricted to a small number of entities, and if so, does the restriction come from training data? If it's tied to training data distribution, it remains even more unclear if the method can potentially apply to generic settings where >>2 objects are moving, which is fairly common in dynamic scenes.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the lack of 6-DoF controllability of existing video generation methods. The method is well-motivated and method designs are clearly explained. The advantage of 6-DoF control over 2D motion control is clearly demonstrated in experiments.", "weaknesses": "* The section on related works discusses prior methods on motion control and motion synthesis tasks, but could also include discussions on techniques for injecting controls to video foundation models, including ControlNet [1] and methods that allow 2D image editing by manipulation attention maps. In particular, ControlNet [1] is currently mentioned but not cited in the paper. \n* The proposed dataset is restricted to human and animal categories, and locations remain to be in cities. Whether it's feasible to scale this method to generic object categories and generic scenes remains an open question.  \n\n[1] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1w7E9QYIUG", "reviewer": "Reviewer_WQpa", "summary": "3DTrajMaster\" introduces a method for controlling multi-entity 3D motion in video generation using 6DoF pose sequences. The authors propose a novel plug-and-play 3D-motion grounded object injector that fuses entity descriptions with corresponding 3D trajectories using a gated self-attention mechanism. They address the lack of suitable training data by constructing a 360°-Motion Dataset, combining collected 3D assets with GPT-generated trajectories. The method is tested against prior 2D motion control approaches and shows state-of-the-art performance in both motion control accuracy and generalization.", "questions": "Will the dataset and code will be public after acceptance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed 3D-motion grounded object injector, combining 6DoF pose sequences with entity descriptions, is an innovative contribution that extends beyond 2D control limitations.\n\n**Dataset Creation**: The construction of the 360°-Motion Dataset addresses a notable gap in available training data, particularly for multi-entity scenarios, using an innovative combination of GPT and UE.\n\n**Flexibility**: The plug-and-play nature of the proposed object injector facilitates broader applicability across different generative models, with the gated self-attention mechanism ensuring entity-specific trajectory adherence.", "weaknesses": "**Dataset Limitation**: The reliance on synthetic data and a limited number of assets may hinder real-world generalization. The \"city\" setting constraint for the dataset also limits the diversity of possible outputs.\n\n**Generalizability**: The model's performance for generalized 3D scenes beyond those captured in the MatrixCity platform remains unclear. More evaluation of real-world, diverse datasets would strengthen the contributions.\n\n**Evaluation Scope**: While evaluation metrics like FVD and CLIP Similarity are used, the lack of real-world evaluations or qualitative feedback from human users makes it hard to gauge practical effectiveness fully. Also, the author could consider comparing with recent 4D generation methods such as TC4D which also can control trajectory.\n\nI will also check other reviewers's feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7RHOk1bOO", "reviewer": "Reviewer_2Hag", "summary": "1. This paper aims to control the entities’ motion with 3D control signals in video generation.  3D control signals are a more natural representation compared to 2D signals as the motion is in 3D space. \n2. This paper proposed a plug-and-play module to integrate the entities with their respective 3D trajectories into a pre-trained video generative model to control the 3D motion. \n3. To avoid video quality degradation during the fine-tuning, they use a Lora-like domain adaptor and an annealed sampling strategy.\n4. They construct a synthetic dataset collecting dynamic 3D human and animal assets with ground-truth 3D motion for training.\n5. Experiments show that the proposed methods can achieve state-of-the-art performance in 3D motion control.", "questions": "In Table 2, could you clarify the distinction between “Multiple Entities” and “All Entities”? Does “All Entities” include both “single” and “multiple” cases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is the first to control entities’ motion with 3D trajectories in video generation. The task is novel and reasonable as 3D control signals can fully express the inherent 3D nature of motion and offer better controllability in video generation compared to 2D control signals.\n2. The method design is clear and reasonable.\n3. The paper constructs a new synthetic dataset for this task. The dataset potentially benefits the following video generation with 3D entity control. \n4. The experiments are thorough and solid. Plenty of visualization results as well as the videos in the supplementary demonstrate the effectiveness and generability of the proposed method.\n5. The paper is well-written and easy to follow.", "weaknesses": "1. The dataset lacks diversity in terms of background and motion types. The setting is restricted to a \"City\" environment (as noted in the paper's Limitations section), and the actions are primarily limited to walking. Consequently, models trained on this dataset are also constrained in their generalizability.\n2. Foot skating/floating issues are prevalent in the dataset. This appears to result from inconsistencies between the relative motion and global motion of the dynamic entities, which could negatively impact model training by introducing artifacts.\n\nMinor:\n\n1. The explanation of the \"ControlNet-like architecture\" is vague and lacks clarity. The paper references this term in Lines 20 and 115, suggesting it pertains to the Object Injector, whose initial weights stem from the 2D spatial self-attention layer in the video generative model. However, this does not align with a true ControlNet-like module, which would typically function as a parallel module with zero-initialized layers designed to adjust the original features. Instead, the Object Injector is positioned after the 2D spatial self-attention layer, differing fundamentally from ControlNet-like behavior.\n2. It would have been better to elaborate on Line 334: “This phenomenon reflects the model’s ability to learn 3D motion representations”? The reasoning behind this statement isn’t entirely clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRuIKAr56P", "reviewer": "Reviewer_95Hu", "summary": "Paper tackles the problem of multi-entity 3D control signals for video generation\n\nApproach works by constructing an Unreal Engine dataset of assets with GPT4V generated trajectories and various camera angles, and then LoRAing it to do domain adaptation to prevent it from looking too much like the UE assets.\n\nExtensive qualitative results indicate the method clearly works, and quantitative results indicate value of components + favorable results compared to several baselines.", "questions": "- Why does the alligator gait look so much worse than the other animals? Its legs seem to be almost static in some scenes.\n - It seems that the proprietary base video generation model is extremely strong (congrats!). Do you think this recipe will work with less strong video generation models, such as current open source models?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method clearly works. The supplemental provides extensive videos paired with prompts showcasing multiple walking agents. While generations are imperfect, I was shocked at the quality -- the assets seem to properly interact with light in the scenes, including full long shadows, as well as the terrain below, such as water. \n\nThe method used to achieve this adaptation seems straightfoward, and the recipe seems generally replicable and applicable to other video generation models despite the fact that the generation model used is proprietary.", "weaknesses": "- Pose evals are human only; however, I think this is well motivated given the stated lack of general video pose predictors.\n\nQuite honestly, I am not an expert in this domain and I lack the background to provide meaningful criticism. The results look good, the actors clearly follow the given trajectories, and the recipe given to achieve this generally makes sense and feels general enough to be useful for arbitrary video generation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)