Trajectory attention for fine-grained video motion control
Authors: Zeqi Xiao
Wenqi Ouyang
Yifan Zhou
Shuai Yang
Lei Yang
Jianlou Si
Xingang Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Trajectory attention
video generation
motion control
Primary area: generative models

Abstract: Recent advancements in video generation have been greatly driven by video diffusion models, with camera motion control emerging as a crucial challenge in creating view-customized visual content. This paper introduces trajectory attention, a novel approach that performs attention along available pixel trajectories for fine-grained camera motion control. Unlike existing methods that often yield imprecise outputs or neglect temporal correlations, our approach possesses a stronger inductive bias that seamlessly injects trajectory information into the video generation process. Importantly, our approach models trajectory attention as an auxiliary branch alongside traditional temporal attention. This design enables the original temporal attention and the trajectory attention to work in synergy, ensuring both
precise motion control and new content generation capability, which is critical when the trajectory is only partially available. Experiments on camera motion control for images and videos demonstrate significant improvements in precision and long-range consistency while maintaining high-quality generation. Furthermore, we show that our approach can be extended to other video motion control tasks, such as first-frame-guided video editing, where it excels in maintaining content consistency over large spatial and temporal ranges.

Published as a conference paper at ICLR 2025
TRAJECTORY ATTENTION FOR FINE-GRAINED VIDEO MOTION CONTROL
Zeqi Xiao', Wenqi Ouyang', Yifan Zhou', Shuai Yang". Lei Yang'. Jianlou Si², Xingang Pan' L--,a, Nanyang Technological University,
Wangxuan Institute of Computer Technology, Peking University
TimeeTime Research
{zeqi001, yifanC06, munginouyang, xingang. pan)eaaunnnntuuu edu. sg
wi milliamyang@pku.. edu. en (jianlousi, ( imeeetommccoooom
ABSTRACT
Recent advancements in video generation have been greatly driven by video dif- fusion models, with camera motion control emerging creating view-customized visual content. This paper introduces trajectory atten- tion, novel approach that performs attention along available pixel trajectories for fine-grained camera motion control. Unlike existing methods that often yield im- precise outputs neglect temporal correlations, our approach possesses stronger inductive bias that seamlessly injects trajectory information into the video genera- tion process. Importantly, our approach models trajectory attention an auxiliary branch alongside traditional temporal attention. This design enables the original temporal attention and the trajectory attention to work in synergy, ensuring both precise motion control and new content generation capability, which is critical when the trajectory is only partially available. Experiments on camera motion control for images and videos demonstrate significant improvements in precision and long-range consistency while maintaining high-quality generation. Further- more, show that our approach can be extended to other video motion control tasks, such as first- fras--rrm-- video editing, where excels in maintaining con- tent consistency over large spatial and temporal ranges.
crucial challenge in
INTRODUCTION
Video generation has experienced remarkable advancements in recent years, driven by sophisticated deep learning models such as video diffusion models and temporal attention mechanisms (OpenAl 2024 Chen et al. 2024] Wang et al., 2023a Guo et al. 2023b). These innovations have enabled the synthesis of increasingly realistic videos, fueling fields in areas such as filmmaking (Zhao al. 2023 Zhuang et al. 2024) and world modeling (OpenAI 2024 Valevski et al., 2024). Video motion control, which aims to produce customized motion in video generation, has emerged
crucial aspect (Yang et al. 2023b Ling et al., 2024 Ouyang et al. 2024 Ku et al., 2024 Zhao et af. 2023).
Among various control signals, camera motion control has garnered increasing attention due to its wide applications in creating view-customized visual content. However, effectively conditioning generation results on given camera trajectories remains non-trivial. Researchers have explored sev- eral approaches to address this challenge. One method involves encoding camera parameters into embeddings and injecting them into the model via cross-attention or addition (Wang al.) 2024c He et al. 2024 Bahmmni et al., 2024). While straightforward, this approach often yields imprecise and amb uunbiguuu outputs due to the high-level constraints and implicit control mechanisms it employs. Another strategy involves rendering partial frames based on camera trajectories and using these ei- ther as direct input (Hu et al.] 2024 Yu et al., 2024) or as optimization targets (You et al.) (024) for trame-wise conditioning, Although this method provides more explicit control, it often neglects temporal correlations across frames, leading to inconsistencies in the generated sequence.
Project page this URL

---

Published as a conference paper at ICLR 2025
Camera motion control on images
Camera motion control on videos
First-frame featfrrrm-quiedd video editing
Figure 1: Trajectory attention injects partial motion information by making content along trajecto- ries consistent. facilitates various tasks such as camera motion control on images and videos, and first-rame--uidde video editing. Yellow boxes indicate reference contents. Green boxes indicate input frames. Blue boxes indicate output frames.
In response to these limitations, recent methods have begun to address temporal relationships by leveraging 3D inductive biases (Xu et al. 2024 et. 2024). These approaches focus on narrowed domains, utilizing specific settings such row-wise attention (Li et al. (002)) or epipolar constraint attention (Xu et al. 2024). As we consider the trajectory of a camera moving around scenes, becomes apparent that certain parts of the moving trajectories of pixels, represented as a sequence of 2D coordinates across frames, are predictable due to 3D consistency constraints. This observation raises intriguing question:
exploit such trajectories strong inductive bias achieve more fine- mmne-grined motion control?
Revisiting the temporal attention mechanism, which is central to video models for synthesizing dynamic motions with consistent content, we can view the dynamics as pixel trajectories across frames. The temporal attention mechanism, with its generic attention design, functions by implicitly synthesizing and attending to these trajectories. Building on this observation, when parts of the trajectories available, the attention along these trajectories can be modeled explicitly as strong inductive bias to produce controlled motion with consistent content.
To this end, we propose trajectory attention that performs attention along the available trajectories across frames for fine-grained camera motion control. Instead of directly adapting the temporal attention to operate on trajectories, which yields suboptimal results in practice, we model trajectory attention as an auxiliary branch alongside the original temporal attention. This design is critical due to the distinct goals of these two attention mechanisms. Temporal attention, which must balance motion synthesis and content consistency, typically focuses short-range dynamics and attends to adjacent frames within local window. In contrast, trajectory attention designed to ensure long- range consistency across features along trajectory (see Fig. 2 The trajectory attention branch can inherit the parameters of the original temporal attention for efficient tuning, and its output is added the output of temporal attention residuals. This whole design offers several merits: 1) allows better division of tasks: trajectory attention manages motion control and ensures long- range consistency along specified paths, while temporal attention synthesizes motion for the rest regions; 2) it can integrate seamlessly without modifying the original parameters; 3) it supports sparse trajectories, the condition injected moderately, meaning available trajectories do have to cover all pixels.
Our experiments on camera motion control for images and videos demonstrate that our designs significantly enhance precision and long-range consistency. As shown in Fig. [ our approach lever- ages a stronger inductive bias that optimizes the attention mechanism. This results in improved control precision while maintaining high-quality generation. The proposed trajectory attention can be extended to other video motion control tasks, such as cest-frame-guides video editing. Existing techniques often struggle to maintain content consistency over large spatial and temporal ranges (Ku et al. 2024 Ouyang et al.) 2024). In contrast, our method's ability to model long-range, con- sistent correspondences achieves promising results in these challenging scenarios. Moreover, the efficiency of our design allows for training with limited data and computational resources, making it generalizable diverse contexts and frame ranges.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Video Diffusion Models. The field of video generation has seen significant advancements in recent
years, especially in the of video diffusion models (Ho et al., 2022 Guo et al. 2023b
Chen
et al.
2023a Wang et al. 2023b a OpenAl 2024 Blattmann et al. 2023 Guo et al. 2023a
Chen
al.
2024 Hong al. (022).
The core of motion modeling of video diffusion models is the temporal attention module. Some approaches (Guo al., 2023b; Chen et al. 2023a Wang al., 2023b a) decompose attention into spatial and temporal components, where temporal attention aligns features across different frames. Others (Yang al., 2024b OpenAl] 2024 Lab etc. 2024) integrate spatial and temporal attention into a unified mechanism, capturing both types of information simultaneously. While these methods rely on data-driven techniques to implicitly learn dynamic video priors within the attention mecha- nism, how to leverage such priors for explicit and precise motion control remains under-explored.
Motion Control in Video Generation. Prior works have explored various control signals for video motion control (Guo et al.] [2024] Niu et al. [2024] Yu et al.
[0233] CCen et al. [2023b] Yang et al.)
2024a] Zhao et al. | Chen et al. [203cc] Zhang et al. 2023b)) including sketches (Wang et al. 2024b), depth maps
2024a Zuo et al. [2024] Zhu et al.
Wang et al. (0024b), drag vectors Yin ct al.
2023 Teng et al.] 2023] Deng et al. (003)). human
pose (Zhang et al. 2024
Zhu et al. 2024b),
object trajectory (Qiu et al. 2024 Wang et al.
2024a
Wu et al. 2024
Gu et al.
2024), and features
extracted from reference videos (Yatim et al.
2023 Xiao et al. 2024 Yang et al. 2023b
Ouyang
al. 2024
Ku et al. 2024).
One important branch of video motion control is camera motion control, also known as novel view synthesis. In this regard, |Wann et al. (2024c); HH al.) (2024); |Bahmani al. (2024) Wu et al. (2024) utilize high-level condition signals by encoding camera pose parameters into conditional features. However, these methods often lack precision in capturing detailed temporal dynamics, as they impose weak constraints on the resulting motion. Hoo et al. (2024) enables camera control by rendering incomplete warped views followed by re-denoising, Mülller al. (2024); Yu et al. (2024); You et al. (2024) render partial videos as guidance and leverage video generation models to inpaint the remaining frames. Despite these innovations, their approaches suffer from temporal inconsis- tency due to the lack of consideration for sequential coherence. Methods such as those proposed by (hi et al. )204);; Xu et al. (2024); [Cong et al. (2023); Kuang et al. (2024) explicitly modify attention using optical flow epipolar constraints. These solutions can be viewed as a weaker vari- constraint. Our approach introduces a trajectory attention mechanism
ant of for motion information injection. Such mechanism imposes strong inductive bias the tem- poral dimension, as also explored by /atatrick et al. (2021) for video recognition. By leveraging the attention mechanism, our method affords precise control over video generation and improves effi- ciency-as all without requiring specially annotated datasets, such as camera pose annotations. This approach enhances motion control throughout the generation process while preserving the fidelity of temporal dynamics.
3 METHODOLOGY
This section introduces trajectory attention for fine-grained motion control. We first outline video diffusion models with focus on temporal attention (Sec. B.I). then adapt it for trajectory attention and discuss its limitations (Sec. 3.2). We present trajectory attention as an additional branch, with visualizations of its effectiveness (Sec. 3.3 and describe an efficient training pipeline (Sec. 3.4
3.1 PRELIMINARY
The core of vidco motion modeling lies in the temporal attention mechanism within video diffusion models, whether applied through decomposed spatial and temporal attention or full 3D attention, to capture robust motion priors. This paper demonstrates the decomposed setting, which is more widely used and has greater open-source availability. However, our design is also adaptable to full 3D attention, will shown in the experimental results and appendix.
A typical video diffusion architecture for decomposed tional layers, spatial attention blocks, and temporal attention blocks. The temporal attention operates
attention includes convolu-
3

---

Published as a conference paper at ICLR 2025
Prame
Prame
Frame
Frame
Atlenion map of temporal attention
(b) A Atentoo map of trajectory attention
Figure 2: Attention map visualization of temporal attention and trajectory attention. (a) Tem- poral attention tends to concentrate its weight on a narrow, adjacent frame window. (b) In contrast, trajectory attention exhibits a broader attention window, highlighting its capacity to produce more consistent controllable results. Here, the attention map structured with the frame number as the side length. The attention weights normalized within the range of 0 to 1, where higher values (indicated by light yellow) represent stronger attention.
Tempari
Pazz Iqei
Spaiil
Trijettry
ADasia
Milt: hnad bintion
Pronet Back
Hobio Stizs
Hakiiz SuRcs Faclac
Upland Hidlen Stute PScTe/Mis
Triekar
Tog lackas
Figure 3: Overview of the proposed motion control pipeline. Our method allows for conditioning on trajectories from various sources such as camera motion derived from single image, as shown this figure. We inject these conditions into the model through trajectory attention, enabling explicit and fineggriieed control over the motion in the generated video.
as follows. Given an input latent feature Z E RF where F, H, W, and represent number of frames, height, width, and channels, respectively, temporal attention operates along the frame dimension. The feature first projected into query (Q), key (K), and value (V):
Q (Z)). K Pa (Z), V (Z.).
where Pg. Pk, and P. are learnable projection functions. Temporal attention is then applied along the frame dimension as:
Z' Softmax (QK"))
(
(2)
yielding the output latent feature Z'. For simplicity, we omit the details like rescaling factor and multi-head operations. With large-scale training, temporal attention effectively captures dynamic and consistent video motions, making it a natural candidate for motion control in video models.
3.2 TAMING TEMPORAL ATTENTION FOR TRAJECTORY ATTENTION
As shown in Fig. vanilla temporal attention operates on the same spatial position across different frames, where the coordinates in the attention form predefined trajectories across frames.
Since temporal attention has already learned to model motion along pre-defined trajectories, a natu- ral extension is to tame temporal attention for additional trajectory attention. For example, given set of trajectories Tr, where each trajectory is represented by a series of coordinates, we incorporate them into the temporal attention mechanism.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Trajectory-basse sampling
Input:
Hidden states Z E IRx where F is the number of frames, H. W are the spatial dimensions, and the number of channels. trajectories Tr E R where each trajectory specifies F 2D locations. Trajectory masks M E IR PxL, where r E {0, 1 indicates whether a trajectory is valid at frame for trajectory I. foreach trajectory = 1, ,.... do
Sample hidden states Z; = ( vr.a) 1...., F} E R FxC where (z V5,a) are the 2D coordinates from Tr[i] for each frame f.
.end
Stack sampled hidden states: Z, Stack(Z, i-
RPXIXC
Mask out invalid hidden states using M: Z, Z, GM
Output: Masked sampled hidden states Z, E FxLxC
However, this straighwarpward adaptation often yields suboptimal results due to conflict between temporal and trajectory attention. Temporal attention designed to ensure consistency along the trajectory while preserving the dynamism of feature representations. However, achieving both per- fectly is challenging. Consequently, temporal attention often prioritizes natural dynamics at the expense of long-range consistency. This is evident in the attention statisties: as shown in Fig. 2aa), the learned temporal attention predominantly focuses on adjacent frames.
In contrast, trajectory attention, given its known dynam- ies, aims solely to align features along the trajectory. This singular focus on alignment often clashes with the broader objectives of temporal attention. Simply adapt- ing temporal attention to accommodate trajectory infor- mation can therefore introduce conflicts. Experimental results further demonstrate that, even with extensive train- ing, the quality of motion control remains suboptimal when trajectory attention naively integrated.
W
(a) Temporal attention
(b) Trajectory attention
Figure 4: Visualization of vanilla tem- poral attention and trajectory atten- tion.
3.3 MODELING TRAJECTORY ATTENTION AS AN AUXILIARY BRANCH
The above analysis reveals that temporal attention and trajectory attention should not share the same set of weights. Inspired by the recent success of Zhang et al. (2023a), we model temporal attention and trajectory attention into a two-branch structure, where trajectory attention responsible for injecting fine-grained trajectory consistent signal the origin generation process.
As illustrated in Fig. trajectory attention and temporal attention share the same structure, as well identical input and output shapes. The key difference lies in the process: trajectories to sample features from the hidden states (Algorithm [ then apply multi-head attention with distinct parameters, and finally project the results back to the hidden state format after frame- wise attention (Algorithm 2)
first the given
To validate the purpose distinction, compare the attention maps (softmax scores along the frame axis) of temporal and trajectory attention, based on the SVD model (Blattmann et al., (003).. As shown in Fig. aa and (b), trajectory attention clearly provides broader attention window, enabling more consistent and controllable results.
3.4 TRAINING TRAJECTORY ATTENTION EFFICIENTLY
As illustrated in Fig. initialize the weights of the QKV projectors with those from temporal attention layers to harness the motion modeling capabilities learned from large-scale data. Addition- ally, the output projector is initialized with zero weights ensure gradual training process.
The training objective follows the standard approach used fundamental generation models. For instance, in the case of Stable Video Diffusion (Blattmann al. 2023), the objective is:
5

---

Published as a conference paper at ICLR 2025
Algorithm 2: Back projection
Input:
Hidden states after attention Zi RFLXXC L trajectories Tr e RLXFx2 Trajectory masks M E FxL
Initialize:
Zp E RXXXXXXX, U Zp =-=0, 0, U
where H and W are the height and width of the spatial grid.
foreach i == I.....L. do
Add [ E R RFCC
Zp at locations ( gfs) from Tr|i]: (, J 15, -
Update count table U at the same locations: (/ M
end
Normalize Zp element-wise for valid positions (U 0):
xx, for all (,,z,) where u(f > 0 U( x.
2,(,,,,y,;)
Output: Back paoj-proeected hidden states Zp E PxHxxx
Copy Weights
Zero Init
Projector
Q Projector
Mulli- Head
Mulli-
Out Projector
Out Projector
K Projector
Projector
ttention
ttention
Projector
Projector
Temporal ttention
Trajectory A Attention
Figure 5: Training strategy for trajectory attention. To leverage the motion modeling capability learned from large-scale data, we initialize the weights of the QKV projectors with those from temporal attention layers. Additionally, the output projector is initialized with zero weights to ensure a smooth and gradual training process.
E Do(x + +n; -xu xulle).
(3)
where Dg represents the neural network, No denotes the latent features of the target vidcos, n is the noise, is the condition signal, and is the variance parameter.
4 FINE-GRAINED CONTROL OF VIDEO GENERATION
This section delves into the process of extracting trajectories for different task settings. While our primary focus is on camera motion control for both static images and dynamic video content, we also showcase the process of trajectory extraction for video editing.
4.1 CAMERA MOTION CONTROL ON IMAGES
Algorithm 3 outlines the process of extracting trajectories, denoted Tr, along with the corre- sponding validity mask M from a single image. Unlike prior approaches (Wang et al., [024cc He al. 2024), which rely high-level control signals for video manipulation, our method explicitly models camera motion as trajectories across frames. This enables precise and accurate control of camera movement.
4.2 CAMERA MOTION CONTROL ON VIDEOS
The process for camera motion control on videos is more complex than the process for images since the video itself has its motion. We need to extract the original motion with point trajectory estimation methods like Karaev et al. 2023). then combine the original motion with camera motion get the final trajectories. We show the details in Algorithm

---

Published as a conference paper at ICLR 2025
Algorithm 3: Trajectory extraction from single image
Input: Image I E HR xx33, A set of camera pose with intrinsic and extrinsic parameters, (K E Rx33] and {E) t]], where R E R3x3 representations the rotation part of the extrinsic parameters, and t E R3x1 is the translation part. The length of the camera pose equals frame number F. Hp and W are the height and width of the pixel
space
Estimate the depth map D E R 7 from I given camera pose parameters.
Get the translation of pixels T E Fx based on using using D, K, and E. Get trajecories Tr = + C, where C E R is pixel-level grid coordinates of image with shape Hp x Wp.
Get valid trajectory mask M for pixels that within the image space.
Output: Trajectories Tr, Trajectory Masks M
Algorithm 4: Trajectory extraction from video
Input: Video Frames V R Fx,,x W A set of camera pose with intrinsic and extrinsic parameters. {K E 3x33 and EER;t]]: The lenght of camera pose equals to frame number F
Estimate the depth map D E R BFXHHx from given camera pose parameters.
Estimate point trajecotries P E 2xxxx2 and the corresponding occlusion masks M.. Get the translation of pixels T E IR H, M,,,xx using D, K and E.
Sample the translation of point trajectories P. E FxL22 from T using P.
Get trajecories Tr = P.- + P. Get valid trajectory mask M = A,
A M,, where M, is for pixels that within the image space. Output: Trajectories Tr, Trajectory Masks M
4.3
VIDEO EDITING
Video editing based on an edited first frame has gained popularity recently (Ouyang et al. 2024 Ku a 2024). The goal is to generate videos where the content of the first frame aligns with the edited version while inheriting motion from reference videos. Our method is well-suited for this task, as we leverage Image-to- tidiieeo generation models that use the edited first frame as a conditioning input while incorporating trajectories extracted from the original videos to guide the motion.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets. We MiraData (Ju al. 2024 for training, large-scale video dataset with long- duration videos and structured captions, featuring realistic and dynamic scenes from games or daily life. We sample short video clips and apply Yang al. (023a) extract optical flow as trajectory guidance. In total, we train with 10k video clips.
Implementation Details. We conducted our main experiments using SVD (Blattmann al. 2023 employing the Adam optimizer with learning rate of le-5 per batch size, with mixed precision training fp16. We only fine-tune the additional trajectory attention modules which inherit weights from the temporal modules. Our efficient training design allows for approximately 24 GPU hours of training (with batch size of on a single A 100 GPU over the course of one day). We train trajectory attention the 12-frame video generation modules and apply the learned trajectory attention to both 12- frame and 25-frame video generation models. Despite being trained on 12-frame videos, the trajectory attention performs effectively when integrated into the 25-frame model, demonstrating the strong generalization capability of our design.
Metrics. We assessed the conditional generation performance using four distinct metrics: (1) Abso- lute Trajectory Error (ATE) (Goel et al. 1999). which quantifies the deviation between the estimated and actual trajectories of camera robot; and (2) Relative Pose Error (RPE) (Goel et al. 1999...
7

---

Published as a conference paper at ICLR 2025
Table 1: Qualitative comparison image camera motion control. *, Motion12V AnimateDifft
Guo et al., 2023b) while we use SVD (Blattmann et al. 2023 as the base models. Other methods use SVD as default.
Setting
Methods
ATE (m,
RPE trans (m, 4)
RPE Rot (deg,
FID (L)
MotionCtrl
1.2151
0.5213
1.8372
101.3
frames
Ours
0.0212
0.0221
0.1151
104.2
MotionI2V
0.0712
0.0471
0.2853
124.1
16 frames
Ours
0.0413
0.0241
0.1231
108.7
CameraCtrl
0.0411
0.0268
0.3480
115.8
NVS Solver
0.1216
0.0558
0.4785
108.5
25 frames
Ours
0.0396
0.0232
0.1939
103.5
NVSSubbe Fiod
Figure 6: Qualitative comparisons for camera motion control on images. While other meth- ods often exhibit significant quality degradation inconsistencies in camera motion, our approach consistently delivers high-quality results with precise, fine-grained control over camera movements. Regions are highlighted in yellow boxes to reveal camera motion. For a more comprehensive under- standing, we highly recommend viewing the accompanying videos in the supplementary materials.
which captures the drift in the estimated pose by separately calculating the translation (RPE-T) T) and rotation (RPE-R) errors. (3) Fréchet Inception Distance (FID) (Heusel al. 2017), which evaluates the quality and variability of the generated views.
5.2 CAMERA MOTION CONTROL ON SINGLE IMAGES
We compare the results of camera motion control single images with the methods proposed by
Wang et al. ) (Shi et al. (2024); He et al. (2024). The evaluation is based on 230 combinations of diverse scenes and camera trajectories. To ensure a fair comparison, our model tested under varying settings due to the frame limitations of certain models (i.e., [Wang et al. 2024c only releases
a 12-frame version).
Table summarizes the results, showing that our methods consistently achieve higher comparable control precision in terms of ATE and RPF, along with strong fidelity as measured by FID, compared
to other methods (Wang al. 2024c Shi al. 2024 He al. 2024 You et al. 2024). Although MotionCurl (Wang et al., 2024c) generates slightly better results in terms of FID. it compromises significantly control precision. Motion-12V Shi et al. (2024). which uses flow-based attention, only allows frames to attend to the first frame, leading to quality issues in some cases. In contrast, approach maintains better control precision while preserving generation quality. It also performs better over longer time ranges than other recent methods (He et al. 2024: You et al. 2024).
We further provide qualitative results in Fig. 6 which is aligned with the conclusions in Table
8

---

Published as a conference paper at ICLR 2025
Table 2: Qualitative comparison video camera motion control.
Methods
ATE (m, 4)
RPE trans (m, })
RPE Rot (deg, 4)
FID (1)
NVS_Solver
0,5112
0.3442
1.3241
134.5
Ours
0.3572
0.1981
0.7889
129.3
Ours (w. NVS Nolver)
0.3371
0.1972
0.6241
112.2
NVS Solver Refermon Ongin
Oaas
Figure 7: Qualitative comparisons for camera motion control on videos. In the second row, provide video frames after view warping reference. Methods like NVS Solver (You al. 2024 use frame-wise information injection but overlook temporal continuity, leading to inconsistent motion control, especially frames farther from the first one. In contrast, our approach explicitly models attention across frames, which significantly benefits control precision. We highlight the control precision with yellow boxes, where our method aligns better with the reference. we integrate NVS Solver's capability to inject frame-wise information, achieving better video alignment with the original videos.
5.3 CAMERA MOTION CONTROL ON VIDEOS
We compare the video synthesis performance of our method with You et al. (2024), who employ test-time optimization approach. Their method uses view-warped frames optimization targets, injecting partial frame information into the generation process. However, it optimizes on a per-frame basis, neglecting temporal coherence. As result, when large view changes occur, their method often struggles to follow the motion accurately and introduces spatial blur. In contrast, our method precisely handles large motions. Notably, the way You al. (2024) injects frame information is orthogonal to our approach. By combining their optimization technique with our trajectory attention, we achieve higher fidelity in the generated results, as demonstrated in Table 2 and Fig. 7
5.4 VIDEO EDITING
Compared to previous first-frame guided editing methods (Ku al. 2024 Ouyang et al. 2024 our approach explicitly models motion dynamics as trajectories across frames, enabling better con- tent consistency over large spatial and temporal ranges. As shown in Fig. & while other methods struggle to maintain consistency after editing, our method successfully preserves the edited features throughout the entire sequence.
5.5 ABLATION ON TRAJECTORY ATTENTION DESIGNS
To validate the effectiveness of trajectory attention design, conducted ablation study, presented in Table 3 We examined four types of implementations: 1) Directly applying tempo- ral attention trajectory attention, 2) Integrating trajectory attention into temporal attention with weight tine-tuning, 3) Utilizing an add-on branch for modeling trajectory attention, and 4) Inheriting weights from temporal attention (as illustrated in Fig. (5)
The results Table 3 indicate that the vanilla adaptation leads to significantly poor motion tracking and video quality, with some outputs exhibiting complete noise (we omit such invalid results during

---

Published as a conference paper at ICLR 2025
Oligin
Edit
Any
12VE::
Ouns
Figure 8: Results on first-frame guided video editing. We compare our method with those from Ouyang et al. (2024); Ku et al. (2024). The results show that other methods struggle to maintain consistency after editing. In contrast, our method successfully preserves the edited features across frames, thanks to its ability to model trajectory consistency throughout the video.
Table Ablation on trajectory attention design.
Methods
ATE (m, 4)
RPE trans (m, 4)
RPE Rot (deg. 4)
FID (L)
Vanilla
1.7812
2.4258
13.2141
329.6
Tuning
0.3147
0.3169
1.5364
139.2
+ Add-on Branch
0.0724
0.1274
0.3824
112.4
+ Weight Inheriting
0.0396
0.0232
0.1939
103.5
evaluation, otherwise calculating the statistic results is not feasible.). After fine-tuning the temporal weights, the implementation functions better but remains suboptimal. In contrast, using an add-on branch for trajectory attention markedly improves both motion control precision and video quality. Additionally, inheriting weights from temporal attention facilitates faster convergence and better overall performance compared to simply initializing attention weights randomly.
An extreme close-up of an gray- man with beard...
Drone view of waves crushing against the rugged clill's
An drone view of seaside, rock
An drone view of a beautiful village
Figure 9:
Qualitative results on Open-Sora-Plan.. Plaa & etc. 2024) By incorporating trajectory
attention into the 3D attention module, we successfully enable camera motion control.
5.6 RESULTS FULL ATTENTION MODELS.
Our method also has the potential support full 3D attention using similar pipeline shown in Fig. 3 and Fig. 5 with the key difference being the application of trajectory attention to the attention module instead of the temporal attention. As demonstrated in Fig. this enables diverse camera motion control in the generated results. For detailed implementation, please refer to the supplementary materials.
CONCLUSION
In conclusion, we introduced trajectory attention, a novel approach for fine-grained camera mo- tion control in video generation. Our method, which models trajectory attention as an auxiliary branch alongside temporal attention, demonstrates significant improvements in precision and long- range consistency. Experiments show its effectiveness in camera motion control for both images and videos while maintaining high-quality generation. The approach's 'ss extensibility other video motion control tasks, such as fust-frram--ee -uiided video editing, highlights its potential impact on the broader field of video generation and editing.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This research is supported by MOE AcRF Tier (RG97/23) and NTU SUG- NAP. This research is also supported under the RIE2020 Industry Alignment Fund Industry Col- laboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

---

Published as a conference paper at ICLR 2025
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hisi-Yiing Lee, Chaoyang Wang. Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. EXXiv:2407.2278.1 2024.
arXiv preprint
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310. 19512, 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. ar preprint arXiv:2401.0900 2024.
Shai-Shien Chen, Chich Hubert Lin, Hung- Yu Tseng, Tsung Yi Lin, and Ming-Hsuan Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404, 2023b.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023c.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint arXiv:2310.05922 2023.
Yufan Deng, Ruida Wang, Yuhao Zhang, Yu-Wing Tai, and Chi-Keung Tang. Dragvideo: Interactive drag-style video editing. arXiv preprint arXiv:2312.02216, 2023.
Puneet Goel, Stergios Roumeliotis, and Gaurav S Sukhatme. Robust localization using relative and absolute position estimates. In Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No. 99CH36289), volume 2, pp. 1134-114.. IEEE, 1999.
Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjic Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video sub- ject swapping with interactive semantic point correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7621-7630 2024.
Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint w33iv:2407003168. 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint 1.1. 2023a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023b.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15866, 2022.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation. arXiv preprint arXiv:2406.101) 10126, 2024.
Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training -ree camera motion transfer for video generation. 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.cm 006655,, 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07633 2023.
Max Ku, Cong Wei, Weiming Ren, Huan Yang. and Wenhu Chen. Anyv2v: A plug-and- frame- work for any video-to-video editing tasks. arXiv preprint arXiv:2403. 14468, 2024.
Zhengfei Kuang. Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. 2024.
arXiv preprint arXiv
PKU- Yuan Lab and Tuzhan AI etc. Open-sora-plan, April 2024. URL http::///o..ooo
5281/zenodo 10948109
Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, et al. Era3d: High-resolution multiview diffusion using efficient row-wise attention. arXiv preprint arXiv:2405.11616 2024.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-fres frren motion cloning for controllable video generation. arXiv preprint arXiw:2406.05338, 2024.
Norman Müller, Katja Schwarz, Barbara Rössle, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Multidiff: Consistent novel view synthesis from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
10258-10268, 2024.
Muyao Niu, Xiaodong Cun, Xintao Wang. Yong Zhang, Ying Shan, and Yinqiang Zheng, Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model. arXiv preprint 2tli:2400..02222 2024.
OpenAl. Video generation models as world simulators. https: http:///ppaaa..... com/ research/ video-geneeation--nooo.
as wwrld- www.ldddssmlatoor.....m 2024.
Wenqi Ouyang, Yi Dong, Lei Yang. Jianlou Si, and Xingang Pan. 12vedit: -rrt--rram-guuieed video editing via image-to-video diffusion models. arXiv preprint 1XXi:22455.16537 2024.
Mandela Patrick, Dylan Camphell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34: 14:12493-2 2021.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. arXiv preprint XrXiv:2406.16888 2024.
Xiaoyu Shi, Zhaoyang Huang, Fu- Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
13

---

Published as a conference paper at ICLR 2025
Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. Drag-a-video: Non-rigid video editing with point- based interaction. arXiv preprint arXiv:2312.02936, 2023.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, 2024. URL httt::ht:://// /ttp:///aavv..v.. wwt.:bs//////oo/oooo httpp/////aaai.ooo//////////////
118377
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang
Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402001566, 2024a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint (XXixv2330.06577 2023a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang. Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou Videocomposer: Compositional video synthesis with motion con- trollability. Advances in Neural Information Processing Systems, 36, 2024b.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiging Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024c.
Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware customized text-to-video generation. NeurIPS, 2024.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405. 148644 2024.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509 2024.
Lihe Yang, Lei Qi. Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong sistency in semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7236-7246, 2023a.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint arXiv:2402.03162, 2024a.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023b.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06077 2024b.
Danah Yatim, Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. arXiv preprint 1xXiv:2311.17000 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control video generation by integrating text, image, and trajectory. arXiv preprint ar www.:::300000000000 2023.
Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nys-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint arXiv:2405.1536. 2024.
Jiwen Yu, Xiaodong Cun, Chenyang Qi, Yong Zhang, Xintao Wang, Ying Shan, and Jian Zhang. Animatezero: Video diffusion models are zero-shot image animators. arXiv preprint arXiv.2312.03793, 2023.
14

---

Published as a conference paper at ICLR 2025
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409.02048, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077 2023b.
Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimiemotion: High-quality human motion video generation with confidence-aware pose guid- ance. Xrviv preprint arXiv:2406. (0l.:2406.11880) 19680, 2024.
Rui Zhao, Yuchao Gu. Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465 2023.
Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, and Jiang Bian. Compositional 3d- aware video generation with Ilm director. Advances in Neural Information Processing Systems, 2024a.
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision (ECCV), 2024b.
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang- 2024.
Vlogger: Make your dream vlog. arXiv preprint
Yi Zuo, Lingling Li. Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, and Yuwei Guo. Edit-your-motion: Space-time time diffusion decoupling learning for video motion editing, 2024. URL https: raaxxxivv
www.wwwww.wwv..oo..ooo///.. www.orgg/bbb...... 2405. 04496
15


Reviews:
[{"review_id": "p660r7ZjIW", "reviewer": "Reviewer_CnyS", "summary": "The paper introduces a novel approach called Trajectory Attention for fine-grained video motion control, particularly aiming to enhance camera motion control in video generation tasks. By modeling trajectory attention as an auxiliary branch alongside traditional temporal attention, the method leverages available pixel trajectories to inject precise motion information into the video generation process. This design allows the original temporal attention and the trajectory attention to work synergistically. The proposed method demonstrates strong adaptability, e.g., being transferable to architectures like DiT. Experiments across various tasks show significant improvements in control precision and content consistency while maintaining high-quality generation. Extensive ablation studies validate the effectiveness of each module.", "questions": "I suggest to review and correct the mathematical formulations and notation to enhance the paper's clarity and reliability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is lightweight, requiring low training costs, making it practical and efficient for real-world applications without the need for extensive computational resources.\n2. The method demonstrates strong transferability, showing effectiveness with different architectures such as DiT. \n3. The paper conducts thorough exploration at application level, showcasing the method's effectiveness in multiple tasks, including camera motion control and video editing. Abalation studies are sufficient.", "weaknesses": "1. The method heavily relies on dense optical flow information, as shown in Figure 3 of the supplementary material. This dependency can significantly increase inference time due to the computational cost of processing dense optical flow, especially in real-time applications. \n2. The reliance on dense optical flow makes it challenging to adapt the method to user inputs of sparse trajectories. As noted in DragNUWA, it's difficult for users to input precise trajectories at key points in practical applications, leading to a gap between training and inference. This limitation reduces the method's practicality in scenarios where only sparse motion cues are available.\n3. In line 158, H and W represent the dimensions of the latent features, but in Algorithm 3, H and W are used for image dimensions, which is confusing.\n4. Some examples in Fig.6 and Fig.9 are not significant, like the second example in Fig.6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WA0DWl7xQ0", "reviewer": "Reviewer_v5kt", "summary": "The paper introduces Trajectory Attention, a novel approach designed to enhance fine-grained motion control in video generation, particularly focusing on precise camera motion control within video diffusion models. Traditional methods often struggle with imprecise outputs and neglect temporal correlations, leading to inconsistencies in generated videos. This work addresses these challenges by explicitly modeling trajectory attention as an auxiliary branch alongside the standard temporal attention mechanism. By modeling trajectory attention as an auxiliary branch alongside the standard temporal attention, the method explicitly injects available pixel trajectory information into the video generation process. This design allows the temporal attention to focus on motion synthesis and short-range dynamics, while the trajectory attention ensures long-range consistency along specified paths. The approach efficiently integrates trajectory information without modifying the original model parameters and supports sparse trajectories, meaning it can handle partial trajectory data. Experiments demonstrate that this method significantly improves motion control precision and video quality across various tasks, including camera motion control on images and videos, as well as first-frame-guided video editing.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel concept of Trajectory Attention for fine-grained motion control in video generation. This auxiliary attention mechanism enhances the existing temporal attention in video diffusion models by explicitly incorporating trajectory information, which is a significant advancement in the field.\n2. By modeling trajectory attention as an auxiliary branch that works alongside the original temporal attention, the approach allows for seamless integration without modifying the original model parameters. This design choice is both practical and efficient, leveraging pre-trained models and enabling efficient fine-tuning.\n3. The proposed method demonstrates significant improvements in motion control precision and long-range consistency over existing methods. The experimental results, including quantitative metrics like Absolute Trajectory Error (ATE) and Relative Pose Error (RPE), validate the effectiveness of the approach.\n4. The paper includes thorough experiments and ablation studies that not only demonstrate the superior performance of the proposed method but also validate the design choices. This strengthens the credibility of the findings and provides valuable insights into the method's effectiveness.", "weaknesses": "1. The method is primarily designed for video diffusion models that use decomposed spatial-temporal attention. It is less clear how well the approach generalizes to models with integrated spatial-temporal attention (e.g. 3D DiTs) or other architectures. Expanding the evaluation to include such models would strengthen the contribution.\n2. The paper compares the proposed method with a limited set of existing approaches. Including discussions with more recent or state-of-the-art methods, especially those that have emerged concurrently, would provide a more comprehensive evaluation of the method's relative performance. For example, Collaborative Video Diffusion [1] uses epipolar attention to align contents of different camera trajectories, and Camco [2] also uses epipolar, but to enhance the 3D consistency of generated contents.\n3. The experimental evaluations are primarily conducted on the MiraData dataset. While this dataset may offer certain advantages, relying on a single dataset limits the ability to generalize the findings. Evaluating the method on additional, diverse datasets would strengthen the claims about its general applicability.\n4. While the method supports sparse trajectories, the paper does not extensively explore how it performs when the trajectory information is highly sparse, incomplete, or noisy. Real-world applications often involve imperfect data, so robustness to such conditions is important. Going back to my point 2, this is especially concerning since the model is trained on MiraData, which mostly consists of synthetic videos.\n\n[1] Kuang et al. Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control, in NeurIPS, 2024.\n\n[2] Xu et al. CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation, in arXiv, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQ5wuAMyFC", "reviewer": "Reviewer_EFMe", "summary": "This paper proposes injecting a new attention layer along the trajectory into the model to support camera motion control in video generation. During training, optical flow is used as the trajectory, and the new attention operation is performed only along this trajectory. The trained model achieves good results in camera control for image-to-video and video-to-video tasks.", "questions": "1) How do you ensure that when attention is applied along the trajectory, the generated pixel also follows the trajectory? Have you observed any cases where control fails?\n\n2) In Algorithm 3, are you feeding \\{I_r\\} to the model in any particular format? The same question applies for Algorithm 4 with \\{V_r\\}.\n\n3) Is the comparison to other work (motion control/camera control) fair? They are trained on different datasets, and they may have some issue generalizing to the evaluation dataset used here. How did you select the evaluation set? Were you able to evaluate on the test set of other papers? \n\n4) In training, optical flow is used as a trajectory, but in inference, the model takes the camera trajectory as input. Could this cause a mismatch between training and inference? Why not use the camera trajectory as guidance during training as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Metric-wise, it seems the model achieves better camera control.\n- The model can be used for first-edited-frame + original-video-guided editing, though how this is achieved is not very clear.", "weaknesses": "1) Figure-1 is confusing. It takes some time to understand the input and output of each task. It would be better to reorganize this figure to make it clearer. Each task could be separated into a small sub-figure with a clear indication of the input and output.\n\n2) In Figure-3, it’s unclear what the model’s input is in two scenarios: (1) when you have multiple frames as input, i.e., ‘camera motion control on videos’ in Figure-1, and (2) when you have multiple frames plus edited frames as input, i.e., ‘first-frame-guided video editing’ in Figure-1.\n\n3) The trajectory attention mechanism operates only in 2D space, making it challenging to distinguish motion orthogonal to the image plane—for example, whether an centered object is moving towards or away from the camera. In such cases, the coordinates remain the same across frames. Could this be a limitation of this method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6yfHNlROR", "reviewer": "Reviewer_1VgT", "summary": "The paper focuses on fine-grained camera motion control in video generation. It has the following contributions:\n1. Trajectory Attention Mechanism: Proposes a novel trajectory attention branch alongside the original temporal attention branch. It models attention along available pixel trajectories for camera motion control.\n2. Improved Performance: Demonstrates significant improvements in precision and long-range consistency for camera motion control in both images and videos while maintaining high-quality generation.\n3. Extension to Other Tasks: Shows that the approach can be extended to other video motion control tasks, such as first-frame-guided video editing, where it excels in maintaining content consistency over large spatial and temporal ranges.", "questions": "According to the weaknesses, you can take the suggestions below to make your paper more convincing:\n1. In Algorithm 3, please elaborate on which depth estimation method you take in step 1 and how you render a set of views $I_{r}$ and get the translation of pixels $T$ in step 2. In Algorithm 4, please elaborate on which point trajectory estimation method you take in step 2. Meanwhile, could you provide the visual results of the trajectory extraction from a single image and a video to demonstrate the correctness of Algorithms 3 and 4? \n2. Provide results of your method on videos with occlusions, rapid camera movements, and multiple moving objects, respectively.\n3. Provide a comparison with more related works, such as [MotionBooth](https://arxiv.org/abs/2406.17758) and [CamTrol](https://arxiv.org/abs/2406.10126). More comparisons to concurrent work are also encouraged but not mandatory.\n\nIf the authors could solve my problems, I would raise the score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Originality: The paper demonstrates originality in its approach to video motion control. The concept of trajectory attention, modeled as an auxiliary branch to traditional temporal attention, is a novel way to incorporate pixel trajectories for fine-grained camera motion control. This approach differs from existing methods that either rely on high-level constraints or neglect temporal correlations.\n2. Quality: The experimental setup is comprehensive, using a large-scale dataset and multiple evaluation metrics. The results are presented in a clear and organized manner, with both quantitative comparisons and qualitative visualizations. The ablation study further validates the effectiveness of the proposed components, indicating a high level of quality in the research design and execution.\n3. Significance: The significance of the paper lies in its potential impact on the field of video generation and motion control. The proposed method shows improved performance in camera motion control for both images and videos, which is crucial for creating high-quality and customized visual content.", "weaknesses": "1. The paper does discuss trajectory extraction for different tasks such as camera motion control on images and videos, and video editing. However, the description of the extraction process could be more detailed and clear. For example, in Algorithm 3 for trajectory extraction from a single image, some steps might require further clarification for a reader who is not familiar with the underlying concepts. The estimation of the depth map and the rendering of views are steps that could be explained in more detail, including the methods and algorithms used. Similarly, in Algorithm 4 for video trajectory extraction, the point trajectory estimation and the combination with camera motion could be more clearly described.\n2. While the proposed trajectory attention method shows promising results in the presented experiments, there is a lack of exploration of more complex scenarios. For example, in real-world video data, there may be occlusions, rapid camera movements, or multiple moving objects, and it is not clear how the method would perform in such situations.\n3. The comparison with existing methods, although extensive to some extent, could be more comprehensive. There are many other techniques in the field of video motion control that were not included in the comparison, and it is possible that some of these methods may have unique features or advantages that could have provided a more nuanced understanding of the proposed method's superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjiL99z1IZ", "reviewer": "Reviewer_KvZ7", "summary": "This paper introduces Trajectory Attention, an innovative method for fine-grained camera motion control that attends to available pixel trajectories. The authors identify conflicts between the original temporal attention modules in diffusion models and supplementary trajectory-conditioned temporal modules. To resolve these conflicts, the paper employs optical-flow data to define trajectories, samples the most correlated points along them, and applies a copy attention mechanism to enhance trajectory precision. The original temporal module is retained for consistency. Comprehensive experiments on camera motion control for both images and videos demonstrate significant improvements in precision and long-range consistency without compromising high-quality generation. Furthermore, the approach is shown to be extensible to other video motion control tasks, including first-frame-guided video editing, where it maintains content consistency over extensive spatial and temporal dimensions", "questions": "- How can one obtain or customize the appropriate intrinsic and extrinsic parameters when performing trajectory extraction for a single image or video? Does the camera always need to be directed at the center of the image?\n\n- Is it necessary to adjust the camera's intrinsic and extrinsic parameters based on the depth information available?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The Trajectory Attention module is intuitive and offers flexibility in capturing temporal correlations in camera motion. This innovative approach effectively addresses the challenges associated with fine-grained control of camera motion.\n\n- The experiments on camera motion control for both images and videos are impressive. They demonstrate significant improvements in precision and long-range consistency, all while maintaining high-quality generation. These results underscore the effectiveness of the proposed method in handling complex camera motion scenarios.\n\n- The paper effectively shows that the approach can be extended to other video motion control tasks. For instance, in first-frame-guided video editing, the method excels at maintaining content consistency over large spatial and temporal ranges. This versatility is a testament to the robustness and general applicability of the Trajectory Attention framework.", "weaknesses": "- The paper raises concerns about object dynamics in the image-to-video case presented in the supplementary material. The examples, such as the dog and the cat, lack additional motion, which could be a limitation. It would be beneficial to see how objects with more complex dynamics are handled by the method.\n\n- There is a concern regarding the generalization of camera pose. In the Image-to-Video (first-frame) scenario, the trajectory module is trained with optical-flow data from only 10K video clips. It's unclear how the method would perform under challenging motions, such as clockwise rotation, high-speed zooming in and out, or 360-degree rotations like those seen in NVS-Solver GitHub. In these extreme trajectories, points visible in the first frame may become invisible, potentially leading to anti-aliasing issues. Additional results or a discussion of the necessary limitations would aid in a more comprehensive assessment of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)