GenDoP: Auto-regressive Camera Trajectory Generation as a Director of
  Photography
Authors: name: Mengchen Zhang, name: Tong Wu, name: Jing Tan, name: Ziwei Liu, name: Gordon Wetzstein, name: Dahua Lin

Camera trajectory design plays a crucial role in video production, serving as
a fundamental tool for conveying directorial intent and enhancing visual
storytelling. In cinematography, Directors of Photography meticulously craft
camera movements to achieve expressive and intentional framing. However,
existing methods for camera trajectory generation remain limited: Traditional
approaches rely on geometric optimization or handcrafted procedural systems,
while recent learning-based methods often inherit structural biases or lack
textual alignment, constraining creative synthesis. In this work, we introduce
an auto-regressive model inspired by the expertise of Directors of Photography
to generate artistic and expressive camera trajectories. We first introduce
DataDoP, a large-scale multi-modal dataset containing 29K real-world shots with
free-moving camera trajectories, depth maps, and detailed captions in specific
movements, interaction with the scene, and directorial intent. Thanks to the
comprehensive and diverse database, we further train an auto-regressive,
decoder-only Transformer for high-quality, context-aware camera movement
generation based on text guidance and RGBD inputs, named GenDoP. Extensive
experiments demonstrate that compared to existing methods, GenDoP offers better
controllability, finer-grained trajectory adjustments, and higher motion
stability. We believe our approach establishes a new standard for
learning-based cinematography, paving the way for future advancements in camera
control and filmmaking. Our project website:
https://kszpxxzmc.github.io/GenDoP/.