DreamCinema: Cinematic Transfer with Free Camera and 3D Character
Authors: name: Weiliang Chen; name: Fangfu Liu; name: Diankun Wu; name: Haowen Sun; name: Haixu Song; name: Yueqi Duan
Published: 2024-08-22 17:59:44

Abstract: We are living in a flourishing era of digital media, where everyone has the
potential to become a personal filmmaker. Current research on cinematic
transfer empowers filmmakers to reproduce and manipulate the visual elements
(e.g., cinematography and character behaviors) from classic shots. However,
characters in the reimagined films still rely on manual crafting, which
involves significant technical complexity and high costs, making it
unattainable for ordinary users. Furthermore, their estimated cinematography
lacks smoothness due to inadequate capturing of inter-frame motion and modeling
of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC
has opened up the possibility of efficiently generating characters tailored to
users' needs, diversifying cinematography. In this paper, we propose
DreamCinema, a novel cinematic transfer framework that pioneers generative AI
into the film production paradigm, aiming at facilitating user-friendly film
creation. Specifically, we first extract cinematic elements (i.e., human and
camera pose) and optimize the camera trajectory. Then, we apply a character
generator to efficiently create 3D high-quality characters with a human
structure prior. Finally, we develop a structure-guided motion transfer
strategy to incorporate generated characters into film creation and transfer it
via 3D graphics engines smoothly. Extensive experiments demonstrate the
effectiveness of our method for creating high-quality films with free camera
and 3D characters.