Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Authors: name: Shengqu Cai; name: Duygu Ceylan; name: Matheus Gadelha; name: Chun-Hao Paul Huang; name: Tuanfeng Yang Wang; name: Gordon Wetzstein
Published: 2023-12-03 14:17:11

Abstract: Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.