CamViG: Camera Aware Image-to-Video Generation with Multimodal
  Transformers
Authors: name: Andrew Marmon; name: Grant Schindler; name: José Lezama; name: Dan Kondratyuk; name: Bryan Seybold; name: Irfan Essa
Published: 2024-05-21 20:54:27

Abstract: We extend multimodal transformers to include 3D camera motion as a
conditioning signal for the task of video generation. Generative video models
are becoming increasingly powerful, thus focusing research efforts on methods
of controlling the output of such models. We propose to add virtual 3D camera
controls to generative video methods by conditioning generated video on an
encoding of three-dimensional camera movement over the course of the generated
video. Results demonstrate that we are (1) able to successfully control the
camera during video generation, starting from a single frame and a camera
signal, and (2) we demonstrate the accuracy of the generated 3D camera paths
using traditional computer vision methods.