Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
  Pretrained 2D Diffusion Models
Authors: name: James Burgess; name: Kuan-Chieh Wang; name: Serena Yeung
Published: 2023-09-14 18:52:16

Abstract: Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.
  ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts.