SyncDreamer: Generating Multiview-consistent Images from a Single-view
  Image
Authors: name: Yuan Liu; name: Cheng Lin; name: Zijiao Zeng; name: Xiaoxiao Long; name: Lingjie Liu; name: Taku Komura; name: Wenping Wang
Published: 2023-09-07 02:28:04

Abstract: In this paper, we present a novel diffusion model called that generates
multiview-consistent images from a single-view image. Using pretrained
large-scale 2D diffusion models, recent work Zero123 demonstrates the ability
to generate plausible novel views from a single-view image of an object.
However, maintaining consistency in geometry and colors for the generated
images remains a challenge. To address this issue, we propose a synchronized
multiview diffusion model that models the joint probability distribution of
multiview images, enabling the generation of multiview-consistent images in a
single reverse process. SyncDreamer synchronizes the intermediate states of all
the generated images at every step of the reverse process through a 3D-aware
feature attention mechanism that correlates the corresponding features across
different views. Experiments show that SyncDreamer generates images with high
consistency across different views, thus making it well-suited for various 3D
generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.