MVDiffusion: Enabling Holistic Multi-view Image Generation with
  Correspondence-Aware Diffusion
Authors: name: Shitao Tang; name: Fuyang Zhang; name: Jiacheng Chen; name: Peng Wang; name: Yasutaka Furukawa
Published: 2023-07-03 15:19:17

Abstract: This paper introduces MVDiffusion, a simple yet effective multi-view image
generation method for scenarios where pixel-to-pixel correspondences are
available, such as perspective crops from panorama or multi-view images given
geometry (depth maps and poses). Unlike prior models that rely on iterative
image warping and inpainting, MVDiffusion concurrently generates all images
with a global awareness, encompassing high resolution and rich content,
effectively addressing the error accumulation prevalent in preceding models.
MVDiffusion specifically incorporates a correspondence-aware attention
mechanism, enabling effective cross-view interaction. This mechanism underpins
three pivotal modules: 1) a generation module that produces low-resolution
images while maintaining global correspondence, 2) an interpolation module that
densifies spatial coverage between images, and 3) a super-resolution module
that upscales into high-resolution outputs. In terms of panoramic imagery,
MVDiffusion can generate high-resolution photorealistic images up to
1024times1024 pixels. For geometry-conditioned multi-view image generation,
MVDiffusion demonstrates the first method capable of generating a textured map
of a scene mesh. The project page is at https://mvdiffusion.github.io.