MV-Adapter: Multi-view Consistent Image Generation Made Easy
Authors: name: Zehuan Huang; name: Yuan-Chen Guo; name: Haoran Wang; name: Ran Yi; name: Lizhuang Ma; name: Yan-Pei Cao; name: Lu Sheng
Published: 2024-12-04 18:48:20

Abstract: Existing multi-view image generation methods often make invasive
modifications to pre-trained text-to-image (T2I) models and require full
fine-tuning, leading to (1) high computational costs, especially with large
base models and high-resolution images, and (2) degradation in image quality
due to optimization difficulties and scarce high-quality 3D data. In this
paper, we propose the first adapter-based solution for multi-view image
generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
enhances T2I models and their derivatives without altering the original network
structure or feature space. By updating fewer parameters, MV-Adapter enables
efficient training and preserves the prior knowledge embedded in pre-trained
models, mitigating overfitting risks. To efficiently model the 3D geometric
knowledge within the adapter, we introduce innovative designs that include
duplicated self-attention layers and parallel attention architecture, enabling
the adapter to inherit the powerful priors of the pre-trained models to model
the novel 3D knowledge. Moreover, we present a unified condition encoder that
seamlessly integrates camera parameters and geometric information, facilitating
applications such as text- and image-based 3D generation and texturing.
MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
XL (SDXL), and demonstrates adaptability and versatility. It can also be
extended to arbitrary view generation, enabling broader applications. We
demonstrate that MV-Adapter sets a new quality standard for multi-view image
generation, and opens up new possibilities due to its efficiency, adaptability
and versatility.