Auto-Regressively Generating Multi-View Consistent Images
Authors: name: JiaKui Hu, name: Yuxiao Yang, name: Jialun Liu, name: Jinbo Wu, name: Chen Zhao, name: Yanye Lu

Generating multi-view images from human instructions is crucial for 3D
content creation. The primary challenges involve maintaining consistency across
multiple views and effectively synthesizing shapes and textures under diverse
conditions. In this paper, we propose the Multi-View Auto-Regressive (MV-AR)
method, which leverages an auto-regressive model to progressively generate
consistent multi-view images from arbitrary prompts. Firstly, the
next-token-prediction capability of the AR model significantly enhances its
effectiveness in facilitating progressive multi-view synthesis. When generating
widely-separated views, MV-AR can utilize all its preceding views to extract
effective reference information. Subsequently, we propose a unified model that
accommodates various prompts via architecture designing and training
strategies. To address multiple conditions, we introduce condition injection
modules for text, camera pose, image, and shape. To manage multi-modal
conditions simultaneously, a progressive training strategy is employed. This
strategy initially adopts the text-to-multi-view (t2mv) model as a baseline to
enhance the development of a comprehensive X-to-multi-view (X2mv) model through
the randomly dropping and combining conditions. Finally, to alleviate the
overfitting problem caused by limited high-quality data, we propose the
"Shuffle View" data augmentation technique, thus significantly expanding the
training data by several magnitudes. Experiments demonstrate the performance
and versatility of our MV-AR, which consistently generates consistent
multi-view images across a range of conditions and performs on par with leading
diffusion-based multi-view image generation models. Code and models will be
released at https://github.com/MILab-PKU/MVAR.