MV-Adapter: Multi-view Consistent Image Generation Made Easy

Authors: Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, Lu Sheng
Primary area: generative models
Keywords: Multi-view Image Generation, 3D Generation, Diffusion Models

Abstract:
Generating multi-view images of an object has important applications in content creation and perception. Existing methods achieved this by making invasive changes to pre-trained text-to-image (T2I) models and performing full-parameter training, leading to three main limitations: (1) High computational costs, especially for high-resolution outputs; (2) Incompatibility with derivatives and extensions of the base model, such as personalized models, distilled few-step models, and plugins like ControlNets; (3) Limited versatility, as they primarily serve a single purpose and cannot handle diverse conditioning signals such as text, images, and geometry. In this paper, we present MV-Adapter to address all the above limitations. MV-Adapter is designed to be a plug-and-play module working on top of pre-trained T2I models. This enables efficient training for high-resolution synthesis while maintaining full compatibility with all kinds of derivatives of the base T2I model. MV-Adapter provides a unified implementation for generating multi-view images from various conditions, facilitating applications such as text- and image-based 3D generation and texturing. We demonstrate that MV-Adapter sets a new quality standard for multi-view image generation, and opens up new possibilities due to its adaptability and versatility.

Decision: reject
Reviewer scores: 6, 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729234137737

thread_type:
official_review

<thread_object>
creator: reviewer_wkdr

<content>
summary:
This paper proposes a new method for consistent multi-view image generation. The proposed method followed the core idea of T2I-Adapter and made some extensions to it (e.g., conditions, training data) to be applied to the multi-view image generation task. The experimental results demonstrate the effectiveness of the proposed method.

strengths:
- This paper is well-written and easy to follow.
- The results are visually pleasing and show improvement against previous methods.

weaknesses:
- My major concern is that the technical contributions of this paper are a bit thin. Specifically, this paper can be viewed as an extension of T2I-Adapter [1] to the consistent multi-view image generation task. Therefore, most of the merits of the proposed method (e.g., efficiency, adaptability, versatility, performance) are inherited from the T2I-Adapter but are not original. The true contributions of this paper are actually the modifications to T2I-Adapter, which are incremental.

[1] Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z. and Shan, Y., 2024, March. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 5, pp. 4296-4304).

- For the modifications to T2I-Adapter, the conditions (Sec. 4.1) and attentions (Sec. 4.2) are mostly borrowed from previous works (the references are included in the paper) and there are no significant new insights as well.

Therefore, I think the paper is publishable but maybe not at a top conference like ICLR.

questions:
Please see the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730011472624

thread_type:
official_review

<thread_object>
creator: reviewer_vrdn

<content>
summary:
This paper introduces MV-Adapter, a plug-and-play adapter designed to support various conditions when generating multi-view consistent images. It preserves the base T2I model’s priors, improving efficiency by eliminating the need for constant full fine-tuning. Additionally, MV-Adapter supports multiple conditioning inputs, enhancing versatility. All the contributions above make the performance better.

strengths:
1. MV-Adapter supports various conditions, which provides a possible solution for multi-view generation with more conditional control.
2. MV-Adapter is more efficient and has less training costs according to the data shown in the paper.

weaknesses:
1. The paper does not present the metrics results of 3D reconstruction using the generated multiviews, a downstream task that most directly reflects the consistency of the generated views. It would be beneficial to include evaluations using standard 3D reconstruction metrics such as volumetric IoU or Chamfer distance. This would provide quantitative insights into the multi-view consistency and help validate the quality of the generated outputs.
2. The proposed “image cross-attention” mechanism is not novel, as it has already been employed in Zero123++. It would be helpful to explicitly compare the proposed method with the implementation in Zero123++, focusing on any specific differences or improvements.

questions:
1. How does the result of reconstructing 3D with MV-Adapter method compare with other multi-view generation methods? Does it offer any distinct advantages, or are there potential limitations that need to be addressed?
2. I noticed that the paper does not mention whether the text captions used for training are sourced directly from the Cap3D dataset or if they were separately annotated by the authors. Could you clarify the origin of these text captions.
3. For multi-view generation, the primary focus should be on the quality of the generated outputs rather than merely reducing the number of trainable parameters. I am curious how the model would perform if subjected to full-model fine-tuning while keeping the current design unchanged. This could further validate the architecture by demonstrating that comparable results can be achieved without the need to fine-tune all parameters, providing additional insights into the effectiveness of the proposed design.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730106643618

thread_type:
official_review

<thread_object>
creator: reviewer_c83o

<content>
summary:
This paper introduces MV-adapter, a plug-and-play module for pretrained T2I models, enabling them to support multi-view generation. Beyond the base image generation models, the proposed adapter can be integrated with other custom models, such as ControlNet. It allows for camera pose guidance, geometry guidance, and can optionally use an image as a condition. The authors conduct extensive experiments on various use cases and perform ablation studies to investigate network design choices.

strengths:
1. The paper is well-written, clearly organized, and easy to follow.
2. The experiments are comprehensive. The proposed method is tested on multiple base networks, including SD2.1, SDXL, and other fine-tuned models derived from these two.
3. The proposed method supports both geometry-guided and camera pose-guided generation, significantly expanding its potential real-world applications.
4. The module can be integrated with ControlNet, reusing the control signals learned from ControlNet.

weaknesses:
While I do not observe any major weaknesses in the paper, I still have some questions regarding the implementation and experimental details:

1. The paper does not provide much detail about the T2I-UNet used for image cross-attention. Is this UNet identical to the diffusion UNet? What timestep is used for feature extraction?

2. The proposed method is very similar to T2I-adapter in terms of network structure. Can the MV-adapter be combined with a learned T2I-adapter, similar to ControlNet?

3. How is the CFG calculated at inference time, since the network now has multiple conditions.

questions:
Please refer to the weaknesses listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730170027643

thread_type:
official_review

<thread_object>
creator: reviewer_v3nf

<content>
summary:
This paper introduces MV-Adapter, a plug-and-play module for text-to-image diffusion models that facilitates efficient 6-view image generation with multiple conditions. MV-Adapter overcomes the limitations of existing methods, such as high computational costs, incompatibility with T2I derivatives, and limited versatility in handling various conditioning signals. It achieves this through a condition guider that processes camera pose or geometry information and a decoupled attention mechanism that incorporates multi-view and image cross-attention. The authors evaluate MV-Adapter on various T2I models and derivatives, demonstrating its effectiveness in generating high-quality multi-view images from text, image, or geometry prompts.

strengths:
1: The paper is well-written and easy to follow.

2: The motivation is straightforward and intuitive. The study explores various design choices for text-to-multi-view diffusion models, making progress in all examined aspects.

3: The experiments are solid. Qualitative results showcase promising visual fidelity and view consistency across various T2I models and tasks. Quantitative evaluation also demonstrates competitive performance compared to baselines.

weaknesses:
1: A key limitation is the fixed 6-view output of MV-Adapter. While sufficient for some tasks, it falls short compared to video-diffusion models like Emu-Video used in im-3d and vfusion3d (16 views) or SV3D (20 views). Additionally, MV-Adapter cannot synthesize novel views from arbitrary angles, a capability demonstrated by SV3D and Cat3D. This restricts its use in applications requiring more comprehensive 3D understanding or flexible viewpoint control.

2: The quantitative improvements over baselines appear modest (with GSO image-to-multi-view). More compelling evidence is needed to showcase the impact of MV-Adapter. A user study or an anonymous webpage hosting video comparisons for direct assessment would strengthen these claims.

3: Minor: The proposed components draw heavily from existing work, which may lead to a perceived lack of novelty. The condition guider is inspired by T2I-Adapter, and the decoupled attention borrows from Era3D.

questions:
1: The paper primarily focuses on generating a fixed set of views. Did the authors considered enabling novel view synthesis with arbitrary viewing angles, similar to methods like SV3D or Cat3D? If so, how to envision adapting MV-Adapter to achieve this?

2:  Could the authors please provide more details about the experimental setup for the image-to-multiview comparison on the GSO dataset? Specifically, how many gso assets are used? Did top/bot views covered for evaluation?

I am currently leaning towards a weak rejection (more like a 4 score), and my final recommendation hinges on the authors' response to the raised weaknesses and questions in their rebuttal. Specifically, I would appreciate a discussion on the possibility of extending MV-Adapter to handle arbitrary viewpoints and a more in-depth analysis of the quantitative results.

After the rebuttal, I would like to recommend a borderline acceptance for this submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571786049

thread_type:
official_review

<thread_object>
creator: reviewer_fnks

<content>
summary:
The paper proposes a novel module called MV-Adapter, designed as a plug-and-play solution for enhancing pre-trained text-to-image (T2I) models in multi-view image generation. 

MV-Adapter addresses key limitations of existing methods, such as high computational costs, incompatibility with model derivatives, and limited versatility. By integrating with T2I models without invasive modifications or full-parameter training, MV-Adapter enables efficient high-resolution synthesis while maintaining compatibility with various extensions and derivatives like personalized models and plugins. 

It supports diverse conditioning signals, facilitating applications in text and image-based 3D generation and texturing. Experiments show that MV-Adapter sets a new standard in generating multi-view consistent images, combining efficiency, adaptability, versatility, and high performance.

strengths:
- MV-Adapter streamlines the training process by eliminating full fine-tuning, enabling efficient high-resolution image generation with reduced computational costs.

- MV-Adapter seamlessly integrates with various derivatives and extensions of the base T2I model, such as personalized models and plugins like ControlNets, enhancing its versatility across different applications.

- MV-Adapter supports diverse conditioning inputs (e.g., text, images, geometry), and MV-Adapter broadens the scope of multi-view generation and ensures consistent, high-quality output across various tasks.

weaknesses:
I would like to positively acknowledge that the results are impressive. 
- However, the weakness I found in this paper is that the contributions seem somewhat limited. The main contribution appears to be the application of self-attention, and if there are other contributions, could you revise the paper to emphasize those points? It would be helpful to clearly explain what differentiates this work from existing self-attention methods in multi-view generation, and what new methods, beyond self-attention, have been introduced in this paper.

- Additionally, I believe that including an evaluation comparing parallel and serial architectures would make this a stronger paper. Conducting an ablation study to compare the proposed parallel architecture with a serial architecture and evaluating generation quality, multi-view consistency, and computational efficiency would be beneficial.

- Overall, since the consistency of the results is good, enhancing the novelty aspect would make this a better paper. What is the novelty of this paper compared to existing multi-view generation methodologies? Furthermore, mentioning the potential applicability of this paper and highlighting potential new applications would also be helpful.

questions:
It has been stated in the Weaknesses section. I hope the authors will address these points in the rebuttal session.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732173863733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Highlighting Our Key Contributions (2)

comment:
**2. Technical Innovations in Adapter Design**

Our adapter framework incorporates the following two key technical contributions:

**Attention Architecture.** Building on the importance of adapters for the multi-view generation task, our design adheres to the principle of preserving the original network structure and feature space of the base T2I model. To achieve this, we duplicate the self-attention layers (including both architecture and weights) to create new multi-view attention and image cross-attention layers. By initializing the output projections to zero, we ensure that the base model's priors are maintained while allowing the new layers to learn geometric knowledge without interfering with the original model. In contrast, existing methods like MVDream expand the keys and values of the base model's original self-attention layers to include multi-view features, and Zero123++ similarly extends the key and value scopes to incorporate features from the reference image. These existing approaches modify the original network and can disrupt the learned priors, making them incompatible with the adapter principle.

Furthermore, we enhance the effectiveness of our attention layers through a parallel organization structure. The most straightforward way to insert new network layers is to append them after the original layers, connecting them in a serial manner. However, this sequential arrangement may not effectively utilize the image priors modeled by the pre-trained self-attention layers, as it requires the new layers to learn from scratch. Even if we initialize the new layers with the pre-trained self-attention parameters, the features input to these new layers are in a different domain, causing the initialization priors to be ineffective. To fully exploit the effective priors of the spatial self-attention layers, we design a parallel architecture. This design ensures that the new attention layers can inherit powerful prior to learn geometric knowledge efficiently. Detailed discussions of this design are presented in the **"Attention Architecture" section of our paper (Section 4.2)**. Our ablation results in **Table 5** directly demonstrate the superiority of this design.

**Unified Condition Embedding and Encoder.** To enable the adapter's versatility—supporting both camera conditions and geometry conditions for critical 3D applications like 3D generation and 3D texture generation—we innovatively unify camera parameters and geometric information into spatial map representations. We introduce a spatial-aligned condition encoder that encodes both types of maps using the same architecture. Previous methods have only encoded either camera information or geometric information but have not unified both within a single framework.

By adopting this unified representation and encoder, our adapter gains enhanced flexibility and applicability, seamlessly handling various types of guidance within a unified framework. This design allows the model to support multiple functionalities without additional complexity, which previous works have not achieved.

**3. Extensions to Arbitrary View Generation and Inspirations for Future Work**

**Extensibility to Arbitrary View Generation.** Our method is not limited to generating a fixed set of six views; it can be extended to arbitrary view generation, enabling broader applications. We achieve this by first generating multiple anchor views and then using them as conditions to guide the generation of additional views at arbitrary angles. The MV-Adapter framework remains largely unchanged—we adjust the number of input images to one or four (concatenating multiple images into a long image if needed) and replace the row-wise attention with full self-attention in the multi-view attention module. Detailed schemes and experimental results are provided in **Appendix A3.6** and on **our anonymous project page** https://mv-adapter.github.io/.

**Learning New Knowledge Like MV-Adapter.** By decoupling the learning of geometric knowledge from the image prior, our framework efficiently integrates new knowledge without compromising the base model's rich visual capabilities. This principle not only enhances learning from limited data but also inspires other tasks that build upon existing image priors to learn new types of knowledge. For example, beyond multi-view consistency, our approach can be applied to learning zoom in/out effects, consistent lighting, and other viewpoint-dependent properties. It also provides insights for modeling physical or temporal knowledge based on image priors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732174048628

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Highlighting Our Key Contributions (1)

comment:
We thank all reviewers for their constructive feedback and for recognizing the strengths of our work. Specifically, we appreciate Reviewer Fnks' comment that "the results are impressive," as well as Reviewer v3NF and Reviewer C83o's remarks that the paper is "well-written" and that "experiments are solid." These positive remarks encourage us to further clarify and strengthen the key contributions of our work.

We kindly suggest that reviewers visit **our anonymous project page** https://mv-adapter.github.io/ for a deeper understanding of our work. We note that this project page was already provided in our initial submission's supplementary materials for reference.

Additionally, we have uploaded the revised paper and included a highlighted version of the paper (with updated content highlighted) in the supplementary materials to facilitate the reviewers' review.

The primary concern raised is the lack of technical innovation in our adapter design. We would like to address this by highlighting the following key points:

**1. Novel Contribution: First Adapter-Based Solution for Multi-View Image Generation**

The reviewers questioned that the **components** of our method are not very new — this is true — multi-view self-attention, reference attention and spatial-aligned control are not new and have been widely used in prior literatures across different tasks. However, what we’d really like to address as our major contribution, is the design of unifying these techniques as an effective **“adapter”** framework working on top of pre-trained T2I models for the **multi-view image generation task**. Here we’ll show why “adapters” are so important especially for this task we want to solve.

**Why adapters?** Adapters are plug-and-play modules designed to work seamlessly with pre-trained generation models, facilitating downstream tasks by introducing external control signals or constraints. Crucially, they achieve this without invasive changes to the pre-trained models, thereby preserving the prior knowledge encoded in the base models. To adapt a pre-trained model for new tasks, we can always make changes to the network structure and do full fine-tuning. Then why would we need adapters? There are several good properties of adapters which we would favor against full fine-tuning:
- **Adapters are easy to train (efficiency).** Adapters only require updating a small number of parameters, making them both faster to train and memory-efficient. This property has become increasingly critical as state-of-the-art T2I models grow in scale (e.g., ~800M parameters in SD1.5 to ~12B in Flux.1), making full fine-tuning infeasible for smaller research labs.
- **Adapters keep versatility.** Downstream tasks often have limited training data compared to the billions of (text, image) pairs used to train base T2I models. Full fine-tuning on such limited datasets risks overfitting and mode collapse, impairing the generalization capability of the models. Adapters mitigate this risk by constraining the optimization space through fewer trainable parameters.
- **Adapters are easy to use (adaptability).** Adapters are plug-and-play modules and can be directly applied to different variants of base models, including customized fine-tuned versions and LoRAs. They can also interoperate with other adapters, enabling the construction of flexible workflows that accommodate diverse control signals. This is the most important reason why adapters are actively being developed and supported by the community.

These properties make adapters particularly suitable and useful for multi-view image generation.

**Relevance to Multi-view Image Generation.** Generating view-consistent images involves interaction across multiple views, requiring at least N images to be processed simultaneously during training. This is computationally challenging when working with large base models and high-resolution images. Moreover, the scarcity of 3D training data exacerbates optimization difficulties when performing full model fine-tuning. Previous multi-view generation methods are limited to resolutions of 256x256 (MVDream) or 384x384 (Zero123+++) and struggle with resolutions of 512x512 (Instant3D, Era3D). In contrast, modern T2I models can achieve resolutions of 1024x1024 or higher. These limitations primarily stem from the challenges associated with full training.

By leveraging our adapter design, we successfully achieve 768x768 resolution multi-view generation on the SDXL model. As shown in our paper, our trained MVAdapter demonstrates versatility, enabling seamless application to base model variations for stylized or customized multi-view generation, and compatibility with other adapters such as ControlNets.

**The above effects are achieved from our effective adapter framework, and the following innovative design in our framework.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1733142035642

thread_type:
public_comment

<thread_object>
creator: ~yoonwoo_jeong1

<content>
title:
Request for acknowledgment and comparison with NVS-Adapter(ECCV2024) which includes significant overlaps with MV-Adapter.

comment:
I appreciate your impressive and remarkable results in text-to-3D generation tasks.
I am the first author of "NVS-Adapter: Plug-and-Play Novel View Synthesis from a Single Image," which was presented at ECCV 2024 ([link](https://eccv.ecva.net/virtual/2024/poster/1425)). After reading your paper, we noticed that it includes significant overlaps with our core contributions but does not discuss or mention our work at all.

Our initial version has been available on arXiv since 2023 ([link](https://arxiv.org/abs/2312.07315)), and our code is also publicly accessible as an open-source project ([link](https://github.com/kakaobrain/nvs-adapter)). Thus, I believe our work cannot be considered a concurrent effort to your study and SHOULD be mentioned and thoroughly compared. Moreover, considering that all reviewers raised concerns about your paper’s technical novelty, I strongly believe that comparing your study with existing approaches will be a critical factor in evaluating its contributions.

For your reference, the core contributions of NVS-Adapter can be summarized as follows:

1. NVS-Adapter is the first approach to adopt the concept of Adapters for synthesizing multiple novel views in 3D generation.
2. Novel multi-views are generated simultaneously using attention-based architectures.
3. The plug-and-play approach enables a T2I model to be compatible with existing modules, such as ControlNet and LoRA, for various 3D generation scenarios, without requiring additional fine-tuning.

Thank you for considering this feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733312441698

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Conclusion

comment:
First and foremost, we extend our sincere gratitude to the reviewers and the public commenters for their detailed, insightful feedback and engaging discussions. Below, we summarize the key developments and reinforce the core contributions of our paper during the rebuttal phase.

**1. Enhancements Addressed During the Rebuttal Phase**

- **Validation of Technical Innovations:** We have conducted additional quantitative analyses and provided more extensive qualitative visualizations to substantiate the effectiveness of our proposed parallel architecture.
- **Comprehensive Evaluation of MV-Adapter:** We incorporated a user study within the image-conditioned multi-view generation setting and evaluated the geometry quality of 3D reconstruction. These evaluations confirm that MV-Adapter achieves state-of-the-art consistency and quality in multi-view generation.
- **Extension to Arbitrary Viewpoint Generation:** We have successfully extended MV-Adapter to support arbitrary viewpoint generation, accompanied by a substantial number of visualizations. This extension verifies the scalability and broad applicability of MV-Adapter across diverse scenarios.
- **Exploration of New Applications and Future Works:** MV-Adapter's compatibility with various personalized models and spatial control plugins allows for multi-view image customization and controllable generation. Additionally, by decoupling the learning of geometric knowledge from the image prior, MV-Adapter presents a more effective pathway for modeling new knowledge, such as physical or temporal aspects.
- **Supplementary Technical Details:** We have enriched the technical exposition of our paper by elaborating on the image cross-attention mechanisms, detailing the calculation of CFG with multiple conditions, and clarifying the experimental settings for comparisons with baseline methods.

**2. Summary of Responses to Public Comments**

Based on discussions and consensus with the authors of NVS-Adapter, we further confirmed one of our core contributions: presenting the first systematic adapter-based solution for multi-view image generation. We also acknowledge the oversight in not citing and discussing NVS-Adapter adequately in our original submission. We commit to addressing this in the revised version by including a comprehensive discussion of NVS-Adapter and its relation to our work.

**3. Reiteration of Novelty and Core Contributions**

- We introduce **the first systematic adapter-based solution for multi-view image generation**, significantly enhancing its efficiency, adaptability, and versatility.
- **Technical Innovations in Adapter Design:** Our innovative attention architecture, which includes the duplication of self-attention layers and a parallel architecture, along with the unified condition encoder that supports encoding camera and geometry conditions, represents a substantial advancement. Ablation studies have validated the effectiveness of these innovations. We appreciate the recognition of our technical contributions by the public commenters.
- **Extensions to Arbitrary View Generation and Future Inspirations:** The ability of MV-Adapter to effortlessly extend to arbitrary viewpoints paves the way for numerous downstream applications. Furthermore, our approach to decoupling geometric knowledge learning from image priors offers a novel and efficient methodology for modeling new types of knowledge.

In summary, the rebuttal phase has allowed us to strengthen our paper by providing additional validations, expanding our evaluations, and clarifying our contributions. We are confident that these enhancements, along with our novel contributions, substantiate the significance and impact of our work in the field of multi-view image generation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes an adapter-based method for consistent multi-view image generation. The paper received reviews from five reviewers resulting in five borderline scores. Multiple reviewers recognised the comprehensiveness of the experimental work and quality of writing and presentation. The core of the main concerns related to a lack of technical innovation and comments on incremental novelty. 

The author rebuttal could partially address concerns by presenting additional evaluation results and explanations enabling two reviewers to initially raise their scores after extended discussion. Further to this, public comments highlighted that recent highly related work was unacknowledged. The initial failure to acknowledge the relevant NVS-Adapter work, coupled with a somewhat unconvincing distinction justification did not convince some reviewers, causing multiple reviewers to lean towards rejection in the private discussion phase, based on core unresolved concerns regarding incremental contributions and lack of technical innovation in adapter design.

Authors noted their intention to address the relationship between NVS-Adapter and MV-Adapter. Paper modifications are proposed towards providing a thorough comparison and highlighting distinctions, however the proposed changes appear somewhat substantial ('task scope, motivation, methodology, overall design, and generated results'). There was a consensus that such additions are necessary and yet are currently not present in the manuscript version under review.