FlexGen: Flexible Multi-View Generation from Text and Image Inputs

## Abstract

Abstract In this work, we introduce FlexGen, a flexible framework designed to generate controllable and consistent multi-view images, conditioned on a single-view image, or a text prompt, or both. FlexGen tackles the challenges of controllable multi-view synthesis through additional conditioning on 3D-aware text annotations. We utilize the strong reasoning capabilities of GPT-4V to generate 3D-aware text annotations. By analyzing four orthogonal views of an object arranged as tiled multi-view images, GPT-4V can produce text annotations that include 3D-aware information with spatial relationship.
By integrating the control signal with proposed adaptive dual-control module, our model can generate multi-view images that correspond to the specified text.
FlexGen supports multiple controllable capabilities, allowing users to modify text prompts to generate reasonable and corresponding unseen parts. Additionally, users can influence attributes such as appearance and material properties, including metallic and roughness.
Extensive experiments demonstrate that our approach offers enhanced multiple controllability, marking a significant advancement over existing multi-view diffusion models. This work has substantial implications for fields requiring rapid and flexible 3D content creation, including game development, animation, and virtual reality. Project page: https://xxu068.github.io/flexgen.github.io/ .

## 1 Introduction

Recent progress in generative models  (Song et al., ; Ho et al., ) has significantly advanced 2D content creation, thanks to the rapid increase in 2D data volumes.
However, 3D content creation remains challenging due to the limited accessibility of 3D assets, which are essential for diverse downstream applications, including game modeling  (Gregory, ; Lewis & Jacobson, ), computer animation  (Parent, ; Lasseter, ), and virtual reality  (Schuemie et al., ).
Previous 3D generation methods primarily concentrate on optimization-based techniques using multi-view posed images  (Wang et al., ; Yariv et al., ; Ge et al., ; Verbin et al., ), or employ SDS-based distillation approaches derived from 2D generative models  (Lin et al., ; Poole et al., ; Liang et al., ). Although effective, these methods often demand significant optimization time, limiting their practicality in real-world applications.

Figure: Figure 1: FlexGen is a flexible framework designed to generate high-quality, consistent multi-view images conditioned on a single-view image, or a text prompt, or both. Our method allows editing of unseen regions and modification of material properties through user-defined text.
Refer to caption: x1.png

Multi-view diffusion models (Liu et al., ; Long et al., ; Shi et al., ; ; Wang & Shi, ) have demonstrated the potential of pre-trained 2D generative models for 3D content creation through the synthesis of consistent multi-view images. Despite their promising performance, the controllable generation of multi-view images remains under-explored. Most existing multi-view diffusion models typically rely on a single-view image, which lacks 3D-aware controllable guidance and proves insufficient for robust multi-view image generation. For example, the generation of unseen regions continues to pose a significant challenge, often simply replicating information from the input view to unseen areas.

Several studies focus on conditional 3D generation. For instance, Coin3D (Dong et al., ) utilizes basic shapes as 3D-aware guidance, whereas Clay (Zhang et al., ) leverages sparse point clouds and 3D bounding boxes. However, these guidance methods are not user-friendly. Other research (Xu et al., ; ) employs sparse multi-view images as 3D-aware guidance for 3D generation.
These methods effectively supplement most unseen regions due to the incorporation of additional views, thereby achieving promising results. Nevertheless, acquiring sparse multi-view images is not always feasible in practice. These approaches typically depend on a pre-trained multi-view diffusion model for this purpose, yet integrating 3D-aware controllable guidance into multi-view diffusion models remains a significant challenge. Inspired by controllable content creation in 2D domain with text prompts, similar design can be incorporated as extra condition for providing 3D-aware guidance. Text encompasses adequate relationship information, which has been proven in the 2D generative model (Shen et al., ).
Previous SDS-based distillation methods  (Liang et al., ; Lin et al., ; Poole et al., ; Wang et al., ) have successfully employed text prompts to guide 3D asset generation, yielding significant results and validating this approach.

However, designing 3D-aware text annotations for 3D assets is not trivial. Most prior works (Li et al., ; Achiam et al., ) focus on 2D image captioning, which lacks sufficient 3D-aware information. To overcome this limitation, Cap3D (Luo et al., ) leverages the reasoning capabilities of BLIP-2 (Li et al., ) to generate descriptions for each rendered view of a 3D asset. These descriptions are then aggregated by GPT-4 (Achiam et al., ) to form a comprehensive caption. However, this approach often results in a high-level summary that misses detailed local captions. This limitation arises from two main factors: firstly, BLIP-2 primarily produces global descriptions, and secondly, individual view provides limited information about the object, often resulting in redundant or incomplete single-view annotations.
Instant3D (Li et al., ) is the most similar work to ours, utilizing text prompts from Cap3D to generate multi-view images. However, as discussed before, text annotations from Cap3D lack 3D-aware information and Instant3D only supports text-to-3D task, which is not flexible enough.

Alternatively, we propose leveraging the powerful recognition capabilities of GPT-4V(ision) to perform 3D-aware global-to-local captioning. By analyzing four orthogonal rendered views from a object, GPT-4V is capable of generating detailed descriptions that capture both global context and local features with 3D-aware information. This approach ensures the resulting text annotations are enriched with 3D-aware details with spatial relationship, providing a more comprehensive and perceptually accurate understanding. We show a comparison with Cap3D in Figure  2.

Figure: Figure 2: Comparison of the caption between Cap3D and Ours.

With 3D-aware text annotations, we can enhance the generative capabilities of existing multi-view diffusion models, enabling controllable generation guided by text prompts. By modifying the text prompt, our model is capable of generating diverse and consistent multi-view images that accurately correspond to the given textual descriptions. To this end, we propose an adaptive dual-control module that adaptively integrates image and text modalities through reference attention, enabling precise joint control over multi-view image generation by leveraging both visual input and detailed text prompts. In this work, we demonstrate three kinds of controllability of our model: firstly, our model can supplement unseen parts, achieving controllable unseen part generation; secondly, we introduce material controllability by adjusting the text prompt to modify the metallic and roughness properties when rendering multi-view images. For instance, appending the prompt with “high metallic” and “low roughness” can reflect these material characteristics accurately; finally, our model enables part-level control over the texture of the generated unseen parts,
as we incorporate detailed texture information during the text annotation phase.

To summarize, our contributions are listed as follows.

- •
We propose FlexGen, a method for flexible generation of multi-view images, guided by both image and text inputs. This approach offers robust controllability and ensures that the generated images accurately correspond to the given textual descriptions.
- •
We annotate 3D-aware text guidance using GPT-4V and generate material prompts consistent with the materials used during rendering, achieving controllable generation of textures, materials, and unseen parts.
- •
Extensive experiments validate the effectiveness of our proposed methods, demonstrating the ability to controllable generation of multi-view images.

## 2 Related work

### 2.1 Diffusion Models for Multi-view Synthesis

Recent research has extensively explored the generation of multi-view images using diffusion models to achieve efficient and 3D-consistent results. These efforts include both text-based methods, such as MVDiffusion (Deng et al., ), MVDream (Shi et al., ) and ImageDream (Wang & Shi, ), and image-based methods like SyncDreamer (Liu et al., ), Wonder3D (Long et al., ) and Zero123++ (Shi et al., ).
MVDiffusion, for instance, leverages text conditioning to simultaneously generate all images with a global transformer, facilitating cross-view interactions. Similarly, MVDream incorporates a self-attention layer to capture cross-view dependencies, ensuring consistency across different views. For image-based approaches, SyncDreamer constructs a volume feature from the multi-view latent representation to produce consistent multi-view color images. Wonder3D enhances the quality of 3D results by explicitly encoding geometric information and employing cross-domain diffusion.
Several methods (Tang et al., ; ; Zheng et al., ; ) adopt these approaches to first generate multi-view images of an object and then reconstruct the 3D shape from these views using sparse reconstruction techniques.
Despite achieving reasonable results, these methods still face limitations in controllable generation.

### 2.2 Controllable Generative models

In recent years, adding conditional control to generative models has garnered increasing attention for enabling controllable generation. These efforts span both 2D and 3D domains. For instance, several 2D methods (Hess, ; Brooks et al., ; Gafni et al., ; Kim et al., ; Parmar et al., ) focus on text-guided control by adjusting prompts or manipulating CLIP features. Additionally, ControlNet (Zhang et al., ) allows for a series of 2D image hints for control tasks through a parallel model architecture. However, similar controllable capabilities in 3D generation (Bhat et al., ; Cohen-Bar et al., ; Pandey et al., ) remain largely inapplicable. Coin3D (Dong et al., ) introduces a framework for generating 3D assets guided by basic shapes and text. Nonetheless, basic 3D shapes are not user-friendly for general users, whereas text serves as a more intuitive and accessible conditional input.
In this work, we integrate text prompts into the multi-view diffusion model for controllable multi-view image generation, enabling the model to generate controllable unseen regions when a single-view image is simultaneously provided.

## 3 Method

FlexGen is a flexible multi-view generation framework that supports conditioning based on text, single-view images, or a combination of both.
By incorporating 3D-aware text annotations derived from GPT-4V, our method effectively achieve controllable multi-view images generation, including reasonable unseen part generation, texture controllable generation and materials editing.
We begin with a succinct problem formulation in Section 3.1. Then, we introduce how to annotate 3D-aware caption in Section 3.2. After that, adaptive dual-control module is introduced to add textual condition into the framework in Section 3.3. Finally, we introduce training and inference in Sectin 3.4.
An overview framework of FlexGen is shown in Figure 3.

Figure: Figure 3: Overview of the framework. FlexGen is a flexible framework to generate controllable and consistent multi-view images, conditioned on a single-view image, or a text prompt, or both. The system incorporates a 3D-aware annotation method using GPT-4V and an adaptive dual-control module that integrates both a reference input image and text prompts for precise joint control. The condition switcher enhances flexibility, enabling the model to generate multi-view images based on image input, text input, or a combination of both modalities.
Refer to caption: x3.png

### 3.1 Problem Formulation

Given a single-view image $I$ or a user-defined prompt $T$ that describes an object or both, our goal is to develop a generative model $G$ that produces a tiled image $I_{out}$. This image consists of a $2\times 2$ layout, representing 4 views of an object - the front, left, back, and right - with each view at a resolution of $512\times 512$.
The multi-view images are aligned with both the single-view image and the prompt, ensuring consistency among them, as illustrated in Figure 1.
Inspired by  (Li et al., ), our generative model
$G$ is based on a large pre-trained text-to-image diffusion model. The design of the $2\times 2$ image grid aligns better with the original data format used by the 2D diffusion model, facilitating the utilization of more prior knowledge.
Regardless of the focal length and pose of the input image $I$, we consistently generate orthographic images with a fixed focal length and an elevation angle of $5^{\circ}$. For example, when processing an input image captured at an elevation angle $\alpha$ and azimuth angle $\beta$, FlexGen generates multi-view images at azimuth angles $\{\beta,\beta+90^{\circ},\beta-90^{\circ},\beta+180^{\circ}\}$, all with a fixed elevation of $5^{\circ}$.

### 3.2 3D-Aware Caption Annotation

Cap3D (Luo et al., ) utilized BLIP (Li et al., )
to annotate each single rendered view and then leverage GPT4 for holistic description. However, such method leads to text annotation lacking 3D-aware information.
Alternatively, we construct a dataset consists of paired multi-view images and 3D-aware global-local text annotation, as shown in the Figure 4. Our dataset is based on Objaverse (Deitke et al., ), which provides basic shape and textual for each object, lacking high-quality textual description for each 3D asset.
Therefore, we utilized an advanced large-scale multimodal model, GPT-4V, to generate high-quality textual descriptions for each 3D asset. Given these orthogonal views, GPT-4V not only summarizes a comprehensive global description of the object but also captures the intricate 3D relationships between its components. Specifically, the dataset construction process consists of three steps: rendering, captioning, and merging. (1) In the rendering stage, each 3D asset is rendered into four orthogonal multi-view images with a resolution of 512×512, which are tiled into a single image $I_{out}$ in a $2\times 2$ layout. (2) The captioning step is performed using GPT-4V with tiled image, which generates 3D-aware global and local captions. (3) In the final step, the global and local descriptions are merged to form the “global-local text description” of the 3D asset. During training, we randomly select a portion of the local descriptions to simulate user behavior.

Moreover, we incorporate material descriptions, such as metallic and roughness attributes, into the text annotations to enable material-controllable generation. We propose adding material descriptions that correspond to the materials used during rendering by blender Hess (). For example, if multi-view images are rendered with high metallic and low roughness, we enhance the prompt with “high metallic” and “low roughness” to ensure the descriptions match the visual data. More details on the material rendering can be found in the Appendix.

### 3.3 Adaptive Dual-Control Module

Previous approaches, such as those by Li et al. () and Shi et al. (), typically focus on single-modality inputs, conditioning solely on either a text prompt or a single-view image, without enabling joint control over both for generating multi-view images. To overcome this limitation, we propose an adaptive dual-control module that allows for simultaneous conditioning on both image and text inputs, enabling more precise and flexible multi-view image generation.

Our method builds upon the reference attention mechanism (Zhang, ), which we extend to integrate both the reference image and text prompt effectively. This enhanced integration facilitates robust interaction between the two modalities, enabling our model to generate multi-view images that maintain (1) high fidelity to the input image and (2) coherence with both the global and fine-grained descriptions specified in the text.

Reference attention involves running the denoising UNet model on an additional reference image, appending the self-attention key and value matrices from the reference image to the corresponding attention layers during model denoising. We introduce a slight modification to this approach. In addition to the reference image, we inject the prompt information. Specifically, the user-defined prompt is processed through the CLIP encoder to obtain per-token CLIP text embeddings $E$ with a shape of $L\times D$, where $L$ represents the token length and $D$ the embedding dimension. This prompt includes both global descriptions and local features. Using the cross-attention mechanism, we facilitate sufficient information interaction between the image and prompt, enabling more precise joint control. Once this interaction is complete, we append the self-attention key and value matrices from our adaptive dual-control module to the corresponding attention layers during model denoising.

Figure: Figure 4: 3D-aware caption generation pipeline. A 3D object is rendered into four orthogonal multi-view images (front, left, back, right) in a $2\times 2$ grid layout. Using GPT-4V, the agent generates both global and local descriptions. The global description captures the overall attributes of the object and the 3D spatial relationships between its components, while local features detail specific aspects such as color, posture, and texture, thereby enriching the dataset with rich semantic annotations.
Refer to caption: x4.png

### 3.4 Training and inference

Building on the adaptive dual-control module, our framework accommodates both prompt and image conditions to guide the generation of multi-view images. To increase flexibility, we introduced a condition switcher during training that supports both single-mode and dual-mode conditions, allowing for seamless transitions between different input scenarios. With a configurable probability, inputs can be left empty: when the text prompt is absent, it defaults to an empty string to ensure uninterrupted processing by the model. Similarly, in the absence of an image input, we substitute it with a black image.

During inference, this design facilitates flexible and controllable multi-view generation. When both modalities are available, the image and prompt collaborate to provide complementary information, enriching the generated output. If only the image is supplied, the model functions in an image to multi-view mode, generating multiple views based solely on the visual input. Conversely, when only the text is available, the model operates as a text to multi-view generator, producing views that align with the textual description.

Figure: Figure 5: Qualitative comparison of novel view synthesis. Our method achieves superior generation quality by integrating both text and image guidance. The text prompts used in our approach are generated by GPT-4V based on the input view. Some details are enlarged in the red circle
Refer to caption: x5.png

## 4 Experiments

### 4.1 Evaluation Settings

Training Datasets.
Given the inconsistent quality of the original Objaverse dataset (Deitke et al., ), we initially excluded objects lacking texture maps and those with low polygon counts. We subsequently curated a collection of 147k high-quality objects to form the final training set. For rendering these objects, we employed Blender, setting the camera distance at 4.5 units and the field of view (FOV) to 30 degrees. We generated 24 ground-truth images for the target view set, maintaining a fixed elevation of 5 degrees while uniformly distributing the azimuth angles across the range [0, 360]. The input views were randomly sampled with elevation angles between -30 and 30 degrees and azimuth angles evenly distributed across [0, 360]. All images were rendered at a resolution of $512\times 512$. For the purpose of 3D-aware caption annotation, we utilized four orthogonal images from the ground-truth set as inputs for GPT-4V.

Training Details.
We utilized Stable Diffusion 2.1 as our base model, training it on eight NVIDIA A800 80GB GPUs over a period of 10 days, completing 180,000 iterations with a batch size of 32. The Adam optimizer was employed with a learning rate of 1e-5. During training, we configured the probabilities of using both the prompt and image, only the image, and only the text at 0.3 each, while the probability for both modalities being absent was set at 0.1. Sampling involved 75 steps using the DDIM methodology (Song et al., ).

Table: Table 1: The quantitative comparison in novel view synthesis and sparse-view reconstruction. We report PSNR, LPIPS, CD and FS on the GSO dataset.

| Method | PSNR$\uparrow$ | LPIPS$\downarrow$ | CD$\downarrow$ | FS@0.1$\uparrow$ |
| --- | --- | --- | --- | --- |
| Ours | 22.31 | 0.12 | 0.076 | 0.928 |
| Ours(w/o caption) | 21.12 | 0.14 | 0.078 | 0.921 |
| Zero123++ | 18.83 | 0.16 | 0.087 | 0.910 |
| Era3D | 18.52 | 0.19 | 0.245 | 0.713 |
| SyncDreamer | 17.66 | 0.21 | 0.126 | 0.833 |

Table: Table 2: The quantitative comparison with MVDream in text to multi-view synthesis. “Ground truth” refers to the multiple views used to generate text by GPT-4V. Our method outperform MVDream by a large margin.

| Method | FID $\downarrow$ | IS$\uparrow$ | CLIP$\uparrow$ |
| --- | --- | --- | --- |
| Ours | 35.56 | 13.41$\pm$0.87 | 0.83 |
| Ground truth | N/A | 13.81$\pm$1.40 | 0.89 |
| MVDream | 44.42 | 12.98$\pm$1.22 | 0.79 |

Evaluation Metrics.
We evaluated our method across three distinct tasks: text to multi-view image generation, novel view synthesis (NVS), and sparse-view 3D reconstruction using the generated images as input to a reconstruction module. For text to multi-view generation, we employed the Frechet Inception Distance (FID) (Heusel et al., ) and Inception Score (IS) (Salimans et al., ) to assess image quality, and CLIP score (Radford et al., ) to evaluate alignment between the generated images and the textual descriptions.
For NVS, we utilized LPIPS (Zhang et al., ) to compare perceptual similarity between the generated novel-view images and the ground truth. The quality of 3D reconstruction was assessed using Chamfer Distance (CD) and volumetric Intersection over Union (IoU) between the reconstructed meshes and the ground truth ones.

Figure: Figure 6: Qualitative comparison with Era3D, SyncDreamer and Zero123++ on sparse-view 3D reconstruction. Our method generates consistent multi-view images, achieving better results.
Refer to caption: x6.png

Evaluation Datasets
We employed the Google Scanned Object (GSO) dataset (Downs et al., ) for evaluation purposes. For the text to multi-view generation task, we randomly selected 300 samples from the GSO dataset. For each sample, we rendered a set of four orthogonal views and employed our captioning method to generate 300 text prompts that accurately describe the objects. These same 300 samples were also used for evaluation in the novel view synthesis (NVS) and sparse-view 3D reconstruction tasks.

### 4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

Figure: Figure 7: Qualitative comparison of text to multi-view. Our method significantly outperforms MVDream, achieving better generation quality that is consistent with the text.
Refer to caption: x7.png

Figure: Figure 8: Ablation study to demonstrate that caption is capable of supplement unseen part, achieving significant better and reasonable results.
Refer to caption: x8.png

Novel view synthesis and sparse-view 3D reconstruction. First, we quantitatively compare our method with other single-image to multi-view approaches, including Zero123++ (Shi et al., ), Era3D (Li et al., ), and SyncDreamer (Liu et al., ), as shown in Table 1. Our method outperforms the others across several key metrics, such as PSNR and LPIPS, demonstrating that the joint control of text prompts and images allows for more consistent and realistic multi-view generation, particularly in unseen areas. Qualitative results are presented in Figure 5. To further verify the consistency of multi-view generation in three dimensions, we reconstructed the multi-view images generated by all methods using the open-source reconstruction method InstantMesh (Xu et al., ) for a fair comparison. We report Chamfer Distance (CD) and FS metrics in Table 1, which show that our multi-view images lead to more accurate geometry reconstructions. Qualitative comparison of 3D reconstruction results are shown in Figure 6.

Text to multi-view.
For text to multi-view evaluation, we conducted both qualitative and quantitative comparisons with the only existing open-source model, MVDream (Shi et al., ). Table 2 presents quantitative results, highlighting differences in generation quality and text-image consistency. Our model achieved Inception Score (IS) and CLIP score metrics that were comparable to those of the validation set, demonstrating strong image quality and text-image alignment. Furthermore, our model consistently outperformed MVDream across all evaluated metrics. Figure 7 showcases qualitative examples from the validation set, our model, and MVDream, illustrating that our model produces higher-quality, multi-view consistent images that more accurately match the text prompts. The images from our model exhibit superior fidelity and adherence to the input descriptions compared to those generated by MVDream, underscoring the effectiveness of our approach.

Figure: Figure 9: Material editing in generated multi-view images is facilitated by providing prompts such as “high-metallic” and “low-roughness”.
Refer to caption: x9.png

### 4.3 Abaltion Staudy

We conducted an ablation study to evaluate the impact of key components in our approach, specifically focusing on the 3D-aware caption annotation and the Adaptive dual-control module.

Adaptive Dual-Control Module.
The adaptive dual-control module represents a key innovation within our framework, enabling simultaneous control over both image and text inputs. Unlike previous models that typically focus on a single modality, our approach allows for enhanced flexibility and precision in generating outputs. This dual-input capability empowers users to guide the generation process using both image and textual prompts, significantly enhancing the model’s ability to produce coherent and contextually appropriate multi-view images, as demonstrated in Figure 8. This integration of modalities marks a substantial advancement over existing controllable generative models, offering a more intuitive and effective means for users to influence the output. Additionally, we illustrate the module’s capability to edit material properties in the generated multi-view images, as shown in Figure 9.

The impact of 3D-Aware Captioning.
To assess the effectiveness of 3D-aware captioning, we trained our model using annotations from the Cap3D dataset and our own dataset separately, comparing the control capabilities afforded by image and text prompts. As illustrated in Figure 10, training with Cap3D data limits the ability for fine control, whereas our method, through its integration of 3D-aware information, enables more effective editing of text prompts.

Figure: Figure 10: Ablation of 3D-aware Captioning .

## 5 Conclusion

In this work, we present FlexGen, a multi-view diffusion model designed to generate consistent and controllable multi-view images guided by a single image, text, or a combination of both. To achieve this, we harness the powerful recognition capabilities of GPT-4V to perform 3D-aware text annotations by reasoning over orthogonal views of an object, arranged as tiled multi-view images. Additionally, we introduce an adaptive dual-control module that enables text-based conditioning to be incorporated directly into the generation phase. By embedding spatial relationships, texture, and material descriptions into the text annotations, we achieve enhanced control over the output. Extensive experiments demonstrate the effectiveness of our proposed approach.

Limitations and future works. Although FlexGen introduces the innovative capability to jointly control both image and text inputs, our method occasionally encounters difficulties with complex user-defined instructions. This limitation likely stems from the constraints imposed by the availability of high-quality datasets. In the future, we plan to expand the dataset size and enhance the control capabilities to more effectively accommodate intricate instructions.

## Appendix A Appendix

### A.1 Detail of Material Rendering

We integrate material descriptions, including attributes such as metallicity and roughness, into the text annotations to enable material-controllable generation. These descriptions are designed to correspond with the materials utilized during rendering in Blender. With Blender, we can freely adjust the values of metallicity and roughness, allowing us to render corresponding images, as show in Figure 11. The values for metallicity and roughness range from 0 to 1. Specifically, when the values are below 0.3, the corresponding prompt is ”low” and when they exceed 0.6, the corresponding prompt is ”high”.

### A.2 More visualization results

We show more visualization of our model in Figure 12.

Figure: Figure 11: Visualization of an object rendered with different values of metallicity and roughness.
Refer to caption: x11.png

### A.3 Failure Cases

Figure 13 illustrates two failure cases, where the input text is unable to serve as an editor for multiple views. This limitation arises due to the absence of relevant text descriptions in the training data.

Figure: Figure 12: illustration of some success cases
Refer to caption: x12.png

Figure: Figure 13: illustration of some failure cases
Refer to caption: x13.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Does "Zero123++" refer to Zero123-XL? If not, could you clarify why Zero123-XL was not included for comparison?
2. How does this paper address the Janus problem in multi-view image synthesis?
3. Given FlexGen’s reliance on GPT-4V for 3D-aware captions, how does this approach overcome the limitations of generating multi-view consistency solely from text inputs?
soundness: 2
strengths: + FlexGen’s use of GPT-4V for 3D-aware captioning and the adaptive dual-control module offers flexibility in image synthesis, enabling detailed control over multi-view consistency and visual attributes.
summary: The paper presents FlexGen, a framework designed for multi-view image synthesis using single-view images, text prompts, or both. The core methodology leverages GPT-4V to generate 3D-aware text annotations, aiming to achieve more controllable and consistent multi-view image generation.
weaknesses: - The main contribution appears to be the use of GPT-4V for generating detailed captions in multi-view synthesis. This application of existing technology lacks significant innovation and may not constitute a substantial advancement in multi-view generation.
- Qualitative results in Figures 5 and 6 do not clearly demonstrate a marked advantage of FlexGen over existing methods.
- Appendix Section A.2 lacks the corresponding figures and analysis that could further clarify the model’s performance and visual outputs.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - Given that much of the proposed approach relies on established techniques, could the authors clarify what specific aspects of the methodology are novel? In what ways does the integration of text prompts and reference guidance go beyond previous work. 
- While GPT-4V is used for enhanced captioning, did you consider alternative or custom annotation strategies to achieve richer or more context-specific annotations for 3D assets? If so, why were they not pursued, and if not, how might they enhance your model’s performance?
- Currently, the approach requires generating new images in Blender for each material property. Have you considered any strategies to make the model more scalable in terms of material variations, possibly by automating or simulating material properties in the model itself?
- How does your model handle situations where the text and image prompts may conflict or suggest different visual details? Have you tested such cases, and if so, what did you observe about the model’s ability to reconcile or prioritize these inputs?
soundness: 2
strengths: - The use of GPT-4V for multi-view annotation is effective and shows promising results.
- The paper explores an interesting approach by incorporating material properties into multi-view synthesis within generative models.
summary: This paper proposes a method for generating multi-view images conditioned on both image and text prompts. Building on the reference attention mechanism from prior work, it incorporates additional text conditioning to enable controllable generation through text prompts. To enhance text captions, the authors use GPT-4V to annotate 3D assets and render objects with two different material properties, allowing for varied material appearances. Quantitative experiments demonstrate improved performance over several baseline methods in view synthesis, 3D reconstruction, and text-to-multi-view tasks.
weaknesses: - The primary contributions rely on integrating existing techniques (GPT-4V for captioning and a previously established reference-guided mechanism), rather than proposing fundamentally new methodologies.
- While the detailed captioning using GPT-4V is beneficial, the approach does not introduce a novel annotation strategy beyond leveraging GPT’s generative capacity.
- The core of the proposed approach relies heavily on previously established methods, specifically the reference view guidance. The main component, known as the "key-value (k, v) appending mechanism," which enables reference view guidance, was first introduced in prior work by Zhang et al. [1]. This paper primarily extends the mechanism by adding additional text prompts for control, but even this extension is not entirely novel; the concept of using text prompts alongside multi-view guidance has been previously explored in works such as Direct2.5 [2] and MVControl [3].
- The paper lacks quantitative evaluation to assess the effectiveness of material properties (e.g., metallic, roughness). Additionally, the approach does not appear scalable, as supporting a new material requires generating an entirely new set of images in Blender. It is also challenging to predict how well material conditioning via text prompts would perform for a broader range of material combinations.

[1] Lyumin Zhang. Reference-only control. In Reference-only control, pp. https://github.com/Mikubill/sd–webui–controlnet/discussions/1236. github, 2023. 

[2] Lu, Y., Zhang, J., Li, S., Fang, T., McKinnon, D., Tsin, Y., Quan, L., Cao, X. and Yao, Y., 2024. Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8744-8753). 

[3] Li, Z., Chen, Y., Zhao, L. and Liu, P., 2023. Mvcontrol: Adding conditional control to multi-view diffusion for controllable text-to-3d generation. arXiv preprint arXiv:2311.14494.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1: The paper primarily focuses on generating a fixed set of views. Did the authors considered enabling novel view synthesis with arbitrary viewing angles, similar to methods like SV3D or Cat3D? If so, how to envision adapting FlexGen to achieve this?

I find the core idea of this paper interesting and the presented results are solid. I am currently tending towards a borderline accept prior to the rebuttal. I believe the paper has the potential to make a significant contribution to the field, but I would like to see the authors address the raised weaknesses and questions in their rebuttal to solidify my decision.
soundness: 3
strengths: 1: The proposed framework is flexible, supporting generation from single-view images, text prompts, or both. This allows for versatile applications and user interactions. The adaptive dual-control module enables fine-grained control over various aspects of the generated multi-view images, including unseen parts, material properties, and textures, showcasing impressive controllability compared to existing methods.

2: The paper presents extensive experiments on the Objaverse and GSO datasets, demonstrating superior performance in novel view synthesis, sparse-view 3D reconstruction, and text-to-multi-view generation compared to state-of-the-art methods.

3: The writing is very clear and easy to follow. However, the appendix could be slightly refined, as it seems to have been rushed before the deadline.
summary: This paper introduces FlexGen, a novel framework for generating consistent and controllable 4 multi-view images from single-view images, text prompts, or both. The key contributions are: (1) A captioning pipeline that utilizes GPT-4V to generate 3D-aware text annotations from rendered orthogonal views. (2) A new framework that integrates image and text modalities for fine-grained control over the generation process. The results are solid, showing clear performance gains over recent baselines.
weaknesses: 1: While GPT-4V enables rich 3D-aware annotations, generating these can be computationally expensive and relies on a proprietary model. Exploring open-source MLLMs for captioning could be valuable, potentially increasing accessibility and reducing dependence on closed-source solutions. The paper could benefit from discussing the trade-offs between annotation quality and computational cost when using different models.

2: The paper mentions occasional difficulties with complex user-defined instructions. Further investigation is needed to understand the limitations of the current approach and improve its robustness in handling complex scenarios. Including visual examples would be beneficial.

3: A key limitation is the fixed 4-view output. While sufficient for some tasks, it falls short compared to video-diffusion models like Emu-Video used in im-3d and vfusion3d (16 views) or SV3D (20 views). Additionally, FlexGen cannot synthesize novel views from arbitrary angles, a capability demonstrated by SV3D and Cat3D. This restricts its use in applications requiring more comprehensive 3D understanding or flexible viewpoint control.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: - Will you release your annotation? 

- Can you elaborate on the key architectural differences between your model and other multiview diffusion models like Zero123++? From the numbers and visuals, seems the FlexGen outperforms the previous art by a large margin. But from the technical side, there seems to be not clear significant difference with previous work. What specific components or techniques in your approach contribute most significantly to the improvements you observe?
soundness: 3
strengths: - They annotate Objaverse by GPT4v which will be a good addition to the community if the authors would like to open source.
summary: This work annotates Objaverse with GPT4v and trains a mulit-view generation from both image and text inputs.
weaknesses: - Limited technical novelty. ImageDream (Wang and Shi, 2023) trained a multi-view image generation from both image and text and showed similar capability. Note their method can also be used to add new unseen details at the back, check their opensourced code here: https://github.com/bytedance/ImageDream. I find the using both image and text and the shared attention mechanisms are close in two works.  My suggestion: (1) compare to them technically; (2) show ablation study why your design is better than theirs. 

-  Lacks mathematical formulation for Adaptive Dual-Control Module. What is the formulation of condition switcher in Fig. 3?  From Sec. 3.4, is the switcher a simple dropout training and zero inference if missing? Consider providing a formal mathematical description of the Adaptive Dual-Control Module, including the condition switcher. This would enhance the technical depth of the paper and allow for better reproducibility.

- Missing ablation study.  Could you include ablation studies that address: (1) The impact of using the curated vs. regular Objaverse dataset, (2) The importance of the GPT4v caption in the model's performance, (3) The effect of injecting into both self-attention and cross-attentions? and (4) other designs that are crucial / novel to your work. These studies would help readers understand the relative importance of each component in your method."