IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts

Authors: Bohan Zeng, Shanglin Li, Yutang Feng, Ling Yang, Juan Zhang, Hong Li, Jiaming Liu, Conghui He, Wentao Zhang, Jianzhuang Liu, Baochang Zhang, Shuicheng YAN
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D generation, Diffusion model

Abstract:
Recent advances in 3D generation have been remarkable, with methods such as DreamFusion leveraging large-scale text-to-image diffusion-based models to guide 3D object generation. These methods enable the synthesis of detailed and photorealistic textured objects. However, the appearance of 3D objects produced by such text-to-3D models is often unpredictable, and it is hard for single-image-to-3D methods to deal with images lacking a clear subject, complicating the generation of appearance-controllable 3D objects from complex images. To address these challenges, we present IPDreamer, a novel method that captures intricate appearance features from complex **I**mage **P**rompts and aligns the synthesized 3D object with these extracted features, enabling high-fidelity, appearance-controllable 3D object generation. Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts, highlighting its promising capability in appearance-controlled, complex 3D object generation.

Decision: accept (poster)
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730540193334

thread_type:
official_review

<thread_object>
creator: reviewer_4vlr

<content>
summary:
This paper present a novel method to capture intricate appearance features from the Image prompts, which is further used to enhance the alignment of the image prompts with the generated objects. Experiments demonstrate the proposed method generate objects which is well-aligned with image prompts, show better ability in complex generation.

strengths:
* The paper proposes a novel framework for 3D generation by breaking an image prompt into several parts and adopting a multi-guidance optimization process. Experiments demonstrate the effectiveness of the proposed framework.
* The idea of the paper that breaks the complex images into several parts is interesting and good. Breaking a complex thing into parts makes a hard problem much easier.

weaknesses:
* The written of the paper is not so clear, some details are lack:
  * The description on how to adopt GPT-4v to generate localization prompts is lack in the paper.
  * In Figure 1 (b), the author gives comparison between VSD and IPSDS on text-based generation. But is the proposed method IPSDS need an image prompt? How to compare IPSDS with VSD on text-based generation? Moreover, for the cases in Fig (a), could the author provide the images parts extracted from the reference image of the castles. It’s hard to understand how could we break such things into parts.
  * For eq.9 and eq.10,the author highlights that “they localize the features of the multiple images onto 3D object” in many places such as Line 321-322, 349-350, which makes me very confused. I think the author is adopting eq.9 and eq.10 to fuse information from different image parts to do SDS loss. Therefore, this description is inaccurate and leads to misunderstanding. 、
  * Some annotation in the equations are missing, like $Z$ in eq.9.
* In line 360, the author declares that a global optimization is further needed, which is achieved by simply concatenating all the features from the multiple images instead of adopting a mask based strategy. Why we need such a global optimization? What if we directly adopt global optimization without the mask-guided one? I think the author should provide such evaluation.
* Finally, I think the evaluation of the paper is not enough. The accuracy of adopting SAM and GPT-4v to break into parts is not evaluated. Moreover, I think the author should provide more visualization examples on the extracted image parts together with the generation results, which will make overall process easier to understand.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612018781

thread_type:
official_review

<thread_object>
creator: reviewer_sw6c

<content>
summary:
The paper introduced a controllable 3D object generation approach using image prompts (similar to style transfer). The proposed IPDreamer approach is a novel method that could capture the intricacies of appearance features from image prompts, and could generate high fidelity and controllable 3D objects. The approach is tested on some public benchmarks with user studies available as well, and was proven to be effective.

strengths:
- The paper is tackling an important and very challenging 3D genAI problem. Comparing to existing approaches, the IPDreamer could edit the objects using more complex image prompts
- The introduced prompt score distillation sampling approach is a reasonable formulation that builds on existing SDS approaches, and the masked-guided alignment strategy seems to be highly effective 
- Experimental results suggest that the approach is better comparing to other counterparts. User studies is also provided.

weaknesses:
I think this is a nice paper and a good extension to many of the existing approaches. The final output of the algorithm seems to be good enough. I do have a few clarification questions that I hope the authors could address in future revisions of the papers:
- The paper leverages GPT-4v as MLLM inputs. How accurate should the MLLM be, in case people don't have access to this advanced MLLM algorithm? Would the output become much worse? 
- It's very nice to conduct user studies for genAI works in general. Could authors provide more demographics information in the appendix section? (age, gender, background, etc)
- I don't fully understand Fig 1b, especially the right images -- what is the contents in the input and what is the actual real-world application of this particular input/output pair?

questions:
See the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661222883

thread_type:
official_review

<thread_object>
creator: reviewer_zssc

<content>
summary:
The paper introduces a text/image-to-3D approach for controlling the appearance of generated 3D objects given complex input images where the subject is not clearly identified.
The proposed approach encompasses multiple components. 
First, IPAdapter image encoder is used to extract image features that are used as texture guidance within the Score Distillation Sampling (SDS).
To be able to handle complex images with multiple components, a mask-guided compositional alignment strategy exploits a Multi-Modal Language Model (MLLM) to provide localization part labels given the image and the provided coarse Nerf model.
Then, cross-attention maps are used to localize those parts by computing attention between the image feature and the textual labels produced by the MLLM.
Finally, the localized parts are optimized jointly to produce a globally consistent 3D object.
Experiments show that the proposed approach produces high-quality results that abide by the guidance image.

strengths:
- The idea of splitting complex objects into parts that are optimized jointly is interesting and can be potentially employed for more complicated 3D scenes.

- The method section is comprehensive and provides an overview of SDS, making it self-contained.

- The visual quality of the provided results is compelling.

weaknesses:
- The paper primarily focuses on controlling the generation of 3D objects from complex input images. As noted in line 537, "IPDreamer addresses the limitations of existing text-to-3D and **single-image-to-3D** methods." However, the paper does not include comparisons with relevant single-image-to-3D methods, such as [1] and [2]. Could the authors clarify why these comparisons were omitted?

- In Figure 7, the qualitative comparison presents different samples for each method. Conventionally, all methods are evaluated on the same samples to ensure consistency in comparisons. Could the authors provide insight into this choice?

- The proposed method incorporates several additional components beyond the standard SDS pipeline, including ChatGPT, SAM, ControlNet, and IPAdapter. Could the authors provide details on the runtime overhead introduced by each component, as well as the overall runtime?

- The method illustration in Figure 2 appears challenging to interpret. It does not effectively aid in understanding the proposed pipeline, and I found it difficult to correlate it with the text. A more intuitive figure might improve readability and clarity.

[1] Shi, Ruoxi, et al. "Zero123++: a single image to consistent multi-view diffusion base model." arXiv preprint arXiv:2310.15110 (2023).

[2] Voleti, Vikram, et al. "Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion." European Conference on Computer Vision. Springer, Cham, 2025.

questions:
- I do not understand Figure 1b. What is being generated, a 3D shape or an image? both the leaves and the water ripples look like images!

- What is the difference between equations (11-13) and (14-17)? Are both used during optimization?

- What is the impact of employing the super-resolution model, ControlNet tiling, on the final generated quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745687113

thread_type:
official_review

<thread_object>
creator: reviewer_prnj

<content>
summary:
The paper introduces IPDreamer which, by leveraging the complex image prompts for the first time, can generate detailed 3D objects. To achieve this task, IPDreamer first proposes an Image Prompt Score Distillation Sampling (IPSDS) that leverages both RGB features and normal features to help guide the denoising process. The authors further introduce a mask-guided compositional alignment strategy that allows for extracting corresponding features from different images of the same objects, further improving the details of the 3D generation. Extensive qualitative and quantitative experiments have been provided in the paper.

strengths:
+ The paper is the first time to consider generating 3D objects from complex images. It's quite interesting considering the current progress of the current 2D generative models.

+ The paper is well-written and easy to follow.

weaknesses:
- Fig.1 is not clear. It's not able to showcase that existing methods struggle with complex images.

- The results showcased are not quite aligned with the input image.

- The masks in Fig.4 are not quite aligned with the corresponding parts.

- It's hard to see the effectiveness of mask-guided compositional alignment. 

- The results provided in Fig. 5 are not very good.

- What if we apply the best text-to-2D diffusion model to the DreamFusion or other text-to-3D pipeline and carefully design the text prompts? For example, the text-to-2D diffusion model that's capable of generating complex and high-resolution images.

questions:
Based on my comments on the strengths and weaknesses, I currently still lean a little bit toward the positive rating. I would like to hear from the other reviewers and the authors during the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731994199934

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarifications and Summary of Key Contributions

comment:
We'd like to express our gratitude to the reviewers and ACs for their thorough evaluation and valuable feedback. To facilitate a clearer understanding of our work, we have prepared a summary of our key contributions and would like to address several important points that may have been misinterpreted during the review process.

**Contributions**:
1. For cases with unclear boundaries, such as "Leaves flying in the wind," IPDreamer can generate reasonable 3D objects based on these conditions.
2. IPDreamer can utilize complex images to achieve high-quality texture editing on coarse 3D objects.
3. The Mask-guided Compositional Alignment strategy introduced in IPDreamer enables the concurrent integration of multiple image prompts to effectively guide the optimization process of a single 3D object.

Our IPDreamer is capable of achieving the functions mentioned in the contributions, which are difficult for existing text-to-3D and single-image-to-3D methods to accomplish. Then, we would now like to address several points raised during the review process.

**Clarifications**:
1. In the original version of the paper, we compared our method with single-image-to-3D methods, such as LRM and LGM. In the revised paper, we add results from additional single-image-to-3D methods for further comparison.
2. Our method primarily transfers features from complex images to an initialized coarse 3D model for flexible and high-quality 3D object synthesis. It is important to note that in industry applications, it is common first to provide an initial 3D object and then perform texture editing on it. Therefore, IPDreamer has significant practical value for building 3D assets.
3. The Mask-guided Compositional Alignment scheme is essential when substantial semantic difference exists between the reference complex image prompt and the initialized 3D object, or when multiple complex images are used as input conditions. When the reference image and the initial 3D model do not have significant semantic differences, high-quality 3D object synthesis can be achieved using only equations (2) to (7), which demonstrates the robustness of IPSDS.

In response to the insightful feedback from our reviewers, we make minor adjustments to both the main content and the figures and add more analysis in the Appendix. All modified sections, including the updated explaination and figures, are highlighted in blue (with figure captions also marked in blue if any modifications were made).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732947723784

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarifications and Contributions: A Letter to Area Chairs, Reviewers, and 3D Generation Researchers

comment:
Dear Area Chairs, Reviewers, and 3D Generation Researchers,

We sincerely thank all the reviewers and area chairs for their valuable contributions to improving this paper. The purpose of this letter is twofold: on one hand, to assist the reviewers and area chairs in making their final judgments, and on the other hand, **since this work has genuinely helped us solve several practical problems, we hope this paper will assist researchers in the 3D generation field by addressing potential challenges and providing some ideas.** Here, we would like to highlight the motivation, contribution, application, and future work of our paper once again.

**Motivation**

Although we have already clearly stated the motivation in the introduction, we would like to restate it here to provide a more intuitive understanding of our starting point. This will focus on the limitations of text-to-3D and single-image-to-3D methods, and explain the core idea behind our IPDreamer.

- Text-to-3D has achieved impressive realism, but the generated results often lack precision and controllability due to the limited information provided by the input text.

- In contrast, single-image-to-3D methods benefit from the prior information contained in the input image, allowing for better control over the output appearance. However, as researchers familiar with this approach will know, these methods require the subject in the input image to be very clear and unobstructed, often needing to be segmented out from the background. This is a limitation since such images are difficult to obtain in practice, as most images contain occlusions, and the main subject may not be easily isolated. This makes single-image-to-3D methods more restrictive and less stable when combined with text-to-image methods for generating text-to-3D results.

- The core idea of our paper is to extract corresponding features from complex images and align them with 3D objects. This approach not only enables the generation of 3D results for ambiguous subjects, such as “leaves flying in the wind,” but also allows for high-quality 3D object texture editing.

**Contribution**

- First, with the proposed IPSDS and Mask-guided Compositional Alignment, our method IPDreamer can leverage complex images for texture editing of 3D objects, generating high-fidelity 3D results that preserve the initial shape of the 3D object while adapting its appearance based on the complex image. This functionality is further elaborated in the application section.

- Since we can guide the generated results with complex images, our method can handle testing samples that are ambiguous or lack a clear main subject. For instance, we can generate the ideal 3D results for vague descriptions like "leaves flying in the wind" or "splashing wave."

- By leveraging Mask-guided Compositional Alignment, IPDreamer can also use multiple complex images to collaboratively guide the optimization of a single 3D object.

**Application**

Next, we will discuss the practical value of our contributions, especially the texture editing.

- In real-world applications like 3D game character design or scene creation, a common workflow involves creating a base 3D structure, which is then edited by modelers based on designers' sketches. IPDreamer can significantly reduce the amount of work for the modelers.

- For chibi-style character design, where the 3D structure of the character is usually fixed, but the appearance (texture) needs modification, our high-quality texture editing using Mask-guided Compositional Alignment and IPSDS can perfectly achieve this.

- For fixed 3D results, our method can guide the generation of diverse 3D outputs by leveraging a variety of complex images. This is particularly useful for enhancing the diversity of 3D data.

- In academic research, for example, in 4D generation, where we need to generate significantly different appearances of the same person, our texture editing can complement deformation networks, which often struggle to create noticeable appearance changes.

There are many other potential applications, but here we have listed some of the most familiar ones for reference.

**Future Work**

The core idea behind IPDreamer is extracting features from complex image prompts and aligning them with 3D objects. Future work could focus on improving feature extraction and refining the alignment process to enhance the accuracy and versatility of this method.

Finally, we would like to express our sincere gratitude once again to the reviewers and area chairs for their efforts. We deeply appreciate the constructive suggestions that have helped us improve the paper. **We respect the reviewers' opinions and fully support the area chairs' final decision. At the same time, we hope that the problems solved by IPDreamer will help many others and inspire new ideas in the 3D generation community.**

Sincerely, \
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes IPDreamer a novel framework for text-input or image-input based 3D generation using guidance from a diffusion model. The key contribution of this work is to allow for precise control of the appearance of the generated mesh via a provided high-quality prompt image or set of images. To enable this, the authors propose the Image Prompt Score Distillation Sampling (IPSDS) loss and a mask guided compositional alignment module which enables the alignment of a prompt image with significantly differing content from the mesh/input image or to multiple prompt images. The authors show various qualitative and quantitative comparisons to state of the art approaches and superior performance of their approach in comparison to them in precisely optimizing for both the generated object's fine-grained texture and geometry. 

The strength of this paper is that it addresses a gap in the literature on 3D reconstruction in being able to edit the texture and fine grained shape of images using the style and appearance of high resolution prompt images. This is a practical workflow employed by digital artists in may 3D content creation tasks. Additionally the visual results of the proposed method are impressive. 

On the flip side, there are legitimate concerns about the speed of the method, which is optimization-based in comparison to the more recent faster feedforward methods; the method's inability to segment the prompt images correctly into parts and the inability of the method to strictly follow the structure of the provided input image in all circumstances.