Phidias: A Generative Model for Creating 3D  Content from Text, Image, and 3D Conditions with Reference-Augmented  Diffusion

Authors: Zhenwei Wang, Tengfei Wang, Zexin He, Gerhard Petrus Hancke, Ziwei Liu, Rynson W. H. Lau
Primary area: generative models
Keywords: 3D generation, retrieval-augmented generation, multi-view diffusion

Abstract:
Generative 3D modeling has made significant advances recently, but it remains constrained by its inherently ill-posed nature, leading to challenges in quality and controllability. Inspired by the real-world workflow that designers typically refer to existing 3D models when creating new ones, we propose Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Phidias integrates three key components: 1) meta-ControlNet to dynamically modulate the conditioning strength, 2) dynamic reference routing to mitigate misalignment between the input image and 3D reference, and 3) self-reference augmentations to enable self-supervised training with a progressive curriculum.  Collectively, these designs result in significant generative improvements over existing methods. Phidias forms a unified framework for 3D generation using text, image, and 3D conditions, offering versatile applications.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729070725905

thread_type:
official_review

<thread_object>
creator: reviewer_ef4x

<content>
summary:
This paper proposes a new 3D generation method that introduces a meta-ControlNet and dynamic reference routing, leveraging user-provided 3D reference models and images to produce high-quality, controllable 3D object synthesis.

strengths:
- The introduction of meta-ControlNet, combined with 3D reference models, provides strong guidance for the generation process, resulting in high-quality 3D outputs with excellent geometric consistency.

- As a reviewer familiar with this field, I believe that the combination of 3D reference models and images for generating 3D results has significant industrial value and could greatly inform future work in 4D generation as well.

- The method is simple, easy to understand, and the paper is clearly written.

weaknesses:
- Obtaining a reference 3D model can be quite challenging. Although this is valuable in certain application scenarios, it can be difficult to find appropriate reference 3D models in extreme cases. I recommend that the authors add this limitation to the paper.

- While the strategy of combining ControlNet for generation appears straightforward, this could be criticized for lacking novelty. However, in my opinion, simplicity often leads to elegant and effective solutions, and I don’t consider this a reason for rejection.

questions:
- I strongly support the task of combining reference 3D models with images for 3D generation as proposed in the paper, especially considering its relevance to the gaming industry and asset creation. Some related 3D generation methods discuss a texture editing task using complex images to guide coarse 3D object generation, which is highly relevant to this work. Could the authors provide some discussion or analysis on how to improve the method when there are significant differences between the reference image and the reference 3D model? This could be addressed in future work.

- I am inclined to accept this paper, as I also work on related tasks. Currently, I rate it as 6 but would likely increase the rating to 8 during the rebuttal phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729585312000

thread_type:
official_review

<thread_object>
creator: reviewer_ena9

<content>
summary:
This paper introduces Phidias, a RAG-based framework for high-fidelity 3D content creation from multimodal inputs, including text, images, and 3D references. The framework introduces key innovations include Meta-ControlNet to dynamically modulate reference influence, dynamic reference routing to progressively refine outputs through a coarse-to-fine strategy, and self-reference augmentation to enhance generalization. Phidias enables sparse-view reconstruction, theme-based model variation, and user-interactive generation, demonstrating clear superiority over state-of-the-art approaches in both geometric precision and visual fidelity.

strengths:
1. This paper presents a well-founded solution for multi-view generation, effectively addressing challenges such as the generation of unseen views and other ill-posed problems.
2. This paper is well-organized and clearly written, making it accessible to readers.

weaknesses:
1. A significant weakness is the limited analysis of the retrieval mechanism for 3D references, which plays a critical role in guiding generation. The reliance on existing 3D models introduces a potential bottleneck, as Phidias’s performance may degrade if the retrieved references are irrelevant or of poor quality. 
2. The control mechanism introduced via Meta-ControlNet is promising, but the paper does not delve deeply into more challenging settings. For example, scenarios where the input concept conflicts with the reference.

questions:
1. why the point cloud for retrieval only contains 10K points, is it limitation stems from constraints imposed by Uni3D?
2. Why are the performences in Figure 14 so different from the results of the last row of figures in Figure 4? It looks like that some case's pose in Figure 4 doesn't match the input image, either.
3. The time and resource consumption for retrieval during each inference is not clearly stated
</content>
</thread_object>

</thread>
<thread>
creation date:
1729834266559

thread_type:
official_review

<thread_object>
creator: reviewer_mgwn

<content>
summary:
The paper introduces a generative model designed for creating 3D content from text, image, and 3D conditions with reference-augmented diffusion. Phidias aims to enhance the quality, generalization, and controllability of 3D generation by leveraging existing 3D models as references. The model integrates three key components: meta-ControlNet for dynamically adjusting conditioning strength, dynamic reference routing to address misalignments between input images and 3D references, and self-reference augmentations for self-supervised training with a progressive curriculum. 

These components collectively improve generative performance over existing methods, offering a unified framework for 3D generation using diverse inputs. Phidias enables various applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.

strengths:
1. This paper proposes a dynamic reference routing approach. Recognizing that reference models do not perfectly correspond to the conceptual image, this paper utilize a feature pyramid to address this discrepancy. The motivation is that, it has been widely observed during the reverse diffusion process that the coarse structure of a target image is established in the early, high-noise timesteps, while finer details emerge progressively as the timesteps advance. This motivates to start with low-resolution reference CCMs at high noise levels.
2. This paper facilitates a range of downstream applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.

weaknesses:
1. Dependence on Retrieval Quality: As stated by the author The performance of Phidias is contingent upon the quality and relevance of the retrieved 3D reference models, which may not always be optimal.
2. Database Limitations: The current 3D database used for retrieval is limited in size, which could restrict the diversity and accuracy of reference models available for augmentation. Moreover, the model necessitates an additional dataset for inference, which may further restrict its applicability.

questions:
1. In line 186, there is a missing citation regarding the canonical coordinate map. I believe this map is also referred to as the NOCS map, as introduced in reference [a].

[a] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668168368

thread_type:
official_review

<thread_object>
creator: reviewer_tcl1

<content>
summary:
This paper presents a new 3D generation approach that utilizes a reference 3D model to enhance the generation process. The method comprises three key components: (1) Meta-ControlNet (2) Dynamic Reference Routing (3) Self-Reference Augmentation. Experimental results demonstrate the effectiveness of this method in enhancing 3D generation quality, generalization ability, and controllability.

strengths:
- Gathering information from a 3D reference model is straightforward.
- The evaluation of generation quality shows the effectiveness of the approach.
- The paper is well-written and easy to follow.

weaknesses:
- In the evaluation, only generation quality is assessed, with no evaluation of generation speed. For Phidias, retrieving information from the reference 3D model requires both 3D model retrieval and rendering of the canonical coordinate maps, which introduces additional latency. However, the paper does not report specific latency overheads.
- From Table 3, the effectiveness of dynamic routing appears to be limited.

questions:
- With the dynamic reference routing mechanism, the meta-ControlNet should be capable of handling inputs at various resolutions. Is the meta-ControlNet trained on images of different resolutions, or is it trained solely on the highest resolution and expected to generalize to lower resolutions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695803799

thread_type:
official_review

<thread_object>
creator: reviewer_phbc

<content>
summary:
The paper present a reference-augmented method using retrieved 3D objects as auxiliary guidance to reduce the ill-posed nature of the single-view 3D reconstruction task. The method leverages multi-view diffusion zero123++ for the generation of 6 novel view images, and scale-up the LGM for 3D reconstruction. The main contribution of the paper is to use input concept image (typically lack of 3D information of the back or side view, etc.) to refer a similar 3D objects from their 3D database (40K objects selected from Objaverse), the 6 canonical coordinate maps (CCM) from referred 3D object is used to generate better multi-view images. Meta-ControlNet with dynamic resolution of the intake CCM during diffusion sampling contribute to the misalignment dilemma between the concept images and referred 3D object. The results show that the method outperform other SOTA models in image-to-3D tasks given the proper 3D object reference. And it may extend to text-to-3D, and 3D-to-3D applications.

strengths:
1. The framework extends the controllability of current image-to-3D models. The model retrieves an auxiliary 3D object as guidance to improve the 3D reconstruction quality. The results shows outperformance than current SOTA models.

2. The paper propose meta-controlnet, and self-reference inference to let model learn how to resolve the conflict between retrieved objects and concept front-view image. Given the setting by the paper claimed, the model is able to provide reasonable 3D generation results matches on both sides.

3. The model is able to extend to text-to-3D and 3D-to-3D generation tasks, which may contribute to more broad applications.

weaknesses:
1. (model behavior) The model is trained with self-augmented ground-truth 3D objects, in order to handle the misalignment dilemma. However, as the A.4 specified, the augmented ones are still very similar to the ground truth. Moreover, all showcases are with no explicit conflicts between concept image and 3D objects. This raises the question of how a bad retrieval influence the model performance, which is actually common in practical case (as line 878-879). More comprehensive analysis or comparisons are necessary.

2. (retrieval robustness) The quality of the retrieval directly influence the improving of the generation quality, especially on the back or side views. I elaborate in detail in Question-1. If the user have either a out-of-retrieval-database front-view concept image, or the retrieval brings an unreasonable objects, the model output may not be as good as claimed. As Fig.7 shows, random reference as an example of non-similar retrieval may provide unreasonable results. This restricts the usage of the method.

3. (restricted/unfair experiment setting) Followed by weakness-1, the paper constrains the input image as only-front-view (line 221-222) to all competitor models. A random view, or 2-4 views as other related works defined, will contain more geometric information of the 3D details of the object. If not restricting to only-front-view input, using other models which do not require an auxiliary 3D object input like OpenLRM may already provide reasonable results. Their results may be better than the results with no reference in Fig.1, and Fig.7, and SOTA results in Fig. 5.

4. (restricted/unfair experiment setting) In the model evaluation section, all showcase examples meet the optimal situation of the framework. (a) The retrieved 3D objects is very similar to the concept image in the front view, (b) there is no/little conflicts on the other views. Given that, the improvement of the quantitative results in Tab.1 is still marginal. The row "Ours (GT Ref.)" may not be a reasonable comparison, since it provides ground truth geometric information in 6-view. I am not surprise it provides better generation results from the unseen view.

5. (model usage) When there is considerable conflict between retrieved 3D objects and concept images (as line 878-879, it is actually very common), which side would the model follow the most? Potential users may want to see more examples of practical cases. And from a point of view as graphics designer, it will be better to provide a controlling nob for user to decide the extent of guidance from either side, rather let the model itself decides the extent. Right now, it is still uncontrollable.

6. (novelty) The novelty of the paper concentrated on the meta-controlnet and reference-based inference. However, given the weakness aforementioned, it may solve an restricted task or with limited contribution to the real problem, and compare to SOTAs in an unfair setting. Moreover, the reason to use diffusion models, and following reconstruction model may need more elaborations.

questions:
Great work! I elaborate more based on weakness, and provide some suggestions and questions.

1. What is the main research goal of the paper? Is it to improve the quality of the 3D reconstruction on un-seen views (which indicates the output model should faithfully obey the input concept image), or it provides more customization freedom by given auxiliary 3D object reference (which indicates the output should lean to the given 3D objects.). Based on a full reviewing of the paper, my understand is the paper lean to the former one (correct me if I am wrong). So when there is conflict between retrieved 3D objects and concept images, which side would the model follow the most? If I am a graphics designer, a useful one will be a model doing either way, rather than a model-decided interpolation between two sides.
 
2. Is it a correct understand that the meta-controlnet is actually to learn a 3D texture mapping, which utilize the pixel-based texture in the concept image, map it to the retrieved 3D object surface based on 6-view CCMs? In that, the best way to phrase the contribution may like "using the front-view image as 2D texture guidance to generate unseen views given the 6-view CCM priors".

3. Follow Question-2, would you elaborate more about the role and reason to use diffusion model here? My understanding is, because there is conflict if we directly use 3D texture mapping using concept images and 6-view CCMs, so you let the diffusion process handle those conflicts. Any reason to exclude other pixel-based networks or traditional CG methods?

4. There are a lot of necessary details in the supplementary, like details of retrieving, dataset, etc. It will be better to move them into the main paper, even some of details may raise questions from reviewers. There is still a half-page blank now.

 I reserve the change of grades upon authors' responses to my reviewing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732072105453

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We would like to thank all reviewers for their valuable comments. We appreciate that reviewers recognize our ideas for RAG-based 3D generation as novel, with good results and various downstream applications. **We have revised our paper based on the reviewers’ comments in red (see the updated pdf).** Before we proceed with a detailed, point-by-point response for each reviewer, we hope that the reviewers and the Area Chair could browse through this blog content first, which primarily addresses potential misunderstandings and similar concerns about our paper. For other individual questions, we will answer one-by-one under each reviewer’s section.

**1. Regarding the potential impact of our paper**

As the first work that brings the concept of retrieval-augmented generation (RAG) into the field of 3D generation, we achieve significant improvements in generative quality, generalization ability, and controllability. We kindly invite the reviewers and the Area Chair to watch the attached demo video in the supplementary if you haven’t watched it yet. Note that these promising results are achieved using a small retrieval database of only 40K 3D objects, which could be significantly enhanced if we further scale up this database (L304). We believe our paper has industrial values and could greatly inform future work in RAG-based 3D/4D and video generation (as stated by Reviewer-Ef4X).


**2. Common Question 1: Generation robustness given dissimilar 3D reference with explicit conflicts (R-phbC, R-mgWn, R-eNa9, R-Ef4X)**

We have carefully designed the meta-ControlNet and dynamic reference routing for robust inference, given that the 3D references have different similarities. While dynamic reference routing focuses on dealing with local conflicts (L406-410), Meta-ControlNet is designed to produce adaptive control signals so that Phidias can globally adjust the reference strength given 3D reference with different similarity levels (L229-232). Theoretically, with these two designs, Phidias can be robust to references that are significantly different from the concept image. We have also analyzed the influence of 3D reference with varying levels of similarity in Fig. 7 (a) and Sec. 4.2. Phidias can still generate plausible results even with a random reference (L422-424).

Some reviewers suggest including examples using references with more explicit conflicts. First, we have already included such cases in the Appendix, such as the gray robot (1st row, 1st col) and ceramic incense vessel (2nd row, 1st col) in Fig. 17 and the wooden pig (2nd row, 2nd col) and multi-feet robot (4th row, 2nd col) in Fig. 18. Second, as suggested, we further add additional results with more explicit conflicts in Fig. 7 (b) and discussions in L424-426 to demonstrate the robust performance of our method further. As can be seen, **the lower-bound performance of Phidias, when the reference is significantly different from the concept image, equals SOTA image-to-3D without reference**. Note that, in most cases, the retrieved 3D model will not be totally different from the image by retrieval. The performance would be more stable and significantly enhanced if the dataset were scaled to the whole Objaverse, ObjaverseXL, and other 3D databases. 

**3. Common Question 2: Inference speed (R-tCL1, R-eNa9)**

As suggested, we detail the inference speed in Appendix (A.3, L858). In short, similar to existing two-stage image-to-3D generation methods, Phidias can generate a final 3D model from the input image **in less than 10 seconds**, including 9 seconds for multi-view generation, 0.03 seconds for image-based 3D retrieval, 0.4 seconds for rendering the CCM maps, and 0.02 seconds for sparse-view 3D reconstruction. The inference time is tested using a single NVIDIA A100 GPU. The retrieval process is fast as we only need to compare the query vector of an input image (dim=1x1024) with precomputed embeddings of 3D models in the dataset (dim=Nx1024, where N is the size of the dataset) using cosine similarity, as stated in L783-785. The retrieval latency will not significantly increase even if we scale up the dataset thanks to the super-fast matrix multiplication using GPU.

**4. Summary of changes**

Please review the updated paper to see if your concerns are addressed. Notable changes include:
- We have included additional results in Fig. 7 (b) and discussions in L424-426 to demonstrate the robustness of Phidias given reference with explicit conflicts.
- We have added a new application in L483-485 and Fig. 12 to provide a “control knob” for users to decide the reference strength manually, as suggested by R-phbC.
- We have included the details of inference speed in A.3, L858.
- We have included additional discussions to further explain the results of Tab. 1 in L358-361.
- We have included the reference of “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation” in L183, as suggested by R-mgWn.


Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733116493425

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Invitation to Join Rebuttal Discussions (Last Day)

comment:
Dear Reviewer eNa9 and Reviewer mgWn,

I hope this message finds you well. Thank you once again for your valuable comments on our paper. **As the last day for reviewer responses is tomorrow, we kindly invite you to join the rebuttal discussions today.**

We have submitted a revised version of the paper, along with a common response for all reviewers, individual responses for each reviewer, and a demo video in the supplementary materials. We hope these responses and additional materials address your previous concerns.

We would greatly appreciate your engagement, feedback, and suggestions, which will help us refine our work further. If your concerns have already been addressed, please also consider increasing your ratings.

Thank you very much for your time and consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a unified approach for text or image guided 3D generation via multi-view diffusion models, which is further guided with a retrieved 3D object, for the very first time. In order to resolve the fundamental conflict caused by the incongruence between the guiding concept image and the 3D object, the authors propose three strategies to mitigate it: a meta-ControlNet to adaptively control the guidance from the 3D model's CCM images; using multiple resolutions for the 3D object as the reverse diffusion process proceeds and using a perturbation-based strategy for training with progressively more misaligned 3D objects in a self-reconstruction training setting. The work shows state-of-the-art performance and many novel downstream applications.

The significance of this work includes that it is the first unified framework to include a guiding 3D object for 3D generation, which can be helpful in many downstream application such as asset creation by artists in interactive settings. It is also broadly applicable to all 3D generation problems.