Phidias: A Generative Model for Creating 3D  Content from Text, Image, and 3D Conditions with Reference-Augmented  Diffusion
Authors: Zhenwei Wang
Tengfei Wang
Zexin He
Gerhard Petrus Hancke
Ziwei Liu
Rynson W. H. Lau
Venue: ICLR 2025 Poster (2025)
Keywords: 3D generation
retrieval-augmented generation
multi-view diffusion
Primary area: generative models
TL;DR: A 3D diffusion model with RAG, supporting reference-augmented 3D generation from text, image, and 3D conditions.

Abstract: Generative 3D modeling has made significant advances recently, but it remains constrained by its inherently ill-posed nature, leading to challenges in quality and controllability. Inspired by the real-world workflow that designers typically refer to existing 3D models when creating new ones, we propose Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Phidias integrates three key components: 1) meta-ControlNet to dynamically modulate the conditioning strength, 2) dynamic reference routing to mitigate misalignment between the input image and 3D reference, and 3) self-reference augmentations to enable self-supervised training with a progressive curriculum.  Collectively, these designs result in significant generative improvements over existing methods. Phidias forms a unified framework for 3D generation using text, image, and 3D conditions, offering versatile applications.

Published as a conference paper at ICLR 2025
PHIDIAS: A GENERATIVE MODEL FOR CREATING 3D CONTENT FROM TEXT, IMAGE, AND 3D CONDITIONS WITH REFERENCE-AUGMENTE DIFFUSION
Zhenwei Wang- 1 Tengfei Wang Zexin He Gerhard Hancke* Ziwei Liu Rynson W.H. Lau
1 City University of Hong Kong Shanghai AT Lab SCUHK 1S-Lab, NTU
ABSTRACT
Generative 3D modeling has made significant advances recently, but remains constrained by its inherently ill- ill-posed nature, leading to challenges in quality and controllability. Inspired by the real-world workflow that designers typically refer to existing 3D models when creating new ones, propose Phidias, novel gen- erative model that uses diffusion for reférence-augmented 3D generation. Given image, method leverages retrieved or uscr-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, gener- alization ability, and controllability. Phidias integrates three key components: meta-Controlle to dynamically modulate the conditioning strength, 2) dynamic reference routing to mitigate misalignment between the input image and 3D refer- ence, and 3) self-reference augmentations to enable self-supervised training with a progressive curriculum. Collectively, these designs result in significant generative improvements over existing methods. Phidias forms a unified framework for 3D generation using text, image, and 3D conditions, offering versatile applications. Project page: https: RRAG-AD 3D Bt..i...ggttttt github.io/.
image Input
Image Input
Retrieve
Retrieve
3D Reference
Gonerated 30 Model
3D Reference
Generated 3D Model
youngg onfrce roile girl
wire bive huir
Retrieee
Text Input
Generated Image
3D
Image variations
3D
Saiff Reff-Rfference
3D Reference
Generated 3D Variation
Generated Variation
Figure 1:
The proposed method, Phidias, can produce high-quality 3D assets given references. which can be obtained via retrieval (top two rows) specified by users (bottom row). It supports 3D generation from a single image, text prompt, or an existing 3D model.
INTRODUCTION
The goal of 3D generative models is empower artists and beginners effortlessly convert their design concepts into 3D models. Consider the input image in Fig. 1. A skilled craftsman can, through blend of skills and creativity, convert 2D concept image into an exquisite 3D model. This creative process may originate from artists' pure imagination more commonly, through examin- ing one or more existing 3D models as source of inspiration (Bob, 2022; Carvajal, 2023). Artists
Intern Shanghai AT Lab. "Equal Contribution. 'Corresponing Authors.

---

Published as a conference paper at ICLR 2025
often reference pre-existing 3D models to help improve the modeling quality. The question then arises: could we develop a reference- baased 3D generative model that can replicate this capability?
Over the years, a plethora of works (Wang al., 2023; Liu al., 2023b; Hong et al., 2023; Ben- sadoun et al., 2024) steadily expanded the frontiers of 3D generative models. These methods, while yielding stunning performance, still face several challenges. 1) Generation quality. A single image cannot furnish sufficient information for reconstructing full 3D model, due to the ambiguity of this ill-posed task. It necessitates the generative model to "hallucinate" the unseen parts in data-driven manner. However, this hallucination step can lead to view inconsistency and imprecise geometries that appear abrupt and unrealistic. 2) Generalization ability. These models often struggle with out-of out-of-donm oft-of---mmiin cases, such as atypical input views or objects, constrained by the data coverage of existing 3D datasets (Deitke et al., 2023). Meanwhile, the growing variety and quantity of object cat- egories exacerbate the difficulty for generative models to learn implicit shape priors, with limited model capacity v.s. an infinitely diverse array of objects. 3) Controllability. Due to the ambiguity, one input image can produce several plausible 3D models, each differing in shape, geometric style, and local patterns. Existing methods are constrained by limited diversity and controllability, which hinders the ability to predictably generate the desired 3D models.
To address these challenges, we propose to take 3D models as additional inputs to guide the gen- eration, inspired by the success of retrieval augmented generation (RAG) in language (Lewis et al., 2020) and image (Sheynin et al., 2022). Given an input image and a reference 3D model, we present Phidias, novel reference-augneenn augmented diffusion model that unifies 3D generation from text, image, and 3D conditions. As shown in Fig. I, the reference 3D model would help 1) improve generation quality by alleviating ambiguity with richer information for unseen views, 2) enhance generaliza- tion capacity by serving as a shape template or an external memory for generative models, and 3) provide controllability by allowing users indicate desired shape patterns and geometric styles.
Our method introduces referencee-nngeeteed multi-view diffusion model, followed by sparse-view 3D reconstruction. The goal is produce 3D models faithful to the input concept image, with improved quality by incorporating relevant information from the 3D reference. However, it is non- trivial to learn such a generative model due to the Misaligument Dilemma, where the discrepancy between the concept image and the 3D reference can lead to conflicts in the generation process. This requires our model to utilize the misaligned 3D reference adaptively. To tackle this challenge, Phidias leverages three key designs outlined below.
The first is Nentrolnn... Consider the 3D reference as conditions for the diffusion model. Unlike image-to-imen translation works (Zhang et al., 2023; Wang et al., 2022) that demand the generated images to closely follow the conditions, we treat the reference model as auxiliary guidance to provide additional information. The generated multi-view images are expected to be consistent with the concept image, without requiring precise alignment with the reference model. To this end, we build our method on ControlNet and propose meta-control network dynamically modulate the conditioning strength when it conflicts with the concept image, based on their similarity.
The second design is dynamic reference routing for further alleviating the misalignment. Rather than using the same 3D reference for the full diffusion process, we adjust its resolution across de- noise timesteps. This follows the dynamics of the reverse diffusion process (Balaji et al., 2022), which generates the coarse structure in high-noised timesteps and details in low-noised timesteps. Thus, we can alleviate the generation conflicts by starting with coarse 3D reference and progres- sively increasing its resolution as the reverse diffusion process continues.
The final key design is self-reference augmentations. As it is not feasible to gather a large set of 3D models together with their matching references, a practical solution is to use the 3D model itself as its own reference (iee, self-reference) for self-supervised learning. The trained model, however, may not work well when the 3D reference does not align with the concept image. To alleviate this problem, we apply a variety of augmentations to the 3D models to simulate this misalignment. In addition, also introduce a progressive augmentation approach that leverages curriculum learning for the diffusion model to effectively utilize 3D references with varying degrees of similarity.
Taken together, the above ingredients work in concert to enable Phidias to achieve stunning perfor- mances in 3D generation. Several application scenarios are thus supported: 1) Retrieval-augmented image-to-3D generation, 2) Retrieval-augmented text-to-3D generation, 3) Theme-aware 3D-to-31D) generation, 4) Interactive 3D generation with coarse guidance, and 5) High-fidelity 3D completion.
2

---

Published as a conference paper at ICLR 2025
From
Dynamic Reference Routing
Database User...
Meta- Controlleet
30 Reference
Sparse-View 3D
Control Signal
Protrained Multi-View Diffusiinn
Text
Image
3D
Concept Image
Mult-View Images
Input
Reference-Auumennted
Model
Figure 2: Overview of Phidias model. It generates a 3D model in two stages: (1) reference-
augmented multi-view generation and (2) sparse-view 3D reconstruction.
We summarize our contributions as follows: 1) We propose the first reference- based 3D-aware diffu- sion model. 2) We design our model with three key component designs enhance the performance. 3) Our model serves as a unified framework for 3D generation, which enables a variety of applica- tions with text, image, and 3D inputs. 4) Extensive experiments show that our method outperforms existing approaches qualitatively and quantitatively.
2 RELATED WORKS
Image to 3D. Pioneering works (Melas-Kyrian Kyriazi al., 2023; Tang et al., 2023; Chen al., 2024b) perform 3D synthesis by distilling image diffusion priors (Poole et al., 2023), but are time- consuming. Recent advancements have leveraged feed-forward models with 3D datasets. Some works use diffusion models to generate points (Nichol et al., 2022), neural radiance fields (Wang al., 2023; Jun Nichol, 2023; Gupta al., 2023; Hong et al., 2024). SDF (Cheng et al., 2023; Zhang et al., 2024b), and gaussian splatting (Zhang et al., 2024a). Another line of works uses trans- formers for auto-regressive generation (Siddiqui reconstruction (Hong et al., 2023; Tang et al., 2024; Zou al., 2023; Wang et al., 2024a; Xu et al., 2024), which often rely on multi-view diffusion for better performance.
al., 2023; Chen al., 2024a) or sparse-view
Multi- View Diffusion Models. Multi-view models reduce the complexities of 3D synthesis to con- sistent 2D synthesis. Seminal works (Liu al., 2023b) have shown novel view synthesis capabilities with pre-trained image diffusion models (Rombach et al., 2022). Later, plethora of works explored multi-view diffusion models with better consistency (Shi et al., 2023a; Wang Shi, 2023; Shi al., 20236; Long et al., 2023; Liu et al., 2023a) by introducing cross-view communication. More recent works (Voleti et al., 2024; Chen et al., 2024c; You et al., 2024; Han et al., 2024) leverage video pri- for multi-view generation by injecting cameras into video diffusion models. However, they still struggle with generalized and controllable generation due to the ill-posed nature of this problem.
Reference-Auumennte Generation. Retrieval-augmenter generation (RAG) emerges enhance the generation of both language (Lewis et al., 2020) and image (Sheynin et al., 2022; Blattmann et al., 2022) by incorporating relevant external information during the generation process. Under the context of 3D generation, the concept of reference-based generation is also widely applied. Some works (Chaudhuri et al., 2011; Kim et al., 2013; Schor et al., 2019) probe into the database for compatible parts and assemble them into 3D shapes. Some works (Xie et al., 2024; Richardson et al., 2023; Ych et al., 2024; Zeng et al., 2023; 2024) utilize reference texts or images as guidance generate textures for existing 3D meshes. Some works refer to a 3D exemplar model (Wu & Zheng, 2022; Wang et al., 2024b) to produce customized 3D assets. Despite success in specific contexts, they are cime-consuming with per-case optimization. contrast, our method focuses on learning generalized feed-forward model that applies to reference-augmenter 3D generation.
APPROACH
Given concept image, aim leveraging additional 3D reference model alleviate 3D inconsistency issues and geometric ambiguity that exist in 3D generation. The 3D reference model be either provided by the user retrieved from large 3D database for different applications.
3

---

Published as a conference paper at ICLR 2025
(a) Mita-ControoNNet
(b) Dynamic Routing
Concent
ProsiAfie
CettaCottrfeer
CCM
Zero
Adaptive Control Signal
Zero Coriee
Noke leves
Middle koise Leweh
High Noke Level:
Mult-Siale nlgement Feations
Reforance
CCM
Midil kes. COM
Low Res
Zoro
Figure 3: Architectural designs for (a) meta- Meta-ControllNe and (b) dynamic reference routing.
The overall pipeline of Phidias is shown in Fig. 2, which involves two stages: reference-augmented multi-view generation and sparse-view 3D reconstruction.
3.1 REFERENCE AUGMMETEED MULTI- VIEW DIFFUSION
Multi-view diffusion models incorporate camera conditions into well-trained image diffusion mod- els for novel-view synthesis with supervised fine-tuning, We aim to weave additional 3D references into these multi-view models for better generation quality, generalization ability, and controllability. Our approach can be built on arbitrary multi-view diffusion models, enabling reference- augmmeete 3D content creation from text, image, and 3D conditions. Specifically, we initialize our model with 7ero123++ (Shi et al., 2023a), which simply tiles multi-view images for efficient generation condi- tioned on one input image Cimage
To integrate 3D reference models Cref into the diffusion process, we transform them into multi-view canonical coordinate maps (CCM) (Wang et al., 2024a), also known as NOCS maps (Wang et al., 2019), to condition the diffusion model. The choice of CCMs as the 3D representation is based on two reasons: 1) Multi-view images serve
efficient and compatible inputs for diffusion models than meshes or voxels, as they have embedded camera viewing angles that correspond with the output images. 2) Reference models often share similar shapes with the concept image but vary significantly in texture details. By focusing on the geometry while omitting the texture, CCMs conditions can reduce generation conflicts arising from texture discrepancies. We add conditioner branch to incorporate reference CCMs into the base multi-view diffusion model. The objective for training diffusion model Eg can be then formulated as:
(1) To leverage the powerful pertaining capability, only the additional conditioner for reference CCMs trainable while the base multi-view diffusion is frozen. However, challenge in task is that the 3D reference may not strictly align with the concept image or, more commonly, vary in most local parts. We found naive conditioner designs such as ControlNet (Zhang et al., 2023) tend to produce undesirable artifacts, as they were originally designed for image-to-image translation where the gen- erated images strictly align with the condition images. To mitigate this problem, introduce three key designs for our reference-augee augmented diffusion model: (1) Meta- ControlNer for adaptive control of the conditioning strength (Sec. 3.2); (2) Dynamic Reference Routing for dynamic adjustment of the 3D reference (Sec. 3.3); (3) Self- Reference Augmentation for self-supervised training (Sec. 3.4).
L E, (0(,. [[lc co (æz, f, Cinage: Crot) [
3.2 META-CONTROLNET.
ControlNet designed to add additional controls to pre-trained diffusion models for image-to-ing to-iag translation. The conditions are derived from the ground-truth images for self-supervised learning, and thus the generated images are expected to follow the conditions. However, our settings, the conditions are from the reference model, which often misaligns with the target 3D models want to generate. The vanilla ControlNet fails handle such cases. This necessitates further architecture advancement accordingly adjust conditioning strength when the reference conflicts with the con- cept image. To this end, we propose meta- meta-ControlNet.. as shown in Fig. (a). Meta-ControlNe is comprised of two collaborative subnets, base ControlNet and additional meta-controller.
Base ControlNet is comprised of an image encoder, a trainable copy down-sampling blocks and middle blocks of the base multi-view diffusion, denoted Thase (.), and series of 1 x
4

---

Published as a conference paper at ICLR 2025
convolution layers (Zero Convs) 2 (.). It takes reference CCM maps Cref input to produce the control signal. To deal with misaligned 3D reference, we introduce an additional meta- cont-controllrr to modulate the conditioning strength according to different similarity levels.
Meta-controller shares a similar architecture but has different parameters O', It works as a knob that dynamically modulates base ControlNet generate adaptive control signals. Meta-controller takes pair Spair of the concept image and the front-view reference CCM as input to produce meta-control signals based on their similarities. The meta-control signals are injected into diffusion models in two ways. On the one hand, meta-controller produces multi-scale alignment features ymeetl 2 Zmotall (Fey (zpaiz)) to be injected into base ControlNet. These features are applied the down- sampling blocks of base ControlNet (Eq. 2) at each scale to guide the encoding of reference and help produce base-signals as:
Ybase
Zbase
(Foase (ymetall Zreot).
(2)
where Zref and paii are the feature maps of Cref and Cpair via the trainable encoders in Fig. 3 (a).
On the other hand, meta-controller produces meta-signals ymeta2 = 2gr (For (zpaiz) to be injected to the pretrained multi-view diffusion models. These features added up to base- signal Ybase to directly apply for the pretrained diffusion models. Totally, the final outputs of meta- ControlNet are adaptive control signals Yadaptive based the similarity between the concept image and the 3D reference, as:
Yadaptive
Xyyhase + 3metn2)
where À is the coefficient to adjust the strength of control signal for different applications.
3.3 DYNAMIC REFERENCE ROUTING
Reference models typically align roughly with the concept image in terms of coarse shape, but diverge significantly in local details. This misalignment can cause confusion and conflicts, as the generation process relies on both the image and reference model. To address this issue, we propose a dynamic reference routing strategy that adjusts the reference resolution across denoise timesteps, as shown in Fig. 3 (b). As widely observed during the reverse diffusion process, the coarse structure of target image is determined in high-noised timesteps and fine details emerge later as the timestep goes on. This motivates us to start with low- resolutttni reference CCMs at high noise levels t4 By lowering the resolution, reference models provide fewer details but exhibit smaller misalignment with the concept image. This enables reference models to assist in generating the global structure of 3D objects without significant conflicts. We then gradually increase the resolution of reference CCMs as the reverse diffusion process goes into middle noise levels i'm and low noise levels tı to help refine local structures, e.g., progressively generating a curly tail from a straight one (Fig. 3 (b))- This design choice would ensure effective usage of both concept image and 3D reference during the multi-view image generation process while avoiding degraded generation caused by misalignment.
3.4 HELF-REFERENCE AUGMENTATION
A good reference model should resemble the target 3D model (with varied details) to provide addi- tional geometric cues, but it is impractical to collect sufficient target-reference pairs for training. An intuitive solution is to retrieve a similar model from a large 3D database as the training reference. However, due to the limited variety in current databases, finding a perfect match is challenging. The retrieved reference can vary greatly in orientation, size and semantics. While this is a common situ- ation in inference scenarios, where very similar reference is often unavailable, we found training with these challenging pairs fails to effectively use the 3D reference. We conjecture that the learning process struggles due to the significant differences between the reference and target 3D, leading the diffusion model to disregard the references. To avoid the 'idleness' of reference, we developed a self-reference scheme that uses the target model as its own reference by applying various augmen- tations to mimic misalignment (refer to Appendix A.5). This approach ensures that the reference models are somewhat aligned with the target and more compatible, alleviating the learning difficulty. We further design curriculum training strategy, which begins with minimal augmentations (very similar references) to force the diffusion model to rely on the reference for enhancement. Over time, gradually increase augmentation strength and incorporate retrieved references, challenging the diffusion model to learn from references that do not closely match the target. Once trained, our model performs well with variety of references, even those retrieved ones that are not very similar.
5

---

Published as a conference paper at ICLR 2025
P
Input Image
3D Reference
Retrieved
Generated Model
3D Reference
Retrieved
Generated Model 2
Figure 4: Diverse retrievallaugmmentdd image-to-3ll results. With a single input image, Phidias can generate diverse 3D models with different 3D references.
3.5 SPARSE-VIES 3D RECONSTRUCTION
With multi-view images generated the first stage, we can obtain final 3D models via sparse- view 3D reconstruction. This step can be built upon arbitrary sparse-view reconstruction models. Specifically, we finetune LGM (Tang et al., 2024) by expanding the number input views from to and the resolution of each view from 256 x 256 to 320 320 that the trained reconstruction model aligns with the multi-view images generated in our first stage.
4 EXPERIMENTS
In this section, we evaluate our method on image-to-3D generation, significant area in 3D gen- eration research. For each image, retrieve 3D reference model from a 3D database based on similarity (Zhou et al., 2024). The database used is subset Objaverse, containing 40K models. We anticipate that performance could be further enhanced with a larger database in the future. For the rest of this section, we compare Phidias with state-of-the-arr methods and conduct ablation anal- ysis. More results and implementation details can be found in Appendix. Results text-to-3D and 3D-to-3D generation can be found in Sec. 5.
4.1 COMPARISONS WITH THATE-OF-THE- METHODS
We compare Phidias with five image-to-3D baselines: CRM (Wang al., 2024a), LGM (Tang et al., 2024), InstantMesh eXu al., 2024), SV3D (Voleti et al., 2024), and OpenLRM (He & Wang, 2023).
Qualitative Results. For visual diversity (Fig. 4), given the same concept image, Phidias can gener- ate diverse 3D assets that are both faithful to the concept image and conforming to specific retrieved 3D reference in geometry. For visual comparisons (Fig. 5), while the baseline methods can generate plausible results, they suffer from geometry distortion (e.g., horse legs). Besides, none of the exist- ing methods can benefit from the 3D reference for improved generalization ability (e.g., excavator's dipper) and controllability (e.g., cat's tail) ours.
Quantitative Results. Following previous works, we conduct quantitative evaluation on google scanned objects (GSO) (Downs et al., 2022). We remove duplicated objects with the same shape

---

Published as a conference paper at ICLR 2025
Ours Image + 3D
Input
OpenLRM
CRM
InstantMiesh
SV3D
Figure 5: Qualitative comparison on image-to- Soage-to-3DD 3D generation.
Table 1: Quantitative comparison with baselines image-to Image-to-3D synthesis.
Method
PSNR
LPIPS
CLIP-P
CLIP-11
F-Score
OpeaLRM
16.15
0.194
0.866
0.847
0.805
LGM
14,80
0,807
0.219
0.869
0.871
0,0398
0.831
CRM
16.35
0.841
0.182
0.855
0.843
0.0443
0.796
SV3D
16.24
0.838
0,203
0.879
0.866
InstantMesh
14.63
0,796
0.235
0.882
0.880
0.0450
0.788
Ours (GT Ret.)
20.37
0,870
0.117
0,911
0.885
0,0391
0.840
Ours (Retrieved Ref.)
17.02
0.845
0.174
0.887
0.885
0.0402
0.833
and randomly select 200 objects for evaluation. For visual quality, report reconstruction met- rics (PSNR, SSIM and LPIPS) on 20 novel views. We also report novel views' CLIP similarity with paired GT (CLIP-P) and input image (CLIP-I). For geometry quality, sample 50K points from mesh surface and compute Chamfer Distance (CD) and F-Score (with a threshold 0.05). To align the generated mesh and GT, unify their coordinate systems and re-scale them into unit box. We report our results with the retrieved reference, i.e., Ours (Retrieved Ref.), and GT mesh as reference, i.e., Ours (GT Ref.), respectively. As shown in Tab. 1. ours, with either retrieved GT reference, outperforms all baselines, benefiting from the proposed augmenel- method. The results of Ours (Retrieved Ref.) seems marginal although Phidias produces plausible 3D models given various references that differ from GT (Fig. 7 (a)). We argue that this is caused by the dif- ferences between the retrieved references and GT when computing the reconstruction metrics. The actual performance of Phidias should be between Ours (GT Ref.) and Ours (Retrieved Ref.).
User Study. We further conduct a user study to evaluate human preferences among different meth- ods. We publicly invite 30 users to complete questionnaire for pairwise comparisons. We show the preference rate (i.e., the percentage users prefer ours compared to a baseline method) Tab. 2, which suggests that approach significantly outperforms existing methods in the image-to-3D task based on human preferences.
4.2 ABLATION STUDY AND ANALYSIS
Ablation Studies. We conduct ablation studies across four settings: a base model employing a standard ControlNet trained with self-referenee and three variants (each integrating proposed component into the base model). The quantitative results in Tab. demonstrate clear improvements in both visual and geometric metrics with our proposed components.

---

Published as a conference paper at ICLR 2025
Table 2: User study,
Table 3: Quantitative ablation study of the proposed components.
Baseline
Pref. Rate
Method
PSNR
SSMM
I.PPS
CLIPP-P
CLIP-I
CD
F-Score
OpenLRM
Model
0.0424
LGM
95.8%
Metn- ControlNet
16.35
0,833
0.190
0.881
0.878
0.0407
0.829
CRM
93.7%
Dynamic Rosting
14.76
0.816
0.221
0.0420
0.826
SV3D
88.4%
Self-Ref. Augmentation
16.57
0.840
0.182
0,880
0.883
0,0414
0.830
InstamiMesh
91.6%%
Full Model
17.02
0.845
0.174
0.887
0.885
0.0402
0,833
(a) Meta-ControlNet
Inputs
Base Model Retrieval
Mett--CottoolNet
(b) Dynamic Reference Routing
Inputs
Base Model
Dynamic Reference Routing
Self-Reference Augmentation
Inputs
Base Model
Self-Reference Augmentation
Figure 6: Qualitative ablation study of the proposed components.
Effectiveness of Ceta--ontroolee.. To evaluate meta-ControlNet, we use both self-reference and retrieved reference for training, the learning of Meta-Controller (Fig. (a) top) requires reference models with varying levels similarity. As shown in Fig. 6 (a), the base model trained with retrieved reference often ignores the reference, failing follow the shape pattern (disconnected boat). This phenomenon stems from the considerable similarity variation among retrieved references, which confuses the diffusion model. The base model thereby struggles determine when and how to use the reference as it lacks the ability to adjust to different levels of similarity. Consequently, they often end up with ignoring the reference models entirely. In contrast, meta Cetta-CottroNett equips the model with the capability to dynamically modulate the conditioning strength of the reference model, thereby effectively utilizing available references for improving or controlling the generation process.
Effectiveness of Dynamic Reference Routing. Dynamic reference routing aims to alleviate local conflicts between the reference and concept images. As illustrated in Fig. 6 (b), when given a highly similar reference, the base model tends rely heavily on it, leading missing specific local details within the concept image, e.g., the rope on the left. By addressing these conflicts with dynamic routing, the model maintains the essential details of the concept image, while still benefiting from the guidance of the 3D reference.
Effectiveness Self-Reference Augmentation. As shown in Fig. 6 (c), without self-reference aug- mentation, the base model predominantly depends on the provided reference for generation. When given significantly misaligned reference, the model tends to follow the reference's 's structure, sulting in an undesired outcome. Conversely, self-reference augmentation ensures that the generated models remain faithful to the concept image, while using the reference geometry guidance.
Analysis on Similarity Levels of the 3D Reference. We analyze how similarity levels of 3D ref- erences would affect the performance. For each input, we retrieve three models ranked first (top-1) third (top-3), and fifth (top-5) in similarity scores, and randomly choose one model, to serve as 3D references. Quantitative results in Tab. indicate that Phidias performs better with more similar ref- erences. Fig. 7 (a) shows Phidias generates diverse plausible results with different references. All results remain faithful to the input image in the front view, but show variations shapes influenced by the specific reference used. Also, we found Phidias can still generate plausible results even with random 3D reference (Fig. 7 (a) middle), indicating robustness to reference with different similarity levels. Moreover, shown in Fig. (b), Phidias will ignore inappropriate 3D reference with explicit conflicts and generate satisfying results following the imput image, demonstrating the robustness to some extreme cases when the retrieved 3DD reference significantly differs from the image.
8

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative analysis on similarity levels of the 3D reference.
Reference
PSNR 1
SSIM
LPIPS
CLIPPP
CLIP-1 T
CD 4.
F-Score
Top-1 Retrieval
17.02
0.845
0.174
0.887
0,885
0,0402
0,833
Top-3 Retrieval
16.75
0.841
0.172
0.887
0.886
0.0395
0.830
Top-5 Retricval
0.835
0.0408
Random Reference
14.74
0.820
0.226
0.884
0
0.0424
0.810
Without Reference
[a] 3D Reference with Various Similarity Levels
No 3D Reference
Random 3D Reference
GT Reference
Input Image
Top-3
Top-5
(b) 3D Reference with Explicit Conflicts
Input Image
3D Reference
Generated 3D Model
Input Image
3D Reference
Generated 3D Model
Figure 7:
Qualitative analysis similarity levels of the 3D Reference.
"Gray fnmaood
"Large. amatte warioae Chiosee buildingg
character with goiden hooded
Input
3D Reference
Generated 3D Model
Input
3D Reference
Generated 3D Model
Figure 8: Phidias enables retrieval-augmmeneed text-to-31 -D generation by first converting the input text into concept image, and then retrieving 3D reference based on both the text and image.
APPLICATIONS
Phidias supports versatile applications beyond image-to-3D, such as text-to-3D, theme-awane aware 3D- to-3D, interactive 3D generation with coarse guidance, and eigh-fidlity 3D completion.
Text to 3D. Text-to-3D generation can converted to image-conditions generation by transform- ing text prompt into concept image. However, the generated concept image sometimes be atypical and may lose some information compared with original text input. To enhance generative quality, Phidias employs retrieval-augmented text-to-3D generation, as illustrated in Fig. This involves first retrieving a set of 3D references based on the concept image, and then selecting the that most closely matches the text description the final reference.
Theme-Aware 3D-to-3D Generation. This task aims to create a gallery of theme-consistent 3D variations from existing 3D models. Previous work (Wang et al., 2024b) proposed an optimization- based approach, which is time-consuming Phidias supports fast generation by first generating im- age variations based on the input 3D model, and then transforming these variant images into 3D variations with the original 3D model itself as reference. The results are shown in Fig. 9, using 3D models from Sketchfab and previous works as inputs.
Interactive 3D Generation with Coarse Guidance. Interactive generation gives users more control over the outputs, empowering them to make quick edits and receive rapid feedback. Phidias also provides this functionality, allowing users continually adjust the geometry of generated 3D models using manually created coarse 3D shapes as reference models, as shown in Fig. 10.
High-Fidelity 3D Completion. Given incomplete 3D models, shown in Fig. 11, Phidias can be used to restore the missing components. Specially, by generating a complete front view through image inpainting and referencing the original 3D model, Phidias precisely predict and fill in the missing parts in novel views while maintaining the integrity and details the origin, resulting in seamlessly and coherently structured 3D model.

---

Published as a conference paper at ICLR 2025
3D Input
Self-Reference
Image Variations
Generated 3D Variation
Generated 3D Variation
Figure 9: Phidias facilitates rapid, theme- aware 3D-to-3D generation by using an existing 3D model as reference to transform its image variations into corresponding variations.
Input Image
Coarse Shape
Generated
Coarse Shape
Generated 3D
Figure 10: Phidias enables interactive 3D generation with coarse 3D shapes as guidance.
Self Reference
Image Inpainting
Incomplete 3D
Completed Front Views
Completed 3D
Figure 11: Phidias supports high-fidelin 3D completion by using the completed front views guide the restoration of the missing parts and the original 3D model to help preserve the original details.
Input Image
30 Reference
=
2-1
2-2
-
Figure 12:
Phidias allows users to explicitly decide the reference strength during inference.
3D RAG with Adjustable Reference Strength. Phidias offers users enhanced control over the ref- erence strength by adjusting À in Eq. This feature works as "control knob" to vary the emphasis on the reference, which is particularly useful for scenarios like image-based 3D re-texturing and interpolation between the concept and reference, as illustrated in Fig. 12.
CONCLUSION
In this work, we introduce Phidias, 3D-aware diffusion model enhanced by 3D references. By corporating meta-ControlNer dynamic reference routing, and self-referene augmentations, Phidias effectively leverages reference models with varying degrees similarity for 3D generation. The proposed approach boosts the quality of 3D generation, expands its generalization capabilities, and improves user control. Phidias offers unified framework for creating high-tyality 3D content from diverse modalities, which enables versatile applications beyond image-to-3D, such as, but not lim- ited to, text to 3D, theme-aware 3D-to-3D generation. interactive 3D generation, and high-fidelity 3D completion. More discussions of limitations and failure cases can be found in Appendix B.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the National Key R&D Program of China (2022ZD016020) and Shanghai Artificial Intelligence Laboratory. This work is also in part supported by a GRF grant from the Research Grants Council of Hong Kong (Ref. No.: 11205620).
REFERENCES
Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint urXiv:2211.0133 2022.
Raphael Bensadoun, Tom Monnier, Yanir Kleiman, Filippos Kokkinos, Yawar Siddiqui, Mahendra Kariya, Omri Harosh, Roman Shapovalov, Benjamin Graham, Emilien Garreau, al. Meta 3d gen. arXiv preprint arXiv:2407.02599 2024.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval- augmented diffusion models. Advances Neural Information Processing Systems, 35:15309 15324, 2022.
Bob. 3D modeling 101: Comprehensive beginners guide, 2022. URL https:// /tt////wwwooooooooooo hww.hhh...oooooooooo. articles/3d mxtiieles/ddmndellnggg 11111111-amiii comprehensive 1tt.-p-------coppreeesseieeeeee www.reeeensiiee-beeeeeeeee
Carlos 2023.
Carvajal.
The
importance
of references in 3d projects,
https://wee https://wwww.ikkeee... www.keii...c..mmm cttpiinncco/c/e////////p importance- references 3d crrojects-cariie -araaaacaaooaoaaaalala carvajal/.
URL
Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Probabilistic reasoning for assembly-based 3d modeling. ACM Trans. Graph., 30(4), jul 2011. ISSN 0730- 0301.
Yiwen Chen, Tong Hc, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, and Chi Zhang. Meshanything: Artist-created mesh generation with autoregressive transformers, 2024a.
Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Compositional 3d assets creation using spatially-awar diffusion guidance. ECCV, 2024b.
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.00 067388 2024c.
Yen-Chi Cheng, Hsin- Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4456-4465, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2553-2560. IEEE, 2022.
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05377 2023.
Junlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models from video diffusion models. European Conference on Computer Vision (ECCV), 2024.
Zexin He and Tengfei Wang. Openlmm: Open-source large reconstruction models. https://
gi githb... com 3DTopi httpp////pppp..... 2023.
11

---

Published as a conference paper at ICLR 2025
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang. Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint arXi::2403.0221 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311. 1.0:204000 04400, 2023.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Vladimir G. Kim, Wilmot Li, Niloy J. Mitra, Siddhartha Chaudhuri, Stephen Direrdi, and Thomas Funkhouser. Learning part-based templates from large collections of 3d shapes. ACM Truns. Graph., jul 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. eetrieval-augmenee genera- tion for knowledge-in wowleege-intensive intensive nlp tasks. Advances in Neural Information P Processing Systems, 33: 9459-9474, 2020.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint 1xxx:23311008880 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-to-1 Zero-shot one image 3d object. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2298-93091 2023b.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15000 2023.
Luke Melas- Mariaz,,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. RealFusion: 360 recon- struction of any object from a single image. 2023.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.0875// 2022.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In International Conference on Learning Representations (ICLR), 2023.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- guided texturing of 3d shapes. In ACM SIGGRAPII 2023 conference proceedings, pp. 1-11, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10699 2022.
Nadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen-Or. Componet:
Learning to generate the unseen by part synthesis and composition. In 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 8758-8767, 2019. doi: 10. N0.1109//CCV22211000088
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv preprint
MrXiv:2204.02849 2022.
Ruoxi Shi, Hansheng Chen. Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23+t: a single image to consistent multi-view diffusion base model. arXiv preprint arXi::2310.151111 2023a.
12

---

Published as a conference paper at ICLR 2025
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long. Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.1651/1 2023b.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder- only transformers. arXiv preprint arX1v:2311.1545,, 2023.
Sketchfab. Sketchfab. https: httts////eeetcfhhb.oooo
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Igm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22819-22829, Octo- ber 2023.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. 12008, 2024.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2642- 2651, 2019.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201, 2023.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang. Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. TrXiv:2205.12952 2022.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 4563-4573, 2023.
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.050:: 2024a.
Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Themestation: Generating theme-aware 3d assets from few exemplars. SIGGRAPH, 2024b.
Rundi Wu and Changxi Zheng. Learning to generate 3d shapes from single example. ACM Transactions on Graphics (TOG), (6)),, 2022.
Zhiyu Xie, Yuqing Zhang, Xiangjun Tang, Yiqian Wu, Dehan Chen, Gongsheng Li, and Xao- gang Jin. Styletex: Style image-guided texture generation for 3d models. arXiv preprint arXi::2411.00399 2024.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.0719) 2024.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martin Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 1179-1189, 2023.
13

---

Published as a conference paper at ICLR 2025
Yu- Ying Yeh, Jia-Bin Huang, Changil Kim, Lei Xiao, Thu Nguyen-Phuoe, Numair Khan, Cheng Zhang, Manmohan Chandraker, Carl Marshall, Zhao Dong, et al. Texturedreamer: Image- guided texture synthesis through geometry- diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7304-4314, 2024.
Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint 4rX1v:2405.15364 2024.
Bohan Zeng, Shanglin Li, Yutang Feng, Hong Li, Sicheng Gao, Jiaming Liu, Huaxia Li, Xu Tang, Jianzhuang Liu, and Baochang Zhang. Ipdreamer: Appearaace--onntoolaabll 3d object generation with image prompts. arXiv preprini urXiv:2310.05375. 2023.
Xianfang Zeng, Xin Chen, Zhongqi Qi, Wen Liu, Zibo Zhao, Zhibin Wang, Bin Fu, Yong Liu, and Gang Yu. Paint3d: Paint anything 3d with lighting-less texture diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4252-4262, 2024.
Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube: Structuring gaussian splatting using optimal transport for 3d generative modeling. arXiv preprint arXiv:2403. 2024a.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets, 2024b.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3836-3847, October 2023.
Junsheng Zhou, Jinsheng Wang, Baorui Ma Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. In International Conference on Learning Represen- tations (ICLR), 2024.
Zi-Xin Zou, Zhipeng Yu, Yuan-Cher Guo, Yangguang Li, Ding Liang. Yan-Pei Cao, and Song-Har Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.091444 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
A.1 DATASET
Training set. To train our reference-uugmentiee multi-view diffusion model, we use a filtered sub- set of the Objaverse (Deitke et al., 2023) dataset, excluding low-quality 3D models as described in (Tang et al., 2024). Additionally, eliminate data originating from scans, both of which are intended to ensure the quality of subsequent retrieval. We also exclude objects with an excessively high number of vertices or faces to optimize the costly point cloud extraction process and reduce computational time. These refinements result in final training set comprising approximately 64K 3D objects. For each object, normalize it within a unit sphere, and render concept image, 6 canonical coordinate maps (CCMs), and 6 target RGBA images, following the camera distribution protocol of Zero 133++ (Shi et al., 2023a). In particular, the concept image is rendered using randomly sampled azimuth and elevation angles from predefined range. The poses of the six corresponding CCMs and target images consist of interleaving absolute elevations of {20'. -10", 20". - 100 20". -10%,, and relative azimuths of {o 10+300 Φ + 90% @+ + 150", Φ + 210", + 270%, Φ + 330%). where & represents the azimuth of the concept image. To train our sparse- sperse-view 3D reconstruction model, we adopt the same training set and render images from 32 randomly sampled camera views. All images are rendered at a resolution of 512 x 512, a fixed absolute field of view (FOV) of 30 and a fixed camera distance of 1.866.
apply further filtering to remove objects that are too thin and
Retrieval data and method. We leverage Uni3D (Zhou et al., 2024) to retrieve a 3D reference from an input image. In Uni3D, the latent space of the point cloud encoder is aligned to the OpenCLIP (II- harco et al., 2021) image embedding space, facilitating seamless image-to-Pone PointCCood retrieval. Be- fore retrieval, point clouds are sampled from meshes according to the probability distribution of face areas, ensuring denser sampling in regions with larger surface areas. Each point cloud contains 10K points. As point cloud preprocessing is time-consuming, we limit our retrieval to a subset of 40K objects from Objaverse. Our retrieval database contains precomputed embeddings generated by the Uni3D point cloud encoder, which are compared with the query vector of an input image using cosine similarity. To obtain the query vector, we first apply normalization transforms to align the input image with the pre-trained EVA02-E-14-plus model from OpenCLIP, which acts as the query encoder. The normalized image is then encoded into a feature vector. The top candidates are selected based on the highest similarity scores, and softmax function is applied to the top-k scores to enable probabilistic sampling, ensuring efficient and accurate matching between the input image and the corresponding point clouds.
A.2 TRAINING
Reference-augmeented multi-view diffusion model. White- Backe-Back Zerol23++. As discussed in Sec. 3.1, we select (ecro123++++ as our initial multi-view diffusion model. Upon receiving an input image, Zero1 123++ generates tailored multi-view image at resolution of 960 x 640, comprising six 320 x 320 views arranged in a x332 2 grid. The original Zero123++ produces images with gray back- ground, which can result in floaters and cloud-like artifacts during the subsequent sparse-view 3D reconstruction phase. To mitigate this issue, we initialize our model with a variant of Zero aero123++ (Xu al., 2024), which is finetuned to generate multi-view images with a white background.
Training Details. During the training of our refireene-augmenneed multi-view diffusion model, we use the rendered concept image and six CCMs of a 3D object as conditions, and six corresponding target images tailored to a 960 x 640 image as ground truth image for denoising. All images and CCMs have a white background. We concatenate the concept image and the front-view CCM along the RGB channel as the input for meta- Neta--ontrollNet. For the proposed dynamic reference routing, we dynamically downsample the original CCMs to lower resolutions and then upsample them to 320 x 320, using the nearest neighbor. Specifically, we start with a resolution of 16 at noise levels of [0, 0.05) and gradually increase the resolution to 32 and 64 at noise levels of [0.05, 0.4) and [0.4, 1.0], respectively. For self-reference augmentations (Sec. A.5), the probabilities of applying random resize, flip horizontal, grid distortion, shift, and retrieved reference are set to 0.4, 0.5, 0.1, 0.5, and 0.2, respectively. We train the model for 10,000 steps, beginning with 1000 warm-up steps
15

---

Published as a conference paper at ICLR 2025
Condition Gref
Condition Epair
Image Eynage
Time E
Image Encoder 120x80
Image Encoder 120x80
Image Eacedor
Time Eacoder
Zye/
Epoly
zero convolution
zero convolution
Input x;
Image&Ting
ImagedTing Time
Encoder Block 60x40
MVD Encoder 60x40 (trainable copy)
Eneoder Block 60x40 (trainable copy)
MVD Encoder Block 30x20
MVD Encoder Block 30x20 (trainable capy)
MVD Encoder Block 30x20 (trainable copy)
MVD Encoder Block C
MVD3 Encoder Block C 15x10 (trainable copy)
MV13 Hicoeder Hlock
15x10
15x10 (trainable copy)
Encoder
MVD Encoder Block 15x10 (traizable copy)
MVD Encoder Block 15x10 (trainable copy)
Block D 15x10
MVD Middle Block 15x10
MVD Midcle Block 15x10 ttrainalle copy)
MVD Middle Block 15x10 (tminaale copy)
convolution
zero convolution
MVD Decoder Block D 15x10
zero convolution
zero convolition
Decoder Black 15x10
zero convolution
zero convalution
MVD Decoder Black 30x20
zero convolution
zero convelution
MVD Decoder Block A 60x40
convolution
zero convolution
Output eg (x,, Cinage- Fref)
(a) Pretrained Multi-view Diffusion
(b) Base ControlNet
(e) Meta-Controller
Figure 13: Detailed architecture design of meta-ControlNet.
with minimal augmentations. We use the AdamW optimizer with a learning rate of 1.0 x 10-5 and a total batch size of 48. The whole training process takes around 10 hours on 8 NVIDIA A100 (80G) GPUs
viarse-view 3D reconstruction model. As discussed in Sec. 3.5, we employ LGM to convert the synthesized multi images into a 3D model. The original LGM is designed to reconstruct 3D model from four input views a resolution of 256 x 256. However, this does not align with the multi-view images generated in our first stage, which consist of six views at resolution of 320 320. To adapt LGM to our specific inputs, we take its pretrained weights as initialization and finetune it to support six input images at 320 x 320. Simultancously changing the number of input views and image resolutions can destabilize the training process. We therefore separate the finetuning of number of input views and input resolution. Specifically, first finetune the model with six input views at the original resolution for 60 epochs and then further finetune the model at higher resolution of 320 x 320 for another 60 epochs. The finetuning process is conducted on 32 NVIDIA A100 (80G) GPUs using the AdamW optimizer with a learning rate of 2.0 x 10-4 and a total batch size of 192. The whole finetuning process takes around four days.
A.3 INFERENCE SPEED
Similar to existing image-to-3D generation methods Wang et al. (2024a); Xu et al. (2024); Tang et al. (2024) that combine multi-view generation and sparse-view 3D reconstruction, Phidias can generate final 3D model from the input image in less than 10 seconds. The major latency arises from the diffusion- based multi-view generation process, which takes 9 seconds to generate an image with views in 3 x 2 grid by 75 denoising steps. Other inference-related processes nearly real-time,
16

---

Published as a conference paper at ICLR 2025
3D Reference
Frame
Frame
Frame 3
Frame 4
Ours
ORe
CRM
LGM
InstantMesh
SV3D
Figure 14: Analysis on different input viewpoints. We compare the performance Phidias with five baseline methods by reconstructing 3D objects from video frames with various viewpoints. For each case, we show two rendered images at novel views.
including 0.03 seconds for image-based 3D retrieval, 0.4 seconds for rendering the CCM maps, and 0.02 seconds for sparse-view 3D reconstruction. The inference time is tested using single NVIDIA A100 GPU.
A.4 MRTA-CONTROLNET
detailed figure of the proposed meta-ControlNet in the style of vanilla ControlNet shown in Fig. 13, where Cpair is pair of the concept image and the front-view reference CCM.
A.5 AUGMENTATION DETAILS
We implement a series augmentations to facilitate the training our diffusion model in a self- reference manner, where the ground truth 3D model serves its own reference. These augmenta- tions are designed to simulate the misalignment between the 3D reference and the concept image.
Resize and horizontal flip. Due to the self-reference strategy, reference CCMs always pixel-wise aligned with the concept image. However, during inference, references often differ in scale or exhibit mirror symmetry. For example, reference 3D character might hold weapon in the opposite hand compared to the concept image. To address this, we apply random resizing and horizontal flipping to the reference model, simulating scale variations and shimor--yymmeric structures.
Grid distortion and shift. During inference, the reference may exhibit asymmetric similarity with the target 3D model across different views. For instance, reference building might closely resemble
17

---

Published as a conference paper at ICLR 2025
(a) Angle deviation between input image and 3D reference
(b) Semantic a but structural-misalineed 3D reference
(97.86)
Input Image
3D Ref. CCM
Generated 3D Model
Input Image
3D Ref. CCM
Generated 3D Model
Figure 15: Failure cases. There are two typical failure cases due to bad retrieval: (a) misaligned pose and (b) misaligned structure.
the concept image from the front but differ significantly from the side. To address this, we apply multi-view jitter through grid distortion and shifting. Specifically, we independently distort and shift each view of the reference CCMs using a random grid and random shift offset during training, simulating such asymmetric similarity across views.
Retrieved Reference. Although the retrieved 3D reference alone is insufficient for model training, as discussed in Sec. 3.4, it can still serve as a strong augmentation simulate significant misalignment. Therefore, we assign small probability of using the retrieved model as the reference during training.
B LIMITATION AND FAILURE CASES
Despite promising results, Phidias still has several limitations for further improvement. As retrieval-augmentedd generation model, the performance can be affected by the retrieval method and the scale and quality of 3D reference datahase. Currently, the 3D database we used for retrieval only consists of 40K objects, making it difficult to find a very similar match. Also, mainstream 3D retrieval methods (Zhou et al., 2024; Xue et al., 2023) rely semantic similarity, which may not always yield the best match. For example, retrieved reference models with misaligned poses structures can lead to undesired outcomes, retrieval accuracy and expand the 3D reference database could mitigate these issues. Additionally, the limited resolution of the backbone multi-view diffusion model (320 320) restricts the handling of high-resolution images. Enhancing the resolution of the diffusion model could further improve the quality of the generated 3D models.
shown in Fig. 15. Future works that improve the
C ADDITIONAL RESULTS
C.I ADDITIONAL ANALYSIS ENHANCED GENERALIZATION ABILITY
Phidias takes an additional 3D reference as input improve generative quality (Fig. 5) and provide greater controllability (Fig. 4) for 3D generation. We argue that Phidias can also enhance general- ization ability when given input images from atypical viewpoints. When reconstructing 3D objects from video frames with varying views (Fig. 14), we observe that the baseline methods perform well with typical view angles (i.e., frame 1) but struggle with atypical input view angles (e.g., frame 3 and 4). Conversely, Phidias produces plausible results given all four input views, demonstrating robust generalization ability across both typical and atypical viewpoints.
C.2 MORE RESULTS
More results on theme-aware 3D-to-3D generation are shown in Fig. 16. More results on text-to-3D and image-to-3D generation are shown Fig. 17 and Fig. 18.
18

---

Published as a conference paper at ICLR 2025
Input
Self-Reference
Generated Variation
Generated 3D Variation
Figure 16: Additional results on theme-aware 3D-to-3D generation.
19

---

Published as a conference paper at ICLR 2025
Buily Tno mechaniiaa urmis
armored robot with anguiar design
GGeet and yellow cenamic vessel"
"Golden and sileer medieval Emight's Aeliee
"Glowing mushroom forest with Text Input
"Red and stlver
Text Input
3D Reference
Generated 3D Model
3D Reference
Generated 3D Model
Figure 17: Additional results retrieval-augmenter text-to-3D generation.
Image Input
3D Reference
Generated 3D Model
Input
3D Reference
Generated 3D Model
Figure 18: Additional results on retrieval- augmented image-to-3D generation.
20


Reviews:
[{"review_id": "KztSsjkYNO", "reviewer": "Reviewer_phbC", "summary": "The paper present a reference-augmented method using retrieved 3D objects as auxiliary guidance to reduce the ill-posed nature of the single-view 3D reconstruction task. The method leverages multi-view diffusion zero123++ for the generation of 6 novel view images, and scale-up the LGM for 3D reconstruction. The main contribution of the paper is to use input concept image (typically lack of 3D information of the back or side view, etc.) to refer a similar 3D objects from their 3D database (40K objects selected from Objaverse), the 6 canonical coordinate maps (CCM) from referred 3D object is used to generate better multi-view images. Meta-ControlNet with dynamic resolution of the intake CCM during diffusion sampling contribute to the misalignment dilemma between the concept images and referred 3D object. The results show that the method outperform other SOTA models in image-to-3D tasks given the proper 3D object reference. And it may extend to text-to-3D, and 3D-to-3D applications.", "questions": "Great work! I elaborate more based on weakness, and provide some suggestions and questions.\n\n1. What is the main research goal of the paper? Is it to improve the quality of the 3D reconstruction on un-seen views (which indicates the output model should faithfully obey the input concept image), or it provides more customization freedom by given auxiliary 3D object reference (which indicates the output should lean to the given 3D objects.). Based on a full reviewing of the paper, my understand is the paper lean to the former one (correct me if I am wrong). So when there is conflict between retrieved 3D objects and concept images, which side would the model follow the most? If I am a graphics designer, a useful one will be a model doing either way, rather than a model-decided interpolation between two sides.\n \n2. Is it a correct understand that the meta-controlnet is actually to learn a 3D texture mapping, which utilize the pixel-based texture in the concept image, map it to the retrieved 3D object surface based on 6-view CCMs? In that, the best way to phrase the contribution may like \"using the front-view image as 2D texture guidance to generate unseen views given the 6-view CCM priors\".\n\n3. Follow Question-2, would you elaborate more about the role and reason to use diffusion model here? My understanding is, because there is conflict if we directly use 3D texture mapping using concept images and 6-view CCMs, so you let the diffusion process handle those conflicts. Any reason to exclude other pixel-based networks or traditional CG methods?\n\n4. There are a lot of necessary details in the supplementary, like details of retrieving, dataset, etc. It will be better to move them into the main paper, even some of details may raise questions from reviewers. There is still a half-page blank now.\n\n I reserve the change of grades upon authors' responses to my reviewing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The framework extends the controllability of current image-to-3D models. The model retrieves an auxiliary 3D object as guidance to improve the 3D reconstruction quality. The results shows outperformance than current SOTA models.\n\n2. The paper propose meta-controlnet, and self-reference inference to let model learn how to resolve the conflict between retrieved objects and concept front-view image. Given the setting by the paper claimed, the model is able to provide reasonable 3D generation results matches on both sides.\n\n3. The model is able to extend to text-to-3D and 3D-to-3D generation tasks, which may contribute to more broad applications.", "weaknesses": "1. (model behavior) The model is trained with self-augmented ground-truth 3D objects, in order to handle the misalignment dilemma. However, as the A.4 specified, the augmented ones are still very similar to the ground truth. Moreover, all showcases are with no explicit conflicts between concept image and 3D objects. This raises the question of how a bad retrieval influence the model performance, which is actually common in practical case (as line 878-879). More comprehensive analysis or comparisons are necessary.\n\n2. (retrieval robustness) The quality of the retrieval directly influence the improving of the generation quality, especially on the back or side views. I elaborate in detail in Question-1. If the user have either a out-of-retrieval-database front-view concept image, or the retrieval brings an unreasonable objects, the model output may not be as good as claimed. As Fig.7 shows, random reference as an example of non-similar retrieval may provide unreasonable results. This restricts the usage of the method.\n\n3. (restricted/unfair experiment setting) Followed by weakness-1, the paper constrains the input image as only-front-view (line 221-222) to all competitor models. A random view, or 2-4 views as other related works defined, will contain more geometric information of the 3D details of the object. If not restricting to only-front-view input, using other models which do not require an auxiliary 3D object input like OpenLRM may already provide reasonable results. Their results may be better than the results with no reference in Fig.1, and Fig.7, and SOTA results in Fig. 5.\n\n4. (restricted/unfair experiment setting) In the model evaluation section, all showcase examples meet the optimal situation of the framework. (a) The retrieved 3D objects is very similar to the concept image in the front view, (b) there is no/little conflicts on the other views. Given that, the improvement of the quantitative results in Tab.1 is still marginal. The row \"Ours (GT Ref.)\" may not be a reasonable comparison, since it provides ground truth geometric information in 6-view. I am not surprise it provides better generation results from the unseen view.\n\n5. (model usage) When there is considerable conflict between retrieved 3D objects and concept images (as line 878-879, it is actually very common), which side would the model follow the most? Potential users may want to see more examples of practical cases. And from a point of view as graphics designer, it will be better to provide a controlling nob for user to decide the extent of guidance from either side, rather let the model itself decides the extent. Right now, it is still uncontrollable.\n\n6. (novelty) The novelty of the paper concentrated on the meta-controlnet and reference-based inference. However, given the weakness aforementioned, it may solve an restricted task or with limited contribution to the real problem, and compare to SOTAs in an unfair setting. Moreover, the reason to use diffusion models, and following reconstruction model may need more elaborations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AvCmWoveid", "reviewer": "Reviewer_tCL1", "summary": "This paper presents a new 3D generation approach that utilizes a reference 3D model to enhance the generation process. The method comprises three key components: (1) Meta-ControlNet (2) Dynamic Reference Routing (3) Self-Reference Augmentation. Experimental results demonstrate the effectiveness of this method in enhancing 3D generation quality, generalization ability, and controllability.", "questions": "- With the dynamic reference routing mechanism, the meta-ControlNet should be capable of handling inputs at various resolutions. Is the meta-ControlNet trained on images of different resolutions, or is it trained solely on the highest resolution and expected to generalize to lower resolutions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Gathering information from a 3D reference model is straightforward.\n- The evaluation of generation quality shows the effectiveness of the approach.\n- The paper is well-written and easy to follow.", "weaknesses": "- In the evaluation, only generation quality is assessed, with no evaluation of generation speed. For Phidias, retrieving information from the reference 3D model requires both 3D model retrieval and rendering of the canonical coordinate maps, which introduces additional latency. However, the paper does not report specific latency overheads.\n- From Table 3, the effectiveness of dynamic routing appears to be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JkDaF31Xu", "reviewer": "Reviewer_mgWn", "summary": "The paper introduces a generative model designed for creating 3D content from text, image, and 3D conditions with reference-augmented diffusion. Phidias aims to enhance the quality, generalization, and controllability of 3D generation by leveraging existing 3D models as references. The model integrates three key components: meta-ControlNet for dynamically adjusting conditioning strength, dynamic reference routing to address misalignments between input images and 3D references, and self-reference augmentations for self-supervised training with a progressive curriculum. \n\nThese components collectively improve generative performance over existing methods, offering a unified framework for 3D generation using diverse inputs. Phidias enables various applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.", "questions": "1. In line 186, there is a missing citation regarding the canonical coordinate map. I believe this map is also referred to as the NOCS map, as introduced in reference [a].\n\n[a] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a dynamic reference routing approach. Recognizing that reference models do not perfectly correspond to the conceptual image, this paper utilize a feature pyramid to address this discrepancy. The motivation is that, it has been widely observed during the reverse diffusion process that the coarse structure of a target image is established in the early, high-noise timesteps, while finer details emerge progressively as the timesteps advance. This motivates to start with low-resolution reference CCMs at high noise levels.\n2. This paper facilitates a range of downstream applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.", "weaknesses": "1. Dependence on Retrieval Quality: As stated by the author The performance of Phidias is contingent upon the quality and relevance of the retrieved 3D reference models, which may not always be optimal.\n2. Database Limitations: The current 3D database used for retrieval is limited in size, which could restrict the diversity and accuracy of reference models available for augmentation. Moreover, the model necessitates an additional dataset for inference, which may further restrict its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8FtBHgXwBP", "reviewer": "Reviewer_eNa9", "summary": "This paper introduces Phidias, a RAG-based framework for high-fidelity 3D content creation from multimodal inputs, including text, images, and 3D references. The framework introduces key innovations include Meta-ControlNet to dynamically modulate reference influence, dynamic reference routing to progressively refine outputs through a coarse-to-fine strategy, and self-reference augmentation to enhance generalization. Phidias enables sparse-view reconstruction, theme-based model variation, and user-interactive generation, demonstrating clear superiority over state-of-the-art approaches in both geometric precision and visual fidelity.", "questions": "1. why the point cloud for retrieval only contains 10K points, is it limitation stems from constraints imposed by Uni3D?\n2. Why are the performences in Figure 14 so different from the results of the last row of figures in Figure 4? It looks like that some case's pose in Figure 4 doesn't match the input image, either.\n3. The time and resource consumption for retrieval during each inference is not clearly stated", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a well-founded solution for multi-view generation, effectively addressing challenges such as the generation of unseen views and other ill-posed problems.\n2. This paper is well-organized and clearly written, making it accessible to readers.", "weaknesses": "1. A significant weakness is the limited analysis of the retrieval mechanism for 3D references, which plays a critical role in guiding generation. The reliance on existing 3D models introduces a potential bottleneck, as Phidias’s performance may degrade if the retrieved references are irrelevant or of poor quality. \n2. The control mechanism introduced via Meta-ControlNet is promising, but the paper does not delve deeply into more challenging settings. For example, scenarios where the input concept conflicts with the reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7gElG5ASDo", "reviewer": "Reviewer_Ef4X", "summary": "This paper proposes a new 3D generation method that introduces a meta-ControlNet and dynamic reference routing, leveraging user-provided 3D reference models and images to produce high-quality, controllable 3D object synthesis.", "questions": "- I strongly support the task of combining reference 3D models with images for 3D generation as proposed in the paper, especially considering its relevance to the gaming industry and asset creation. Some related 3D generation methods discuss a texture editing task using complex images to guide coarse 3D object generation, which is highly relevant to this work. Could the authors provide some discussion or analysis on how to improve the method when there are significant differences between the reference image and the reference 3D model? This could be addressed in future work.\n\n- I am inclined to accept this paper, as I also work on related tasks. Currently, I rate it as 6 but would likely increase the rating to 8 during the rebuttal phase.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of meta-ControlNet, combined with 3D reference models, provides strong guidance for the generation process, resulting in high-quality 3D outputs with excellent geometric consistency.\n\n- As a reviewer familiar with this field, I believe that the combination of 3D reference models and images for generating 3D results has significant industrial value and could greatly inform future work in 4D generation as well.\n\n- The method is simple, easy to understand, and the paper is clearly written.", "weaknesses": "- Obtaining a reference 3D model can be quite challenging. Although this is valuable in certain application scenarios, it can be difficult to find appropriate reference 3D models in extreme cases. I recommend that the authors add this limitation to the paper.\n\n- While the strategy of combining ControlNet for generation appears straightforward, this could be criticized for lacking novelty. However, in my opinion, simplicity often leads to elegant and effective solutions, and I don’t consider this a reason for rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)