Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion

## Abstract

Abstract In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias , a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in significant generative improvements over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions, offering versatile applications. Demo videos are at: https://RAG-3D.github.io/ .

## 1 Introduction

The goal of 3D generative models is to empower artists and even beginners to effortlessly convert their design concepts into 3D models.
Consider the input image in Fig. 1. A skilled craftsman can, through a blend of skills and creativity, convert a 2D concept image into an exquisite 3D model. This creative process can originate from artists’ pure imagination or, more commonly, through examining one or more existing 3D models as a source of inspiration (Bob, ; Carvajal, ). Artists often refer to these pre-existing 3D models to improve the modeling quality. The question then arises: could we develop a reference-based 3D generative model that can replicate this capability?

Over the years, a plethora of works (Wang et al., ; Liu et al., ; Hong et al., ; Bensadoun et al., ) steadily expanded the frontiers of 3D generative models.
These methods, while yielding stunning performance, still face several challenges. 1) Generation quality.
A single image cannot furnish sufficient information for reconstructing a full 3D model, due to the ambiguity of this ill-posed task. This necessitates the generative model to “hallucinate” the unseen parts in a data-driven manner. However, this hallucination can lead to view inconsistency and imprecise geometries that appear abrupt and unrealistic. 2) Generalization ability.
These models often struggle with out-of-domain cases, such as atypical input views or objects, constrained by the data coverage of existing 3D datasets (Deitke et al., ).
Also, the growing variety and quantity of object categories exacerbate the difficulty for generative models to learn implicit
shape priors, with a limited model capacity v.s. an infinitely diverse array of objects. 3) Controllability. Due to the ambiguity, one input image can produce several plausible 3D models, each differing in shape, geometric style, and local patterns. Existing methods are constrained by limited diversity and controllability, which hinders the ability to predictably generate the desired 3D models.

To address these challenges, we propose to take 3D models as additional inputs to guide the generation, inspired by the success in retrieval augmented generation (RAG) for language (Lewis et al., ) and image (Sheynin et al., ).
Given an input image and a reference 3D model, we present *Phidias*, a novel reference-augmented diffusion model that unifies 3D generation from text, image, and 3D conditions. As shown in Fig. 1, the reference 3D model would help 1) improve quality by alleviating ambiguity with richer information for unseen views, 2) enhance generalization capacity by serving as a shape template or an external memory for generative models, and 3) provide controllability by indicating desired shape patterns and geometric styles.

Our method proposed a reference-augmented multi-view diffusion model,
followed by sparse-view 3D reconstruction.
The goal is to produce 3D models faithful to the concept image with improved quality by incorporating relevant information from the 3D reference.
However, it is non-trivial to learn such a generative model due to the *Misalignment Dilemma*, where the discrepancy between the concept image and the 3D reference can lead to conflicts in the generation process. This requires our model to utilize the misaligned 3D reference adaptively.
To tackle this challenge, *Phidias* leverages three key designs outlined below.

The first is meta-ControlNet.
Consider 3D reference as conditions for diffusion models. Unlike previous image-to-image translation works (Zhang et al., ; Wang et al., ) that demand the generated images to closely follow the conditions, we treat reference model as auxiliary guidance to provide additional information. The generated multi-view images are expected to be consistent with the concept image, without requiring precise alignment with the reference model.
To this end, we build our method on ControlNet and propose a meta-control network that dynamically modulates conditioning strength when it conflicts with the concept image, based on their similarity.

The second design is dynamic reference routing for further alleviating the misalignment.
Rather than using the same 3D reference for the full diffusion process, we adjust its resolution across denoise timesteps. This follows the dynamics of the reverse diffusion process (Balaji et al., ), which generates coarse structure in high-noised timesteps and details in low-noised timesteps.
Thus, we can alleviate the generation conflicts by starting with a coarse 3D reference and progressively increasing its resolution as the reverse diffusion process goes on.

The final key design is self-reference augmentations. It is not feasible to gather large sets of 3D models and their matching references. A practical solution is to use the 3D model itself as its own reference (i.e., self-reference) for self-supervised learning. The trained model, however, does not work well when the 3D reference does not align with the target image.
To avoid overfitting to a trivial solution, we apply a variety of augmentations to 3D models that simulate this misalignment. Furthermore, we introduce a progressive augmentation approach that leverages curriculum learning for diffusion models to effectively utilize references that vary in similarity.

Taken together, the above ingredients work in concert to
enable *Phidias* to achieve stunning performance in 3D generation. Several application scenarios are thus supported: 1) Retrieval-augmented image-to-3D generation, 2) Retrieval-augmented text-to-3D generation, 3) Theme-aware 3D-to-3D generation, 4) Interactive 3D generation with coarse guidance, and 5) High-fidelity 3D completion.

We summarize our contributions as follows: 1) We propose the first reference-based 3D-aware diffusion model. 2) We design our model with three key component designs to enhance the performance. 3) Our model serves as a unified framework for 3D generation, which provides a variety of applications with text, image, and 3D inputs. 4) Extensive experiments show our method outperforms existing approaches qualitatively and quantitatively.

## 2 RELATED WORKS

Image to 3D.
Pioneering works (Melas-Kyriazi et al., ; Tang et al., ; Chen et al., ) perform 3D synthesis by distilling image diffusion priors (Poole et al., ), but are time-consuming. Recent advancements have leveraged feed-forward models with 3D datasets. Some works use diffusion models to generate points (Nichol et al., ), neural radiance fields (Wang et al., ; Jun & Nichol, ; Gupta et al., ; Hong et al., ), SDF (Cheng et al., ; Zhang et al., ), and gaussian splatting (Zhang et al., ). Another line of works uses transformers for auto-regressive generation (Siddiqui et al., ; Chen et al., ) or sparse-view reconstruction (Hong et al., ; Tang et al., ; Zou et al., ; Wang et al., ; Xu et al., ), which often rely on multi-view diffusion for better performance.

Multi-View Diffusion Models.
Multi-view models reduce the complexities of 3D synthesis to consistent 2D synthesis. Seminal works (Liu et al., ) have shown novel view synthesis capabilities with pre-trained image diffusion models (Rombach et al., ). Later, a plethora of works explored multi-view diffusion models with better consistency (Shi et al., ; Wang & Shi, ; Shi et al., ; Long et al., ; Liu et al., ) by introducing cross-view communication. More recent works (Voleti et al., ; Chen et al., ; You et al., ; Han et al., ) leverage video priors for multi-view generation by injecting cameras into video diffusion models. However, they still struggle with generalized and controllable generation due to the ill-posed nature of this problem.

Reference-Augmented Generation.
Retrieval-augmented generation (RAG) emerges to enhance the generation of both language (Lewis et al., ) and image (Sheynin et al., ; Blattmann et al., ) by incorporating relevant external information during the generation process. Under the context of 3D generation, the concept of reference-based generation is also widely applied. Some works (Chaudhuri et al., ; Kim et al., ; Schor et al., ) probe into the database for compatible parts and assemble them into 3D shapes. Some works refer to a 3D exemplar model (Wu & Zheng, ; Wang et al., ) to produce customized 3D assets. Despite success in specific contexts, they are time-consuming with per-case optimization. In contrast, our method focuses on learning a generalized feed-forward model that applies to reference-augmented 3D generation.

Figure: Figure 2: Overview of the Phidias model. It generates a 3D model in two stages: (1) reference-augmented multi-view generation and (2) sparse-view 3D reconstruction.
Refer to caption: extracted/5861237/Figures/overview.png

## 3 Approach

Given one *concept image*, we aim at leveraging an additional *3D reference model* to alleviate 3D inconsistency issues and geometric ambiguity that exist in 3D generation.
The 3D reference model can be either provided by the user or retrieved from a large 3D database for different applications.
The overall pipeline of Phidias is shown in Fig. 2, which involves two stages: reference-augmented multi-view generation and sparse-view 3D reconstruction.

### 3.1 Reference-Augmented Multi-View Diffusion

Multi-view diffusion models incorporate camera conditions into well-trained image diffusion models for novel-view synthesis with supervised fine-tuning.
We aim to weave additional 3D references into these multi-view models for better generation quality, generalization ability, and controllability. Our approach can be built on arbitrary multi-view diffusion models, enabling reference-augmented 3D content creation from text, image, and 3D conditions. Specifically, we initialize our model with Zero123++ (Shi et al., ), which simply tiles multi-view images for efficient generation conditioned on one input image $\bm{c}_{\mathrm{image}}$.

To integrate 3D reference models $\bm{c}_{\mathrm{ref}}$ into the diffusion process, we transform them into multi-view canonical coordinate maps (CCM) to condition the diffusion model. The choice of CCMs as the 3D representation is based on two reasons: 1) Multi-view images serve as more efficient and compatible inputs for diffusion models than meshes or voxels, as they have embedded camera viewing angles that correspond with the output images. 2) Reference models often share similar shapes with the concept image but vary significantly in texture details. By focusing on the geometry while omitting the texture, CCMs conditions can reduce generation conflicts arising from texture discrepancies. We add a conditioner branch to incorporate reference CCMs into the base multi-view diffusion model. The objective for training our diffusion model $\epsilon_{\theta}$ can be then formulated as:

$$ $\mathcal{L}=\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1)}\left[\|\epsilon- \epsilon_{\theta}\left(\bm{x}_{t},t,\bm{c}_{\mathrm{image}},\bm{c}_{\mathrm{ ref}}\right)\|^{2}\right]$ (1) $$

To leverage the powerful pertaining capability, only the additional conditioner for reference CCMs is trainable while the base multi-view diffusion is frozen.
However, a challenge in our task is that the 3D reference may not strictly align with the concept image or, more commonly, vary in most local parts. We found naive conditioner designs such as ControlNet (Zhang et al., ) tend to produce undesirable artifacts, as they were originally designed for image-to-image translation where the generated images strictly align with the condition images. To mitigate this problem, we introduce three key designs for our reference-augmented diffusion model: (1) Meta-ControlNet for adaptive control of the conditioning strength (Sec. 3.2); (2) Dynamic Reference Routing for dynamic adjustment of the 3D reference (Sec. 3.3); (3) Self-Reference Augmentation for self-supervised training (Sec. 3.4).

### 3.2 Meta-ControlNet.

Figure: Figure 3: Architectural designs for meta-ControlNet (a) and dynamic reference routing (b).
Refer to caption: x1.png

ControlNet is designed to add additional controls to pre-trained diffusion models for image-to-image translation. The conditions are derived from the ground-truth images for self-supervised learning, and thus the generated images are expected to follow the conditions. However, in our settings, the conditions are from the reference model, which often misaligns with the target 3D models we want to generate. The vanilla ControlNet fails to handle such cases.
This necessitates further architecture advancement to accordingly adjust conditioning strength when the reference conflicts with the concept image. To this end, we propose meta-ControlNet, as shown in Fig. 3 (a). Meta-ControlNet is comprised of two collaborative subnets, a base ControlNet and an additional meta-controller.

Base ControlNet is comprised of an image encoder, a trainable copy of down-sampling blocks and middle blocks of the base multi-view diffusion, denoted as $\mathcal{F}^{{\mathrm{base}}}_{\Theta}\left(\cdot\right)$, and a series of $1\times 1$ zero convolution layers (Zero Convs) $\mathcal{Z}^{{\mathrm{base}}}_{\Theta}(\cdot)$. It takes reference CCM maps $\bm{c}_{\mathrm{ref}}$ as input to produce the control signal. To deal with misaligned 3D reference, we introduce an additional meta-controller to modulate the conditioning strength according to different similarity levels.

Meta-controller shares a similar architecture but has different parameters $\Theta^{\prime}$. It works as a knob that dynamically modulates base ControlNet to generate adaptive control signals. Meta-controller takes a pair $\bm{c}_{\mathrm{pair}}$ of the concept image and the front-view reference CCM as input to produce meta-control signals based on their similarities. The meta-control signals are injected into diffusion models in two ways. On the one hand, meta-controller produces multi-scale alignment features $\bm{y}_{\mathrm{meta1}}=\mathcal{Z}^{{\mathrm{meta1}}}_{\Theta^{\prime}}\left(
\mathcal{F}^{{\mathrm{meta}}}_{\Theta^{\prime}}\left(\bm{z}_{\mathrm{pair}}
\right)\right)$ to be injected into base ControlNet. These features are applied to the down-sampling blocks of base ControlNet (Eq. 2) at each scale to guide the encoding of reference and help produce base-signals as:

$$ $\bm{y}_{\mathrm{base}}=\mathcal{Z}^{{\mathrm{base}}}_{\Theta}\left(\mathcal{F} ^{{\mathrm{base}}}_{\Theta}\left(\bm{y}_{\mathrm{meta1}},\bm{z}_{\mathrm{ref}} \right)\right),$ (2) $$

where $\bm{z}_{\mathrm{ref}}$ and $\bm{z}_{\mathrm{pair}}$ are the feature maps of $\bm{c}_{\mathrm{ref}}$ and $\bm{c}_{\mathrm{pair}}$ via the trainable encoders in Fig. 3 (a).

On the other hand, meta-controller produces meta-signals $\bm{y}_{\mathrm{meta2}}=\mathcal{Z}^{{\mathrm{meta2}}}_{\Theta^{\prime}}\left(
\mathcal{F}^{{\mathrm{meta}}}_{\Theta^{\prime}}\left(\bm{z}_{\mathrm{pair}}
\right)\right)$ to be injected to the pretrained multi-view diffusion models. These features are added up to base-signal $\bm{y}_{\mathrm{base}}$ to directly apply for the pretrained diffusion models. Totally, the final outputs of meta-ControlNet are adaptive control signals $\bm{y}_{\mathrm{adaptive}}$ based on the similarity between the concept image and the 3D reference, as:

$$ $\bm{y}_{\mathrm{adaptive}}=\bm{y}_{\mathrm{base}}+\bm{y}_{\mathrm{meta2}}.$ (3) $$

### 3.3 Dynamic Reference Routing

Reference models typically align roughly with the concept image in terms of coarse shape, but diverge significantly in local details. This misalignment can cause confusion and conflicts, as the generation process relies on both the image and reference model. To address this issue, we propose a dynamic reference routing strategy that adjusts the reference resolution across denoise timesteps, as shown in Fig. 3 (b).
As widely observed during the reverse diffusion process, the coarse structure of a target image is determined in high-noised timesteps and fine details emerge later as the timestep goes on. This motivates us to start with low-resolution reference CCMs at high noise levels $t_{h}$. By lowering the resolution, reference models provide fewer details but exhibit smaller misalignment with the concept image. This enables reference models to assist in generating the global structure of 3D objects without significant conflicts. We then gradually increase the resolution of reference CCMs as the reverse diffusion process goes into middle noise levels $t_{m}$ and low noise levels $t_{l}$ to help refine local structures, e.g., progressively generating a curly tail from a straight one (Fig. 3 (b)).
This design choice would ensure effective usage of both concept image and 3D reference during the multi-view image generation process while avoiding degraded generation caused by misalignment.

### 3.4 Self-Reference Augmentation

A good reference model should resemble the target 3D model (with varied details) to provide additional geometric cues, but it is impractical to collect sufficient target-reference pairs for training. An intuitive solution is to retrieve a similar model from a large 3D database as the training reference. However, due to the limited variety in current databases, finding a perfect match is challenging. The retrieved reference can vary greatly in orientation, size and semantics. While this is a common situation in inference scenarios, where a very similar reference is often unavailable, we found training with these challenging pairs fails to effectively use the 3D reference. We conjecture that the learning process struggles due to the significant differences between the reference and target 3D, leading the diffusion model to disregard the references. To avoid the ‘idleness’ of reference, we developed a self-reference scheme that uses the target model as its own reference by applying various augmentations to mimic misalignment (refer to Appendix A.4). This approach ensures that the reference models are somewhat aligned with the target and more compatible, alleviating the learning difficulty.
We further design a curriculum training strategy, which begins with minimal augmentations (very similar references) to force the diffusion model to rely on the reference for enhancement. Over time, we gradually increase augmentation strength and incorporate retrieved references, challenging the diffusion model to learn from references that do not closely match the target. Once trained, our model performs well with a variety of references, even those retrieved ones that are not very similar.

### 3.5 Sparse-View 3D Reconstruction

With multi-view images generated in the first stage, we can obtain final 3D models via sparse-view 3D reconstruction. This step can be built upon arbitrary sparse-view reconstruction models. Specifically, we finetune LGM (Tang et al., ) by expanding the number of input views from 4 to 6 and the resolution of each view from $256\times 256$ to $320\times 320$ so that the trained reconstruction model aligns with the multi-view images generated in our first stage.

## 4 Experiments

In this section, we evaluate our method on image-to-3D generation, a significant area in 3D generation research. For each image, we retrieve a 3D reference model from a 3D database based on similarity (Zhou et al., ). The database used is a subset of Objaverse, containing 40K models. We anticipate that performance could be further enhanced with a larger database in the future.
For the rest of this section, we compare Phidias with state-of-the-art methods and conduct ablation analysis.
More results and implementation details can be found in Appendix.
Results on text-to-3D and 3D-to-3D generation can be found in Sec. 5.

Figure: Figure 4: Diverse retrieval-augmented image-to-3D results. Phidias can generate diverse 3D models with different references for a single input image.
Refer to caption: x2.png

Figure: Figure 5: Qualitative comparisons on image-to-3D generation.
Refer to caption: x3.png

### 4.1 Comparisons with State-of-the-Art Methods

We compare Phidias with five image-to-3D baselines: CRM (Wang et al., ), LGM (Tang et al., ), InstantMesh (Xu et al., ), SV3D (Voleti et al., ), and OpenLRM (He & Wang, ).

Qualitative Results.
For visual diversity (Fig. 4), given the same concept image, Phidias can generate diverse 3D assets that are both faithful to the concept image and conforming to a specific retrieved 3D reference in geometry. For visual comparisons (Fig. 5), while the baseline methods can generate plausible results, they suffer from geometry distortion (e.g., horse legs). Besides, none of the existing methods can benefit from the 3D reference for improved generalization ability (e.g., excavator’s dipper) and controllability (e.g., cat’s tail) as ours.

Quantitative Results.
Following previous works, we conduct quantitative evaluation on google scanned objects (GSO) (Downs et al., ). We remove duplicated objects with the same shape and randomly select 200 objects for evaluation.
For visual quality, we report reconstruction metrics (PSNR, SSIM and LPIPS) on 20 novel views. We also report novel views’ CLIP similarity with paired GT (CLIP-P) and input image (CLIP-I). For geometry quality, we sample 50K points from mesh surface and compute Chamfer Distance (CD) and F-Score (with a threshold of 0.05). To align the generated mesh and GT, we unify their coordinate systems and re-scale them into a unit box.
We report our results with the retrieved reference, i.e., Ours (Retrieved Ref.), and GT mesh as reference, i.e., Ours (GT Ref.), respectively. As shown in Tab. 1, ours, with either retrieved or GT reference, outperforms all baselines, benefiting from the proposed retrieval-augmented method. While the CD is slightly larger, we argue that our approach produces plausible 3D models given different references (Fig. 7), though they can differ from GT mesh when computing chamfer distance.

**Table 1: Quantitative comparison with baselines on image-to-3D synthesis.**
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | CLIP-P $\uparrow$ | CLIP-I $\uparrow$ | CD $\downarrow$ | F-Score $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OpenLRM | 16.15 | 0.843 | 0.194 | 0.866 | 0.847 | 0.0446 | 0.805 |
| LGM | 14.80 | 0.807 | 0.219 | 0.869 | 0.871 | 0.0398 | 0.831 |
| CRM | 16.35 | 0.841 | 0.182 | 0.855 | 0.843 | 0.0443 | 0.796 |
| SV3D | 16.24 | 0.838 | 0.203 | 0.879 | 0.866 | - | - |
| InstantMesh | 14.63 | 0.796 | 0.235 | 0.882 | 0.880 | 0.0450 | 0.788 |
| Ours (GT Ref.) | 20.37 | 0.870 | 0.117 | 0.911 | 0.885 | 0.0391 | 0.840 |
| Ours (Retrieved Ref.) | 17.02 | 0.845 | 0.174 | 0.887 | 0.885 | 0.0402 | 0.833 |

Figure: Table 2: User study.

User Study.
We further conduct a user study to evaluate human preferences among different methods. We publicly invite 30 users to complete a questionnaire for pairwise comparisons. We show the preference rate (i.e., the percentage of users prefer ours compared to a baseline method) in Tab. 4.1, which suggests that our approach significantly outperforms existing methods in the image-to-3D task based on human preferences.

Figure: Figure 6: Qualitative ablation study of the proposed components.
Refer to caption: x4.png

### 4.2 Ablation Study and Analysis

Ablation Studies.
We conduct ablation studies across four settings: a base model employing a standard ControlNet trained with self-reference, and three variants (each integrating one proposed component into the base model). The quantitative results in Tab. 4.1 demonstrate clear improvements in both visual and geometric metrics with our proposed components.

Effectiveness of Meta-ControlNet.
To evaluate meta-ControlNet, we use both self-reference and retrieved reference for training, as the learning of Meta-Controller (Fig. 3 (a) top) requires reference models with varying levels of similarity. As shown in Fig. 6 (a), the base model trained with retrieved reference often ignores the reference, failing to follow the shape pattern (disconnected boat). This phenomenon stems from the considerable similarity variation among retrieved references, which confuses the diffusion model. The base model thereby struggles to determine when and how to use the reference as it lacks the ability to adjust to different levels of similarity. Consequently, they often end up with ignoring the reference models entirely. In contrast, meta-ControlNet equips the model with the capability to dynamically modulate the conditioning strength of the reference model, thereby effectively utilizing available references for improving or controlling the generation process.

Effectiveness of Dynamic Reference Routing.
Dynamic reference routing aims to alleviate local conflicts between the reference and concept images. As illustrated in Fig. 6 (b), when given a highly similar reference, the base model tends to rely heavily on it, leading to missing specific local details within the concept image, e.g., the rope on the left.
By addressing these conflicts with dynamic routing, the model maintains the essential details of the concept image, while still benefiting from the guidance of the 3D reference.

Effectiveness of Self-Reference Augmentation.
As shown in Fig. 6 (c), without self-reference augmentation, the base model predominantly depends on the provided reference for generation. When given a significantly misaligned reference, the model tends to follow the reference’s structure, resulting in an undesired outcome. Conversely, self-reference augmentation ensures that the generated models remain faithful to the concept image, while using the reference as geometry guidance.

**Table 4: Quantitative analysis on similarity levels of 3D reference.**
| Reference | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | CLIP-P $\uparrow$ | CLIP-I $\uparrow$ | CD $\downarrow$ | F-Score $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Top-1 Retrieval | 17.02 | 0.845 | 0.174 | 0.887 | 0.885 | 0.0402 | 0.833 |
| Top-3 Retrieval | 16.75 | 0.841 | 0.172 | 0.887 | 0.886 | 0.0395 | 0.830 |
| Top-5 Retrieval | 15.96 | 0.835 | 0.185 | 0.886 | 0.884 | 0.0408 | 0.819 |
| Random Reference | 14.74 | 0.820 | 0.226 | 0.884 | 0.882 | 0.0424 | 0.810 |
| Without Reference | 15.90 | 0.836 | 0.188 | 0.886 | 0.880 | 0.0416 | 0.814 |

Analysis on Similarity Levels of 3D Reference.
We analyze how similarity levels of 3D references would affect the performance. For each input, we retrieve three models ranked first (top-1), third (top-3), and fifth (top-5) in similarity scores, and randomly choose one model, to serve as 3D references. Quantitative results in Tab. 4 indicate that Phidias performs better with more similar references. Fig. 7 shows Phidias generates diverse plausible results with different references. All results remain faithful to the input image in the front view, but show variations in shapes influenced by the specific reference used. Also, we found Phidias can still generate plausible results even with a random 3D reference, indicating robustness to reference with different similarity levels.

Figure: Figure 7: Qualitative analysis on similarity levels of 3D Reference.
Refer to caption: extracted/5861237/Figures/abs_similarity_levels.png

## 5 Applications

Figure: Figure 8: Phidias enables retrieval-augmented text-to-3D generation by first converting input text into a concept image, and then retrieving a 3D reference based on both the text and image.
Refer to caption: extracted/5861237/Figures/app_text_to_3d.png

Phidias supports versatile applications beyond image-to-3D, such as text-to-3D, theme-aware 3D-to-3D, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.

Text to 3D.
Text-to-3D generation can be converted to image-conditioned generation by transforming a text prompt into a concept image. However, the generated concept image can sometimes be atypical and may lose some information compared with original text input. To enhance generative quality, Phidias employs retrieval-augmented text-to-3D generation, as illustrated in Fig. 8. This involves first retrieving a set of 3D references based on the concept image, and then selecting the one that most closely matches the text description as the final reference.

Figure: Figure 9: Phidias facilitates rapid, theme-aware 3D-to-3D generation by using an existing 3D model as a reference to transform its image variations into corresponding 3D variations.
Refer to caption: x5.png

Theme-Aware 3D-to-3D Generation.
This task aims to create a gallery of theme-consistent 3D variations from existing 3D models. Previous work (Wang et al., )
proposed an optimization-based approach, which is time-consuming. Phidias supports fast generation by first generating image variations based on the input 3D model, and then transforming these variant images into 3D variations with the original 3D model itself as reference.
The results are shown in Fig. 9, using 3D models from Sketchfab(^1^11[https://sketchfab.com/](https://sketchfab.com/)) and previous works as inputs.

Figure: Figure 10: Phidias enables interactive 3D generation with coarse 3D shapes as guidance.
Refer to caption: x6.png

Interactive 3D Generation with Coarse Guidance.
Interactive generation gives users more control over the outputs, empowering them to make quick edits and receive rapid feedback. Phidias also provides this functionality, allowing users to continually adjust the geometry of generated 3D models using manually created coarse 3D shapes as reference models, as shown in Fig. 10.

Figure: Figure 11: Phidias supports high-fidelity 3D completion by using the completed front views to guide the missing parts restoration and the original 3D model to help preserve the origin details.
Refer to caption: extracted/5861237/Figures/app_3d_completion.png

High-Fidelity 3D Completion.
Given incomplete 3D models, as shown in Fig. 11, Phidias can be used to restore the missing components. Specially, by generating a complete front view through image inpainting and referencing to the original 3D model, Phidias can precisely predict and fill in the missing parts in novel views while maintaining the integrity and details of the origin, resulting in a seamlessly and coherently structured 3D model.

## 6 Conclusion

In this work, we introduced Phidias, a 3D-aware diffusion model enhanced by 3D reference. By incorporating meta-ControlNet, dynamic reference routing, and self-reference augmentations, Phidias effectively leverages reference models with varying degrees of similarity for 3D generation. The proposed approach boosts the quality of 3D generation, expands its generalization capabilities, and improves user control. Phidias offers a unified framework for creating high-quality 3D content from diverse modalities, such as text, images, and pre-existing 3D models, enabling versatile applications. We believe that Phidias will inspire further research to advance the field of 3D generation.

#### Acknowledgments

This work is partially supported by the National Key R&D Program of China (2022ZD0160201) and Shanghai Artificial Intelligence Laboratory. This work is also in part supported by a GRF grant from the Research Grants Council of Hong Kong (Ref. No.: 11205620).

## Appendix

## Appendix A Implementation Details

### A.1 dataset

Training set. To train our reference-augmented multi-view diffusion model, we use a filtered subset of the Objaverse (Deitke et al., ) dataset, excluding low-quality 3D models as described in (Tang et al., ). Additionally, we apply further filtering to remove objects that are too thin and eliminate data originating from scans, both of which are intended to ensure the quality of subsequent retrieval. We also exclude objects with an excessively high number of vertices or faces to optimize the costly point cloud extraction process and reduce computational time. These refinements result in a final training set comprising approximately 64K 3D objects. For each object, we normalize it within a unit sphere, and render 1 concept image, 6 canonical coordinate maps (CCMs), and 6 target RGBA images, following the camera distribution protocol of Zero123++ (Shi et al., ). In particular, the concept image is rendered using randomly sampled azimuth and elevation angles from a predefined range. The poses of the six corresponding CCMs and target images consist of interleaving absolute elevations of {$20$°, $-10$°, $20$°, $-10$°, $20$°, $-10$°}, and relative azimuths of {$\phi+30$°, $\phi+90$°, $\phi+150$°, $\phi+210$°, $\phi+270$°, $\phi+330$°}, where $\phi$ represents the azimuth of the concept image.
To train our sparse-view 3D reconstruction model, we adopt the same training set and render images from 32 randomly sampled camera views. All images are rendered at a resolution of $512\times 512$, a fixed absolute field of view (FOV) of $30$°, and a fixed camera distance of $1.866$.

Retrieval data and method. We leverage Uni3D (Zhou et al., ) to retrieve a 3D reference from an input image. In Uni3D, the latent space of the point cloud encoder is aligned to the OpenCLIP (Ilharco et al., ) image embedding space, facilitating seamless image-to-PointCloud retrieval.
Before retrieval, point clouds are sampled from meshes according to the probability distribution of face areas, ensuring denser sampling in regions with larger surface areas. Each point cloud contains 10K points. As point cloud preprocessing is time-consuming, we limit our retrieval to a subset of 40K objects from Objaverse. Our retrieval database contains precomputed embeddings generated by the Uni3D point cloud encoder, which are compared with the query vector of an input image using cosine similarity. To obtain the query vector, we first apply normalization transforms to align the input image with the pre-trained EVA02-E-14-plus model from OpenCLIP, which acts as the query encoder. The normalized image is then encoded into a feature vector. The top candidates are selected based on the highest similarity scores, and a softmax function is applied to the top-k scores to enable probabilistic sampling, ensuring efficient and accurate matching between the input image and the corresponding point clouds.

### A.2 training

Reference-augmented multi-view diffusion model.
White-Background Zero123++. As discussed in Sec. 3.1, we select Zero123++ as our initial multi-view diffusion model. Upon receiving an input image, Zero123++ generates a tailored multi-view image at a resolution of $960\times 640$, comprising six $320\times 320$ views arranged in a $3\times 2$ grid. The original Zero123++ produces images with a gray background, which can result in floaters and cloud-like artifacts during the subsequent sparse-view 3D reconstruction phase. To mitigate this issue, we initialize our model with a variant of Zero123++ (Xu et al., ), which is finetuned to generate multi-view images with a white background.

Training Details. During the training of our reference-augmented multi-view diffusion model, we use the rendered concept image and six CCMs of a 3D object as conditions, and six corresponding target images tailored to a $960\times 640$ image as ground truth image for denoising. All images and CCMs have a white background. We concatenate the concept image and the front-view CCM along the RGB channel as the input for meta-ControlNet. For the proposed dynamic reference routing, we dynamically downsample the original CCMs to lower resolutions and then upsample them to $320\times 320$, using the nearest neighbor. Specifically, we start with a resolution of 16 at noise levels of $[0,0.05)$ and gradually increase the resolution to 32 and 64 at noise levels of $[0.05,0.4)$ and $[0.4,1.0]$, respectively. For self-reference augmentations (Sec. A.4), the probabilities of applying random resize, flip horizontal, grid distortion, shift, and retrieved reference are set to 0.4, 0.5, 0.1, 0.5, and 0.2, respectively. We train the model for 10,000 steps, beginning with 1000 warm-up steps with minimal augmentations. We use the AdamW optimizer with a learning rate of $1.0\times 10^{-5}$ and a total batch size of 48. The whole training process takes around 10 hours on 8 NVIDIA A100 (80G) GPUs.

Sparse-view 3D reconstruction model.
As discussed in Sec. 3.5, we employ LGM to convert the synthesized multi-view images into a 3D model. The original LGM is designed to reconstruct a 3D model from four input views at a resolution of $256\times 256$. However, this does not align with the multi-view images generated in our first stage, which consist of six views at a resolution of $320\times 320$. To adapt LGM to our specific inputs, we take its pretrained weights as initialization and finetune it to support six input images at $320\times 320$.
Simultaneously changing the number of input views and image resolutions can destabilize the training process. We therefore separate the finetuning of number of input views and input resolution. Specifically, we first finetune the model with six input views at the original resolution for 60 epochs and then further finetune the model at a higher resolution of $320\times 320$ for another 60 epochs. The finetuning process is conducted on 32 NVIDIA A100 (80G) GPUs using the AdamW optimizer with a learning rate of $2.0\times 10^{-4}$ and a total batch size of 192. The whole finetuning process takes around four days.

### A.3 Meta-controlnet

Figure: Figure 12: Detailed architecture design of meta-ControlNet.
Refer to caption: extracted/5861237/Figures/detailed_meta-controlnet.png

A detailed figure of the proposed meta-ControlNet in the style of vanilla ControlNet is shown in Fig. 12, where $\bm{c}_{pair}$ is a pair of the concept image and the front-view reference CCM.

### A.4 Augmentation Details

We implement a series of augmentations to facilitate the training of our diffusion model in a self-reference manner, where the ground truth 3D model serves as its own reference. These augmentations are designed to simulate the misalignment between the 3D reference and the concept image.

Resize and horizontal flip. Due to the self-reference strategy, reference CCMs are always pixel-wise aligned with the concept image. However, during inference, references often differ in scale or exhibit mirror symmetry. For example, a reference 3D character might hold a weapon in the opposite hand compared to the concept image. To address this, we apply random resizing and horizontal flipping to the reference model, simulating scale variations and mirror-symmetric structures.

Grid distortion and shift. During inference, the reference may exhibit asymmetric similarity with the target 3D model across different views. For instance, a reference building might closely resemble the concept image from the front but differ significantly from the side. To address this, we apply multi-view jitter through grid distortion and shifting. Specifically, we independently distort and shift each view of the reference CCMs using a random grid and a random shift offset during training, simulating such asymmetric similarity across views.

Retrieved Reference. Although the retrieved 3D reference alone is insufficient for model training, as discussed in Sec. 3.4, it can still serve as a strong augmentation to simulate significant misalignment. Therefore, we assign a small probability of using the retrieved model as the reference during training.

Figure: Figure 13: Analysis on different input viewpoints. We compare the performance of Phidias with five baseline methods by reconstructing 3D objects from video frames with various viewpoints. For each case, we show two rendered images at novel views.
Refer to caption: x7.png

## Appendix B Limitation and Failure Cases

Despite promising results, Phidias still has several limitations for further improvement. As a retrieval-augmented generation model, the performance can be affected by the retrieval method and the scale and quality of 3D reference database. Currently, the 3D database we used for retrieval only consists of 40K objects, making it difficult to find a very similar match. Also, mainstream 3D retrieval methods rely on semantic similarity, which may not always yield the best match. For example, retrieved reference models with misaligned poses or structures can lead to undesired outcomes, as shown in Fig. 14. Future works that improve the retrieval accuracy and expand the 3D reference database could mitigate these issues. Additionally, the limited resolution of the backbone multi-view diffusion model ($320\times 320$) restricts the handling of high-resolution images. Enhancing the resolution of the diffusion model could further improve the quality of the generated 3D models.

## Appendix C Additional Results

### C.1 Additional Analysis on Enhanced Generalization Ability

Phidias takes an additional 3D reference as input to improve generative quality (Fig. 5) and provide greater controllability (Fig. 4) for 3D generation. We argue that Phidias can also enhance generalization ability when given input images from atypical viewpoints. When reconstructing 3D objects from video frames with varying views (Fig. 13), we observe that the baseline methods perform well with typical view angles (i.e., frame 1) but struggle with atypical input view angles (e.g., frame 3 and 4). Conversely, Phidias produces plausible results given all four input views, demonstrating robust generalization ability across both typical and atypical viewpoints.

### C.2 More Results

More results on theme-aware 3D-to-3D generation are shown in Fig. 15. More results on text-to-3D and image-to-3D generation are shown in Fig. 16 and Fig. 17.

Figure: Figure 14: Failure cases. There are two typical failure cases due to bad retrieval: (a) misaligned pose and (b) misaligned structure.
Refer to caption: x8.png

Figure: Figure 15: Additional results on theme-aware 3D-to-3D generation.
Refer to caption: x9.png

Figure: Figure 16: Additional results on retrieval-augmented text-to-3D generation.
Refer to caption: x10.png

Figure: Figure 17: Additional results on retrieval-augmented image-to-3D generation.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Great work! I elaborate more based on weakness, and provide some suggestions and questions.

1. What is the main research goal of the paper? Is it to improve the quality of the 3D reconstruction on un-seen views (which indicates the output model should faithfully obey the input concept image), or it provides more customization freedom by given auxiliary 3D object reference (which indicates the output should lean to the given 3D objects.). Based on a full reviewing of the paper, my understand is the paper lean to the former one (correct me if I am wrong). So when there is conflict between retrieved 3D objects and concept images, which side would the model follow the most? If I am a graphics designer, a useful one will be a model doing either way, rather than a model-decided interpolation between two sides.
 
2. Is it a correct understand that the meta-controlnet is actually to learn a 3D texture mapping, which utilize the pixel-based texture in the concept image, map it to the retrieved 3D object surface based on 6-view CCMs? In that, the best way to phrase the contribution may like "using the front-view image as 2D texture guidance to generate unseen views given the 6-view CCM priors".

3. Follow Question-2, would you elaborate more about the role and reason to use diffusion model here? My understanding is, because there is conflict if we directly use 3D texture mapping using concept images and 6-view CCMs, so you let the diffusion process handle those conflicts. Any reason to exclude other pixel-based networks or traditional CG methods?

4. There are a lot of necessary details in the supplementary, like details of retrieving, dataset, etc. It will be better to move them into the main paper, even some of details may raise questions from reviewers. There is still a half-page blank now.

 I reserve the change of grades upon authors' responses to my reviewing.
soundness: 3
strengths: 1. The framework extends the controllability of current image-to-3D models. The model retrieves an auxiliary 3D object as guidance to improve the 3D reconstruction quality. The results shows outperformance than current SOTA models.

2. The paper propose meta-controlnet, and self-reference inference to let model learn how to resolve the conflict between retrieved objects and concept front-view image. Given the setting by the paper claimed, the model is able to provide reasonable 3D generation results matches on both sides.

3. The model is able to extend to text-to-3D and 3D-to-3D generation tasks, which may contribute to more broad applications.
summary: The paper present a reference-augmented method using retrieved 3D objects as auxiliary guidance to reduce the ill-posed nature of the single-view 3D reconstruction task. The method leverages multi-view diffusion zero123++ for the generation of 6 novel view images, and scale-up the LGM for 3D reconstruction. The main contribution of the paper is to use input concept image (typically lack of 3D information of the back or side view, etc.) to refer a similar 3D objects from their 3D database (40K objects selected from Objaverse), the 6 canonical coordinate maps (CCM) from referred 3D object is used to generate better multi-view images. Meta-ControlNet with dynamic resolution of the intake CCM during diffusion sampling contribute to the misalignment dilemma between the concept images and referred 3D object. The results show that the method outperform other SOTA models in image-to-3D tasks given the proper 3D object reference. And it may extend to text-to-3D, and 3D-to-3D applications.
weaknesses: 1. (model behavior) The model is trained with self-augmented ground-truth 3D objects, in order to handle the misalignment dilemma. However, as the A.4 specified, the augmented ones are still very similar to the ground truth. Moreover, all showcases are with no explicit conflicts between concept image and 3D objects. This raises the question of how a bad retrieval influence the model performance, which is actually common in practical case (as line 878-879). More comprehensive analysis or comparisons are necessary.

2. (retrieval robustness) The quality of the retrieval directly influence the improving of the generation quality, especially on the back or side views. I elaborate in detail in Question-1. If the user have either a out-of-retrieval-database front-view concept image, or the retrieval brings an unreasonable objects, the model output may not be as good as claimed. As Fig.7 shows, random reference as an example of non-similar retrieval may provide unreasonable results. This restricts the usage of the method.

3. (restricted/unfair experiment setting) Followed by weakness-1, the paper constrains the input image as only-front-view (line 221-222) to all competitor models. A random view, or 2-4 views as other related works defined, will contain more geometric information of the 3D details of the object. If not restricting to only-front-view input, using other models which do not require an auxiliary 3D object input like OpenLRM may already provide reasonable results. Their results may be better than the results with no reference in Fig.1, and Fig.7, and SOTA results in Fig. 5.

4. (restricted/unfair experiment setting) In the model evaluation section, all showcase examples meet the optimal situation of the framework. (a) The retrieved 3D objects is very similar to the concept image in the front view, (b) there is no/little conflicts on the other views. Given that, the improvement of the quantitative results in Tab.1 is still marginal. The row "Ours (GT Ref.)" may not be a reasonable comparison, since it provides ground truth geometric information in 6-view. I am not surprise it provides better generation results from the unseen view.

5. (model usage) When there is considerable conflict between retrieved 3D objects and concept images (as line 878-879, it is actually very common), which side would the model follow the most? Potential users may want to see more examples of practical cases. And from a point of view as graphics designer, it will be better to provide a controlling nob for user to decide the extent of guidance from either side, rather let the model itself decides the extent. Right now, it is still uncontrollable.

6. (novelty) The novelty of the paper concentrated on the meta-controlnet and reference-based inference. However, given the weakness aforementioned, it may solve an restricted task or with limited contribution to the real problem, and compare to SOTAs in an unfair setting. Moreover, the reason to use diffusion models, and following reconstruction model may need more elaborations.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - With the dynamic reference routing mechanism, the meta-ControlNet should be capable of handling inputs at various resolutions. Is the meta-ControlNet trained on images of different resolutions, or is it trained solely on the highest resolution and expected to generalize to lower resolutions?
soundness: 3
strengths: - Gathering information from a 3D reference model is straightforward.
- The evaluation of generation quality shows the effectiveness of the approach.
- The paper is well-written and easy to follow.
summary: This paper presents a new 3D generation approach that utilizes a reference 3D model to enhance the generation process. The method comprises three key components: (1) Meta-ControlNet (2) Dynamic Reference Routing (3) Self-Reference Augmentation. Experimental results demonstrate the effectiveness of this method in enhancing 3D generation quality, generalization ability, and controllability.
weaknesses: - In the evaluation, only generation quality is assessed, with no evaluation of generation speed. For Phidias, retrieving information from the reference 3D model requires both 3D model retrieval and rendering of the canonical coordinate maps, which introduces additional latency. However, the paper does not report specific latency overheads.
- From Table 3, the effectiveness of dynamic routing appears to be limited.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In line 186, there is a missing citation regarding the canonical coordinate map. I believe this map is also referred to as the NOCS map, as introduced in reference [a].

[a] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
soundness: 3
strengths: 1. This paper proposes a dynamic reference routing approach. Recognizing that reference models do not perfectly correspond to the conceptual image, this paper utilize a feature pyramid to address this discrepancy. The motivation is that, it has been widely observed during the reverse diffusion process that the coarse structure of a target image is established in the early, high-noise timesteps, while finer details emerge progressively as the timesteps advance. This motivates to start with low-resolution reference CCMs at high noise levels.
2. This paper facilitates a range of downstream applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.
summary: The paper introduces a generative model designed for creating 3D content from text, image, and 3D conditions with reference-augmented diffusion. Phidias aims to enhance the quality, generalization, and controllability of 3D generation by leveraging existing 3D models as references. The model integrates three key components: meta-ControlNet for dynamically adjusting conditioning strength, dynamic reference routing to address misalignments between input images and 3D references, and self-reference augmentations for self-supervised training with a progressive curriculum. 

These components collectively improve generative performance over existing methods, offering a unified framework for 3D generation using diverse inputs. Phidias enables various applications, including retrieval-augmented image-to-3D, text-to-3D, theme-aware 3D-to-3D generation, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.
weaknesses: 1. Dependence on Retrieval Quality: As stated by the author The performance of Phidias is contingent upon the quality and relevance of the retrieved 3D reference models, which may not always be optimal.
2. Database Limitations: The current 3D database used for retrieval is limited in size, which could restrict the diversity and accuracy of reference models available for augmentation. Moreover, the model necessitates an additional dataset for inference, which may further restrict its applicability.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. why the point cloud for retrieval only contains 10K points, is it limitation stems from constraints imposed by Uni3D?
2. Why are the performences in Figure 14 so different from the results of the last row of figures in Figure 4? It looks like that some case's pose in Figure 4 doesn't match the input image, either.
3. The time and resource consumption for retrieval during each inference is not clearly stated
soundness: 3
strengths: 1. This paper presents a well-founded solution for multi-view generation, effectively addressing challenges such as the generation of unseen views and other ill-posed problems.
2. This paper is well-organized and clearly written, making it accessible to readers.
summary: This paper introduces Phidias, a RAG-based framework for high-fidelity 3D content creation from multimodal inputs, including text, images, and 3D references. The framework introduces key innovations include Meta-ControlNet to dynamically modulate reference influence, dynamic reference routing to progressively refine outputs through a coarse-to-fine strategy, and self-reference augmentation to enhance generalization. Phidias enables sparse-view reconstruction, theme-based model variation, and user-interactive generation, demonstrating clear superiority over state-of-the-art approaches in both geometric precision and visual fidelity.
weaknesses: 1. A significant weakness is the limited analysis of the retrieval mechanism for 3D references, which plays a critical role in guiding generation. The reliance on existing 3D models introduces a potential bottleneck, as Phidias’s performance may degrade if the retrieved references are irrelevant or of poor quality. 
2. The control mechanism introduced via Meta-ControlNet is promising, but the paper does not delve deeply into more challenging settings. For example, scenarios where the input concept conflicts with the reference.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - I strongly support the task of combining reference 3D models with images for 3D generation as proposed in the paper, especially considering its relevance to the gaming industry and asset creation. Some related 3D generation methods discuss a texture editing task using complex images to guide coarse 3D object generation, which is highly relevant to this work. Could the authors provide some discussion or analysis on how to improve the method when there are significant differences between the reference image and the reference 3D model? This could be addressed in future work.

- I am inclined to accept this paper, as I also work on related tasks. Currently, I rate it as 6 but would likely increase the rating to 8 during the rebuttal phase.
soundness: 3
strengths: - The introduction of meta-ControlNet, combined with 3D reference models, provides strong guidance for the generation process, resulting in high-quality 3D outputs with excellent geometric consistency.

- As a reviewer familiar with this field, I believe that the combination of 3D reference models and images for generating 3D results has significant industrial value and could greatly inform future work in 4D generation as well.

- The method is simple, easy to understand, and the paper is clearly written.
summary: This paper proposes a new 3D generation method that introduces a meta-ControlNet and dynamic reference routing, leveraging user-provided 3D reference models and images to produce high-quality, controllable 3D object synthesis.
weaknesses: - Obtaining a reference 3D model can be quite challenging. Although this is valuable in certain application scenarios, it can be difficult to find appropriate reference 3D models in extreme cases. I recommend that the authors add this limitation to the paper.

- While the strategy of combining ControlNet for generation appears straightforward, this could be criticized for lacking novelty. However, in my opinion, simplicity often leads to elegant and effective solutions, and I don’t consider this a reason for rejection.

Meta-review: This paper proposes a unified approach for text or image guided 3D generation via multi-view diffusion models, which is further guided with a retrieved 3D object, for the very first time. In order to resolve the fundamental conflict caused by the incongruence between the guiding concept image and the 3D object, the authors propose three strategies to mitigate it: a meta-ControlNet to adaptively control the guidance from the 3D model's CCM images; using multiple resolutions for the 3D object as the reverse diffusion process proceeds and using a perturbation-based strategy for training with progressively more misaligned 3D objects in a self-reconstruction training setting. The work shows state-of-the-art performance and many novel downstream applications.

The significance of this work includes that it is the first unified framework to include a guiding 3D object for 3D generation, which can be helpful in many downstream application such as asset creation by artists in interactive settings. It is also broadly applicable to all 3D generation problems.

Decision: Accept (Poster)