3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

## Abstract

Abstract Multi-view image diffusion models have significantly advanced open-domain 3D object generation. However, most existing models rely on 2D network architectures that lack inherent 3D biases, resulting in compromised geometric consistency. To address this challenge, we introduce 3D-Adapter, a plug-in module designed to infuse 3D geometry awareness into pretrained image diffusion models. Central to our approach is the idea of 3D feedback augmentation : for each denoising step in the sampling loop, 3D-Adapter decodes intermediate multi-view features into a coherent 3D representation, then re-encodes the rendered RGBD views to augment the pretrained base model through feature addition. We study two variants of 3D-Adapter: a fast feed-forward version based on Gaussian splatting and a versatile training-free version utilizing neural fields and meshes. Our extensive experiments demonstrate that 3D-Adapter not only greatly enhances the geometry quality of text-to-multi-view models such as Instant3D and Zero123++, but also enables high-quality 3D generation using the plain text-to-image Stable Diffusion. Furthermore, we showcase the broad application potential of 3D-Adapter by presenting high quality results in text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks.

## 1 Introduction

Diffusion models (Ho et al., ; Song et al., ) have recently made significant strides in visual synthesis, achieving production-quality results in image generation (Rombach et al., ). However, the success of 2D diffusion does not easily translate to the 3D domain due to the scarcity of large-scale datasets and the lack of a unified, neural-network-friendly representation (Po et al., ). To bridge the gap between 2D and 3D generation, novel-view or multi-view diffusion models (Liu et al., ; Long et al., ; Shi et al., ; Li et al., ; Chen et al., ; Voleti et al., ) have been finetuned from pretrained image or video models, facilitating 3D generation via a 2-stage paradigm involving multi-view generation followed by 3D reconstruction (Liu et al., ; ; Li et al., ; Wang et al., ; Xu et al., ). While these models generally exhibit good *global semantic consistency* across different view angles, a pivotal challenge lies in achieving *local geometry consistency*. This entails ensuring precise 2D–3D alignment of local features and maintaining geometric plausibility. Consequently, these two-stage methods often suffer from floating artifacts or produce blurry, less detailed 3D outputs (Fig. 1 (c)).

To enhance local geometry consistency, previous works have explored inserting 3D representations and rendering operations into the denoising sampling loop, synchronizing either the denoised outputs (Gu et al., ; Xu et al., ; Zuo et al., ) or the noisy inputs (Liu et al., ; Gao et al., ) of the network, a process we refer to as *I/O sync*. However, we observe that I/O sync generally leads to less detailed, overly smoothed textures and geometry (Fig. 1 (a)). This phenomenon can be attributed to two factors:

- •
Diffusion models typically incorporate residual connections (as seen in U-Nets (Ho et al., ) and transformers (Peebles & Xie, )) to ensure that important information is preserved during denoising. However, 3D reconstruction and rendering are lossy operations that disrupt residual connections.
- •
For texture generation methods in Liu et al. (); Gao et al. (), I/O sync is equivalent to multi-view score averaging, which theoretically leads to mode collapse, causing the loss of fine details in the generated outputs. More theoretical analysis can be found in Appendix A.

Figure: Figure 1: Comparison between the results generated by different architectures. Texture refinement is enabled for text-to-3D, image-to-3D, and text-to-avatar.
Refer to caption: x1.png

To overcome the limitations of I/O sync, we propose a novel approach termed *3D feedback augmentation*, which attaches a 3D-aware parallel branch to the base model, while preserving the original network topology and avoiding score averaging. Essentially, this branch decodes intermediate features from the base model to reconstruct an intermediate 3D representation, which is then rendered, encoded, and fed back into the base model through feature addition, thus augmenting 3D awareness. Specifically, when using a denoising U-Net as the base model, we implement 3D feedback augmentation as *3D-Adapter*, which reuses a copy of the original U-Net with an additional 3D reconstruction module to build the parallel branch. Thanks to its ControlNet-like (Zhang et al., ) model reuse, 3D-Adapter requires minimal or, in cases where suitable off-the-shelf ControlNets are available, zero training.

To demonstrate its flexibility, we present two variants of 3D-Adapter in this paper.

#### Fast 3D-Adapter using feed-forward Gaussian reconstruction.

Given an off-the-shelf multi-view diffusion model, we use its original U-Net and VAE to decode intermediate denoised images, which are then fed into the feed-forward Gaussian Reconstruction Model (GRM) (Xu et al., ) to obtain a 3D Gaussian splatting (3DGS) (Kerbl et al., ). Subsequently, the rendered RGBD images are re-encoded using a finetuned U-Net encoder (ControlNet), with features fused back into the original U-Net decoder to produce consistent denoised outputs. We finetune the GRM and ControlNet model in 2 phases, and demonstrate that 3D-Adapter can greatly enhance the geometry consistency of the base model, such as Instant3D (Li et al., ) and Zero123++ (Shi et al., ).

#### Flexible training-free 3D-Adapter using 3D optimization and pretrained ControlNets.

Aside from using feed-forward reconstruction models, we also explore aggregating intermediate images of multiple independent views by optimizing an Instant-NGP neural radiance field (NeRF) (Müller et al., ; Mildenhall et al., ) and DMTet mesh (Shen et al., ), which enables highly flexible choice of camera layouts. For popular base models (e.g., Stable Diffusion (Rombach et al., )) with off-the-shelf ControlNets, the rendered RGBD images can be then re-encoded using a combination of “tile” and depth ControlNets, eliminating the need for further finetuning. For texture generation, optimization can be replaced by the more efficient texture backprojection approach.

## 2 Related Work

#### 3D-native diffusion models.

We define 3D-native diffusion models as injecting noise directly into the 3D representations (or their latents) during the diffusion process. Early works (Bautista et al., ; Dupont et al., ) have explored training diffusion models on low-dimensional latent vectors of 3D representations, but are highly limited in model capacity. A more expressive approach is training diffusion models on triplane representations (Chan et al., ), which works reasonably well on closed-domain data (Chen et al., ; Shue et al., ; Gupta et al., ; Wang et al., ). Directly working on 3D grid representations is more challenging due to the cubic computation cost (Müller et al., ), so an improved multi-stage sparse volume diffusion model is proposed in Zheng et al. (). In general, 3D-native diffusion models face the challenge of limited data, and sometimes the extra cost of preprocessing the training data into 3D representations (e.g., NeRF), which limit their scalability.

#### Novel-/multi-view diffusion models.

Trained on multi-view images of 3D scenes, view diffusion models inject noise into the images (or their latents) and thus benefit from existing 2D diffusion research. Watson et al. () have demonstrated the feasibility of training a conditioned novel view generative model using purely 2D architectures. Subsequent works (Shi et al., ; ; Liu et al., ; Long et al., ) achieve open-domain novel-/multi-view generation by fine-tuning the pre-trained 2D Stable Diffusion model (Rombach et al., ). However, 3D consistency in these models is generally limited to global semantic consistency because it is learned solely from data, without any inherent architectural bias to support detailed local alignment.

#### Two-stage 3D generation.

Two-stage methods (Fig. 2 (a)) link view diffusion with multi-view 3D reconstruction models, offering a significant speed advantage over score distillation sampling (SDS) (Poole et al., ). Liu et al. () initially combine Zero-1-to-3 (Liu et al., ) with SparseNeuS (Long et al., ), and subsequent works (Liu et al., ; Xu et al., ; Long et al., ; Tang et al., ) have further explored more effective multi-view diffusion models and enhanced reconstruction methods. A common issue with two-stage approaches is that existing reconstruction methods, often designed for or trained under conditions of perfect consistency, lack robustness to local geometric inconsistencies. This may result in floaters and blurry textures.

#### 3D-aware view diffusion and I/O sync.

To introduce 3D-awareness in single-image diffusion models, Anciukevicius et al. (); Tewari et al. () elevate image features into 3D NeRFs to render denoised views. Extending this concept to multi-view diffusion, DMV3D (Xu et al., ) represents an end-to-end feed-forward output sync approach. However, these methods often produce blurry outputs due to the information loss in the absence of residual connections. Liu et al. () attempt to preserve these connections through input sync and attention-based feature fusion, yet it lacks a robust architecture, leading to subpar quality as noted in Liu et al. (). Alternatively, employing feed-forward Gaussian reconstruction models (Xu et al., ; Tang et al., ) for output sync preserves more information but often struggles to coherently fuse inconsistent views, resulting in only marginal improvements in 3D consistency as seen in Zuo et al. (). On the other hand, optimization-based I/O sync methods in  Gu et al. (); Liu et al. (); Gao et al. () either require strong local conditioning or suffer from the pitfalls of score averaging, resulting in overly smoothed textures.

## 3 Preliminaries

Let $p({\bm{x}}|{\bm{c}})$ denote the real data distribution, where ${\bm{c}}$ is the condition (e.g., text prompts) and ${\bm{x}}\in\mathbb{R}^{V\times 3\times H\times W}$ denotes the $V$-view images of a 3D object. A Gaussian diffusion model defines a diffusion process that progressively perturb the data point by adding an increasing amount of Gaussian noise $\bm{\epsilon}\sim\mathcal{N}(0,{\bm{I}})$, yielding the noisy data point ${\bm{x}}_{t}\coloneqq\alpha_{t}{\bm{x}}_{i}+\sigma_{t}\bm{\epsilon}$ at diffusion timestep $t$, with pre-defined noise schedule scalars $\alpha_{t},\sigma_{t}$. A denoising network $D$ is then tasked with removing the
noise from ${\bm{x}}_{t}$ to predict the denoised data point. The network is typically trained with an L2 denoising loss:

$$ $\mathcal{L}_{\text{diff}}=\mathbb{E}_{t,{\bm{c}},{\bm{x}},\bm{\epsilon}}{ \mathopen{}\mathclose{{}\left[\frac{1}{2}w_{t}^{\text{diff}}\mathopen{} \mathclose{{}\left\|D\mathopen{}\mathclose{{}\left({\bm{x}}_{t},{\bm{c}},t} \right)-{\bm{x}}}\right\|^{2}}\right]},$ (1) $$

where $t\sim\mathcal{U}(0,T)$, and $w_{t}^{\text{diff}}$ is an empirical time-dependent weighting function (e.g., SNR weighting $w_{t}^{\text{diff}}=\mathopen{}\mathclose{{}\left(\alpha_{t}/\sigma_{t}}\right
)^{2}$). At inference time, one can sample from the model using efficient ODE/SDE solvers (Lu et al., ) that recursively denoise ${\bm{x}}_{t}$, starting from an initial noisy state ${\bm{x}}_{t_{\text{init}}}$, until reaching the denoised state ${\bm{x}}_{0}$. Note that in latent diffusion models (LDM) (Rombach et al., ), both diffusion and denoising occur in the latent space. For brevity, we do not differentiate between latents and images in the equations and figures.

#### I/O sync baseline.

We broadly define I/O sync as inserting a 3D representation and a rendering/projecting operation at the input or output end of the denoising network to synchronize multiple views. Input sync is primarily used for texture generation, and it is essentially equivalent to output sync, assuming linearity and synchronized initialization
(detailed in the Appendix A). Therefore, for simplicity, this paper considers only output sync as the baseline. As depicted in Fig. 2 (b), a typical output sync model can be implemented by reconstructing a 3D representation from the denoised outputs $\hat{{\bm{x}}}_{t}$, and then re-rendering the views from 3D to replace the original outputs.

Figure: Figure 2: Comparison between different architectures. For brevity and generality, we omit the VAE decoders in LDMs Rombach et al. (), the condition encoders such as text encoders, the rendered alpha channel, and the noisy RGB input for the ControlNet.
Refer to caption: x2.png

## 4 3D-Adapter

To overcome the limitations of I/O sync, our key idea is the 3D feedback augmentation architecture, which involves reconstructing a 3D representation midway through the denoising network and feeding the rendered views back into the network using ControlNet-like feature addition. This architecture preserves the original flow of the base model while effectively leveraging its inherent priors.

Based on this idea, we propose the 3D-Adapter, as illustrated in Fig. 2 (c). For each denoising step, after passing the input noisy views ${\bm{x}}_{t}$ through the base U-Net encoder, we use a copy of the base U-Net decoder to first output intermediate denoised views $\hat{{\bm{x}}}^{\prime}_{t}$. A 3D reconstruction model then lifts these intermediate views to a coherent 3D representation, from which consistent RGBD views $\tilde{{\bm{x}}}_{t}$ are rendered and fed back into the network through a ControlNet encoder. The output features from this encoder are added to the base encoder features, which are then processed again by the base decoder to produce the final denoised output $\hat{{\bm{x}}}_{t}$. The full denoising step can be written as:

$$ $\hat{{\bm{x}}}_{t}=D_{\text{aug}}\mathopen{}\mathclose{{}\left({\bm{x}}_{t},{ \bm{c}},t,\smash[tb]{\underbrace{R(\overbrace{D({\bm{x}}_{t},{\bm{c}},t)}^{ \hat{{\bm{x}}}^{\prime}_{t}})}_{\tilde{{\bm{x}}}_{t}}}}\right)\vphantom{ \underbrace{R(\overbrace{D({\bm{x}}_{t},{\bm{c}},t)}^{\hat{{\bm{x}}}^{\prime}_ {t}})}_{\tilde{{\bm{x}}}_{t}}}.$ (2) $$

where $R$ denotes 3D reconstruction and rendering, and $D_{\text{aug}}$ denotes the augmented U-Net with feedback ControlNet.

Various 3D-Adapters can be implemented depending on the choice of base model and 3D reconstruction method, as described in the following subsections.

### 4.1 3D-Adapter Using Feed-Forward GRM

GRM (Xu et al., ) is a feed-forward sparse-view 3D reconstruction model based on 3DGS. In this section, we describe the method to train GRM-based 3D-Adapters for the text-to-multi-view model Instant3D (Li et al., ) and image-to-multi-view model Zero123++ (Shi et al., ).

#### Training phase 1: finetuning GRM.

GRM is originally trained on consistent ground truth input views, and is not robust to low-quality intermediate views, which are often highly inconsistent and blurry. To overcome this challenge, we first finetune GRM using the intermediate images $\hat{{\bm{x}}}^{\prime}_{t}$ as inputs, where the time $t$ is randomly sampled just like in the diffusion loss. In this training phase, we freeze the base encoder and decoder of the U-Net, and initialize GRM with the official checkpoint for finetuning. As shown in Fig. 2 (c), a rendering loss $\mathcal{L}_{\text{rend}}$ is employed to supervise GRM with ground truth novel views. Specifically, both the appearance and geometry are supervised using the combination of an L1 loss $L_{1}^{\text{RGBAD}}$ on RGB/alpha/depth maps, and an LPIPS loss $L_{\text{LPIPS}}^{\text{RGB}}$ (Zhang et al., ) on RGB only. The loss is computed on 16 rendered views $\tilde{{\bm{X}}}_{t}\in\mathbb{R}^{16\times 5\times 512\times 512}$ and the corresponding ground truth views ${\bm{X}}_{\text{gt}}$, given by:

$$ $\mathcal{L}_{\text{rend}}=\mathbb{E}_{t,{\bm{c}},{\bm{x}},\epsilon}{\mathopen{ }\mathclose{{}\left[w^{\text{rend}}_{t}\mathopen{}\mathclose{{}\left(L_{1}^{ \text{RGBAD}}\mathopen{}\mathclose{{}\left(\tilde{{\bm{X}}}_{t},{\bm{X}}_{ \text{gt}}}\right)+L_{\text{LPIPS}}^{\text{RGB}}\mathopen{}\mathclose{{}\left( \tilde{{\bm{X}}}_{t},{\bm{X}}_{\text{gt}}}\right)}\right)}\right]},$ (3) $$

where $w^{\text{rend}}_{t}$ is a time-dependent weighting function. We use $w^{\text{rend}}_{t}=\mathopen{}\mathclose{{}\left.\alpha_{t}\middle/\sqrt{{
\alpha_{t}}^{2}+{\sigma_{t}}^{2}}}\right.$. The L1 RGBAD loss also employs channel-wise weights, which are detailed in our code.

#### Training phase 2: finetuning feedback ControlNet.

In this training phase, we freeze all modules except the feedback ControlNet encoder, which is initialized with the base U-Net weights for finetuning. Following standard ControlNet training method, we employ the diffusion loss in Eq. (1) to finetune the RGBD feedback ControlNet. To accelerate convergence, we feed rendered RGBD views of a less noisy timestep $\tilde{{\bm{x}}}_{0.1t}$ to the ControlNet during training.

#### Inference: guided 3D feedback augmentation.

One potential issue is that the ControlNet encoder may overfit the finetuning dataset, resulting in an undesirable bias that persists even if the rendered RGBD $\tilde{{\bm{x}}}_{t}$ is replaced with a zero tensor. To mitigate this issue, inspired by classifier-free guidance (CFG) (Ho & Salimans, ), we replace $\hat{{\bm{x}}}_{t}$ with the guided denoised views $\hat{{\bm{x}}}_{t}^{\text{G}}$ during inference to cancel out the ControlNet bias:

$$ $\hat{{\bm{x}}}_{t}^{\text{G}}=\lambda_{\text{aug}}\mathopen{}\mathclose{{} \left(D_{\text{aug}}\mathopen{}\mathclose{{}\left({\bm{x}}_{t},{\bm{c}},t, \tilde{{\bm{x}}}_{t}}\right)-D_{\text{aug}}\mathopen{}\mathclose{{}\left({\bm{ x}}_{t},{\bm{c}},t,\bm{0}}\right)}\right)+\lambda_{\bm{c}}D\mathopen{} \mathclose{{}\left({\bm{x}}_{t},{\bm{c}},t}\right)+(1-\lambda_{\bm{c}})D \mathopen{}\mathclose{{}\left({\bm{x}}_{t},\bm{0},t}\right),$ (4) $$

where $\lambda_{\bm{c}}$ is the regular condition CFG scale, and $\lambda_{\text{aug}}$ is our feedback augmentation guidance scale. During training, we feed zero tensors to the ControlNet with a 20% probability, so that $D_{\text{aug}}\mathopen{}\mathclose{{}\left({\bm{x}}_{t},{\bm{c}},t,\bm{0}}\right)$ learns a meaningful dataset bias.

#### Training details.

We adopt various training techniques to reduce the memory footprint, including mixed precision training, 8-bit AdamW (Dettmers et al., ; Loshchilov & Hutter, ), gradient checkpointing, and deferred back-propagation (Xu et al., ; Zhang et al., ). The adapter is trained with a total batch size of 16 objects on 4 A6000 GPUs. In phase 1, GRM is finetuned with a small learning rate of $5\times 10^{-6}$ for 2k iterations (for Instant3D) or 4k iterations (for Zero123++). In phase 2, ControlNet is finetuned with a learning rate of $1\times 10^{-5}$ for 5k iterations. 47k (for Instant3D) or 80k (for Zero123++) objects from a high-quality subset of Objaverse (Deitke et al., ) are rendered as the training data.

### 4.2 3D-Adapter Using 3D Optimization/Texture Backprojection

Feed-forward 3D reconstruction methods, like GRM, are typically constrained by specific camera layouts. In contrast, more flexible reconstruction approaches, such as optimizing an Instant-NGP NeRF (Müller et al., ; Mildenhall et al., ) and DMTet mesh (Shen et al., ), can accommodate diverse camera configurations and achieve higher-quality results, although they require longer optimization times.

In this subsection, we design an optimization-based 3D adapter, using Stable Diffusion v1.5 (Rombach et al., ) as the base model, where each view is denoised independently, with the adapter being the only component that shares information across views.

During the sampling process, the adapter performs NeRF optimization for the first 60% of the denoising steps. It then converts the color and density fields into a texture field and DMTet mesh, respectively, to complete the remaining 40% denoising steps. All optimizations are incremental, meaning the 3D state from the previous denoising step is retained to initialize the next. As a result, only 96 optimization steps are needed per denoising step. Alternatively, for texture generation only, multi-view aggregation can be achieved by backprojecting the views into UV space and blending the results according to visibility. For feedback augmentation, the off-the-shelf “tile” ControlNet, originally trained for superresolution, performs effectively for RGB feedback due to its robustness to blurry inputs, while the off-the-shelf depth ControlNet can naturally handle depth feedback.

It should be noted that, when using single-image diffusion as the base model, 3D-Adapter alone cannot provide the necessary global semantic consistency for 3D generation. Therefore, it should be complemented with other sources of consistency, such initialization with partial noise like SDEdit (Meng et al., ) or extra conditioning from ControlNets. For text-to-avatar generation, we use rendered views of a human template for SDEdit initialization with the initial timestep $t_{\text{init}}$ set to $0.88T$, and employ an extra pose ControlNet for conditioning. For text-to-texture generation, global consistency is usually good due to ground truth depth conditioning.

#### Optimization loss functions.

For NeRF/mesh optimization, we employ L1 and LPIPS losses on RGB and alpha maps, and total variation (TV) loss on normal maps. Additionally, we enforce stronger geometry regularization using ray entropy loss for NeRF, and Laplacian smoothing loss (Sorkine et al., ) plus normal consistency loss for mesh, making the optimization more robust to imperfect intermediate views $\hat{{\bm{x}}}^{\prime}_{t}$. More details can be found in Appendix C.

### 4.3 Texture Post-Processing

To further enhance the visual quality of objects generated from text, we implement an optional texture refinement pipeline as a post-processing step. First, when using the GRM-based 3D-Adapter, we convert the generated 3DGS into a textured mesh via TSDF integration. With the initial mesh, we render six surrounding views and apply per-view SDEdit refinement ($t_{\text{init}}=0.5T$) using Stable Diffusion v1.5 with “tile” ControlNet.
Finally, the refined views are aggregated into the UV space using texture backprojection. For fair comparisons in the experiments, this refinement step is not used by default unless specified otherwise.

## 5 Experiments

### 5.1 Evaluation Metrics

To evaluate the results generated by 3D-Adapter and compare them to various baselines and competitors, we compute the following metrics based on the rendered images of the generated 3D representations:

- •
CLIP score (Radford et al., ; Jain et al., ): Evaluates image–text alignment in text-to-3D, text-to-texture, and text-to-avatar tasks. We use CLIP-ViT-L-14 for all CLIP-related metrics.
- •
Aesthetic score (Schuhmann et al., ): Assesses texture details. The user study in Wu et al. () revealed that this metric highly correlates with human preference in texture details.
- •
FID (Heusel et al., ): Measures the visual quality when reference test set images are available, applicable to text-to-3D models trained on common dataset and all image-to-3D models.
- •
CLIP similarity (Radford et al., ), LPIPS (Zhang et al., ), SSIM (Wang et al., ), PSNR: Evaluates novel view fidelity in image-to-3D.
- •
Mean depth distortion (MDD) (Yu et al., ; Huang et al., ): Assesses the geometric quality of generated 3DGS. Lower depth distortion indicates less floaters or fuzzy surfaces, reflecting better geometry consistency. More details can be found in Appendix D.
- •
CLIP t-less score: Assesses the geometric quality of generated meshes by computing the CLIP score between shaded textureless renderings and texts appended with “textureless 3D model”.

**Table 1: Text-to-3D: comparison with baselines, parameter sweep, and ablation studies.**
| ID | Method | CLIP↑ | Aesthetic↑ | FID↓ | MDD<br>$/10^{-7}$<br>↓ |
| --- | --- | --- | --- | --- | --- |
| A0 | Two-stage (GRM) | 27.02 | 4.48 | 34.19 | 0232.4 |
| A1 | A0 + I/O sync | 24.62 | 4.35 | 63.22 | 1239.7 |
| A2 | A1 w/ finetuned GRM | 22.57 | 4.16 | 70.35 | 0001.7 |
| B0 | 3D-Adapter $\lambda_{\text{aug}}$=1 | 27.31 | 4.54 | 32.81 | 0004.7 |
| B1 | 3D-Adapter $\lambda_{\text{aug}}$=2 | 27.22 | 4.52 | 33.46 | 0003.9 |
| B2 | 3D-Adapter $\lambda_{\text{aug}}$=4 | 26.99 | 4.45 | 34.34 | 0003.2 |
| B3 | 3D-Adapter $\lambda_{\text{aug}}$=8 | 25.47 | 4.28 | 39.36 | 0025.3 |
| C0 | B0 w/o feedback | 27.18 | 4.55 | 33.13 | 0007.6 |
| C1 | B0 w/o bias canceling | 25.49 | 4.36 | 42.20 | 0003.6 |

Figure: Figure 3: Comparison on text-to-3D generation
Refer to caption: x3.png

### 5.2 Text-to-3D Generation

For text-to-3D generation, we adopt the GRM-based 3D-Adapter with Instant3D U-Net as the base model. All results are generated using EDM Euler ancestral solver (Karras et al., ) with 30 denoising steps and mean latent initialization (Appendix B.2). The inference time is around 0.7 sec per step, and detailed inference time analysis is presented in Appendix B.3. For evaluation, we first compare 3D-Adapter with the baselines and conduct ablation studies on a validation set of 379 BLIP-captioned objects sampled from a high-quality subset of Objaverse (Li et al., ; Deitke et al., ). The results are shown in Table 2, with the rendered images from the dataset used as real samples when computing the FID metric. Subsequently, we benchmark 3D-Adapter on the same test set as GRM (Xu et al., ), consisting of 200 prompts, to make fair comparisons to the previous SOTAs in Table 2. Qualitative results are shown in Fig. 3.

#### Baselines.

The two-stage GRM (A0) exhibits good visual quality, but the MDD metric is magnitudes higher than that of our 3D-Adapter (B0–B3) due to the highly ambiguous geometry caused by local misalignment. Naively rewiring it into an I/O sync model (A1) worsens the results, as the original GRM is trained only on rendered ground truths ${\bm{x}}$ and cannot handle the imperfections of the denoised views $\hat{{\bm{x}}}$. However, when using the GRM model fine-tuned according to our method (Eq. 3), the model (A2) achieves the lowest possible MDD with nearly perfect geometry consistency, but it suffers significantly from mode collapse and struggles to generate meaningful content.

**Table 3: Image-to-3D: comparison with previous SOTAs.**
| Method | Type | PSNR↑ | SSIM↑ | LPIPS↓ | CLIP<br>sim<br>↑ | FID↓ |
| --- | --- | --- | --- | --- | --- | --- |
| One-2-3-45 | Mesh | 17.84 | 0.800 | 0.199 | 0.832 | 89.4 |
| TriplaneGaussian | GS | 16.81 | 0.797 | 0.257 | 0.840 | 52.6 |
| Shap-E | Mesh | 15.45 | 0.772 | 0.297 | 0.854 | 56.5 |
| LGM | GS | 16.90 | 0.819 | 0.235 | 0.855 | 42.1 |
| DreamGaussian | Mesh | 19.19 | 0.811 | 0.171 | 0.862 | 57.6 |
| Wonder3D | Mesh | 17.29 | 0.815 | 0.240 | 0.871 | 55.7 |
| One-2-3-45++ | Mesh | 17.79 | 0.819 | 0.219 | 0.886 | 42.1 |
| GRM | GS | 20.10 | 0.826 | 0.136 | 0.932 | 27.4 |
| 3D-Adapter (ours) | GS | 20.38 | 0.840 | 0.135 | 0.936 | 20.2 |
| 3D-Adapter<br>+ TSDF (ours) | Mesh | 20.34 | 0.840 | 0.135 | 0.933 | 21.7 |

Figure: Figure 4: Comparison of mesh-based image-to-3D methods on the GSO test set.
Refer to caption: x4.png

#### Parameter sweep on λ aug subscript 𝜆 aug \lambda_{\text{aug}} italic_λ start_POSTSUBSCRIPT aug end_POSTSUBSCRIPT and ablation studies.

The 3D-Adapter with a feedback augmentation guidance scale $\lambda_{\text{aug}}=1$ (B0) achieves the best visual quality among all variants and significantly better geometry quality than A0. As $\lambda_{\text{aug}}$ increases, the MDD metric continues to improve, but at the expense of visual quality. A very large $\lambda_{\text{aug}}$ (B3) unsurprisingly worsens the results, similar to a large CFG scale. Disabling feedback augmentation (C0, equivalent to $\lambda_{\text{aug}}=0$) notably impacts the geometric quality, as evidenced by the worse MDD metric, although it still outperforms the baseline (A0) thanks to our robust GRM fine-tuning. Without bias canceling (C1), all visual metrics degrade significantly, which substantiates the effectiveness of 3D feedback guidance (Eq. (4)).

#### Comparison with other competitors.

Built on top of GRM, our 3D-Adapter ($\lambda_{\text{aug}}$=1) further advances the benchmark, outperforming previous SOTAs in text-to-3D (Jun & Nichol, ; Tang et al., ; Li et al., ; Shi et al., ; Xu et al., ) as shown in Table 2 and Fig. 3.

### 5.3 Image-to-3D Generation

For image-to-3D generation, we adopt the same approach used for text-to-3D generation, except for employing Zero123++ U-Net as the base model. We follow the same evaluation protocol as in Xu et al. (), using 248 GSO objects (Downs et al., ) as the test set. As shown in Table 4, 3D-Adapter ($\lambda_{\text{aug}}$=1) outperforms the two-stage GRM and other competitors (Liu et al., ; Zou et al., ; Jun & Nichol, ; Tang et al., ; ; Long et al., ; Liu et al., ; Xu et al., ) on all metrics. Moreover, the quality loss in converting the generated 3DGS to mesh via TSDF is almost negligible. We present qualitative comparisons of the meshes generated by 3D-Adapter and other methods in Fig. 4.

**Table 5: Text-to-texture: comparison with baselines.**
| Methods | CLIP↑ | Aesthetic↑ |
| --- | --- | --- |
| Two-stage baseline | 25.82 | 4.85 |
| I/O sync baseline | 26.05 | 4.68 |
| 3D-Adapter | 26.40 | 4.85 |

Figure: Figure 5: Comparison on text-to-texture generation.
Refer to caption: x5.png

### 5.4 Text-to-Texture Generation

For text-to-texture evaluation, 3D-Adapter employs fast texture backprojection to blend multiple views for intermediate timesteps, and switches to high-quality texture field optimization (similar to NeRF) for the final timestep. A community Stable Diffusion v1.5 variant, DreamShaper 8, is adopted as the base model. During the sampling process, 32 surrounding views are used initially, and this number is gradually reduced to 7 views during the denoising process to reduce computation in later stages. We adopt the EDM Euler ancestral solver with 24 denoising steps. The total inference time is around 1.5 minutes per object on a single RTX A6000 GPU (including UV unwrapping), which is faster than the competitors (SyncMVD (Liu et al., ): $\sim$1.9 min, TEXTure (Richardson et al., ): $\sim$2.0 min, Text2Tex (Chen et al., ): $\sim$11.2 min). 92 BLIP-captioned objects are sampled from a high-quality subset of Objaverse as our test set.

#### Comparison with baselines.

As shown in Table 6 and Fig. 5, the two-stage baseline has good texture details but notably worse CLIP score due to poor consistency. The I/O sync baseline has much better consistency, but it sacrifices details, resulting in the worst aesthetic score. In comparison, 3D-Adapter excels in both metrics, producing detailed and consistent textures.

#### Comparison with other competitors.

Quantitatively, Table 6 demonstrates that 3D-Adapter significantly outperforms previous SOTAs on both metrics. Interestingly, even our two-stage baseline in Table 6 surpasses the competitors, which can be attributed to our use of texture field optimization and community-customized base model. Qualitative results in Fig. 5 reveal that previous methods are generally less robust compared to 3D-Adapter and may produce artifacts in some cases.

Figure: Figure 6: Comparison on text-to-avatar generation using the same pose template.
Refer to caption: x6.png

### 5.5 Text-to-Avatar Generation

For text-to-avatar generation, the optimization-based 3D-Adapter is adopted with a custom pose ControlNet for Stable Diffusion v1.5, which provides extra conditioning given a human pose template. 32 full-body views and 32 upper-body views are selected for denoising, capturing both the overall figure and face details. These are later reduced to 12 views during the denoising process. We use the EDM Euler ancestral solver with 32 denoising steps, with an inference time of approximately 7 minutes per object on a single RTX A6000 GPU (including 0.5 minutes on CPU-based UV unwrapping). Texture editing (using text-to-texture pipeline and SDEdit with $t_{\text{init}}=0.3T$) and refinement can be optionally applied to further improve texture details, which costs 1.4 minutes. For evaluation, we compare 3D-Adapter with baselines using 21 character prompts on the same pose template. As shown in Table 4, 3D-Adapter achieves the highest scores across all three metrics, indicating superior appearance and geometry. Fig. 6 reveals that I/O sync produces overly smoothed texture and geometry due to mode collapse, while the two-stage baseline results in noisy, less coherent texture and geometry. These observations also align with the quantitative results in Table 4.

## 6 Conclusion

In this work, we have introduced 3D-Adapter, a plug-in module that effectively enhances the 3D geometry consistency of existing multi-view diffusion models, bridging the gap between high-quality 2D and 3D content creation. We have demonstrated two variants of 3D-Adapter: the fast 3D-Adapter using feed-forward Gaussian reconstruction, and the flexible training-free 3D-Adapter using 3D optimization and pretrained ControlNets. Experiments on text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks have substantiated its all-round competence, suggesting great generality and potential in future extension.

#### Limitations.

3D-Adapter introduces substantial computation overhead, primarily due to the VAE decoding process before 3D reconstruction. In addition, we observe that our finetuned ControlNet for 3D feedback augmentation strongly overfits the finetuning data, which may limit its generalization despite the proposed guidance method. Future work may focus on developing more efficient, easy-to-finetune networks for 3D-Adapter.

#### Acknowledgements

We thank Yinghao Xu and Zifan Shi for sharing the data, code, and results for text-to-3D and image-to-3D evaluation, and other members of Geometric Computation Group, Stanford Computational Imaging Lab, and SU Lab for useful feedback and discussions. This project was in part supported by Vannevar Bush Faculty Fellowship, ARL grant W911NF-21-2-0104, Google, Samsung, and Qualcomm Innovation Fellowship.

## Appendix A Theoretical Analysis of the I/O Sync Technique

When performing diffusion ODE sampling using the common Euler solver, a linear input sync operation is equivalent to syncing the output $\hat{{\bm{x}}}_{t}$ as well as the initialization ${\bm{x}}_{t_{\text{init}}}$. This is because the input ${\bm{x}}_{t}$ can be expressed as a linear combination of all previous outputs $\{{\bm{x}}_{t-\Delta t},{\bm{x}}_{t-2\Delta t},\dots\}$ and the initialization ${\bm{x}}_{t_{\text{init}}}$ by expanding the recursive Euler steps.

Furthermore, linear I/O sync is also equivalent to linear score sync, since the learned score function ${\bm{s}}_{t}\mathopen{}\mathclose{{}\left({\bm{x}}_{t}}\right)$ can also be expressed as a linear combination of the input ${\bm{x}}_{t}$ and output $\hat{{\bm{x}}}_{t}$:

$$ ${\bm{s}}_{t}\mathopen{}\mathclose{{}\left({\bm{x}}_{t}}\right)=-\frac{\hat{\bm {\epsilon}}_{t}}{\sigma_{t}}=\frac{\alpha_{t}\hat{{\bm{x}}}_{t}-{\bm{x}}_{t}}{ {\sigma_{t}}^{2}}$ (5) $$

However, synchronizing the score function, a.k.a. score averaging, is theoretically problematic. Let $p({\bm{x}}|{\bm{c}}_{1}),p({\bm{x}}|{\bm{c}}_{2})$ be two independent probability density functions of a corresponding pixel ${\bm{x}}$ viewed from cameras ${\bm{c}}_{1}$ and ${\bm{c}}_{2}$, respectively. A diffusion model is trained to predict the score function ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{v})$ of the noisy distribution at timestep $t$, defined as:

$$ ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{v})=\nabla_{{\bm{x}}_{t}}\log\int p({\bm{x }}_{t}|{\bm{x}})p({\bm{x}}|{\bm{c}}_{v})d{\bm{x}},$ (6) $$

where $p({\bm{x}}_{t}|{\bm{x}})=\mathcal{N}({\bm{x}}_{t};\alpha_{t}{\bm{x}},\sigma_{t
}^{2}{\bm{I}})$ is a Gaussian perturbation kernel. Ideally, assuming ${\bm{c}}_{1}$ and ${\bm{c}}_{2}$ are independent, combining the two conditional PDFs $p({\bm{x}}|{\bm{c}}_{1})$ and $p({\bm{x}}|{\bm{c}}_{2})$ yields the product $p({\bm{x}}|{\bm{c}}_{1},{\bm{c}}_{2})=\frac{1}{Z}p({\bm{x}}|{\bm{c}}_{1})p({
\bm{x}}|{\bm{c}}_{2})$, where $Z$ is a normalization factor. The corresponding score function should then become ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{1},{\bm{c}}_{2})=\nabla_{{\bm{x}}_{t}}\log
\int p({\bm{x}}_{t}|{\bm{x}})p({\bm{x}}|{\bm{c}}_{1},{\bm{c}}_{2})d{\bm{x}}$.
However, the average of ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{1})$ and ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{2})$ is generally not proportional to ${\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{1},{\bm{c}}_{2})$, i.e.:

$$ $\displaystyle\frac{1}{2}{\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{1})+\frac{1}{2}{ \bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{2})$ $\displaystyle=\frac{1}{2}\nabla_{{\bm{x}}_{t}}\log\mathopen{}\mathclose{{} \left(\int p({\bm{x}}_{t}|{\bm{x}})p({\bm{x}}|{\bm{c}}_{1})d{\bm{x}}}\right) \mathopen{}\mathclose{{}\left(\int p({\bm{x}}_{t}|{\bm{x}})p({\bm{x}}|{\bm{c}} _{2})d{\bm{x}}}\right)$ $\displaystyle\ \cancel{\propto}\ \nabla_{{\bm{x}}_{t}}\log\int p({\bm{x}}_{t}| {\bm{x}})\mathopen{}\mathclose{{}\left(\frac{1}{Z}p({\bm{x}}|{\bm{c}}_{1})p({ \bm{x}}|{\bm{c}}_{2})}\right)d{\bm{x}}={\bm{s}}_{t}({\bm{x}}_{t}|{\bm{c}}_{1}, {\bm{c}}_{2}).$ (7) $$

In Fig. 7, we illustrate a simple 1D simulation, showing that score averaging leads to mode collapse, when compared to the real product distribution. This explains the blurry, mean-shaped results produced by the I/O sync baselines.

Figure: Figure 7: A simple 1D simulation illustrating the difference between the score averaged distribution and the actual perturbed product distribution. $*$ denotes convolution, and $\mathcal{N}(x)$ denotes the Gaussian perturbation kernel.
Refer to caption: x7.png

## Appendix B Details on GRM-Based 3D-Adapter

### B.1 ControlNet

The GRM-based 3D-Adapter trains a ControlNet (Zhang et al., ) for feedback augmentation, which has very large model capacity and can easily overfit our relatively small finetuning dataset (e.g., 47k objects for Instant3D). Therefore, using the CFG-like bias subtraction technique (Eq. (4)) is extremely important to the generalization performance, which is already validated in our ablation studies.
Additionally, we disconnect the text prompt input from the ControlNet to further alleviate overfitting.

### B.2 Mean Latent Initialization

Instant3D’s 4-view UNet is sensitive to the initialization method, as noted in the original paper (Li et al., ), which develops an empirical Gaussian blob initialization method to stabilize the background color. In contrast, this paper adopts a more principled mean latent initialization method by computing the mean value $\bar{{\bm{x}}}$ of the VAE-encoded latents of 10K objects in the training set. The initial state is then sampled by perturbing the mean latent with Gaussian noise $\epsilon$:

$$ ${\bm{x}}_{t_{\text{init}}}=\alpha_{t_{\text{init}}}\bar{{\bm{x}}}+\sigma_{t_{ \text{init}}}\bm{\epsilon}.$ (8) $$

**Table 7: GRM-based 3D-Adapter: Inference times (sec) with guidance on a single RTX A6000.**
| Encode | Adapter<br>Decode | VAE<br>Decode | GRM | Render | Adapter<br>Encode | Decode | Adapter<br>total | Overall<br>total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.055 | 0.120 | 0.215 | 0.091 | 0.023 | 0.082 | 0.121 | 0.531 | 0.707 |

### B.3 Inference Time

Detailed module-level inference times per denoising step is shown in Table 7 (with classifier-free guidance and guided 3D feedback augmentation enabled). Apparently, the SDXL VAE decoder is the most expensive module within 3D-Adapter, which may be replaced by a more efficient decoder in future work.

## Appendix C Details on Optimization-Based 3D-Adapter

The optimization-based 3D-Adapter faces the challenge of potentially inconsistent multi-view inputs, especially at the early denoising stage. Existing surface optimization approaches, such as NeuS (Wang et al., ), are not designed to address the inconsistency.
Therefore, we have developed various techniques for the robust optimization of InstantNGP NeRF (Müller et al., ) and DMTet mesh (Shen et al., ), using enhanced regularization and progressive resolution.

#### Rendering.

For each NeRF optimization iteration, we randomly sample a 128$\times$128 image patch from all camera views. Unlike Poole et al. () that computes the normal from NeRF density gradients, we compute patch-wise normal maps from the rendered depth maps, which we find to be faster and more robust.
For mesh rendering, we obtain the surface color by querying the same InstantNGP neural field used in NeRF.
For both NeRF and mesh, Lambertian shading is applied in the linear color space prior to tonemapping, with random point lights assigned to their respective views.

#### RGBA losses.

For both NeRF and mesh, we employ RGB and alpha rendering losses to optimize the 3D parameters so that the rendered views $\tilde{{\bm{x}}}_{t}$ match the intermediate denoised views $\hat{{\bm{x}}}^{\prime}_{t}$. For RGB, we employ a combination of pixel-wise L1 loss and patch-wise LPIPS loss (Zhang et al., ).
For alpha, we predict the target alpha channel from $\hat{{\bm{x}}}^{\prime}_{t}$ using an off-the-shelf background removal network (Lee et al., ) as in Magic123 (Qian et al., ). Additionally, we soften the predicted alpha map using Gaussian blur to prevent NeRF from overfitting the initialization.

#### Normal losses.

To avoid bumpy surfaces, we apply an L1.5 total variation (TV) regularization loss on the rendered normal maps:

$$ $\mathcal{L}_{\text{N}}=\sum_{chw}\mathopen{}\mathclose{{}\left\|w_{hw}\cdot \nabla_{hw}n^{\text{rend}}_{chw}}\right\|^{1.5},$ (9) $$

where $n^{\text{rend}}_{chw}\in\mathbb{R}$ denotes the value of the $C\times H\times W$ normal map at index $(c,h,w)$, $\nabla_{hw}n^{\text{rend}}_{chw}\in\mathbb{R}^{2}$ is the gradient of the normal map w.r.t. $(h,w)$, and $w_{hw}\in[0,1]$ is the value of a foreground mask with edge erosion.

#### Ray entropy loss for NeRF.

To mitigate fuzzy NeRF geometry, we propose a novel ray entropy loss based on the probability of sample contribution. Unlike previous works (Kim et al., ; Metzer et al., ) that compute the entropy of opacity distribution or alpha map, we consider the ray density function:

$$ $p(\tau)=T(\tau)\sigma(\tau),$ (10) $$

where $\tau$ denotes the distance, $\sigma(\tau)$ is the volumetric density and $T(\tau)=\exp{-\int_{0}^{s}\sigma(\tau)d\tau}$ is the ray transmittance. The integral of $p(\tau)$ equals the alpha value of the pixel, i.e., $a=\int_{0}^{+\inf}p(\tau)d\tau$, which is less than $1$. Therefore, the background probability is $1-a$ and a corresponding correction term needs to be added when computing the continuous entropy of the ray as the loss function:

$$ $\mathcal{L}_{\text{ray}}=\sum_{r}\int_{0}^{+\inf}{\negthickspace-p_{r}(\tau) \log{p_{r}(\tau)}d\tau}-\underbrace{(1-a_{r})\log{\frac{1-a_{r}}{d}}}_{\text{ background correction}},$ (11) $$

where $r$ is the ray index, and $d$ is a user-defined “thickness” of an imaginative background shell, which can be adjusted to balance foreground-to-background ratio.

#### Mesh smoothing losses

As per common practice, we employ the Laplacian smoothing loss (Sorkine et al., ) and normal consistency loss to further regularize the mesh extracted from DMTet.

#### Implementation details

The weighted sum of the aforementioned loss functions is utilized to optimize the 3D representation.
At each denoising step, we carry forward the 3D representation from the previous step and perform additional iterations of Adam (Kingma & Ba, ) optimization. During the denoising sampling process, the rendering resolution progressively increases from 128 to 256, and finally to 512 when NeRF is converted into a mesh (for texture generation the resolution is consistently 512). When the rendering resolution is lower than the diffusion resolution 512, we employ RealESRGAN-small (Wang et al., ) for efficient super-resolution.

## Appendix D Details on the Mean Depth Distortion (MDD) Metric

The MDD metric is inspired by the depth distortion loss in Yu et al. (), which proves effective in removing floaters and improving the geometry quality. The depth distortion loss of a pixel is defined as:

$$ $\mathcal{L}_{\text{D}}=\sum_{m,n}{\omega_{m}\omega_{n}|\tau_{m}-\tau_{n}|},$ (12) $$

where $m,n$ index over Gaussians contributing to the ray, $\omega_{m}$ is the blending weight of the $m$-th Gaussian and $\tau_{m}$ is the distance of the intersection point.

To compute the mean depth distortion of a view, we take the sum of depth distortion losses across all pixels and divide it by the sum of alpha values across all pixels:

$$ $\mathit{MDD}=\frac{\sum_{r}{{\mathcal{L}_{\text{D}}}_{r}}}{\sum_{r}{a_{r}}},$ (13) $$

where $r$ is the pixel index.

## Appendix E More Results

We present more qualitative comparisons in Fig. 8, 9, 10, 11, 12.

Figure: Figure 8: More comparisons on text-to-3D generation (part 1).
Refer to caption: x8.png

Figure: Figure 9: More comparisons on text-to-3D generation (part 2).
Refer to caption: x9.png

Figure: Figure 10: More comparisons on image-to-3D generation.
Refer to caption: x10.png

Figure: Figure 11: More comparisons on text-to-texture generation.
Refer to caption: x11.png

Figure: Figure 12: More comparisons on text-to-avatar generation.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: (1) What is the rationale behind the significant drop in metrics in Table 1 when adding IO sync and GRM finetuning to the two-stage baseline? Also GRM finetuning does not appear to be evaluated in the 3D-Adapter case.

(2) The SOTA selection in Table 2 seems debatable; could the authors clarify the choice of methods over more recent approaches?

(3) How does the output quality compare to I/O sync + tiled ControlNet, and how critical is ControlNet’s role during denoising relative to the 3D prior that could be introduced simply by depth conditioning in IO mode?
soundness: 2
strengths: By integrating depth conditioning via ControlNet, the 3D-Adapter provides a straightforward yet effective solution to incorporate 3D priors within 2D diffusion frameworks as a part of denoising process.

The paper presents extensive quantitative and qualitative evaluations across multiple configurations and baselines.

The proposed 3D-Adapter outperforms several prior methods in 3D generation quality, demonstrating improvements on widely used metrics like CLIP score and aesthetic score.
summary: The paper introduces 3D-Adapter, a plug-in module aimed at improving 3D consistency in multi-view diffusion models for 3D generation. By integrating a combination of 3D representation and ControlNet for depth conditioning into the denoising framework, 3D-Adapter enhances 3D structure coherence across views without modifying the core model's topology / weights. The authors present two 3D-Adapter variants: a fast feed-forward method using Gaussian splatting (GRM) and a flexible, training-free approach utilizing NeRF optimization. Extensive evaluations across text-to-3D, image-to-3D, and text-to-texture tasks demonstrate that 3D-Adapter improves geometry quality and coherence over prior methods, providing a robust solution for multi-view 3D generation tasks.
weaknesses: The paper’s scope is broad, which obscures the clarity of its main contributions and makes it challenging to pinpoint specific innovations with reusable community value. Instead of covering multiple branches, e.g., GRM vs. NeRF optimization, a more focused examination on a single GRM pipeline could provide deeper insights, making the contribution more accessible and actionable for the community. Some comparisons and insights appear irrelevant, e.g. the text-to-avatar section, as the paper primarily addresses general 3D consistency issues rather than avatar-specific issues. This creates confusion regarding the paper’s core contributions.

Introducing per-step 3D → depth → ControlNet prediction comes with computational overhead. The paper could explore strategies to mitigate this, such as reducing the frequency of applying the adapter or skipping initial steps. Since per-step VAE decoding is probably the main bottleneck, adopting a lightweight VAE alternative, e.g. TinyAE, could yield substantial speed gains.

Some design choices around the ControlNet addition appear understudied. Introducing depth ControlNet itself may add a substantial 3D prior, but this effect is not adequately considered and is instead attributed mainly to ControlNet's role in denoising. Additionally, in the GRM branch, ControlNet is trained from scratch, which seems counterintuitive and is not supported with ablation studies.

The paper was fairly difficult to read and navigate, primarily due to its broad scope and, to a lesser extent, its writing style.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - The role of augmentation guidance appears to be significant. Could you provide visual evaluations in addition to Table 1 to better illustrate this effect?
- Please include a dedicated pipeline figure specifically for the process described in Section 4.2 (line 289) to enhance clarity.
- What are the results of using the optimization-based 3D-Adapter without additional conditioning?
soundness: 3
strengths: 1. The idea behind the 3D-Adapter is conceptually novel. It incorporates a lightweight 3D rendering pipeline within the diffusion sampling process through a mechanism that balances computational efficiency and multi-view consistency.


2. While there may be limitations in terms of generalizability, the 3D-Adapter has demonstrated significant versatility through its application across various tasks and base models, as shown through comprehensive experiments.
summary: The paper suggests a novel approach by introducing the 3D feedback augmentation adapter. It presents methods for applying this adapter to both feed-forward 3D generation and pretrained 2D models. The design incorporates a comprehensive 3D rendering process and a fine-tuning stage utilizing ControlNet to enhance the model's 3D awareness and multi-view consistency. Through experiments, the paper demonstrates the superior performance of the 3D-Adapter by applying it to various tasks and diverse base models, showcasing its effectiveness and versatility across different applications.
weaknesses: 1. Clarity of exposition: A more comprehensive explanation of the relationship between GRM and models such as Instant3D and Zero123++ would greatly enhance the reader's understanding. It is particularly important to elucidate whether GRM builds 3D representations from the views generated by Instant3D and Zero123++ and whether fine-tuning of GRM is involved in this process.

2, Limitations in the scalability of 3D-Adapter: 
- According to Equation 2, it appears that the 3D-Adapter could potentially be applied to other I/O sync-based pretrained models. However, it is necessary to discuss whether incorporating ControlNet into I/O sync pretrained models beyond GRM would yield similar results, and why Training Phase 1 is essential. If Training Phase 1 is a crucial step for optimizing the performance of 3D-Adapter, including GRM, it should be examined whether this phase is equally necessary when applied to other models.

- Despite utilizing various 3D representation techniques and loss functions, such as NeRF and DMTet, ensuring global semantic consistency remains challenging. This inherent limitation highlights the need for additional conditions to achieve robust 3D generation. Consequently, this raises concerns about the 3D consistency of 3D-Adapter in more general text-to-3D or image-to-3D tasks outside of specialized domains, such as avatar generation.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Please clarify the following:
  > However, 3D reconstruction and rendering are lossy operations that disrupt residual connections.

  Since residual connections shouldn’t be disrupted if 3D reconstruction is applied before or after the UNet denoising stage, I’m unclear why this is described as a problem. Could you explain?

2. (line 142)

> A common issue with two-stage approaches is that existing reconstruction methods, often designed for or trained under conditions of perfect consistency, lack robustness to local geometric inconsistencies. This may result in floaters and **blurry textures**.

Please provide evidence that two-stage approaches specifically suffer from blurry textures.

3. (line 198)

> assuming linearity

Could you clarify what you mean by "linearity" in this context?

4. Regarding concerns over some visuals: Could you provide a detailed explanation of the I/O-sync implementation in Tables 1 and 5? Table 5 shows a slight improvement when using I/O-sync in the "baseline" model, but in Table 1, there's a significant drop from A0 to A1. The visuals in column 5 appear broken and clearly worse than those in, for example, LGM, yet the CLIP score is significantly higher. Please double-check these evaluations.

5. Could you justify the use of ControlNet? How does it compare to using pure latent fusion?

6. 
> During the sampling process, the adapter performs NeRF optimization for the first 60% of the denoising steps. It then converts the color and density fields into a texture field and DMTet mesh, respectively, to complete the remaining 40% denoising steps. All optimizations are incremental, meaning the 3D state from the previous denoising step is retained to initialize the next. As a result, only 96 optimization steps are needed per denoising step. Alternatively, for texture generation only, multiview aggregation can be achieved by backprojecting the views into UV space and blending the results according to visibility

It's quite complex and specific training process, it would be great to hear about the reasoning/ablations behind these numbers.
soundness: 2
strengths: * Improved quality in downstream tasks over previous state-of-the-art
* Extensive final metrics in text-to-3d evaluation
summary: The authors propose 3D-adapter, a method to improve quality and 3d-consistency of existing text-to-multiview and text-to-image models by having a branch which reconstructs an object in 3D, then applies trained controlnet on rendered rgb and depth.
weaknesses: * The quality improvement in the main text-to-3D task is 1) marginal and 2) visuials are not convincing enough to justify complicating the training pipeline. For instance, in Fig. 1, image-to-3D visuals are comparable b/w 3D-Adapter and "two-stage pipeline"
* The paper lacks significant novelty, as the use of 3D representations for synchronization has been explored before (e.g. NerfDiff). The main contribution appears to be the placement of the adapter in the parallel branch rather than at the input/output stage of the diffusion UNet. However, this seems more like a technical choice, as is the decision to train ControlNet on intermediate outputs.
* Some visuals (e.g. Fig. 3, column 5) raise concerns about the correctness of the implementation.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: - The analysis in Appendix A helps to clarify the motivation of this work. On the other hand, the result is something that has been known in the community. Have any similar results been discussed previously in different domains? It would be convincing to add references to discuss consistency across different works in different applications.

- How much training data should be used to train GRM? As stated in the training details of Sec. 4.1, the authors fine-tuned 2000 iterations with 16 objects in a single batch, i.e., 32000 objects. Considering that a single object yields many views, it seems to be a lot for fine-tuning. What is the minimal training data to make the proposed method work? This question can also be rephrased as why the authors picked 2000 and 4000 iterations for the Instant3D and Zero123++ cases, respectively.

- Extending analysis: The authors showed why I/O sync may be bad, which is interesting. Then, another question naturally arises: why ControlNet-like feature addition (feedback augmentation) used in this work is effective? It is more interesting because the proposed method also does not guarantee a reduction of the gap in Eq. (7).

- Lines 286-288: The authors mentioned two ControlNets for superresolution and depth. Does it mean the authors use both ControlNet or is the depth ControlNet replaced with the super-resolution ControlNet? Then, how can we feed depth rendering to the superresolution ControlNet encoder without training?

- Line 287: Please provide the URL for the superresolution ControlNet as a footnote to give proper credit and to enhance the reproducibility. 

- Line 485 - "texture field optimization": What is texture field optimization referred to? Is the texture field optimization applicable to the other competing methods, TEXTure, Text2Tex, and SyncMVD, in Table 6? Was it applied to them already?
soundness: 4
strengths: - Clear contributions and positioning among existing research
- Clear demonstration of the limitation of I/O sync
- Resource-efficient design (training data efficient and training resource-efficient)
- Demonstration of various applications
- Noticeable improvements in terms of geometry and visual fidelity
- Clarity and presentation. The paper is well-organized and clearly presented
summary: This submission introduces 3D-Adapter, a novel module to address challenges in 3D geometry consistency synchronization in multi-view image diffusion models. 
This submission identifies the fundamental limitation of existing synchronization methods working on input and output domains. Then, the authors propose to add a synchronization mechanism to the intermediate feature level (like ControlNet) to encourage 3D consistency in existing multi-view image diffusion models, instead of input and output levels.
The key idea of the mechanism is that, at each denoising step, intermediate latent representations are decoded, 3D-reconstructed, and projected back to 2D representations (RGBD), where improved 3D consistency is re-encoded, called 3D feedback augmentation.

The study explores two variants of the 3D-Adapter: a feed-forward version using a Gaussian Splatting-based model (GRM) with fine-tuning and a training-free version utilizing neural fields and meshes depending on application scenarios. The extensive experiments across text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks demonstrate that 3D-Adapter improves generation quality in existing pre-trained models like Instant3D and Zero123++.


Overall, the submission is deemed to be well-prepared and positioned.

The proposed method itself would not be very innovative, but this reviewer found that the motivation and its theory behind the motivation are interesting. Although the researchers have been empirically aware of the limitation of the I/O sync method, the authors clearly presented the gap, which grounds the motivation of the proposed approach well.

However, some weaknesses remain that may improve the submission further.
weaknesses: - **Lack of information on computation and memory overhead for competing methods**

  Although the authors briefly address computaitonal overhead in "Text-to-Texture Generation" and "Text-to-Avatar Generation" applications, the authors do not explicitly compare overheads in "Text-to-3D" and "Image-to-3D" cases. While it is not neccessary to win every competing method, providing these comparisons would help readers better understand the position of this work in terms of computation demands.

- **Missing related work**

  The paper is well-position among the existing work, but some recent works are missing. Since this field is very competitive and timely, it would be beneficial to acknowledge recent developments as well so that readers gains a clearer understanding of the current landscape and to discern similarities and differences. 

  [C1] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion, CVPR 2024

  [C2] LRM: Large Reconstruction Model for Single Image to 3D, ICLR 2024

- **Lack of comparison with recent methods in Sec. 5.4 "Text-to-Texture Generation"**

  Texture generation researches have been developed to enhance multi-view consistency quite well. A few samples of the following works are not concurrent but the past researches. The authors missed these very relevant work and only compared with past researches. 

  [C3] DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models, SIGGRAPH 2024

  [C4] TexPainter: Generative Mesh Texturing with Multi-view Consistency, SIGGRAPH 2024

  [C5] Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering, CVPR2024

  [C6] Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models, CVPR2024

  In addition, for the Text-to-3D generation, [C1] released their code, but the authors did not compare with this closely related work.


- **Confusing notation of the proposed method in the tables**

  3D-Adaptor is not a standalone model. The notation "3D-Adaptor" could mislead readers to perceive it as a standalone model. The authors are recommended to change the notations of the proposed method across all the parts: e.g., "3D-Adaptor (ours)" => "Instant3D + 3D-Adaptor (ours)" to clarify the integration.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could the authors elaborate on how 3D-Adapter's 3D feedback augmentation differs fundamentally from existing I/O sync techniques? Specifically, in the claim made around L051, the paper categorizes methods such as Nerfdiff, DMV3D, and VideoMV under synchronizing the denoised outputs, suggesting that these approaches disrupt residual connections and result in poor texture quality. This assertion is somewhat confusing and not evidently supported. For example, DMV3D uses a 3D reconstruction model as a multi-view denoiser, and VideoMV employs a stage-wise re-sampling strategy to fuse 3D reconstruction information. It is unclear how these methods would interfere with the residual connections in the network design. The claim requires further justification or supporting evidence to be convincing.
2. Additionally, it is recommended that the authors consider showing intermediate results during the denoising process, such as outputs from the 3D reconstruction model at various stages. This could help readers better understand the contributions of the proposed method.
soundness: 3
strengths: 1. The 3D feedback augmentation approach and the architecture's integration with diffusion models present a novel contribution that extends beyond typical 2D-to-3D adaptation techniques.
2. The figures effectively demonstrate the qualitative improvements of the proposed approach, particularly in challenging cases.
3. The broad applicability of 3D-Adapter to text-to-3D, image-to-3D, and text-to-texture tasks indicates potential for further research and practical applications.
summary: The paper introduces a novel plug-in module, 3D-Adapter, which enhances multi-view diffusion models to improve the quality of 3D geometry generation. By integrating 3D feedback augmentation, it infuses 3D geometry awareness into pretrained image diffusion models. The 3D-Adapter operates in two main variants: a fast feed-forward version and a flexible training-free version using neural fields and meshes. The proposed method addresses the limitations of previous two-stage approaches by maintaining the original network topology and augmenting the base model through feature addition. Extensive experiments show improvements across various tasks, including text-to-3D, image-to-3D, text-to-texture, and text-to-avatar generation.
weaknesses: 1. **Claims About Existing Methods**: The assertion (L046-L053) that I/O sync methods degrade residual connections and cause texture quality issues is confusing and not adequately supported. More specific explanations with these methods are necessary.
2. **Inadequate Definitions**: The paper lacks clarity on the a) components of the 3D-Adapter (I understand that it includes VAE Decoder, ControlNet, and 3D Reconstruction Model) and b) the “I/O sync” baselines compared in Fig.1. Explicitly listing the modules included and the compared methods or baseline settings would aid understanding.
3. **Comparison Gaps**: The image-to-3D generation experiments do not include comparisons with CRM[1], SV3D[2], InstantMesh[3]. The text-to-texture experiments do not include metrics like FID, KID or comparisons with some established methods such as Paint3D[4] and FlashTex[5].
4. **Limitation Discussion**: Although the paper mentions the concerns about inference efficiency and shows the metric in the Appendix, more discussions about training efficiency would strengthen the paper, because it requires fine-tuning reconstruction model while keeping the multi-view diffusion model in memory.
5. **Insufficient References and Discussion of Prior Work**: a) The paper lacks citations for relevant multi-view generation methods such as Free3D[6], EpiDiff[7], and SPAD[8]. b) Additionally, it does not discuss highly relevant works like IM3D[9] and Carve3D[10]. c) While Ouroboros3D[11] and Cycle3D[12] may be considered concurrent work, it would still be valuable to include a discussion, as there are differences between these methods and the 3D-Adapter that merit discussion.

[1] CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model

[2] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

[3] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models

[4] Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models

[5] FlashTex: Fast Relightable Mesh Texturing with LightControlNet

[6] Free3D: Consistent Novel View Synthesis without 3D Representation

[7] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion

[8] SPAD : Spatially Aware Multiview Diffusers

[9] IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation

[10] Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning

[11] Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion

[12] Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle

Meta-review: The paper presents a method to incorporate depth prior into the framework of 3D generation. The paper receives mixed ratings from the reviewers. The reviewers have some concerns in terms of several perspectives. First, the reviewer argues that the novelty of the presented work is not elaborated clearly. The geometric consistency issue seems to be the key contribution that the paper targets, however, it is a bit unexpected that the depth prior is involved for 3D generation, and also, this step with diffusion will bring additional overhead as mentioned by one of the reviewers. Another issue criticized by one reviewer is that the proposed geometry consistency regularization seems to not improve the performance clearly, and also some visualizations raised potential concerns about implementation correctness. Finally, quite a few comments regarding the presentation quality were raised by the reviewers. Based on these critical comments, AC decided to reject this paper for this time.

Decision: Reject