Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion

## Abstract

Abstract Existing single image-to-3D creation methods typically involve a two-stage process, first generating multi-view images, and then using these images for 3D reconstruction. However, training these two stages separately leads to significant data bias in the inference phase, thus affecting the quality of reconstructed results.
We introduce a unified 3D generation framework, named Ouroboros3D, which integrates diffusion-based multi-view image generation and 3D reconstruction into a recursive diffusion process.
In our framework, these two modules are jointly trained through a self-conditioning mechanism, allowing them to adapt to each other’s characteristics for robust inference.
During the multi-view denoising process, the multi-view diffusion model uses the 3D-aware maps rendered by the reconstruction module at the previous timestep as additional conditions.
The recursive diffusion framework with 3D-aware feedback unites the entire process and improves geometric consistency.
Experiments show that our framework outperforms separation of these two stages and existing methods that combine them at the inference phase. Project page: https://costwen.github.io/Ouroboros3D/ .

## 1 Introduction

page\cref@result
3D content creation from a single image have improved rapidly in recent years with the adoption of large 3D datasets [, , ] and diffusion models [, , ]. A body of research [, , , , , , , ] has focused on multi-view diffusion models, fine-tuning pretrained image or video diffusion models on 3D datasets to enable consistent multi-view synthesis. These methods demonstrate generalizability and produce promising results. Another group of works [, , , , ] propose generalizable reconstruction models, generating 3D representation from one or few views in a feed-forward process, which has led to efficient image-to-3D creation.

Since single-view reconstruction models [] trained on 3D datasets [, ] lack generalizability and often produce blurring at unseen viewpoints, several works [, , , ] combine multi-view diffusion models and feed-forward reconstruction models, so as to extend the reconstruction stage to sparse-view input, boosting the reconstruction quality.
As shown in \creffig:concept-comparison, these methods split 3D generation into two stages: multi-view synthesis and 3D reconstruction. By combining generalizable multi-view diffusion models and robust sparse-view reconstruction models, such pipelines achieve high-quality image to 3D generation. However, combining the two independently designed models introduces a significant “data bias” to the reconstruction model. The data bias is mainly reflected in two aspects: (1) Multi-view bias. Multi-view diffusion models learn consistency at the image level, which makes it difficult to ensure geometric consistency. When it comes to reconstruction, multi-view images that lack geometric consistency affect the subsequent stage. (2) Limited data for reconstruction model. Unlike multi-view diffusion models, reconstruction models which are trained from scratch on limited 3D dataset, lacks the generalization ability.

Figure: Figure 1: Concept comparison between Ouroboros3D and previous two-stage methods. Instead of separating multi-view diffusion model and reconstruction model, our framework involves joint training and inference of these two models, which are established into a recursive diffusion process.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x1.png

Recent works like IM-3D [] and VideoMV [] have attempted to aggregate the rendered views of the reconstructed 3D model into multi-view synthesis by adopting re-sampling strategy in the denoising loop, thus improving the consistency of the generated multi-view images. These methods integrate the aforementioned two stages at the inference phase. But the models at both stages still lack joint training, which prevents the reconstruction model from enhancing its robustness to the generated multi-view images. Moreover, these test-time aggregating methods cannot directly utilize geometric information such as depth maps, normal maps, or coordinates maps [] that can also be obtained from the reconstructed 3D. Notably, these explicit 3D aware maps can better guide the multi-view generation.

In this paper, we propose a unified image-to-3D creation framework, Ouroboros3D, which integrates multi-view generation and 3D reconstruction into a recursive diffusion process (shown in \creffig:teaser). Our framework jointly trains the multi-view diffusion model and the reconstruction model through a self-conditioning mechanism, allowing them to adapt to each other’s characteristics for robust inference.
During multi-view denoising, the multi-view diffusion model uses the 3D-aware maps rendered by the reconstruction module at the previous timestep as additional conditions. By leveraging color maps and spatial canonical coordinates maps from the reconstructed 3D representation, our multi-view diffusion model generates images that better conform to the actual 3D structure. This recursive diffusion framework enhances generating complex scenes by uniting the 3D generation process with 3D-aware feedback.
In our experiments, we use the video diffusion model [] as the multi-view generator and the Large Multi-View Gaussian Model (LGM) [] as the reconstruction module. Tests on the GSO dataset [] show that our framework outperforms separation of these stages and existing methods [] that combine the stages at the inference phase.

Our key contributions are as follows:

- •
We introduce a image-to-3D creation framework Ouroboros3D, which integrates multi-view generation and 3D reconstruction into a recursive diffusion process.
- •
Ouroboros3D employs a self-conditioning mechanism with 3D-aware feedback, using rendered maps to guide the multi-view generation, ensuring better geometric consistency and robustness.
- •
Experiments show that our framework significantly reduces data bias, and outperforms separation of the two stages and existing methods [] that combine the stages at the inference stage.

Figure: Figure 2: Overview of 3D-aware recursive diffusion. During multi-view denoising, the diffusion model uses 3D-aware maps rendered by the reconstruction module at the previous step as conditions.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x2.png

## 2 Related Work

page\cref@result

#### Image/Video Diffusion for Multi-view Generation

Diffusion models [, , , , , , , , , ] have demonstrated their powerful generative capabilities in image and video generation fields. Current research [, , , , , , , , ] fine-tunes pretrained image/video diffusion models on 3D datasets like Objaverse [] and MVImageNet []. Zero123 [] introduces relative view condition to image diffusion models, enabling novel view synthesis from a single image and preserving generalizability. Based on it, methods like SyncDreamer [], ConsistNet [] and EpiDiff [] design attention modules to generate consistent multi-view images. These methods fine-tuned from image diffusion models produce generally promising results.
By considering multi-view images as consecutive frames of a video (e.g., orbiting camera views), it naturally leads to the idea of applying video generation models to 3D generation []. However, since the diffusion model is not explicitly modeled in 3D space, the generated multi-view images often struggle to achieve consistent and robust details.

#### Image to 3D Reconstruction

Recently, the task of reconstructing 3D objects has evolved from traditional multi-view reconstruction methods [, , , ] to feed-forward reconstruction models [, , , , , , ]. Ultilizing one or few shot as input, these highly generalizable reconstruction models synthesize 3D representation, enabling the rapid generation of 3D objects. LRM [] proposes a transformer-based model to effectively map image tokens to 3D triplanes. Instant3D [] further extends LRM to sparse-view input, significantly boosting the reconstruction quality. LGM [] and GRM [] replace the triplane representation with 3D Gaussians [] to enjoy its superior rendering efficiency. CRM [] and InstantMesh [] optimize on the mesh representation for high-quality geometry and texture modeling. These reconstrucion models built upon convolutional network architecture or transformer backbone, have led to efficient image-to-3D creation.

#### Pipelines of 3D Generation

Early works propose to distill knowledge of image prior to create 3D models via Score Distillation Sampling (SDS) [, , ], limited by the low speed of per-scene optimization. Several works [, , , ] fine-tune image diffusion models to generate multi-view images, which are then utilized for 3D shape and appearance recovery with traditional reconstruction methods [, ]. More recently, several works [, , , , ] involve both multi-view diffusion models and feed-forward reconstruction models in the generation process. Such pipelines attempt to combine the processes into a cohesive two-stage approach, thus achieving highly generalizable and high-quality single-image to 3D generation. However, due to the lack of explicit 3D modeling, the results generated by the multi-view diffusion model cannot guarantee strong consistency, which will lead to data deviation for the reconstructed model between the testing phase and the training phase. Compared to them, we propose a unified pipeline, integrating the two stages through a self-conditioning mechanism at the training stage, with 3D aware feedback for high consistency.

## 3 Method

Figure: Figure 3: Overview of Ouroboros3D. We adopt a video diffusion model as the multi-view generator by incorporating the input image and relative camera poses. In the denoising sampling loop, we decode the predicted $\mathbf{\widetilde{x}}_{0}^{f}$ to noise-corrupted images, which are then used to recover 3D representation by a feed-forward reconstruction model. Then the rendered color images and coordinates maps are encoded and fed into the next denoising step. At inference, the 3D-aware denoising sampling strategy iteratively refines the images by incorporating feedback from the reconstructed 3D into the denoising loop, enhancing multi-view consistency and image quality.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x3.png

Given a single image, Ouroboros3D aims to generate multiview-consistent images with a reconstructed 3D Gaussion model. To reduce the data bias and improve robustness of the generation, our framework integrates multi-view synthesis and 3D reconstruction in a recursive diffusion process. As illustrated in \crefframework, the proposed framework involves a video diffusion model (SVD []) as multi-view generator (refer to \crefvideodiffusion) and a feed-forward reconstruction model to recover a 3D Gaussian Splatting (refer to \creffeed-forward model. Moreover, we introduce a self-conditioning mechanism, feeding the 3D-aware information obtained from the reconstruction module back to the multi-view generation process (refer to \cref3dfeedback). The 3D-aware recursive diffusion strategy iteratively refines the multi-view images and the 3d model, thus enhancing the final production.

### 3.1 Video Diffusion Model as Multiview Generator

page\cref@result
Recent video diffusion models [, ] have demonstrated a remarkable capability to generate 3D-aware videos. We employs the well-known Stable Video Diffusion (SVD) Model as our multi-view generator, which generates videos from an image input. Further details about SVD can be found in \crefvideomodel. In our framework, we set the number of the generated frames $f$ to 8.

We enhance the video diffusion model with camera control $c$ to generate images from different viewpoints. Traditional methods encode camera positions at the frame level, which results in all pixels within one view sharing the same positional encoding [, ]. Building on the innovations of previous work [, ], we integrate the camera condition $c$ into the denoising network by parameterizing the rays $\mathbf{r}=(o,o\times d)$. Specifically, we use two-layered MLP to inject Plücker ray embeddings for each latent pixel, enabling precise positional encoding at the pixel level. This approach allows for more detailed and accurate 3D rendering, as pixel-specific embedding enhances the model’s ability to handle complex variations in depth and perspective across the video frames.

In our framework, unlike existing two-stage methods, our multi-view diffusion model does not complete all the denoising steps independently. In contrast, in the denoising sampling loop, we obtain the straightly predicted $\widetilde{\mathbf{x}}_{0}^{f}$ at each timestep, which will be used for subsequent 3D reconstruction, and rendered maps will be used as conditions to guide the next denoising step. Therefore, at each sampling step, we do the reparameterization of the output from the denoising network $F_{\theta}$ to convert it into $\widetilde{\mathbf{x}}_{0}^{f}$. Taking one view as an example, we use the following formula to process the denoised image $c_{\text{in}}(\sigma)\mathbf{x}$ and the associated noise level $c_{\text{noise}}(\sigma)$:

$$ $\cref@constructprefix{page}{\cref@result}\tilde{\mathbf{x}}_{0}=c_{\text{skip}}(\sigma)\mathbf{x}+c_{\text{out}}(\sigma)F_{\theta}(c_{\text{in}}(\sigma)\mathbf{x};c_{\text{noise}}(\sigma)).$ (1) $$

where $\sigma$ indicates the standard deviation of the noise. The above operation adjusts the output of $F_{\theta}$ to $\widetilde{\mathbf{x}}_{0}^{f}$, which will be decoded into images and passed to the subsequent 3D reconstruction module.

### 3.2 Feed-Forward Reconstruction Model

page\cref@result

In the Ouroboros3D framework, the feed-forward reconstruction model is designed to recover 3D models from pre-generated multi-view images, which can be images decoded from straightly predicted $\widetilde{\mathbf{x}}_{0}^{f}$, or completely denoised images. We utilize Large Multi-View Gaussian Model (LGM) [] $\mathcal{G}$ as our reconstruction module due to its real-time rendering capabilities that benefit from 3D representation of Gaussian Splatting. This method integrates seamlessly with our jointly training framework, allowing for quick adaptation and efficient processing.

We pass four specific views from the reparameterized output $\widetilde{\mathbf{x}}_{0}^{f}$ to the Large Gaussian Model (LGM) for 3D Gaussian Splatting reconstruction. To enhance the performance of LGM, particularly its sensitivity to different noise levels $c_{\text{noise}}(\sigma)$ and image details, we introduce a zero-initialized time embedding layer within the original U-Net structure of the LGM. This innovative modification enables the LGM to dynamically adapt to the diverse outputs that arise at different stages of the denoising process, thereby substantially improving its capacity to accurately reconstruct 3D content from images that have undergone partial denoising.

The loss function employed for the fine-tuning of the LGM is articulated as follows:

$$ $\mathcal{L}_{\mathcal{G}}=\mathcal{L}_{\text{rgb}}(\mathbf{x}_{0},\mathcal{G}(\tilde{\mathbf{x}}_{0},c_{\text{noise}}(\sigma)))+\lambda\mathcal{L}_{\text{LPIPS}}(\mathbf{x}_{0},\mathcal{G}(\tilde{\mathbf{x}}_{0},c_{\text{noise}}(\sigma))).$ (2) $$

where we have utilized the mean square error loss $\mathcal{L}_{\text{rgb}}$ for the color channel and a VGG-based perceptual loss $\mathcal{L}_{\text{LPIPS}}$ for the LPIPS term. In practical applications, the weighting factor $\lambda$ is conventionally set to 1.

Additionally, to maintain the model’s reconstruction capability for normal images, we also input the model without adding noise and calculate the corresponding loss. In this case, we set $c_{\text{noise}}(\sigma)$ to 0.

### 3.3 3D-Aware Feedback Mechanism

page\cref@result

As shown in \crefframework, we adopt a 3D-aware feedback mechanism that involves the rendered color images and geometric maps produced by our reconstruction module in a denoising loop to further improve the multi-view consistency of the resulting images and facilitate cyclic adaptation of the two stages. Instead of integrating multi-view generation and 3D reconstruction at the inference stage using re-sampling strategy [, ], we propose to train these two modules jointly to support more informative feedback. Specifically, in addition to the rendered color images, our flexible framework is able to derive additional geometric features to guide the generation process, which brings guidance of more explicit 3D information to multi-view generation.

In practice, we obtain color images and canonical coordinates maps (CCM) [] from the reconstructed 3D model, and utilize them as condition to guide the next denoising step of multi-view generation.
We use position maps instead of depth maps or normal maps as the representative of geometric maps because canonical coordinate maps record the global vertex coordinate values after normalization of the overall 3D model, rather than the normalization of the relative self-view (such as depth maps). This operation enables the rendered maps to be characterized as cross-view alignment, providing the strong guidance of more explicit cross-view geometry relationship. The details of canonical coordinates maps can be found in \crefcoordinates_map.

To encode color images and coordinates maps into the denoising network of multi-view generation module, we design two simple and lightweight encoders for color images and coordinates maps using a series of convolutional neural networks, like T2I-Adapter []. The encoders are composed of four feature extraction blocks and three downsample blocks to change the feature resolution, so that the dimension of the encoded features is the same as the intermediate feature in the encoder of U-Net denoiser. The extracted features from the two conditional modalities are then added to the U-Net encoder at each scale.

We propose a 3D-aware self-conditioning [] training and inference strategy. The original multi-view denoising network $F_{\theta}(\mathbf{x};\sigma)$ is augmented with 3D-aware feedback, formulated as $F_{\theta}(\mathbf{x};\sigma,\mathcal{G}(\tilde{\mathbf{x}}_{0}))$, where $\mathcal{G}(\tilde{\mathbf{x}}_{0})$ is the rendered maps of the reconstruction module.

#### Training Strategy

As illustrated in Algorithm [1](#alg1), to train a 3D-aware multi-view generation network, we use the rendered results from feed-forward model as the self-conditioning input. In practice, we randomly apply this mechanism with a probability of 0.5. When not using the 3D reconstruction result, we set $\mathcal{G}(\tilde{x}_{0})=0$ as the input. This probabilistic approach ensures balanced learning, allowing the model to effectively incorporate 3D information without over-reliance on it.

Figure: Algorithm 1 Training Ouroboros3D with the self-conditioned strategy.\cref@constructprefixpage\cref@result

#### Inference/Sampling Strategy

At the inference stage, as shown in Algorithm [2](#alg2), the 3D feedback $\mathcal{G}(\tilde{\mathbf{x}}_{0})$ is initially set to 0 0. At subsequent timesteps, this condition is updated with the previous reconstruction result $\mathcal{G}(\tilde{\mathbf{x}}_{0})$. This iterative process refines the 3D representation, ensuring each frame benefits from prior reconstructions, leading to more consistent multi-view images and higher quality reconstructed 3D models.

Figure: Algorithm 2 Sampling algorithm of Ouroboros3D.\cref@constructprefixpage\cref@result

## 4 Experiments

page\cref@result

### 4.1 Implementation Details

Datasets
We use a filtered subset of the Objaverse [] dataset to train our model, following LGM [] to remove bad models with bad captions or missing texture. It leads to a final set of around 80K 3D objects. We render 2 16-frame RGBA orbits at $512\times 512$. For each orbit, the cameras are positioned at a randomly sampled elevation between [-5, 30] degrees. During training, we subsample any 8-frame orbit by picking any frame in one orbit as the first frame (the conditioning image), and then choose every 2nd frame after that.

We evaluate the synthesized multi-view images and reconstructed 3D Gaussian Splatting (3DGS) on the unseen GSO [] dataset. We filter 100 objects to reduce redundancy and maintain diversity. For each object, we render ground truth orbit videos and pick the first frame
as the conditioning image.

Experimental Settings
Our Ouroboros3D is trained for 30,000 iterations using 8 A100 GPUs with a total batch size of 32. We clip the gradient with a maximum norm of 1.0. We use the AdamW optimizer with a learning rate of $1\times 10^{-5}$ and employ FP16 mixed precision with DeepSeed[] with Zero-2 for efficient training. At the inference stage, we set the number of sampling steps as 25, which takes about 20 seconds to generate a 3d model.

Metrics
We compare generated multi-view images and rendered views from reconstructed 3DGS with the ground truth frames, in terms of Learned Perceptual Similarity (LPIPS []), Peak Signal-to-Noise Ratio (PSNR), and Structural SIMilarity (SSIM).

Baselines
In terms of multi-view generation, we compare Ouroboros3D with SyncDreamer [], SV3D [], VideoMV []. For image-to-3D creation, we adopt feed-forward reconstruction models or pipelines as baseline methods, including TripoSR [], LGM [] and InstantMesh []. Notably, LGM and InstantMesh adopt two-stage methods to achieve image-to-3D creation, which generate multi-view images and then ultilize large reconstruction models to obtain 3D objects.

### 4.2 Comparison with Existing Alternatives

**Table 1: Quantitative comparison on the quality of generated multi-view images and 3D representation for image-to-multiview and image-to-3D tasks.\cref@constructprefixpage\cref@result**
|  | Method | Resolution | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Image-to-Multiview | SyncDreamer [] | $256\times 256$ | 20.056 | 0.8163 | 0.1596 |
| SV3D [] | $576\times 576$ | 21.042 | 0.8497 | 0.1296 |  |
| VideoMV [] | $256\times 256$ | 18.605 | 0.8410 | 0.1548 |  |
| Ouroboros3D (SVD) | $512\times 512$ | 21.770 | 0.8866 | 0.1093 |  |
| Image-to-3D | TripoSR [] | $256\times 256$ | 18.481 | 0.8506 | 0.1357 |
| LGM [] | $512\times 512$ | 17.716 | 0.8319 | 0.1894 |  |
| VideoMV(GS) [] | $256\times 256$ | 18.764 | 0.8449 | 0.1569 |  |
| InstantMesh (NeRF) [] | $512\times 512$ | 19.948 | 0.8727 | 0.1205 |  |
| Ouroboros3D (LGM) | $512\times 512$ | 21.761 | 0.8894 | 0.1091 |  |

Figure: Figure 4: Qualitative comparisons of generated multi-view images. Our method achieves better consistency and quality.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x4.png

Image-to-Multiview
We first compare against recent image-to-multiview generation methods [, , ]. The quantitative results are shown in \creftab:comp-metrics, and the qualitative results are shown in \creffig:quali-multi-view. The multi-view images produced by ours have better quality and consistency than those which train the multi-view generator independently or leverage reconstruction models to improve consistency at inference.

Figure: Figure 5: Qualitative comparisons for image-to-3D.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x5.png

Image-to-3D
We then compare with recent methods [, , , ] on image-to-3D tasks that are capable of generating 3D Gaussians or other 3D representation like triplane NeRF. \creffig:quali-3d shows images rendered from the generated 3D for comparison. We also present the quantitative results in \creftab:comp-metrics. We observe that our unified framework generates more realistic and high-quality 3D objects than existing methods which separate the two stages of multi-view generation and 3D reconstruction, or unites them at the inference stage. More results are shown in \creffig:more-result.

Generalizability
Our method is applicable to image inputs outside the distribution and can produce high-quality objects with the consistent 3D structure, as results are shown in \creffig:ood and \creffig:more-result.

### 4.3 Ablation Study

**Table 2: Ablation study of different feedback mechanisms. Results show that our 3D-aware feedback mechanism lead to superior generalization performance.\cref@constructprefixpage\cref@result**
| Jointly Training | Feedbacks | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | $\Delta$PSNR$\downarrow$ | $\Delta$SSIM$\downarrow$ | $\Delta$LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ✗ | ✗ | 20.012 | 0.8465 | 0.1287 | 1.067 | 0.0125 | 0.0189 |
| ✓ | ✗ | 20.549 | 0.8651 | 0.1221 | 0.511 | 0.0094 | 0.0070 |
| ✓ | CCM | 21.325 | 0.8738 | 0.1192 | 0.304 | 0.0036 | 0.0018 |
| ✓ | CCM + RGB | 21.761 | 0.8894 | 0.1091 | 0.009 | 0.0028 | 0.0002 |

Figure: Figure 6: Generalizability of our 3D generation. We can generate high-quality 3D models given image inputs outside the distribution.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x6.png

Figure: Figure 7: Qualitative comparison with no-feedback and 3d-aware feedback.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x7.png

To assess the effectiveness of our 3D-aware feedback mechanism, we conducted ablation experiments on the generated 3DGS for different configurations (\creffig:ablation and \creftab:ablation).
We start with a “base framework” that does not jointly trains the multi-view generation module and the reconstruction module, or use feedback mechanism.
We then incrementally add components of our proposed approach.
The full model (the last setting) means that we use both geometry and appearance information as conditions to guide the multi-view generation.
\creftab:ablation reports the ablation results, which demonstrate significant improvements by enhancing both geometric consistency and texture details.
We also report the absolute distances of performance metrics between the generated multiviews and 3DGS. It can be observed that our framework reduces the performance difference between the generated multi-view images and 3D representation, and improves the combined performance.

Figure: Figure 8: More visualization results of our image-to-3D creation.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x8.png

### 4.4 Limitations and Future Works

page\cref@result

Our model currently utilize 3D Gaussian splatting as the generated 3D representation, which is not as widely used in the gaming field as meshes. Replacing the reconstruction module with CRM [] or InstantMesh [] can enable our framework to generate meshes from a single image. In addition, experiments on 3D scene dataset will also be an extension of our framework.

## 5 Conclusion

In this paper, we introduce Ouroboros3D, a unified framework for single image-to-3D creation that integrates multi-view image generation and 3D reconstruction in a recursive diffusion process. We In our framework, these two modules are jointly trained through a self-conditioning mechanism, which allows them to adapt to the inherent characteristic of each stage, leading to more robust generation. By establishing a recursive relationship between these two stages through a self-conditioning mechanism, our approach effectively mitigates the data bias encountered in existing two-stage methods.
Experiments demonstrate that Ouroboros3D not only generates consistent and high-quality multi-view images, but also produces 3D objects with superior geometric consistency and details.

## Appendix A Technical Details

### A.1 Video Model Fine-tuning

page\cref@result
Based on the approach outlined in [], the generation process employs the EDM framework[]. Let $p_{\text{data}}(\mathbf{x}_{0})$ represent the video data distribution, and $p(\mathbf{x};\sigma)$ be the distribution obtained by adding Gaussian noise with variance $\sigma^{2}$ to the data. For sufficiently large $\sigma_{\text{max}}$, $p(x;\sigma_{\text{max}}^{2})$ approximates a normal distribution $\mathcal{N}(0,\sigma_{\text{max}}^{2})$. Diffusion models (DMs) leverage this property and begin with high variance Gaussian noise, $x_{M}\sim\mathcal{N}(0,\sigma_{\text{max}}^{2})$, and then iteratively denoise the data until reaching $\sigma_{0}=0$.

In practice, this iterative refinement process can be implemented through the numerical simulation of the Probability Flow ordinary differential equation (ODE):

$$ $d\mathbf{x}=-\dot{\sigma}(t)\sigma(t)\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma(t))\,dt$ (3) $$

where $\nabla_{\mathbf{x}}\log p((\mathbf{x};\sigma)$ is called as score function.

DM training is to learn a model $s_{\theta}(\mathbf{x};\sigma)$ to approximate the score function $\nabla_{\mathbf{x}}\log p((\mathbf{x};\sigma)$. The model can be parameterized as:

$$ $\nabla_{\mathbf{x}}\log p((\mathbf{x};\sigma)\approx s_{\theta}((\mathbf{x};\sigma)=\frac{D_{\theta}(\mathbf{x};\sigma)-\mathbf{x}}{\sigma^{2}},$ (4) $$

where $D_{\theta}$ is a learnable denoiser that aims to predict ground truth $\mathbf{x}_{0}$.

The denoiser $D_{\theta}$ is trained via denoising score matching (DSM):

$$ $\mathbb{E}_{\mathbf{x}_{0}\sim p_{\text{data}}(\mathbf{x}_{0}),(\sigma,n)\sim p(\sigma,n)}\left[\lambda_{\sigma}\|D_{\theta}(\mathbf{x}_{0}+n;\sigma)-\mathbf{x}_{0}\|_{2}^{2}\right],$ (5) $$

where $p(\sigma,n)=p(\sigma)\mathcal{N}(n;0,\sigma^{2})$, $p(\sigma)$ is a distribution over noise levels $\sigma$, $\lambda_{\sigma}$ is a weighting function. The learnable denoiser $D_{\theta}$ is parameterized as:

$$ $D_{\theta}(\mathbf{x};\sigma)=c_{\text{skip}}(\sigma)\mathbf{x}+c_{\text{out}}(\sigma)F_{\theta}(c_{\text{in}}(\sigma)\mathbf{x};c_{\text{noise}}(\sigma)),$ (6) $$

where $F_{\theta}$ is the network to be trained.

We sample $\log\sigma\sim\mathcal{N}(P_{\text{mean}},P_{\text{std}}^{2})$, with $P_{\text{mean}}=1.0$ and $P_{\text{std}}=1.6$. Then we obtain all the parameters as follows:

$$ $c_{\text{in}}=\frac{1}{\sqrt{\sigma^{2}+1}}$ (7) $$

$$ $c_{\text{out}}=\frac{-\sigma}{\sqrt{\sigma^{2}+1}}$ (8) $$

$$ $c_{\text{skip}}(\sigma)=\frac{1}{\sigma^{2}+1}$ (9) $$

$$ $c_{\text{noise}}(\sigma)=0.25\log\sigma$ (10) $$

$$ $\lambda(\sigma)=\frac{1+\sigma^{2}}{\sigma^{2}}$ (11) $$

We fine-tune the network backbone $F_{\theta}$ on multi-view images of size $512\times 512$. During training, for each instance in the dataset, we uniformly sample 8 views and choose the first view as the input view.
view images of size $512\times 512$.

### A.2 Canonical Coordinates Map

page\cref@result

Figure: Figure 9: The projection process of coordinates map.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/fig/coordindate_map.png

For control networks [, ] of image diffusion models, the conditional maps like depth maps need to be normalized to [0, 1], typically using the formula: $(p-p_{mean})/(p_{max}-p_{min})$. For multi-view generation, each view performs a normalize operation on itself, which results in a scale ambiguity. At the same time, the depth map is relative to a certain view, and the correlation between the depth values is not significant across views.

To avoid the above issues caused by self-normalization, we use canonical coordinate maps (CCM). Coordinate maps transform the depth value $d$ to a common world coordinate system using the camera’s intrinsic and extrinsic parameters, represented as $(X,Y,Z)$. The transformation formula is:

$$ $\begin{pmatrix}X\\ Y\\ Z\end{pmatrix}=K^{-1}\cdot\begin{pmatrix}u\\ v\\ 1\end{pmatrix}\cdot d$ $$

where $(u,v)$ are the pixel coordinates, $d$ is the corresponding depth value, and $K$ is the camera intrinsic matrix.
Then the coordinate values of all views will be multiplied by a global scale and added an offset value to convert to the range of 0 to 1. This representation makes the correlation between different views more significant and is helpful for multi-view generation.

### A.3 3D-aware Feedback

page\cref@result

Figure: Figure 10: Architecture of the residual block used in the feedback stage.\cref@constructprefixpage\cref@result
Refer to caption: /html/2406.03184/assets/x9.png

**Table 3: The detailed structure of all layers in the feedback injection network.\cref@constructprefixpage\cref@result**
| Input | $\text{inp}\in\mathbb{R}^{3\times 512\times 512}$ |
| --- | --- |
| PixelUnshuffle [] | $192\times 64\times 64$ |
| ResBlock $\times 3$ | $320\times 64\times 64$ |
| ResBlock $\times 3$ | $640\times 32\times 32$ |
| ResBlock $\times 3$ | $1280\times 16\times 16$ |
| ResBlock $\times 3$ | $1280\times 8\times 8$ |

With reference to \cref3dfeedback in the main paper, \crefresblock and \crefresnetwork provide a detailed illustration of the feedback injection netwrok. We use two networks to inject the coordinates map and RGB texture map feedback into the score function. Each network consists of four feature extraction blocks and three downsample blocks to adjust the feature resolution. The reconstruction coordinates map and RGB texture map initially have a resolution of $512\times 512$. We employ the pixel unshuffle operation to downsample these maps to $64\times 64$.

At each scale, three residual blocks[] are used to extract the multi-scale feedback features, denoted as $F_{P}=\{F^{1}_{p},F^{2}_{p},F^{3}_{p},F^{4}_{p}\}$ and $F_{T}=\{F^{1}_{t},F^{2}_{t},F^{3}_{t},F^{4}_{t}\}$ for the coordinates map and RGB texture map, respectively. These feedback features match the intermediate features $F_{\text{enc}}=\{F^{1}_{\text{enc}},F^{2}_{\text{enc}},F^{3}_{\text{enc}},F^{4}_{\text{enc}}\}$ in the encoder of the UNet denoiser. The feedback features $F_{P}$ and $F_{T}$ are added to the intermediate features $F_{\text{enc}}$ at each scale as described by the following equations:

$$ $\mathbf{F}_{p}=\mathcal{F}^{0}(P)$ (12) $$

$$ $\mathbf{F}_{t}=\mathcal{F}^{1}(T)$ (13) $$

$$ $\mathbf{F}^{i}_{\text{enc}}=\mathbf{F}^{i}_{\text{enc}}+\mathbf{F}_{p}^{i}+\mathbf{F}_{t}^{i},\quad i\in\{1,2,3,4\}$ (14) $$

where $P$ represents the coordinates map feedback input, and $T$ represents the RGB texture feedback input. $\mathcal{F}^{0}$ and $\mathcal{F}^{1}$ denote the functions of the feedback inject network applied to the coordinates map and RGB texture map, respectively.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please try to answer the questions listed in the weakness (especially point #1). 

I also have concerns about the generalizability of the recursive framework and how other techniques can be adapted into it

In fact, for a fair comparison, it would be beneficial to evaluate the results of the proposed framework equipped different multi-view synthesis models and large 3D generation models, such as those used as baselines.
soundness: 2
strengths: 1. The general motivation of doing joint training of the diffusion and reconstruction models - it might fill the domain gap between the two separately trained models (such as in LGM). 

2. The technical design of the RGB and CCM feedback - it keeps feeding the latest-reconstructed color and geometry info to each step of diffusion generation - thus is able to generate finer and more consistent images for later optimization.
summary: This paper proposes a pipeline that aims to better solve the single-view-to-3D generation task. The method sounds more like modifying based on the LGM framework, but instead tries to do a joint fine-tuning of LGM with its originally freezed off-the-shelf multi-view generation model. To build the connection between both models, a color encoder and a geometry encoder are also proposed and jointly trained, where they recursively pass color and geometry info from LGM to multi-view generation model.
weaknesses: 1. Given the proposed CCM and RGB feedback as major claimed contribution of where the improvements come from, I wonder why even without CCM and RGB feedback (table 2), the scores (PSNR, SSIM, LPIPS) are still outperform those in the baselines (table 1)? Given the PSNR “21.761” appears in both tables, I think they use the same test set.

Without CCM and RGB, the proposed method degrades to a regular one, then where do the improvements over the baselines come from? Please explain if any other proposed techniques contribute to these improvements.

2. Fig. 8 is not really helpful. The same thing can be made using the baseline results, where the geometry is gradually fixed from early to late steps. If want to present the proposed method fixing it faster and better, comparing a baseline result at the same steps could be useful in this figure.

3. The training and inference cost are both higher than the baselines, but still within an acceptable range.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors claim that the framework can accommodate various multi-view generation and reconstruction networks. Could they provide more examples to illustrate this flexibility in practice?

While the proposed framework is capable of deriving additional geometric features beyond the rendered color images,could the authors demonstrate the reliability and effectiveness of these features as guidance in enhancing model performance?
soundness: 2
strengths: The idea of incorporating feedback from the reconstructed 3D model into the denoising loop is well-founded, as seem to enhance multi-view consistency and improve overall image quality. 

The displayed results show improvements over prior work, quantitatively. In particular, the video demonstrating the recursive refinement process effectively illustrates the model's capability to incrementally enhance the reconstruction quality.

The paper is well-written, clearly structured, and easy to follow.
summary: This work addresses the challenge of multi-view bias and misaligned reconstruction data by proposing a unified framework that integrates multi-view generation and 3D reconstruction into a recursive diffusion process. The approach systematically aligns multi-view perspectives while refining 3D geometry, reducing inconsistencies across views. Experimental results demonstrate the framework's ability to achieve multi-view consistency, producing 3D models with enhanced visual realism. Video results are provided.
weaknesses: 1. The qualitative enhancements over LGM and CLAY are not apparent. A more detailed comparative analysis, including side-by-side visualizations, would help clarify the specific advancements introduced by the proposed method.

2. The reconstruction model's ability to reconstruct a 3D Gaussian Splatting from both slightly and highly noisy images is counterintuitive. An explanation of the model's robustness to varying noise levels would be beneficial.

3. The proposed approach utilizes 3D Gaussian Splatting as its 3D representation, which may be less suitable for practical applications compared to meshes.

**Reference**:  
[1] Zhang et al., "CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets," SIGGRAPH 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: * The paper proposes a novel framework for 3D generation by introducing 3D feedback through a process that denoise->reconstruct->render->condition->denoise… . The performance looks good. 
* The idea of the paper that involves 3D feedback mechanism into multi-view generation is reasonable
* The paper is well-written and easy to follow.
summary: This paper presents a novel method by incorporating a 3D-aware feedback mechanism into the multi-view generation process. In comparison with existing methods, this paper proposes a joint training strategy to alleviate the reconstruction bias due to data misalignment between training and inference stages. Experimental results show this methods enhance multi-view consistency and outperform current methods.
weaknesses: * The novelty of this paper is limited. The overall pipeline is very similar to existing works such as VideoMV and SyncDreamer. Integrating a large gaussian reconstruction model into the denoising process of video diffusion model has been proposed by VideoMV and the 3D feedback condition has been proposed by SyncDreamer, except that SyncDreamer adopts depth map while this method adopts RGB and CCM. However, these are very little new contribution for the fast-developing 3D generation research community.
* The evaluation is not thorough. The author highlighted in the paper that joint training alleviates reconstruction bias between training and inference stage. However, the experiments about joint training is not sufficient. There are only four ablation experiments in the paper: 
a) wo joint training and 3D reconstruction, b) 3D reconstruction w RGB,  c) 3D reconstruction CCM, d) full model (with joint training and 3D reconstruction). In such a setting, we could not know the influence of joint training as we do not have a pair experiments of w/wo joint training. 
* As declared by the author, the proposed method also leads to significantly efficiency drop in the training stage.
* Limitation is not discussed in the paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My main doubts lie in the incremental technical novelty, specifically, I am not quite convinced from Figure 1 and accompanying text in the paper that the paper makes such substantial contribution in using 3D (geometry) feedback to guide multiview diffusion. Apparently prior works, at least appeared in 2023/24 use depth or other 2.5/3D proxies, to guide the diffusion process, and/or vice versa, because this is such a straightforward idea even dating back early 2000s in the era of image-based modeling and rendering.  

To give the authors some ideas, during ECCV 2024 (notwithstanding, concurrent works with ICLR 2025) there are maybe 15 papers dealing with this problem with somewhat similar pipeline where multiview images and (some form of) 3D geometry work in tandem to improve their intermediate results and thus final reconstruction, iteratively or recursively. The burden lies on the authors but not the reviewer in delineating the technical differences and novelties to produce a convincing case why this paper as it is written, is ICLR worthy, which is unclear in the main paper, although the implementation steps are clearly described (otherwise I would have scored the paper lower than 5 overall). Authors related work section is some good start, but some passing statements on STOA may not be adequate or convincing. 

Otherwise, to qualify for simple and worthwhile ICLR ideas, I need to see substantial improvement on the results over STOA to speak for themselves, which I am not seeing as explained above.
soundness: 3
strengths: 1. The paper is very readable.

2. STOA results with incremental improvements over recent prior work are presented.
summary: Paper presents (yet another) approach to image-to-3D generation, or generating multi-views with 3D consistency, tapping into the recent development in diffusion models and in particular, stable video diffusion is employed. Authors novel claim consists of jointly training of multiview generation and 3D reconstruction, guided by their 3D-aware feedback or self-conditioning, more consistent multi-views and 3D reconstruction can be achieved than existing approaches where joint training is absent or inadequate. Quantitative and qualitative comparison with SyncDreamer, SV3D, VideeoMV, TripSR, LGM (3DGS) and InstantMesh (NeRF) are conducted. Running times and ablation studies are provided.
weaknesses: Following the emergence of powerful diffusion models in generating or hallucinating highly realistic images 1-2 years ago, many researchers have observed that while the generated images look plausible, they lack 3D consistency and thus we have seen various approaches in recent computer vision and machine learning venues on optimizing image generation and 3D geometric consistency in tandem.                                                                                                                                                                                                                

The work may thus be categorized into this and relevant line of research: simply google "diffusion model image generation mulitiview consistency 3D reconstruction"... The main difference claimed by the authors is summarized in Figure 1 where the feedback is 3D aware, rather than just RGB (losses).

Given this background:

1. The main proposed idea of 3D-aware feedback to guide multiview diffusion, as stated in my summary, is not very novel.

2. Related to the above, the 3D-aware feedback (section 3.3) consists of the RGB images and, unlike others using depth maps, the main new geometric/3D "proxy" in the joint optimization or training proposed are "canonical coordinates maps (CCM) (Li et al., 2023b) from the reconstructed 3D model as the condition to guide the next denoising step of multi-view generation."  Despite the advantages of CCM over depth/normal maps mentioned by the authors, this technicality in my opinion is hardly justified as ICLR contribution, unless we are seeing substantial improvement of the results, which brings us to:

3. While achieving STOA results which arguably contain 3D-consistent fine details such as thin structures, as well as decal and textures on surfaces, the class of showcase results still belongs to that of SynDreamer and the present contenders compared in the paper. In particular,  some flaws still exist in their 360$^\circ$ demos, especially when the reconstruction is viewed from the side which appears to be too flat when the input view is frontal (e.g., check out the stuff toy holding a pink heart, the statuette with two large pony tails, and the cute dinosaur, for instance, in the supplemental material), suffering something reminiscent of single-view reconstruction bias which can be remotely related bas-relief ambiguity when the reconstructed model can only look plausible at frontal views.