SC3D: Self-conditioned Generative Gaussian Model with 3D-aware Feedback

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 13:22:00

Abstract: Existing single image-to-3D creation methods typically involve a two-stage process, first generating multi-view images, and then using these images for 3D reconstruction. However, training these two stages separately leads to significant data bias in the inference phase, thus affecting the quality of reconstructed results.We introduce a unified 3D generation framework, named SC3D, which integrates diffusion-based multi-view image generation and 3D reconstruction through a self-conditioning mechanism. In our framework, these two modules are established as a cyclic relationship so that they adapt to the distribution of each other.During the denoising process of multi-view generation, we feed rendered color images and maps by SC3D itself to the multi-view generation module.
This self-conditioned method with 3D aware feedback unites the entire process and improves geometric consistency.Experiments show that our approach enhances sampling quality, and improves the efficiency and output quality of the generation process.

Main paper: 
 introduction: 
 3D content creation from a single image have improved rapidly in recent years with the adoption of large 3D datasets [1, 2, 3] and diffusion models [4, 5, 6] . A body of research [7, 8, 9, 10, 11, 12, 13, 14] has focused on multi-view diffusion models, fine-tuning pretrained image or video diffusion models on 3D datasets to enable consistent multi-view synthesis. These methods demonstrate generalizability and produce promising results. Another group of works [15, 16, 17, 18, 19] propose generalizable reconstruction models, generating 3D representation from one or few views in a feed-forward process.Theses reconstruction models built upon convolutional network or transformer backbone, have led to efficient image-to-3D creation.Since single-view reconstruction models [15] trained on 3D datasets [1, 20] lack generalizability and often produce blurring at unseen viewpoints, several works [21, 16, 18, 19] extend models to sparse-view input, boosting the reconstruction quality. As shown in Fig. 1 , these methods split 3D generation into two stages: multi-view synthesis and 3D reconstruction. By combining generalizable multi-view diffusion models and robust sparse-view reconstruction models, such pipelines achieve high-quality image to 3D generation. However, combining the two independently designed models introduces a significant "data bias" to the reconstruction model. The data bias is mainly reflected in two aspects: (1) Multi-view bias. Multi-view diffusion models learn consistency at the image level, struggle to ensure geometric consistency. When it comes to reconstruction, multi-view images that lack geometric consistency affect the subsequent stage. (2) Limited data for reconstruction model.Unlike multi-view diffusion models, reconstruction models which are trained from scratch on limited 3D dataset, lacks the generalization ability.Recent works like IM-3D [22] and VideoMV [23] and consistency of the generated multi-view images. These methods integrate the aforementioned two stages at the inference phase. But the models at both stages still lack joint training, which prevents the reconstruction model from enhancing its robustness to the generated poor multiviews. Moreover, these test-time aggregating methods cannot directly utilize geometric information such as depth maps, normal maps, or position maps that can also be obtained from the reconstructed 3D. Notably, these explicit 3D aware maps can better guide the multi-view generation.To address these challenges, we propose a unified single image-to-3D creation framework, named SC3D, which integrates multi-view generation and 3D reconstruction through a self-conditioning mechanism. Our framework involves jointly training the multi-view diffusion model and the reconstruction model. In SC3D, these two modules are established as a cyclic relationship so that they adapt to the characteristics of each other, enabling robust generation at inference. Specifically, during the denoising process, we feed rendered 3D-aware maps from the reconstructed 3D to the multi-view generation module. By leveraging the color maps and spatial canonical coordinates maps from the reconstruction 3D representation as condition, our multi-view diffusion model synthesizes multi-view images that better conform to the actual 3D structure. This self-conditioned framework with 3D aware feedback unites the 3D generation process and enhances the robustness for unseen complex scenes. Experiments on the GSO dataset [24] validate that our SC3D reduces data bias between training and inference, and enhances the overall efficiency and output quality.Our key contributions are as follows:• We introduce SC3D, which unifies multi-view generation and 3D reconstruction in a single framework and involves jointly training these two modules, enabling adaption to each other.• SC3D employs a self-conditioning mechanism with 3D-aware feedback, using rendered 3D-aware maps to guide the multi-view generation, ensuring better geometric consistency and robustness.• Experiments show that SC3D significantly reduces data bias, improves the quality of 3D reconstruction, and enhances overall efficiency in creating 3D content from a single image. 
related work: 
 Image/Video Diffusion for Multi-view Generation Diffusion models [25, 26, 27, 28, 29, 30, 31, 32, 33, 34] have demonstrated their powerful generative capabilities in image and video generation fields. Current research [7, 8, 9, 10, 11, 12, 13, 14, 35] fine-tunes pretrained image/video diffusion models on 3D datasets like Objaverse [1] and MVImageNet [20] . Zero123 [7] introduces relative view condition to image diffusion models, enabling novel view synthesis from a single image and preserving generalizability. Based on it, methods like SyncDreamer [9] , ConsistNet [36] and EpiDiff [11] design attention modules to generate consistent multi-view images. These methods fine-tuned from image diffusion models produce generally promising results. By considering multi-view images as consecutive frames of a video (e.g., orbiting camera views), it naturally leads to the idea of applying video generation models to 3D generation [13] . However, since the diffusion model is not explicitly modeled in 3D space, the generated multi-view images often struggle to achieve consistent and robust details.Image to 3D Reconstruction Recently, the task of reconstructing 3D objects has evolved from traditional multi-view reconstruction methods [37, 38, 39, 40] to feed-forward reconstruction models [15, 41, 42, 16, 17, 18, 19] . Ultilizing one or few shot as input, these highly generalizable reconstruction models synthesize 3D representation, enabling the rapid generation of 3D objects.LRM [15] proposes a transformer-based model to effectively map image tokens to 3D triplanes.Instant3D [21] further extends LRM to sparse-view input, significantly boosting the reconstruction quality.LGM [16] and GRM [17] replace the triplane representation with 3D Gaussians [40] to enjoy its superior rendering efficiency. CRM [18] and InstantMesh [19] optimize on the mesh representation for high-quality geometry and texture modeling. These reconstrucion models built upon convolutional network architecture or transformer backbone, have led to efficient image-to-3D creation.Early works propose to distill knowledge of image prior to create 3D models via Score Distillation Sampling (SDS) [43, 44, 45] , limited by the low speed of per-scene optimization. Several works [9, 11, 14, 22] fine-tune image diffusion models to generate multi-view images, which are then utilized for 3D shape and appearance recovery with traditional reconstruction methods [46, 40] . More recently, several works [21, 16, 18, 19, 23] involve both multi-view diffusion models and feed-forward reconstruction models in the generation process. Such pipelines attempt to combine the processes into a cohesive two-stage approach, thus achieving highly generalizable and high-quality single-image to 3D generation. However, due to the lack of explicit 3D modeling, the results generated by the multi-view diffusion model cannot guarantee strong consistency, which will lead to data deviation for the reconstructed model between the testing phase and the training phase. Compared to them, we propose a unified pipeline, integrating the two stages through a self-conditioning mechanism at the training stage, with 3D aware feedback for high consistency.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See in Section 4.3. 
method: 
 Given a single image, SC3D aims to generate multiview-consistent images with a reconstructed 3D Gaussion model. To reduce the data bias and improve robustness of the generation, we propose SC3D, a unified 3D generation framework which integrates multi-view synthesis and 3D reconstruction through a self-conditioning mechanism. As illustrated in Fig. 2 , the proposed framework involves a video diffusion model (SVD [32] ) as multi-view generator (refer to Section 3.1) and a feed-forward reconstruction model to recover a 3D Gaussian Splatting (refer to Section 3.2. Moreover, we introduce a self-conditioning mechanism, feeding the 3D-aware information obtained from the reconstruction module back to the multi-view generation process (refer to Section 3.3). The 3D-aware denoising sampling strategy iteratively refines the multi-view images and the 3d model, thus enhancing the final production.Recent video diffusion models such as those in [13, 34] have demonstrated a remarkable capability to generate 3D-aware videos by scaling up both the model and dataset. Our research employs the well-known Stable Video Diffusion (SVD) Model, which generates videos from image input.Formally, given an image I ∈ R 3×h×w , the model is designed to generate a video V ∈ R f ×3×h×w .Further details about SVD can be found in Appendix A.1.We enhance the video diffusion model with camera control c to generate images from different viewpoints. Traditional methods encode camera positions at the frame level, which results in all pixels within one view sharing the same positional encoding [47, 13] . Building on the innovations of previous work [11, 35] , we integrate the camera condition c into the denoising network by parameterizing the rays r = (o, o × d). Specifically, we use two-layered MLP to inject Plücker ray embeddings for each latent pixel, enabling precise positional encoding at the pixel level. This approach allows for more detailed and accurate 3D rendering, as pixel-specific embedding enhances the model's ability to handle complex variations in depth and perspective across the video frames. In our framework, unlike existing two-stage methods, our multi-view diffusion model does not complete multiple denoising steps independently. In contrast, in the denoising sampling loop, we obtain the straightly predicted x f 0 at the current timestep, which will be used for subsequent 3Dreconstruction. Then we use rendered 3d-aware view maps as conditions to guide the next denoising step. Therefore, at each sampling step, we do the reparameterization of the output from the denoising network F θ to convert it into x f 0 . Taking a single view as an example, we processes the denoised image c in (σ)x and the associated noise level c noise (σ), which σ indicates the standard deviation of the noise. The reparameterization is formulated as follows:EQUATIONThe above operation process adjusts the output of F θ to x f 0 , which will be decoded into images and passed to the subsequent 3D reconstruction module.In the SC3D framework, the feed-forward reconstruction model is designed to recover 3D models from pre-generated multi-view images, which can be images decoded from straightly predicted x f 0 , or completely denoised images. We utilize Large Multi-View Gaussian Model (LGM) [16] G as our reconstruction module due to its real-time rendering capabilities that benefit from 3D representation of Gaussian Splatting. This method integrates seamlessly with our jointly training framework, allowing for quick adaptation and efficient processing.We pass four specific views from the reparameterized output x f 0 to the Large Gaussian Model (LGM) for 3D Gaussian Splatting reconstruction. To enhance the performance of LGM, particularly its sensitivity to different noise levels c noise (σ) and image details, we introduce a zero-initialized time embedding layer within the original U-Net structure of the LGM. This innovative modification enables the LGM to dynamically adapt to the diverse outputs that arise at different stages of the denoising process, thereby substantially improving its capacity to accurately reconstruct 3D content from images that have undergone partial denoising.The loss function employed for the fine-tuning of the LGM is articulated as follows:EQUATIONwhere we have utilized the mean square error loss L rgb for the color channel and a VGG-based perceptual loss L LPIPS [43] for the LPIPS term. In practical applications, the weighting factor λ is conventionally set to 1.Additionally, to maintain the model's reconstruction capability for normal images, we also input the model without adding noise and calculate the corresponding loss. In this case, we set c noise (σ) to 0.As shown in Fig. 2 , we adopt a 3D-aware feedback mechanism that involves the rendered color images and geometric maps produced by our reconstruction module in a denoising loop to further improve the multi-view consistency of the resulting images and facilitate cyclic adaptation of the two stages. Instead of integrating multi-view generation and 3D reconstruction at the inference stage using re-sampling strategy [22, 23] , we propose to train these two modules jointly to support more informative feedback. Specifically, in addition to the rendered color images, our flexible framework is able to derive additional geometric features to guide the generation process, which brings guidance of more explicit 3D information to multi-view generation.In practice, we obtain color images and canonical coordinates maps [48] from the reconstructed 3D model, and utilize them as condition to guide the next denoising step of multi-view generation. We use position maps instead of depth maps or normal maps as the representative of geometric maps because canonical coordinate maps record the vertex coordinate values after normalization of the overall 3D model, rather than the normalization of the relative self-view (such as depth maps). This operation enables the rendered maps to be characterized as cross-view alignment, providing the strong guidance of more explicit cross-view geometry relationship. The details of canonical coordinates map can be found in Appendix A.2.We adopt a 3D-aware self-conditioning [49] training and inference strategy that leverages reconstruction stage results to enhance multi-view consistency and the quality of generated images. During training, the original denoising network F θ (x; σ) is augmented with a 3D-aware feedback denoising network F θ (G(x 0 ); σ), where G(x 0 ) is the output of the LGM reconstruction.To encode color images and coordinates maps into the denoising network of multi-view generation module, we design two simple and lightweight encoders for color images and coordinates maps using a series of convolutional neural networks, like T2I-Adapter [50] . Training Strategy As illustrated in Algorithm 1, to train a 3D-aware multi-view generation network, we use the rendered maps by the 3D reconstruction module as the self-conditioning input. In practice, we randomly use this self-conditioning mechanism with a probability of 0.5. When not using the 3D reconstruction result, we set G(x 0 ) = 0 as the input. This probabilistic approach ensures balanced learning, allowing the model to effectively incorporate 3D information without over-reliance on it. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the code in the supplemental materials. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We focus on 3D asset content synthesis, training our model on the G-Objaverse [1, 51] dataset and the LVIS subset of Objaverse, which consists of 300K high-quality 3D objects and is widely used in 3D generation. We evaluate SC3D on the Google Scanned Object (GSO) dataset [24] , which consists of approximately 1,000 scanned models, and we randomly select 100 samples for comparison. We adopt TripoSR [42] , SyncDreamer [9] , SV3D [13] , ImageDream [8] combined with LGM [16] as the baseline approach [16] and VideoMV [23] as baseline methods. For each baseline, we report PSNR, SSIM, and LPIPS metrics.For LGM, we utilize the official LGM single-image generation pipeline, which employs ImageDream [52] to transition from a single image input to multiple images. However, the conical coordinate system employed by ImageDream complicates the direct evaluation of the output. To address this, we use the official code to test on the GSO dataset, followed by manual calibration to assess the generated quality, as illustrated in Fig. 3 . The misalignment between the two stages of ImageDream and LGM often results in generated models with blurred linear edges and geometric ambiguities. Rendered multi-views from Generated 3DGS To validate the effectiveness of the proposed SC-3D framework, we conducted a series of ablation studies comparing PSNR, SSIM, and LPIPS metrics for different configurations ( We also demonstrate the impact of incorporating feedback mechanisms on the two models, as shown in Table 3 . It can be observed that when no feedback mechanism is used, there is a significant discrepancy between the two models' modalities, which leads to a degradation in their combined performance.Delta PSNR Delta SSIM Delta LPIPS SVD 0.511 0.0094 0.0070 SVD + Coordinates Map Feedback 0.304 0.0036 0.0018 SVD + 3D-aware Feedback 0.009 0.0028 0.0002 Table 3 : The absolute differences in performance metrics between GS and SVD generation results..Current models utilize Gaussian splatting as a 3D representation, mapping and rendering coordinates to textures for feedback. Although algorithms for converting Gaussian Splatting to meshe are under development, achieving high quality in converting Gaussian models to general meshes remains challenging. Directly employing a NeRF-based feed-forward model during the training process significantly reduces training speed due to the computational demands of volumetric rendering. Our model currently lacks the ability to generalize to the scene level, a limitation we intend to address in future research. 
conclusion: 
 In this paper, we introduce SC3D, a unified framework for 3D generation from a single image that integrates multi-view image generation and 3D reconstruction through a self-conditioning mechanism.By establishing a cyclic relationship between these two stages, our approach effectively mitigates the data bias encountered in traditional methods. The self-conditioned method with 3D-aware feedback enhances geometric consistency throughout the generation process.Our experiments demonstrate that SC3D not only improves the quality and efficiency of the generation process but also achieves superior geometric consistency and detail in the reconstructed 3D models.By jointly training the multi-view diffusion model and the reconstruction model, SC3D adapts to the inherent biases of each stage, resulting in more robust and accurate outputs.σ 0 = 0.In practice, this iterative refinement process can be implemented through the numerical simulation of the Probability Flow ordinary differential equation (ODE):EQUATIONwhere ∇ x log p((x; σ) is called as score function.DM training is to learn a model s θ (x; σ) to approximate the score function ∇ x log p((x; σ). The model can be parameterized as:EQUATION)where D θ is a learnable denoiser that aims to predict ground truth x 0 .The denoiser D θ is trained via denoising score matching (DSM):EQUATIONwhere p(σ, n) = p(σ)N (n; 0, σ 2 ), p(σ) is a distribution over noise levels σ, λ σ is a weighting function. The learnable denoiser D θ is parameterized as:EQUATIONwhere F θ is the network to be trained.We sample log σ ∼ N (P mean , P 2 std ), with P mean = 1.0 and P std = 1.6. Then we obtain all the parameters as follows:EQUATIONEQUATIONc noise (σ) = 0.25 log σ (10)EQUATIONWe fine-tune the network backbone F θ on multi-view images of size 512 × 512. During training, for each instance in the dataset, we uniformly sample 8 views and choose the first view as the input view.view images of size 512 × 512.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See in Appendix B. 
experiment statistical significance: 
 • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the Section 1, we discuss how 3D generation can accelerate various industries by enhancing design processes, improving simulations, and reducing production costs. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper does not involve the release of data or models that have a high risk for misuse.Guidelines: The paper focuses on foundational research and does not have direct societal implications. It does not address societal impacts.• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In conditional control models such as ControlNet [54] , T2IAdapter, when depth maps are used as input, their range needs to be normalized to the [0, 1] interval, typically using the formula: (pp mean )/(p max -p min ). However, this normalization process may introduce scale ambiguity, which can affect the multi-view generation performance. To avoid the issues caused by normalization, we use coordinate maps. Coordinate maps transform the depth value d to a common world coordinate system using the camera's intrinsic and extrinsic parameters, represented as (X, Y, Z). The transformation formula is:X Y Z = K -1 • u v 1 • dwhere (u, v) are the pixel coordinates, d is the corresponding depth value, and K is the camera intrinsic matrix. At each scale, three residual blocks [56] are used to extract the multi-scale feedback features, denoted asF P = {F 1 p , F 2 p , F 3 p , F 4 p } and F T = {F 1 t , F 2 t , F 3 t , F 4 t }for the coordinates map and RGB texture map, respectively. These feedback features match the intermediate features F enc = {F 1 enc , F 2 enc , FEQUATIONEQUATIONEQUATIONwhere P represents the coordinates map feedback input, and T represents the RGB texture feedback input. F 0 and F 1 denote the functions of the feedback inject network applied to the coordinates map and RGB texture map, respectively.Implementation As illustrate in Table 5 , all models are trained for 30,000 iterations using 8 A100GPUs with a total batch size of 32. We clip the gradient with a maximum norm of 1.0. We use the AdamW optimizer with a learning rate of 1 × 10 -5 and employ FP16 mixed precision with DeepSeed [57] with Zero-2 for efficient training. We adjust the cameras in each batch so that the initial input view consistently represents the reference frame, using an identity rotation matrix and a fixed translation for alignment.The inference settings are shown in Table 6 .Hyperparameter Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper received four reviews from expert reviewers. The rebuttal and discussion have cleared up many concerns about structure, content, clarity, comparisons, and visual results. This has led to a slight increase in the overall rating of the paper.
However, the promised changes needed to bring the paper to the level of NeurIPS go beyond what can be accepted without another review. The AC find the additional results, clarifications, and comparisons essential, as well as the promised improvements in writing, structure, typos, and other fixes and encourages the authors to resubmit a revised version to the next suitable venue.

Reviews: review: paper_summary: paper_summary: This paper proposes SC3D for the single-image-to-3D generation, which integrates the diffusion-based multi-view generation and Gaussians-based 3D reconstruction through a self-conditioning mechanism. Specifically, during each denoising step, SC3D injects the rendered image and geometric map from the reconstruction model into the denoising process to enhance the multi-view consistency of the multi-view generated images. Experiments on GSO dataset demonstrates its superiority over existing methods mentioned in this paper.
strength_weakness: strength_weakness: 1. Lack detailed visual comparisons with baseline methods. The authors only compare SC3D with LGM but do not show results generated from other baselines, making the visual comparison results less convincing.

2. The paper suffers from poor organization. For example, Figure 4 and Figure 5 are not referenced anywhere in the text. The purpose of Figure 6 is confusing, as its caption suggests it shows results from another work, and it is difficult to discern differences among the three rows. Additionally, the paper's typesetting is of poor quality. There are many blank spaces in the text.
questions: questions: 1. Will jointly training the multi-view generation network and 3D reconstruction network make the training unstable? Will the jointly training mechanism increase the training time? More details about potential disadvantages of jointly training should be discussed in the paper.

2. As mentioned in Line 211-218, the setting of two ablated experiments seems same, what is the difference between them?
limitations: limitations: Please refer the weaknesses and questions above.
paper_summary: paper_summary: The paper observes that the current state-of-the-art image-to-3D generation models consist of two separate parts: generate multi-view images from a single image and run on top the 3D reconstruction. This process has no feedback loop, i.e. the reconstruction does not inform the image generation which in turn leads to a worse quality of reconstruction. They propose a method that builds in a feedback to loop back the feedback of the reconstruction into the diffusion process. They report superior 3D reconstruction quality over the usual two separate step method.
strength_weakness: strength_weakness: 1. The presentation of the paper is not clear. 
    - In Fig. 1 the paper is describing an iterative process. Fig. 1 has also no output, but suggests the 3D representation is the output. However, Fig. 2 suggests the (multi-view) images are the final output? Are the two decoders the same? In the paper you are referring to different models G and F. They are not mapped to the figure to get a better picture.
    - Lines 169-172: this is describing the training strategy. That should be moved to the part starting from line 180 where you are actually describing the training strategy.
    - Equation 1: c_skip is not explained
    - The paper has many typos. Especially in part 4, they appear in almost every paragraph.
    - Lines 227-228: This statement seems contradicting: “Directly employing a NeRF-based feed-forward model during the training process significantly reduces training speed due to the computational demands of volumetric rendering.”
    - Replacing the algorithm code with more concise pseudo code may make it much easier for more readers to understand.
2. Comparisons are not very comprehensive
    - None of the methods in Figure 1 are qualitatively compared. 
    - In Figure 3, it seems different views are compared in the first and second column
    - Minor: It would be also really helpful to introduce some visual cues into figure 9 to easier grasp the results.
3. Some claims are not justified
    - Is the section on augmenting the diffusion model with camera control in 3.1 a claimed contribution of the paper? The statement that: “This approach allows for more detailed and accurate 3D rendering, as pixel-specific embedding enhances the model’s ability to handle complex variations in depth and perspective across the video frames.” Is not justified at all, as other types of embeddings are not ablated.
questions: questions: My questions are described in the weaknesses section, but the most important is fleshing out the comparisons to understand what sort of improvement this method makes. The quantitative comparisons are dependent on the generated multi-view images to compare with for PSNR, so the reconstructed results aren’t necessarily good, but rather just match what the model generated. Having qualitative comparisons to existing methods would go a long way in answering my question of whether or not the proposed method is an improvement.
limitations: limitations: Overall the paper does describe some limitations of the method, but it’s not clear if they are relevant. For example, is using the gaussian splatting method really a limitation in this case? I’d be interested to know how long this method takes (is it much slower than LGM baseline), how computationally intensive it is, or how sensitive it is to the initial generation by the multi-view diffusion model.
paper_summary: paper_summary: This paper proposes a method for 3D asset generation conditioned on a single image. The approach follows the recent trend of a two-stage feed-forward model – first generating multi-view images and then using a sparse-view reconstructor to reconstruct the 3D object (specifically, LGM in this paper). This two-stage model has a significant drawback: the inconsistency of the multi-view generation model may result in an imperfect input for the reconstructor, thus causing quality degradation of the final generated 3D assets.

To address this issue, the authors propose adding a 3D-aware feedback mechanism to improve multi-view consistency and enhance the final reconstructed results. Specifically, a self-conditioned mechanism is introduced, where the output of the reconstruction model is fed into the diffusion model. This output is involved in the diffusion process, leading to better 3D consistency.

Overall, the method seems sound to me.
strength_weakness: strength_weakness: (1) Some related works lack citation and discussion:
(a) In "Dmv3d: Denoising Multi-View Diffusion Using 3D Large Reconstruction Model" [ICLR 2024], the paper uses a similar mechanism (though not entirely the same) by employing a 3D reconstructor as a multi-view image denoiser.
(b) “Carve3D: Improving Multi-View Reconstruction Consistency for Diffusion Models with RL Finetuning” [CVPR 2024] enhances multi-view consistency through RL fine-tuning.

(2) I encourage the authors to provide more visual results to help readers understand and appreciate the diffusion/reconstruction process. For example, could the authors provide some visual results of $\tilde{x}_0$ at different denoising steps?

(3) In the comparisons, although quantitative results are provided, could the authors include some qualitative (visual) comparisons to the baseline methods?

Other minor issues:

(1)	Line 118, The Plucker coordinate should be (d, o x d)

(2)	Line 206, We -> we

(3)	Line 225, meshe -> meshes
questions: questions: Not applicable
limitations: limitations: (1)	The method is limited to object-level reconstruction with a clean background. Though this is a common limitation in recent related works, I encourage the authors to explore this issue in future work.

(2)	As discussed in the paper, extracting high-quality surface geometry from the Gaussian model remains an open problem. This is an interesting topic for future research.
paper_summary: paper_summary: This paper (SC3D) proposes a single image-to-3D reconstruction method. It combines multi-view diffusion model and a 3D reconstruction model, and uses the 3D reconstruction results as self condition to improve the multi-view generation process. The motivation of the proposed method is to improve the geometric consistency previous single image reconstruction pipeline, namely first generate multi-view images then perform sparse view reconstruction. The core idea proposed in the paper, 3D-aware feedback, is reasonable and also appear in concurrent works IM-3D and VideoMV. Several ablation studies need to be included to prove that the proposed 3D-feedback (including RGB and coordinate maps) are improving the reconstruction quality. Authors also need to justify more about the contribution w.r.t. related work VideoMV. Furthermore, there is still space to improve the readability in the submission.
strength_weakness: strength_weakness: - Claim about key contributions: the 3D-feedback idea appears already in VideoMV. Since the VideoMV is already available on Arxiv in March, authors need to justify more clearly about the difference and contribution w.r.t. VideoMV.
- lacks generalization results: the method is evaluated on google scan objects, which is standard. However, i am curious to see if the approach generalizes to real world images.
- lacks one ablation: SVD+RGBs feedback, which is missing in Tab. 2 and Tab. 3.
- the readability of the Alg.1 and Alg.2 can be improved. Currently it is too specific and looks like python program. A more abstract algorithm is expected in a scientific paper.
- A typo: in line 213, after comma, "we" instead of "We" (wrong capitalized "W")
questions: questions: - Why there is no SVD+RGB feedback in the ablation study? I think that is important to see how does the coordinates map contribute compared to multi-view RGB images as feedback.
limitations: limitations: - No obvious limitations are found in the proposed method.

Comments: title: Official Comment by Reviewer KsJi
comment: We appreciate your feedback and recognition of the improvements in our revised submission. We've revised the figures for clarity, improved textual organization and clarity, provided additional technical explanations, and thoroughly proofread the manuscript to enhance overall quality. Thank you for your guidance in improving our submission!
Thanks for providing the detailed rebuttal and additional experiments. My concerns are addresses. I will change my rating to Borderline accept.
Thank you for your detailed rebuttal. The additional comparisons and ablations are very insightful, and have increased my perception of the paper. Thus, I am increasing my score. However, the paper will require a significant amount of improvement in the presentation, which makes me less certain that it is ready for publication now.
Thank you for your feedback and for bringing these concerns. Our previous response to your questions may have led to your misunderstanding of the core contribution of our approach. We would like to clarify the distinct contributions and the unique value of our method.
- **Comparison with SyncDreamer.** In terms of specific practices, our 3D-aware feedback differs from the volume features used in SyncDreamer[1]. While many multi-view generation methods, such as MVDream[2], EpiDiff[3], and SPAD[4], employ "3D-aware" features to enforce multi-view consistency, the 3D information they use does **not derive from explicit and accurate 3D models.** Instead, these methods typically rely on implicit multi-view interactions—like full multi-view self-attention in MVDream, epipolar-constraint attention in EpiDiff and SPAD, or SyncDreamer’s use of latent volumes and projection features. However, these approaches do not accurately reflect the true 3D geometric structure. **Our SC3D framework focuses on explicitly modeling 3D geometric structures with more accurate 3D-aware information derived from the reconstruction module.** Moreover, the explicit geometric structure provided by our SC3D framework allows it to be adaptable and generalizable across various network designs other than the implicit geometric features that should be trained or fine-tuned in each applied scenario.
- **Discussion about LGM.** The SC3D framework is designed to seamlessly integrate multi-view generation with reconstruction, demonstrating robust adaptability across different models. While the LGM[5] model is employed in our reconstructions, SC3D's inherent versatility allows it to also **incorporate alternative models** such as GRM[6], GS-LRM[7], among others. This flexibility enhances the framework's ability to adapt to a wide range of reconstruction models, distinguishing it from more rigid systems that are tightly coupled to specific models.
- **Our key contributions.** Our SC3D **framework** integrates multi-view generation with 3D reconstruction, aiming to reduce the "data bias" between these two stages. To address these challenges,  we implement a self-conditioning mechanism and employ a joint training approach. Our multi-view generator in our framework models explicit 3D geometric features by incorporating **accurate 3D-aware** information from the reconstruction model. Our framework is **highly extensible** and can accommodate various multi-view generation networks and reconstruction networks.

We sincerely hope that you will reconsider the contributions and value of our approach. Looking forward to your feedback again.

[1] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024)

[2] MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024)

[3] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion (CVPR 2024)

[4] SPAD : Spatially Aware Multiview Diffusers (CVPR 2024)

[5] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (ECCV 2024)

[6] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

[7] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
Thank you for providing a detailed rebuttal and for conducting additional experiments and the new ablation study, which demonstrates how integrating RGB and CCM images from 3D reconstructions into the 2D diffusion process can enhance reconstruction capabilities.

While the improvement in reconstruction ability is appreciated, my primary concern regarding the innovative contribution of the proposed approach remains unaddressed. The core concept of utilizing 2D multiview diffusion supplemented with 3D-aware features closely mirrors methodologies already explored in SyncDreamer, where a similar technique of constructing a 3D feature volume and projecting it to each view as an additional 3D-aware condition is applied. Additionally, the 3D reconstruction model employed closely resembles the sparse view reconstructor used in LGM, further diluting the distinctiveness of the proposed method.

The idea of combining a 3D reconstruction model to provide 3D-aware features for a 2D multiview generation process is conceptually sound and holds potential. However, for this approach to make a significant impact and to truly advance the field, it requires further development to distinguish it from existing methods and to deepen its methodological insights.

In light of these considerations, I have decided to maintain my initial evaluation score. I believe that with significant refinements, particularly in developing unique aspects and insights beyond those provided by SyncDreamer and LGM, the proposed method could potentially offer a noteworthy contribution to the field.