Coherent 3D Scene Diffusion From a Single RGB Image

Authors: manuel  dahnert
angela  dai
norman  müller
matthias  nießner
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 09:26:07

Abstract: We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. 
Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene.

Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and by allowing the model to learn inter-object relationships throughout the diffusion process.

We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions.

By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04\% improvement in AP3D on SUN RGB-D and a 13.43\% increase in F-Score on Pix3D.

Main paper: 
 introduction: 
 Figure 1 : Given a single RGB image of an indoor scene, our model reconstructs the 3D scene by jointly estimating object arrangements and shapes in a globally consistent manner. Our novel diffusion-based 3D scene reconstruction approach achieves highly accurate predictions by utilizing a novel generative scene prior that captures scene context and inter-object relationships, and by employing an efficient surface alignment loss formulation for joint pose-and shape-synthesis.Holistic 3D scene understanding is crucial for various fields and lays the foundation for many downstream tasks in robotics, 3D content creation, and mixed reality. It bridges the gap between 2D perception and 3D understanding. Despite impressive advancements in 2D perception and 3D reconstruction of individual objects [56, 5, 12, 38] , 3D scene reconstruction from a single RGB observation remains a challenging problem due to its ill-posed nature, heavy occlusions, and the complex multi-object arrangements found in real-world environments. While previous works [15, 32, 33] have shown promising results, they often recover 3D shapes independently and thus do not leverage the scene context nor inter-object relationships. This leads to unrealistic and intersecting object arrangements. Additionally, common feed-forward reconstruction methods [48, 77, 37] struggle with heavy occlusions and weak shape priors, resulting in noisy or incomplete 3D shapes, which hinders immersion and hence limits the applicability in downstream tasks. To address these challenges and to advance 3D scene understanding, we propose a novel generative approach for coherent 3D scene reconstruction from a single RGB image. Specifically, we introduce a new diffusion model that learns a generative scene prior capturing the relationships between objects in terms of arrangement and shapes. When conditioned on a single image, this model simultaneously reconstructs poses and 3D geometries of all scene objects. By framing the reconstruction task as a conditional synthesis process, we achieve significantly more accurate object poses and sharper geometries. Publicly available 3D datasets [47, 62] typically only provide partial ground-truth annotations, which complicates joint training of shape and pose. To overcome this, we propose a novel and efficient surface alignment loss formulation L align that enables joint training of shape and pose even under the lack of full ground-truth supervision. Unlike previous methods [48, 77] that involve costly shape decoding and point sampling on the reconstructed surface, our approach employs an expressive intermediate shape representation that enables direct point sampling from the conditional shape prior. This provides additional supervision and results in more globally consistent 3D scene reconstructions. Our method not only outperforms current state-of-the-art methods by 12.04% in AP 15 3D on SUN RGB-D [62] and by 13.43% in F-Score on Pix3D [64] but also generalizes to other indoor datasets without further fine-tuning. In summary, our contributions include:• A novel diffusion-based 3D scene reconstruction approach that jointly predicts poses and shapes of all visible objects within a scene. • A novel way for modeling a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and inter-object relationships. • An efficient surface alignment loss formulation L align that leverages an expressive intermediate shape representation for additional supervision, even in the absence of full ground-truth annotation. 
related works: 
 The task of 3D scene reconstruction from a single view combines the fundamental domains of 2D perception and 3D modeling into a unified challenge of holistic 3D understanding. Given the multi-faceted nature of the task, we are providing a comprehensive overview of the relevant research directions and contextualizing our contributions.Object Reconstruction.Since the foundational work by Roberts [54] , numerous methods have been developed to learn cues for deriving 3D object structures, thereby bridging the gap between 2D perception and the 3D world. These methods typically involve an image encoder network that processes the input image of a single object, capturing its features. The extracted features are either correlated with an encoded shape database to retrieve a suitable shape [32, 33, 17] , or used by a 3D decoder to reconstruct the object in a specific 3D representation, such as voxel grids [8, 72] , point clouds [14, 43] , meshes [70, 66] , or neural fields [73, 27] . [19] uses a message-passing graph network between geometric primitves to reason about the structure of the shape.Early works formulated single-view scene reconstruction as 3D scene completion from given or estimated depth information [63, 10, 78, 9] in a volumetric grid. While these methods have produced promising results, their representational power to model fine details is limited by the spatial resolution of the 3D grid. Multi-object reconstruction and scene parsing methods represented objects using primitives [13, 23] , voxel grids [68, 35, 52] , or CAD models [26, 24] , while also considering the relation between the objects [31] . The approach presented by Nie et al. [48] is particularly relevant, proposing a holistic method for joint pose and shape estimation from a single image. Zhang et al. [77] extended this idea by incorporating an implicit shape representation and an additional pose refinement using a graph neural network. Although these methods provided significant advances in holistic scene understanding, they struggled with accurate pose estimation and produced noisy scene objects, leading to intersecting or incomplete objects. In contrast to these previous works, we are proposing a generative method to obtain a strong scene prior and formulate the reconstruction task as a conditional synthesis task. This allows for more robust reconstruction that is less prone to object insections or implausible object geometries.In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as a versatile class of generative models, demonstrating impressive results in image and video generation. Unlike other classes of generative models such as auto-regressive models [46, 75, 59] , Generative Adversarial Networks (GANs) [71, 79] and Variational Autoencoders (VAEs), diffusion models iteratively reverse a Markovian noising process. This method ensures stable training and has the ability to capture diverse modes while producing detailed outputs. Several approaches have utilized diffusion models to learn the distribution of individual 3D shapes using various 3D representations, including volumetric grids [6, 7, 25] , point clouds [42, 74] , meshes [2] , implicit functions [30] , neural fields [45, 58, 29] or hybrid representations [80, 76] . [53] propose a hierarchical voxel diffusion model, which is capable of modelling large-scale and fine-detailed geometry. While these methods can synthesize high-quality 3D shapes, they typically focus on single objects in canonical space. In contrast, we are proposing a diffusion-based approach that addresses the more challenging problem of multi-object scene reconstruction, encompassing accurate pose estimations and an understanding of inter-object relationships.Recent works also use diffusion models for single-view object reconstruction [6, 7, 44] . For instance, [65] learns the shape distribution of a single category by denoising a set of 2D images for each object, while [44] projects image features onto noisy point clouds during the diffusion process to ensure geometric plausibility. Recently, several works proposed to leverage multi-view consistency within pre-trained text-conditional 2D image diffusion models to reconstruct individual 3D objects [38, 51, 57] . Similar to our work, Tang et al. [67] use a diffusion model to learn scene priors from synthetic data, showing unconditional scene synthesis of a single room type and text-conditional generation. However, their approach does not support image-based scene reconstruction. Furthermore, it depends on clean synthetic data, which provides full 3D ground truth supervision and CAD model retrieval, thereby limiting shape diversity. While these existing methods have shown promising results on single objects or synthetic scenes, our approach targets real-world scenes. By framing the reconstruction task as a conditional generation process, our scene prior accurately delivers poses and shapes of multiple objects, even in the presence of strong occlusions, significant clutter, and challenging lighting conditions.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our approach in Sec. 4.6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Our method takes a single RGB image of an indoor scene as input and generates a globally consistent 3D scene reconstruction that matches the input image. To this end, we are framing the reconstruction task as a conditional generation problem using a diffusion model conditioned on the input view (Sec. 3.2), which simultaneously predicts the poses (Sec. 3.3) and shapes (Sec. 3.4) of all objects in the scene. Given the ill-posed nature of single-view reconstruction, such a probabilistic formulation is particularly well-suited for this task. To ensure accurate reconstructions and to learn a strong scene prior, we model inter-object relationships within the scene using an intra-scene attention module (Sec. 3.5). Additionally, recognizing the incomplete ground truth in many 3D indoor scene datasets, we introduce a loss formulation for joint shape and pose training, which enables training under only partially available supervision (Sec. 3.6 ). An overview of our approach is illustrated in Fig. 1 . In the following sections, we describe each individual contribution in more detail.We frame the scene reconstruction task as a conditional generation process via a diffusion formulation [22] . Given an instance-segmented RGB image I containing a variable number of 2D objects b i for i ∈ {1, . . . , n}, our model Φ simultanteously estimates all 3D objects o i = (ρ i , σ i ) with 7-DoF poses ρ i and 3D geometries σ i :(ô 1 , . . . , ôn ) = Φ(I|(b 1 , . . . , b n )).(1)During the forward process, we gradually add Gaussian noise to a data point x 0 to x T over a series of discrete time steps T . For a given data point x 0 , e.g., shapes σ i and poses ρ i , the noisy version x t at time step t is given by a Markovian process [22, 60] q(x t |x t-1 ) and its joint distribution q(x 1:T |x 0 ) can be expressed as:EQUATIONEQUATIONwith t ∈ [1, T ] and β t a pre-defined linear variance schedule.During the reverse process, the denoising network Φ tries to remove the noise and recover x 0 from x T as p Φ (x t-1 |x t , y)EQUATIONEQUATIONwith y being the conditional information from the input image I.Conditioning. To effectively guide the diffusion process p Φ (x 0:T |y), it is crucial to accurately model the conditional information y. First, we encode the input image I using a 2D backbone Θ I and apply 2D instance segmentation to get n detected 2D objects b i , comprising of its 2D bounding box, image feature patch, and semantic class (cls). Each element is encoded using a specific embedding function Θ. The per-instance y i and scene condition y is then formed as:EQUATIONEQUATIONTo learn a scene prior over all objects in the scene, we condition the denoising network on the scene condition y. This not only enables learning the individual object representations o i but also facilitates learning to capture the scene context and inter-object relationships (Sec. 3.5). Furthermore, we adopt classifier-free guidance [21] for our model by dropping the condition y with probability p = 0.8, i.e., using a special 0-condition ∅. This allows our model to function as a conditional model p Φ (x 0 |y) and unconditional model p Φ (x 0 ) at the same time, thus enabling unconditional synthesis (Appendix B).Loss Formulation. Unlike related works like [23, 48, 77 ] that regress object poses ρ i and shape parameters σ i using a multitude of highly-tuned losses, we train our model Φ to minimize simple diffusion and alignment losses:L joint (I) = L pose (I) + L shape (I) + λL align , (8) L pose (I) = E ϵ∼N (0,1),t ∥ε ρ (ρ(t), t, I, b) -ϵ∥, (9) L shape (I) = E ϵ∼N (0,1),t ∥ε σ (σ(t), t, I, b) -ϵ∥, (10) where we define z(t) = √ ᾱt z + √ 1 -ᾱt ϵ for z ∈ {ρ, σ} with pre-defined noise coefficients ᾱt , while εz denotes the predicted noise. We use λ = 0.01 to balances the effect of L align .Due to the lack of full ground truth supervision in publically available 3D datasets, we introduce an additional alignment loss L align for joint training of pose and shape (Sec. 3.6) . Depending on the availability of ground-truth data (see Sec. 4.2, we mask out individual losses.We adopt the object pose parameterization of [23] , defining the pose ρ i = (c i , s i , θ i ) of an object by its 3D center c i ∈ R 3 , the spatial size s i ∈ R 3 , and orientation θ i ∈ [-π, π) in . The 3D center c i is further represented by the 2D offset δ i ∈ R 2 between the 2D bounding box center coordinate and the projected coordinate of the 3D center on the image plane, along with the distance d i ∈ R from the object center to the projected center. Our model learns to denoise this 7-dim. pose representation.We represent object shapes using the disentangled shape representation from [20] . A shape is represented as a shape code σ i ∈ R 256 which is factorized into a set of g oriented, anisotropic 3D Gaussians G j , j ∈ {1, ..., g} and an associated 512-dim. latent feature vector per Gaussian. Each Gaussian consist of 16 main parameters: µ j ∈ R 3 (center), factorized covariance matrix U j ∈ R 3×3 (rotation), λ j ∈ R 3 (scale) and π j ∈ R 1 ("mixing" weight). We use g = 16 Gaussians to form a scaffolding of the shape's geometry. Together with their latent features, these Gaussians are decoded into high-fidelity occupancy fields, and the final mesh is extracted by applying marching cubes [40] .While similar to [30] , our model learns to denoise this shape parameterization σ i , our additional surface alignment loss L align (Sec. 3.6) provides relational signal between predicted shapes and poses. This enables additional guidance in the face of missing joint pose and shape annotations as in SUN RGB-D dataset [62] .Given the ill-posed nature of single-view reconstruction, a robust scene prior is essential for achieving good performance. Effectively capturing the scene context and modeling the relationships between objects within the scene is crucial for learning this strong scene prior [31, 77] . Previous methods either reconstruct each object individually [15] or refine their features using graph networks [77] . In contrast, our approach considers the entire scene by conditioning on all scene objects simultaneously p Φ (x 0 |y) and y = (y 1 , . . . , y N ) and additionally allows objects to exchange relational information throughout the entire process. We model the inter-object relationships using an attention formulation [69] , which has proven to be powerful for aggregating contextual information. We denote this formulation as Intra-Scene Attention (ISA), which allows all objects within the scene to attend to each other, effectively modeling their relationships. Please refer to Appendix E for more details and to Tab. 2 for the corresponding ablation study, which demonstrates the effectiveness of our learned scene prior.Publically available 3D scene datasets often only provide partial ground-truth annotations [47, 62] . To facilitate joint training of our model on pose and shape estimation, even in the absence of complete ground-truth annotations, we propose to leverage our expressive intermediate shape representation to provide additional supervision and to align shapes efficiently with the available partial depth information D. An illustration of the surface alignment loss formulation is provided in Fig. 2 . During training, for each object o i , we use the expected shape code σi estimation by our model to obtain the predicted Gaussian Ĝi,j distribution. Given this scaffolding representation, we directly sample m = 1000 points p (j,l) ∼ N (µ j , Σ j ) per Gaussian Ĝi,j resulting in a shape point cloud P i = {p (j,l) |j ∈ {1, . . . , g}, l ∈ {1, . . . , m}}. We transform the resulting shape points P i into the camera frame by the predicted object pose ρi . Using the instance segmentations and ground-truth depth maps, we obtain K i surface points q i k for object o i and define the surface alignment loss for all visible objects as 1-sided Chamfer Distance [16, 48] EQUATIONUnlike previous works such as [48] that perform costly sampling of points on the decoded shape surface, our approach enables direct point sampling from the conditional shape prior Ĝi,j . This loss formulation facilitates joint training of pose and shape for all objects simultaneously and its efficancy is demonstrated through ablation studies in Tab. 2.Our architecture consists of a pre-trained image backbone, a novel image-conditional scene prior diffusion model, and a conditional shape decoder diffusion module. We utilize an off-the-shelf 2D instance segmentation model, Mask2Former [5] , which is pre-trained on COCO [36] using a Swin Transformer [39] backbone, to obtain instance segmentation and image features. Please refer to Appendix E for details about the condition embedding functions. To denoise object poses ρ i , we use a 1-dim. UNet [55] architecture with 8 encoding and decoding blocks with skip connections. Each block consists of a time-conditional ResNet [18] layer, multi-head attention between the per-object condition y i and the pose representation, and our intra-scene attention module (Sec. 3.5) to enable relational information exchange and effectively train a scene prior. We use 8 attention heads, with 64 features per head. To estimate object shapes σ i from the input view I, we denoise the unordered set of Gaussian G i,j using a Transformer [69] model with 2 encoder layers, 6 decoder layers, and multi-head attention with 4 heads to the object condition information, similar to [30] . The per-Gaussian latent features are denoise with a shape decoder diffusion model, realized as another Transformer model with 6 encoder and decoder layers, which is conditioned on the shape Gaussians.For all diffusion training processes, we uniformly sample time steps t = 1, ...T, T = 1000, and use a linear variance schedule with β 1 = 0.0001 and β T = 0.02. We implement our model in PyTorch [50] and use the AdamW [41] optimizer with a learning rate of 1 × 10 -4 and β 1 = 0.9, β 2 = 0.999. We train our models on a single RTX3090 with 24GB VRAM for 1000 epochs on Pix3D, for 500 epochs on SUN RGB-D and for 50 epochs of additional joint training using L align .During inference, we employ DDIM [61] with 100 steps to accelerate sampling speed. For classifierfree guidance [21] , we drop the condition y with probability p = 0.8.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release the code after cleaning up and documenting it for easy usage.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 In the following sections, we will demonstrate the advantages of our method and contributions by evaluating it against common 3D scene reconstruction benchmarks.We compare our method against current state-of-the-art methods for holistic scene understanding: Total3D [48] , Im3D [77] , and InstPIFu [37] . Total3D [48] directly regresses 3D object poses from image features and uses a mesh deformation and edge-removal approach [49] to reconstruct a shape. Im3D [77] utilizes an implicit shape representation and a graph neural network to refine the pose predictions. InstPIFu [37] focuses on single-object reconstruction and proposes to query instancealigned features from the input image in their implicit shape decoder to handle occlusion. For scene reconstruction, they rely on the predicted 3D poses of Im3D. We use the official code and checkpoints provided by the authors of these baseline methods and evaluate with ground truth 2D instance segmentation and camera parameters to ensure a fair comparison. We further compare against a retrieval-based method, ROCA [17] in Appendix D.Following [23, 48, 77] , we train and evaluate the performance of our 3D pose estimation on the SUN RGB-D [62] dataset with the official splits. This dataset consists of 10,335 images of indoor scenes (offices, hotel rooms, lobbies, furniture stores, etc.) captured with four different RGB-D cameras. Each image is annotated with 2D and 3D bounding boxes of objects in the scene. During joint training, we use the provided depth maps together with instance masks to compute L align .We train and evaluate the performance of our 3D shape reconstruction on the Pix3D [64] dataset, which contains images of common furniture objects with pixel-aligned 3D shapes from 9 object classes, comprising 10,046 images. We use the train and test splits defined in [37] , ensuring that 3D models between the respective splits do not overlap.For quantitative comparison against baseline methods, we follow the evaluation protocol of [48] . For pose estimation, we report the intersection over union of the 3D bounding box (IoU 3D ) and average precision with an IoU 3D threshold of 15% (AP 15 3D ) on the SUN RGB-D dataset [62] . In line with previous works [48, 77] , we evaluate with oracle 2D detections but also provide camera parameters to all methods during evaluation. To further assess the alignment of the 3D shapes in the scene, we calculate L align between reconstructed shapes and the instance-segmented ground-truth depth map. For single-view 3D shape reconstruction, we follow evaluate on the Pix3D [64] dataset. We follow [37] and sample 10,000 points on the predicted shape surface, extracted with Marching Cubes [40] at a resolution of 128 3 , and on the ground truth shapes and evaluate Chamfer distance (CD ×10 3 ) and F-score after mesh alignment. In Fig. 3 , we present qualitative comparisons of our approach against state-of-the-art methods for single-view 3D scene reconstruction. The results from Total3D often exhibit intersecting objects and lack global structure. Additionally, their deformation and edgeremoval approach results in 3D shapes with visible artifacts and limited details. While the implicit shape representation of Im3D is more flexible, it often produces incomplete and floating surfaces. In contrast, our diffusion-based reconstruction method, as shown in Tab. 1, learns strong scene priors, resulting in a +0.2 improvement in L align and more coherent 3D arrangements of the objects in the scene (+12.04% AP 15 3D ), as well as high-quality and clean shapes (+13.43% F-Score). Furthermore, we demonstrate the generalizability of our model to other indoor datasets. We evaluate our approach on individual frames from the ScanNet [11] dataset using 2D instance predictions from Mask2Former without additional fine-tuning. As shown in Fig. 4 , our method accurately reconstructs the given input view with matching poses and high-quality 3D geometries. In Appendix D, we additionally train on ScanNet and compare against ROCA [17] . Due to its retrieval approach, the shapes are complete. However, the resulting quality can limited by the diversity of the shape database, which can lead to suboptimal results, see Fig. 11 .As shown in Tabs. 1 and 6, our method outperforms all baseline methods by a significant margin in terms of IoU 3D and AP 15 3D , i.e., improving mAP 15 3D by 12.04% over Im3D [77] . Detailed per-class results are provided in Tabs. 6 and 8. Figs. 3 and 7 demonstrate that our approach effectively learns common object arrangements, such as multiple chairs surrounding a table, while ensuring that furniture pieces do not intersect or float in the air. We attribute these improvements to our model's robust scene understanding, which is derived from learning a strong scene prior that accounts for inter-object relationships. In Tab. 1, we quantitatively compare the single-view shape reconstruction performance of our approach against baseline methods on the Pix3D dataset. The results demonstrate that modeling single-view reconstruction as conditional generation over a robust shape prior leads to significant improvements in Chamfer Distance (+9.6%) and F-Score (+13.43%). Detailed per-class results can be found in Tabs. 7 and 9. Fig. 9 illustrates that InstPiFU often reconstructs noisy and incomplete shapes. In contrast, our approach produces clean 3D geometries with fine details, such as thin chair legs and the crease between pillows of a sofa.In Fig. 5 , we show unconditional results by injecting ∅ as a condition (Sec. 3.2), showcasing that our shape prior models detailed and diverse shape modes across several semantic classes. In Fig. 10 , we additionally visualize the shape decomposition capabilities resulting from our shape encoding and the scaffolding Gaussian representation.We conduct a series of detailed ablation studies to verify the effectiveness of our design decisions and contributions. The quantitative results are provided in Tab. 2.To assess the benefits of the denoising diffusion formulation, we construct a 1-step feed-forward regression model that uses the same conditional information as input features and model architecture but regresses the object outputs directly in a single timestep. As shown in Tab. 2, modeling 3D scene reconstruction as a conditional diffusion process, rather than using a feed-forward regression formulation, results in significant improvements of +11.08% AP 15 3D and +0.19 L align .What is the effect of our scene prior modeling?We evaluate the impact of learning a scene prior by modeling the distribution of all objects and their relationships compared to learning the marginal per-object distribution, i.e., predicting each object individually. As shown in Tab. 2, our joint-object scene prior yields a significant improvement of +9.30% AP 15 3D over per-object prediction. This improvement underscores the importance of learning a robust scene prior that effectively captures inter-object relationships.We investigate the benefit of joint training for pose and shape using L align compared to individual training of pose estimation and shape reconstruction. Although our model already learns strong scene and shape priors, Tab. 2 shows that joint training provides additional benefits, resulting in an improvement of +2.11% in AP 15 3D and +0.07 in L align .Figure 3 : Qualitative comparison of 3D scene reconstruction on SUN RGB-D [62] . While the baselines often produce noisy or incomplete shape reconstruction of intersecting or misplaced objects, our method produces plausible object arrangements as well as high-quality shape reconstructions.Figure 4 : Inference results on ScanNet [11] . We use our model trained on SUN RGB-D [62] and perform inference on individual frames of ScanNet without fine-tuning. We observe strong generalization capabilities with respect to different camera parameters and scene arrangements.While our conditional scene diffusion approach for single-view 3D scene reconstruction demonstrates significant improvements, there are some limitations. First, our method relies on accurate 2D object detection, making it dependent on the performance of 2D perception models. Upcoming state-of-theart 2D detection models [1] can be seamlessly integrated to enhance the performance of our approach. Second, our shape prior, trained on a diverse set of semantic classes using 3D shape supervision, does not generalize to unseen object categories. This can be mitigated by combining our model for known categories with single-object diffusion models that leverage pre-trained text-image generation models for 3D shape synthesis [38] of uncommon shape categories. While accurate 3D scene reconstruction forms the foundation for subsequent downstream tasks like mixed reality applications, our current model assumes a static scene geometry. Future work could integrate object affordance and articulation into our shape prior [34] to enable more immersive human-scene interactions.We do not anticipate any societal consequences or negative ethical implications arising from our work. Our approach advances the holistic understanding of 2D perception and 3D modeling, benefiting various research areas. 
conclusion: 
 In this paper, we present a novel diffusion-based approach for coherent 3D scene reconstructions from a single RGB image. Our method combines a simple yet powerful denoising formulation with a robust generative scene prior that learns inter-object relationships by exchanging relational information among all scene objects. To address the issue of missing ground-truth annotations in publicly available 3D datasets, we introduce a surface alignment loss L align to jointly train shape and pose, effectively leveraging our shape representation. Our approach significantly enhances 3D scene understanding, outperforming current state-of-the-art methods across various benchmarks, with +12.04% AP 15 3D on SUN RGB-D and +13.43% F-Score on Pix3D. Extensive experiments demonstrate that our contributions -3D scene reconstruction as a conditional diffusion process, scene prior modeling, and joint shape-pose training enabled by L align -collectively contribute to the overall performance gain. Additionally, we show that our model supports unconditional synthesis and generalizes well to other indoor datasets without further fine-tuning. We believe these advancements lay a solid foundation for future progress in holistic 3D scene understanding and open up exciting applications in mixed reality, content creation, and robotics. [62] . The input image is displayed on the left, and the predicted and ground-truth 3D arrangements are visualized as topdown orthographic views of the scene. We observe that Total3D frequently lacks a globally consistent structure, while Im3D predicts globally structured results but occasionally produces intersecting or floating objects. In contrast, our approach successfully recovers a coherent arrangement of objects within the scene by learning a robust scene prior. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Given limited computational resources, we are not able to train multiple models using different initializations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Sec. 4.6, we discuss why we do not anticipate any negative impacts of our work on single-view 3D scene reconstruction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our model only relies on a 2D instance segmentation model, which was pre-trained on COCO.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: For training and testing, we use multiple publically available datasets, namely SUN RGB-D [62] , Pix3D [64] , and ScanNet [11] , which were properly explained and cited in the main paper (Sec. 4.2).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.  
 
 Appendix: In Tab. 4, we show detailed comparisons of our approach against baseline methods, Total3D [48] and Im3D [77] , on the 10 most common classes of SUN RGB-D. Our approach consistently outperforms all baseline methods on all classes except the "bed" class. We attribute this exception to the fact that beds are often only partially visible in the input view due to their spatial extent, which introduces higher variability. In contrast, Im3D employs a series of geometric losses and regularization terms, which seems to help in extreme amodal cases at the cost of additional loss balancing. Nevertheless, our method achieves a significant overall improvement of 12.04% in AP 15 3D on these 10 classes, with particularly notable gains for "dressers" (+26.03%), "chairs" (+21.91%) and "cabinets" (+19.37%), showcasing the effect of our robust scene prior.Tabs. 6 and 8 show the per-class comparisons and ablation studies on all 37 NYU classes in terms of IoU 3D and mAP 15 3D . Our approach improves compared to Im3D by a +7.57% increase in mAP 15 3D and +4.56% increase in class-mean IoU 3D across all 37 classes. The ablation results highlight the importance of our diffusion formulation (+7.67% mAP 15 3D ), scene prior modeling (+7.11% mAP 15 3D ), and joint training using the surface alignment loss L align (+0.72 mAP 15 3D ).For single-view object reconstruction, we evaluate Chamfer Distance and F-Score on Pix3D and show per-class comparisons in Tabs. 7 and 9. Our image-conditional shape prior leads to significant improvements, +9.6% in Chamfer Distance and +13.43 in F-Score, while outperforming InstPIFu in most categories, except sofas and wardrobes in F-Score.Room Layout [48, 77] also predict the room bounding box with a separate network head. We study, how our model can also predict the room layout. For that we include the room bounding box pose as part of the object poses during the diffusion process. We follow the room layout parameterization of [48, 77] and model the 3D room center directly instead of decomposing it as 2D offset & distance, which is done for the objects. In Tab. 3, we demonstrate that by denoising the pose of room layout, we outperform the regression-based methods. We compare with a shape retrieval baseline, namely ROCA [17] . Since ROCA requires full groundtruth supervision during training, we adopt their setup and train our model on the same 25,000 frames from the ScanNet [11] dataset with pose annotations derived from Scan2CAD [3] , as well as the same CAD pool from ShapeNet [4] . We additionally adopt their full 9-DoF pose parameterization by predicting all 3 rotation angles. Following ROCA, we quantitatively evaluate the Alignment Accuracy in Tab. 5. Please refer to [3, 17] for the details of the evaluation. In Fig. 11 , we can see that ROCA retrieves clean and complete shapes by definition. However, due to its limited shape database, it cannot capture all shape modes accurately, leading to shape mismatches. Our reconstruction-based approach instead can recover faithful shape results while simultaneously predicting a coherent object arrangement.To ensure a reasonable signal-noise ratio [28] among the object pose parameters, we normalize the parameters to [-1, 1] by dividing them by its max value and shift the range using a parameter-specific µ value. For this, we calculate the min-max ranges of all pose parameters, i.e., rotation θ, 3D scale s, and projected distance d, within the train Table 6 : 3D pose estimation results for all NYU-37 classes on SUN RGB-D [62] . We report the Average Precision (AP) at 15% 3D-IoU threshold of the baseline and different variants of our approach: Our approach outperforms Total3D and Im3D on most semantic categories, especially on frequent classes likes chairs (+21.9%) or tables (+12.7%).Ours EQUATIONs : µ = 3.5, max = 7.0, (13) θ : µ = 0.0, max = 3.14.(EQUATIONDuring training, the loss is computed on the un-normalized parameter ranges. After inference and for evaluation, we un-normalize each parameter according to its original range.During training, for each object o i , we use the predicted shape σi to estimate its scaffolding Gaussians Ĝj . From each 3D Gaussian distribution, we directly draw 3D point samples p (j,l) ∼ N (µ j , Σ j ). This shape point cloud P i approximates the shape. With the predicted and un-normalized object pose ρi , we define a 3D rigid transformation R 4×4 and transform the shape point cloud P i to the camera coordinate system. We use this transformed shape pointcloud P cam i and the instance-segmented ground-truth depth map from Table 7 : Per-class comparisons of shape reconstruction on Pix3D [64] . We report F-Score using the non-overlapping 3D model split from [37] We use the multi-head attention mechanism [69] between the scene objects to allow them to attend to each other, effectively learning their inter-object relationships and the scene context. Specifically, given an unordered set S = [o 1 , o 2 , ..., o n ], o i ∈ R n per-object n-dimensional feature vectors, projection layers (W Q , W K and W V ) and features Q = S × W Q , K = S × W K and V = S × W V after projection. we define the intra-scene attention as:EQUATIONCondition: Embedding Functions After cropping the 2D image feature patch R W ×H×C from the frozen image backone Θ I , we apply adaptive average pooling to resize the per-object feature patches to a common 2D size leading to resized per-object feature crop of 8 × 8 and C = 256. This feature crop is further embedded using a small 2D CNN Θ feat with 3 blocks of convolutional layers with 512 features, group norm, and leaky ReLU activation. The embedded feature crop is reshaped to a 4096-dim vector.Θ box is implemented as sinusoidal position encoding with 10 frequencies. This function is applied on a 2D bounding box, represented by the top-left and bottom-right corners, leading to an 84-dim vector per object. For Θ cls , we use a simple 1-hot encoding to embed the semantic class information. The final per-object condition information is the concatenation, resulting in a 4127-dim vector for each object.Since the official code of SPAGHETTI does not include the training code and only provides checkpoints for two different shape classes (chairs, Figure 9 : Qualitative comparison of 3D shape reconstruction on the Pix3D [64] . While InstPIFu often produces noisy surfaces, our image-conditional 3D diffusion model synthesizes high-quality shapes that closely match the target geometries.Figure 10 : Shape decomposition visualization. We assign each vertex of the reconstructed mesh to the closest 3D Gaussian center and visualize the assignment with individual colors. Our scaffolding representation decomposes the shape into distinctive regions and aligns well with certain semantic parts, e.g., individual chair legs or the arm rests of a sofa. airplanes), we re-implement the training procedure, loss function, and disentanglement loss following the description in the papers to train the full shape prior over all relevant shape categories. Random geometric augmentations are essential during training to achieve self-supervised disentanglement into extrinsic and intrinsic shape properties. We apply full 360-degree random rotations, uniform scale augmentation between 0.7 and 1.3, and translation jitter of ∓0.3 on the disentangled extrinsic and target pointcloud. Further, we do not utilize the symmetry options of the original implementation.Figure 11 : Comparison with retrieval baseline method ROCA [17] on frames from ScanNet [11] . While ROCA cannot always retrieve a matching mode from the shape database, such as the desk in the first row, our diffusion-based reconstruction approach reconstructs accurate shapes and poses. Table 8 : Per-class pose estimation results for all NYU-37 classes on SUN RGB-D [62] . We evaluate the pose estimation quality in terms of 3D IoU. Our scene prior formulation achieves improvements across all categories which particular high gains on common object classes like "chair" (+16.6%) or "desk" (+16.4%).Ours Justification: We provide experimental results demonstrating the performance improvements by following the evaluation protocol of common 3D scene reconstruction benchmarks (Tab. 1). We further ablate the impact of the individual contributions -denoising formulation, scene prior modeling, and surface alignment loss in Tab. 2. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper received four reviews from expert reviewers. The rebuttal has cleared up most remaining questions, and in the end, all reviewers retain their positive rating and recommend acceptance. 
The reviewers highlight the novelty of the formulation, the clarity of the experiments and the writing of the paper.
The AC strongly encourages the authors to include all additional results, comparisons, and clarifications in the appropriate sections of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes an approach to reconstruct the 3D surfaces of multiple objects from a single RGB image. Given an instance-segmented RGB image as input, the poses and shapes of objects are jointly predicted and denoised using a diffusion model conditioned on the input image. A one-sided chamfer distance (from available ground truth surface point to sampled points in camera space) is used to provide supervision on incomplete surfaces. The output is geometrically evaluated against Total3D (3D detection-based composition), Im3D (implicit surface), and InstPIFu (pixel-aligned implicit surface) and shows improved scene coherence and shape quality.
strength_weakness: strength_weakness: In order for the simple loss (one-sided chamfer distance) to work well, you need strong category-specific priors on generated shapes; it requires the estimated shape gaussians to be stable in the first place. The joint learning of poses is also inherently category-specific and might not generalize well to unseen or uncommon object categories. While this limitation is mentioned in the paper, I think this is an inherent limitation that is not easy to mitigate.

Limits of one-sided chamfer distance: In qualitative results containing office chair images for example, I noticed that many of them have extra leg parts hallucinated. I think this is an example of something that the one-sided chamfer loss won't be able to fix since the error isn't reflected in the loss. I'm surprised the method worked well despite this; this isn't necessarily a weakness, but tells us something about the problem we're trying to solve and evaluation methods (maybe "alignment is all you need"?). I think a retrieval-based baseline (with alignment) would have been interesting for the reasons mentioned.

I tend to think the improvements are mostly due to architectural advancements such as attention models (that can process arbitrary unordered variable-length objects) and better off-the-shelf shape models available. The paper appropriately applied those techniques and that is a good thing, but maybe it does not fundamentally improve our understanding of the problem. Perhaps this isn't a fair criticism considering the limitations and quality of other papers accepted in similar venues. I currently feel borderline about the paper. The holistic nature of the problem is difficult enough and the paper proposes one simple solution that I think can be used by other researchers still.
questions: questions: (please see weaknesses section)
limitations: limitations: I think they are addressed well.
paper_summary: paper_summary: In this work, the authors propose a diffusion-based method for generating coherent 3D scenes from a single input image. They use a diffusion model to jointly denoise the 3D poses and geometries of all objects. To address the incomplete ground truth of existing datasets, they also introduce a surface alignment loss to better learn the 3D geometry. Experimental results show that the proposed approach significantly outperforms state-of-the-art methods.
strength_weakness: strength_weakness: - I am unclear on how to learn the shape information:

(i) For the shape code sigma_i (16 Gaussians x 16 dimensions per Gaussian), do you have the ground truth for it? If not, how do you apply the shape diffusion loss mentioned in Equation 10?

(ii) To decode the shape code from Gaussians to an occupancy grid, you apply a shape decoder diffusion model (mentioned in Line 213). How do you train this model? Do you include any details about this model?

- I am interested in how the proposed method learns the intermediate scene representation (i.e., anisotropic 3D Gaussians). You use the surface alignment loss as supervision. Are there other indirect supervisions? How do you ensure that the Gaussians do not intersect each other or become trivial (e.g., the same as each other)?

- If I understand correctly, you add inter-object attentions to the intra-scene attention to allow the pose and shape model to run in parallel. If so, this operator is quite common.

- I find the author may miss the citation and discussion with an important literature [a].

[a] CoReNet: Coherent 3D scene reconstruction from a single RGB image. ECCV 2020.
questions: questions: - It is suggested to add notation for cls in Equation 6.

- It is suggested to add some figures for the ablation study.

- I find [b] to be quite related to this work. It is suggested to discuss this paper in the related work section. Combining some 3D architecture with the proposed method could be a nice extension.

[b] XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.
limitations: limitations: I think the authors properly discuss the limitations of this work.
paper_summary: paper_summary: This paper proposes a novel framework for 3D scene reconstruction from single images based on diffusion models. This framework jointly predicts 3D object poses and shapes with seperate diffusion models and captures global scene context with a standard mutli-head attention module. To train this model, the paper proposes a surface alignment loss based on one-sided Chamfer Distance, which makes their method able to utilize depth maps as supervision. The experimental results on Sun RGB-D, ScanNet, and Pix3D and numerical ablation studies demonstrate the efficacy of proposed method.
strength_weakness: strength_weakness: My concerns are mainly about the description of the method:
1. The model architecture is not clear enough in Section 3.7. It seems that each object pose, each object shape, and each Gaussian, are denoised using an independent diffusion model? Where is ISA added and how is it integrated with other modules? Are the inputs of ISA from Equation (7)? 
2. In Figure 1, are the noisy shape Gaussians as in shape with guassian noise? Or are they initial estimation directly from image features? It's a bit confusing.
3. Are the diffusion models for shape and pose trained with full 3D supervision before joint training with alignment loss? If yes, what if there are only ground truth depth maps in training data?

Besides, the overall idea of 3D generation from features as in Equation (6) combined with global context via attention has been explored in previous related works, such as part-based 3D object reconstruction, e.g., He, Qian, et al. "Single Image 3D Object Estimation with Primitive Graph Networks." Proceedings of the 29th ACM International Conference on Multimedia. 2021. This should be included in related works.
questions: questions: 1. According to the alignment loss, the predicted Gaussians should be converged to centered on the surface of 3D objects? Otheriwse, the direct sampling won't be around the surface?
2. How do you deal with varying number of objects within a single image? How do you deal with false positives and false negatives from detection?
3. Have you tried other normalization methods besides Equation (12-14)? Does it affect the learning efficiency?
4. Does the disentanglement of pose and shape helps with learning convergence?
5. How is the proposed method comparing to other SOTAs in single object reconstruction? Can this method be applied to part-level object reconstruction?
limitations: limitations: The authors adequately addressed the limitations and I do not have more to add here.
paper_summary: paper_summary: The task is to perform 3D scene reconstruction from a single RGB views, such that they can output both scene object poses and geometries. Prior work cannot jointly output both poses and geometries, one design choice that is validated by this work to be effective. Method-wise, extending on SALAD, this work introduces a diffusion model that learns a generative scene prior that can capture contextual information across scene elements. To achieve joint learning, they introduce a alignment loss such that they only need to single view supervision. The author conducts extensive experiments to justify their design choices.
strength_weakness: strength_weakness: 1. Minor concern is that the paper over-claim a bit on the contribution. Despite phrasing as 3D scene reconstruction (as is mentioned in teaser, abstract and intro) from single view, the proposed framework in the method actually only reconstruct poses and geometries of foreground objects in the image without the background part, e.g., ground or wall. 

2. Global shape prior cannot always fit the observed object perfectly, with different local details, e.g., Fig.3 shelf. This could accumulate error in the downstream alignment. 

3. Furniture kinds of 3D objects, e.g., those in ShapeNet, are simple, which usually is easier to get good shape reconstruction. How would the proposed framework work on more intricate 3D objects, e.g., plants, trees?
questions: questions: 1. Could you clarify on how the proposed framework resolve with scale ambiguity given the single view setup?

2. Writing needs more polish. For example, Equation 1 seems incorrect (feel free to clarify in the rebuttal). Based on the description in Line 121, RGB image I seems to be given, which indicates it is one item to be conditioned on. And in Line 150, missing parenthesis for (See Sec. 4.2

3. Could you clarify on how the gradient being back-propagated through point sampling (Line 190)? "We directly sample m = 1000 points p(j,l) ∼ N (μj , Σj ) per Gaussian Gˆi,j resulting in a shape point cloud Pi = {p(j,l)|j ∈ {1,...,g},l ∈ {1,...,m}}." In SALAD, it has another cascaded diffusion Transformer that condition on extrinsic and intrinsic parameters to get the part-level point cloud from Transformer. But I am not sure if the process is the same in this paper. 

4. Could you clarify how the texture of the 3D objects are learned? The qualitative results show objects with color but there is no supervision on the color.
limitations: limitations: Yes.

Comments: comment: Thanks for your rebuttal. I would like to keep my rate.
Thank you for the response. The architecture diagram is very helpful and most of my concerns are resolved. I do believe this is a good work and would like keep my rating.
Thanks for the detailed clarification. I retain my rating. Nice work!
Thank you for the rebuttal. My "borderline accept" rating remains the same.
I agree with other reviewers that the paper is above threshold.