Title: Coherent 3D Scene Diffusion From a Single RGB Image
Venue: NeurIPS 2024
Accepted: yes
Keywords: Single RGB Image 3D Scene Reconstruction

Diffusion Models

Scene Understanding

3D Scene Prior
Abstract: We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. 
Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene.

Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and by allowing the model to learn inter-object relationships throughout the diffusion process.

We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions.

By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04\% improvement in AP3D on SUN RGB-D and a 13.43\% increase in F-Score on Pix3D.

Coherent 3D Scene Diffusion From a Single RGB Image

Abstract

We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene. Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process. We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions. By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04% improvement in AP3D on SUN RGB-D and a 13.43% increase in F-Score on Pix3D.

1 Introduction

Input

Output

RGBImage

IntermediateScene

Gaussians

Representation

BoX20

Joint

Shape

Pose&Shape

BOX2D

Diffusion

Decoding

BoX20

Holistic 3D scene understanding is crucial for various fields and lays the foundation for many downstream tasks in robotics, 3D content creation, and mixed reality. It bridges the gap between 2D perception and 3D understanding. Despite impressive advancements in 2D perception and 3D reconstruction of individual objects [56, 5, 12, 38], 3D scene reconstruction from a single RGB observation remains a challenging problem due to its ill-posed nature, heavy occlusions, and the

complex multi-object arrangements found in real-world environments. While previous works [15, 32, 33] have shown promising results, they often recover 3D shapes independently and thus do not leverage the scene context nor inter-object relationships. This leads to unrealistic and intersecting object arrangements. Additionally, common feed-forward reconstruction methods [48, 77, 37] struggle with heavy occlusions and weak shape priors, resulting in noisy or incomplete 3D shapes, which hinders immersion and hence limits the applicability in downstream tasks. To address these challenges and to advance 3D scene understanding, we propose a novel generative approach for coherent 3D scene reconstruction from a single RGB image. Specifically, we introduce a new diffusion model that learns a generative scene prior capturing the relationships between objects in terms of arrangement and shapes. When conditioned on a single image, this model simultaneously reconstructs poses and 3D geometries of all scene objects. By framing the reconstruction task as a conditional synthesis process, we achieve significantly more accurate object poses and sharper geometries. Publicly available 3D datasets [47, 62] typically only provide partial ground-truth annotations, which complicates joint training of shape and pose. To overcome this, we propose a novel and efficient surface alignment loss formulation L align that enables joint training of shape and pose even under the lack of full ground-truth supervision. Unlike previous methods [48, 77] that involve costly shape decoding and point sampling on the reconstructed surface, our approach employs an expressive intermediate shape representation that enables direct point sampling from the conditional shape prior. This provides additional supervision and results in more globally consistent 3D scene reconstructions. Our method not only outperforms current state-of-the-art methods by 12.04% in AP 15 3D on SUN RGB-D [62] and by 13.43% in F-Score on Pix3D [64] but also generalizes to other indoor datasets without further fine-tuning. In summary, our contributions include:

A novel diffusion-based 3D scene reconstruction approach that jointly predicts poses and shapes of all visible objects within a scene.

A novel way for modeling a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and inter-object relationships.

An efficient surface alignment loss formulation L align that leverages an expressive intermediate shape representation for additional supervision, even in the absence of full ground-truth annotation.

2 Related Works

The task of 3D scene reconstruction from a single view combines the fundamental domains of 2D perception and 3D modeling into a unified challenge of holistic 3D understanding. Given the multi-faceted nature of the task, we are providing a comprehensive overview of the relevant research directions and contextualizing our contributions.

2.1 Single-View 3D Reconstruction

Object Reconstruction. Since the foundational work by Roberts [54], numerous methods have been developed to learn cues for deriving 3D object structures, thereby bridging the gap between 2D perception and the 3D world. These methods typically involve an image encoder network that processes the input image of a single object, capturing its features. The extracted features are either correlated with an encoded shape database to retrieve a suitable shape [32, 33, 17], or used by a 3D decoder to reconstruct the object in a specific 3D representation, such as voxel grids [8, 72], point clouds [14, 43], meshes [70, 66], or neural fields [73, 27]. [19] uses a message-passing graph network between geometric primitves to reason about the structure of the shape.

Scene Reconstruction. Early works formulated single-view scene reconstruction as 3D scene completion from given or estimated depth information [63, 10, 78, 9] in a volumetric grid. While these methods have produced promising results, their representational power to model fine details is limited by the spatial resolution of the 3D grid. Multi-object reconstruction and scene parsing methods represented objects using primitives [13, 23], voxel grids [68, 35, 52], or CAD models [26, 24], while also considering the relation between the objects [31]. The approach presented by Nie et al . [48] is particularly relevant, proposing a holistic method for joint pose and shape estimation from a single image. Zhang et al . [77] extended this idea by incorporating an implicit shape representation and

an additional pose refinement using a graph neural network. Although these methods provided significant advances in holistic scene understanding, they struggled with accurate pose estimation and produced noisy scene objects, leading to intersecting or incomplete objects. In contrast to these previous works, we are proposing a generative method to obtain a strong scene prior and formulate the reconstruction task as a conditional synthesis task. This allows for more robust reconstruction that is less prone to object insections or implausible object geometries.

2.2 3D Diffusion Models

In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as a versatile class of generative models, demonstrating impressive results in image and video generation. Unlike other classes of generative models such as auto-regressive models [46, 75, 59], Generative Adversarial Networks (GANs) [71, 79] and Variational Autoencoders (VAEs), diffusion models iteratively reverse a Markovian noising process. This method ensures stable training and has the ability to capture diverse modes while producing detailed outputs. Several approaches have utilized diffusion models to learn the distribution of individual 3D shapes using various 3D representations, including volumetric grids [6, 7, 25], point clouds [42, 74], meshes [2], implicit functions [30], neural fields [45, 58, 29] or hybrid representations [80, 76]. [53] propose a hierarchical voxel diffusion model, which is capable of modelling large-scale and fine-detailed geometry. While these methods can synthesize high-quality 3D shapes, they typically focus on single objects in canonical space. In contrast, we are proposing a diffusion-based approach that addresses the more challenging problem of multi-object scene reconstruction, encompassing accurate pose estimations and an understanding of inter-object relationships.

Conditional Diffusion for 3D Reconstruction. Recent works also use diffusion models for single-view object reconstruction [6, 7, 44]. For instance, [65] learns the shape distribution of a single category by denoising a set of 2D images for each object, while [44] projects image features onto noisy point clouds during the diffusion process to ensure geometric plausibility. Recently, several works proposed to leverage multi-view consistency within pre-trained text-conditional 2D image diffusion models to reconstruct individual 3D objects [38, 51, 57]. Similar to our work, Tang et al . [67] use a diffusion model to learn scene priors from synthetic data, showing unconditional scene synthesis of a single room type and text-conditional generation. However, their approach does not support image-based scene reconstruction. Furthermore, it depends on clean synthetic data, which provides full 3D ground truth supervision and CAD model retrieval, thereby limiting shape diversity. While these existing methods have shown promising results on single objects or synthetic scenes, our approach targets real-world scenes. By framing the reconstruction task as a conditional generation process, our scene prior accurately delivers poses and shapes of multiple objects, even in the presence of strong occlusions, significant clutter, and challenging lighting conditions.

3 Method

3.1 Overview

Our method takes a single RGB image of an indoor scene as input and generates a globally consistent 3D scene reconstruction that matches the input image. To this end, we are framing the reconstruction task as a conditional generation problem using a diffusion model conditioned on the input view (Sec. 3.2), which simultaneously predicts the poses (Sec. 3.3) and shapes (Sec. 3.4) of all objects in the scene. Given the ill-posed nature of single-view reconstruction, such a probabilistic formulation is particularly well-suited for this task. To ensure accurate reconstructions and to learn a strong scene prior, we model inter-object relationships within the scene using an intra-scene attention module (Sec. 3.5). Additionally, recognizing the incomplete ground truth in many 3D indoor scene datasets, we introduce a loss formulation for joint shape and pose training, which enables training under only partially available supervision (Sec. 3.6). An overview of our approach is illustrated in Fig. 1. In the following sections, we describe each individual contribution in more detail.

3.2 Conditional 3D Scene Diffusion

We frame the scene reconstruction task as a conditional generation process via a diffusion formulation [22]. Given an instance-segmented RGB image I containing a variable number of 2D objects b i

ScenePrior

SurfaceAlignmentLoss

Pose

Shape

Anisotropic

PointSamples

PointSamples

1-SidedChamferDistance

Shape Gaussians

inCanonicalSpace

inCameraspace

ObjectRelationships

SceneContext&

Estimate

shapeo;

Pij~N(utyEt.)

Samplepoints

withposepi

Transform

Lalign

Pose

Shape

Pose

Shape

for i ∈ { 1 , . . . , n } , our model Φ simultanteously estimates all 3D objects o i = ( ρ i , σ i ) with 7-DoF poses ρ i and 3D geometries σ i :

During the forward process , we gradually add Gaussian noise to a data point x 0 to x T over a series of discrete time steps T . For a given data point x 0 , e.g ., shapes σ i and poses ρ i , the noisy version x t at time step t is given by a Markovian process [22, 60] q ( x t | x t -1 ) and its joint distribution q ( x 1: T | x 0 ) can be expressed as:

with t ∈ [1 , T ] and β t a pre-defined linear variance schedule.

During the reverse process , the denoising network Φ tries to remove the noise and recover x 0 from x T as p Φ ( x t -1 | x t , y )

with y being the conditional information from the input image I .

Conditioning. To effectively guide the diffusion process p Φ ( x 0: T | y ) , it is crucial to accurately model the conditional information y . First, we encode the input image I using a 2D backbone Θ I and apply 2D instance segmentation to get n detected 2D objects b i , comprising of its 2D bounding box, image feature patch, and semantic class ( cls ) . Each element is encoded using a specific embedding function Θ . The per-instance y i and scene condition y is then formed as:

To learn a scene prior over all objects in the scene, we condition the denoising network on the scene condition y . This not only enables learning the individual object representations o i but also facilitates learning to capture the scene context and inter-object relationships (Sec. 3.5). Furthermore, we adopt classifier-free guidance [21] for our model by dropping the condition y with probability p = 0 . 8 , i.e ., using a special 0-condition ∅ . This allows our model to function as a conditional model p Φ ( x 0 | y ) and unconditional model p Φ ( x 0 ) at the same time, thus enabling unconditional synthesis (Appendix B).

Loss Formulation. Unlike related works like [23, 48, 77] that regress object poses ρ i and shape parameters σ i using a multitude of highly-tuned losses, we train our model Φ to minimize simple diffusion and alignment losses:

where we define ˜ z ( t ) = √ ¯ α t z + √ 1 -¯ α t ϵ for z ∈ { ρ, σ } with pre-defined noise coefficients ¯ α t , while ˆ ϵ z denotes the predicted noise. We use λ = 0 . 01 to balances the effect of L align .

Due to the lack of full ground truth supervision in publically available 3D datasets, we introduce an additional alignment loss L align for joint training of pose and shape (Sec. 3.6). Depending on the availability of ground-truth data (see Sec. 4.2, we mask out individual losses.

3.3 Object Pose Parameterization

We adopt the object pose parameterization of [23], defining the pose ρ i = ( c i , s i , θ i ) of an object by its 3D center c i ∈ R 3 , the spatial size s i ∈ R 3 , and orientation θ i ∈ [ -π, π ) in . The 3D center c i is further represented by the 2D offset δ i ∈ R 2 between the 2D bounding box center coordinate and the projected coordinate of the 3D center on the image plane, along with the distance d i ∈ R from the object center to the projected center. Our model learns to denoise this 7 -dim. pose representation.

3.4 Shape Encoding

We represent object shapes using the disentangled shape representation from [20]. A shape is represented as a shape code σ i ∈ R 256 which is factorized into a set of g oriented, anisotropic 3D Gaussians G j , j ∈ { 1 , ..., g } and an associated 512 -dim. latent feature vector per Gaussian. Each Gaussian consist of 16 main parameters: µ j ∈ R 3 (center), factorized covariance matrix U j ∈ R 3 × 3 (rotation), λ j ∈ R 3 (scale) and π j ∈ R 1 ('mixing' weight). We use g = 16 Gaussians to form a scaffolding of the shape's geometry. Together with their latent features, these Gaussians are decoded into high-fidelity occupancy fields, and the final mesh is extracted by applying marching cubes [40].

While similar to [30], our model learns to denoise this shape parameterization σ i , our additional surface alignment loss L align (Sec. 3.6) provides relational signal between predicted shapes and poses. This enables additional guidance in the face of missing joint pose and shape annotations as in SUN RGB-D dataset [62].

3.5 Scene Prior Modeling

Given the ill-posed nature of single-view reconstruction, a robust scene prior is essential for achieving good performance. Effectively capturing the scene context and modeling the relationships between objects within the scene is crucial for learning this strong scene prior [31, 77]. Previous methods either reconstruct each object individually [15] or refine their features using graph networks [77]. In contrast, our approach considers the entire scene by conditioning on all scene objects simultaneously p Φ ( x 0 | y ) and y = ( y 1 , . . . , y N ) and additionally allows objects to exchange relational information throughout the entire process. We model the inter-object relationships using an attention formulation [69], which has proven to be powerful for aggregating contextual information.

We denote this formulation as Intra-Scene Attention (ISA), which allows all objects within the scene to attend to each other, effectively modeling their relationships. Please refer to Appendix E for more details and to Tab. 2 for the corresponding ablation study, which demonstrates the effectiveness of our learned scene prior.

3.6 Surface Alignment Loss

Publically available 3D scene datasets often only provide partial ground-truth annotations [47, 62]. To facilitate joint training of our model on pose and shape estimation, even in the absence of complete ground-truth annotations, we propose to leverage our expressive intermediate shape representation

to provide additional supervision and to align shapes efficiently with the available partial depth information D . An illustration of the surface alignment loss formulation is provided in Fig. 2.

During training, for each object o i , we use the expected shape code ˆ σ i estimation by our model to obtain the predicted Gaussian ˆ G i,j distribution. Given this scaffolding representation, we directly sample m = 1000 points p ( j,l ) ∼ N ( µ j , Σ j ) per Gaussian ˆ G i,j resulting in a shape point cloud P i = { p ( j,l ) | j ∈ { 1 , . . . , g } , l ∈ { 1 , . . . , m }} . We transform the resulting shape points P i into the camera frame by the predicted object pose ˆ ρ i . Using the instance segmentations and ground-truth depth maps, we obtain K i surface points q i k for object o i and define the surface alignment loss for all visible objects as 1-sided Chamfer Distance [16, 48]

Unlike previous works such as [48] that perform costly sampling of points on the decoded shape surface, our approach enables direct point sampling from the conditional shape prior ˆ G i,j . This loss formulation facilitates joint training of pose and shape for all objects simultaneously and its efficancy is demonstrated through ablation studies in Tab. 2.

3.7 Architecture

Our architecture consists of a pre-trained image backbone, a novel image-conditional scene prior diffusion model, and a conditional shape decoder diffusion module. We utilize an off-the-shelf 2D instance segmentation model, Mask2Former [5], which is pre-trained on COCO [36] using a Swin Transformer [39] backbone, to obtain instance segmentation and image features. Please refer to Appendix E for details about the condition embedding functions.

To denoise object poses ρ i , we use a 1-dim. UNet [55] architecture with 8 encoding and decoding blocks with skip connections. Each block consists of a time-conditional ResNet [18] layer, multi-head attention between the per-object condition y i and the pose representation, and our intra-scene attention module (Sec. 3.5) to enable relational information exchange and effectively train a scene prior. We use 8 attention heads, with 64 features per head.

To estimate object shapes σ i from the input view I , we denoise the unordered set of Gaussian G i,j using a Transformer [69] model with 2 encoder layers, 6 decoder layers, and multi-head attention with 4 heads to the object condition information, similar to [30]. The per-Gaussian latent features are denoise with a shape decoder diffusion model, realized as another Transformer model with 6 encoder and decoder layers, which is conditioned on the shape Gaussians.

3.8 Training and Implementation Details

For all diffusion training processes, we uniformly sample time steps t = 1 , ...T, T = 1000 , and use a linear variance schedule with β 1 = 0 . 0001 and β T = 0 . 02 . We implement our model in PyTorch[50] and use the AdamW [41] optimizer with a learning rate of 1 × 10 -4 and β 1 = 0 . 9 , β 2 = 0 . 999 . We train our models on a single RTX3090 with 24GB VRAM for 1000 epochs on Pix3D, for 500 epochs on SUN RGB-D and for 50 epochs of additional joint training using L align .

During inference, we employ DDIM [61] with 100 steps to accelerate sampling speed. For classifierfree guidance [21], we drop the condition y with probability p = 0 . 8 .

4 Experiments

In the following sections, we will demonstrate the advantages of our method and contributions by evaluating it against common 3D scene reconstruction benchmarks.

4.1 Baseline Methods

We compare our method against current state-of-the-art methods for holistic scene understanding: Total3D [48], Im3D [77], and InstPIFu [37]. Total3D [48] directly regresses 3D object poses from image features and uses a mesh deformation and edge-removal approach [49] to reconstruct a shape. Im3D [77] utilizes an implicit shape representation and a graph neural network to refine the pose

predictions. InstPIFu [37] focuses on single-object reconstruction and proposes to query instancealigned features from the input image in their implicit shape decoder to handle occlusion. For scene reconstruction, they rely on the predicted 3D poses of Im3D. We use the official code and checkpoints provided by the authors of these baseline methods and evaluate with ground truth 2D instance segmentation and camera parameters to ensure a fair comparison. We further compare against a retrieval-based method, ROCA [17] in Appendix D.

4.2 Datasets

Following [23, 48, 77], we train and evaluate the performance of our 3D pose estimation on the SUN RGB-D [62] dataset with the official splits. This dataset consists of 10,335 images of indoor scenes (offices, hotel rooms, lobbies, furniture stores, etc.) captured with four different RGB-D cameras. Each image is annotated with 2D and 3D bounding boxes of objects in the scene. During joint training, we use the provided depth maps together with instance masks to compute L align . We train and evaluate the performance of our 3D shape reconstruction on the Pix3D [64] dataset, which contains images of common furniture objects with pixel-aligned 3D shapes from 9 object classes, comprising 10,046 images. We use the train and test splits defined in [37], ensuring that 3D models between the respective splits do not overlap.

4.3 Evaluation Protocol

For quantitative comparison against baseline methods, we follow the evaluation protocol of [48]. For pose estimation, we report the intersection over union of the 3D bounding box (IoU3D) and average precision with an IoU3D threshold of 15% (AP 15 3D ) on the SUN RGB-D dataset [62]. In line with previous works [48, 77], we evaluate with oracle 2D detections but also provide camera parameters to all methods during evaluation. To further assess the alignment of the 3D shapes in the scene, we calculate L align between reconstructed shapes and the instance-segmented ground-truth depth map. For single-view 3D shape reconstruction, we follow evaluate on the Pix3D [64] dataset. We follow [37] and sample 10,000 points on the predicted shape surface, extracted with Marching Cubes [40] at a resolution of 128 3 , and on the ground truth shapes and evaluate Chamfer distance (CD × 10 3 ) and F-score after mesh alignment.

4.4 Comparison to State of the Art

3D Scene Reconstruction. In Fig. 3, we present qualitative comparisons of our approach against state-of-the-art methods for single-view 3D scene reconstruction. The results from Total3D often exhibit intersecting objects and lack global structure. Additionally, their deformation and edgeremoval approach results in 3D shapes with visible artifacts and limited details. While the implicit shape representation of Im3D is more flexible, it often produces incomplete and floating surfaces. In contrast, our diffusion-based reconstruction method, as shown in Tab. 1, learns strong scene priors, resulting in a +0.2 improvement in L align and more coherent 3D arrangements of the objects in the scene (+12.04% AP 15 3D ), as well as high-quality and clean shapes (+13.43% F-Score).

Furthermore, we demonstrate the generalizability of our model to other indoor datasets. We evaluate our approach on individual frames from the ScanNet [11] dataset using 2D instance predictions from Mask2Former without additional fine-tuning. As shown in Fig. 4, our method accurately reconstructs the given input view with matching poses and high-quality 3D geometries.

In Appendix D, we additionally train on ScanNet and compare against ROCA [17]. Due to its retrieval approach, the shapes are complete. However, the resulting quality can limited by the diversity of the shape database, which can lead to suboptimal results, see Fig. 11.

3D Pose Estimation & Scene Arrangement. As shown in Tabs. 1 and 6, our method outperforms all baseline methods by a significant margin in terms of IoU3D and AP 15 3D , i.e ., improving mAP 15 3D by 12.04% over Im3D [77]. Detailed per-class results are provided in Tabs. 6 and 8. Figs. 3 and 7 demonstrate that our approach effectively learns common object arrangements, such as multiple chairs surrounding a table, while ensuring that furniture pieces do not intersect or float in the air. We attribute these improvements to our model's robust scene understanding, which is derived from learning a strong scene prior that accounts for inter-object relationships.

3D Object Reconstruction. In Tab. 1, we quantitatively compare the single-view shape reconstruction performance of our approach against baseline methods on the Pix3D dataset. The results demonstrate that modeling single-view reconstruction as conditional generation over a robust shape prior leads to significant improvements in Chamfer Distance (+9.6%) and F-Score (+13.43%). Detailed per-class results can be found in Tabs. 7 and 9. Fig. 9 illustrates that InstPiFU often reconstructs noisy and incomplete shapes. In contrast, our approach produces clean 3D geometries with fine details, such as thin chair legs and the crease between pillows of a sofa.

In Fig. 5, we show unconditional results by injecting ∅ as a condition (Sec. 3.2), showcasing that our shape prior models detailed and diverse shape modes across several semantic classes. In Fig. 10, we additionally visualize the shape decomposition capabilities resulting from our shape encoding and the scaffolding Gaussian representation.

4.5 Ablations Studies

We conduct a series of detailed ablation studies to verify the effectiveness of our design decisions and contributions. The quantitative results are provided in Tab. 2.

What is the effect of the denoising formulation? To assess the benefits of the denoising diffusion formulation, we construct a 1-step feed-forward regression model that uses the same conditional information as input features and model architecture but regresses the object outputs directly in a single timestep. As shown in Tab. 2, modeling 3D scene reconstruction as a conditional diffusion process, rather than using a feed-forward regression formulation, results in significant improvements of +11 . 08 %AP 15 3D and +0 . 19 L align .

What is the effect of our scene prior modeling? We evaluate the impact of learning a scene prior by modeling the distribution of all objects and their relationships compared to learning the marginal per-object distribution, i.e ., predicting each object individually. As shown in Tab. 2, our joint-object scene prior yields a significant improvement of +9 . 30 %AP 15 3D over per-object prediction. This improvement underscores the importance of learning a robust scene prior that effectively captures inter-object relationships.

What is the effect of joint training? We investigate the benefit of joint training for pose and shape using L align compared to individual training of pose estimation and shape reconstruction. Although our model already learns strong scene and shape priors, Tab. 2 shows that joint training provides additional benefits, resulting in an improvement of +2 . 11 %in AP 15 3D and +0 . 07 in L align .

Input

Total3D

Im3D

Ours

Input

Ours

Input

Ours

4.6 Limitations

While our conditional scene diffusion approach for single-view 3D scene reconstruction demonstrates significant improvements, there are some limitations. First, our method relies on accurate 2D object detection, making it dependent on the performance of 2D perception models. Upcoming state-of-theart 2D detection models [1] can be seamlessly integrated to enhance the performance of our approach. Second, our shape prior, trained on a diverse set of semantic classes using 3D shape supervision, does not generalize to unseen object categories. This can be mitigated by combining our model for known categories with single-object diffusion models that leverage pre-trained text-image generation models for 3D shape synthesis [38] of uncommon shape categories. While accurate 3D scene reconstruction

forms the foundation for subsequent downstream tasks like mixed reality applications, our current model assumes a static scene geometry. Future work could integrate object affordance and articulation into our shape prior [34] to enable more immersive human-scene interactions.

Broader Impact We do not anticipate any societal consequences or negative ethical implications arising from our work. Our approach advances the holistic understanding of 2D perception and 3D modeling, benefiting various research areas.

5 Conclusion

In this paper, we present a novel diffusion-based approach for coherent 3D scene reconstructions from a single RGB image. Our method combines a simple yet powerful denoising formulation with a robust generative scene prior that learns inter-object relationships by exchanging relational information among all scene objects. To address the issue of missing ground-truth annotations in publicly available 3D datasets, we introduce a surface alignment loss L align to jointly train shape and pose, effectively leveraging our shape representation. Our approach significantly enhances 3D scene understanding, outperforming current state-of-the-art methods across various benchmarks, with +12.04% AP 15 3D on SUN RGB-D and +13.43% F-Score on Pix3D. Extensive experiments demonstrate that our contributions - 3D scene reconstruction as a conditional diffusion process, scene prior modeling, and joint shape-pose training enabled by L align - collectively contribute to the overall performance gain. Additionally, we show that our model supports unconditional synthesis and generalizes well to other indoor datasets without further fine-tuning. We believe these advancements lay a solid foundation for future progress in holistic 3D scene understanding and open up exciting applications in mixed reality, content creation, and robotics.

References

Coco leaderboard. URL https://cocodataset.org/#detection-leaderboard .

A. Alliegro, Y. Siddiqui, T. Tommasi, and M. Nießner. Polydiff: Generating 3d polygonal meshes with diffusion models. arXiv preprint arXiv:2312.11417 , 2023.

A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nießner. Scan2cad: Learning cad model alignment in rgb-d scans. In CVPR , 2019.

A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.

Y.-C. Cheng, H.-Y. Lee, S. Tulyakov, A. G. Schwing, and L.-Y. Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4456-4465, 2023.

G. Chou, Y. Bahat, and F. Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. 2023.

C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Vision-European Conference on Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 628-644. Springer, 2016.

T. Chu, P. Zhang, Q. Liu, and J. Wang. Buol: A bottom-up framework with occupancy-aware lifting for panoptic 3d scene reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4937-4946, 2023.

M. Dahnert, J. Hou, M. Nießner, and A. Dai. Panoptic 3d scene reconstruction from a single rgb image. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR , 2017.

M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR , 2023.

Y. Du, Z. Liu, H. Basevi, A. Leonardis, B. Freeman, J. Tenenbaum, and J. Wu. Learning to exploit stability for 3d scene parsing. In Conference on Neural Information Processing Systems , 2018.

H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 605-613, 2017.

G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2018.

C. Gümeli, A. Dai, and M. Nießner. Roca: Robust cad model retrieval and alignment from a single image. 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Q. He, D. Zhou, B. Wan, and X. He. Single image 3d object estimation with primitive graph networks. In Proceedings of the 29th ACM International Conference on Multimedia , pages 2353-2361, 2021.

A. Hertz, O. Perel, R. Giryes, O. Sorkine-Hornung, and D. Cohen-Or. Spaghetti: Editing implicit shapes through part aware generation. ACM Transactions on Graphics (TOG) , 41(4):1-20, 2022.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

S. Huang, S. Qi, Y. Xiao, Y. Zhu, Y. N. Wu, and S.-C. Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation. In Conference on Neural Information Processing Systems , 2018.

S. Huang, S. Qi, Y. Zhu, Y. Xiao, Y. Xu, and S.-C. Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In European Conference on Computer Vision , 2018.

K.-H. Hui, R. Li, J. Hu, and C.-W. Fu. Neural wavelet-domain diffusion for 3d shape generation. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022.

H. Izadinia, Q. Shan, and S. M. Seitz. Im2cad. In CVPR , 2017.

W. Jang and L. Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12949-12958, 2021.

T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

S. W. Kim, B. Brown, K. Yin, K. Kreis, K. Schwarz, D. Li, R. Rombach, A. Torralba, and S. Fidler. Neuralfield-ldm: Scene generation with hierarchical latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

J. Koo, S. Yoo, M. H. Nguyen, and M. Sung. Salad: Part-level latent diffusion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14441-14451, 2023.

N. Kulkarni, I. Misra, S. Tulsiani, and A. Gupta. 3d-relnet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2212-2221, 2019.

W. Kuo, A. Angelova, T.-y. Lin, and A. Dai. Mask2cad: 3d shape prediction by learning to segment and retrieve. In Proceedings of the European Conference on Computer Vision (European Conference on Computer Vision) , 2020.

W. Kuo, A. Angelova, T.-Y. Lin, and A. Dai. Patch2cad: Patchwise embedding learning for in-the-wild shape retrieval from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12589-12599, 2021.

J. Lei, C. Deng, W. B. Shen, L. J. Guibas, and K. Daniilidis. Nap: Neural 3d articulated object prior. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 31878-31894. Curran Associates, Inc., 2023.

L. Li, S. Khan, and N. Barnes. Silhouette-assisted 3d object instance reconstruction from a cluttered scene. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (Proceedings of the IEEE/CVF International Conference on Computer VisionW) , pages 2080-2088, 2019. doi: 10.1109/ ProceedingsoftheIEEE/CVFInternationalConferenceonComputerVisionW.2019.00263.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-European Conference on Computer Vision 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

H. Liu, Y. Zheng, G. Chen, S. Cui, and X. Han. Towards high-fidelity single-view holistic reconstruction of indoor scenes. In European Conference on Computer Vision , 2022.

R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Z. Liu, Y. Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Trans. Gr. , 21(4):163-169, 1987.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

P. Mandikal, N. KL, and R. Venkatesh Babu. 3d-psrnet: Part segmented 3d point cloud reconstruction from a single image. In Proceedings of the European Conference on Computer Vision (European Conference on Computer Vision) Workshops , pages 0-0, 2018.

L. Melas-Kyriazi, C. Rupprecht, and A. Vedaldi. Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12923-12932, 2023.

N. Müller, Y. Siddiqui, L. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328-4338, 2023.

C. Nash, Y. Ganin, S. A. Eslami, and P. Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning , pages 7220-7229. PMLR, 2020.

P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision , 2012.

Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. J. Zhang. Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In CVPR , 2020.

J. Pan, X. Han, W. Chen, J. Tang, and K. Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems , 2019.

B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR , 2023.

S. Popov, P. Bauszat, and V. Ferrari. Corenet: Coherent 3d scene reconstruction from a single rgb image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 366-383. Springer, 2020.

X. Ren, J. Huang, X. Zeng, K. Museth, S. Fidler, and F. Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

L. Roberts. Machine perception of threedimensional solids. PhD thesis, Massachusetts Institute of Technology , 1963.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

E. Sella, G. Fiebelman, P. Hedman, and H. Averbuch-Elor. Vox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 430-440, 2023.

J. R. Shue, E. R. Chan, R. Po, Z. Ankner, J. Wu, and G. Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20875-20886, 2023.

Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V . Rosov, A. Dai, and M. Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE , 2024.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR , 2015.

S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from a single depth image. arXiv preprint arXiv:1611.08974 , 2016.

X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In CVPR , 2018.

S. Szymanowicz, C. Rupprecht, and A. Vedaldi. Viewset diffusion: (0-)image-conditioned 3d generative models from 2d data. International Conference on Computer Vision , 2023.

J. Tang, X. Han, J. Pan, K. Jia, and X. Tong. A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages 4541-4550, 2019.

J. Tang, Y. Nie, L. Markhasin, A. Dai, J. Thies, and M. Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207 , 2023.

S. Tulsiani, S. Gupta, D. F. Fouhey, A. A. Efros, and J. Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 302-310, 2018.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision , 2018.

J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Advances in Neural Information Processing Systems , pages 82-90, 2016.

H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2690-2698, 2019.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR , 2021.

X. Zeng, A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, and K. Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978 , 2022.

B. Zhang, M. Nießner, and P. Wonka. 3DILG: Irregular latent grids for 3d generative modeling. In Thirty-Sixth Conference on Neural Information Processing Systems , 2022.

B. Zhang, J. Tang, M. Niessner, and P. Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445 , 2023.

C. Zhang, Z. Cui, Y. Zhang, B. Zeng, M. Pollefeys, and S. Liu. Holistic 3d scene understanding from a single image with implicit representation. In CVPR , 2021.

X. Zhang, Z. Chen, F. Wei, and Z. Tu. Uni-3d: A universal model for panoptic 3d scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (Proceedings of the IEEE/CVF International Conference on Computer Vision) , pages 9256-9266, October 2023.

X. Zheng, Y. Liu, P. Wang, and X. Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum , volume 41, pages 52-63. Wiley Online Library, 2022.

L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826-5835, 2021.

A Appendix

In the following, we show more qualitative results for scene reconstruction on SUN RGB-D [62] (Appendix B) and object reconstruction on Pix3D [64]. We provide detailed quantitative per-class comparisons supplementing the tables in the main paper (Appendix C). We additionally compare against a retrieval baseline on the ScanNet [11] dataset in Appendix D. Finally, we provide additional details on the architecture of our diffusion model in (Appendix E).

For a comprehensive overview of our approach and results, we encourage the reader to watch the supplemental video.

B Additional Qualitative Results

Scene Reconstruction In Fig. 6, we show additional qualitative results of our method on test frames from SUN RGB-D. Despite strong occlusions and challenging viewing angles, our model predicts accurate scene reconstructions. Our generative scene prior learns common scene patterns, such as parallel object placements between the table and sofa or a bed and neighboring nightstands. In Fig. 8, we also demonstrate that our robust conditional scene prior can recover clean and matching shape reconstruction even for heavily occluded objects, e.g ., a chair for which only the back seat is barely visible.

Input

Ours

Input

Ours

Object Reconstruction & Unconditional Synthesis In Fig. 9, we show a qualitative comparison of single-view 3D object reconstruction on the Pix3D dataset. Unlike InstPIFu, which often produces noisy and incomplete surfaces, our image-condition diffusion model reconstructs clean and highfidelity objects. Such a visual quality allows these reconstructions to be integrated into e.g ., mixed reality applications.

To probe the learned shape prior and investigate its shape synthesis capabilities, we input the 0condition ∅ instead of extracted image features to our model. As shown in Fig. 5, our model learns a high-quality shape prior with fine details across various semantic classes.

Input

Total3D

Im3D

Ours

Ground Truth

日

口□

C Additional Quantitative Results

Scene Reconstruction In Tab. 4, we show detailed comparisons of our approach against baseline methods, Total3D [48] and Im3D [77], on the 10 most common classes of SUN RGB-D. Our approach consistently outperforms all baseline methods on all classes except the 'bed' class. We attribute this exception to the fact that beds are often only partially visible in the input view due to their spatial extent, which introduces higher variability. In contrast, Im3D employs a series of geometric losses and regularization terms, which seems to help in extreme amodal cases at the cost of additional loss balancing. Nevertheless, our method achieves a significant overall improvement of 12.04% in AP 15 3D on these 10 classes, with particularly notable gains for 'dressers' ( +26 . 03 %), 'chairs' ( +21 . 91 %) and 'cabinets' ( +19 . 37 %), showcasing the effect of our robust scene prior.

Tabs. 6 and 8 show the per-class comparisons and ablation studies on all 37 NYU classes in terms of IoU3D and mAP 15 3D . Our approach improves compared to Im3D by a +7 . 57 %increase in mAP 15 3D and +4 . 56 %increase in class-mean IoU3D across all 37 classes. The ablation results highlight the importance of our diffusion formulation ( +7 . 67 %mAP 15 3D ), scene prior modeling ( +7 . 11 %mAP 15 3D ), and joint training using the surface alignment loss L align ( +0 . 72 mAP 15 3D ).

Object Reconstruction For single-view object reconstruction, we evaluate Chamfer Distance and F-Score on Pix3D and show per-class comparisons in Tabs. 7 and 9. Our image-conditional shape prior leads to significant improvements, +9.6% in Chamfer Distance and +13.43 in F-Score, while outperforming InstPIFu in most categories, except sofas and wardrobes in F-Score.

Room Layout [48, 77] also predict the room bounding box with a separate network head. We study, how our model can also predict the room layout. For that we include the room bounding

box pose as part of the object poses during the diffusion process. We follow the room layout parameterization of [48, 77] and model the 3D room center directly instead of decomposing it as 2D offset & distance, which is done for the objects. In Tab. 3, we demonstrate that by denoising the pose of room layout, we outperform the regression-based methods.

D Comparison to shape retrieval baseline on ScanNet

We compare with a shape retrieval baseline, namely ROCA [17]. Since ROCA requires full groundtruth supervision during training, we adopt their setup and train our model on the same 25,000 frames from the ScanNet [11] dataset with pose annotations derived from Scan2CAD [3], as well as the same CAD pool from ShapeNet [4]. We additionally adopt their full 9-DoF pose parameterization by predicting all 3 rotation angles. Following ROCA, we quantitatively evaluate the Alignment Accuracy in Tab. 5. Please refer to [3, 17] for the details of the evaluation. In Fig. 11, we can see that ROCA retrieves clean and complete shapes by definition. However, due to its limited shape database, it cannot capture all shape modes accurately, leading to shape mismatches. Our reconstruction-based approach instead can recover faithful shape results while simultaneously predicting a coherent object arrangement.

E Architecture Details

Object Pose Parameterization: Normalization To ensure a reasonable signal-noise ratio [28] among the object pose parameters, we normalize the parameters to [ -1 , 1] by dividing them by its max value and shift the range using a parameter-specific µ value. For this, we calculate the min-max ranges of all pose parameters, i.e ., rotation θ , 3D scale s , and projected distance d , within the train

set of SUN RGB-D. The 2D offsets to the 2D bounding box center are normalized by the image dimensions.

During training, the loss is computed on the un-normalized parameter ranges. After inference and for evaluation, we un-normalize each parameter according to its original range.

Surface Alignment Loss: Point Sample Transformation During training, for each object o i , we use the predicted shape ˆ σ i to estimate its scaffolding Gaussians ˆ G j . From each 3D Gaussian distribution, we directly draw 3D point samples p ( j,l ) ∼ N ( µ j , Σ j ) . This shape point cloud P i approximates the shape. With the predicted and un-normalized object pose ˆ ρ i , we define a 3D rigid transformation R 4 × 4 and transform the shape point cloud P i to the camera coordinate system. We use this transformed shape pointcloud P cam i and the instance-segmented ground-truth depth map from

Input

CameraView

SideView

#1

#2

SUN RGB-D as the partial target pointcloud to measure the 1-sided Chamfer distance and to compute the surface alignment loss L align .

Scene Prior Modeling: Inter-Object Relationships via Intra-Scene Attention We use the multi-head attention mechanism [69] between the scene objects to allow them to attend to each other, effectively learning their inter-object relationships and the scene context. Specifically, given an unordered set S = [ o 1 , o 2 , ..., o n ] , o i ∈ R n per-object n -dimensional feature vectors, projection layers ( W Q , W K and W V ) and features Q = S × W Q , K = S × W K and V = S × W V after projection. we define the intra-scene attention as:

Condition: Embedding Functions After cropping the 2D image feature patch R W × H × C from the frozen image backone Θ I, we apply adaptive average pooling to resize the per-object feature patches to a common 2D size leading to resized per-object feature crop of 8 × 8 and C = 256 . This feature crop is further embedded using a small 2D CNN Θ feat with 3 blocks of convolutional layers with 512 features, group norm, and leaky ReLU activation. The embedded feature crop is reshaped to a 4096 -dim vector.

Θ box is implemented as sinusoidal position encoding with 10 frequencies. This function is applied on a 2D bounding box, represented by the top-left and bottom-right corners, leading to an 84 -dim vector per object. For Θ cls, we use a simple 1-hot encoding to embed the semantic class information. The final per-object condition information is the concatenation, resulting in a 4127 -dim vector for each object.

Reimplementation of SPAGHETTI [20] Since the official code of SPAGHETTI does not include the training code and only provides checkpoints for two different shape classes (chairs,

Input

InstPIFu

Ours

Ground Truth

airplanes), we re-implement the training procedure, loss function, and disentanglement loss following the description in the papers to train the full shape prior over all relevant shape categories. Random geometric augmentations are essential during training to achieve self-supervised disentanglement into extrinsic and intrinsic shape properties. We apply full 360-degree random rotations, uniform scale augmentation between 0.7 and 1.3, and translation jitter of ∓ 0.3 on the disentangled extrinsic and target pointcloud. Further, we do not utilize the symmetry options of the original implementation.

InputImage

ROCA

Ours

Image crop

4096-dim condition

Input

ndano

XEIR3

Output

MLP

Transformer

Decoder

MLP

[Decoder Block] x6

Occupancy

dim=512

Decoder

MLP

cond-dim = 1024

Noisy Extrinsics

Denoised Extrinsics

G × N(0; 1)16

G×R16

3DShape

Extrinsic Diffusion

G×16-dimcondition

Input

ndano

Input

Transformer

Encoder

Transformer

Decoder

ndano

XEIR3

Transformer

Transformer

Noisy Intrinsics

[Encoder Block]x6

[Decoder Block] x6

Denoised Intrinsics

Extrinsics

Encoder

Decoder

Inside/

G × N(0:; 1)256

dim=512

cond-dim=1024

dim=512

G×IR256

[Encoder Block]x4

dim=512

[Decoder Block]x6

dim=512

MLP

Outside

Intrinsics

cond-dim = 1024

Intrinsic Diffusion

Occupancy Decoder

TransformerEncoderBlock

TransformerDecoderBlock

MLP

MLP

Weight

Self-Attn

MLP

LayerNorm

Self-Attn

Cross-Attn

MLP

LayerNorm

Linear

LayerNorm

Linear

LayerNorm

x6

Linear

Norm

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide experimental results demonstrating the performance improvements by following the evaluation protocol of common 3D scene reconstruction benchmarks (Tab. 1). We further ablate the impact of the individual contributions - denoising formulation, scene prior modeling, and surface alignment loss in Tab. 2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our approach in Sec. 4.6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work does not introduce new theorems. We provide an empirical evaluation of our model through a series of experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed description of the architecture in Sec. 3.7, Appendix E and implementation instructions in Sec. 3.8.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code after cleaning up and documenting it for easy usage. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give detailed explanations of training and test details in Secs. 3.7 and 3.8 and Appendix E. We use common data splits for training and testing (Sec. 4.2) and follow the evaluation protocols of [48, 37] (Sec. 4.3).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Given limited computational resources, we are not able to train multiple models using different initializations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Sec. 3.8, we explain the setup (hardware, training duration) to train our models. Given the rich design space of the challenging task of single-view 3D scene reconstruction, we conducted a wide range of experiments to reach the performance of the presented work. However, we did not keep track of the total compute of all individual runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Sec. 4.6, we discuss why we do not anticipate any negative impacts of our work on single-view 3D scene reconstruction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our model only relies on a 2D instance segmentation model, which was pre-trained on COCO.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For training and testing, we use multiple publically available datasets, namely SUN RGB-D [62], Pix3D [64], and ScanNet [11], which were properly explained and cited in the main paper (Sec. 4.2).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Use the official train/test splits of publically available datasets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not perform research with human subjects, nor did the data pre-processing and experiments require crowdsourcing tasks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not perform research with human subjects, nor did the data pre-processing and experiments require crowdsourcing tasks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe task is to perform 3D scene reconstruction from a single RGB views, such that they can output both scene object poses and geometries. Prior work cannot jointly output both poses and geometries, one design choice that is validated by this work to be effective. Method-wise, extending on SALAD, this work introduces a diffusion model that learns a generative scene prior that can capture contextual information across scene elements. To achieve joint learning, they introduce a alignment loss such that they only need to single view supervision. The author conducts extensive experiments to justify their design choices.\n\nSTRENGTHS:\n1. The introduction of the surface alignment loss (Sec 3.6) provides a nice solution to jointly improve object poses and geometry estimation. This kind of supervision only needs image level supervision (no need for 3D supervision). \n\n2. The experiments on comparison and ablation is informative, which reasons well the key design decision made in this paper. \n\n3. The qualitative demo is multifaceted, which makes it easier to evaluate their pose and geometry quality.\n\nWEAKNESSES:\n1. Minor concern is that the paper over-claim a bit on the contribution. Despite phrasing as 3D scene reconstruction (as is mentioned in teaser, abstract and intro) from single view, the proposed framework in the method actually only reconstruct poses and geometries of foreground objects in the image without the background part, e.g., ground or wall. \n\n2. Global shape prior cannot always fit the observed object perfectly, with different local details, e.g., Fig.3 shelf. This could accumulate error in the downstream alignment. \n\n3. Furniture kinds of 3D objects, e.g., those in ShapeNet, are simple, which usually is easier to get good shape reconstruction. How would the proposed framework work on more intricate 3D objects, e.g., plants, trees?\n\nQUESTIONS:\n1. Could you clarify on how the proposed framework resolve with scale ambiguity given the single view setup?\n\n2. Writing needs more polish. For example, Equation 1 seems incorrect (feel free to clarify in the rebuttal). Based on the description in Line 121, RGB image I seems to be given, which indicates it is one item to be conditioned on. And in Line 150, missing parenthesis for (See Sec. 4.2\n\n3. Could you clarify on how the gradient being back-propagated through point sampling (Line 190)? \"We directly sample m = 1000 points p(j,l) ∼ N (μj , Σj ) per Gaussian Gˆi,j resulting in a shape point cloud Pi = {p(j,l)|j ∈ {1,...,g},l ∈ {1,...,m}}.\" In SALAD, it has another cascaded diffusion Transformer that condition on extrinsic and intrinsic parameters to get the part-level point cloud from Transformer. But I am not sure if the process is the same in this paper. \n\n4. Could you clarify how the texture of the 3D objects are learned? The qualitative results show objects with color but there is no supervision on the color.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for 3D scene reconstruction from single images based on diffusion models. This framework jointly predicts 3D object poses and shapes with seperate diffusion models and captures global scene context with a standard mutli-head attention module. To train this model, the paper proposes a surface alignment loss based on one-sided Chamfer Distance, which makes their method able to utilize depth maps as supervision. The experimental results on Sun RGB-D, ScanNet, and Pix3D and numerical ablation studies demonstrate the efficacy of proposed method.\n\nSTRENGTHS:\nThis paper shows its strengths from several aspects:\n1. It proposes an overall solid method for 3D scene reconstruction from single images and deals with the ill-posed nature of this problem with powerful generative prior from diffusion models. \n2. The proposed alignment loss makes the method able to utilize partial supervision from depth maps for model training.\n3. This paper conducted comprehensive experiments in both scene and object reconstruction and achieve good results comparing to previous methods.\n4. The presentation and writing are overall good and clear.\n\nWEAKNESSES:\nMy concerns are mainly about the description of the method:\n1. The model architecture is not clear enough in Section 3.7. It seems that each object pose, each object shape, and each Gaussian, are denoised using an independent diffusion model? Where is ISA added and how is it integrated with other modules? Are the inputs of ISA from Equation (7)? \n2. In Figure 1, are the noisy shape Gaussians as in shape with guassian noise? Or are they initial estimation directly from image features? It's a bit confusing.\n3. Are the diffusion models for shape and pose trained with full 3D supervision before joint training with alignment loss? If yes, what if there are only ground truth depth maps in training data?\n\nBesides, the overall idea of 3D generation from features as in Equation (6) combined with global context via attention has been explored in previous related works, such as part-based 3D object reconstruction, e.g., He, Qian, et al. \"Single Image 3D Object Estimation with Primitive Graph Networks.\" Proceedings of the 29th ACM International Conference on Multimedia. 2021. This should be included in related works.\n\nQUESTIONS:\n1. According to the alignment loss, the predicted Gaussians should be converged to centered on the surface of 3D objects? Otheriwse, the direct sampling won't be around the surface?\n2. How do you deal with varying number of objects within a single image? How do you deal with false positives and false negatives from detection?\n3. Have you tried other normalization methods besides Equation (12-14)? Does it affect the learning efficiency?\n4. Does the disentanglement of pose and shape helps with learning convergence?\n5. How is the proposed method comparing to other SOTAs in single object reconstruction? Can this method be applied to part-level object reconstruction?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and I do not have more to add here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a diffusion-based method for generating coherent 3D scenes from a single input image. They use a diffusion model to jointly denoise the 3D poses and geometries of all objects. To address the incomplete ground truth of existing datasets, they also introduce a surface alignment loss to better learn the 3D geometry. Experimental results show that the proposed approach significantly outperforms state-of-the-art methods.\n\nSTRENGTHS:\n- I like the formulation of this method. I think anisotropic 3D Gaussians are a good representation for 3D reconstruction. We could treat the 3D reconstruction in a coarse-to-fine manner and use the 3D Gaussians as the 3D proxy to guide the fine reconstruction.\n\n- The results look promising. With partial ground truth for supervision, the proposed method learns complete geometry from a single RGB image with high occlusion.\n\nWEAKNESSES:\n- I am unclear on how to learn the shape information:\n\n(i) For the shape code sigma_i (16 Gaussians x 16 dimensions per Gaussian), do you have the ground truth for it? If not, how do you apply the shape diffusion loss mentioned in Equation 10?\n\n(ii) To decode the shape code from Gaussians to an occupancy grid, you apply a shape decoder diffusion model (mentioned in Line 213). How do you train this model? Do you include any details about this model?\n\n- I am interested in how the proposed method learns the intermediate scene representation (i.e., anisotropic 3D Gaussians). You use the surface alignment loss as supervision. Are there other indirect supervisions? How do you ensure that the Gaussians do not intersect each other or become trivial (e.g., the same as each other)?\n\n- If I understand correctly, you add inter-object attentions to the intra-scene attention to allow the pose and shape model to run in parallel. If so, this operator is quite common.\n\n- I find the author may miss the citation and discussion with an important literature [a].\n\n[a] CoReNet: Coherent 3D scene reconstruction from a single RGB image. ECCV 2020.\n\nQUESTIONS:\n- It is suggested to add notation for cls in Equation 6.\n\n- It is suggested to add some figures for the ablation study.\n\n- I find [b] to be quite related to this work. It is suggested to discuss this paper in the related work section. Combining some 3D architecture with the proposed method could be a nice extension.\n\n[b] XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.\n\nLIMITATIONS:\nI think the authors properly discuss the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to reconstruct the 3D surfaces of multiple objects from a single RGB image. Given an instance-segmented RGB image as input, the poses and shapes of objects are jointly predicted and denoised using a diffusion model conditioned on the input image. A one-sided chamfer distance (from available ground truth surface point to sampled points in camera space) is used to provide supervision on incomplete surfaces. The output is geometrically evaluated against Total3D (3D detection-based composition), Im3D (implicit surface), and InstPIFu (pixel-aligned implicit surface) and shows improved scene coherence and shape quality.\n\nSTRENGTHS:\n+ Efficient training\n+ Appropriate use of category-specific object shape priors\n+ Clean ablation results\n\nWEAKNESSES:\nIn order for the simple loss (one-sided chamfer distance) to work well, you need strong category-specific priors on generated shapes; it requires the estimated shape gaussians to be stable in the first place. The joint learning of poses is also inherently category-specific and might not generalize well to unseen or uncommon object categories. While this limitation is mentioned in the paper, I think this is an inherent limitation that is not easy to mitigate.\n\nLimits of one-sided chamfer distance: In qualitative results containing office chair images for example, I noticed that many of them have extra leg parts hallucinated. I think this is an example of something that the one-sided chamfer loss won't be able to fix since the error isn't reflected in the loss. I'm surprised the method worked well despite this; this isn't necessarily a weakness, but tells us something about the problem we're trying to solve and evaluation methods (maybe \"alignment is all you need\"?). I think a retrieval-based baseline (with alignment) would have been interesting for the reasons mentioned.\n\nI tend to think the improvements are mostly due to architectural advancements such as attention models (that can process arbitrary unordered variable-length objects) and better off-the-shelf shape models available. The paper appropriately applied those techniques and that is a good thing, but maybe it does not fundamentally improve our understanding of the problem. Perhaps this isn't a fair criticism considering the limitations and quality of other papers accepted in similar venues. I currently feel borderline about the paper. The holistic nature of the problem is difficult enough and the paper proposes one simple solution that I think can be used by other researchers still.\n\nQUESTIONS:\n(please see weaknesses section)\n\nLIMITATIONS:\nI think they are addressed well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}