Coherent 3D Scene Diffusion From a Single RGB Image
Authors: Manuel Dahnert
Angela Dai
Norman Müller
Matthias Nießner
Venue: NeurIPS 2024 poster (2024)
Keywords: Single RGB Image 3D Scene Reconstruction
Diffusion Models
Scene Understanding
3D Scene Prior
Primary area: machine_vision
TL;DR: A novel diffusion-based 3D scene diffusion model for 3D scene reconstruction from a single RGB image

Abstract: We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. 
Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene.

Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture scene context and by allowing the model to learn inter-object relationships throughout the diffusion process.

We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions.

By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04\% improvement in AP3D on SUN RGB-D and a 13.43\% increase in F-Score on Pix3D.

Coherent 3D Scene Diffusion From a Single RGB Image
Manuel Dahnert'
Angela Dai
Norman Müller?
Matthias Nießner
'Technical University of Munich, Germany
Meta Reality Labs Zurich, Switzerland
Abstract
We present novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method utilizes an image-conditioned 3D scene diffusion model simultaneously denoise the 3D poses and geometries of all objects within the scene. Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process. We further propose efficient surface alignment loss to facilitate training even in the absence of full ground-trute annotation, which is common in publicly available datasets. This loss leverages expressive shape representation, which enables direct point sampling from intermediate shape predictions. By framing the task of single RGB image 3D scene reconstruction as conditional diffusion process, our approach surpasses current state-of-the-ae methods, achieving a 12.04% improvement in AP on SUN RGB-D and 13.43% increase F-Score on Pix3D.
Introduction
3D Scene Diffusion
RSS
30 Recoistucfoo
Kols Shape Gaurclans
Intermediatt Scane Regresentatinn
Junns Pase & Sloppe
Shace Decodio
Linisies
Figure 1: Given single RGB image of an indoor scene, model reconstructs the 3D scene by jointly estimating object arrangements and shapes in a globally consistent manner. Our novel diffusion-based 3D scene reconstruction approach achieves highly accurate predictions by utilizing a novel generative scene prior that captures scene context and inter-object relationships, and by employing an efficient surface alignment loss formulation for joint pose- and shape-synthesis.
Holistic 3D scene understanding crucial for various fields and lays the foundation for many downstream tasks in robotics, 3D content creation, and mixed reality. It bridges the gap between 2D perception and 3D understanding. Despite impressive advancements in 2D perception and 3D reconstruction of individual objects [56]
S 12 38), 3D scene reconstruction from a single RGB observation remains a challenging problem due its ill pol-po nature, heavy occlusions, and the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

complex multi--ojject arrangements found in real-world environments. While previous works [1]] have shown promising results. they often recover 3D shapes independently and thus do not leverage the scene context nor inter-object relationships. This leads to unrealistic and intersecting object arrangements. Additionally, common feed-forward reconstruction methods 48 77 37 struggle with heavy occlusions and weak shape priors, resulting in noisy or incomplete 3D shapes, which hinders immersion and hence limits the applicability in downstream tasks. To address these challenges and to advance 3D scene understanding, we propose novel generative approach for coherent 3D scene reconstruction from a single RGB image. Specifically, we introduce a new diffusion model that learns generative scene prior capturing the relationships between objects in terms of arrangement and shapes. When conditioned on a single image, this model simultaneously reconstructs poses and 3D geometries of all scene objects. By framing the reconstruction task as a conditional synthesis process, we achieve significantly more accurate object poses and sharper geometries. Publicly available 3D datasets
32 33
[47 62 typically only provide partial ground-truth annotations, which complicates joint training of shape and pose. To overcome this, we propose novel and efficient surface alignment loss formulation Lalign that enables joint training of shape and pose even under the lack of full ground-truth supervision. Unlike previous methods [48] (77) that involve costly shape decoding and point sampling on the reconstructed surface, our approach employs an expressive intermediate shape representation that enables direct point sampling from the conditional shape prior. This provides additional supervision and results in more globally consistent 3D scene reconstructions. Our method not only outperforms current state-of-the- the-art methods by 12.04% in API on SUN RGB-D [6] and by 13.43% in F-Score on Pix3D 664 but also generalizes to other indoor datasets without further fine-tuning. In summary, our contributions include:
A novel ieffuusion-baed 3D scene reconstruction approach that jointly predicts poses and shapes of all visible objects within a scene.
A novel way for modeling a generative scene prior by conditioning on all scene objects simultancously to capture scene context and inter-object relationships.
An efficient surface aligmment loss formulation Lalign that leverages an expressive intermedi- ate shape representation for additional supervision, even in the absence of full ground-truth annotation.
2 Related Works
The task of 3D scene reconstruction from a single view combines the fundamental domains of 2D perception and 3D modeling into unified challenge of holistic 3D understanding. Given the multi-faceted nature of the task, we are providing a comprehensive overview of the relevant research directions and contextualizing our contributions.
2.1 Single- Single-View 3D Reconstruction
Since the foundational work by Roberts [54], numerous methods have been developed to learn cues for deriving 3D object structures, thereby bridging the gap between 2D perception and the 3D world. These methods typically involve an image encoder network that processes the input image of single object, capturing its features. The extracted features are either correlated with an encoded shape database retrieve a suitable shape [32 17. or used by a 3D decoder to reconstruct the object in a specific 3D representation. such as voxel grids [8, (2], point clouds 44 4 meshes [70] 66), or neural fields [73 27 19 uses a message- paaggggggsinngggg graph network between geometric primitves to reason about the structure of the shape.
Object Reconstruction.
Scene Reconstruction. Early works formulated single-view scene reconstruction as 3D scene completion from given or estimated depth information 63 10
78 in volumetric grid. While these methods have produced promising results, their representational power to model fine details is limited by the spatial resolution of the 3D grid. Multi-object reconstruction and scene parsing methods [2) or CAD models (26 24], while
represented objects using primitives [3 23 voxel grids (68 35
also considering the relation between the objects [31] The approach presented by Nie et al. 48 is particularly relevant, proposing a holistic method for joint pose and shape estimation from a single image. Zhang et al. [77 extended this idea by incorporating implicit shape representation and

---

an additional pose refinement using a graph neural network. Although these methods provided significant advances in holistic scene understanding, they struggled with accurate pose estimation and produced noisy scene objects, leading to intersecting or incomplete objects. In contrast to these previous works, we are proposing a generative method to obtain a strong scene prior and formulate the reconstruction task as conditional synthesis task. This allows for more robust reconstruction that is less prone to object insections or implausible object geometries.
2.2 3D Diffusion Models
In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as a versatile class of generative models, demonstrating impressive results in image and video generation. Unlike other classes of generative models such as auto-regressive models [46] 75, 59 Generative Adversarial Networks (GANs) 71 79] and Variational Autoencoders (VAEs), diffusion models iteratively reverse a Markovian noising process. This method ensures stable training and has the ability to capture diverse modes while producing detailed outputs. Several approaches have utilized diffusion models to learn the distribution of individual 3D shapes using various 3D representations, including volumetric
grids 66 7 255, point clouds 42 [4], meshes (2). implicit functions [30], neural fields 45
58 29]
53 propose a hierarchical voxel diffusion model, which is capable of modelling large-scals scale and fine-detailed geometry. While these methods can synthesize high-quality 3D shapes, they typically focus on single objects in canonical space. In contrast, we are proposing a diffusion-basee approach that addresses the more challenging problem of multi-object scene reconstruction, encompassing accurate pose estimations and an understanding of inter-object relationships.
or hybrid representations 80 76..
Conditional Diffusion for 3D Reconstruction. Recent works also use diffusion models for single-view object reconstruction (6/7]44, For instance, [65] learns the shape distribution of single category by denoising a set of 2D images for each object, while [44] projects image features onto noisy point clouds during the diffusion process ensure geometric plausibility. Recently, several works proposed to leverage multi-view consistency within pre-trained text-conditional 2D image diffusion models to reconstruct individual 3D objects [88 5T 57]. Similar to our work, Tang er al. [67] use diffusion model to learn scene priors from synthetic data, showing unconditional scene synthesis of single room type and text-conditional generation. However, their approach does not support mmage-based scene reconstruction. Furthermore, it depends on clean synthetic data, which provides full 3D ground truth supervision and CAD model retrieval, thereby limiting shape diversity. While these existing methods have shown promising results on single objects or synthetic scenes, our approach targets real-world scenes. By framing the reconstruction task as a conditional generation process, our scene prior accurately delivers poses and shapes of multiple objects, even in the presence of strong occlusions, significant clutter, and challenging lighting conditions.
3 Method
3.1
Overview
Our method takes single RGB image of an indoor scene as input and generates globally consistent 3D scene reconstruction that matches the input image. To this end, we are framing the reconstruc- tion task conditional generation problem using diffusion model conditioned on the input view (Sec. 3.2). which simultaneously predicts the poses (Sec. 3.3) and shapes (Sec. 3..4 of all objects in the scene. Given the ill-posed nature of single-view reconstruction, such probabilistic formulation is particularly well-suited for this task. To ensure accurate reconstructions and to learn a strong scene prior, we model inter-object relationships within the scene using an intra-scene attention module 3Scc.3.5. Additionally, recognizing the incomplete ground truth in many 3D indoor scene datasets, we introduce Joss formulation for joint shape and pose training, which enables training under only partially available supervision (Sec. 3.6). An overview of our approach is illustrated in Fig. In the following sections, we describe each individual contribution in more detail.
3.2 Conditional 3D Scene Diffusson
We frame the scene reconstruction task as a conditional generation process via diffusion formula- tion (22]. Given an instance-segmentt RGB image containing a variable number of 2D objects 6,
3

---

Scene Prior
Surface Alignment Loss
Anisotropic Shape
Point Samples
Point Samples
1-Sided Chamfer Distance
Canonical
space
Object Relationships
Figure 2: Scene Prior and Surface Aligmment Loss Overview. (Left) We propose novel way to model scene priors (Sec. (35) by modeling the scene context and the relationships between all objects during the denoising process. (Right) For additional supervision and joint training, we use surface alignment loss (Sec. (36)) between given ground truth depth map and point samples directly drawn from the intermediate shape representation o; and transformed to camera space with the predicted object pose Pú.
for E {1, (1,..., n, model + simultanteously estimates all 3D objects o; (p), with 7-DoF poses and 3D geometries oj:
( o..) =
b,)).
(1)
During the forward process, gradually add Gaussian noise to data point zy to TT over series of discrete time steps T. For a given data point zgg, e.g., shapes o; and poses Pt, the noisy version 2; at time step is given by Markovian process [22 60 (( and its joint distribution g (mo) can be expressed as:
q(1r(x=-1) N(ai v 9,
(2)
q = II g((x)// ((-))
(3)
i=1
with t E [1, and 8r pre-defined linear variance schedule.
During the reverse process, the denoising network tries to remove the noise and recover To from pa (I:-11/...1
Pa 10(21-1/xe v) -NI- (21, v), Ex (2s,t,
(4)
(zorrll)) = Pa (IT)
II pa (1e-
(5)
with being the conditional information from the input image I.
Conditioning. To effectively guide the diffusion process pa (2o:r ly), is crucial to accurately model the conditional information y. First, encode the input image using 2D backbone θ, and apply 2D instance segmentation to get n detected 2D objects b, comprising of its 2D bounding box, image feature patch, and semantic class (cls). Each element is encoded using specific embedding function θ. The per-instance y; and scene condition y is then formed as:
Y/ concat Obox (box:), I (feat;). Ocu(cls;)) y (3) (y...... Vn.)
To learn scene prior over all objects in the scene, we condition the denoising network on the scene condition This not only enables learning the individual object representations but also facilitates learning to capture the scene context and inter-object relationships (Sec. (.5)). Furthermore, we adopt classifier-free guidance [2 for our model by dropping the condition with probability p 0.8, i.e. using a special 0-condition Ø. This allows our model to function a conditional model p y) and unconditional model pa (xo) at the same time, thus enabling unconditional synthesis (Appendix B

---

Loss Formulation. Unlike related works like [23 48 771 that regress object poses Pj and shape
parameters oj using& a multitude of highly-tuned losses, we train our model & to minimize simple diffusion and alignment losses:
Cjoin! (I) P Shupe (I) + a
(8)
Lpsse (T) = E En~N 13 (p(t), f, I, - c|l,
(9)
Lshape (I) E 1,, E,e (i,t),t,,,b) - -ell,
(10)
where we define Z(t) Vātz + vI- āGe for E o with pre-defined noise coefficients az, while & denotes the predicted noise. We use X = 0.01 to balances the effect of Lalign-
Due to the lack of full ground truth supervision in publically available 3D datasets, we introduce an additional alignment loss Lalign for joint training of pose and shape (Sec. (.6). Depending on the availability of ground-truth data (see Sec. 4.2 we mask out individual losses.
3.3 Object Pose Parameterization
We adopt the object pose parameterization of 23 defining the pose p; = (c,, 8,, 0.) of an object by its 3D center c; R the spatial size R and orientation E [-TT, πT) in The 3D center c; is further represented by the 2D offset 6, E R between the 2D bounding box center coordinate and the projected coordinate of the 3D center the image plane, along with the distance d, E R from the object center to the projected center. Our model learns to denoise this 7-dim. pose representation.
3.4 Shape Encoding
We represent object shapes using the disentangled shape representation from 20 A shape is represented as shape code d; E 256 which is factorized into set of g oriented, anisotropic 3D Gaussians Gy.j {1..... g) and an associated 512-dim. latent feature vector per Gaussian. Each Gaussian consist of 16 main parameters: H4y R (center), factorized covariance matrix U, R3x3 (rotation), AÚ E R (scale) and π. E R² ("mixing" weight). We g 16 Gaussians to form scaffolding of the shape's geometry. Together with their latent features, these Gaussians are decoded into high-fidelit occupancy fields, and the final mesh is extracted by applying marching cubes [40).
While similar 30), our model learns denoise this shape parameterization d; our additional surface alignment loss falign (Sec. (.6) provides relational signal between predicted shapes and poses. This enables additional guidance in the face of missing joint pose and shape annotations as in SUN RGB-D dataset 62)
3.5
Scene Prior Modeling
Given the ill poo-dd nature of single-view reconstruction, a robust scene prior is essential for achieving good performance. Effectively capturing the scene context and modeling the relationships between objects within the scene is crucial for learning this strong scene prior [7].77. Previous methods either reconstruct each object individually [15) refine their features using graph networks 777.. In contrast, our approach considers the entire scene by conditioning on all scene objects simultaneously pe (xo ly) and y ....... yN) and additionally allows objects to exchange relational information throughout the entire process. We model the inter-object relationships using an attention formulation has proven be powerful for aggregating contextual information.
669 which
We denote this formulation as Intra-Scene Attention (ISA), which allows all objects within the scene attend each other, effectively modeling their relationships. Please refer to Appendix E for more details and to Tab, Lab.2)nm for the corresponding ablation study, which demonstrates the effectiveness of our learned scene prior.
3.6 Surface Alignment Loss
Publically available 3D scene datasets often only provide partial ground-truth annotations 47
62
To facilitate joint training our model on pose and shape estimation, even in the absence of complete ground-truth annotations, propose leverage expressive intermediate shape representation
5

---

to provide additional supervision and to align shapes efficiently with the available partial depth information D. An illustration of the surface alignment loss formulation is provided Fig. 2 During training, for each object o,, we use the expected shape code o estimation by our model to obtain the predicted Gaussian Gig distribution. Given this scaffolding representation, we directly sample m 1000 points Pú.n (N,,, E,) per Gaussian G.,; resulting in a shape point cloud P: = (P(s.)y Ii {1, ........ I), f.....,m]] m. We transform the resulting shape points P. into the camera frame by the predicted object pose P3- Using the instance segmentations and ground-truth depth maps, obtain K; surface points qj. for object o; and define the surface alignment loss for all
visible objects as I-sided Chamfer Distance [16 48
1 71
Calign
min PER
pl
(11)
Unlike previous works such as [48] that perform costly sampling of points on the decoded shape surface, our approach enables direct point sampling from the conditional shape prior Gz.y. This loss formulation facilitates joint training of pose and shape for all objects simultaneously and its efficancy
demonstrated through ablation studies Tab.
3.7 Architecture
Our architecture consists a pre-trained image backbone, a novel image-conditional scene prior diffusion model, and a conditional shape decoder diffusion module. We utilize off-the-shelf 2D instance segmentation model, Mask2Former (5), which is pre-trained on COCO (36] using a Swin Transformer [39] backbone, to obtain instance segmentation and image features. Please refer to Appendix [] for details about the condition embedding functions.
To denoise object poses Pt, blocks with skip connections. Each block consists of a time-conditional ResNet [88 layer, multi-head attention between the per-object condition yi and the pose representation, and our intra-scene attention module (Sec. 3.5) to enable relational information exchange and effectively train a scene prior. We attention heads, with 64 features per head.
1-dim. UNet [55] architecture with 8 encoding and decoding
To estimate object shapes o; from the input view I, we denoise the unordered set of Gaussian G using a Transformer [69] model with encoder layers, 6 decoder layers, and multi-head attention with heads to the object condition information, similar to [30.. The per-Gaussian latent features are denoise with a shape decoder diffusion model, realized as another Transformer model with 6 encoder and decoder layers, which is conditioned on the shape Gaussians.
3.8 Training and Implementation Details
For all diffusion training processes, we uniformly sample time steps / 1,..7, = 1000, and use a
linear variance schedule with 0.0001 and 8r 0.02. We implement model in PyToorch (50) and use the AdamW [41 optimizer with a learning rate of x 10-4 and B1 = 0.9, ,32 = 0.999. We train our models on single RTX3090 with 24GB VRAM for 1000 epochs on Pix3D, for 500 epochs on SUN RGB-D and for 50 epochs of additional joint training using Calign-
During inference, we employ DDIM [6]l with 100 steps accelerate sampling speed. For classifier- free guidance 21, we drop the condition with probability 0.8.
4 Experiments
In the following sections, will demonstrate the advantages of our method and contributions by evaluating against common 3D scene reconstruction benchmarks.
4.1 Baseline Methods
We compare our method against current state-of-the-art methods for holistic scene understanding: Total3D [48. Im3D (77), and InstPIFu (37. Total3D [48) directly regresses 3D object poses from
image features and uses mesh deformation and edge-removal approach [49 reconstruct shape. Im3D [77 utilizes an implicit shape representation and a graph neural network to refine the pose

---

predictions. InstPIFu [37 focuses single-object reconstruction and proposes to query instance- aligned features from the imput image in their implicit shape decoder to handle occlusion. For scene reconstruction, they rely on the predicted 3D) poses of Im3D. We use the official code and checkpoints provided by the authors of these baseline methods and evaluate with ground truth 2D instance segmentation and camera parameters to ensure fair comparison. We further compare against a retrieval-based method, ROCA [17 in Appendix
4.2 Datasets
Following
23 48 77], we train and evaluate the performance of our 3D pose estimation on the SUN RGB-D
62 dataset with the official splits. This dataset consists of 10,335 images of indoor scenes (offices, hotel rooms, lobbies, furniture stores, etc.) captured with four different RGB-D cameras. Each image is annotated with 2D and 3D bounding boxes of objects in the scene. During joint training, we use the provided depth maps together with instance masks to compute Lulign- We train and evaluate the performance of our 3D shape reconstruction which contains images of common furniture objects with pixel-aligned 3D shapes from object classes, comprising 10,046 images. We use the train and test splits defined in [37, ensuring that 3D models between the respective splits do not overlap.
the Pix3D [64] dataset,
4.3 Evaluation Protocol
For quantitative comparison against baseline methods, we follow the evaluation protocol of [4]] For pose estimation, we report the intersection union of the 3D bounding box (IoU3p) and average precision with an IoU threshold of 15% (AP on the SUN RGB-D dataset 62 In line with previous works 48 (77). we evaluate with oracle 2D detections but also provide camera parameters to all methods during evaluation. To further assess the alignment of the 31D shapes in the scene, calculate Lalign between reconstructed shapes and the instance-segmented ground-truth depth map. For single-view 3D shape reconstruction, low (37) and sample 10,000 points on the predicted shape surface, extracted with Marching Cubes [40] resolution of 128 and on the ground truth shapes and evaluate Chamfer distance (CD x10*) and F-score after mesh alignment.
follow evaluate on the Pix3D [64] dataset. We fol-
4.4 Comparison to State of the Art
3D Scene Reconstruction. In Fig. 3. we present qualitative comparisons of our approach against state-of-the-ar methods for single-view 3D scene reconstruction. The results from Total3D often exhibit intersecting objects and lack global structure. Additionally, their deformation and edge- removal approach results in 3D shapes with visible artifacts and limited details. While the implicit shape representation of Im3D is more flexible, it often produces incomplete and floating surfaces. In contrast, our diffusion-based reconstruction method, as shown in Tab. T learns strong scene priors, resulting in a +0.2 improvement in Latign and more coherent 3D arrangements of the objects in the scene (+12.04% API5). as well as high-quality and clean shapes (+ 13.44% F-Score).
Furthermore, we demonstrate the generalizability approach individual frames from the ScanNet [11] dataset using 2D instance predictions from Mask2Former without additional fine-tuning. As shown in Fig. 4 our method accurately reconstructs the given input view with matching poses and high-quality 3D geometries.
our model to other indoor datasets. We evaluate
Due to its retrieval approach, the shapes are complete. However, the resulting quality can limited by the diversity of the shape database, which can lead to suboptimal results, see Fig.
In Appendix D we additionally train on ScanNet and compare against ROCA 17
3D Pose Estimation Scene Arrangement. As shown in Tabs. I Mand 6 our method outperforms all baseline methods by significant margin in terms of IoUnD and !PI5 aD. i.e., improving mAP!5 by 12.04% over Im3D 777.. Detailed per-class results are provided in Tabs. and Figs. and demonstrate that our approach effectively learns common object arrangements, such as multiple chairs surrounding a table, while ensuring that furniture pieces do not intersect or float in the air. We attribute these improvements to our model's robust scene understanding, which is derived from learning strong scene prior that accounts for onter-object relationships.
7

---

Table 1: Quantitative evaluation of 3D scene reconstruction on SUN RGB-D 62 (left) and 3D
shape reconstruction on Pix3D 64 (right). Our 3D scene diffusion approach outperforms all baseline methods on both tasks on common 3D scene reconstruction metrics.
SUN RGB-D 62
Pix3D 64
IoUan f
AP
Calign
CD
F-Score f
Total3D 48
20.52
(15.55
30.56
(2762)
1.35
(0.36)
44.32
(29.27)
36.20
(22.1)
Tm3D
77
28.31
46.14
(-1209
1.24
(4.25)
51.31
(-6226)
21.45
(37226)
InsiPIFu
37
26.14
(9.96)
45.02
613.16)
1.19
(4.00)
24.65
(9.6)
45.28
64344)
Ours
36.10
58.18
0.99
15.05
58.71
Table 2: Ablations. We ablate the effect of our contributions and design decisions. We observe significant gains by introducing our proposed scene prior and intra-scene attention module, using denoising diffusion compared to regression, and jointly training shape and pose together.
Diftussio
ISA
Joint
IoUnD
AP
Calign
/
X
28.98
(-.12)
47.10
(-11.08)
1.18
(..19)
x
x
28.82
(-7.25
48.88
(9.30)
1.12
(0.13)
v
x
35.16
(4:34)
56.07
(2.11)
1.06
(0.07
/
v
36.10
58.18
0.99
3D Object Reconstruction. In Tab. 1 we quantitatively compare the single-view shape reconstruction performance of our approach against baseline methods on the Pix3D dataset. The results demonstrate that modeling single-view reconstruction conditional generation shape prior leads to significant improvements Chamfer Distance (+9.6%) and F-Score (+13.43%) Detailed per-class results reconstructs noisy and incomplete shapes. In contrast, our approach produces clean 3D geometries with fine details, such thin chair legs and the crease between pillows of sofa.
a robust
be found in Tabs. and Fig. illustrates that InstPiFU often
In Fig. show unconditional results by injecting Ø condition (Sec. 3.2), showcasing that shape prior models detailed and diverse shape modes across several semantic classes. In Fig, [00] we additionally visualize the shape decomposition capabilities resulting from our shape encoding and the scaffolding Gaussian representation.
4.5 Ablations Studies
We conduct series detailed ablation studies to verify the effectiveness of our design decisions and contributions. The quantitative results are provided in Tab. 2
What the effect of the denoising formulation? To assess the benefits of the denoising diffusion formulation, we construct 1-step feed-forward regression model that uses the same conditional information as input features and model architecture but regresses the object outputs directly in single timestep. As shown in Tab, 2 modeling 3D scene reconstruction conditional diffusion process, rather than using forward regression formulation, results in significant improvements of +11088% AP!5 and +0.19 Calign-
What is the effect of our scene prior modeling? We evaluate the impact of learning scene prior by modeling the distribution of all objects and their relationships compared to learning the marginal per-object distribution, i.e., predicting each object individually. As shown in Tab. 2 our joint-object scene prior yields significant improvement of +9.30% AP!S over per-object prediction. This improvement underscores the importance of learning a robust scene prior that effectively captures inter-object relationships.
What is the effect of joint training? We investigate the benefit of joint training for pose and shape using Lalign compared to individual training of pose estimation and shape reconstruction. Although our model already learns strong scene and shape priors, Tab. shows that joint training provides additional benefits, resulting in an improvement of +2.11% in AP!S and +0.07 in Latign.

---

Input
Total3D
lm30
Ours
Figure 3: Qualitative comparison of 3D scene reconstruction on SUN RGB-D 62 While the baselines often produce noisy or incomplete shape reconstruction of intersecting or misplaced objects, our method produces plausible object arrangements as well as high-quality shape reconstructions.
Input
Input
Figure 4: Inference results on ScanNet 111. We use our model trained on SUN RGB-D 62 and perform inference on individual frames of ScanNet without fine-tuning. We observe strong generalization capabilities with respect to different camera parameters and scene arrangements.
4.6 Limitations
While our conditional scene diffusion approach for single-view 3D scene reconstruction demonstrates significant improvements, there are some limitations. First, our method relies on accurate 2D object detection, making it dependent on the performance of 2D perception models. Upcoming state-of-the art 2D detection models [1] can be seamlessly integrated to enhance the performance of our approach. Second, our shape prior, trained on a diverse set of semantic classes using 3D shape supervision, does not generalize to unseen object categories. This can be mitigated by combining our model for known categories with single-object diffusion models that leverage pre-trained text-image generation models for 3D shape synthesis [38) of uncommon shape categories. While accurate 3D scene reconstruction

---

Figure 5: Unconditional results. Injecting Ø as a condition to our conditional diffusion model, i.e., effectively disabling the conditioning mechanism, results in high-quality and diverse results.
forms the foundation for subsequent downstream tasks like mixed reality applications, our current model assumes static scene geometry. Future work could integrate object affordance and articulation into our shape prior 34 to enable more immersive human-scemt interactions.
We do not anticipate any societal consequences or negative ethical implications arising from our work. Our approach advances the holistic understanding of 2D perception and 3D modeling, benefiting various research areas.
Broader Impact
Conclusion
In this paper, we present novel diffusion-based approach for coherent 3D scene reconstructions from single RGB image. Our method combines simple yet powerful denoising formulation with robust generative scene prior that learns inter-object relationships by exchanging relational information among all scene objects. To address the issue of missing ground-truth annotations in publicly available 3D datasets, we introduce a surface alignment loss Lalign to jointly train shape and pose, effectively leveraging scene understanding, outperforming current state-of-the-art methods across various benchmarks, with +1 +1..04% AP!S on SUN RGB-D and +13.43% F-Score on Pix3D. Extensive experiments demonstrate that our contributions 3D scene reconstruction as conditional diffusion process, scene prior modeling, and joint shape-pose training enabled by Lalign collectively contribute to overall performance gain. Additionally, we show that model supports unconditional synthesis and generalizes well to other indoor datasets without further fine-tuning. We believe these advancements lay a solid foundation for future progress in holistic 3D scene understanding and open up exciting applications in mixed reality, content creation, and robotics.
shape representation. Our approach significantly enhances 3D
Acknowledgements
This work was funded by the ERC Starting Grant Scan2CAD (804724) of Matthias Nießner and the ERC Starting Grant SpatialSem (101076253) of Angela Dai.
10

---

References
Coco leaderboard. URL htl.atppp//ooaaaaattrrr/eeeeeeee
[2] A. Alliegro, Y. Siddiqui, T. Tommasi, and M. Nießner. Polydiff: Generating 3d polygonal meshes with diffusion models. arXiv preprint arXiv:23 1417, 2023.
[3] A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nicßner. Scan2cad: Learning cad model alignment in rgb-d scans. In CVPR, 2019.
[4] A. X. Chang. T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, Xiao, L. Yi. and Yu. Shapenet: An information rich model repository. arXiv preprint 2015.
[5] B. Cheng, I. Misra, A. G. Schwing, Kirillov, and Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[6] Y.-C. Cheng, H.-Y. Lee, S. Tulyakov, A. Schwing, and L.-Y. Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[7] G. Chou, Y. Bahat, and F. Heide. Diffusion-sdf:. Conditional generative modeling of signed distance functions. 2023.
[8] C. B. Choy, D. Xu. J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Earopn-Ereppean Conference Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628 644, Springer, 2016.
[9] T. Chu, P. Zhang. Q. Liu, and J. Wang. Buol: A bottom-up framework with occupancy-aware lifting for panoptic 3d scene reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4937-4946. 2023.
[10] M. Dahnert, J. Hou, M. Nießner, and A. Dai. Panoptic 3d scene reconstruction from a single rgb image. In Thiriy-Fifth Conference Neural Information Processing Systems, 2021.
[11] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[12] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kemb- havi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[13] Y. Du, Z. Liu, H. Basevi, A. Leonardis, B. Freeman, J. Tenenbaum, and J. Wu. Learning to exploit stability for 3d scene parsing. In Conference Neural Information Processing Systems, 2018.
[14] H. Fan, H. Su, and L. J. Guibas. A point generation network for 3d object reconstruction from single image. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[15] G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[16] Т. Groueix, M. Fisher, G. Kim, Russell, and M. Aubry. AtlasNet: Papier-Mäché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[17] C. Gümeli, A. Dai, and M. Nießner. Roca: Robust cad model retrieval and alignment from a single image. 2022.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual Icarning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[19] Q. He, D. Zhou, B. Wan, and X. He. Single image 3d object estimation with primitive graph networks. In Proceedings of the 29th ACM International Conference on Multimedia, pages
2021.
[20] A. Hertz, O. Perel, R. Giryes, O. Gorkine-HHrnune and D. Cohen-Or. Spaghetti: Editing implicit shapes through part aware generation. ACM Transactions on Graphics (TOG), 41(4):1-20, 2022.
[21] J. Ho and T. Salimans. Classifier-fie free diffuuson guidance. arXiv preprint arXhv:2207.155%8 2022.

---

[22] J. Ho. A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:6840-6881, 2020.
[23] S. Huang. S. Qi. Y. Xiao, Y. Zhu, Y. N. Wu, and S.-C. Zhu. Cooperative holistic scene understanding: Unifying object, layout, and camera pose estimation. Conference on Neural Information Processing Systems, 2018.
[24] S. Huang, S. Qi, Y. Zhu, Y. Xiao, Y. Xu, and S.-C. Zhu. Holistic 3d scene parsing and reconstruction from single rgb image. In European Conference Computer Vision, 2018.
[25] K.-H. Hui, R. Li, J. Hu, and C.-W. Fu. Neural wavelet-domain diffusion for 3d shape generation. In SIGGRAPH Asia 2022 Conference Papers, pages 2022.
[26] H. Izadinia, Q. Shan, and S. Seitz. Im2cad. In CVPR, 2017.
[27] Jang and L.. Agapito. Codenerf: Disentangled neural radiance fields for object categories. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12949-12958, 2021.
[28] T. Karras, M. Aittala, T. Aila, and S. Laine. Flucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565526577, 2022.
[29] S. W. Kim, B. Brown, K. Yin, K. Kreis, K. Schwarz, D. Li, R. Rombach, A. Torralba, and S. Fidler. Neuralficl--ldm: Scene generation with hicrarchical latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[30] J. Koo, S. Yoo, M. H. Nguyen, and M. Sung. Salad: Part-level latent diffusion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14441-14451, 2023.
[31] N. Kulkarni, I. Misra, S. Tulsiani, and A. Gupta. 3d-relnet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2212 2221, 2019.
[32] W. Kuo, A. Angelova, T-y. Lin, and A. Dai. Mask2ead: 3d shape prediction by learning to segment and retrieve. In Proceedings of the European Conference on Computer Vision (European Conference on Computer Vision), 2020.
[33] W. Kuo, A. Angelova, T.-Y. Lin, and A. Dai. Patch2ead: Patchwise embedding learning for in-the-wild shape retrieval from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12589-12599 2021.
[34] J. Lei, C. Deng. W. B. Shen, L. J. Guibas, and K. Daniilidis. Nap: Neural 3d articulated object prior. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages
Curran Associates, Inc., 2023.
[35] L. Li, S. Khan, and N. Barnes. Silhouette-assisted 3d object instance reconstruction from cluttered scene. 2019 IEEE/CVE International Conference on Computer Vision Workshop (P (Proceedings of the IEEE/CVF International Conference on Computer VisionW), pages 2080-2088, 2019. doi: 10.1 wwnprfffceteeeeeeeeeeeeeaonnnnnn
[36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. T.. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-European Conference Computer Vision 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part 13, pages 740-755. Springer. 2014.
[37] H. Liu, Y. Zheng, G. Chen, S. Cui, and X. Han. Towards high-fidelity single-view holistic reconstruction of indoor scenes. In European Conference on Computer Vision, 2022.
[38] R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. 7akharov, and C. Vondrick. Zero-1-to-3: Zero-shot one image 3d object. In Proceedings of the IEFF/CVF International Conference Computer Vision, 2023.
[39] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Zhang, S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[40] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Trans. Gr., 21(4):163-16) 1987.
12

---

[41] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations 2018.
[42] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Puttern Recognition, pages 2837-28455, 2021.
[43] P. Mandikal, N. KL, and R. Venkatesh Babu. 3d-psrnet: Part segmented 3d point cloud reconstruction from single image. Proceedings of European Conference on Computer Vision (European Conference on Computer Vision) Workshops, pages 0 0, 2018.
[44] L.. Keriaz,, C. Rupprecht, and Vedaldi. Pe2: Projection-conditinne point cloud diffusion for single-image 3d reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12923-1222 12932, 2023.
[45] N. Müller, Y. Siddiqui, L. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4328 4338, 2023.
[46] C. Nash, Y. Ganin, S. A. Eslami, and P. Battaglia. Polygen: An autoregressive generative model of 3d meshes. International conference on machine learning, pages 7220-7229, PMLR, 2020.
[47] P. K. Nathan Silberman, Derek Hoiem and Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, 2012.
[48] Y. Nic, X. Ham, S. Guo, Zheng, J. Chang, and J. Zhang. TotalBdunderstanddig Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In CVPR, 2020.
[49] J. Pan, X. Han, W. Chen, J. Tang, and K. Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[50] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems, 2019.
[51] B. Poole, A. Jain, J. T. Barron, and Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[52] S. Popov, Bauszat, and Ferrari. Corenet: Coherent 3d scene reconstruction from single rgb image. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pages 366-386 300--833.00 Springer, 2020.
[53] X. Ren, J. Huang, X. Zeng, K. Museth, S. Fidler, and E. Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hicrarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2024.
[54] L. Roberts. Machine perception of threedimensional solids. PhD thesis, Massachusetts Institute of Technology. 1963.
[55] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-AAse Assisteed InterventionMMICC. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[56] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252. 2015.
[57] E. Sella, G. Fiebelman, P. Hedman, and H. Averbuch-Elor. Vox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 430 440, 2023.
[58] J. R. Shue, E. R. Chan, R. Po, Ankner, J. Wu, and G. Wetzstein. 3d neural field generation using triplanc diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20875-20886, 2023.
[59] Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. NieBner. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2024.
13

---

[60] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[61] J. Song. C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[62] S. Song. S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
[63] S. Song, Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from single depth image, arXiv preprint arXiv:1611.08974 2016.
[64] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang. T. Xue, J. B. Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. CVPR, 2018.
[65] S. Szymanowicz, C. Rupprecht, and A. Vedaldi. Viewset diffusion: (0- -o-iimag--onnditoneed 3d generative models from 2d data, International Conference on Computer Vision, 2023.
[66] J. Tang, X. Han, J. Pan, K. Jia, and X. Tong. A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In Proceedings of the ieeelcvf conference on computer vision and pattern recognition. pages 4541-4550, 2019.
[67] Tang. Nie, L. Markhasin, A. Dai, J. Thies, and M. Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303. 14207, 2023.
[68] S. Tulsiani, S. Gupta, D. F. Fouhey, A. A. Efros, and J. Malik. Factoring shape, pose, and layout from the 2d image of 3d scene. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 302 110, 2018.
[69] A. Vaswani. N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[70] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.G.. Jiang, Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference Computer Vision, 2018
[71] J. Wu. C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning probabilistic latent space of object shapes via 3d aeeeraaiveeadvessariie modeling. In Advances in Neural Information Processing Systems, pages 82-90, 2016.
[72] H. Xie, H. Yao. X. Sun. S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and nulti-view images. Proceedings of the IEEE/CVF international conference on computer vision, pages 2690- 2898, 2019.
[73] A. Yu, V. Ye, M. Tancik, and Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, 2021.
[74] X. Zeng, A. Vahdat, Williams. Z. Gojeic, Litany, Fidler, and Kreis, Lion: Latent point diffusion models for 3d shape generation. arXiv preprint urXiv:2210.06978, 2022.
[75] B. Zhang, M. Nießner, and P. Wonka. 3DILG: Irregular latent grids for generative modeling. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
[76] B. Zhang, Tang, M. Niessner, and Wonka. 3dshape2vecset: 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301. 2023.
[77] C. Zhang, 7. Cui, Zhang, B. Zeng, M. Pollefeys, and S. Liu. Holistic 3d scene understanding from a single image with implicit representation. In CVPR, 2021.
[78] X. Zhang, Z. Chen, F. Wei, and Tu. Uni-3d: A universal model for panoptic 3d scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (Proceedings of the IEEF/CVF International Conference on Computer Vision), pages 9256-9266, October 2023.
[79] X. Zheng, Y. Liu, P. Wang, and X. Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum, volume 41, pages 52-63. Wiley Online Library, 2022.
[80] L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021. 2021.
14

---

A Appendix
In the following, we show more qualitative results for scene reconstruction on SUN RGB-D 62 (Appendix (B) and object reconstruction Pix3D [64]. We provide detailed quantitative per-class comparisons supplementing the tables in the main paper (Appendix C). We additionally compare against retrieval baseline the ScanNet [1] dataset in Appendix D Finally, provide additional details on the architecture of our diffusion model in (Appendix B
For comprehensive overview of approach and results, encourage the reader watch the supplemental video.
B Additional Qualitative Results
Scene Reconstruction In Fig. [ we show additional qualitative results of our method on test frames from SUN RGB-D. Despite strong occlusions and challenging viewing angles, our model predicts accurate scene reconstructions. Our generative scene prior learns common scene patterns, such as parallel object placements between the table and sofa In Fig-B] we also demonstrate that our robust conditional scene prior can recover clean and matching shape reconstruction even for heavily occluded objects, e.g., a chair for which only the back seat barely visible.
bed and neighboring nightstands.
Our diffusion- based scene layout and shape prediction approach achieves accurate results even for strongly occluded objects.
Figure 6: Additional qualitative scene reconstruction results on SUN RGB 62
Object Reconstruction Unconditional Synthesis Fig. 9 of single-view 3D object reconstruction on the Pix3D dataset. Unlike InstPIFu, which often produces noisy and incomplete surfaces, our image-condition diffusion model reconstructs clean and high- fidelity objects. Such visual quality allows these reconstructions be integrated into e.g., mixed reality applications.
we show qualitative comparison
To probe the learned shape prior and investigate its shape synthesis capabilities, we input the 0- condition instead extracted image features to our model. As shown in Fig. 5] our model learns high-quality shape prior with fine details across various semantic classes.
15

---

Input
The input image displayed the left, and the predicted and ground-truti 3D arrangements visualized top- down orthographic views of the scene. We observe that Total3D frequently lacks globally consistent structure, while Im3D predicts globally structured results but occasionally produces intersecting or floating objects. In contrast, our approach successfully recovers a coherent arrangement of objects within the scene by learning a robust scene prior.
Figure 7: Qualitative comparison of 3D pose estimation on the SUN RGB-D [62].
C
Additional Quantitative Results
Scene Reconstruction In Tab, 4 we show detailed comparisons of our approach against baseline methods, Total3D (88 and Im3D 77.. on the 10 most common classes of SUN RGB-D. Our approach consistently outperforms all baseline methods on all classes except the "bed" class. We attribute this exception to the fact that beds are often only partially visible in the input view due to their spatial extent, which introduces higher variability. and regularization terms, which seems to help in extreme amodal cases the cost of additional loss balancing. Nevertheless, our method achieves significant overall improvement of 12.04% in AP!5 on these 10 classes, with particularly notable gains for "dressers" (4-26.03%) "chairs" (+-21.915 %) and "cabinets" (+- (+139.3%) showcasing the effect of our robust scene prior.
contrast, Im3D employs series of geometric losses
Tabs. 6 and g show the per-class comparisons and ablation studies on all NYU classes in terms of loUsn and mAP!s Our approach improves compared Im3D by +7.57% increase in MAP!5 and +4.56% increase in class-mean loU across all 37 classes. The ablation results highlight the importance of our diffusion formulation (+7.67% mAP!5). scene prior modeling (+7.11 mAP!5). and joint training using the surface alignment loss Latign (+0.72 MAP!5)
For single-view object reconstruction. we evaluate Chamfer Distance and F-Score on Pix3D and show per-class comparisons in Tabs. 7 and [ Our image-conditional shape prior leads significant improvements, +9.6% in Chamfer Distance and +13.43 in F-Score, while outperforming InstP[Fu in most categories, except sofas and wardrobes in F-Score.
Object Reconstruction
77 also predict the room bounding box with a separate network head. We study, how our model can also predict the room layout. For that include the room bounding
Room Layout
16

---

box pose as part of the object poses during the diffusion process. We follow the room layout parameterization of [88 77 and model the 3D room center directly instead of decomposing as 2D offset & distance, which is done for the objects. In Tab. 3 we demonstrate that by denoising the pose of room layout, we outperform the regression-
methods.
We evaluate the 3D IoU of the orientied room bounding box. Our diffusion-based pose estimation lead to an improvement of + +1.7% in Room Layout loU.
Table Additional 3D room layout estimation on SUN RGB-D 62
Layout IoU
Total3D 48
59.2
Im3D 77
64.4
Ours
66.1
Table 4: Additional per-class comparisons of 3D layout estimation on SUN RGB-D [62]. Our method outperforms the baselines in most categories with overall strong improvements in mAP3 evaluated at an loU-threshold of 15%.
bed
chair
sofa
table
desk
dresser
n.stand
sink
cabinet
lamp
mAP
Total3D 48
72.47
22.74
53.56
41.49
32.74
17.45
20.06
24.67
16.83
3.63
32.54
Im3D
88.73
36.77
72.81
58.64
49.80
29.73
44.10
34.71
32.72
13.34
46.14
Ours
86.58
58.68
74.13
71.36
62.81
55.76
48.14
50.44
52.09
21.82
58.18
Table 5: Quantitative comparison with ROCA [17 the ScanNet dataset [11 While ROCA esti- mated each object's pose individually, our generative scene prior can reason about object relationships,
leading to +3.1%% improvement class-wise alignment accuracy.
bathtub
bed
bin
b.shelf
cabinet
chair
display
sofa
table
cls.
inst.
ROCA 17
22.5
10.0
29.3
14.2
15.8
41.0
30.9
16.8
14.5
21.7
27.4
Ours
28.7
18.3
19.1
17.6
36.9
39.7
19.2
24.5
19.2
24.8
29.5
D
Comparison to shape retrieval baseline on ScanNet
compare with shape retrieval baseline, namely ROCA 17.. Since ROCA requires full ground- truth supervision during training, adopt their setup and train our model on the same 25,000 frames from the ScanNet [TT] dataset with pose annotations derived from Scan2CAD 3 as well as the same CAD pool from ShapeNet [ We additionally adopt their full 9-DoF pose parameterization by predicting rotation angles. Following ROCA, we quantitatively evaluate the Alignment Accuracy in Tab. 5 Please refer to 3 17 for the details of the evaluation. In Fig. II] we see that ROCA retrieves clean and complete shapes by definition. However, due to its limited shape database, cannot capture all shape modes accurately, leading to shape mismatches. Our reconstruction- sassuettooooooddddddeee based approach instead can recover faithful shape results while simultaneously predicting coherent object arrangement.
E
Architecture Details
Object Pose Parameterization: Normalization
ensure a reasonable signal-noise ratio 28
among the object pose parameters, we normalize the parameters to [-1, 1] by dividing them by its max value and shift the range using a parameter-speciffe μ value. this, we calculate the min-max ranges of all pose parameters, i.e., rotation 0, 3D scale s, and projected distance d, within the train
17

---

We report the Average Precision (AP) 15% 3D-IoU threshold of the baseline and different variants of our approach: Our approach outperforms Total3D and Im3D on most semantic categories, especially on frequent classes likes chairs (+21.9%) tables (+12.7%).
Table 6: 3D pose estimation results for all NYU-37 classes on SUN RGB-D 62
Total3D
Im3D
Ours no ISA
MZ2
8o diET.
joint
full
cabinet
16.83
32.72
35.43
37.32
40.48
48.48
52.09
bed
72.47
88.73
76.23
84.58
86.50
90.71
86.58
chair
22.74
36.77
46.97
49.38
48.82
55.80
58.68
sofa
53.56
72.81
64.83
66.44
66.27
72.43
74.13
lable
41.49
58.64
62.31
59.34
58.47
69.70
71.36
door
1.18
5.85
6.25
3.58
5.58
7.73
5.44
window
2.72
0.57
0.51
3.08
2.57
2.62
2.72
bookshelf
4.95
18.02
19.56
25.07
20.99
30.81
30,81
picture
1.21
1.66
0.99
2.04
1.31
1.80
3.95
counter
41.29
62.48
62.58
62.30
56.47
69.78
72.44
blinds
0.00
2.79
1.67
2.27
3.64
4.27
5.20
desk
32.74
49.80
52.31
48.78
48.93
60.20
62.81
shelves
9.72
18.16
14.58
16.31
14.51
25.31
28.01
curtain
1,30
7.69
9.19
3.94
6.76
11.93
10.43
dresser
17.45
29.73
36.07
41.86
50.91
53.06
55.76
pillow
9.41
19.48
19.37
23.10
20.54
33.45
28.99
mirror
0.50
0.84
4.22
1.11
2.04
8.15
9.98
clothes
0.00
0.00
0.0
0.00
0.00
0.00
0.0
books
4.23
7.16
5.42
11.26
10.73
17.18
12.76
fridge
25.00
40.47
27.13
42.66
37.59
45.90
46.17
television
10.88
14.49
13.89
11.95
10.71
19.81
23.55
paper
3.47
1.14
1.97
4.96
4.75
4.97
5.75
towel
4.35
14.80
2.68
8.11
8.19
11.02
12.99
s.curtain
0.00
0.00
0.00
0.00
0.00
0.00
0.00
box
7.40
11.52
15.86
17.43
17.72
29.02
24.42
whiteboard
1.40
2.59
2.68
1.66
4.18
5.44
person
22.12
19.22
38.32
31.48
28.45
55.10
56.39
nightstand
20.06
44.10
28.76
38.41
36.32
45.50
48.14
toilet
64.36
73.14
65.11
61.56
71.57
71.19
66.30
sink
24.67
34.71
30.49
32.01
39.60
42.94
50.44
lamp
3.63
13.34
12.90
12.88
12.48
21.84
21.82
bathtub
46.86
66.54
30.51
36.47
40.87
50.46
52.77
bag
13.67
8,45
8.66
13.78
16.52
18.89
21.69
mAP!5
3p 6l)
17.63
26.01
24.17
25.91
26.47
32.86
33.58
mAPls 3D (1937)
30.56
46.14
44.63
47.10
48.88
56.07
58.18
set of SUN RGB-D. The 2D offsets to the 2D bounding box center are normalized by the image dimensions.
d 2.7, max 2.5,
(12)
$ 1L 3.5, 0: 0.0, max = 3.14.
7.0,
(13)
(14)
During training, the loss is computed on the un-normalized parameter ranges. After inference and for evaluation, we un-normalize each parameter according to original range.
Surface Alignment Loss: Point Sample Transformation During training, for each object o,,
we use the predicted shape o; to estimate its scaffolding Gaussians Gj. From each 3D Gaussian distribution, we directly draw 3D point samples P(j.)) .N(u, E,). This shape point cloud P; approximates the shape. With the predicted and un-normalized object pose Å., we define a 3D rigid transformation R and transform the shape point cloud P. to the camera coordinate system. We this transformed shape pointcloud peam and the instance-segmentee ground-trus -trtt depth map from
18

---

Table 7: Per-class comparisons of shape reconstruction on Pix3D 64).
We report F-Score using
the non-overlapping 3D model split from 37 We observe noticeable improvements or comparable results on all categories.
bed
b.case
chair
desk
misc
sofa
table
tool
w.robe
F-Score
Total3D [88]
34.69
28.42
35.67
34.90
10.41
51.15
17.05
57.16
52.04
36.20
Tm3D 777
37.13
15.51
25.70
26.01
11.04
49.71
21.16
5.85
59.46
31.45
InstPIFu 37
54.99
62.26
35.30
47.30
27.03
56.54
37.51
64.24
94.62
45.62
Ours
62.47
65.32
60.05
56.67
30.89
55.87
56.28
69.11
92.56
58.71
Input
Camera View
Side View
#2
Figure 8: Probabilistic behavior for partially occluded shapes. In the input image, the left chair is heavily occluded, which allows for multiple plausible interpretations of the non-visible part of the shape. Our diffusion-basee method derives faithful modes.
SUN RGB-D the partial target pointcloud to measure the 1-sided Chamfer distance and compute the surface alignment loss Calign-
Scene Prior Modeling:
Inter-Object Relationships via Intra-Scene Attention We use the multi-head attention mechanism [69] between the scene objects to allow them attend to each other, effectively learning their inter-object relationships and the scene context. Specifically, given an unordered set S = [01,02, 02, ..., on], O; E R per-object n-dimeensinal feature vectors, projection layers (WQ, W and WV) and features Q = S x Wº, K = S x WK and V S x WV after projection. we define the intra-scene attention as:
QKT
ISA(S) coftmax' dds V
(15)
After cropping the 2D image feature patch Rxx from the frozen image backone O₁ we apply adaptive average pooling to resize the per-object feature patches to a common 2D size leading resized per-object feature crop of 8 and = 256. This feature crop is further embedded using a small 2D CNN Oteat with 3 blocks of convolutional layers with 512 features, group norm, and leaky ReLU activation. The embedded feature crop is reshaped to a 4096-dim vector.
Condition: Embedding Functions
Ob.x is implemented sinusoidal position encoding with 10 frequencies. This function is applied on a 2D bounding box, represented by the top-left and bottom-right corners, leading to an 84-dim vector per object. For Ocs, use simple 1-hot encoding embed the semantie class information. The final per-object condition information is the concatenation, resulting in a 4127-dim vector for each object.
Since the official code of SPAGHETTI does not include the training code and only provides checkpoints for two different shape classes (chairs,
Reimplementation of SPAGHETTI 20
19

---

Figure 9: Qualitative comparison of 3D shape reconstruction on the Pix3D (64). While InstPIFu often produces noisy surfaces, our srage-conditions 3D diffusion model synthesizes high-quality shapes that closely match the target geometries.
Figure 10: Shape decomposition visualization. We assign each vertex of the reconstructed mesh to the closest 3D Gaussian center and visualize the assignment with individual colors. Our scaffolding representation decomposes the shape into distinctive regions and aligns well with certain semantic parts, e.g., individual chair legs or the arm rests of sofa.
airplanes), re-implement the training procedure, loss function, and disentanglement loss following the description the papers to train the full shape prior over all relevant shape categories. Random geometric augmentations are essential during training to achieve self-supervised disentanglement into extrinsic and intrinsic shape properties. We apply full 360-degree random rotations, uniform scale augmentation between 0.7 and 1.3, and translation jitter of T0.3 the disentangled extrinsic and target pointcloud. Further, we do not utilize the symmetry options the original implementation.
20

---

ROCA
Figure Comparison with retrieval baseline method ROCA 17 on frames from ScanNet 11 While ROCA cannot always retrieve a matching mode from the shape database, such as the desk in first row, our diffusion-basee reconstruction approach reconstructs accurate shapes and poses.
Image crop
4096 dim condition
Inpot
Output
c R
Output
MLP
Decoder
MLP
Occupancy
MLP
2004
Decoder
Noisy Extrinsies ")
Denoised Extrinsics R
Extrinsic Diffusion
Shape Decoding
Input R
Output
transformer
transformer
Impet
Output
Transformer
Transformer
Extrinsics
Noisy Intrinsics G 1)35
Enater Bocdji
Pecado Block (lewk?) ard dase 2024
Denoised Intrinsics
G x 3.
Unodee Bioot
MLP
Inside/ Outide
Intrinsic Diffusion
Occupancy Decoder
Transformer Frooder Block
Transformer Decoder Block
MIP
MLP
Figure 12: Architecture Diagram the Shape Diffusion Model. The shape diffusion model consists of 3 sub-parts: An image-conditioned diffusion model, denoising the 3D Gaussians; 3D Gaussian-condittonen diffusion model, denoising the intrisic vectors; and an Occupancy Decoder, which takes as input 3D point coordinate and the denoised extrinsics intrinsies and outputs an occupancy value indicating whether the 3D point is inside/outside of the shape.
21

---

Table 8: Per-class pose estimation results for all NYU-37 classes on SUN RGB-D 62 We evaluate the pose estimation quality in terms of 3D loU. Our scene prior formulation achieves improvements across all categories which particular high gains on common object classes like "chair" (+16.6%) or "desk" (+16.4%)
Total3D)
Im3D
Ours
no MEP
Do ditf.
no ISA
no joint
full
cabinet
13.68
21.96
23.16
26.06
24.54
33.07
32.97
bed
32.28
42.65
41.53
48.98
44.87
52.67
52.25
chair
19.85
26.87
30.62
33.94
30.97
42.92
43.52
sofa
28.32
36.00
32.98
32.69
34.91
38.72
39.48
table
25.70
33.74
32.55
30.41
32.31
40.11
39.95
door
3,91
7.84
7.35
10.01
7.76
10.33
6.73
window
3.52
2.10
3.12
6.86
15.45
18.17
bookshelf
9.07
16.76
17.16
16.75
18.15
24.45
19.43
picture
2.35
5.30
4.69
5.70
4.32
3.36
6.32
counter
21.72
26.82
30.87
28.25
30.92
42.56
38.43
blinds
1.90
7.11
8.38
0.00
5.53
0.00
0.00
desk
21.09
28.21
28.12
34.57
27.51
44.22
44.68
shelves
10.33
14.92
14.01
16.35
14.81
24.32
17.60
curtain
5.09
9.46
10.40
7.99
9.39
2.55
0.00
dresser
16.84
23.29
23.08
22.86
27.82
29.19
32.56
pillow
11.07
17.65
16.62
18.12
16.77
19.05
16.69
mirror
2.05
4.45
5.65
4.83
4.11
9.03
5.81
clothes
0.00
0.00
0.00
0.00
0.00
0.00
books
6,81
8.97
9.59
14.00
15.63
12.30
20.48
fridge
18.41
27.02
19.92
16.18
24.61
26.85
23.36
television
9.59
14.11
12.74
12.60
11.62
19.73
18.93
paper
5.16
4.86
8.76
8.10
17.11
12.54
10.40
towel
7.46
10.53
7.26
10.83
8.32
18.79
13.71
s.curtain
33.12
13.49
30.53
0.00
0.00
0.00
9.41
box
9.40
12.04
16.18
17.91
16.47
24.55
23.82
whiteboard
4.06
6.27
5.94
4.07
6.39
6.46
5.47
person
24.14
23.33
28.94
15.40
21.89
19.50
28.91
nightstand
17.93
29.12
21.06
25.80
24.92
25.59
24.81
toilet
34.11
39.46
38.15
28.95
39.63
51.58
50.91
sink
19.92
25.40
21.50
20.54
24.99
20.81
26.60
lamp
9.63
15.90
12.92
13.94
13.20
24.33
24.22
bathtub
24.64
29.56
24.06
27.38
24.80
34.26
35.17
bag
11.18
11.70
13.63
18.41
16.38
22.74
21.60
mloUap Gll
14.15
18.10
18.53
17.30
18.76
22.79
22.66
mloUsp (1077)
20.52
28.31
26.75
28.98
28.82
35.16
36.10
Table 9: Per-class comparisons of shape reconstruction on Pix3D 64] We report Chamfer Distance using the non-overlapping 3D model split from (37). Across most categories, our model achieves strong improvements compared to the baselines. Especially for frequent classes like "chair" or "table", we see a reduction of more than 45%.
bed b.case
chair
desk
misc
sofa
table
tool
w.robe
CD
Total3D 48
22.91
36.61
56.47
33.95
137.50
9.27
81.19
94.70
10.43
44.32
Im3D 77
11.88
29.61
40.01
65.36
144.06
10.54
146.13
29.63
4.88
51.31
InstPIFu 37
10.90
7.55
32.44
22.09
47.31
8.13
45.82
10.29
1.29
24.65
Ours
8.43
7.11
17.63
19.81
65.29
8.41
21.06
8.07
2.01
15.05
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provide experimental results demonstrating the performance improve- ments by following the evaluation protocol of common 3D reconstruction bench- marks (Tab. D. We further ablate the impact of the individual contributions - denoising formulation, scene prior modeling, and surface alignment loss in Tab.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our approach in Sec. 4.6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: Our work does not introduce new theorems. We provide an empirical evaluation of our model through a series of experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed description of the architecture in Sec. 3.7 Appendix
and implementation instructions in Sec. 3.8
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code after cleaning up and documenting it for easy usage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
hubblic//uilass/ddeeebbmm..sss httssSubmmmsiioooooooo.oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ccippi.cc guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We give detailed explanations of training and test details in Secs.
3.7 and 3.8
and Appendix [ We use common data splits for training and testing (Sec. 4.2) and follow the evaluation protocols of 48
(Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Given limited computational resources, we are not able to train multiple models using different initializations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Sec. 3.8 we explain the setup (hardware, training duration) to train our models. Given the rich design space of the challenging task of single-view 3DD scene reconstruction, we conducted wide range of experiments to reach the performance of the presented work. However, did not keep track of the total compute of all individual runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////..aaarss..ccp////./aaas
Answer: [Yes]
Justification:
We reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Sec, 4.6) we discuss why we do not anticipate any negative impacts of our work on single-view 3D scene reconstruction.
26

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our model only relies on 2D instance segmentation model, which was pre-trained on COCO.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For training and testing, we use multiple publically available datasets, namely SUN RGB-D (62. Pix3D (64), and ScanNet [1]. which were properly explained and cited in the main paper (Sec. 4.2).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
27

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
Use the official train/test splits of publically available datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not perform research with human subjects, nor did the data pre-processing and experiments require crowdsourcing tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not perform research with human subjects, nor did the data prepprocessing and experiments require crowdsourcing tasks.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "7PyRiIp6tR", "reviewer": "Reviewer_W8nX", "summary": "This paper proposes an approach to reconstruct the 3D surfaces of multiple objects from a single RGB image. Given an instance-segmented RGB image as input, the poses and shapes of objects are jointly predicted and denoised using a diffusion model conditioned on the input image. A one-sided chamfer distance (from available ground truth surface point to sampled points in camera space) is used to provide supervision on incomplete surfaces. The output is geometrically evaluated against Total3D (3D detection-based composition), Im3D (implicit surface), and InstPIFu (pixel-aligned implicit surface) and shows improved scene coherence and shape quality.", "questions": "(please see weaknesses section)", "limitations": "I think they are addressed well.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Efficient training\n+ Appropriate use of category-specific object shape priors\n+ Clean ablation results", "weaknesses": "In order for the simple loss (one-sided chamfer distance) to work well, you need strong category-specific priors on generated shapes; it requires the estimated shape gaussians to be stable in the first place. The joint learning of poses is also inherently category-specific and might not generalize well to unseen or uncommon object categories. While this limitation is mentioned in the paper, I think this is an inherent limitation that is not easy to mitigate.\n\nLimits of one-sided chamfer distance: In qualitative results containing office chair images for example, I noticed that many of them have extra leg parts hallucinated. I think this is an example of something that the one-sided chamfer loss won't be able to fix since the error isn't reflected in the loss. I'm surprised the method worked well despite this; this isn't necessarily a weakness, but tells us something about the problem we're trying to solve and evaluation methods (maybe \"alignment is all you need\"?). I think a retrieval-based baseline (with alignment) would have been interesting for the reasons mentioned.\n\nI tend to think the improvements are mostly due to architectural advancements such as attention models (that can process arbitrary unordered variable-length objects) and better off-the-shelf shape models available. The paper appropriately applied those techniques and that is a good thing, but maybe it does not fundamentally improve our understanding of the problem. Perhaps this isn't a fair criticism considering the limitations and quality of other papers accepted in similar venues. I currently feel borderline about the paper. The holistic nature of the problem is difficult enough and the paper proposes one simple solution that I think can be used by other researchers still.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Generalization to unseen/uncommon object categories**\n\nOur shape prior is learned across all categories and in our experiments, we have seen that the shape Gaussians are indeed quite stable and align well with geometric parts (Figs. 1 & 2). However, we do rely on strong priors for seen categories, making generalization to unseen categories challenging. This limitation could be mitigated by training on larger shape datasets such as Objaverse [1].\n\n**2. Comparison to a retrieval baseline**\n\nWe include qualitative results of the CAD retrieval method ROCA [2] in the rebuttal PDF (Fig. 3). While the retrieved CAD models are complete by definition, ROCA struggles to capture the correct mode of instances and produces misaligned and intersecting objects. We will include a detailed evaluation and comparison of this baseline in the final paper.\n\n**3. Improvements**\n\nIn contrast to the regression-based baselines, such as Total3D [3], Im3D [4] and Mesh R-CNN [4], we introduce a fully probabilistic approach to model the joint distribution of object arrangements and shapes. As our quantitative and qualitative results show, our formulation leads to more accurate shapes and object arrangements. In particular, for ambiguous and challenging cases, our diffusion prior still yields plausible results where the baselines tend to produce inaccurate geometries and unrealistic poses.\n\n**References**\n\n- [1] Deitke et al. \"Objaverse: A universe of annotated 3d objects.\", CVPR 2023.\n- [2] Gümeli et al. - \"ROCA: Robust cad model retrieval and alignment from a single image.\", CVPR 2022 \n- [3] Nie et al. - “Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image”, CVPR 2020\n- [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021\n- [5] Gkioxari et al. - “Mesh R-CNN”, ICCV 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6fp6FGpmaL", "reviewer": "Reviewer_U6My", "summary": "In this work, the authors propose a diffusion-based method for generating coherent 3D scenes from a single input image. They use a diffusion model to jointly denoise the 3D poses and geometries of all objects. To address the incomplete ground truth of existing datasets, they also introduce a surface alignment loss to better learn the 3D geometry. Experimental results show that the proposed approach significantly outperforms state-of-the-art methods.", "questions": "- It is suggested to add notation for cls in Equation 6.\n\n- It is suggested to add some figures for the ablation study.\n\n- I find [b] to be quite related to this work. It is suggested to discuss this paper in the related work section. Combining some 3D architecture with the proposed method could be a nice extension.\n\n[b] XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.", "limitations": "I think the authors properly discuss the limitations of this work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- I like the formulation of this method. I think anisotropic 3D Gaussians are a good representation for 3D reconstruction. We could treat the 3D reconstruction in a coarse-to-fine manner and use the 3D Gaussians as the 3D proxy to guide the fine reconstruction.\n\n- The results look promising. With partial ground truth for supervision, the proposed method learns complete geometry from a single RGB image with high occlusion.", "weaknesses": "- I am unclear on how to learn the shape information:\n\n(i) For the shape code sigma_i (16 Gaussians x 16 dimensions per Gaussian), do you have the ground truth for it? If not, how do you apply the shape diffusion loss mentioned in Equation 10?\n\n(ii) To decode the shape code from Gaussians to an occupancy grid, you apply a shape decoder diffusion model (mentioned in Line 213). How do you train this model? Do you include any details about this model?\n\n- I am interested in how the proposed method learns the intermediate scene representation (i.e., anisotropic 3D Gaussians). You use the surface alignment loss as supervision. Are there other indirect supervisions? How do you ensure that the Gaussians do not intersect each other or become trivial (e.g., the same as each other)?\n\n- If I understand correctly, you add inter-object attentions to the intra-scene attention to allow the pose and shape model to run in parallel. If so, this operator is quite common.\n\n- I find the author may miss the citation and discussion with an important literature [a].\n\n[a] CoReNet: Coherent 3D scene reconstruction from a single RGB image. ECCV 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Clarification of shape learning**\n\nTo support our latent diffusion approach for modeling shapes, we represent 3D shapes following the disentangled formulation of SPAGHETTI [1]. SPAGHETTI learns the disentanglement into 16 Gaussians and per-Gaussian “intrinsics” in a self-supervised way through augmentation with rigid transformations of the shapes. Since the official code only provides the inference part and checkpoints for chairs and airplanes, we re-implement the method according to the paper and train it across all categories on Pix3D [2]. From this trained model, we extract the shape codes $\\sigma_i$ for supervision of the shape diffusion model. \nThe shape decoder mentioned in L213 refers to the one of SPAGHETTI. \nWe will add a more detailed description of our re-implementation and training in the paper as well as release the code and checkpoints together with the final paper.\n\n\n**2. Intermediate scene representation**\n\nThe intermediate scene presentation consists of the per-shape 3D Gaussians from the shape diffusion model and the object pose from the object pose diffusion model. The 3D Gaussians are transformed to world space by the predicted object poses. Following baseline protocols and Sec. 4.2, we train our model in stages and use the respective supervision: The scene prior and object poses are trained on SUN RGB-D [3]; Shape reconstruction is trained on Pix3D [2] using the preprocessed 3D shape codes $\\sigma_i$ (see above). The final model is then jointly trained on SUN RGB-D. We do not constrain the 3D Gaussians to be non-overlapping or  non-trivial, however by restricting the number of Gaussians (16 in our experiments), this encourages the shape model to distribute the Gaussians effectively in 3D space.\n\n**3. Intra-Scene Attention**\n\nSince we condition the model on all scene objects simultaneously, the intra-scene attention allows our model to learn a joint scene prior, e.g., properties such as object-object relationships and scene contexts. This is different from methods that individually regress the objects’ pose/shape (MeshRCNN [5], ROCA [6]) or refine the object poses in a second step (Im3D [4]).\n\n**4. Notion of cls in Eq. 6**\n\n‘cls’ in Eq. 6 refers to the semantic class in L135. We will clarify this in the paper.\n\n**5. Figures for ablation studies**\n\nWe included two additional figures (Fig. 1 & Fig. 2) for qualitative results of our ablation studies in the rebuttal PDF. \n\n**6. Additional related works**\n\nThanks for pointing us to additional related works, we are happy to include and discuss them in the paper.\n\n**References**\n\n- [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022\n- [2] Sun et al. - “Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”, CVPR 2018\n- [3] Song et al. - “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite”, CVPR 2015\n- [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021\n- [5] Gkioxari et al. - “Mesh R-CNN”, ICCV 2019\n- [6] Gümeli et al. - \"ROCA: Robust cad model retrieval and alignment from a single image.\", CVPR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v661bkTvek", "reviewer": "Reviewer_scCV", "summary": "This paper proposes a novel framework for 3D scene reconstruction from single images based on diffusion models. This framework jointly predicts 3D object poses and shapes with seperate diffusion models and captures global scene context with a standard mutli-head attention module. To train this model, the paper proposes a surface alignment loss based on one-sided Chamfer Distance, which makes their method able to utilize depth maps as supervision. The experimental results on Sun RGB-D, ScanNet, and Pix3D and numerical ablation studies demonstrate the efficacy of proposed method.", "questions": "1. According to the alignment loss, the predicted Gaussians should be converged to centered on the surface of 3D objects? Otheriwse, the direct sampling won't be around the surface?\n2. How do you deal with varying number of objects within a single image? How do you deal with false positives and false negatives from detection?\n3. Have you tried other normalization methods besides Equation (12-14)? Does it affect the learning efficiency?\n4. Does the disentanglement of pose and shape helps with learning convergence?\n5. How is the proposed method comparing to other SOTAs in single object reconstruction? Can this method be applied to part-level object reconstruction?", "limitations": "The authors adequately addressed the limitations and I do not have more to add here.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper shows its strengths from several aspects:\n1. It proposes an overall solid method for 3D scene reconstruction from single images and deals with the ill-posed nature of this problem with powerful generative prior from diffusion models. \n2. The proposed alignment loss makes the method able to utilize partial supervision from depth maps for model training.\n3. This paper conducted comprehensive experiments in both scene and object reconstruction and achieve good results comparing to previous methods.\n4. The presentation and writing are overall good and clear.", "weaknesses": "My concerns are mainly about the description of the method:\n1. The model architecture is not clear enough in Section 3.7. It seems that each object pose, each object shape, and each Gaussian, are denoised using an independent diffusion model? Where is ISA added and how is it integrated with other modules? Are the inputs of ISA from Equation (7)? \n2. In Figure 1, are the noisy shape Gaussians as in shape with guassian noise? Or are they initial estimation directly from image features? It's a bit confusing.\n3. Are the diffusion models for shape and pose trained with full 3D supervision before joint training with alignment loss? If yes, what if there are only ground truth depth maps in training data?\n\nBesides, the overall idea of 3D generation from features as in Equation (6) combined with global context via attention has been explored in previous related works, such as part-based 3D object reconstruction, e.g., He, Qian, et al. \"Single Image 3D Object Estimation with Primitive Graph Networks.\" Proceedings of the 29th ACM International Conference on Multimedia. 2021. This should be included in related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Clarification of model architecture**\n\nThank you for the feedback regarding the architecture description of our model in Section 3.7. We will improve the clarity of this section in the final paper. In the rebuttal PDF, we have included an architecture figure of the shape model (Fig. 4). \n\nIn particular, the pose diffusion model is conditioned on all objects’ image features of a scene simultaneously (Eq. 6 & 7). This setup allows the model to learn a scene prior via ISA by exchanging relational information between the objects and to denoise the individual object poses. \nDuring joint training, the entire model takes as input all scene object features (Eq. 6 & 7), estimates the scene context implicitly via ISA, predicts the object poses and the intermediate shape Gaussians of each object. These Gaussians are transformed into world space using the specific object pose to compute the Alignment Loss (Fig. 2, right).\n\n\n**2. Clarification of “Noisy Shape Gaussians” in Figure 1**\n\nThe “Noisy Shape Gaussians” in Figure 1 depict the anisotropic 3D Gaussians of 3 objects at the beginning of the denoising process. During the backward diffusion process, these 3D Gaussians get denoised to form the intermediate, scaffolding representation of each shape. We will clarify it in the figure and the caption.\n\n\n**3. 3D supervision**\n\nYes, the pose and shape models are first trained with respective 3D supervision before joint fine-tuning.\nWe follow the training scheme of previous works, Total3D [3], Im3D [4], and train the poses on SUN RGB-D [5] and shapes on Pix3D [6]. Hence, our approach learns marginal distributions for layout and shape first. We then fine-tune for the joint distribution by using the provided depth map from SUN RGB-D and the proposed alignment loss.\n\nWe agree that training on depth data only poses an interesting and challenging problem setting. In order to directly learn the joint distribution, image-based representation methods, such as EG3D [7], could be employed to learn meaningful shape priors from RGB-(D) data. These could be combined with depth and pose estimation like NOCs [8]. We will address this in the limitation section.\n\n\n\n**4. Context-learning via attention**\n\nThanks for pointing us to this relevant work, we will include it in the related work section.\n\n**5. Surface alignment of Gaussians**\n\nIn our experiments, we observed that the intermediate 3D Gaussians usually align well with the object’s parts (see Fig. 1 and Fig. 2), leading to sampled points that are close to the surface.\nFor future work, it would be interesting to integrate surfel-based representations [9, 10] as promising alternatives for binding Gaussians to the surface of the object (2DGS [11]). \n\n\n\n**6. Varying number of scene objects**\n\nOur model can in fact handle a varying number of objects in the scene, because the dot-product attention in our ISA formulation is agnostic to the number of objects during train and inference time. In our experiments, we have not experienced negative impacts of false positives/negatives from the image detection backbone, however it is possible to perform NMS on the reconstructed 3D shapes to further filter out false detections.\n\n**7. Object pose normalizations**\n\nDuring our experiments, we did not try different normalization factors for the object pose parameters. We will include a comparison with different, e.g., unnormalized, object poses in the final paper.\n\n**8. Convergence of disentangled of pose and shape**\n\nThank you for the interesting question. Investigating the convergence properties of the presented disentangled pose and shape formulation is indeed a meaningful addition. \nWe will perform a comparison in a controlled environment, e.g., using synthetic data with full 3D ground truth such as 3D-FRONT [12], and include it in the final paper.\n\n\n**9. Comparison with SOTA single-object methods**\n\nWe show comparisons for single object reconstruction with InstPIFU [13] in Figure 8.\n\n**10. Can it be applied to part-level object reconstruction**\n\nIn our experiments, the scaffolding 3D Gaussians align well with individual objects parts. For a part-level reconstruction, each vertex of the output shape mesh can be assigned to its nearest 3D Gaussian to get a part-level segmentation of the object. We will include a visualization of that in the final paper.\n\n**References**\n\n- [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022\n- [2] Koo et al. - “SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation”, ICCV 2023\n- [3] Nie et al. - “Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image”, CVPR 2020\n- [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021\n- [5] Sun et al. - “Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”, CVPR 2018\n- [6] Song et al. - “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite”, CVPR 2015\n- [7] Chan et al. - “EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks”, CVPR 2022\n- [8] Wang et al. - “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation”, CVPR 2019\n- [9] Pfister et al. - “Surfels: Surface elements as rendering primitives”, SIGGRAPH 2000\n- [10] Zwicker et al. - “EWA volume splatting”, Visualization 2001\n- [11] Huang et al. - “2DGS: 2D Gaussian Splatting for Geometrically Accurate Radiance Fields”, SIGGRAPH 2024\n- [12] Fu et al. - “3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics”, ICCV 2021\n- [13] Liu et al. - “Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes”, ECCV 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAyrMUvyuV", "reviewer": "Reviewer_BXMX", "summary": "The task is to perform 3D scene reconstruction from a single RGB views, such that they can output both scene object poses and geometries. Prior work cannot jointly output both poses and geometries, one design choice that is validated by this work to be effective. Method-wise, extending on SALAD, this work introduces a diffusion model that learns a generative scene prior that can capture contextual information across scene elements. To achieve joint learning, they introduce a alignment loss such that they only need to single view supervision. The author conducts extensive experiments to justify their design choices.", "questions": "1. Could you clarify on how the proposed framework resolve with scale ambiguity given the single view setup?\n\n2. Writing needs more polish. For example, Equation 1 seems incorrect (feel free to clarify in the rebuttal). Based on the description in Line 121, RGB image I seems to be given, which indicates it is one item to be conditioned on. And in Line 150, missing parenthesis for (See Sec. 4.2\n\n3. Could you clarify on how the gradient being back-propagated through point sampling (Line 190)? \"We directly sample m = 1000 points p(j,l) ∼ N (μj , Σj ) per Gaussian Gˆi,j resulting in a shape point cloud Pi = {p(j,l)|j ∈ {1,...,g},l ∈ {1,...,m}}.\" In SALAD, it has another cascaded diffusion Transformer that condition on extrinsic and intrinsic parameters to get the part-level point cloud from Transformer. But I am not sure if the process is the same in this paper. \n\n4. Could you clarify how the texture of the 3D objects are learned? The qualitative results show objects with color but there is no supervision on the color.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of the surface alignment loss (Sec 3.6) provides a nice solution to jointly improve object poses and geometry estimation. This kind of supervision only needs image level supervision (no need for 3D supervision). \n\n2. The experiments on comparison and ablation is informative, which reasons well the key design decision made in this paper. \n\n3. The qualitative demo is multifaceted, which makes it easier to evaluate their pose and geometry quality.", "weaknesses": "1. Minor concern is that the paper over-claim a bit on the contribution. Despite phrasing as 3D scene reconstruction (as is mentioned in teaser, abstract and intro) from single view, the proposed framework in the method actually only reconstruct poses and geometries of foreground objects in the image without the background part, e.g., ground or wall. \n\n2. Global shape prior cannot always fit the observed object perfectly, with different local details, e.g., Fig.3 shelf. This could accumulate error in the downstream alignment. \n\n3. Furniture kinds of 3D objects, e.g., those in ShapeNet, are simple, which usually is easier to get good shape reconstruction. How would the proposed framework work on more intricate 3D objects, e.g., plants, trees?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reconstructing background objects**\n\nIndeed, structural elements can be considered part of holistic scene reconstruction.\nIncluding structural elements in the scene prior can certainly be beneficial and can improve the coherence of the scene. \nWe will extend our formulation with the structural elements of the scene and train a layout estimator to reconstruct them. We will report our findings in the final paper - thanks a lot for this suggestion!\n\n\n\n**2. Global shape prior**\n\nThanks for the comment. While our model produces clean shape modes even under heavy occlusions\ncompared to blurry or incomplete shapes of prior work, our learned shape prior does not necessarily match the input image perfectly. \nIn our alignment loss formulation, we sample points from the 3D Gaussians, which represent the coarse shape of the object. Incorporating the per-Gaussian intrinsics, which encode local details, into the loss formulation is an interesting experiment.\n\n\n\n**3. Reconstruction of natural objects, e.g. plants, trees** \n\nThanks for the interesting question. In Fig. 3, 5 and 9, we show that our geometry-based model is able to reconstruct certain fine details, such as individual braces at the back of a wooden chair or thin table legs. However, for natural, intricate objects such as plants, geometric approaches generally struggle to reconstruct very fine details. To study the limits of our approach, we will include a comparison with an increased number of 3D Gaussians in the shape model. \nTo further mitigate this issue, our approach can reconstruct the geometry of the object, and additional fine details, such as individual leaves, can be further represented by local textures, as has been done in computer games, or with 3D Gaussian splats. \n\n**4. Resolving scale ambiguity**\n\nSince our model learns object poses and scales together with a strong scene prior, our model naturally reasons about common scene scales and hence is less prone to ambiguities of individual objects given the scene context.\n\n\n\t\n**5. RGB image input**\n\nThanks for the comment. We will go through the manuscript thoroughly and improve the writing and fix minor typos. \nEq. 1 defines the overall problem setup of reconstructing multiple objects from a single RGB image ($\\mathbf{I}$, L121). Instead of conditioning on the input image directly, we use an image feature extractor $\\Theta_i$ and instance segmentation on the input image to extract per-detection image features, which are then used to condition the diffusion model ($\\text{feat}_i$, L135, Eq. 6). \n\n**6. Gradient backpropagation through point sampling**\n\nFor sampling we use `torch.distributions.MultivariateNormal` function with the estimated per-Gaussian centers $\\mu$ and covariance matrices $\\Sigma$. While the sampling itself is not differentiable, we can compute the gradients on the sampled points. \nDuring joint training with the proposed efficient alignment loss, we omit the additional denoising of the intrinsic vectors and the costly rejection of point samples using the shape decoder. Hence, our point sampling formulation is different from the shape decoding of SPAGHETTI [1] or SALAD [2].\n\n\n\n**7. Textures of objects**\n\nOur model does not predict object colors or textures, however, this is an interesting future direction.\nFor visual clarity, we color each object according to its semantic class - we will indicate this in the captions of the figures.\n\n**References**\n- [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022\n- [2] Koo et al. - “SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation”, ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their constructive and valuable feedback. We are pleased that *all* reviewers recognized the soundness of our approach for the challenging problem of single-image 3D reconstruction. 

Our method was highlighted as solid (scCV) and efficient (W8nX) by appropriately employing strong diffusion priors (W8nX, scCV). The proposed surface alignment loss was well-received (scCV, U6My, BXMX), as was the use of anisotropic Gaussians for shape representation (U6My). These contributions led to results described as “multi-faceted” (BXMX), “good” (scCV) and  “promising” (U6My). Additionally, we are glad that our experimental setup and ablation studies were received as “comprehensive” (scCV), informative (BXMX) and “clean” (W8nX).

Meta-review: The paper received four reviews from expert reviewers. The rebuttal has cleared up most remaining questions, and in the end, all reviewers retain their positive rating and recommend acceptance. 
The reviewers highlight the novelty of the formulation, the clarity of the experiments and the writing of the paper.
The AC strongly encourages the authors to include all additional results, comparisons, and clarifications in the appropriate sections of the paper.
Decision: Accept (poster)