Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors
Authors: Ziwei Liao
Binbin Xu
Steven L. Waslander
Venue: CoRL 2024 (2024)
Keywords: Mapping
Objects Reconstruction
Pose Estimation
Diffusion

Abstract: Object-level mapping builds a 3D map of objects in a scene with detailed shapes and poses from multi-view sensor observations.   Conventional methods struggle to build complete shapes and estimate accurate poses due to partial occlusions and sensor noise. They require dense observations to cover all objects, which is challenging to achieve in robotics trajectories.  Recent work introduces generative shape priors for object-level mapping from sparse views, but is limited to single-category objects. In this work, we propose a General Object-level Mapping system, GOM, which leverages a 3D diffusion model as shape prior with multi-category support and outputs Neural Radiance Fields (NeRFs) for both texture and geometry for all objects in a scene. 
GOM includes an effective formulation to guide a pre-trained diffusion model with extra nonlinear constraints from sensor measurements without finetuning. We also develop a probabilistic optimization formulation to fuse multi-view sensor observations and diffusion priors for joint 3D object pose and shape estimation. 
Our GOM system demonstrates superior multi-category mapping performance from sparse views, and achieves more accurate mapping results compared to state-of-the-art methods on the real-world benchmarks. 
We will release our code and model upon publication.

Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors
Ziwei Liao
Binbin Xu
Steven L. Waslander
Institute for Aerospace Studies & Robotics Institute
University of Toronto
(ziwei fiwei.liao, binbin.xu, xu, steven. www.laandeerroooootic..ooo utias. utoronto.ca ca
Abstract: Object-level mapping builds a 3D map of objects in scene with detailed shapes and poses from multi-view sensor observations. Conventional methods struggle to build complete shapes and estimate accurate poses due partial occlusions and sensor noise. They require dense observations to cover all objects, which is challenging achieve in roboties trajectories. Recent work introduces generative shape priors for object-level mapping from sparse views, but is limited to single-category objects. In this work, we propose a General Object-level Mapping system, GOM, which leverages a 3D diffusion model as shape prior with multi-category support and outputs Neural Radiance Fields (NeRFs) for both texture and geometry for all objects scene. GOM includes an effective formulation to guide a pre-trained diffusion model with extra nonlinear constraints from sensor measurements without finetuning. We also develop probabilistic optimization formulation fuse multi-view sensor observations and diffusion priors for joint 3D object pose and shape estima- tion. GOM demonstrates superior multi-category mapping performance from sparse views, and achieves more accurate mapping results compared to state- of-the-art art methods on the real-world benchmarks. We will release our code: htgggg/twooooool..llaaaaeeee///e
Keywords: Mapping, Objects Reconstruction, Pose Estimation, Diffusion
Introduction
Object-level mapping [1, 2, 3, 4, 5, 6, 7, 8, 9] builds a 3D map of multiple object instances scene, which is critical for scene understanding [10] and has various applications robotic manipu- lation [11], semantic navigation [12, 13] and long-term dynamic map maintenance [14]. addresses two closely coupled tasks: 3D shape reconstruction [15, 16] and pose estimation [17]. Conventional methods [18, 19, 20] approach these tasks from a perspective of state estimation [21], solving an inverse problem where low-dimensional observations (RGB and Depth images) are used to recover high-dimensional unknown variables (3D poses and shapes) through known observation process (e.g., projection, and differentiable rendering). However, these methods require dense observations (e.g., hundreds views for NeRF [18]) to fully constrain the problem. In roboties or AR appli- cations, obtaining such dense observations challenging due to limitations in the robot's or user's observation angle and occlusions in clustered scenarios. Therefore, it is crucial develop methods that can map from sparse (fewer than 10) even single observations.
Human vision can infer complete 3D objects from images despite occlusions by using prior knowl- edge the objects, which represents the prior distributions of the shapes specific categories, such as chairs, based on thousands of instances observed in daily life. aim to introduce generative models [22] as providers of prior knowledge constrain the 3D object mapping. Generative mod- els have demonstrated impressive abilities to generate high-quality multi-modal data by learning distributions in large-scale datasets, including texts [23]. images [24], videos [25]. and 3D mod-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

els [26, 27, 28, 29]. Previously, researchers have explored methods, to introduce generative priors to constrain inverse problems in the image field, such as restoration [30] and controllable genera- tion [31]. However, applying generative models to 3D inverse problems remains an open problem.
Two types of generative priors have been explored in the context of 3D inverse problem. (1) Im- age Generation Models as 2D Priors. Researchers [32, 33] use image generation models [24] to generate -iew images to optimize NeRF parameters for shape reconstruction. However, these models suffer from poor 3D consistency (e.g., multi-face problem [32]) as it is difficult to preserve 3D consistency and identity in image generation models. Additionally, they require dense observa- tions, which limit computation efficiency (e.g., taking hours per object). Achieving high-quality re- sults often necessitates complex modifications, such as retraining the generation model [34, 35, 36]. Furthermore, these methods typically focus on shape generation from texts, rather than recon- struction and pose estimation from multi-view real observations.
(2) 3D Shape Generation Models as 3D Priors. Recent object-level mapping systems [3, 4] directly inject 3D priors from a 3D gener- ation model like DeepSDF [27] by jointly optimizing pose and shape (represented in learned latent space). Compared to 2D priors, which provide only partial and low-dimensional shape information, 3D priors achieve better 3D consistency and completeness. However, limited by the capacity of an autodecoder model [27], they support only single-category objects, and model only the geometries of shapes without textures, limiting the generalization ability.
We aim to propose an object-level mapping system that introduces valuable generative priors for diverse objects in any scene. Our system is therefore designed to be: (1) General: It supports multi- category objects by using a pre-trained generative model as shape prior; (2) Flexible: It supports multi-view, multi-modality observation signals (e.g., RGB, depth images and pointclouds), without the need for finetuning the generative model. Retraining large-scal: scale 3D generative models a process that is both time-consuming and resource-intensive. The attributes of being General and Flexible essential when integrating into complex robotics system for downstream tasks.
Recently, diffusion-based 3D generative models trained on millions of 3D objects and supporting multiple object categories have been released [37, 29]. Additionally, new public large-scale 3D object datasets [38, 39] are becoming available. However, employing diffusion-based models to solve inverse problems remains theoretically unsolved, since diffusion models use score functions to model distributions [24] and cannot directly output probability densities like the VAF-variant models [27]. In this paper, we make key theoretical contribution by proposing lation to fuse 3D diffusion priors with 3D sensor observations. As far as we know, we are the first to explore 3D object-level mapping with multi-category 3D diffusion model for both pose and shape estimation. Our contributions are summarized as follows:
effective formu-
We propose a 3D object-level mapping system from sparse RGB-D observations, leverag- ing pre-trained diffusion-based 3D generation model as a multi-category shape prior;
We propose an effective optimization formulation that jointly fuses information from multi- view sensor observations and diffusion priors;
We demonstrated our superior multi-category mapping performance over the state-of-the- art baselines through extensive experiments on the real ScanNet dataset.
2 Related Work
2.1 3D Object-level Mapping
3D scene understanding is crucial for robots to perform high-level object- tasks such as navi- gation and manipulation [10]. Early methods in the field of SLAM and SfM [40] built 3D scene mod- els from dense multi-view observations, leveraging multi-view geometric consistency [41]. How- ever, they used single model represent both foreground objects and backgrounds, unaware of separate instances [16, 42, 43]. Object-level mapping addresses the problem of estimating objects' shapes and poses independent instances. SLAM+ [1] was pioneering in this area but requires

---

database of CAD models and could not handle unseen objects outside this database. To generalize to unseen objects, researchers have used compact object representation such as quadrics [6, 7] or cuboids [5], but these approaches lack detailed shape information and texture. Methods [44, 2, 45] produce dense representations, such as Signed Distance Function (SDF) fields, and Neural Radiance Fields (NeRFs) [46]. However, they require dense observations to constrain the high-dimensional shape variables, and struggle achieve complete and accurate results in the presence of occlu- sions and noise. Differing from the methods above, we develop our method to address object-level mapping from sparse observations, which is more relevant for robotic applications.
2.2 Generative Models as Shape Priors
Introducing generative models as prior constraints for 3D object pose and shape estimation is an open problem. Currently, the literature explores two kinds of priors. (1) Prior for Specific Single- category. Early 3D generative models [27, 28, 47] train separate models for single-category objects. Object-level mapping systems [8, 3, 4, 48, 9] introduce DeepSDF-like [27] generative models as pri- ors to constrain object shapes. They estimate complete shapes and poses from sparse observations under occlusions, however, limited by the shortcomings of these generative priors, they are limited to a single category. (2) General Prior for Multi-categorics. Recent research, including DreamFu- sion [32] and Magic3D [33], optimizes 3D shapes represented by NeRF through multi-view images from an image generation model [49]. However, since the prior is inherently 2D, has limited 3D consistency, leading to multi-face artifacts and dientity-preservingg problems [32]. Although they generate multi-category 3D shapes from texts, they can not reconstruct from real images esti- mate object poses to form map with multiple instances. In contrast to the methods above, we aim to leverage a 3D prior model with multi-category ability for improved 3D consistency. Recently, diffusion-baar based 3D generative models [37, 29] have become available, such as Shap-E [29] which is trained on millions of objects in thousands of categories. However, unlike VAB-variant models [27], diffusion-barr based models are not straightforward integrate into object-level mapping system. We discuss this further in the next section.
2.3 Pretrained Diffusion Models as Optimization Constraints
Training a diffusion model on millions of 3D objects is time-consuming [37, 29]. We aim to lever- age pre-trained diffusion model without finetuning. One method to control pre-trained diffusion model through conditional generation. introduces extra constraints to guide the original diffu- sion process, and has been investigated in image generation [31], restoration [30], and 3D human pose generation [50]. However, a conditional generation formulation is not flexible enough to esti- mate extra variables (i.e. poses) beyond the variable modeled by the diffusion model (i.e. shapes). Another method is to leverage the diffusion model as constraint in optimization, which is flexible to introduce extra variables and constraints. However, unlike VAE-variant methods that have ex- plicit latent spaces [27, 3, 9], the latent space of diffusion models still under investigation [51]. Diffusion models output scores instead of densities and require a complex sampling process to gen- erate sample [24], making it nontrivial formulate joint optimization with additional variables. Recently, some work optimization with diffusions, in the fields of images [51, 52], 3D human poses [53, 54], and medical CT reconstruction [55]. Most similar to ours, Yang al. [56] optimized a NeRF with single-categon diffusion model, but did not include pose estimation. Differing from the work above, we are the first to use a pre-trained multi-categories diffusion model in an optimiza- tion framework for both 3D object pose and shape estimation.
3 Methods
3.1 Framework Overview
Our system, visualized in Figure/ 1, estimates 3D object shapes and poses in scene using two information sources; multiple RGB-D observations and generative shape prior model. Following
3

---

Inputs
Optimization with Diffusion Prior and Observations
Outputs
Pose Gradients
3D Pose
Trunsform
Render
NeRF Parameters
RGB-D
Rendered Images
World Coordinate
Shape Gradients
Text Prompt
''A Chair")
3D Diffusion Prior
Prior Condition
3D Diffusion Model
Loss
Predicted Noise
Gaussian
Distribution
Figure 1: Framework overview. We propose object-level mapping framework that fuses both multi-view observations and a pre-trained diffusion shape prior model. generalizes to multi- categories objects, and multiple multi-modalities observations without the need of fine-tuning.
Shap-E [29], object shapes are represented as NeRFs with SDFs, modeling both the geometry and texture, which can be rendered into images or generate meshes.
3.2 3D Objects Shapes and Poses Representations
We aim to build map of the 3D objects O in scene. For each object, estimate its 3D model (a NeRF), Q,, in the object canonical coordinate frame, Fo, and relative pose, Tow, from the world coordinate frame, J F,,, to the object coordinate frame H.O.
Pose Representation. pose transformation with scale, tion vector, 123, rotation vector, so(3), and scaling vector, s 13:
RAx4, constructed from a transla-
T exp
1 [iag (s)
[]
i
0"
where exp (.) is the exponential mapping from the Lie Algebra to the corresponding Lie Group. The operator (.)* converts vector to skew-symmetrie matrix. Thus, we can represent the pose with 9-DoF vector:
[t. o. s| Rº. As mapping system, we assume camera poses, Two E SE(3), are known, e.g., by off-the-shelf SLAM methods [43], or by robot kinematics [57].
Shape Representation. We parameterize the object's shape with Neural Radiance Field (NeRF) [18] through neural network fe(-) with weights O For any given 3D) point, it outputs its density, a, color, and illumination, i, and can be rendered into RGB and depth images. Follow- ing Shap-E [29], it further outputs SDF values, and can generate meshes via Marching Cubes:
O. é, s fe(xx d)
(2)
where x E Rd is the given 3D coordinate and E R is the viewing direction.
3.3 Leveraging Generative Models as Shape Priors
Common 3D objects, such as chairs and tables, exhibit diverse shapes and textures. This translates to high-dimensional NeRF parameters θ To further constrain the process, propose leverage generative model with learnable parameters B, denoted by g-(-), to learn approximate posterior distribution P(OIC)) of the true posterior distribution P(O[C). Here C' represents the conditioning information, e.g., image text prompt. The generative model ga(-) takes this conditioning information, C, as input and outputs conditional distribution of the NeRF parameters O;
p(o|C) (C)
33)

---

Shape Gradients
Pose Gradients
Step
Step N
Generation
Prompe ["a chair") ]
Emply
Diffusion Prior
Mapping
Obstrvations
In State
Trajecoory
Optimized State
An illustration of the gradient fields and optimization process. The gradient fields from two sources, a diffusion prior originally for Generation, and multi-view non-linear observation straints, are effectively fused into a shape and pose optimization formulation for Mapping.
Figure 2:
Diffusion Models. We leverage Shap-E [29], a pre-trained diffusion model as ga(.). which has been trained on millions 3D objects. As shown in Figure 2, to generate a sample O E P(OIC), we start from a random noise Or and refine through a series of diffusion steps e-) for timestamp t = T,,.... 1, according noise schedule. This process produces the final noise-free sample Op:
1 = c, C,4)
3.4 Optimization with Diffusion and Extra Observations
To update θ with multi-view RGB-D observations and further estimate a new unknown pose vari- able T. we propose an optimization formulation that integrates additional observations into the pre-trained diffusion model, as shown in Figure 2. We refer T as Tow without further notice. Given M observation frames { and a condition C, we aim to estimate a Maximum Like- lihood Estimation for the unknown variables pose T and shape O, from joint distribution of P(T, θ Fi,..., C), By derivation (Proof provided in Appendix), get convenient form for numerical optimization:
í Ô = arg max T,O
P( + log P(oIC)
(5)
Observations
Diffusion Priors
Diffusion Priors. The term log P(e|C) measures the likelihood of current θ under condition C' by the diffusion model. However, diffusion models can not directly output the probability den- sity [24]. We derive a method to calculate gradients from diffusion priors for optimization, following a variational sampler [58], originally designed for controlling image generation. A diffusion model trained to predict the noise in noisy variable θ, at timestamp 4, under condition C:
min Er,e [|ea (O,, C, ell2, I (E,,T) e (O,
(6)
We fix the weights of the preiteained diffusion model, and it to get the gradients De log P(O|C) of the current shape O. First, we add noise to the current shape parameter O to get a noisy O with the predefined noise schedule of the diffusion model at timestamp t:
θ a + ore, EE
(7)
where and are constants of the schedule, and c is a randomly sampled noise. Then, we predict the added noise and evaluate the predicted error (9 C,t) with the diffusion model:
A5 = o (0, C,t) = (a + ore, C,t) -, (0,
(8)
(9)
To eliminate the timestamps f, we randomly sample K times from a uniform distribution (4( and take an expectation:
A Σ CO,,,, 1 4EU(0,T)
(10)
5

---

RGB-D D Reference View
Rendered View
Novel Views
Mech
Methods
(Ours)
vMap
GOM (Ours)
Figure 3: Prior Effectiveness: Using priors, GOM (Ours) can render higher-quality 3D consistent views and generate 3D meshes with fewer artifacts compared to vMap. Results are based 10 RGB-D views.
As illustrated in Figure 2, the predicted noise of diffusion model naturally models the gradient field [24] of the prior distribution. Thus, we take the predicted error A,(O,, C)) as the gradient to update & under a prior condition C::
V lo P(eIC) = (e(e,C)
(11)
We also present ablations with other alternatives in the Appendix.
Observations. The term Σ log P θ) measures the geometric and texture consistency from multi-view observations. For each observation F, {[,, D.. Pwc. we use differentiable render- ing [18] to render RGB image 1,T,ow, O, Pwc.) ) and depth image D2 O, Pwc,) with the camera projection matrix P PwC.- We then formulate an L2 loss:
log P(F θ) x 14, 112 + D.: D,12
(12)
This term can propagate gradients to update both poses Tow and shapes θ shown in Figure 2. Iterative Update. We first coarsely initialize the unknown variables shape O, and pose To. Then, we iteratively refine for steps considering a prior condition C and observations (F;}:
, T, =
C, (FF)), for j = l,....
(13)
The Rini process has two steps. First, a Prior Step to update θ with diffusion using Eq. 11:
3 = - + Ve log P(o|C)le=e,-, 05- + (0A
(14)
where 0 is a middle shape fused with prior. Then, an Observation Step to update both and
, = 0 + Ao Σ (F)
(15)
T = T Ao Σ VT log P(F 0)
(16)
where Ao and Ao are hyperparameters to balance the influence of priors and observations. Finally, we obtain the refined T and ,, after steps optimization.
Experiments
Datasets. We conducted experiments utilizing the ScanNet [59] dataset, which comprises RGB-D scans of real indoor scenes. It presents significant challenges object mapping methods due to real-world occlusions, clusters, blurs, and sensor noise. We use ground-truth shape annotations of ShapeNet [60] meshes provided by Scan2CAD [61] for evaluation. We focus on objects that have

---

View
Pointoloud
Shap-E (Toxt)
Shap-E (Image)
BundleSDF
vMap
(Ours)
Opposite View
Opposita View
Figure 4: Mapping Performance: GOM (Ours) outputs 3D object shapes and poses that align with the inputs and further completes the occluded parts. GOM can also generalize to multiple categories. Results are based on 10 RGB-D views.
minimum of 10 available views, where at least 50% the object is visible in each view. This results in total of 206 scenes, encompassing 1036 object instances across 7 categories. These categories include chairs, tables, cabinets, sofas, bookshelves, beds, and bathtubs.
Baselines and Metrics. We compared our method against the following state-of-art RGB-D object- level mapping methods: (1) Shap-E+/CP, which uses Shap-E [29] to generate a NeRF model condi- tioned on an imput image, and then matches RGB-D inputs with ICP [62]; (2) DSP [3], which opti- mizes objects shapes and poses with DeepSDF [27] as shape priors; (3) vMap [45], which optimizes
an independent NeRF [18] from scratch with a fixed pose for each 3D object; (4) BundleSDF [20], which optimizes objects' NeRF [18] and poses for tracking. To ensure fair comparison,
utilize
the ground truth camera poses, segmentation masks, and data association provided by the dataset across all systems. For pose accuracy, we use Intersection over Union (IoU) to evaluate the bound- ing boxes of the estimated and ground truth shapes in world coordinates. For shape accuracy, we calculate Chamfer Distance (CD) after transforming objects into world coordinates.
Implementation Details. We optimize for 200 steps, with fixed learning rate of 0.5. For every 2 steps, diffuse once with timestamp randomly sampled from the Shap-E diffusion schedule. We use the text prompt "a [category]" as the condition for Shap-E, where the category comes from the segmentation mask. We randomly initialize the shape variable from a Gaussian distribution and a coarse pose using ICP matching between the input depth point cloud and an average shape of the corresponding category, generated from Shap-E with the prompt "a {category]". The same ICP pose initialization method is applied for DSP and vMap to ensure fair comparison. To test the potential performance of vMap [45], replace its iMAP representation with NeRF [18] and run more optimization iterations until it converges. We provide more details in the Appendix.
4.1 Multi-view Mapping Performance
We present the mapping performance with 1, 3, and 10 RGB-D views chairs in Table 1. We show qualitative results in Figure 3 and Figure 4. Compared to vMap [45], which lacks priors as additional constraints, our approach yields significant improvements. This is particularly evident with fewer views, where observations are compromised by occlusions and noise. In these instances, prior information serves as valuable additional constraint. Compared to DSP [3), our method pro- duces more detailed shapes with textures, and outperforms it in both 3 and 10 views. DSP achieves better CD in single-view cases due to smaller latent space with stronger constraints specifically trained on Chairs. However, it can not generalize other categories. Shap-E can generate reason-
7

---

Views
1
3
10
Methods
IoU
CD +
1oUt
CD 4
loU t
CD
Shap-E [29] ICP
0.337
0.169
DSP [3] vMap [45]
0.202
0.173
0.171
0.370
0.282
0.384
0.184
0.413
0.158
GOM (Ours)
0.401
0.267
0.409
0.166
0.429
0.157
Table I: Mapping performance from I, and 10 RGB-D views ScanNet Chairs dataset. Since
Shap-E can only take in one image, we present a single-view result for Shap-E+ICP.
Tasks
Methods
Chairs
Tables
Cahinets
Sofas
Bookshelfs
Beds
Bathtubs
Average
Shap-E [2app-E[9]] ICP
0.169
0.430
0.268
0.162
0.392
0.813
1.91
0.437
DSP [3]
0.171
Mapping
vMap [45]
0.158
0.249
0.200
0.220
0.315
0.729
1.53
0.350
GOM (Ours)
0.157
0.173
0.194
0.136
0.258
0.500
1.32
0.290
Shap-E [29] Image
0,206
0.184
0.101
0.186
0.084
0.266
0.139
0.188
Recon
Shap-E (29) Text
0.211
0.189
0.260
0.187
0.143
0,362
0.207
0,211
GOM (Ours)
0.102
0.119
0.096
0.137
0.098
0.132
0.1
0.107
Table 2: Mapping performance 7 object categories on ScanNet dataset from 10 RGB-D views for
DSP, Maap, and GOM (Ours). Since Shap-E can only take in one image, we present a single-view result for Shap-E+ICP and Shap-E Image for reference. Metrics: Chamfer Distance (CD).
able chairs from image conditions, as shown Figure 4. When combined with ICP, it exhibits a low CD, but struggles faithfully reconstruct objects. Its performance decreases significantly for large objects such as tables, beds and bathtubs, as shown in Table GOM (Ours) achieves better IoU and can continue to improve when provided additional observations. We qualitatively compare to BundleSIFF [20] in Figure 4 and we see that GOM delivers better mapping results as it better completes the unseen parts of objects with prior information.
4.2 Multi-categories Performance
We benchmark the open-vocabulary mapping performance for objects across 7 categories using 10 RGB-D views in Table 2. Our approach outperforms the naive use ICP matching with shapes generated from Shap-E. The ICP matching stuck to local minimum for the pose with a fixed shape. DSP [3] a single-category system that requires separate network weights for each category, and since only the chair model is officially available, we present its performance solely for this category. This limitation also restricts the system's applicabilityy. Compared to vMap [45], our approach yields an improved average CD and enhancements across most categories. This highlights the efficacy of our optimization formulation in integrating prior information with observations. We further demonstrate the Reconstruction performance in Table 2 (Recon), given the ground truth object poses. Our performance significantly improves compared with Mapping, demonstrating the potential for enhanced performance when more accurate initial poses are available. Our approach outperforms both the text and image-conditioned Shap-E model. This underscores that additional observations enhance a pre-trained generative model for reconstruction tasks. We present more experiments the CO3D dataset and analysis in the Supplementary Materials.
5 Conclusion
We present a General Object-level Mapping system, GOM, leveraging a pre-trained diffusion-based 3D generation model as shape priors. We propose an optimization formulation to couple multi-view RGB-D observations, and diffusion priors to constrain shapes and poses for 3D objects. We achieve state-of-the-arn mapping performances among multiple categories without further finetuning. Fur- ther exploring the diffusion shape priors into inverse problems with more constraints, e.g., temporal constraints for dynamic tracking, and spatial constraints for complete SLAM, and the application to downstream robotics tasks such robotics manipulation, will be valuable future work directions.
8

---

References
[1] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, and A. J. Davison. Slam++: Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1352-1359, 2013.
[2] B. Xu, W. Li, D. Tzoumanikas, M. Bloesch, A. Davison, and S. Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In 2019 International Conference Robotics and Aulomation (ICRA), 2019.
[3] Wang, M. Rünz, and L. Agapito. Dsp-slam: Object oriented slam with deep shape priors. In 2021 International Conference on 3D Vision (3DV), pages 1362-1371. IEEE, 2021.
[4] E. Sucar, K. Wada, and A. Davison. Nodeslam: Neural object descriptors for multi-view shape reconstruction. In 2020 International Conference on 3D Vision (3DV), pages 949-958, IEEE, 2020.
[5] S. Yang and Scherer. Cubeslam: Monocular 3-d object slam. Robotics, 55(4):925-993) 2019.
IEEE Transactions on
[6] L. Nicholson, M. Milford, and N. Sünneerhauf. Quadricslam: Dual quadrics from object detec- tions as landmarks object-oriented slam. IEEE Robotics and Automation Letters, 4(1):1-8, 2018.
[7] 7. Liao, Y. Hu, J. Zhang, X. Qi, X. Zhang, and W. Wang. So-slam: Semantic object slam with scale proportional and symmetrical texture constraints. IEEE Robotics and Automation Letters, 7(2):4008-401 2022.
[8] M. Runz, K. Li, M. Tang. L. Ma, C. Kong, T. Schmidt, Reid, L. Agapito, Straub, S. Love- grove, et al. Frodo: From detections to 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14720-14729, 2020.
[9] Z. Liao, J. Yang. J. Qian, A. P. Schoellig, and S. Waslander. Uncertainty-aware 3d object- level mapping with deep shape priors. 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024.
[10] M. Naseer, S. Khan, and F. Porikli. Indoor scene understanding in 2.5/3d for autonomous agents: A survey. IEEE access, 7:1859-1887, 2018.
[11] Y. Wu, Y. Zhang, D. Zhu, X. Chen, S. Coleman, W. Sun, X. Hu, and Z. Deng. Object slam- based active mapping and robotic grasping. In 202/ International Conference 3D Vision (3DV), pages 1372-1381. IEEE, 2021.
[12] D. Chaplot, D. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using gral-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247-4258, 2020.
[13] S. Papatheodorou, N. Funk, D. Tzoumanikas, C. Choi, B. Xu, and S. Leutenegger. Finding things in the unknown: Semantic object-centric exploration with an mav. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), 2023.
[14] J. Qian, V. Chatrath, J. Servos, A. Mavrinac, W. Burgard, S. L. Waslander, and A. P. Schoel- lig. Pov-slam: Probabilistic object-aware variational slam in semi-static environments. arXiv preprint arXiv:2307.004883 2023.
[15] C. Choy, Xu, Gwak, K. Chen, and Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628-644, Springer, 2016.

---

[16] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton. S. Hodges, D. Freeman, A. Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. Proceedings of the 24th annual ACM symposium on User interface software and technology. pages 559-568, 2011.
[17] Y. Hu, Fua, W. Wang, and M. Salzmann. Single-stage 6d object pose estimation. Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2930- 2939, 2020.
[18] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference computer vision, pages 405-42) Springer, 2020.
[19] Y. Jiang, D. Ji, Z. Han, and M. Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1251-1261, 2020.
[20] B. Wen, J. Tremblay, V. Blukis, S. Tyree, Müller, A. Evans, D. Fox, J. Kautz, and S. Birch- field. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 606-617, 2023.
[21] D. Barfoot. State estimation for robotics. Cambridge University Press, 2024.
[22] G. Harshvardhan, M. K. Gourisaria, M. Pandey, and S. S. Rautaray, A comprehensive survey and analysis of generative models in machine learning. Computer Science Review, 38: 300105, 2020.
[23] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[24] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[25] Ho, W. Chan, C. Saharia, J. Whang, Gao, A. Gritsenko, D. Kingma, B. Poole, M. Norouzi, J. Fleet, et al. Imagen video: High definition video generation with diffu- sion models. arXiv preprint arXiv:2210.02303 2022.
[26] Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and gen- erative models for 3d point clouds. In International conference machine learning, pages 40-49. PMLR, 2018.
[27] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
(28) L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning reconstruction function space. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4460-4470, 2019.
[29] H. Jun and A. Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint 23X1v:2305.02463, 2023.
[30] Z. Shi and R. Liu. Conditional velocity score estimation for image restoration. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 179-188, 2024.
10

---

[31] C. Meng, Y. He. Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01072 2021.
[32] B. Poole, A. Jain, T. Barron, and Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 1e022299999900 14988, 2022.
[33] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[34] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. lero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[35] N. Ruiz, Y. Li, V. Jampani, Pritch, M. Rubinstein, and Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[36] M. Liu, C. Xu, H. Jin, L. Chen, M. Varma T, Z. Xu, and H. Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[37] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprini arXiv:2212.0875:1 2022.
[38] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023.
[39] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre, E. VanderBilt, A. Kembhavi, C. Vondrick, G. Gkioxari, K. Ehsani, L. Schmidt, and A. Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. In A. Oh, Naumann, A. Globerson, Saenko, M. Hardt, and Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 35799-35813. Curran Associates, Inc., filef/llllllell /211ee//paaer//23////52 _nd
2023. URL https::/pproceeddnngg www.:eeeddddnne.nneneeeeeeee..ee wwwripe.enncmmm wttp::/sww.urrrrereeeeeeeeeeeeee 009-004877b67777/42222211eeeeee2 Datteppee-aaatttssaaaadaaeeeeeee Benchmarks pdf.
[40] C. Cadena, L. Carlone, H. Carrillo, Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the rebust-peroeption age. IEEE Transactions on roboties, 32(6):130--1333 2016.
[41] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge univer- sity press, 2003.
[42] T. Whelan, Salas-Moreno, B. Glocker, A. Davison, and Leutenegger. Elasticfu- sion: Real-time dense slam and light source estimation. The International Journal of Robotics Research, 35(14) 19:14)))697-7111 2016.
[43] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, (7(6):1874-118
2021.
[44] McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger. Fusion++: Volumetric object-level slam. In 2018 international conference on 3D vision (3DV), pages 32 -1. IEEE, 2018.

---

[45] X. Kong, S. Liu, M. Taher, and A. J. Davison. vmap: Vectorised object mapping for neural field slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 952-961, 2023.
[46] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1))99-106 106, 2021.
[47] Z. Erkoç, F. Ma, Q. Shan, M. Nießner, and A. Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. arXiv preprint arXiv:2303.1700 2023.
[48] B. Xu, A. Davison, and Leutenegger. Learning to complete object shapes for object-level mapping in dynamic scenes. In 2022 JEEE/RSI International Conference Intelligent Robots and Systems (IROS), 2022.
[49] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[50] H. Ji and H. Li. 3d human pose analysis via diffusion synthesis. arXiv:2401.08930. 2024.
arXiv preprint
[51] M. Kwon, J. Jeong, and Y. Uh. Diffusion models already have a semantic latent space. arXiv preprint 22Xiv:2210.10900 2022.
[52] A. Graikos, N. Malkin, N. Jojic, and D. Samaras. Diffusion models as plug-and-play priors. Advances in Neural Information Processing Systems, 35:14715-14728, 2022.
[53] Z. Jiang. Z. Zhou, L. Li, W. Chai, C.-Y. Yang, and J.-N. Hwang. Back to optimization: Diffusion- zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6142-6152, 2024.
[54] Lu, Lin, H. Dou, Y. Zhang, Y. Deng, and H. Wang. Dposer: Diffusion model as robust 3d human pose prior. arXiv preprint drXiv:2312.05541, 2023.
[55] Y. Song, L. Shen, Xing, and S. Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representattins, 2021.
[56] G. Yang, A. Kundu, L. J. Guibas, J. T. Barron, and B. Poole. Learning a diffusion prior for nerfs. arXiv preprint httv:2304.14473. 2023.
[57] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3665-3671, 2020.
[58] M. Mardani, J. Song, Kautz, and A. Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
[59] A. Dai, A. X. Chang, M. Savva, M. Halber, Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5888 5828,5839, 2017.
[60] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv1151.033022, 2015.
12

---

[61] A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nießner. Scan2cad: Learn- ing cad model alignment in rgh-d scans. In Proceedings of the IEEF/CVF Conference on computer vision and pattern recognition, pages 2614 2019.
[62] S. Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145-152, 2001.
[63] Reizenstein, Shapovalov, P. Henzler, L. Sbordone, Labatut, and D. Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference computer vision, pages 10901- 10911, 2021.
[64] T. Wu, Zhang, X. Fu, Wang. J. Ren, L. Pan. W. Wu, L. Yang, J. Wang, C. Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 803-814, 2023.
[65] Z. Liao and S. Waslander. Multi-view 3d object reconstruction and uncertainty modelling with neural shape prior. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5098-3101 2024.
[66] L. Bowman, N. Atanasov, K. Daniilidis, and G. Pappas. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1722-1729. IEEE, 2017.
[67] L. Schmid, M. Abate, Y. Chang, and L.. Carlone. Khronos: unified approach for spatio- temporal metric-semantic slam in dynamic environments. arXiv preprint #rXiv:2402.13817 2024.
[68] T. Ren, S. Liu, A. Zeng. J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint 1rXVv22401.1415. 2024.
[69] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[70] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https: //repo- sam inria. fr fr/fungraph//d http:aaaaaasssisppppaaaeaaeeeeee
13

---

6 Appendix
6.1 Experiments on CO3D dataset
To demonstrate the generalization ability across diverse categories, we conducted additional experi- ments on 10 categories from the CO3D dataset [63]: Toy Truck, Bench, Donut, Broccoli, Toy Train, Apple, Teddy Bear, Hydrant, Book, and Toaster. We compared our results with the baseline method vMap [45], as shown in Figure Our approach consistently outperformed the baseline, benefiting from the incorporation of generative priors from a single pre-trained model.
6.2 More Qualitative Visualizations
Visualization on More Categories on ScanNet dataset. We further illustrate the effectiveness of our approach in generalizing to a variety of categories through qualitative visualizations, as shown in Figure 6. Unlike the DSP method [3], which is based on single-category model, DeepSDF [27], our system is capable of supporting multiple object categories using single pre-trained diffusion model. Shap-E [29].
Scene-level Visualization. Figure 7 showcases a 3D map visualization of a scene on the ScanNet dataset, containing multiple objects from various categories, including four chairs, a sofa, and table. Each instance is independently reconstructed using 10 RGB-D views.
Visualization of Multi-view Inputs. We visualized 10 input images, 3D camera poses, and cor- responding object meshes in Figure 8, which includes two objects from the ScanNet dataset and one object from the CO3D dataset. The ScanNet dataset presents a challenging camera trajectory, with all inputs gathered from single direction relative to the objects. This scenario is typical in real-world applications such as robotics and augmented reality, where robot or user cannot easily circle an object. It also underscores the importance of sparse view mapping, where prior informa- tion is crucial for completing and providing reasonable estimates for occluded parts. An extreme case is illustrated where all camera views have small baselines, resembling single-view mapping. In the CO3D dataset, the cameras are intentionally positioned to circle the objects; however, we randomly sampled only ten views from this trajectory, resulting in a sparse coverage the objects. We demonstrate that our system can effectively handle all these situations.
6.3 More System Evaluations and Discussions
Performance on Single-view Inputs. Single-view mapping is particularly challenging due to its highly ill-posed nature. All systems experience significant performance losses when relying on only single view. Our main challenges stem from the high-dimensional shape representation required for NeRF, which models multiple categories of objects, encompassing both texture and shape. In contrast, DSP employs DeepSDF, SDF-based representation specifically trained on single cat- egory, focusing solely on shapes. This approach has much smaller parameter space and requires fewer constraints. However, this limited parameter space restricts its ability to generalize to other categories. Our method, on the other hand, can generalize across multiple categories using a sin- gle model and can continue to improve with additional observations. With the assistance of prior knowledge, we achieve effective enhancements method, vMap. Additionally, have parameter that allows to increase the weight of prior con- straints during optimization, enabling to place greater trust in the prior information for producing complete and reasonable mapping results.
single-view scenarios compared the baseline
Performance on Out-of-distribution Objects. Dealing with -ut-of-distribution objects a critical challenge for developing a general mapping system. this paper, we present formulation that leverages prior knowledge from pre-trained generative model. While we use Shap-E as a repre- sentative example at the time of writing, it's important note that method is not specifically tailored to Shap-E but applies to broader class of diffusion models. The field of 3D generation
14

---

RGB a Beferance View
Randered Viena
RGB-O Tlaherncce View
Randered Vieas
Methodk
GOW (Ouns
GOM (Our)
MMap
GOM (Durs]
WWpp
GOM
(Ouns)
+Map
Figure 5: Effectiveness of Priors Across 10 Categories on the CO3D Dataset: Toy Truck, Bench, Donut, Broccoli, Toy Train, Apple, Teddy Bear, Hydrant, Book, Toaster, compared vMap. The results are hased on 10 RGB-D views.
models is rapidly evolving, with larger datasets (e.g., ObjaverseXL [39], OmniObject3D [64]) and more powerful generative models emerging.
Our proposed formulation allows for seamless integration of more advanced models without requir- ing any fine-tuning, enabling plug-and-play approach. We demonstrate that the current Shap-E model is effective for diverse range of objects, as shown by our results both ScanNet and CO3D. While generative models may not provide detailed structural constraints for jects, they still offer valuable general priors, such as completeness and smoothness. This is illustrated in the experiment in Figure 9, where a prompt like "an object" yields useful results.
ob-
Our system effectively combines observations and prior constraints to identify an optimal solution from both sources. When the prior information is less accurate, to rely more observations for didtt---iitiittiion objects, tailored to specific applications. Fur- thermore, would be valuable future work to quantify the level of out-of outto--diitribution characteristics, such as uncertainty [9, 65], and to self-adjust the weights to enhance mapping performance.
adaptively adjust the weights
Text Prompts and Segmentation Methods. When selecting text prompts as prior conditions, our system demonstrates certain flexibility regarding the content of the text. In our experiments, provide an example using the prompt "a category," where category off-the-shelf object detection or segmentation algorithms. We also conducted new experiment to analyze the sensitivity to the quality of the text prompt, as shown in Figure 9, In this qualitative anal- ysis, we focused on a chair that was only observed from the front, using three types of prompts: (1) Precise label: "a chair"; (2) General label: "an object"; and (3) Two incorrect labels: "a table" and "a ball." Our findings indicate that our system can robustly leverage prior knowledge from varying levels of label specificity by optimizing both the observations and the prior information converge on solution that integrates both aspects. Particularly for occluded areas with insufficient obser- vational data, the prior knowledge effectively constrains the reconstruction to produce reasonable
be easily obtained from
15

---

RGB O Reference View
Rendered View
Novel Views
Mesh
Methods
vWlp
GOM (Oun)
viap
MMpp
(Ours)
(Ours)
Figure 6: Effectiveness of Priors Across Multiple Categories: Leveraging priors, method (GOM) can produce higher-quality, 3D-consistent views and generates 3D meshes with fewer artifacts com- pared vMap. The results are based 10 RGB-D views.
and smooth surface compared to vMap without any prior, even when using a general prompt like "an object."
In where incorrect labels were used, such "a table" and "a ball," some artifacts in- troduced into the structure (e.g., ball-like back part of the chair). However, the results were still significantly more coherent than the blurry areas observed in the baseline vMap. For the segmented areas, we utilized the provided "ground truth" masks from the ScanNet dataset, which are not man- ually annotated but projected from segmented point clouds. These masks may still contain artifacts and missing parts, and our experiments on ScanNet highlight the system's performance under im- perfect input conditions. Additionally, we have parameters in place to balance the weight between observation confidence and prior knowledge (as detailed in Eq. 14-16 of the main paper). For future work, it will be valuable to model the uncertainty of the segmentation algorithm to enable adaptive weight balancing for improved robustness.
Global Map Consistency. We provide a qualitative scene-level result in Figure 7, demonstrating that our reconstructed shapes from multiple objects are consistent with the overall scene point cloud. In our main experiments, we evaluate each object individually focus on validating the effective- ness of the generative prior model, which influences each object independently. Our contribution in this paper is distinct from other research efforts aimed at constructing globally consistent map.
16

---

Figure 7: Scene-Level Visualization: An example of a reconstructed 3D map of scene including four chairs, one sofa, and one table, all constrained with the same prior network.
TK
Figure 8: Input images from 10 RGB-D views. Sequences from the ScanNet dataset contain only observations from one side, which is more challenging for the occluded unseen parts. Sequences from CO3D contain de0-de observations of objects.
Inputs
Outputs
RGB-D Reference View
Text Prompt
Precise Label
"A cheir"
General Label
*An object'
Label "A table
Wrong Label 2
"A ball"
vMap (wio Prior)
Figure 9: Sensitive analysis of the input text prompt, when giving precise, general, and wrong labels for the generative prior model. We find that our system can robustly leverage prior from different information-level labels, by optimizing both information from observations and prior knowledge to find a converged point that can match both parts.
Thanks to our optimization- based formulation, which combines both priors and observations, we can flexibly introduce additional constraints for global consistency. We can leverage well-established SLAM and Structure from Motion (SfM), such as sliding win- dow optimization and loop closure, to enhance global map consistency. We believe our approach can be extended to incorporate further constraints for global consisteney, including cross-object mantic relationships and geometric supporting relationships, which can be represented as non-lime non-linear
techniques from optimization-bas
17

---

constraints in an optimization framework. However, these enhancements require non-trivial engi- neering and are currently beyond the scope of this paper.
Deployment in real applications. We provide guidance for deploying our proposed method in real- world applications such as robotics to facilitate multi-object mapping. Data association-assigniin observations of the same object from multiple frames-is crucial. There are various well-studied methods for data association, including the use of image features (such as manually defned ORB/SIFT or learned features like SuperPoint), ICP matching with point clouds, or probabilistic data association approaches [66, 67]. For camera pose estimation, we can employ offline calibration to determine intrinsic and extrinsic parameters, possibly using additional sensors such as IMUs, or implement online methods like SLAM [43]. We support two types of input for prior conditions: text prompts and segmented images. Both inputs can be easily derived from the input RGB-D images, for example, by utilizing off-the-shelf segmentation algorithms like groundedSAM [68]. Our text prompts are highly flexible; demonstrated in the experiments in Figure 9, even a general prompt like "an object" can effectively constrain occluded parts. In the context of an autonomous robot, integrating extra text descriptions opens up new avenues for fusing information to construct a map. For instance, in human-robo interaction scenarios, a robot could ask the user, "What is the object in front of me?" and the user might respond, "It's a chair." This interaction provides valuable prior knowledge for the object mapping system. This capability enhances traditional SLAM and mapping systems by allowing them to gather information and constraints beyond mere sensor observations.
Relative Pose Definition and Evaluation. In our experiments, the Intersection over Union (IoU) is calculated between the bounding boxes of the estimated and ground truth shapes in world coordi- nates, rather than between voxelized meshes. This choice means that the metric is not sensitive to the detailed reconstruction quality. Evaluating independent pose errors- swors-such-e as directly assessing translation, rotation, and scale relative to ground truth poses-does not provide a reasonable metric in our pose formulation. This limitation is shared by other object mapping systems like vMap, DSP, and Shap-E+ICP. In these systems, the pose is defined as transformation from canonical object coordinate system to the world coordinate system, which is closely linked to the shapes of objects in that canonical space. However, shape representations are not unique. For example, when using DeepSDF (for DSP) or NeRF representations (for vMap, Shap-E+ICP, and our method), shapes can exhibit different size ratios and orientations (e.g., 90-degree rotation around the UP axis). Conse- quently, reconstructing a 3D object in the world involves multiple combinations of estimated poses and shapes in canonical space that can lead pose to evaluate against "ground truth" pose. It is important to emphasize that, in applications such as robotics, our primary concern is the performance of the final 3D representation in the world, which encompasses both shape and pose, rather than evaluating relative "pose" alone. Thus, we opt to directly evaluate the final outputs in world coordinates, reporting IoU and Chamfer Distance (CD) as quantitative metrics, accompanied by qualitative images to demonstrate performance relative to the baselines.
correct result. Therefore, there is no single, unique
6.4 Ablation Study
Methods to Fuse Observations and Diffusion Priors. We compare three strategies to fuse obser- vations and diffusion priors, as shown in Table 3. (1) Optimize then diffuse, which first optimizes shape and pose with geometric loss only for given number of steps, and then uses the diffusion model diffuse the shape. We notice that the information from observations is often lost during the post-diffusion process. Consequently, the ultimate shape diverges from the ground truth, resulting in large metric error. (2) Diffusee then optimize, which first uses the diffusion model to generate a shape with a text condition, then uses the geometric loss to optimize both shape and pose. We ob- serve that the unobserved segment of the shape is prone to corruption during the post-optimization process. Ultimately, this leads to a performance level that is similar to optimizing using only geo- metric observations without priors, which also remains more artifacts the meshes and renderings.
(3) Jointly Optimize and Diffuse, which fuses prior and observations iteratively during optimization with both diffusion prior and geometric loss so that both sources of information active. This com-
18

---

Items
loU 1
CD
Ours w/ Optimize then Diffuse
0.344
0.182
Ours w/ Diffuse then Optimize
0.416
0.160
Ours w/ Jointly Optimize and Diffuse
0.429
0.157
Gradients DirectDiffuse
0.338
0.222
Gradients NrisePredict (Ours)
0.429
0.157
Ours w/ Image Condition
0.436
0.160
Ours w/ Text Condition
0.429
0.157
Table 3: Ablation study the strategies fuse both observations and diffusion prior. Results from 10 RGB-D views on Chairs of ScanNet dataset.
bined optimization can effectively merge constraints from both sources, thereby achieving superior performance compared to the other strategies.
Methods to Calculate Gradients from a Pre-trained Diffusion Model. The gradients derived from both the diffusion model and the observations are high-dimensional. Employing method to effectively combine these gradients to guide the variable toward convergence point is not straightforward task. We compare our method with another to demonstrate the effectiveness of our approach, as shown in Table 3. (1) NoisePredict (Ours). In Section 3.4 of the main content, we discuss our method to use the pre-trained diffusion model to predict the added noise and propagate back the error as gradients. This implicitly constrains the shape variable to lie inside the distribu- tion modeled by the diffusion model, where trained to accurately predict the added noise. (2) DirectDiffuse, which directly uses the diffusion model to predict less noisy version of the cur- rent shape for step, θ,- ca (O,, C,t). Yang et al. [56] also use this method to leverage shape prior constraints from a single cingle-category diffusion model for object reconstruction. Our task is more difficult with the extra unknown variable of pose. As shown in Table 3, DireciDiffuse un- derperforms in comparison to NoisePredict (Ours). We attribute this to two primary factors. Firstly, making pre-trained diffusion model to accurately predict a denoised variable a challenging task. Secondly, each step of DirectDiffuse necessitates precise timestamp within the current variable, which becomes particularly complex when jointly optimized with gradi- ents from observations. In contrast, gradients can be derived from randomly sampled, uniformly distributed timestamps. This allows for flexible diffusion across arbitrary steps without the stringent requirement to adhere to the noise schedule from T to O.
denote the level of noise
Input Conditions. We evaluate both input conditions supported by Shap-E model [29], image and text, as shown in Table Each has its unique strengths and weaknesses, contingent on the specific applications. The image modality, which contains detailed prior information of specific instance such as texture and shape, is nonetheless limited by the quality of the segmentation task. A corrupted or occluded mask can result in corrupted 3D shape prior. On the other hand, a simple text prompt like "a chair" can provide general distribution of complete shapes within the category, albeit without some instance-specifi details. This approach allows the details to be constrained by the observations. Future work could explore the use of more complex text prompts and the fusion of multiple multi-modal priors to enhance the effectiveness and accuracy of prior constraints.
6.5 Computation Analysis
We conducted an evaluation of the system's computation using 10 RGB-D views on a 16GB V100 GPU. For each instance, GOM (Ours) requires 43.0 seconds for 200 optimization iterations, which includes 100 diffusion steps. In comparison, vMap [45] requires 38.1 seconds to complete 200 op- timization iterations, utilizing only geometric constraints. Our method, leveraging diffusion prior, significantly enhances the quality with minimal computational overhead. The Shap-E model [29] requires 45.6 seconds to generate single instance by diffusing from random noise via compu- tationally intensive sampling process. Our method, utilizing the prior information stored inside
19

---

RGB-0 Referance View
Rendered View
Novel Views
Mush
Methods
(Ours)
MMap
Figure 10: Failure Case: An instance where input observations are occluded and contain corrupted masks. While our method manages complete part the object compared to the baseline, it fails to fully complete thin elements such as legs and handles.
Shap-E, can achieve faster reconstruction than the original generation model. DSP [3] requires 32.3 seconds for 200 iterations, speed benefiting from smaller latent space provided by DeepSDF [27].
However, it is constrained to single category and lacks texture information. We adopt strategy of averaging the total number of rays sampled from multiple frames. Consequently, when more frames are available, the computation time remains nearly identical for 3, and 10 views. BundleSDF [20] reconstructs an object's Signed Distance Function (SDF) and appearance field from scratch, without leveraging any prior information. It utilizes 17 keyframes from the same test scene reconstruct the object and carries out 2500 iterations to optimize both the neural fields and the object pose. The cu- mulative running time is 188.6 seconds, partitioned into 118.2 seconds for pose graph optimization and 70.4 seconds for global optimization.
Depending on the specific applications, parameters such as the number of optimization steps, diffu- sion steps, and sampled rays can adjusted to balance accuracy and computation. As a direction for future work, the implementation of an incremental mapping framework, as opposed to batch optimization from scratch, could further expedite online applications.
6.6 Failure Case and Limitation
The ScanNet dataset presents challenges due to occlusions and sensor noise. We illustrate a repre- sentative failure case in Figure 10. When the input observations are severely occluded contain incomplete masks, part by the tissue placed on the chair), but it fails to complete thin elements like legs and handles that the mask does not cover. The paper and pen placed on the chair are reconstructed as part of the texture. Despite these challenges, our method still generates a smoother surface with significantly fewer artifacts compared to the baselines. Future improvements could include the use of a more powerful segmentation model, such as SAM [69], and adaptively increasing the weights of the prior in areas with corrupted observations. Further, more flexible shape representation beyond NeRF, such Gaussian Splatting [70] can be explored to better model the details of objects.
method partially complete the object (for instance, the center occluded
We outline the limitations of our work to inform future research directions. Before deploying our method real-world applications, practices such as data association and camera pose estimation are necessary for effective multi-object mapping. Our approach relies on text descriptions and seg- mented images from a segmentation algorithm as additional inputs. While these can be relatively easy acquire using off-the-shelf models like Segment Anything, they do introduce extra input requirements. Additionally, the computation is not yet real-time, primarily due to the diffusion process and NeRF rendering. The challenges posed by the diffusion process could be mitigated by employing more lightweight generative models, such as those with smaller latent spaces. For the NeRF rendering, more efficient representations, like Gaussian splatting, could improve perfor- mance. Parameters such as the number of optimization steps can adjusted to balance efficiency and effectiveness. Our current focus is on the priors of individual objects, leading us evaluate them independently. However, more scene-level information and priors could enhance global con- sistency, such cross-object relationships, structural knowledge about objects, and scene graphs These aspects could provide valuable avenues for future work.
20

---

Figure 11: Latent Space Interpolation: visualizing the transition of Shap-E generated models from (1) chair another chair; (2) chair
table; (3) chair to
Text Prompt
Generated Shapes
A chair
A green chair
Figure 12:
Text-Conditioned Generation: Shap-E can generate diverse shapes based on given text prompts. The application of more detailed text prompts presents an intriguing future direction for further constraining the shape and pose mapping process.
6.7 Analysis of the Generative Model Shap-E
Latent Space Interpolation. We illustrate a visualization of latent space interpolation from one chair to another, from a chair to table, and from a chair to a plane in Figure 11. Unlike DeepSDF [27], which utilizes 64-dimensional latent vector for an SDF-based shape, Shap-E em- ploys considerably larger latent space for NeRe-b shape, with dimension of 1024 x 1024. Despite its high dimensionality, linear interpolation still provides meaningful transition for changes in both texture and geometry. A smooth latent space aids the optimization process when incorporat- ing gradients from both observations and priors.
Generation from Text Prompt. The Shap-E model is capable of generating variety shapes based a given prompt, demonstrated in Figure 12. The attributes specified in the text prompts, such color, influence the output shapes certain degree. The use of more complex text prompts, such descriptions from large language models (LLMs) to assist in mapping object shapes and poses, presents an intriguing avenue for future research.
6.8 Derivation of Optimization with Prior
We provide the proof for Equation 5 in the main content. Given M observation frames [FJ.M. and condition C, we aim to estimate a Maximum Likelihood Estimation for the unknown variable pose T and shape θ We start from a joint distribution of P(T, θ Fi....FFM C), and aim to get:
T, Ô = arg max P(T, F1. FMs C)
(17)
T,8
According to Bayes' rule:
P(T, FI....... FM, 0) P(F, ..... FM T, T,0,C)P(TT,O P(F Fa CM
(18)
21

---

Considering that any observation frames F1...., Fat are independent to the prior condition C, and we can assume the prior of the observation P(F.) a constant, thus, P..,,..., FMr is constant. We can get:
P(T, olF,, PM; C) x P(Fi, P..T.0)))(
(19)
Then, we consider the observation part P(Fi...., Fas T, θ C). Since the observations F1, FM are conditionally independent among each other given T and O and are independent to C, the likelihood be factorized as:
P(Fi.., =
(20)
Since we model the pose T and shape separately, they are independent each other. Further considering that the condition C only applies the shape, have:
P(T,OIC) P(TIC) ((T)
(21)
We assume uniform distribution for the object pose T, so that P(T) is a constant. So we have:
P(T. @|C) x P(O|C)
(22)
Inserting the observation part (Eq 20) and the prior part (Eq 22) into the joint distribution (Eq 19), we can estimate the unknown variables through:
t. Ô arg max II ((
(23)
Finally, taking the logarithm, get more convenient form for numerical optimization:
T, = arg max T,e
Σ [ooyP(FTTT θ) + log((()C)
(24)
22


Reviews:
[{"review_id": "4F4vsNMp9N", "reviewer": "Reviewer_vp6P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a General Object-level Mapping system (GOM) which uses a diffusion model to generate shape prior and outputs NeRF representation for both texture and geometry of the scene. A probabilistic optimization scheme is proposed to fuse the sensor observations with the diffusion prior. The proposed method is evaluated on the ShapeNet dataset and is shown to outperform the baselines both qualitatively and quantitatively.", "summary_of_recommendation": "The paper proposes an interesting approach to fuse shape prior from a diffusion model with multi frame measurements. The system is shown to outperform the baselines in different metrics. Although there are some questions to be clarified, I find the proposed method a good addition to the current literature. As a result, I’m recommending a weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Can the method perform well if only a single view is provided to the system?\n2. How would the proposed system perform for out of distribution objects? (For example, if the Shape-E provides an incorrect shape prior because the input object is out of distribution.)\n3. What is the runtime of the proposed system?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZSQBJ4U5P", "reviewer": "Reviewer_3JtG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel method of object-centric mapping that uses a 3D diffusion prior as a shape prior. Given a set of RGB-D views, an iterative optimization process is proposed that uses a diffusion prior as well as a rendering loss to estimate the object pose and shape. The diffusion prior update is approximated by the estimates of a 3D diffusion model (in this case, Shap-E). The authors present experiments on the ScanNet dataset, and extensive ablations in the Appendix.", "summary_of_recommendation": "Given the novelty of the idea and corresponding quantitative results, I recommend a score of SA.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "- L233 notes that Shap-E + ICP “exhibits a low CD but struggles to faithfully reconstruct objects”. The CD is significantly lower than the proposed method; it would be informative to expand on why CD does not correlate to faithful reconstruction in this case.\n- Are the gradient fields shown in Figure 2 actual fields (as implied by L181), or for illustration? Very interesting if actual fields, but there should be some details on how the figure is generated in 2D.\n- L29 in the Appendix indicates that the “Jointly Optimize and Diffuse” paradigm “simultaneously considers” both the measurements and the prior. However, L190-196 in the main text seems to indicate that the process is iterative (i.e., the prior is applied first, then the observation). Is it important to update the prior and observation updates iteratively? Or can they be updated in a single consolidated step?\n- A limitations section has not been provided; this must be addressed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D6JRGsSEwC", "reviewer": "Reviewer_UoFm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Author's propose a novel method (GOM) to identify an object's 3D shape and pose from a small subset of RGBD images.", "summary_of_recommendation": "Paper introduces a novel method to integrate diffusion into 3D shape mapping for RGBD, validated by results. This is relevant research for CoRL, and concerns adressed by rebuttal, so strong accept", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1) The paper acknowledges diffusion models are slow to train, however I am curious what is the inference time of your method compared to the other ones? In the context of image generation, VAE and GAN are \"Fast\" while Diffusion is slow (but accurate) in inference. If your improved results similarly come at the cost of longer processing time this should be clearly outlined as a limitation.\n\n2) Similarly Figures 1 and 2 both imply through their use of arrows that the text prompt e.g. \"A Chair\" is provided not just during training but during inference.\n\nIf the proposed method requires an extra input alongside the RGBD photos (brief text description of the object) then this should be clearly outlined in the limitations section. An autonomous robot for example, would not necessarily know that the shape it is trying to identify is a chair.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a General Object-level Mapping (GOM) system that uses a diffusion model to generate shape priors and NeRF representations for scene texture and geometry. The system includes a probabilistic optimization scheme to fuse sensor observations with the diffusion prior. The method estimates object pose and shape from RGB-D views through iterative optimization, leveraging a pre-trained 3D diffusion model (Shap-E).

All reviewers acknowledge that the paper is well-written and well-organized. The idea is timely and relevant to both the robotics and computer vision communities. Concrete comparisons with baselines demonstrate the method's advancement both qualitatively and quantitatively.

Before the rebuttal, several concerns were raised: the method's sensitivity to the quality of the segmentation, the need for additional metrics such as pose errors, the consistency of the map, and the quality of data association.

During the rebuttal, the authors did an excellent job addressing these concerns by providing detailed clarifications regarding the method and evaluation details. While the AC acknowledges the contribution of the work, they also echo some reviewers' concerns about the lack of concrete results in larger-scale mapping and the absence of actual robotic applications enabled by the proposed framework. The inclusion of these additional results would further enhance the impact of this work in the robot learning community.
Decision: Accept