3D Neural Embedding Likelihood for Robust Sim-to-Real Transfer in Inverse Graphics
Authors: Guangyao Zhou
Nishad Gothoskar
Lirui Wang
Joshua B. Tenenbaum
Dan Gutfreund
Miguel Lazaro-Gredilla
Dileep George
Vikash Mansinghka
Venue: Submitted to ICLR 2023 (2023)
Keywords: 3D inverse graphics
probabilistic inference
likelihood
RGB-D
neural embedding
object pose estimation

Abstract: A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data. We propose a novel 3D Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap. 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner. 3DNEL is trained entirely from synthetic images and generalizes to real-world data. To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images. Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions. Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes. This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.

Under review as a conference paper at ICLR 2023
3D NEURAL EMBEDDING LIKELIHOOD FOR ROBUST SIM-TO-REAL TRANSFER IN INVERSE GRAPHICS
Anonymous authors
Paper under double-blind review
ABSTRACT
A central challenge in 3D scene perception via inverse graphics is robustly mod- eling the gap between 3D graphics and real-world data. We propose a novel 3D Neural Embedding Likelihood (3DNEL. over RGB-D images to address this gap. 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in principled manner. 3DNEL is trained entirely from synthetic images and generalizes to real-world data. To showcase this ca- pability, develop multi-stage inverse graphies pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images. Our method outperforms the previous state-of- ffate-of-the-ar art in sim-to-real pose estimation on the YCB- YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions. Unlike existing bottom-up, discriminative approaches that are specialized for pose esti- mation, 3DNEL adopts probabilistic generative formulation that jointly mod- els multi-object scenes. This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using prin- cipled inference in the same probabilistic model without task specific retraining.
INTRODUCTION
There is a widespread need for models that bridge the gap between 3D graphics and real RGB-D data. Accurate simulation environments exist in domains such as autonomous driving, augmented reality, and robotic manipulation, yet robust 3D scene perception remains a central bottleneck. "In- verse graphics" is an appealing approach to 3D scene understanding that treats scene perception as the inverse problem to 3D graphics. However, in practice these methods have been outperformed by more bottom-up, discriminative approaches, especially those using deep learning (LeCun et al., 2015). A key challenge inverse graphics is modeling the gap between rendered images and ob- served real-world images. This paper aims address this gap with 3D Neural Embedding Like- lihood (3DNEL), a likelihood model of RGB-D images that trained entirely from synthetic data and generalizes to real-world data. 3DNEL uses learned neural embeddings predict dense 2D-3D correspondences from RGB and combines this with 3D information from depth in a principled way.
To showcase 3DNEL's capabilities in sim-to-real transfer, we develop a multi-stage inverse graphics pipeline (MSIGP) that uses 3DNEL for 6D object pose estimation. We demonstrate that 3DNEL can be applied real RGB-D images without having to train on any real data. Our 3DNEL MSIGP con- sists of (1) coarse enumerative procedure that generates pose hypotheses and an initial estimate of the 3D scene and (2) an iterative Markov Chain Monte Carlo (MCMC) process that finetunes the 3D scene. We empirically evaluate 3DNEL MSIGP on the popular YCB-Video (YCB-V V) dataset (Xiang al,,, 2018). 3DNEL MSIGP outperforms the previous state-of-the--eeeee (SOTA) SurfEMB (Haugaard & Buch, 2022) in sim-to-real 6D pose estimation, albeit at the cost of increased computation. It is also significantly more robust: we show over 50% reduction high-error pose predictions compared to SurfEMB. Extensive ablation studies illustrate the source of performance improvements.
Existing approaches for 6D pose estimation are predominantly discriminative and bottom-up, and are specialized to the specific task of 6D pose estimation. In contrast, 3DNEL adopts probabilistic generative formulation which extends beyond just pose estimation. To demonstrate the value of 3DNEL's generative formulation, present additional experiments 3DNEL's easy extension to object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
3D Inverse Graphics Our method follows a long line of work in the "analysis-b-ssyateesss"" paradigm that treats perception as the inverse problem to computer graphics (Kersten & Yuille, 1996: Yuille & Kersten, 2006; Lee & Mumford, 2003; Kersten et al., 2004; Mansinghka et al., 2013; Kulkarni et al., 2015). While elegant and conceptually appealing, robustly modeling the gap between 3D graphics and real-world data, especially using appearance information, remains a cen- tral challenge in 3D inverse graphics. Our key observation is that, dense 2D-3D correspondences, widely used in many recent 6D pose estimation methods (Hodan et al., 2020; Li et al., 2019; He et al., 2020; Tremblay et al., 2018; Florence et al., 2018; Haugaard & Buch, 2022), provide a natu- ral way model appearance information in 3D inverse graphics, and can be combined with depth information into a unified probabilistic generative model to effectively bridge the sim-to-real gap.
Sim-to-Re Real Transfer Most computer vision systems require annotated real-world data for train- ing (Hodan et al., 2017; Brachmann et al., 2014; Xiang et al., 2018; Krizhevsky et al., 2017) in order to achieve strong performance real-world data at test time, since without it the "sim-to-real gap" is too difficult to overcome. Practically, it is tedious and expensive to collect and annotate real data. Recent advances in photorealistic rendering and physics-based simulations (Hodan al., 2018; Tremblay et al., 2018: Denninger et al., 2019) have resulted in strong performance of models trained entirely on synthetic data. SurfEMB (Haugaard & Buch, 2022) one such model that obtains SOTA performance in 61) pose estimation, outperforming numerous approaches that use real data for train- ing- 3DNEL builds upon SurfEMB's learned dense 2D>3D correspondences, and expands previous work (Gothoskar et al., 2021) on probabilistic modeling of real depth data. It combines RGB and depth information into a unified probabilistic generative model, and improves SurfEMB nontrivially in both accuracy and robustness to achieve new SOTA in sim-to-real 6D pose estimation.
6D Object Pose Estimation 6D object pose estimation aims to infer the rigid SE(3) transfor- mation (position and orientation) of an object in the camera frame, given an image observation. Discriminatively trained deep learning approaches (Xiang et al., 2018; Li et al., 2018; Deng et al., 2019; He et al., 2020; Sundermeyer et al., 2018) have, in general, outperformed more traditional feature or template matching methods (Besl & McKay, 1992; Rusinkiewicz & Levoy, 2001; Lowe, 1999; Rothganger et al., 2006; Collet et al., 2011). With the advent of RGB-D cameras, there has been a growing interest in leveraging both the appearance information from RGB and the geometric shape information from depth. Existing methods either use depth post process estimations from RGB (Xiang et al., 2018; Haugaard & Buch, 2022), or fuse learned features from both RGB and real, noisy depth (Wang et al.. 2019; He al., 2021). 3DNEL differs from prior work in two impor- tant ways. (1) 3DNEL combines RGB and depth information in principled probabilistic model, which enables superior sim-to-real transfer. (2) 3DNEL jointly models multi-object scenes using a probabilistic generative formulation, allowing easy extension to tasks beyond 6D pose estimation.
3 METHODS
3.1 PRELIMINARIES
Likelihood for 3D Inverse Graphics 3D inverse graphics formulates the perception problem as searching for the 3D scene description that can be rendered by a graphics engine to best recon- struct the input image. A central challenge in applying the 3D inverse graphics approach to real images is robustly modeling the "gap" between rendered and real images. In this paper we aim to develop a likelihood IP Observed RGB-D Image| scene description) that can combine shape and appearance information in a principled way to robustly assess how well an observed RGB-D image explained by a 3D scene description. A naive approach is to render the 3DD scene description to an RGB-D image and define the likelihood as a noise model that directly compares the rendered and real RGB-D images. Recent work 3DP3 (Gothoskar et al., 2021) demonstrates promising per- formance of this approach when applied to depth images. However, it is much more challenging to specify a sensible noise model operating directly on RGB images. Intuitively, the "gap" between rendered and real depth images is mainly due to small spatial displacements, yet the "gap" between rendered and real RGB images comes from a variety of different factors. In addition, a principled combination of RGB and depth information in unified likelihood remains open problem.
2

---

Under review as a conference paper at ICLR 2023
3D Neural Embedding Likelihood (3DNEL)
Rendered Point Cbud Image
Depth
Query Embeddinngs
Description
Observed RGB
Number of objects
facteo ola Object dasses
Embeiding Modil
Pa Py
Oijjet poses
Semaritk Segneriaatio
3DNEL Energy Function
Object
Observed Depth
Observed Point Cloud Image
Object Ccordincte Image
Key Embadkiing
Embedding Model
Figure I: Evaluating 3DNEL 3DNEL defines the probability of an observed RGB-D image condi- tioned on 3D scene description. We first render the 3D scene description into: (1) depth image, which is transformed to rendered point cloud image, (2) a semantic segmentation map, and (3) the object coordinate image (each pixel contains the object frame coordinates of the object surface point from which the pixel originates). The object coordinate image is transformed, via the key models, into key embeddings. The observed RGB image is transformed, via the query models, into query embeddings. The observed depth is transformed into observed point cloud image. The 3DNEL. Energy Function (Equation 1) is evaluated using the rendered point cloud image, semantic segmen- tation, key embeddings, the observed point cloud image, and query embeddings.
3D Scene Description We describe 3D scenes terms of the objects the scene and their poses. Each object is associated with textured mesh, which captures the 3D shape and appearance infor- mation the object. The poses the objects in the scene are represented in SE (3) (rigid transfor- mations in 3D) which define an object's position and orientation in space. The 3D scene description specified as the number N of objects in the scene, their classes f1, ... ty E {1. .M. and corresponding poses Pi. PN SE(3). We assume uniform prior distributions poses (uniform over a bounded volume for position and uniform on 80(3) for orientation).
object
Noise Model Depth Information We the probabilistic model Pdepth (c(ē;r)
11ll--E11a50 from 3DP3 as our noise model on depth information. Peepth is a uniform distribution 3x
in a radius-r ball centered at a rendered point é E R and models the small spatial displacements in the observed point 3R3 hyperparameter that controls the variance of the noise model.
Dense 2D-3D Correspondence Distributions as Noise Models on RGB Information Instead of directly operating RGB images, leverage leamned dense 2D-3D correspondences specify the noise model on RGB information. Concretely, we build on SurfEMB (Haugaard & Buch (2022)) and the proposed dense 2D-3D correspondence distributions over object surfaces. For each object class E {1, .1. M), SurfEMB learns two neural embedding models: (1) a query embedding model which maps RGB image {0, .... "x to a of query embeddings Q' H xHxWXx one for each 2D pixel location, and (2) key embedding model
R3 + R which maps each 3D location x E R33 (coordinate in the object frame) on the object surface a key R. SurfEMB defines surface distribution P RGB (x|q. t) x exp(q" ge (x)) describing which point x on the object surface the given pixel corresponds to. Importantly, these models can be trained entirely from synthetic data (with photorealistic rendering and physics-ban based simulations). See Appendix A.1 for a more detailed review of the query and key embedding models and the dense 2D-3D correspondence distributions.
embedding 9: (x) RR Given pixel with query embedding
3.2 3D NEURAL EMBEDDING LIKELIHOOD (3DNEL)
Processing 3D Scene Description for 3DNEL evaluation For given 3D scene description, we use graphics engine to render it into: (1) A rendered point cloud image HxWx3 where Č, E R² represents the camera frame coordinate at pixel (i.j). (2) A semantic segmentation map Š 10,1,... ..., , where Š., represents the class which the pixel (i.j) belongs. Here
3

---

Under review as a conference paper at ICLR 2023
3DNEL Multi-Stage Inverse Graphics Pipeline (MSIGP)
Input RGB-D Image
Quarr Emberdiings
Initialization
Finetuning
Pore Hypotheaes Preporsk
Faunooiivaaee Gerentiie
Pose Hypotheses
ICP Phoposolss
Rundon Walk Propossi
SuriEMB Prediction
Inferred 3D Scone Description
Observed Po
Couud
Figure 2: Using 3DNEL for 3D Scene Parsing The 3DNEL MSIGP pipeline starts by computing the query embeddings for each object from the observed RGB, and the observed point cloud image from the depth. Then, fast enumerative procedure produces the pose hypotheses for the objects, and construct an initial 3D scene description. We further perform MCMC finetuning of the 3D scene description with 3DNEL using three types of MH proposals (I) pose hypotheses proposals (2) ICP proposals to align an object to point cloud data, and (3) random walk proposals that refines poses with local perturbations. The result is 3D scene description that explains the observed RGB-D image. 3DNEL's joint modeling of multiple objects through the mixture model formulation enables robust estimation this challenging scene with two similar-looking clamps, while SurfEMB (bot- tom left) makes per-object predictions and incorrectly predicts both clamps to be in the back.
represents background. (3) An object coordinate image X E "xxxxx where X,, represents the coordinate at pixel (i, in the object frame of the object of class Sa.j-
Processing Observed RGB-D Image for 3DNEL Evaluation For an observed RGB image (0. 255 H xx and depth image, use the learned query embedding models obtain M sets of query embeddings Q' E xxxxWxx E {1.... M). one for each object class, and use camera intrinsics unproject the depth image into observed point cloud image C R x
3DNEL evaluation Figure 1 visualizes 3DNEL evaluation using processed 3D scene descriptions and observed RGB-D images. 3DNEL combines the noise model Peepth on depth information from 3DP3 and the dense 2D-3D correspondence distributions P PROH from SurfEMB into a unified likelihood for sim-to-real transfer. 3DNEL jointly models multiple objects in scene through mixture model formulation, resulting in a probabilistic generative model of real RGB-D images.
Intuitively, we assess how well each pixel (i.j) in the observed point cloud image C explained by a pixel (i,5) in the rendered point cloud image Č, by combining the noise model Fdepth on depth and the noise model F FRBB on RGB. To jointly model multiple objects in a scene, assume each pixel (i in C can be explained by multiple pixels in Ć.
We formalize this with a mixture model formulation, where the mixture component associated with the rendered pixel (i,j) combines IP deppt and IP RGB assess how well the observed pixel (i.j) is explained by the rendered pixel (i,5). To model background pixels in C, we assume the observed point cloud image C resides in bounded region of volume B, and introduce a uniform distribution Pbackground c; B) = 1/B on the bounded region with mixture probability as an additional mixture component for background modeling. Representing the total number of non- non-background pixels in the rendered images K Σj,; 1 > 0), the mixture probability for the mixture component

---

Under review as a conference paper at ICLR 2023
(a)
(d)
(e)
Figure 3: Enumeration-basel Pose Hypotheses Generation We visualize our cmumeration-basee pose hypotheses generation process using a mug object as an example. Figure 3(a) visualizes spher- ical voting for identifying the mug center. The red points in the observed point cloud represent points associated with the mug. The 3 colored spheres illustrate how votes from points on the mug combine to identify the mug center. The same procedure can also be used to identify the ny keypoints on the mug surface. Figure 3(b) visualizes 20 top-scoring voxels from the voxel grid associated with the mug center. Figure 3(c) visualizes 20 top-scoring voxels from the voxel grids associated with the ny keypoints. Figures 3(d)(e) show how we score pose hypotheses by summing the scores of the voxels the corresponding n& keypoints fall into. Figure 3(d) shows top-scoring pose hypotheses. Figure 3(e) shows how pose hypotheses far away from ground truth get low scores. In Figures 3(b) to (e), light yellow represents low voxel scores, while dark red represents high voxel scores.
associated with rendered pixel ( is given by (1 e)/K. Since the query embedding pixel depends on the entire image I, the mixture components are not properly normalized. This leads to the following energy-based formulation for 3DNEL:
(1) where we denote Ci,y by c, O by by 3, X by E, and Q by q'. The product is over all observed pixels, and the sum is over all non-background rendered pixels. e, B and are hyper- parameters that we pick in the experiments. See Appendix A.2 for more details.
P(I. C(Č, x II EP @Pacckroound((; B)
1- Σ Peteps (el:)))eee
K
3.3 INFERRING THE 3D SCENE DESCRIPTION FROM RGB-D
Figure gives an overview of 3DNEL MSIGP for 6D object pose estimation.
Enumeration- Pose Hypotheses Generation We develop novel spherical voting proce- dure and a heuristic scoring using the query embeddings and observed point cloud image C defined in Section 3.2, and use them in an enumerative procedure efficiently generate pose hypotheses. We use the object center and πı points sampled using farthest point sampling from the object surface as our keypoints, and discretize the camera frame space into L, L, L, voxel grid.
For given keypoint, our spherical voting procedure aggregates information from the entire image to score how likely the keypoint is present different voxel locations, and stores the scores in a voxel grid. Figure 3(a) visualizes spherical voting for the center x² of the mug object: for pixel location (i,j) with camera frame coordinate c 3 and query embedding q ER we identify
most arg max; IP PRGB (T)q, t), calculate the distance Ta = | z² l|2 from x to I² in the object frame, and cast votes (Qi et al., 2019) with weight Pi.j max; IP "GGB B (ā (.lqq t) towards all points visualizes the 20 top-scoring voxels from the voxel grid for the mug center, and Figure 3(c) visualizes the 20 top-scoring voxels from the voxel grids for the ny keypoints on the mug surface.
likely corresponding point on the mug surface
sphere of radius r. centered
Figure 3(b)
We coarsely discretize the object pose space. We reuse the same camera frame space discretization into voxel grid, and use the Tag x Try Toz voxel centers discretize the location space. We use
5

---

Under review as a conference paper at ICLR 2023
customized procedure (Appendix A.5) to generate n, rotations and discretize the rotation space. We use the voxel grid for the object center to identify top-scoring object locations, and score all n, rotations at these locations. We score a given object pose with the sum of the scores of the voxels the corresponding n. keypoints fall into. Figure 3(d) visualizes 3 example top pose hypotheses from the enumerative procedure. Figure 3(e) visualizes how poses far away from ground truth get low scores from our heuristic scoring. See Appendix A.3 for formal description of the process.
MCMC Finetuning We generate an initial 3D scene description using top scoring pose hypothesis for each object instance in the scene based on the pose hypotheses for its class, and use MCMC, in particular the Metropolis- Hasttngl (MH) algorithm, to iteratively finetune the 3D scene description. For scene with N objects, at each iteration, MH proposes to change the pose P. of single object i, and accepts the proposal with the MH acceptance probability. See Figure 2 for visualizations.
We use three different proposals, all of which sample a pose's position from spherical normal distribution centered at a desired position and the pose's orientation from a von Mises-Fisher dis- tribution centered at a desired orientation. multiple times. For object i of class 4, the pose hypotheses proposals sequentially go through the generated pose hypotheses for object class 4, sample pose centered at each of the pose hypotheses, and propose to change the pose of object to the sampled pose. (2) ICP proposals iterate over all objects and use ICP to align each object's s 3D model to the observed 3D point cloud with the point cloud from all the other objects masked out, and make a proposal centered at the resulting pose. (3) Finally, random walk proposals that randomly perturb object poses. See Appendix A.4 for details.
(1) Pose hypotheses proposals go through the N objects
4 EXPERIMENTS
We conduct extensive experiments on YCB-V, V, a popular benchmark for 6D object pose estimation. We establish that 3DNEL MSIGP significantly improves the previous SOTA SurfEMB in sim-to- real 6D object pose estimation, and is more robust in terms of reducing large pose estimation errors and overcoming failures of 2D detectors. Detailed ablations illustrate the source of performance im- provements, and additional experiments on easy extension of 3DNEL to object and camera tracking from video demonstrates the value of 3DNEL's formulation as a probabilistic generative model.
Evaluation We follow the evaluation protocol of the Benchmark for 6D Object Pose Estimation (BOP) challenge (Hodan et al., 2018). The task is estimate the 6D poses of objects in a scene, from single RGB-D image. The BOP challenge assumes knowledge of the number of instances of each object class in the scene. For a predicted pose, we calculate three error metries with re- spect to the ground truth pose: Visible Surface Discrepancy (VSD) (Hodañ et al., 2016; Hodan et al., 2018), Maximum Symmetry-Aware Surface Distance (MSSD) (Drost et al., 2017), and Max- imum Symmetry-Aww Aware Projection Distance (Brachmann et al., 2016). Average recalls ARysD. ARMSSD, ARMSPD are computed for each error metric across a range of error thresholds. The ag- gregate Average Recall (as reported in Table 1) is the average of ARy SD. ARMSSD. and ARMSPD-
Training To allow easy comparison with baselines, we use publicly released pretrained SurfEMB models for our experiments. These include query models, key models, mask predictors for different object classes, and an additional 2D detector from CosyPose (Labbé et al., 2020), all trained entirely from synthetic data. While sharing the same underlying models, our 3DNEL MSIGP is significantly different from SurfEMB's ' PnP+RANSAC procedure, and demonstrates nontrivial improvements in overall accuracy as well as robustness. We expect additional training tailored towards the 3DNEL formulation can further improve performance, but leave this to future work.
3DNEL MSIGP We follow He et al. (2021) and use Ku et al. (2018) to fill in missing depth. Recent strong pose estimation methods commonly rely on 2D detection and mask prediction. Since our spherical voting procedure can robustly generate pose hypotheses by aggregating information from the entire image, 3DNEL MSIGP can work well even without separate 2D detection and mask prediction (Table 1, 3DNEL MSIGP (No 2D Detection)) while SurTEMB completely fails without these. However, empirically we find that we can leverage additional 2D detection and mask predic- tion to allow spherical voting to aggregate information only from regions of the query embedding images that are likely relevant to the objects, and significantly outperform SurfEMB using identical information. We pick hyperparameters by visually inspecting inference results on a small number of real training images outside the test set. See Appendix A.5 for details.

---

Under review as a conference paper at ICLR 2023
[b)
(6)
Objects
2000
1860
Predictions above Error Thineshold
SurIEMS 30NEL
1750
1500
65-
1230
1352
0.50
1000
730
phchei_Jem
banina
500
405
146
250
146
0.00
0.30
Erte Threehols
1058L
8.6
Figure 4: 3DNEL MSIGP yields greater robustness than SurFEMB (a) Comparison of prediction error (measured by VSD) between SurfEMB and 3DNEL MSIGP across 4123 object instances in YCB-V. Each instance is represented as a point on the scatter plot. Points above the dashed line rep- resent instances for which 3DNEL MSIGP has lower prediction error. (b) The number of instances with prediction error above a certain error threshold, across multiple thresholds. 3DNEL MSIGP makes significantly less high-error pose predictions than SurfEMB (over 50% less above 0.5). (c) Scatter plots per object classes for 6 representative object classes out of the 21 object classes.
Categury
Method
Average Recall
3DNEL MSIGP (Ours)
K3.6%
Sin-to-Real
SurEMBB (Haugaard & Buch, 2022)
80.00%
FFB6D
2021)
JDNEL MSIGP (No RGB in Liselihood)
61.57%
3DNEL MSIGP Depth Likelihood)
50.85%
Ablations
3DNKL MSIGP (SurIEMB initialization + MCMC)
82,7348
3DNEL MSIGP (Inirialization only)
62.36%
3DNEL MSIGP (No 21> Detection)
76.53%
FFB6D ((Hc et al., 2021)
85.50%
CasyPose (Labhé
2020) Views)
84.00%
Real and Synthetic Training
SurfEMB (Haugaurd & Buch, 2022)
82.40%
Pix2Pose (Park et al., 2019)
78.00%
W-PoseNet (Xu al., 2019)
77. 77.90%%
Koenig-Hybrid (König Dross, 20200
Table 3DNEL MSIGP yields greater accuracy than baselines and ablations We report Average Recall on the YCB- Video dataset. 3DNEL MSIGP is the top-performing sim-to-real method for 6D pose estimation. Results are averaged over 5 runs. Standard deviation is below 0.2% for all setups.
Our superior performance comes at the cost of increased computation. Empirically we observe our current (highly unoptimized) implementation can be 6x slower than SurfEMB (20 30s on static images, 15 to 20s for tracking, on machine with single NVIDIA A100 GPU). In Section 5 we discuss this limitation and propose some potential solutions.
4.1 ACCURACY
In Table I, we report the Average Recall for 3DNEL MSIGP, its various ablations and representative bottom-up up baselines. In the first section we compare with two baselines trained entirely on synthetic data: the current SOTA SurfEMB (Haugaard & Buch, 2022) which uses depth heuristically refine pose estimates with RGB information, and FFB6D (He et al., 2021) which is a bottom-up approach that fuses appearance information from RGB with geometry information from depth. 3DNEL for- mulates a probabilistic 3D likelihood over depth data and appearance information from RGB in the form of 2D-3D correspondence distributions, and significantly outperforms SuriEMB, FFB6D.
In the second section, we evaluate various ablations of 3DNEL MSIGP. We first investigate the im- portance of principled combination of RGB and depth. We evaluate 3DNEL MSIGP (No RGB in Likelihood), where we drop P PRGB in Equation 1. and 3DNEL MSIGP (No Depth in Likelihood) where we replace the 3DD ball [|]]C,, - C,gllz S r] in Equation 1 with 3 3 2D patch. Both substantially worse. We next evaluate the importance of different components in 3DNEL MSIGP. We evaluate 3DNEL MSIGP (SurfEMB initialization + .MMC), which replaces the enumeration- based pose hypotheses generation with SurfEMB predictions and does MCMC finetuning with 3DNEL. This leads to 2.73% improvements over SurfEMB but does not outperform 3DNEL MSIGP, illustrating the effectiveness of MCMC finetuning and the value of having multiple pose hypothe-

---

Under review as a conference paper at ICLR 2023
Scene ID
48
49
50
51
52
53
54
55
56
57
58
59
SurEMBB Single Frame
73.6%
67.00
83.7%
91.3%
80.0%
59.89
88.4%
76.7%
70.5%
77.3%
92.4%
84.1%
ADNEL MSIOP Single Frame
71.9%
77.5%
87.5%
84.1%
88.4%
82.8%
85.3%
94.3%
3DNEL Object Trackinng
73.9%
78.5%
84.6%
90.0%
91.3%
92.0%
90.7%
83.7%
91.1%
87.9%
95.6%
64.7%
3DNEL Cameru Tracking
81.5%
94.7%
97.5%
97.0%
97.0%
97.0%
97.2%
97.5%
96.8%
92.2%
98.0%
97.0%
Table 2: to SurTEMB and ablations We apply 3DNEL to object and camera tracking from video. Our results show that we can leverage temporal information further improve pose estimation accuracy over the single frame setting, which is impossible for bottom-up up pose estimation methods like SurfEMB.
Extending 3DNEL to object and camera tracking improves performance compared
ses. We evaluate 3DNEL MSIGP (Initialization only), which uses initialiations from the coarse enumerative procedure as pose estimates. This is substantially worse, demonstrating the importance of MCMC finetuning. We finally evaluate 3DNEL MSIGP (No 2D Detection), which uses spherical voting to aggregate evidence from entire query embedding images to generate pose hypotheses, This uses much less information than SurfEMB (no 2D detection and mask prediction), yet results are not far off, demonstrating the effectiveness of our enumeration-basee pose hypotheses generation.
In the last section, we contextualize 3DNEL MSIGP's performance in relation methods that de- pend on real data. 3DNEL MSIGP outperforms several methods, despite no exposure to real data before test time. We also observe that previous bottom-up approaches that combine RGB and depth information rely heavily annotated real data. FFB6D, for example, has average recall of 85.5% but this drops to 75.8% without real data, We hypothesize this is because these methods essentially overfit to the characteristics of noisy depth and struggle to generalize to real RGB-D images without having seen them at training. In contrast, as a result of our principled probabilistic modeling of depth information, we can transfer from simulated data to real data without sacrificing performance.
4.2 ROBUSTNESS
Figure 4 contains a more detailed comparison between SurfEMB and 3DNEL MSIGP. Across all YCB-V test images, there are 4123 object instances. Each point on the scatter plots corresponds an object instance, and the point's x and y coordinates are the pose prediction error of 3DNEL MSIGP and SurfEMB, respectively. Points above the dashed line correspond to object instances for which 3DNEL MSIGP had a lower prediction error than SurfEMB. Figure 4(a) shows the scatter plot for all 4123 predictions and Figure 4(c) shows the scatter plots for representative object classes. Figure 4(b) shows, across a range of error thresholds, the number of pose predictions with error above that threshold. 3DNEL MSIGP significantly and consistently reduces the the number of high-error pose predictions, highlighting 3DNEL MSIGP's robustness compared to bottom-up approaches.
We find many instances, especially concentrated within a few object classes, for which SurIEMB is systematically making bad pose predictions. This is primarily result of the noisy neural em- beddings, which SurfEMB's ' PoP+RANSAC procedure does not handle well. 3DNEL MSIGP is able to correctly estimate the object's 's pose result of cnumeration-basee pose hypotheses gener- ation, which can robustly handle noisy neural embeddings and identify a variety of pose hypotheses. 3DNEL MSIGP also improves
SurfEMB on test images where the 2D object detector that Sur- fEMB relies on fails. SurfEMB uses a neural 2D object detector and only makes pose predictions within the predicted bounding box. In contrast, 3DNEL MSIGP considers the full image using spherical voting and therefore can make reasonable pose estimates in spite of missing or wrong 2D detections. Out of the 4123 object instances, there are 174 object instances for which there is no 2D detection and therefore no prediction from SurfEMB. Figure 5 and Figure A4 in the appendix visualize 3DNEL MSIGP and SurfEMB's pose predictions for representative YCB-V test images.
4.3 EXTENSION TO OBJECT AND CAMERA TRACKING FROM VIDEO
We demonstrate that, due to its probabilistic generative formulation, we easily extend 3DNEL to support object and camera tracking from video using principled probabilistic inference in the same model without task specific retraining. We extend the single-frame 3DNEL to the multi-timest timestep setup by introducing a simple dynamics prior that samples the object pose at time t + 1 uniformly from the space of poses with position most 3cm away from the object pose at time l.
8

---

Under review as a conference paper at ICLR 2023
(a)
Input RGB
Input Depth
Ground-truth Reconstruction
SurfEMB Output
3DNEL MSIGP Output
T
Figure 5: Qualitative results of 3DNEL used for pose estimation and tracking (a) Results of 3DNEL MSIGP for 6D pose estimation (b) Results of camera tracking from video using 3DNEL. Predicted object poses are overlayed on each video frame. The r-axis indicates time evolution.
For object tracking, we initialize the object poses the first frame using our single-frame prediction, and apply MCMC finetuning with ICP and random walk proposals for subsequent frames, taking into account the dynamics prior. For camera tracking, we initialize object poses at the first frame to ground truth annotations to avoid introducing systematic errors. We again apply MCMC finetuning with ICP and random walk proposals. However, we further assume we know the scene is static and only the camera moves, which translates into jointly updating the poses of all objects in scene.
Results in Table show that the same inference procedure can readily handle such extensions, taking into account the dynamics prior for object tracking and the knowledge of a static scene for camera tracking within the same probabilistic generative model. We observe comprehensive improvements over single frame predictions, except for object tracking in scene 59 where we make large errors in the first frame with our single frame predictions and are not able to recover in later frames.
Finally, to highlight the benefits of modeling uncertainty, key advantage enabled by 3DNEL's probabilistic formulation, we consider a challenging object tracking setup with highly occluded objects. We generate synthetic videos containing two YCB objects, in which object moves across the scene and becomes fully ocluded by the other static object before reappearing. Methods that maintain only single object pose estimation for each object tend lose track of the moving object once it becomes occluded. However, by leveraging Sequential Monte Carlo (Liu & Chen, 1998) to properly model the uncertainty in the pose of the moving object when it is occluded, we are able to achieve reliable tracking in this challenging setup. See Appendix A.7 for details.
5 DISCUSSION
This paper presents 3DNEL, a 3D likelihood for RGB-D images based on neural embeddings, and demonstrated SOTA performance in sim-to-real 6D object pose estimation on YCB-V. However, the current work has several limitations. Since 3DNEL is formulated to explain the whole scene, inference with 3DNEL sometimes moves highly occluded objects to explain visible backgrounds. In addition, certain objects are difficult to capture with depth cameras because of their reflective sur- faces, which can lead degraded pose estimation due to the importance of depth cues in 3DNEL. We expect background model and an outlier model that can properly fall back to use only RGB data when depth is missing will help address these issues. Finally, while 3DNEL improves pose estimation accuracy and robustness, this comes at the cost of increased computation. This is due in part to the lack of performance engineering efforts, and to the non-standard nature of our infer- ence pipeline, which lacks highly optimized software implementations, In future work, plan to additionally leverage batched GPU rendering and compilation, which will enable a large number of MCMC proposals to be evaluated in parallel and therefore significantly speed up our pipeline.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Paul Besl and Neil D McKay. Method for registration of 3D shapes. Sensor Fusion IV: Control Paradigms and Data Structures, volume 1611, pp. 586-606. International Society for Optics and Photonics, 1992.
Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In European conference on computer vision, pp. 536-551. Springer, 2014.
Eric Brachmann, Frank Michel, Alexander Krull, Michael Ying Yang, Stefan Gumhold, al. Uncertainty-driver 6d pose estimation of objects and scenes from single rgb image. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3364-3372 2016.
Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The moped framework: Object recog- nition and pose estimation for manipulation. The international journal of robotics research, 30 (10):1284-1366, 2011.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http:://pppt http://pyballet.oom 2016-2024
Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. Poserbpf: A mno-blackwellized particle filter for 6d object pose tracking. arXiv preprint arXiv: 09304, 2019.
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Ole- fir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv:1911.01911 2019.
Bertram Drost, Markus Ulrich, Paul Bergmann, Philipp Hartinger, and Carsten Steger. Introducing mvtec itodd-a dataset for 3d object recognition in industry. In Proceedings of the IEEE interna- tional conference on computer vision workshops, pp. 2200-2208, 2017.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv:1806.08756. 2018.
Nishad Gothoskar, Marco Cusumano-Toowe
Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 3dp3: 3d scene per- ception via probabilistic programming. Advances in Neural Information Processing Systems, 34: 9600-9612, 2021.
Rasmus Laurvig Haugaard and Anders Glent Buch. Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6749-6758, 2022.
Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. Pvn3d: A
deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11632 11641, 2020.
Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun. Fíb6d: A full flow bidi- rectional fusion network for 6d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3003-3013, 2021.
Tomáš Hodañ, Jifí Matas, and Stěpán Obdržálek. On evaluation of 6d object pose estimation. In European Conference on Computer Vision, pp. 606-619. Springer, 2016.
Tomáš Hodan, Pavel Haluza, Štepán Obdržálek, Jiri Matas, Manolis Lourakis, and Xenophon Zab- ulis. T-les: An rgb-d dataset for 6d pose estimation of texture-less objects. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 880-888. IEEE, 2017.
Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al. Bop: Benchmark for 6d object pose estimation. In Proceedings of the European conference on computer vision (ECCV), pp. 19-34, 2018
10

---

Under review as a conference paper at ICLR 2023
Tomas Hodan, Daniel Barath, and Jiri Matas. Epos:
Estimating 6d pose of objects with symmetries. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11703- 11712, 2020.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Bagan-Kelley and Frédo Durand. Taichi: a language for bigh-performance computation on spatially sparse data structures. ACM Transac- fions on Graphics (TOG), 38(6):201, 2019.
D Knill D Kersten and A Yuille. Introduction: A Bayesian Formulation of Visual Perception. Per- ception as Bayesian inference, pp. 1-21, 1996.
Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object Perception Bayesian Inference. Annu. Rev. Psychol., 15:271-304, 2004.
Rebecca König and Bertram Drost. A hybrid approach for 6dof pose estimation. In European Conference on Computer Vision, pp. 700-706. Springer, 2020.
Alex Krizhevsky, Ilya Sutskever. and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90) 2017.
Jason Ku, Ali Harakeh, and Steven I. Waslander. defense of classical image processing: Fast depth completion on the cpu. In 2018 15th Conference on Computer and Robot Vision (CRV), pp- 16-22 IEFE, 2018.
Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: A Probabilistic Programming Language for Scene Perception. In CVPR 2015: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4390-4399, 2015.
Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. CosyPose: Consistent multi-view multi-object 6D pose estimation. In ECCV 2020: Proceedings of the European Conference Computer Vision, pp. 574-591. Springer, 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
Tai Sing Lee and David Mumford. Hierarchical Bayesian Inference in the Visual Cortex. JOSA A, 20(7):1434-144, 2003.
Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In European Conference on Computer Vision (ECCV), 2018.
Zhigang Li, Gu Wang, and Xiangyang Ji. Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7678-17777 7687, 2019.
Jun S Liu and Rong Chen. Sequential monte carlo methods for dynamic systems. Journal of the American statistical association, 93(443):1032-1044 1998.
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp. 1150-1157. Ieee, 1999.
Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs. NIPS 2013:Ad- vances in Neural Information Processing Systems, 26: 26:1520-1528 2013.
Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7668-7677, 2019.
Charles R Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277-9286. 2019.

---

Under review as a conference paper at ICLR 2023
Fred Rothganger, Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 3d object modeling and recognition using local aifineinvvarian image descriptors and multi-view spatial constraints. In- ternational journal of computer vision, 66(3):331-255 2006.
Szymon Rusinkiewicz and Marc Levoy. Efficient variants of the iep algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pp. 145-152. IEEE, 2001.
Martin Sundermeyer, Zoltan-Csabe Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceed- ings of the european conference on computer vision (ECCV), pp. 699-715,2011
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birch- field. Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. In CORL 2018: Conference Robot Learning, 2018. URL hhttps://arxiii org/ abs/ 1809.10790,
Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese. DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. CVPR 2019: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
3343-3352, 2019.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. 2018.
Zelin Xu, Ke Chen, and Kui Jia. W-posenet: Dense correspondence regularized pixel pair pose regression. arXiv preprint 1912::1112.1188888 2019.
Alan Yuille and Daniel Kersten. Vision as Bayesian Inference: Analysis by Synthesis? Trends in Cognitive Sciences, 10(7):301-308, 2006.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 REVIEW OF SURFEMB
Our noise model on RGB information build on SurIEMB (Haugaard & Buch, 2022) which learns neural embeddings establish dense 2D-3D correspondences
contrastive learning. For each object class E {1.- ... MM.. J. SurfEMB learn a pair of neural networks: the query model fi {0, [0,--.. 255) xx Y IR RXX and the key model 9: : R R The query model transforms an observed RGB image into query embeddings Q', while the key model transforms rendered object coordinate image X into set of key embeddings.
For given 2D pixel location on the observed image with query embedding q. SurfEMB specifies surface correspondence distribution PROB (ū/q, 1) x exp(q' (x)) for each object class 1. To normalize this surface correspondence distribution, for each object class 4, we subsample uniformly across the object's surface get set Ze of surface 3D coordinates in object frame. Then, given pixel with query embedding q, we calculate the probability that this pixel corresponds a surface point & Ze object class as:
PRGB(X1q Z,,t) = exp(q" (x))
(2)
ExEr, exp(q" gc
A.2 MORE DETAILS ON THE ENERGY-BASED FORMULATION OF 3DNEL
Since we are working with an energy-based formulation (Equation 1), to make the probability distri- bution properly defined we need to make sure the normalization constant, i.c. the sum of the energy function over all and
I/II
(Bnnckrround(c:B)
1 1
Σ 2:300
Pdeptn (e 3)
K
is finite and well-defined.
For RGB images of size Il W, since each pixel has only 256 values, there are at most 256 RGB images of size H x W which is finite number. Since the value of the energy function less than for any given I and C, summing over a finite number of and integrating over bounded region for C gives us finite normalization constant, making the probability distribution well-defined.
A.3 DETAILS ON
BASED POSE HYPOTHESES GENERATION
In Algorithm we present a detailed description of our enumeration pose hypotheses gen- eration process that introduced in Section 3.3. In practice we also optionally apply non-max suppression in the TopPosin it ons function in the algorithm to make sure the promising positions identify are well spread out to cover different parts of the image and better represent uncertainty.
A.4 DETAILS ON MCMC FINETUNING
Our pose hypotheses proposals go through the objects present in a scene multiple times. For object i of class l, the pose hypotheses proposals sequentially go through all pose hypotheses for class 4. For each one of these pose hypotheses, the pose hypotheses proposals sample pose centered the given pose hypothesis, and propose to change the pose of object pose. Going through the objects more times help improve performance. at the cost of increased computation. practice we find going through all objects twice strikes good balance between performance and speed.
the sampled
A.5 DETAILS ON INFERENCE PIPELINE IMPLEMENTATION
A.5.1 DISCRETIZING THE ROTATION SPACE
13

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Iinueeration-base Pose Hypotheses Generation
* Basic c setups
*
Object-specific A set of surface points Z for object class t, query model ft, key model St, n keypoints with object frame coordinates o;; ... ,"""",, E R
Spatial discretization: Camera frame coordinates of the center of the boundary voxel y R.,
size of the voxel grid (Lz, Ly. Lz). diameter of the voxels d 0.
Orientation discretization: representative orientations R..... Ra, SO(3).
Parameters: Number of pose hypotheses to generate "p. number of top positions ni.
/* Enumeration -ased pose hypotheses generation
Input: RGB image I E R observed point cloud C E Rxxxx Output: Top scoring pose hypotheses P, .P,, SE((3)
Q - f((I;; // Get query embeddings Q E R from the RGB image I // Aggregation for object center (0,0,0)
v t Voting (Q, C, (0,0,))
for i +- to ng do
// Aggregation for ni keypoints.
V < Voting (Q, C,z,);
// V W'RR Rlaxblxl
/* Identify top positions based on vorg s largest entires. // h. In, E ER
*/
In, + TopPositions (V"); for f 1--1oo,
to n, do
11 Heuristic pose scoring Pf.... Pt. + RankByScore (h, I Si.. 1, ,,,j = =1, 1, mr):;
L SL Scoring V'...,
10 return Pl,..., P
"
/* Voting and heuristic scoring def Voting (Q, C,
*/
12
// roaind evidence aggregation // Initialize V RL,XL, to all 0 array for i + to H to W do
13
V
14
15
I f arg max; PRGB(i Ze, t), Pt. max; PRGB(EIQu,, Z,, for u 4 1 to La 0 x-1 to by t 1 to L2 do
17
e (yl + +(u (u- 1)d, 82 + ( 1)d, us + (w 1)d);
if C -cl2 2 || x 12112 then
19
a,, = Vu... + PiJ
return 29
21
def Scoring (1, R, V1 V*E)
1/ Heuristic pose scoring // Initialize score to
22
+ 0:
23
for i +- 1 to ny do R I; (u, v. w) + Ro v)/d3; // Identify corresponding voxel of x
// Location of of in world frame for pose l,R
25
26
8 s U
// Heuristic scoring
27
return s
We discretize S0(3) into 6400 representative orientations. generate these orientations by first picking 200 points roughly uniformly the unit sphere using the Fibonacci sphere. The 6400 representative orientations are genearted by first rotating the axis (0, 0, 1.0) to point to one of the 200 points, followed by one of 32 in-plane rotations around the axis.
A.5.2 USING ADDITIONAL 2D DETECTION AND MASK PREDCITION FROM SURFEMB
In the best performing setup, we leverage the same 2D detector used in SurtEMB part of the pose hypotheses generation process. Although our spherical voting procedure can robustly aggregate information from the entire image to generate pose hypotheses, demonstrated by the competitive performance of 3DNEL MSIGP (No 2D detection) (Abalations in Table 1), in practice the query emhedding images for many objects are very noisy and tend to hurt performance. Empirically,
14

---

Under review as a conference paper at ICLR 2023
master, cheel can
craceer, box
sugar,
tomaaoo soup, can
mustard bottle
tuna, fish
puadiiing box
3.0
SUITEH Erro
meat
bleach,
SWEEHB Eroor
0.5
3.0
1.0
3.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
nower_dril
wood block
scissors
large aarkee
large_ dlamp
extra Jarge dlmpp
Feam, feam_brick brick
1.0
1.0
1.0
1.0
1.0
1.0
1.0
SUrTEM Enor
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.5
10
3.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.3
1.0
0.0
0.5
1.0
0.0
0.5
1.0
SOMEL Erorr
3DMEL IDXELENS Etroe
SONEL Erter
SONEL trrer
3DND Eror
3DNEL Enar
3DNEL Enor
Figure A1: We compare the prediction error (using the VSD error metric) of SurfEMB and 3DNEL MSIGP across all 4123 object instances in the YCB-V V test dataset, with each instance represented as a point on this scatter plot. In the main text Figure 4(a), we show the scatter plot across all objects. Here, show the results per object.
observe that by additionally using 2D detections, can focus the spherical voting process regions of the observed image that likely relevant for the objects, and further improve performance.
For each object class in the scene, there be multiple 2D detections. For each 2D detection, we do spherical voting just within the detector crop and generate 80 pose hypotheses per detector obtain the query embeddings by upsampling the input RGB image by 1.5x, and do spherical voting on the whole image. In such cases, when we identify top positions,
crop. When there is missing 2D detection,
additionally do non-max suppression with a filter size of 10 to spread the top-scoring positions out. For each such top-scoring position we identify top 2 orientation, and we consider all top-scoring positions and generate total 30 pose hypotheses.
SurfEMB's query embedding model was trained on image crops from 2D detections rescaled to 224 224, but can be applied RGB images of any size. Empirically we observe that due to Sur- fEMB's rescaling of the 2D detection crops to 224 224, the query embeddings for some objects are only informative when we properly rescale the input RGB image. To account for this, we esti- mate the best scale by scoring the top pose hypotheses (using heuristic scoring described in Section 3.3) from each 2D detector crop and pick the scale the highest-scoring 2D detector crop. Concretely, when there multiple 2D detections for object class, score the corresponding scales with the mean heuristic score of the top 5 pose hypotheses from that detector crop, and use the highest scoring scale obtain the query embeddings for the object class. When there is no 2D detections we fall back to a default scale of 1.5.
A.5.3 IMPLEMENTATIO DETAILS AND HYPERPARAMETERS
We use OpenGL for rendering, and use Taichi (Hu et al., 2019) for both spherical voting and 3DNEL evaluation. A Python implementation reproducing the results reported in the paper will released upon acceptance.
As we describe in the main text, we pick hyperparameters by visually inspecting detection results on small number of real training images that outside the test set.
We select 74 = keypoints from the surface of each object class, and use (-350.0, 220.00, 530.0), L, 129, Ly 87, L, 168 and d = 5.0 to make the voxel grids large enough to cover all the keypoints that can be present the camera frame. Here the units for the values y and dl are
15

---

Under review as a conference paper at ICLR 2023
Figure A2: Visualizing an example synthetic video we use to demonstrate the benefits of modeling uncertainty. We render a short video containing two YCB objects, the cracker box and the sugar box. The cracker box is static, while the sugar box is moving and gets completely occluded by the cracker box before reappearing. Tracking is challenging in such setups for methods that only use a single object pose estimation. However by properly modeling uncertainty using the posterior distribution on object poses from 3DNEL through Sequential Monte Carlo, able to achieve successful tracking. The z-axis represents time evolution.
We use = 5.0 in evaluating 3DNEI for all our experiments. We implement 3DNEL with Taichi to leverage powerful GPUs. When identifying points in the rendered point cloud C (organized as an H W x 3 image) that within distance r from point (i,j) the observed point cloud C, to further speed up 3DNEL evaluation, we only look at the points from the patch of size (10, 10) centered at ji.
For MCMC finetuning, we go through all pose hypotheses twice in the pose hypotheses proposals. We go through all objects 5 times the ICP proposals. We go through all objects 50 times in the random walk proposals. Each time we go through an object in the random walk proposals, we propose 7 types of random pose perturbations.
A.6 ADDITIONAL ROBUSTNESS RESULTS
In Figure Al we include additional robustness results.
A.7 BENEFITS OF MODELING UNCERTAINTY: AN ADDITIONAL TRACKING EXPERIMENT
To highlight the benefits of modeling uncertainty, a key advantage enabled by 3DNEL's probabilistic formulation, present additional experiment object tracking in synthetic video containing two YCB objects, the cracker box and the sugar box. The cracker box remains static while the sugar box moves across the scene. The sugar box gets fully occluded by the cracker box before reappearing. We generated this video in PyBullet (Coumans Bai, 2016-2021) using the textured mesh models from YCB-V dataset for the cracker box and sugar box. Some example frames from the synthetic video are shown in Figure A2.
This is challenging setup for methods that do not model uncertainty but only rely single estimation of the object pose, as such methods tend to lose track once the moving object becomes severely occluded. We find that when 3DNEL is used only produce single estimation of the object pose, tracking indeed fails. However, when we apply 3DNEL for tracking using Sequential Monte Carlo, we are able approximate the posterior distribution over the object's pose at each timestep and capture the uncertainty in the pose of the moving object when it gets occluded (rather than single estimation of the object pose). We find that this can help us accurately trace the object and regain track of the moving object when it reappears from occlusion.
For Sequential Monte Carlo, we work particle filtering setup, and assume a simple dynamics model in which the object's translation is sampled from multivariate Gaussian centered at the translation from the previous timestep, and the orientation is sampled from a Von-Mises-Fishiee distribution centered at the orientation from the previous timestep.
In Figure A3, we visualize the tracking results and the inferred posterior distribution (approximated via particles in particle filtering setup) different timesteps. When the sugar box becomes cluded, we see the posterior distribution on the pose of the sugar box spread out across the many positions the sugar box could be behind the cracker box. When the sugar box eventually reappears
16

---

Under review as a conference paper at ICLR 2023
Figure A3: Visualizing object tracking in the challenging setup with highly occluded objects. We use particle filtering setup. Top row visualizes overlays of the predicted object poses and the RGB images. Object poses are derived from the particle that has the highest posterior probability. Bottom row visualizes how the posterior distribution on the pose of the sugar box (represented using the particles) evolves as the sugar box moves across the scene. As we can see, once the sugar box gets occluded, the posterior distribution spreads out across the scene to model the uncertainty we have on the pose of the sugar box. When the sugar box reappears, the posterior distribution becomes concentrated again and the system is able to regain track of the sugar box. The æ-axis indicates time evolution.
can regain track of its pose. On the other hand, methods that maintain only single object pose estimation will almost surely lose track once the moving object gets occluded.
Note that for this experiment, leverage an improved pure JAX implementation of the 3DNEL likelihood. We implemented JAX depth image renderer, which enables a large number of 3DNEL evaluations in parallel. This allows us to achieve real-time tracking performance for 480 640 images on machine with single NVIDIA A100 GPU. Of course the involved data are synthetic and much simpler. However this is promising early step towards significantly speeding up the 3DNEL inference process.
A.8 ADDITIONAL VISUALIZATIONS OF SURFEMB AND 3DNEL MSIGP PREDICTIONS ON YCB-V V TEST IMAGES
In Figure A4 we include additional visualizations of SurfEMB and 3DNEL MSIGP predictions YCB-V test images.
17

---

Under review as a conference paper at ICLR 2023
Ground-truth Reconstruction
SurfEMB Inferred Scene
3DNEL Inferred Scene
Input RGB
Input Depth
20.00
ORITOR
Figure A4: Additional visualizations SurfEMB and 3DNEL MSIGP predictions on YCB-V test images
18


Reviews:
[{"review_id": "yoO6ESLxSU8", "reviewer": "Reviewer_4Xvb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The presented paper aims to introduce a method for the robust sim-to-real transfer to address the domain gap between synthetically generated data and real data. The goal of the method is to support training a model with synthetic data so as to generalize to real data. The proposed method -- called 3DNEL -- uses neural embeddings to predict 2D-3D correspondences from RGB and depth images. 3DNEL is used as part of a multi-stage inverse graphics pipeline that is tested on a 6D pose prediction task.  \n", "summary_of_the_review": "Overall, I have the impression that the topic this paper address is important and interesting to the ICLR community. However, it its current form I cannot recommend accepting this work. This is due to the following reasons: \n\n- The technical description of the method appears quite convoluted and is difficult to follow. The method section needs to be rewritten carefully and more details need to be provided. \n- While the method seems to be sound for the setup defined in the paper, it is not clear how it could be used for other tasks or setups. \n- Compared to SurfEMB the technical contribution appears somewhat marginal.", "strength_and_weaknesses": "Strengths: \n\n- The topic of using synthetic data for effectively training neural networks is interesting and important. \n- The proposed approach of combining RGB and depth images by employing neural embeddings seems novel although somewhat marginal compared to SurfEMB.  \n- The presented experiments are mostly sounds and the results seem promising. \n\nWeaknesses: \n\n- The technical section of the paper is quite convoluted and difficult to follow. \n- Contrary to the author's claims it does not seem as if the method is easy to apply to other tasks and training setups. \n- The method is only shown to work on pose estimation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ypo3IAfTQp", "reviewer": "Reviewer_aQXb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a probabilistic generative method leveraging learned 2d-to-3d mappings from SurfEMB and depth (3DP3) that generates and optimizes (MCMC) 6d pose hypotheses based on RGB-D images. The method is purely trained on synthetic images and evaluated on real data showing significant improvements to the baseline method by evaluating multiple hypothesis and refining them using MCMC.", "summary_of_the_review": "After the rebuttal of the authors the explanation of the method is very clear and the probabilistic formulation shows advantages over existing methods especially also for estimating the uncertainties. There remain a few open points that would be interesting to follow up on, but the proposed idea and results are already showing the benefits of the proposed approach.", "strength_and_weaknesses": "Strength:\n\nThe probabilistic formulation is simple and seems robust.\nProposal generation method based on spherical voting from depth and correspondences is independent of 2d detections priors.\nThe authors show the usefulness of the probabilistic model to generate possible poses under occlusion.\n\nWeaknesses:\n\nA remaining question is which hypothesis are actually sampled in the MCMC optimization. An ablation for the impact of the different methods Voting, ICP and Random Walk would help to evaluate their impact on the final result.\n\nIt seems that at the moment the proposed method leverages the knowledge of how many objects are present in the scene to create the initial proposals, which is a hugh advantage over other methods that do not have this assumption. While it seems like the assumption could be relaxed, it is unclear if the method remains stable in this case.\n\nIn 3.3 the authors write \"ICP proposals iterate over all objects and use ICP to align each object’s 3D model to the observed 3D point cloud with the point cloud from all the other objects masked out,\", however it is unclear how they can mask out all other objects if they only have semantic masks and not instance masks. While using synthetic data learning instance masks would probably not pose a larger problem than predicting SufEmbeddings, it is unclear how these masks are generated at the moment.\n\n\nCorrections:\nIn 3.1 the dimensions of Q should probably be HxWxE instead of HxWx3", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KRYEguTFYK", "reviewer": "Reviewer_a254", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed an inverse-graphics approach for robust object pose estimation: the model first evaluates pose performance on discretized pose hypothesis with a spherical voting, then finetunes the pose with MCMC. The performance on YCB-dataset is strong and achieves the state-of-the-art.", "summary_of_the_review": "In summary, the author proposed an interesting idea of 2D/3D-correspondence-based  approach to solve the object pose estimation problem. The idea is clearly presented and easy to follow. And the experimental results on YCB is strong.\n\nI have a few questions:\n* Any insight why the proposed approach can be better than RANSAC+SurfEMB? the evidence model proposed in Eqn (1) could be covered by RANSAC if a correct correspondence is proposed to generate a hypothesis.\n* Robustness: if a RANSAC-based algorithm tries enough hypothesis, will it be more robust as well?", "strength_and_weaknesses": "+ A 2d/3d correspondence approach to handle sim-to-real pose estimation.\n+ the probabilistic formulation in Eqn (2) and (3) makes sense.\n+ Spherical voting idea is interesting, related in spirit to hough voting. \n+ Experimentally, the model achieves strong performance compared to SOTA Ransac-based-proposal approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject