3DP3: 3D Scene Perception via Probabilistic Programming
Year: 2021

Abstract:
We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images. 3DP3 uses (i) voxel models to represent the 3D shape of objects, (ii) hierarchical scene graphs to decompose scenes into objects and the contacts between them, and (iii) depth image likelihoods based on real-time graphics. Given an observed RGB-D image, 3DP3’s inference algorithm infers the underlying latent 3D scene, including the object poses and a parsimonious joint parametrization of these poses, using fast bottom-up pose proposals, novel involutive MCMC updates of the scene graph structure, and, optionally, neural object detectors and pose estimators. We show that 3DP3 enables scene understanding that is aware of 3D shape, occlusion, and contact structure. Our results demonstrate that 3DP3 is more accurate at 6DoF object pose estimation from real images than deep learning baselines and shows better generalization to challenging scenes with novel viewpoints, contact, and partial observability.

heading: 1 Introduction
text: A striking feature of human visual intelligence is our ability to learn representations of novel objects from a limited amount of data and then robustly percieve 3D scenes containing those objects. We can immediately generalize across large variations in viewpoint, occlusion, lighting, and clutter. How might we develop computational vision systems that can do the same?
This paper presents a generative model for 3D scene perception, called 3DP3. Object shapes are learned via probabilistic inference in a voxel occupancy model that coarsely captures 3D shape and uncertainty due to self-occlusion (Section 4). Scenes are modeled via hierarchical 3D scene graphs that can explain planar contacts between objects without forcing scenes to fit rigid structual assumptions (Section 3). Images are modeled by real-time graphics and robust likelihoods on point clouds. We cast 3D scene understanding as approximate probabilistic inference in this generative model. We develop a novel inference algorithm that combines data-driven Metropolis-Hastings kernels over object poses, involutive MCMC kernels over scene graph structure, pseudo-marginal integration over uncertain object shape, and existing deep learning object detectors and pose estimators (Section 5). This architecture leverages inference in the generative model to provide common sense constraints that fix errors made by bottom-up neural detectors. Our experiments show that 3DP3 is more accurate and robust than deep learning baselines at 6DoF pose estimation for challenging synthetic and real-world scenes (Section 6). Our model and inference algorithm are implemented in the Gen [11] probabilistic programming system.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 2 Related Work
text: Analysis-by-synthesis approaches to computer vision A long line of work has interpreted computer vision as the inverse problem to computer graphics [22, 39, 26, 23]. This ‘analysis-by-synthesis’ approach has been used for various tasks including character recognition, CAPTCHA-breaking, lane detection, object pose estimation, and human pose estimation [40, 36, 27, 30, 18, 31]. To our knowledge, our work is the first to use an analysis-by-synthesis approach to infer a hierarchical 3D object-based representation of real multi-object scenes while exploiting inductive biases about the contacts between objects.
Hierarchical latent 3D scene representations We use a scene graph representation [41] that is closely related to hierarchical scene graph representations in computer graphics [9]. Unlike in graphics, we address the inverse problem of inferring hierarchical scene graphs from observed image data. Inferring hierarchical 3D scene graphs from RGB or depth images in a probabilistic framework is relatively unexplored. One concurrent1 and independent work, Generative Scene Graph Networks (GSGN [13]), proposes a variational autoencoder architecture for decomposing images into objects and parts using a tree-structured latent scene graph that is similar to our scene graph representation. However, GSGN learns RGB appearance models of objects and their parts, uses an inference network instead of a hybrid of data-driven and model-based inference, was not evaluated on real images or scenes, and uses more restricted scene graphs that cannot represent objects with independent 6DoF pose. GSGN builds on an earlier deep generative model [17] that generates multi-object scenes but does not model dependencies between object poses and was not quantitatively evaluated on real 3D scenes. Incorporating a learned inference network for jointly proposing scene graphs into our framework is an interesting area for future work. The term ‘scene graph’ has also been used in computer vision to refer to various less related graph representations of scenes [3, 8, 32].
Probabilistic programming for computer vision Prior work has used probabilistic programs to represent generative models of images and implemented inference in these models using probabilistic programming systems [27, 25]. Unlike these prior works, which relied on manually specified and/or semi-parametric shape models, 3DP3 learns object shapes non-parametrically. 3DP3 also models occlusion of one 3D object by another; uses a novel hierarchical scene graph prior that allows for dependencies between object poses in the prior; uses a novel involutive MCMC [10] kernel for inferring scene graph structure; and uses a novel pseudo-marginal approach for handling uncertainty about object shape during inference. We also present a proof of concept that our system can infer the presence and pose of fully occluded objects.
6DoF object pose estimation We use 6DoF estimation of object pose from RGBD images as an example application. Registration of point clouds [5] can be used to estimate the 6DoF pose of objects with known 3D geometry from depth images. Many recent 6DoF object pose estimators use deep learning [38, 35] and many also take depth images [37, 34]. Some pose estimation methods model scene structure, contact relationships, stability, or other semantic information [21, 8, 24, 16, 4], and some use probabilistic inference [15, 7, 19, 14]. To our knowledge, we present the first 6DoF pose estimator that uses Bayesian inference about the structure of hierarchical 3D scene graphs.
Learning models of novel 3D objects Classic algorithms for structure-from-motion infer a 3D model of a scene from multiple images [33, 1]. Our approach for learning the shape of novel 3D objects produces coarse-grained probabilistic voxel models of objects that can represent uncertainty about the occupancy of self-occluded volumes. Integrating other representations of object shape and object appearance [29] with our scene graph representation is a promising area of future work.

heading: 3 3DP3 generative modeling framework
text: The core of 3DP3 is a generative modeling framework that represents a scene in terms of discrete objects, the 3D shape of each object, and a hierarchical structure called a scene graph that relates the poses (position and orientation) of the objects. This section describes 3DP3’s object shape and scene graph latent representations, a family of prior distributions on these latent representations, and an observation model for image-based observations of scenes. Figure 1 shows the combined generative model written as a probabilistic program.
1An early version of our work [41] is concurrent with an early version of GSGN [12]
box
Input RGB Image Input Depth Image Inferred Poses Inferred Scene Graph Depth Reconstruction
Inference
(a) Inferring a hierarchical 3D scene graph from an RGB-D image with 3DP3. Our model knows that objects often lay flat on other objects, which allows for the depth pixels of one object to inform the pose of other objects. Our algorithm also infers when this knowledge is relevant (e.g. the clamp on the left, represented by the purple node, is laying flat on the box), and when it is not (e.g. the clamp on the right, represented by the red node, is not laying flat on any other object).
(b) 3DP3 uses a structured generative model of 3D scenes, represented as a probabilistic program. The model uses a prior over object shapes that can be learned from data, a prior over scene structure that is a probability distribution on graphs, a traversal of the scene graph starting at the world node r to compute object poses, and a robust likelihood model for depth images. In the graph at right, the world node r (not shown) is the parent of the grey node (box) and the red node (right clamp) because those objects are not layting flat on other objects.
Figure 1: (a) A scene graph inference task and (b) the 3DP3 generative model.

heading: 3.1 Objects
text: The most basic element of our generative modeling framework are rigid objects. The first stage in our generative model prior encodes uncertainty about the 3D shape of M types of rigid objects that may or may not be encountered in any given scene.
Voxel 3D object shapes We model the coarse 3D shape of rigid objects using a voxel grid with dimensions h,w, l ∈ N and cells indexed by (i, j, `) ∈ [h]×[w]×[l]. Each cell has dimension s×s×s for resolution s ∈ R+, so that the entire voxel grid represents the cuboid [0, h ·s]× [0, w ·s]× [0, l ·s]. All objects are assumed to fit within the cuboid. An object’s shape is defined by a binary assignment O ∈ {0, 1}h×w×l of occupancy states to each cell in the voxel grid, where Oij` = 1 indicates that cell (i, j, `) is occupied and Oij` = 0 indicates it is free. Each object also has a finite set of contact planes through which the object may be in flush contact with the contact planes of other objects in physically stable scenes. For example, in Figure 2, the table has a contact plane for its top surface, the yellow sugar box has six contact planes, one for each of its six faces, and the bottom contact plane of the sugar box is in flush contact with the top contact plane of the table. The pose of a contact plane relative to its object is a function of the object shape O. To simplify notation, we denote the set of contact planes for any object by F .
Prior distributions on 3D object shape We assume there are M distinct object types, and each object type m ∈ {1, . . . ,M} has an a-priori unknown shape, denoted O(m). Let O(1:M) :=
(O(1), . . . ,O(M)). The prior distribution on the shape of each object type m is denoted p(O(m)). Although our inference algorithm (Section 5) only requires the ability to sample jointly from p(O(1:M)), we assume shapes of object types are independent in the prior (p(O(1:M)) = ∏M m=1 p(O
(m))). Section 4 shows how to learn a specific shape prior p(O(m)) for an object type from depth images.

heading: 3.2 Scenes
text: Given a collection of M known object types and their shapes, our model generates scenes with N objects by randomly selecting an object type for each object and then sampling a 6DoF object pose for each object. Instead of assuming that object poses are independent, our model encodes an inductive bias about the regularities in real-world scenes: objects are often resting in flush contact with other objects (e.g. see Figure 2). We jointly sample dependent object poses using a flexible hierarchical scene graph, while maintaining uncertainty over the structure of the graph.
Hierarchical scene graphs We model the geometric state of a scene as a scene graph G (Figure 2), which is a tuple G = (G,θ) where G = (V,E) is a directed rooted tree and θ are parameters. The vertices V := {r, v1, . . . , vN} represent N + 1 3D coordinate frames, with r representing the world coordinate frame. An edge (u, v) ∈ E indicates that coordinate frame v is parametrized relative to frame u, with parameters θv . The 6DoF pose of frame v relative to frame u with pose xu ∈ SE(3) is given by a function ∆xv, where ∆xv(θv) ∈ SE(3) and xv := xu ·∆xv(θv). Here, · is the SE(3) group operation, and the world coordinate frame is defined as the identity element (xr := I).
Modeling flush contact between rigid objects While the vertices of scene graphs G can represent arbitrary coordinate frames in a scene (e.g. the coordinate frames of articulated joints, object poses), in the remainder of this paper we assume that each vertex v ∈ V \ {r} corresponds to the pose of a rigid object. We index objects by 1, . . . , N , with corresponding vertices v1, . . . , vN . We assume that each object i has an object type ci ∈ {1, . . . ,M}. For vertices v that are children of the root vertex r, θv ∈ SE(3) defines the absolute 6DoF pose of the corresponding object (∆xv(θv) = θv). For vertices v that are children of a non-root vertex u, the parameters take the form θv = (fv, f ′v, av, bv, zv, φv) and represent a contact relationship between the two objects: fv and f ′v indicate which contact planes of the parent and child objects, respectively, are in contact. (av, bv) ∈ R2 is the in-plane offset of the origin of plane fv of object v from the origin of plane f ′v of object u. zv ∈ R is the perpendicular distance of the origin of plane fv of object v from plane f ′v of object u. φv ∈ S2 × S1 represents the deviation of the normal vectors of the two contact planes from anti-parallel (in S2) and a relative in-plane rotation of the two contact planes (in S1). The relative pose ∆xv(θv) of v with respect to u is the composition (in SE(3)) of three relative poses: (i) v with respect to its plane fv , (ii) v’s plane fv with respect to u’s plane f ′v , and (iii) u’s plane f ′ v with respect to u. The 6DoF poses of all objects (xv for v ∈ V \ {r}) are computed by traversing the scene graph while taking products of relative poses along paths from the root r.
Prior distributions on scene graphs We now describe our prior on scene graphs, given object models O(1:M). We assume the number of objects N in the scene is known (see the supplement for a generalization to unknown N ). We first sample the types ci ∈ {1, . . . ,M} of all objects from an
exchangeable distribution p(c) where c := (c1, . . . , cN ). This includes as a special case distributions where all types are represented at most once among the objects ( ∑N i=1 1[ci = c] ≤ 1), which is the case in our experiments. Next, we sample the scene graph structure G from p(G). We experiment with two priors p(G): (i) a uniform distribution on the set of (N + 1)N−1 directed trees that are rooted at a vertex r, and (ii) δG0(G), where G0 is a graph on N + 1 vertices where (r, v) ∈ E for all v ∈ V \ {r} so that each object vertex has an independent 6DoF pose. For objects whose parent is r (the world coordinate frame), we sample the pose θv ∼ punif , which samples the translation component uniformly from a cuboid scene extent, and the orientation uniformly over SO(3). For objects whose parent is another object u, we sample the choice of contact planes (fv, f ′v) ∈ F × F uniformly, (av, bv) ∼ Uniform([−50cm, 50cm]2), zv ∼ N(0, 1cm), the S2 component of φv from a von Mises–Fisher (vMF) distribution concentrated (κ = 250) on anti-parallel plane normals, and the S1 component from Uniform(S1). We denote this distribution pcont(θv). Note that the parameters of pcont were not tuned or tailored in any detailed way—they were chosen heuristically based on the rough dimensions of table-top objects. The resulting prior over all of the latent variables is:
p(O(1:M), c, G,θ) =
( M∏
m=1
p(O(m))
) 1
(N + 1)N−1 p(c) ∏ v∈V :
(r,v)∈E
punif(θv) ∏
(u,v)∈E: u 6=r
pcont(θv) (1)

heading: 3.3 Images
text: Our generative model uses an observation model that generate synthetic image data given object shapes O(1:M) and a scene graph G containing N objects. We now describe the observation model for depth images that is used in our main experiments (Section 6).
Likelihood model for depth images We first convert an observed depth image into a point cloud Y. To model a point cloud Y ∈ RK×3 with K points denoted yi ∈ R3, we use a likelihood model based on rendering a synthetic depth image of the scene graph. Specifically, given the object models O(m) for each m ∈ {1, . . . ,M}, the object types c, the scene graph G, and the camera intrinsic and extrinsic parameters relative to the world frame, we (i) compute meshes from each O(m), (ii) compute the 6DoF poses (xv) of objects with respect to the world frame by traversing the scene graph G, and (iii) render a depth image Ĩ of G using an OpenGL depth buffer, and (iv) unproject the rendered depth image to obtain a point cloud Ỹ with K̃ points (K̃ is the number of pixels in the depth image). We then generate an observed point cloud Y ∈ RK×3 by drawing each point from a mixture:
p(Y|O(1:M), c, G,θ) := K∏ i=1 C · 1 B + 1− C K̃ K̃∑ j=1 1[||yi − ỹj ||2 ≤ r] 4 3πr 3  (2) for some 0 < C < 1 and some r > 0. The components of this mixture are uniform distributions over the balls of radius r centered at each point in Ỹ (with weights (1−C)/K̃) and a uniform distribution over the scene bounding volume B (weight C).2

heading: 4 Learning object shape models
text: 3DP3 does not require hard-coded shape models. Instead, it uses probabilistic inference to learn non-parametric models of 3D object shape p(O(m)) that account for uncertainty due to self-occlusion. We focus on the restricted setting of learning from scenes containing a single isolated object (N = 1) of known type (c1). Our approach works best for views that lead to minimal uncertainty about the exterior shape of the object; more general, flexible treatments of shape learning and shape uncertainty are beyond the scope of this paper.
First, we group the depth images by the object type (c1), so that we have M independent learning problems. Let I1:T := (I1, . . . , IT ) denote the depth observations for one object type, with object shape denoted O. The learning algorithm uses Bayesian inference in another generative model p′. The posterior p′(O|I1:T ) produced by this algorithm becomes the prior p(O) used in Section 3.1.
2By using a distribution that is uniform over a small, spherical region rather than a Gaussian distribution, we avoid (via k-d trees) computing pairwise distances between all points in Y and Ỹ, resulting in ≈ 10x speedup.
We start with a uninformed prior distribution p′(O) := ∏h
i=1 ∏w j=1 ∏l `=1 p Oij` occ (1 − pocc)(1−Oij`)
on the 3D shape of an object type, for a per-voxel occupancy probability pocc (in our experiments, 0.5). We learn about the object’s shape by observing a sequence of depth images I1:T that contain views of the object, which is assumed to be static relative to other contents of the scene, which we call the ‘map’ M. (In our experiments the map contains the novel object, a floor, a ceiling, and four walls of a rectangular room). We posit the following joint distribution over object shape (O) and the observed depth images, conditioned on the map (M) and the poses of the camera relative to the map over time (x1, . . . ,xT ∈ SE(3)): p′(O, I1:T |M,x1:T ) := p′(O) ∏T t=1 p
′(It|O,M,xt). The likelihood p′ is a depth image likelihood on a latent 3D voxel occupancy grid (see supplement for details). For this model, we can compute p′(O|M,x1:T , I1:T ) = ∏ ij` p
′(Oij`|M,x1:T , I1:T ) exactly using ray marching to decide if a voxel cell is occupied, unoccupied, or unobserved (due to being occluded by another occupied cell), and the resulting distribution on O can be compactly represented as an array of probabilities (∈ [0, 1]h×w×l). However, in real-world scenarios the map M and the camera poses x1:T are not known with certainty. To handle this, our algorithm takes as input uncertain beliefs about M and x1:T (qSLAM(M,x1:T ) ≈ p′(M,x1:T |I1:T )) that are produced by a separate probabilistic SLAM (simultaneous localization and mapping) module, and take the form of a weighted collection of K particles (M(k),x(k)1:T ): qSLAM(M,x1:T ) = ∑K k=1 wkδM(k)(M)δx(k)1:T
(x1:T ). Various approaches to probabilistic SLAM can be used; we implemented it using sequential Monte Carlo (SMC) in Gen (more detail in supplement). From the beliefs qSLAM(M,x1:T ) produced by SLAM, we approximate the object shape posterior via: p̂′(O|I1:T ) := ∫∫ p′(O|M,x1:T , I1:T )qSLAM(M,x1:T )dMdx1:T = K∑
k=1
wkp ′(O|M(k),x(k)1:T , I1:T )
Note that while p′(O|M(k),x(k)1:T , I1:T ) for each k can be compactly represented, the mixture distribution p̂′(O|I1:T ) lacks the conditional independencies that make this possible. To produce a more compact representation of beliefs about the object’s shape, we fit a variational approximation qϕ(O) that assumes independence among voxels (qϕ(O) := ∏ i∈[h] ∏ j∈[w] ∏ `∈[l] ϕ Oij` ij` ·(1−ϕij`)(1−Oij`)) to p̂′(O|I1:T ) using ϕ∗ := arg minϕ KL(p̂′(O|I1:T )||qϕ(O)) (see supplement for details). This choice of variational family is sufficient for representing uncertainty about the occupancy of voxels in the interior of an object shape. Note that our shape-learning experiments did not result in significant uncertainty about the exterior shape of objects3, and in the presence of such uncertainty, a less severe variational approximation may be needed for robust inference of scene graphs from depth images. Fig. 3 shows input depth images (I1:T ) and resulting shape prior learned from T = 5 observations. After learning these shape distributions qϕ(O) ≈ p̂′(O|I1:T ) for each distinct object type, we use them as the shape priors p(Oi) within the generative model of Section 3. The supplement includes the results of a quantitative evaluation of the accuracy of shape learning.
3The lack of significant exterior shape uncertainty in shape-learning experiments allowed us to implement an optimization: Instead of the relative poses of an object’s contact planes depending on O as described in Section 3, we assign each object type a set of six contact planes derived from the faces of the smallest axis-aligned bounding cuboid that completely contains all occupied voxels in one sample O from the learned prior p(O) := qϕ(O).

heading: 5 Building blocks for approximate inference algorithms
text: This section first describes a set of building blocks for approximate inference algorithms that are based on the generative model of Section 3. We then describe how to combine these components into a scene graph inference algorithm that we evaluate in Section 6.
Trained object detectors It is possible to infer the types of objects in the scene (c) via Bayesian inference in the generative model (see supplement for an example that infers c as well as N in a scene with a fully occluded object, via Bayesian inference). However, for scenes where objects are not fully or nearly-fully occluded, and where object types have dissimilar appearance, it is possible to train fast object detectors that produce an accurate point estimate of c given an RGB image.
Trained pose estimators In scenes without full or nearly-full occlusion, it is also possible to employ trained pose estimation methods [37] to give independent estimates of the 6DoF pose of each object instance in the image. However, inferring pose is more challenging than inferring c, and occlusion, self-occlusion, and symmetries can introduce significant pose uncertainty. Therefore, we only use trained pose estimators (e.g. [37]) to (optionally) initialize the poses of objects before Bayesian inference in the generative model, using the building blocks below.
Data-driven Metropolis-Hastings kernels on object pose We employ Metropolis-Hastings (MH) kernels, parametrized by choice of object i ∈ {1, . . . , N}, that take as input a scene graph G, propose new values (θ′vi ) for the scene graph parameters of object i, construct a new proposed scene graph G
′, and then accept or reject the move from G to G′ based on the MH rule. For objects v whose parent is the world frame ((r, v) ∈ E), we use a data-driven proposal distribution centered on an estimate (x̂v) of the 6DoF object pose obtained with ICP (a spherical normal distribution concentrated around the estimated position, and a vMF distribution concentrated around the estimated orientation). We also use kernels with random-walk proposals centered on the current pose. For objects whose parent is another object ((u, v) ∈ E for u 6= r), we use a random-walk proposal on parameters (avi , bvi , zvi). Note that when the pose of an object is changed in the proposed graph G′, the pose of any descendant objects is also changed.4 Each of these MH kernels is invariant with respect to p(G,θ|c,Y).
Involutive MCMC kernel on scene graph structure To infer the scene graph structure G, we employ a family of involutive MCMC kernels [10] that propose a new graph structure G′ while keeping the poses (xv) of all objects fixed. The kernel takes a graph structure G and proposes a new graph structure G′ (Figure 4) by: (i) randomly sampling a node v ∈ V \ {r} to ‘sever’ from the tree, (ii) randomly choosing a node u ∈ V \ {v} that is not a descendant of the severed node on which to graft v, (iii) forming a new directed graph G′ over vertices V by grafting v to u; by Lemma O.7.1 the resulting graph G′ is also a tree. Note that there is an involution g on the set of all pairs (G, v, u) satisfying the above constraints. That is, if (G′, v′, u′) = g(G, v, u) then (G, v, u) = g(G′, v′, u′). (This implies, for example, that u′ is the parent of v in G.) Note that this set of transitions is capable of changing the parent vertex of an object to a different parent object, changing the parent vertex of an object from the root (world frame) to any other object, or changing the parent vertex from another object to the root, depending on the random choice of v and u. We compute new values for parameters (θv) for the severed node v and possibly other vertices such that the poses of all vertices are unchanged. See supplement for the full kernel and a proof that it is invariant w.r.t. p(G,θ|c,Y).
4It is possible to construct an involutive MCMC kernel that does not change the poses of descendant objects.
Approximately Rao–Blackwellizing object shape via pseudo-marginal MCMC The acceptance probability expressions for our involutive MCMC and MH kernels targeting p(G,θ|c,Y) include factors of the form p(Y|c, G,θ), which is an intractable sum over the latent object models: p(Y|c, G,θ) = ∑ O(1:M) p(O
(1:M))p(Y|O(1:M), c, G,θ). To overcome this challenge, we employ a pseudo-marginal MCMC approach [2] that uses unbiased estimates of p(Y|c, G,θ) obtained via likelihood weighting (that is, sampling several times from p(O(1:M)) and averaging the resulting p(Y|O(1:M), c, G,θ)). The resulting MCMC kernels are invariant with respect to an extended target distribution of which p(G,θ|c,Y) is a marginal (see supplement for details). We implemented an optimization where we sampled 5 values for O(1:M) and used these samples within every estimate of p(Y|c, G,θ) instead of sampling new values for each estimate. Because our learned shape priors did not have significant exterior shape uncertainty, this optimization did not negatively impact the results.
Scene graph inference and implementation The end-to-end scene graph inference algorithm has three stages. First, we obtain c from either an object detector or because it is given as part of the task (this is the case in our experiments; see Section 6 for details). Second, we obtain initial estimates x̂v of 6DoF object poses xv for all object vertices v via maximum-a-posteriori (MAP) inference in a restricted variant of the generative model with graph structure G fixed to G0 (so there are no edges between object vertices). This MAP inference stage uses the data-driven Metropolis-Hastings kernels on poses, and (optionally) trained pose estimators (see Section 6 for the details, which differ between experiments). Third, we use the estimated poses to initialize an MCMC algorithm targeting p(G,θ|c,Y) with state G← G0 and θv ← x̂v for each v ∈ V \ {r}. The Markov chain is a cycle of the involutive MCMC kernel described above with a mixture of the Metropolis-Hastings kernels described above, uniformly mixed over objects. We wrote the probabilistic program of Figure 1 in Gen’s built-in modeling language. We implemented the data-driven and involutive MCMC kernels, and pseudo-marginal likelihood, and integrated all components together, using Gen’s programmable inference support. Our code is available at https://github.com/probcomp/ThreeDP3.

heading: 6 Experiments
text: We evaluate our scene graph inference algorithm on the YCB-Video [6] dataset consisting of real RGBD images and YCB-Challenging, our own synthetic dataset of scenes containing novel viewpoints, occlusions, and contact structure. We use the evaluation protocol of the Benchmark for 6DoF Object Pose Estimation (BOP) Challenge [20], in which an RGB-D image and the number of objects in the scene and their types are given, and the task is to estimate the 6DoF pose of each object.

heading: 6.1 Pose estimation from real RGB-D images
text: YCB-Video is a standard robotics dataset for training and evaluating 3D perception systems [6]. We first learn shape priors (Section 4) from just 5 synthetic images for each object type. We use DenseFusion [37], a neural 6DoF pose estimator, for pose initialization in the MAP phase of our inference algorithm. To measure pose estimation accuracy, we use the average closest point distance (ADD-S [38, 37]) which estimates the average closest point distance between points on the object model placed at the predicted pose and points on the model placed at the ground-truth pose. Table 1
shows the quantitative results. For almost all objects, our algorithm (3DP3) is more accurate than an ablation (3DP3*) that fixes the structure so that there are no contact relationships, and the ablation is more accurate than DenseFusion. This suggests that both the rendering-based likelihood and inference of structure contribute to 3DP3’s more accurate 6DoF pose estimation. Figure 5 shows examples of corrections that 3DP3 makes to DenseFusion’s estimates.

heading: 6.2 Generalization to challenging scenes
text: Next, we evaluated our algorithm’s performance on challenging scenes containing novel viewpoints, occlusions, and contact structure. Our synthetic YCB-Challenging dataset consists of 2000 RGB-D images containing objects from the YCB object set [6] in the following 4 categories of challenging scenes: (i) Single object: Single object in contact with table, (ii) Stacked: Stack of two objects on a table, (iii) Partial view: Single object not fully in field-of-view, (iv) Partially Occluded: One object partially occluded by another. For this experiment, the MAP stage of our algorithm uses an alternative initialization (see supplement) that does not use DenseFusion. We evaluate 3DP3 and the 3DP3* ablation alongside DenseFusion [37] and another state-of-the-art baseline, Robust6D [34]. For most scenes and objects, our approach significantly outperforms the baselines (Table 2). In Table 3, we assess 3DP3’s robustness by inspecting the error distribution at the 1st, 2nd, and 3rd quartile for each scene type and object type. At Q3, 3DP3 consistently outperforms the baselines and we find that the drop in performance from Q1 and Q3 is less for 3DP3 than the baselines.

heading: 7 Discussion
text: This paper presented 3DP3, a framework for generative modeling, learning, and inference with structured scenes and image data; and showed that it improves the accuracy of 6DoF object pose estimation in cluttered scenes. We used probabilistic programs to conceive of our generative model and represent it concisely; and we used a probabilistic programming system [11] with programmable inference [28] to manage the complexity of our inference and learning algorithm implementations. The current work has several limitations: Our algorithm runs ≈ 20x slower than the DenseFusion baseline. Our shape-learning algorithm requires that the training scenes contain only the single novel object, whose identity is known across training frames. Adding the ability to segment and learn models of novel objects in cluttered scenes and automatically train object detectors and pose estimators for these objects from short RGB-D video sequences, is an ongoing direction of work. The model also does not yet incorporate some important prior knowledge about scenes—interpenetration of objects is permitted, and constraints on physical stability are not incorporated. More experiments are also needed to understand the implications of a Bayesian treatment of 3D scene perception.

heading: 8 Acknowledgements
text: The authors acknowledge Javier Felip Leon (Intel) for helpful discussions and a prototype depth renderer, and Omesh Tickoo (Intel) for helpful discussions. This work was funded in part by the DARPA Machine Common Sense program (Award ID: 030523-00001); by the Singapore DSTA / MIT SCC collaboration; by Intel’s Probabilistic Computing Center; and by philanthropic gifts from the Aphorism Foundation and the Siegel Family Foundation. We thank Alex Lew, Tan Zhi-Xuan, Feras Saad, Cameron Freer, McCoy Becker, Sam Witty, and George Matheos for helpful feedback.

Review summaries:
This paper proposes a probabilistic model for reconstructing a scene of multiple 3D objects. The key advantage of the approach is that it is able to reason about the shapes of objects, the relations between them (via the scene graph) and their contacts. This ability to use physical constraints for scene parsing is a major limitation of deep learning systems. The results show that the proposed method is significantly more accurate than two recent approaches that are based on deep models (i.e. DenseFusion and Robust6D).  The main limitation of the model seems to be the inference speed, which is currently not realtime. 

This paper presents a method for multi-object 6DoF pose estimation with a "probabilistic programming" model. Before inference, the model learns priors on 3D occupancy/shape for objects. Then, given the number of objects in the scene and the classes for objects in a test scene (observed in a depth map), inference proceeds by MCMC sampling scene graphs and poses and occupancies, to minimize a depth reconstruction error. The results look good on YCB and on a new synthetic dataset based on YCB objects, against two recent 6DoF pose estimators, showing good results. 

The paper presents an approach for inferring the 3D pose and shape of multi-object tabletop scenes from a set of multi-view depth frames as input.  The approach is based on a generative model incorporating 3D scene graph generation, as well as per-object category voxelized shape priors.  At test time, the inference algorithm parses the input to a 3D scene graph using MCMC-based updates to the scene graph structure, constrained by physical contact relations between the objects and the object shape priors.  The approach is evaluated on the 6DoF 3D pose estimation task using two datasets: a set of real scenes from the YCB-video dataset, and a set of synthetically generated scenes using reconstructed YCB objects (YCB-Challenging).  The experiments compare the proposed approach against baselines from prior work (specifically, DenseFusion and Robust6D -- both trained in a supervised fashion on larger RGB-D training datasets).  An ablation of the approach without scene graph structure inference is also used as a comparison point.  The experiments show that the approach mostly outperforms the baselines and ablation in terms of accuracy of the predicted 6DoF pose for the objects in the scene.  An additional analysis incorporates one of the baselines (DenseFusion) to initialize the estimated pose used by the proposed method. Improvements to the estimated pose are observed in some challenging scenarios from the YCB-Video test scenes.