3DP3: 3D Scene Perception via Probabilistic Programming
Authors: Nishad Gothoskar
Marco Cusumano-Towner
Ben Zinberg
Matin Ghavamizadeh
Falk Pollok
Austin James Garrett
Joshua B. Tenenbaum
Dan Gutfreund
Vikash Mansinghka
Venue: NeurIPS 2021 Poster (2021)
Keywords: probabilistic programming
approximate inference
scene understanding
object pose estimation

Abstract: We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images. 3DP3 uses (i) voxel models to represent the 3D shape of objects, (ii) hierarchical scene graphs to decompose scenes into objects and the contacts between them, and (iii) depth image likelihoods based on real-time graphics. Given an observed RGB-D image, 3DP3's inference algorithm infers the underlying latent 3D scene, including the object poses and a parsimonious joint parametrization of these poses, using fast bottom-up pose proposals, novel involutive MCMC updates of the scene graph structure, and, optionally, neural object detectors and pose estimators. We show that 3DP3 enables scene understanding that is aware of 3D shape, occlusion, and contact structure. Our results demonstrate that 3DP3 is more accurate at 6DoF object pose estimation from real images than deep learning baselines and shows better generalization to challenging scenes with novel viewpoints, contact, and partial observability.

3DP3: 3D Scene Perception via Probabilistic Programming
Nishad Gothoskar!
Marco Cusumano-Towner
Ben Zinberg'
Matin Ghavamizadeh
Falk Pollok2
Austin Garrett
Joshua Tenenbaum'
Dan Gutfreund
Vikash K. Mansinghka'
MIT
MIT-IBB Watson Lab
(nishad, marcoct bzinberg mghavami, jbt, ( (falk, pollok, austin. t
vkm) Ceit.eedie
dgutfre@us bmm com
Abstract
We present 3DP3, framework for inverse graphics that uses inference in struc- tured generative model of objects, scenes, and images. 3DP3 uses (i) voxel models represent the 3D shape of objects, (ii) hierarchical scene graphs to decompose scenes into objects and the contacts between them, and (iii) depth image likelihoods based real-time graphics. Given observed RGB-D image, 3DP3's inference al algorithm infers the underlying latent 3D scene, including the object poses and parsimonious joint parametrization of these poses, using fast bottom-up pose proposals, novel involutive MCMC updates of the scene graph structure, and, op- tionally, neural object detectors and pose estimators. We show that 3DP3 enables scene understanding that is aware of 3D shape, occlusion, and contact structure. Our results demonstrate that 3DP3 is more accurate at 6DoF object pose estimation from real images than deep learning baselines and shows better generalization to challenging scenes with novel viewpoints, contact, and partial observability.
Introduction
A striking feature of human visual intelligence is our ability to learn representations of novel objects from a limited amount of data and then robustly pereieve 3D scenes containing those objects. We can immediately generalize across large variations in viewpoint, occlusion, lighting, and clutter. How might we develop computational vision systems that can the same?
This paper presents a generative model for 3D scene perception, called 3DP3. Object shapes are learned via probabilistic inference in voxel occupancy model that coarsely captures 3D shape and uncertainty due to self-occlusion (Section 4). Scenes are modeled via hierarchical 3D scene graphs that can explain planar contacts between objects without foreing scenes fit rigid struetual assumptions (Section 3). Images are modeled by real-time graphics and robust likelihoods on point clouds. We cast 3D scene understanding as approximate probabilistic inference in this generative model. We develop a novel inference algorithm that combines data-driven Metropolis-Hasting kernels over object poses, involutive MCMC kernels over scene graph structure, pseudo-marginal integration over uncertain object shape, and existing deep learning object detectors and pose estimators (Section 5). This architecture leverages inference the generative model to provide common sense constraints that fix errors made by bottom-up neural detectors. Our experiments show that 3DP3 more accurate and robust than deep learning baselines 6DoF pose estimation for challenging synthetic and real-world scenes (Section 6). Our model and inference algorithm are implemented in the Gen [11] probabilistic programming system.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2 Related Work
Analysis-byy by-sisss-y-yythhsiss approaches to computer vision A long line of work has interpreted computer
vision as the inverse problem computer graphics [22. 39, 26, 23]. This heallysss-b--yyyhhhhee approach has been used for various tasks including character recognition, CAPTCHA-breakee lane detection, object pose estimation, and human pose estimation [40, 36, 27. 30, 18. 31]. To our knowledge, our work is the first to use an analysis-by-sye bayyss--y-ssnthesiss approach to infer a hierarchical 3D) object-based representation of real multi-object scenes while exploiting inductive biases about the contacts between objects.
Hierarchical latent 3D scene representations We use a scene graph representation [41] that is closely related to hierarchical scene graph representations in computer graphics [9]. Unlike in graphics, we address the inverse problem of inferring hierarchical scene graphs from observed image data. Inferring hierarchical 3D scene graphs from RGB or depth images in probabilistic framework relatively unexplored. One concurrent' and independent work, Generative Scene Graph Networks (GSGN [13). proposes a variational autoencoder architecture for decomposing images into objects and parts using tree-structured latent scene graph that similar to our scene graph representation. However, GSGN learns RGB appearance models of objects and their parts, uses an inference network instead of a hybrid of data-driven and model-based inference, was not evaluated on real images or scenes, and uses more restricted scene graphs that cannot represent objects with independent 6DoF pose. GSGN builds on an earlier deep generative model [17] that generates multi-object scenes but does not model dependencies between object poses and was not quantitatively evaluated on real 3D scenes. Incorporating a learned inference network for jointly proposing scene graphs into our framework is interesting area for future work. The term 'scene graph' has also been used in computer vision to refer to various less related graph representations of scenes [3, 8, 32].
Probabilistic pregramming for computer vision Prior work has used probabilistic programs to represent generative models of images and implemented inference in these models using probabilistic programming systems [27, 25). Unlike these prior works, which relied on manually specified and/or cani-parametric shape models, 3DP3 learns object shapes non- parametrically. 3DP3 also models occlusion of one 3D object by another; uses a novel hierarchical scene graph prior that allows for dependencies between object poses in the prior; uses novel involutive MCMC [10] kernel for inferring scene graph structure; and uses novel pseudo-marginal approach for handling uncertainty about object shape during inference. We also present a proof of concept that our system can infer the presence and pose of fully occluded objects.
6DoF object pose estimation We use 6DoF estimation of object pose from RGBD images as an example application. Registration of point clouds [5] can be used to estimate the 6DoF pose of objects with known 3D geometry from depth images. Many recent 6DoF object pose estimators use deep learning [38, 35] and many also take depth images [37, 34]. Some pose estimation methods model scene structure, contact relationships, stability, or other semantic information [21, 8, 24, 16, 4], and some use probabilistic inference [15, 7, 19, 14). To our knowledge, we present the first 6DoF pose estimator that uses Bayesian inference about the structure hierarchical 3D scene graphs.
Learning models of novel 3D objects Classic algorithmss for structure- fructure-from-iie from-motion infer 3D model of a scene from multiple images [33, 1]. Our approach for learning the shape of novel 3D objects produces coarse- gonse-grrined probabilistic voxel models of objects that can represent uncertainty about the occupancy of self-occluded volumes. Integrating other representations of object shape and object appearance [29] with our scene graph representation is a promising area of future work.
3 3DP3 generative modeling framework
The core of 3DP3 is generative modeling framework that represents scene in terms of discrete objects, the 3D shape of each object, and a hierarchical structure called scene graph that relates the poses (position and orientation) of the objects. This section describes 3DP3's s object shape and scene graph latent representations, a family of prior distributions on these latent representations, and an observation model for image-based observations of scenes. Figure 1 shows the combined generative model written as probabilistic program.
An early version of work [41] concurrent with early version of GSGN [12]

---

Input RGB Image
Input Depth Image
Inferred Poses
Inferred Scene Graph
Depth Reconstruction
Inference
(a) Inferring hierarchical 3D seene graph from an RGB-D image with 3DP3, Our model knows that objects often lay flat on other objects, which allows for the depth pixels of one object to inform the pose of other objects. Our algorithm also infers when this knowledge is relevant (c.g. the clamp the left, represented by the purple node, is laying flat the box), and when it is not (e.g. the clamp the right, represented by the red node, is not laying flat any other object).
procedure GENNEGNNIIVVV-OOOEEEEE
parameters: M (= num object types),
(= num objects in scenc)
for (I,..., M) do
Q SOAP--SSAPPPPPOOO((
end for
~ ODBECI-CCASSE AEELLASS-ASSCMMMETSPPPPPRRE M)
= EV ~ STR.CCCUUU-UPPOOOO///
for Toro-SoRT(V, E) where (u, do
if = then
0, ~ Uxrrons--DDof-Pooo---eee
X, 8,
else
9, CONTACT-PRUOR(G,U. ((:M)
X, x. COMMUTE-REAATIISSOOooo. (L
end if end for
Y UNPROJECT (DRPTH RENDERRx, O(1M)) .))
NoSSE-MODIL(Y)
end procedure
(b) 3DP3 uses structured generative model of 3D scenes, represented as probabilistic program. The model uses prior over object shapes that can be learned from data, prior over scene structure that is a probability distribution graphs, traversal of the scene graph starting the world node compute object poses, and robust likelihood model for depth images. the graph right, the world node (not shown) grey node (box) and the red node (right clamp) because those objects are not layting flat on other objects.
parent of the
Figure 1: (a) A scene graph inference task and (b) the 3DP3 generative model.
3.1
Objects
The most basic element of our generative modeling framework are rigid objects. The first stage in our generative model prior encodes uncertainty about the 3D shape of M types of rigid objects that may or may not be encountered in any given scene.
Voxel 3D object shapes We model the coarse 3D shape of rigid objects using a voxel grid with dimensions h, w, L E IN and cells indexed by (i.j. E) [h] x [w] [1]. Each cell has dimension for resolution & E RR+, so that the entire voxel grid represents the cuboid [0, h- s] x [0, w s) x s.]. All objects are assumed to fit within the cuboid. An object's shape is defined by binary assignment O E (0, 11 1 hxxxl of occupancy states to each cell in the voxel grid, where O.ze = indicates that cell (i,j, E) is occupied and O.ze indicates is free. Each object also has a finite set of contact planes through which the object may be in flush contact with the contact planes of other objects in physically stable scenes. For example, in Figure 2, the table has a contact plane for its top surface, the yellow sugar box has six contact planes, one for each of its six faces, and the bottom contact plane of the sugar box is in flush contact with the top contact plane of the table. The pose of contact plane relative to its object is function of the object shape O. To simplify notation, we denote the set of contact planes for any object by F.
x
Prior distributions 3D object shape We assume there M distinct object types, and each object type m E {I...., M) has an a-priori unknown shape, denoted (m), Let O (1:M)
3

---

Nonation
Meaning
Section
Object shage
3.1
Number of object types
3.1
soup can coord. frame
N
Number of objects
3.2
Ony
Type of object 8
3.2
EV.
Scene gruph structure
3.2
(Jr firs Powe Poss Pecs B
World courd. frume Cbjjcct coord. frame Parametors
3.2
sugar box coord. frame
3.2
O.
3.2
contact planes
table coord. frame
(avi bus zu th
Planor contact pommeters
32
Bag F (E(3)
(x.(O.)
E SR(3)
cDoF
6Dol puse
was.
purent
322
world coord. frame
Rendered depth imuge
33
Y
Rendered cloud
33
Ohserved point cloud
Our hierarchical scene graphs encode a tree of coordinate frames representing entities in a scene and their geometric relationships (c.g. flush contact between faces of two objects).
Figure 2:
(O ..... (A4)) The prior distribution on the shape of each object type m is denoted p( (m)). Al- though our inference algorithm (Section 5) only requires the ability to sample jointly from p O (1:M)). we assume shapes of object types are independent in the prior (p(O(1:)) Section 4 shows how to learn a specific shape prior p( (((m)) for an object type from depth images.
I p O (m))).
3.2 Scenes
Given a collection of M known object types and their shapes, our model generates scenes with N objects by randomly selecting an object type for each object and then sampling a 6DoF object pose for each object. Instead of assuming that object poses independent, our model encodes an inductive bias about the regularities in real-world scenes: objects are often resting in flush contact with other objects (e.g. see Figure 2). We jointly sample dependent object poses using a flexible hicrarchical scene graph, while maintaining uncertainty over the structure of the graph.
Hierarchical scene graphs We model the geometric state of a scene as a scene graph 9 (Figure 2), which is a tuple G = (G, where G = (V, E`) is a directed rooted tree and 0 are parameters. The vertices V = [r, .1...., UN] }o represent N + 1 3D coordinate frames, with r representing the world coordinate frame. An edge (u,v) E E indicates that coordinate frame v is parametrized relative to frame u, with parameters 0g. The 6DoF pose of frame v relative to frame u with pose x,, E (3 is given by function Ax,, where Ax, E SE(3) and x, =- X., (O.). Here, is the SE) group operation, and the world coordinate frame is defined as the identity element (x, := I).
Modeling flush contact between rigid objects While the vertices of scene graphs G can represent arbitrary coordinate frames in scene (e.g. the coordinate frames of articulated joints, object poses), in the remainder of this paper we assume that each vertex v E {r} corresponds to the pose of a rigid object. We index objects by I,....N, N. with corresponding vertices ......, UN. We assume that each object é has an object type c, E M.. For vertices v that are children of the root vertex 0g E SE(3) defines the absolute 6DoF pose of the corresponding object (Ax, (0.) = 0.). For vertices e that are children of a non-root vertex u, the parameters take the form 0, = (Ju, S': an; by, Zu, (x) and represent contact relationship between the two objects: fe and Jr indicate which contact planes of the parent and child objects, respectively, are in contact. (a., b,) E RR2 is the in-plane offset of the origin of plane fo of object from the origin of plane f of object u. 2, E IR is the perpendicular distance of the origin of plane fu of object e from plane f of object u. dy E S2 S represents the deviation of the normal vectors of the two contact planes from anti-parallel (in 5²) and relative in-plane rotation of the two contact planes (in St). The relative pose Ax.(0.) (0.)) of u with respect to u is the composition (in SE)) of three relative poses: (i) u with respect to its plane fe. (ii) e's plane fo with respect to u's plane fo and (iii) u's plane f with respect u. The 6DoF poses of all objects (x,, for V (r}) are computed by traversing the scene graph while taking products of relative poses along paths from the root r.
Prior distributions on scene graphs We now describe our prior on scene graphs, given object models (1:M), We assume the number of objects N in the scene is known (see the supplement for generalization to unknown N). We first sample the types Ci E {1....,M M)
of all objects from an

---

exchangeable distribution p(c) where c == (cI....,,cy) This includes as special case distributions where all types represented most among the objects (E== [ c S 1), which is the case in our experiments. Next, we sample the scene graph structure G from p( We experiment with priors p(G): (i) uniform distribution on the of (N + 1) directed trees that are rooted at a vertex r, and (ii) oG (G), where Go is a graph on N + +1 vertices where (r, v) E E for all E V {r} that each object vertex has independent 6DoF pose. For objects whose parent is (the world coordinate frame), we sample the pose 0, Punif, which samples the translation component uniformly from a cuboid scene extent, and the orientation uniformly over SO(3). For objects whose parent is another object u, we sample the choice of contact planes (fe.f) 50cm]²). 2, N(0, lem), the S component o, from
x F
uniformly, (a.,, b,) ~ Uniform([-50eeem von Hises-Fisher (vMF) distribution concentrated (k 250) on anti anti-parallel plane normals, and the S component from Uniform(S') We denote this distribution Peont (.,). Note that the parameters of Peont were not tuned or tailored in any detailed way way-they were chosen heuristically based on the rough dimensions of table-top objects. The resulting prior over all of the latent variables is:
(OO G,
M II mm
II
(()))))
(N 1)N-1 p(c) II (uni
II (u,v)EE: usr
Pront (0x) (1)
DEV: (Me)EE
3.3 Images
Our generative model uses an observation model that generate synthetic image data given object shapes (1:M) and scene graph containing N objects. We now describe the observation model for depth images that is used in our main experiments (Section 6).
Likelihood model for depth images We first convert an observed depth image into point cloud Y. To model a point cloud Y E R with K points denoted y, E TR3, we use a likelihood model based on rendering synthetic depth image of the scene graph. Specifically, given the object models (mm) for each m {1,.... M) the object types c, the scene graph G, and the camera intrinsic and extrinsic parameters relative to the world frame, we (i) compute meshes from each O (mm) (ii) compute the 6DoF poses (x,) of objects with respect to the world frame by traversing the scene graph G, and (iii) render depth image I of G using an OpenGL. depth buffer, and (iv) unproject the rendered depth image to obtain a point cloud with K points (K is the number of pixels the depth image). We then generate an observed point cloud Y E R x3 by drawing each point from a mixture:
K P (1AM) c, G. = II
1- K
y: 81112 s)
B
TT
J=1
for some 0 < < 1 and some r > O. The components of this mixture are uniform distributions over the balls of radius r centered each point in Y (with weights (1 ()-O)/B K) and a uniform distribution over the scene bounding volume B (weight C).
4 Learning object shape models
3DP3 does not require hard-coded shape models. Instead, it uses probabilistic inference to learn pan-paraetrric models of 3D object shape (m)) that account for uncertainty due self-occlusion. We focus on the restricted setting of learning from scenes containing single isolated object (N = 1) of known type (c1). Our approach works best for views that lead to minimal uncertainty about the exterior shape of the object; more general, flexible treatments of shape learning and shape uncertainty
are beyond the scope of this paper.
First. we group the depth images by the object type (c1), so that we have M independent learning problems. Let 1 ;= (I,.... Ir) denote the depth observations for one object type. with object shape denoted O. The learning algorithm uses Bayesian inference in another generative model p'. The posterior p' (O]I.r) produced by this algorithm becomes the prior POO) used Section 3.1.
By using distribution that uniform over small, spherical region rather than a Gaussian distribution, we avoid (via k-d trees) computing pairwise distances between all points in Y and Y, resulting in z 10x speedup.
5

---

Inferred Shape Posterior
(Cross section)
Certain
Uncertain
Figure Learning a voxel- bassed shape models (m)) for novel object from a set of depth images. Our shape priors capture uncertainty about voxel occupancy due to self-occlusion (right).
Oige Poce ( (1-) We start with uninformed prior distribution p'( IT Poeé
on the 3D shape of object type, for per-voxel occupancy probability Pone (in our experiments, 0.5). We learn about the object's shape by observing a sequence of depth images 1 that contain views of the object, which assumed to static relative other contents of the scene, which we call the 'map' M. (In our experiments the map contains the novel object, a floor, a ceiling, and four walls of a rectangular room). We posit the following joint distribution over object shape (O) and the observed depth images, conditioned on the map (M) and the poses of the camera relative to the map over time (X1..... XT E SE(3)): p'(O. M, x1.7) p0( I (I, O, M,
The likelihood p' depth image likelihood on latent 3D voxel occupancy grid (see supplement for details). For this model, we can compute p' (' XI.7, H:r) Il,ee p' ('OOjrl M, (1:)) ex- actly using ray marching to decide if voxel cell is occupied, unoccupied, or unobserved (due to being occluded by another occupied cell), and the resulting distribution on O can be compactly represented as an array of probabilities (E [o. 1] 1]xxwx(). However, in real-world scenarios the map M and the camera poses X1:T are not known with certainty. To handle this, our algorithm takes as input uncertain beliefs about M and X1:7 (ASLAM (M, X1.7) = p' (M, X1.2 [1:r) that are produced by a separate probabilistic SL.AM (simultaneous localization and mapping) module, and take the form of weighted collection of K particles (M((), >((r): 4SLAM (M, x1.T) = Ek= w:00 (M)) x (X1:T). Vari- ous approaches to probabilistic SLAM can be used: implemented using sequential Monte Carlo (SMC) in Gen (more detail in supplement). From the beliefs QSLAM (M, x1:r) produced by SLAM, we approximate the object shape posterior via:
p' D'(O = // p/ (O)M, x1.r I
)ddxx
Σ wkp' (O
(k) x L.r)
Note that while (O X1-T. I for each i can be compactly represented, the mixture distri- bution O'OO lacks the conditional independencies that make this possible. To produce more compact representation of beliefs about the object's s shape, we fit variational approximation 4 (O)
Osze that assumes independence among voxels (g)(O) (0) = [ILE(N) [I:EWw [Lee() PijE -(1 (1--)ee)(-Oupp)
(see supplement for details). This choice of variational family is sufficient for representing uncertainty about the occupancy of voxels in the interior of an object shape. Note that our shape-leaming experiments did not result in significant uncertainty about the exterior shape of objects?, and the presence of such uncertainty, less severe variational approximation may be needed for robust inference of scene graphs from depth images. Fig. shows input depth images (IIrr) and resulting shape prior learned from T = 5 observations. After learning these shape distributions (O) ~ f (O[]e)) for each distinct object type, we use them as the shape priors (O,) within the generative model of Section The supplement includes the results of quantitative evaluation of the accuracy of shape learning.
to ÿ' (O][): using 80 == arg min, KL (L))((O)) (l))
'The lack significant exterior shape uncertainty in shape-learnng experiments allowed us to implement an optimization: Instead of the relative poses of an object's 's contact planes depending on O as described in Section 3, we assign each object type set of six contact planes derived from the faces of the smallest axis axis-aligned bounding cuboid that completely contains all occupied voxels in sample O from the learned prior p(O) ge

---

G
Randomly choose: vertex sever a graft vertex (o)
G¹
Randomly choose:
G
1] vertex
2) graft vertex
Figure 4: A reversible transition between scene graph structure G and scene graph structure G'.
Building blocks for approximate inference algorithms
This section first describes a set of building blocks for approximate inference algorithms that are based on the generative model of Section 3. We then describe how combine these components into a scene graph inference algorithm that we evaluate in Section 6.
Trained object detectors is possible infer the types of objects in the scene (c) via Bayesian inference in the generative model (see supplement for an example that infers с as well as N in a scene with fully occluded object, via Bayesian inference). However, for scenes where objects are not fully or nearly-fully occluded, and where object types have dissimilar appearance, it is possible to train fast object detectors that produce an accurate point estimate of e given an RGB image.
Trained pose estimators In scenes without full or nearly-full occlusion, it is also possible to employ trained pose estimation methods [37] to give independent estimates of the 6DoF pose of each object instance in the image. However, inferring pose is more challenging than inferring and occlusion, self-occlusion, and symmetries can introduce significant pose uncertainty. Therefore, we only use trained pose estimators (e.g. [37]) to (optionally) initialize the poses of objects before Bayesian inference in the generative model, using the building blocks below.
Data-driven Metssiglis-Hastingg kernels on object pose We employ Metropolis-Hasting (MH) kernels, parametrized by choice object E (I.....M) ..). that take as input scene graph G. propose new values (0',) for the scene graph parameters of object construct new proposed scene graph G', and then accept reject the move from G to g' based on the MH rule. For objects e whose parent is the world frame ((r, v) E F.), we use a data-driven proposal distribution centered on an estimate (x,) of the 6DoF object pose obtained with ICP (a spherical normal distribution concentrated around the estimated position, and a vMF distribution concentrated around the estimated orientation). We also use kernels with random-war walk proposals centered the current pose. For objects whose parent is another object ((u, v) E E for u + r), we use a random-walk proposal on parameters (a.,, br, 2o.). Note that when the pose of an object is changed in the proposed graph G', the pose of any descendant objects is also changed. 4 Each of these MH kernels is invariant with respect to p(G. c Y).
Involutive MCMC kernel employ a family of involutive MCMC kernels [10] that propose a new graph structure G' while keeping the poses (x,) of all objects fixed. The kernel takes graph structure G and proposes new graph structure G' (Figure 4) by: (i) randomly sampling a node v E V [r] to 'sever' from the tree, (ii) randomly choosing node
graph structure To infer the scene graph structure G,
V (u) that is not descendant of the severed node which to graft v. (iii) forming a new directed graph G' over vertices V by grafting v to u; by Lemma O.7.1 the resulting graph G' is also a tree. Note that there is an involution g on the of all pairs (G, U, v,,) satisfying the above constraints. That is if (G', v', u') = g(G, v, u) then (G, v, u) = g(" v', w(). (This implies, for example, that u' is the parent of v in G.) Note that this set of transitions is capable of changing the parent vertex of an object to a different parent object, changing the parent vertex of an object from the root (world frame) any other object, or changing the parent vertex from another object to the root, depending on the random choice of v and u. We compute new values for parameters (f..) for the severed node v and possibly other vertices such that the poses of all vertices are unchanged. See supplement for the full kernel and proof that it is invariant w.r.t. p(G, 0lc, Y).
'It is possible to construct involutive MCMC kernel that does not change the poses of descendant objects.
7

---

Figure 5: Qualitative comparison between DenseFusion's 's pose estimates (top row) and estimates from 3DP3-based algorithm that is initialized with DenseFusion (bottom row) for YCB-Video frames where DenseFusion gives incorrect results. 3DP3's depth-rendering likelihood and scene graph prior can correct large errors made by DenseFusion.
Approximately Rao-Blackwellizinn object shape via pseudo-marginal MCMC The accep- tance probability expressions for our involutive MCMC and MH kernels targeting p(G, 0lc. Y) include factors of the form p( p(Yic, c, G, f). which is intractable sum over the latent object models: employ
(YYc, 0) Eoarr, p(O (OO(1:M) (1)M))))p)) (YO (I:M), G. 0). To overcome this challenge, a pseudo-marginal MCMC approach [2] that uses unbiased estimates of pYlcc G, 0) obtained via likelihood weighting (that is, sampling several times from ((O(M))) and averaging the resulting p (Y c, G, 9)). The resulting MCMC kernels are invariant with respect to an extended target distribution of which p(G, Olc, Y) a marginal (see supplement for details). We implemented an sampled values for O (1:M) and used these samples within every estimate of p( (Y|c, G, 0) instead of sampling new values for each estimate. Because our learned shape priors did not have significant exterior shape uncertainty, this optimization did not negatively impact the results.
optimization where
Scene graph inference and implementation The end-to-end scene graph inference algorithm has three stages. First, we obtain from either an object detector or because it given as part of the task (this is the case in our experiments; see Section 6 for details). Second, we obtain initial estimates &, of 6DoF object poses x, for all object vertices u via maximum-a- -a-tuunn----tttriortrio (MAP) inference in restricted variant of the generative model with graph structure G' fixed to Go (so there are no edges between object vertices). This MAP inference stage uses the data-driven kernels on poses, and (optionally) trained pose estimators (see Section 6 for the details, which differ between experiments). Third, we use the estimated poses to initialize an MCMC algorithm targeting p(G, O|c, Y) with state G +- Go and 0g, +- X, for each v E V \ {r). The Markov chain is a cycle of the involutive MCMC kernel described above with a mixture of the Heatroooli-HHasting kernels described above, uniformly mixed over objects. We wrote the probabilistic program of Figure I in Gen's 's built-in modeling language. We implemented the data-driven and involutive MCMC kernels, and pseudo- marginal likelihood, and integrated all components together, using Gen's programmable inference support. Our code available at ht attps: / //gtthtb....//////// com/ connprooocmp. /ThreeDP3
Cetropolis-Hastinnn
6 Experiments
We evaluate our scene graph inference algorithm on the VCB-Video [6] dataset consisting of real RGB- D images and YCB- (CB-Challenging, our own synthetic dataset of scenes containing novel viewpoints, occlusions, and contact structure. We use the evaluation protocol of the Benchmark for 6DoF Object Pose Estimation (BOP) Challenge [20], in which RGB-D image and the number of objects in the scene and their types are given, and the task is to estimate the 6DoF pose of each object.
6.1 Pose estimation from real RGB-D images
YCB- Video is a standard robotics dataset for training and evaluating 3D perception systems [6]. We first learn shape priors (Section 4) from just 5 synthetic images for each object type. We use DenseFusion [37], neural 6DoF pose estimator, for pose initialization in the MAP phase of our inference algorithm. To measure pose estimation accuracy, we use the average closest point distance (ADD-S [38, 37)) which estimates the average closest point distance between points on the object model placed at the predicted pose and points the model placed at the ground-truth pose. Table I
8

---

0.5cmm Threshold
1.0em Threshold
2.0cm Thresbold
Object Type
Scenes
3D33
Accuracy 31333"
31PS
Accuracy 313P3
Accuracy S13P3
D3
000. master chef
1006
0.74
0.79
0.84
0.59
1,00
1,00
1.00
1.00
1.00
cracker__boroo
868
sugar box
1005 tomato soup can
0.93
0.93
0.97
0.97
006 mustard botile
357
0.99
0.98
0.94
0.99
0,99
0.98
1.00
1,00
1.00
007 Juma fish, can
1148
0.81
0.80
0.91
1.00
1.00
0.99
1.00
1.00
1.00
108 pudding box
214
1,00
0.97
0.70
1,00
1.00
1.00
1.00
1.00
1.00
009 wew.geellboooo box
214
1,00
1,00
1.00
1.00
1,00
1.00
1.00
1,00
1.00
D10 potted ment can
766
0.80
0.78
0.79
0.89
0.88%
0.87
0..93
0.93
0.92
011 banana
379
0.98
0.96
0.52
1.00
1,00
0.97
1.00
1,00
1.00
019 109__piccer_bee___ base
570
1,00
0.99
0.99
1.00
1.00
1.00
1.00
1,00
1.00
021 blench, cleanser
1.00
mug
633
133 power, drill
1057
0.98
0.96
0.88
0.99
0.99
0.98
0.99
0.99
0.99
036 wood_ blockblock
242
0.36
0.33
0.07
0.96
0.93
0.88
1.00
1.00
1,00
037 saissons
181
0.75
0.69
0.20
0.87
0.84
0.70
0.99
0.99
0.98
140 large, marker
648
1,00
1.00
0.99
1.00
1,00
1.00
1,00
1.00
1,00
051 large_.laam
712
0.68
0.64
0.25
0.71
0.70
0.33
0.79
0.79
0.79
D52 extra large clamp
682
0.33
0.227
0.12
0.38
0.34
0.17
0.69
0.70
0.74
261_Josm_bricc
288
0.26
0.24
0.01
1.00
1,00
0.99
1.00
1,00
1,00
Table 1: Accuracy results on the real YCB-Video test set, for accuracy thresholds 0.5cm, 1.0cm, and 2.0cm, and per object type. 3DP3 is our full scene graph inference algorithm and 3DP3* is an ablation that does not infer contact relationships. '# of Scenes' = The number of test images in which that object appears, out of the total 2,949 images. 'DF" DenseFusion [37], a deep learning baseline.
shows the quantitative results. For almost all objects, our algorithm (3DP3) is more accurate than an ablation (3DP3*) that fixes the structure so that there are no contact relationships, and the ablation is more accurate than DenseFusion. This suggests that both the rendering cendering-based likelihood and inference of structure contribute to 3DP3's more accurate 6DoF pose estimation. Figure shows examples of corrections that 3DP3 makes to DenseFusion's 's estimates.
6.2 Generalization to challenging scenes
Next, we evaluated our algorithm's performance on challenging scenes containing novel viewpoints, occlusions, and contact structure. Our synthetic YCB- (CB-Challenging dataset consists of 2000 RGB-D images containing objects from the YCB object set [6] in the following 4 categories of challenging scenes: (i) Single object: Single object in contact with table, (ii) Stacked: Stack of two objects on table, (ii) Partial view: Single object not fully in beld-ffview. (iv) Partially Occluded: One object partially occluded by another. For this experiment, the MAP stage of our algorithm uses an alternative initialization (see supplement) that does not use DenseFusion. We evaluate 3DP3 and the 3DP3* ablation alongside DenseFusion [37] and another wate-offtheearr baseline, Robust6D [34]. For most scenes and objects, our approach significantly outperforms the baselines (Table 2). In Table 3, we assess 3DP3's robustness by inspecting the error distribution at the 1st, 2nd, and 3rd quartile for each scene type and object type. At Q3, 3DP3 consistently outperforms the baselines and we find that the drop in performance from QI and Q3 is less for 3DP3 than the baselines.
7 Discussion
This paper presented 3DP3, a framework for generative modeling, learning, and inference with structured scenes and image data; and showed that it improves the accuracy of 6DoF object pose estimation in cluttered scenes. We used probabilistic programs to conceive of our generative model and represent it concisely; and we used a probabilistic programming system [11] with programmable inference [28] to manage the complexity of our inference and learning algorithm implementations. The current work has several limitations: Our algorithm runs 20x slower than the DenseFusion baseline. Our shape-learing algorithm requires that the training scenes contain only the single novel object, whose identity is known across training frames. Adding the ability to segment and learn models of novel objects in cluttered scenes and automatically train object detectors and pose estimators for these objects from short RGB-D video sequences, is an ongoing direction of work. The model also does not yet incorporate some important prior knowledge about scenes intrreeesnneeennttattion of objects is permitted, and constraints on physical stability are not incorporated. More experiments are also needed to understand the implications of a Bayesian treatment of 3D scene perception.
9

---

0.5cm Threshold
1.0cm Threshold
2.0cm Threshold
Scene Type Object Type
Scenes
SDPS
Accuracy 3DP3*
313P3
303P3*
Accuracy DF
313P3
3DPS5
Accuracy
OF
R61>
011, master chef_
94
0.99
0.95
0.45
0.03
1,000
1.000
0..60
0,46
1.00
1.00
1.00
0.98
Single Object
003 cra.ecreeebrroo box
0.55
0.39
0.16
0.00
0.98
0.98
0.39
0.02
1.00
1.00
0.78
0.42
014 sugar box
0.90
0.87
0.17
0.00
1.00
1.000
0.72
0.32
1.00
1.00
1,00
1.00
lomalo, can
0.81
1.00
mossmmmttrrdbae bottle
0.86
0.79
1.00
1.00
chef
0.86
0.79
0.28
0.012
0.94
0.95
0.56
0.39
0.95
1,00
0.24
Stacked
sugar box
0.61
0.91
00S tomato soup can
0.67
0.52
0.13
0.00
0.89
0.86
0.22
0.06
0.90
0.88
0.75
0.66
006_mastard__ botile
199
0.73
0.60
0.44
0.03
0.94
0.90
0.54
0.30
0.94
0.94
0.85
0.88
0.80
1.00
1.00
Partial View
0.16
0.57
sugar box
0.63
0.59
0.00
0.00
0.89
0.89
0.08
0.04
1.00
1.00
0.73
0.68
005 Jomato soup can
0.34
0.33
0.00
0.00
0.72
0.71
0.13
0.00
0.83
0.82
0.40
0.13
006, mustard bottle
97
0.55
0.62
0.08
0.00
0.87
0.85
0.23
0.00
0.96
0.95
0.37
0.26
master cheefeheeeee
0.52
0.90
Partially Occluded
003 cracker box
0.37
0.35
0.00
1.00
1.000
1.00
0.02
1.00
1.00
1.00
1.00
[0l__suaa___ box
117
0.02
0.01
0.06
0.00
0.30
0.27
0.40
0.12
0.94
0.93
0.84
0.77
045, tomato, soup can
124
0.04
0.00
0.01
0.00
0.31
0.25
0.14
0.06
0.81
0.75
0.50
0.49
006, mostard bottlc
129
0.70
0.43
0.55
0.03
0.84
0.74
0.95
0.29
0.94
0.90
1,00
0.99
Table 2: Accuracy results on our synthetic YCB-Challenging data set. We report the number of scenes over which this accuracy is computed for each object and scene type. Accuracy is shown for 3DP3 and 3DP3*, which are our full method and an ablation that does not model contact relationships, respectively, and two deep learning baselines (DenseFusion (DF) [37] and Robust6D (R6D) [34]).
Tomato Soup
Cracker Box
Potted Meat
Sugar Box
Master Chef
ADD-S
ADD-S
ADD-S
ADD-S
ADD-S Q2
Scene Type
Method
Q1
Q2
Q3
Q1
Q2
Q3
Q1
Q2
Q3
Q1
Q2
Q3
Q1
Q3
3133 (ours)
0.35
0.39
0.41
0.43
0.49
0.54
0.36
0)40
0.45
0.38
0.43
0.48
0.37
01.4
0.48
Single object
3DP3* (ours)
0.35
0.40
0.43
0.47
0.52
0.62
0.39
0.44
0.40
0.45
0.49
0.35
0.49
DenseFusion
0,35
0.55
1.11
0.65
1.35
1.72
0.58
0.88
1.02
0.67
1.25
1.72
0.32
0.61
1.85
RobustóD
0.84
1.05
1.29
1.65
2.22
2.90
0.97
1.09
1.21
1.25
1.61
2.02
0.83
1.48
1.89
3DP3 (ours)
0.37 0.42
0.46
0.46
0.52
0.60
0.39
0.45
0.60
0.40
0.46
0.52
0.40
0.45
0.51
Stacked
3DP3 (ours)
0,38
0.42
0.48
0.50
0.60
0.79
0.40
0.46
0.64
0.43
0.49
0.61
0.41
0.47
0.56
DenseFusion
0.49
0.87
1.21
0.66
1.33
1.97
0.63
0.92
1.16
0.93
1.42
1.96
0.37
0.66
1.83
RobustóD
0.84
1.15
1.36
1.68
2.21
2.86
0.92
1.11
1.26
1.37
1.72
2.21
0.94
1.38
1.82
3DP3 (ours)
0.34
0.40
0.47
0.54
0.76
1.56
0.36
0.45
0.59
0.47
0.55
1.80
0.36
0.47
0.57
Partial view
3DP3 (ours)
0,33
0.40
0.47
0.56
0,90
1,54
0.37
0.45
0.63
0.46
0.59
181
0.36
0.46
0.58
DenseFusion
0.79
1.52
2.10
2.33
2.93
3.81
1.26
1.65
2.07
1.52
2.14
2.78
1.05
2.22
2.71
Robust6D
1.43
2.25
2.93
3,40
4.03
4.77
1.51
1.83
2.13
2.24
2.99
4,30
1.97
2.50
3.27
3DP3 (ours)
0.36
0.42
0.52
0.48
0.52
0.55
0.91
1.25
1.57
0.89
1.69
1.97
0.36
0.42
0.55
Partially Occluded
3DP3* (ours)
0.39
0.49
0.64
0.48
0.53
0.58
1.29
1.66
1.03
1.72
1.99
DenseFusion
0.41
0.48
0.58
0.81
1.11
1.53
1.47
2.01
3.18
0.38
0.48
0.64
Robust6D
1,30
1.48
1.61
1.15
1.46
1.87
1.48
2.02
3.24
0.94
1.10
1.33
Table 3: Robustness of inference. We quantify the ADD-S error stt, 2nd, and 3rd quartiles for each scene type and object type in the synthetic dataset of hard scenes. A value of - indicates the method made no prediction for the object's pose. 3DP3* denotes an ablated version of our method without inference of the scene graph structure and thus object-object contact.
8 Acknowledgements
The authors acknowledge Javier Felip Leon (Intel) for helpful discussions and a prototype depth renderer, and Omesh Tickoo (Intel) for helpful discussions. This work was funded in part by the DARPA Machine Common Sense program (Award ID: 030523-00001); by the Singapore DSTA / MIT SCC collaboration; by Intel's Probabilistic Computing Center, and by philanthropic gifts from the Aphorism Foundation and the Siegel Family Foundation. We thank Alex Lew, Tan Zhi-Xuan Feras Saad, Cameron Freer, McCoy Becker, Sam Witty, and George Matheos for helpful feedback.
10

---

References
[1] Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In Proceedings of the European Conference Computer Vision (ECCV), pages 29-42. Springer, 2010.
[2] Christophe Andrieu, Gareth Roberts, et al. The aeeudo-margina approach for efficient monte carlo computations. The Annals of Statistics, 37(2):6977225, 2009.
[3] Iro Armeni, Zhi- Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semanties, 3d space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5664-5673, 2019.
[4] Sid Yingze Bao and Silvio Savarese. Semantic structure from motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2025-2032. IEEE, 2011.
[5] Paul J Besl and Neil McKay. Method for registration of 3d shapes. In Sensor Fusion IV: Control Paradigms and Data Structures, volume 1611, pages 386-606. International Society for Optics and Photonics, 1992.
[6] Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set. IEEE Robotics Automation Magazine, 22(3):36-52, 2015.
[7] Xiaotong Chen, Rui Chen, Zhiqiang Sui, Zhefan Ye, Yanqi Liu, R Iris Bahar, and Odest Chad- wicke Jenkins. Grip: Generative robust inference and perception for semantic robot manipulation in adversarial environments. In 2019 IEEE/RSJ International Conference on Intelligent Robots IEEE, 2019.
and Systems (IROS), pages
[8] Yixin Chen, Siyuan Huang. Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holis- tic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with hujan-object interaction and physical commonsense. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8648-8657, 2019.
[9] James H. Clark. Hierarchical geometric models for visible surface algorithms. Commun. ACM, 19(10):5475554 October 1976.
[10] Marco Cusumano-Towwe Alexander K Lew, and Vikash K Mansinghka. Automating involutive meme using probabilistic and differentiable programming. arXiv preprint (rXtv:0007.09871 2020.
[11] Marco F Missumano-Towner Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: general-purpost probabilistic programming system with programmable inference. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 221-236, 2019.
[12] Fei Deng, Zhuo Zhi, and Sungjin Ahn. Generative hierarchical models for parts, objects, and scenes. arXiv preprint arXiv: 190..091109910010 19, 2019.
[13] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations (ICLR), 2021.
[14] Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. Poserbpf:
A rao-blackwellized particle filter for 6-d object pose tracking. IEEE Transactions Robotics, 2021.
[15] Karthik Desingh, Shiyang Lu, Anthony Opipari, and Odest Chadwicke Jenkins. Efficient nonparametrie belief propagation for pose estimation and manipulation of articulated objects. Science Robotics, 4(30), 2019.
[16] Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, Bill Freeman, Josh Tenenbaum, and Jiajun Wu. Learning to exploit stability for 3d scene parsing. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
11

---

[17] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29:3225-3211 2016.
[18] Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Greeill,,,,,lla,, Christopher Laan, Bhaskara Marthi, Xinghua Lou, Zhaoshi Meng. Yi Liu, Huayan Wang, Alex Lavin, and D. Scott Phoenix. A generative vision model that trains with high data efficiency and breaks text-based captchas. Science, (58(666))eeaag266 2017.
[19] Jared Glover, Gary Bradski, and Radu Bogdan Rusu. Monte carlo pose estimation with quaternion kernels and the bingham distribution. In Robotics: science and systems, volume page 97, 2012.
[20] Tomáš Hodañ, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jifí Matas, and Carsten Rother. BOP: Benchmark for 6D object pose estimation. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[21] Siyuam Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song- Chun Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In Proceedings of the European Conference on Computer Vision (ECCV), pages 187-203, 2018.
[22] D Knill D Kersten and A Yuille. Introduction: A bayesian formulation of visual perception. Perception as Bayesian inference, pages 1-21, 1996.
[23] Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception bayesian inference. Annu. Rev. Psychol., 1-304, 2004.
[24] Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, and Abhinav Gupta. 3d-reinet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2212-2221, 2019.
[25] Tejas D Kulkarni, Pushmeet Kohli, Joshua Tenenbaum, and Vikash Mansinghka. Picture: probabilistic programming language for scene perception. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4390-4399, 2015.
[26]
Tai Sing Lee and David Mumford. Hierarchical bayesian inference in the visual cortex. JOSA
A, 20(7):1434-1444 1448, 2003.
[27] Vikash K Mansinghka. Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. Approximate bayesian image interpretation using generative probabilistic graphics programs. Advances in Neural Information Processing Systems, 26:1520-1528, 2013.
[28) Vikash K Mansinghka, UIrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. Probabilistic programming with programmable inference. Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. pages 603-616, 2018.
[29] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 405-421. Springer, 2020.
[30] Pol Moreno, Christopher KI Williams, Charlie Nash, and Pushmeet Kohli. Overcoming occlusion with inverse graphics. In Proceedings of the European Conference on Computer Vision (ECCV), pages 170-185. Springer, 2016.
[31] Lukasz Romaszko, Christopher KI Williams, Pol Moreno, and Pushmeet Kohli. Vision-as- graarse-rraphic: Obtaining a rich explanation of a scene from single image. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 851 851-859, 2017.
12

---

[32] Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. arXiv preprint arXiv:2002.06289, 2020.
[33] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-tte revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[34] Meng Tian, Liang Pan, Marcelo H Ang Jr, and Gim Hee Lee. Robust 6d object pose estimation by learning rgb-d features. In International Conference on Robotics and Automation (ICRA), 2020.
[35] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In Conference on Robot Learning (CoRL), 2018
[36] Zhuowen Tu and Song-Chun Zhu. Image segmentation by data-driven markov chain monte carlo. IEEE Transactions pattern analysis and machine intelligence, 24(5):657-673, 2002.
[37] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martin, Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3343 3343-3352, 2019.
[38] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. 2018.
[39] Alan Yuille and Daniel Kersten. Vision as bayesian inference: Analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006.
[40] Song Chun Zhu and Alan Yuille. Region competition: Unifying snakes, region growing, and bayes/mdl for multiband image segmentation. IEEE transactions on pattern analysis and machine intelligence, 18(9):884-900) 1996.
[41] Ben Zinberg, Marco Gusumano-Towne and Vikash K Mansinghka. Structured differentiable models of 3d scenes via generative scene graphs. In Workshop on Perception as Generative Reasoning, NeurIPS, Submitted September 2019 and presented December 2019.
13


Reviews:
[{"review_id": "jRNpuhOQMMP", "reviewer": "Reviewer_NB9L", "summary": "This paper proposes a probabilistic model for reconstructing a scene of multiple 3D objects. The key advantage of the approach is that it is able to reason about the shapes of objects, the relations between them (via the scene graph) and their contacts. This ability to use physical constraints for scene parsing is a major limitation of deep learning systems. The results show that the proposed method is significantly more accurate than two recent approaches that are based on deep models (i.e. DenseFusion and Robust6D).  The main limitation of the model seems to be the inference speed, which is currently not realtime.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**TL;DR:** An intriguing, principled, probabilistic approach for object shape and pose estimation that is able to utilize object relations and physical constraints. In the main review below I have labelled each point as positive (+), negative (-) or mixed (+/-).\n \n**Originality**: \n\n(+) As far as I can tell the method is new in the sense that this is the first probabilistic, generative model that takes into account both object shapes, physical constraints such as their spatial relations and contact between individual objects. This is interesting as physical reasoning can help to overcome the limitations of traditional object reconstruction methods that only take into account multi-view geometry and simple shape priors. The results presented in the paper confirm this as the proposed system is significantly more accurate than the two approaches against which it is compared. \n\n(+) Although this type of analysis-by-synthesis approach based on MCMC has been very popular and well explored in computer vision in the past, the probabilistic model presented in the paper is novel. Furthermore, the meticulously crafted MCMC kernels used for inference are also novel and seem to be effective.  \n\n(+/-) In terms of related work, the authors do a good job at describing which contributions are novel. I think the related work Section 2 on “Probabilistic programs for vision” is particularly clear and gives a concise description of what is novel compared to general probabilistic programming works that have addressed computer vision problems. However, I would probably like to have seen a discussion about more closely related work such as [A,B] that are probabilistic approaches for addressing the same problem described in this paper (i.e. joint scene parsing and object reconstruction). At the moment the related work doesn’t cover such approaches.\n\n[A] Du, Yilun, et al. \"Learning to Exploit Stability for 3D Scene Parsing.\" NeurIPS. 2018.\n\n[B] Bao, Sid Yingze, and Silvio Savarese. \"Semantic structure from motion: A novel framework for joint object recognition and 3D reconstruction.\" Outdoor and Large-Scale Real-World Scene Analysis. Springer, Berlin, Heidelberg, 2012. 376-397.\n\n**Quality**: \n\n(+) Although I have not checked every detail of the proofs in the submission (particularly the ones in the supplementary), the general approach appears to be technically sound as far as I can tell.  \n\n(+) The claims in the paper are well supported. Particularly the results show that the proposed method is able to learn new objects from a very small number of views i.e. 5 and using this model is able to obtain state of the art pose estimation results on the YCB dataset. \n\n(-) There is a caveat with these results though, in that the method appears to be initialized with a state-of-the-art deep learning model, which makes the results slightly less impressive.  However this still shows that the proposed approach is able to refine the poses to a level beyond that which can be obtained with current methods. I would suggest that the authors make it clear in the paper where the method has been initialized using an existing approach and where it hasn’t because this isn’t very clear at the moment. Also, what is the performance of the system when no initialization is used?\n\n(-) Overall I would say that the authors could do a better job of describing the weaknesses of the approach. The most obvious weakness that comes to mind is computational efficiency. There should probably be some form of inference time comparison in the paper to show how the proposed approach compares to the existing methods (especially deep learning based ones) which are known to be very fast. Another weakness is the complexity of the approach - is it not possible to simplify the components such as the transition kernels or the scene graph structure and obtain similar results?\n    \n**Clarity**: \n\n(-) The paper is very dense and not always that easy to follow.  This is partly due to the nature of the approach - the model contains many quantities and distributions that need to be defined, but it might be useful for the reader if there was some summary or overview that describes all the components of the method in one place.\n\n(+/-) Overall, the paper is well organized and the individual sections are quite verbose and not that easy to follow. I don’t have a great suggestion to fix this, but maybe separating the definitions from the actual description might help. \n\n(+) As far as I can tell, the paper provides enough information for an expert to reimplement the method (with the help of a probabilistic programming language, like Gen) and reproduce results. The authors have also supplied the source code which helps with reproducibility.\n\n**Significance**: \n\n(+) In my view, the results presented in the paper are significant. This is one of the best examples of how probabilistic programming can be applied to solve challenging vision tasks.  It is quite surprising how data-efficient and accurate the approach is compared to deep learning systems that need many training samples to achieve similar accuracy for object pose estimation. \nI think the great results obtained in the paper could help to encourage more computer vision researchers to explore probabilistic programming approaches. There are also quite a few areas in which the proposed method could be improved (such as inference time etc.) which might inspire other researchers to try to improve upon the method.\n\n(+) In summary, the method addresses a difficult task of jointly estimating object shape and pose, and also  advances the state of the art in terms of accuracy.\n\n# Post author response and discussion\nAfter considering the other reviews, and the authors' responses, the main concerns raised in my review have been addressed. Specifically, the issues related to how the initialization is done and a more thorough discussion about the weaknesses of the approach have been addressed. In terms of accessibility / reproducibility the authors have committed to open-sourcing their approach. Overall, I think this is a good paper which opens up lots of possibilities for future research at the intersection of probabilistic modelling and computer vision. Therefore, my final rating is \"7: Good paper, accept\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper could do with a better description of the limitations of the approach (as mentioned above), and there is no mention of the potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "X3K3J8Dwp4K", "reviewer": "Reviewer_cyy4", "summary": "The paper presents an approach for inferring the 3D pose and shape of multi-object tabletop scenes from a set of multi-view depth frames as input.  The approach is based on a generative model incorporating 3D scene graph generation, as well as per-object category voxelized shape priors.  At test time, the inference algorithm parses the input to a 3D scene graph using MCMC-based updates to the scene graph structure, constrained by physical contact relations between the objects and the object shape priors.\n\nThe approach is evaluated on the 6DoF 3D pose estimation task using two datasets: a set of real scenes from the YCB-video dataset, and a set of synthetically generated scenes using reconstructed YCB objects (YCB-Challenging).  The experiments compare the proposed approach against baselines from prior work (specifically, DenseFusion and Robust6D -- both trained in a supervised fashion on larger RGB-D training datasets).  An ablation of the approach without scene graph structure inference is also used as a comparison point.  The experiments show that the approach mostly outperforms the baselines and ablation in terms of accuracy of the predicted 6DoF pose for the objects in the scene.  An additional analysis incorporates one of the baselines (DenseFusion) to initialize the estimated pose used by the proposed method. Improvements to the estimated pose are observed in some challenging scenarios from the YCB-Video test scenes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nOn the plus side, the proposed approach is quite comprehensive and formalized in a way that admits an impressive level of generality.  As such, I can see the approach being applicable to domains and datasets beyond what has been presented in the paper.  I believe that the community would benefit from more work in this vein, leveraging probabilistic programming and generative models for 3D scene understanding.  On the other hand, there is a highly relevant line of prior work that is not cited or discussed: \"Attend, Infer, Repeat:\nFast Scene Understanding with Generative Models\" by Eslami et al. NIPS 2016 have proposed a similar probabilistic inference framework and demonstrated that it can be used to parse 3D scenes (i.e. infer the object that are present, and their 6DoF poses).  A detailed discussion of how the proposed approach relates to this prior work, and potentially comparison against it in the evaluation is in my opinion necessary.\n\nQuality:\nThe paper is technically sound, as far as I can tell and the claims are mostly supported by the empirical evidence.  That being said, there is no evaluation of the object shape priors either independently, or as part of the proposed approach.  In other words, the impact of Section 4 on the results is not clear in the current paper.\n\nClarity:\nOverall, the paper is very well written and organized coherently.  There were a few minor typos etc. that could be resolved in a proofreading pass.  A couple of examples that I detected: L98 \"corresponds to rigid object with\" should be \"corresponds to a rigid object with\"; and L201 \"a uninformed\" should be \"an uninformed\".\n\nSignificance:\nThe paper has an impressive scope in terms of the presented formalism and its generality.  However, the empirical results only partially back up the complexity of the proposed approach, and some discussion of relation to prior work is missing (see above points).  Therefore, I am not as positive as I could be with respect to acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are discussed at a high level in the last section.  The speed of inference, and integrating object prior learning with scene graph parsing were identified as avenues for future work.  Potential negative societal impacts were not discussed (the checklist notes that the authors do not believe there are any at this stage of the work).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "U-JeROuOJcm", "reviewer": "Reviewer_6dXH", "summary": "This paper presents a method for multi-object 6DoF pose estimation with a \"probabilistic programming\" model. Before inference, the model learns priors on 3D occupancy/shape for objects. Then, given the number of objects in the scene and the classes for objects in a test scene (observed in a depth map), inference proceeds by MCMC sampling scene graphs and poses and occupancies, to minimize a depth reconstruction error. The results look good on YCB and on a new synthetic dataset based on YCB objects, against two recent 6DoF pose estimators, showing good results. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper was very hard for me to understand, and there still may be parts I have misunderstood. I will set up my review as a list of confusion points, which might all be easily answered by the authors.\n\n- What is the purpose of assigning the names top/bottom/left/right/front/back to the planes of the box? It doesn't seem like this information is ever used. \n\n- How exactly do you model contact relationships, and learn priors about them? I found many variables that relate to contacts in the \"Scene graphs\" paragraph (89-109) but they never come up again, either in the generative model or the inference algorithm. In general there is a lot of notation for the reader to keep track of, and it seemed like much of it was declared and then forgotten, so perhaps it can just be deleted. (But I do very much want to learn about those contact relationships.) \n \n- The voxel grids have \"fixed dimensions h, w, l ∈ N\", but they also have a \"grid resolution s ∈ R\". What is the difference between a dimension and a resolution?\n\n- The text says that the scene graph prior (equation 1) \"favors scenes that can be parsimoniously described via either fewer objects, or fewer objects with independently-sampled 6DoF poses\". The idea of \"fewer objects\" occurs in both sides of the \"either\", so it seems that part should always be true, but actually the number of objects is held constant by an assumption stated earlier: \"an a-priori known number of objects N0 (used in the experiments in Section 5)\". So the parsimonious/fewer part seems inactive here, and we are just left with \"independently-sampled 6DoF poses\". It seems like little-to-no information or inductive bias has actually been introduced here. Am I reading that right?\n\n- Is the pre-trained object occupancy prior conditioned on class? \n\n- The paper says the model performs well \"despite being trained on only a few synthetic depth images\". How many is a few? (Is it the \"5 depth images of each object type\" mentioned later? Why does the abstract say \"5 or fewer\"?) Also, why was it trained on only a few? Since they are synthetic, I expect them to be plentiful. \n\n- Do the baselines also assume known class and number of objects? They seem to involve detection and segmentation as well as pose estimation, so maybe they are a harder problem, and it's not an apples-to-apples comparison. \n\n- I do not quite follow Table 1, and how to compare it with prior work. The Robust6D and DenseFusion papers report ADD-S accuracy across far more classes (~21 instead of 5), their values are in the range 90-100, and they also report the mean across all classes. Table 1 reports \"ADD-S error\" at three quartiles, and the values are under 2, and it looks like the mean across objects is never calculated. It would be good to do the same kind of evaluation as the previous work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns. \n", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for multi-object 6DoF pose estimation with a "probabilistic programming" model. Before inference, the model learns priors on 3D occupancy/shape for objects. Then, given the number of objects in the scene and the classes for objects in a test scene (observed in a depth map), inference proceeds by MCMC sampling scene graphs and poses and occupancies, to minimize a depth reconstruction error. 
The reviewers raised concerns regarding fairness of comparisons to baselines, known versus unknown number of objects at inference time, violation of priors, (lack of) comparisons to other object-centric generative models. The rebuttal of the authors addressed those concerns and generated extensive discussion. The paper is suggested for publication. 

Decision: Accept (Poster)