3DP3: 3D Scene Perception via Probabilistic Programming

1 Introduction . A striking feature of human visual intelligence is our ability to learn representations of novel objects from a limited amount of data and then robustly percieve 3D scenes containing those objects . We can immediately generalize across large variations in viewpoint , occlusion , lighting , and clutter . How might we develop computational vision systems that can do the same ? This paper presents a generative model for 3D scene perception , called 3DP3 . Object shapes are learned via probabilistic inference in a voxel occupancy model that coarsely captures 3D shape and uncertainty due to self-occlusion ( Section 4 ) . Scenes are modeled via hierarchical 3D scene graphs that can explain planar contacts between objects without forcing scenes to fit rigid structual assumptions ( Section 3 ) . Images are modeled by real-time graphics and robust likelihoods on point clouds . We cast 3D scene understanding as approximate probabilistic inference in this generative model . We develop a novel inference algorithm that combines data-driven Metropolis-Hastings kernels over object poses , involutive MCMC kernels over scene graph structure , pseudo-marginal integration over uncertain object shape , and existing deep learning object detectors and pose estimators ( Section 5 ) . This architecture leverages inference in the generative model to provide common sense constraints that fix errors made by bottom-up neural detectors . Our experiments show that 3DP3 is more accurate and robust than deep learning baselines at 6DoF pose estimation for challenging synthetic and real-world scenes ( Section 6 ) . Our model and inference algorithm are implemented in the Gen [ 11 ] probabilistic programming system . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2 Related Work . Analysis-by-synthesis approaches to computer vision A long line of work has interpreted computer vision as the inverse problem to computer graphics [ 22 , 39 , 26 , 23 ] . This ‘ analysis-by-synthesis ’ approach has been used for various tasks including character recognition , CAPTCHA-breaking , lane detection , object pose estimation , and human pose estimation [ 40 , 36 , 27 , 30 , 18 , 31 ] . To our knowledge , our work is the first to use an analysis-by-synthesis approach to infer a hierarchical 3D object-based representation of real multi-object scenes while exploiting inductive biases about the contacts between objects . Hierarchical latent 3D scene representations We use a scene graph representation [ 41 ] that is closely related to hierarchical scene graph representations in computer graphics [ 9 ] . Unlike in graphics , we address the inverse problem of inferring hierarchical scene graphs from observed image data . Inferring hierarchical 3D scene graphs from RGB or depth images in a probabilistic framework is relatively unexplored . One concurrent1 and independent work , Generative Scene Graph Networks ( GSGN [ 13 ] ) , proposes a variational autoencoder architecture for decomposing images into objects and parts using a tree-structured latent scene graph that is similar to our scene graph representation . However , GSGN learns RGB appearance models of objects and their parts , uses an inference network instead of a hybrid of data-driven and model-based inference , was not evaluated on real images or scenes , and uses more restricted scene graphs that can not represent objects with independent 6DoF pose . GSGN builds on an earlier deep generative model [ 17 ] that generates multi-object scenes but does not model dependencies between object poses and was not quantitatively evaluated on real 3D scenes . Incorporating a learned inference network for jointly proposing scene graphs into our framework is an interesting area for future work . The term ‘ scene graph ’ has also been used in computer vision to refer to various less related graph representations of scenes [ 3 , 8 , 32 ] . Probabilistic programming for computer vision Prior work has used probabilistic programs to represent generative models of images and implemented inference in these models using probabilistic programming systems [ 27 , 25 ] . Unlike these prior works , which relied on manually specified and/or semi-parametric shape models , 3DP3 learns object shapes non-parametrically . 3DP3 also models occlusion of one 3D object by another ; uses a novel hierarchical scene graph prior that allows for dependencies between object poses in the prior ; uses a novel involutive MCMC [ 10 ] kernel for inferring scene graph structure ; and uses a novel pseudo-marginal approach for handling uncertainty about object shape during inference . We also present a proof of concept that our system can infer the presence and pose of fully occluded objects . 6DoF object pose estimation We use 6DoF estimation of object pose from RGBD images as an example application . Registration of point clouds [ 5 ] can be used to estimate the 6DoF pose of objects with known 3D geometry from depth images . Many recent 6DoF object pose estimators use deep learning [ 38 , 35 ] and many also take depth images [ 37 , 34 ] . Some pose estimation methods model scene structure , contact relationships , stability , or other semantic information [ 21 , 8 , 24 , 16 , 4 ] , and some use probabilistic inference [ 15 , 7 , 19 , 14 ] . To our knowledge , we present the first 6DoF pose estimator that uses Bayesian inference about the structure of hierarchical 3D scene graphs . Learning models of novel 3D objects Classic algorithms for structure-from-motion infer a 3D model of a scene from multiple images [ 33 , 1 ] . Our approach for learning the shape of novel 3D objects produces coarse-grained probabilistic voxel models of objects that can represent uncertainty about the occupancy of self-occluded volumes . Integrating other representations of object shape and object appearance [ 29 ] with our scene graph representation is a promising area of future work . 3 3DP3 generative modeling framework . The core of 3DP3 is a generative modeling framework that represents a scene in terms of discrete objects , the 3D shape of each object , and a hierarchical structure called a scene graph that relates the poses ( position and orientation ) of the objects . This section describes 3DP3 ’ s object shape and scene graph latent representations , a family of prior distributions on these latent representations , and an observation model for image-based observations of scenes . Figure 1 shows the combined generative model written as a probabilistic program . 1An early version of our work [ 41 ] is concurrent with an early version of GSGN [ 12 ] box Input RGB Image Input Depth Image Inferred Poses Inferred Scene Graph Depth Reconstruction Inference ( a ) Inferring a hierarchical 3D scene graph from an RGB-D image with 3DP3 . Our model knows that objects often lay flat on other objects , which allows for the depth pixels of one object to inform the pose of other objects . Our algorithm also infers when this knowledge is relevant ( e.g . the clamp on the left , represented by the purple node , is laying flat on the box ) , and when it is not ( e.g . the clamp on the right , represented by the red node , is not laying flat on any other object ) . ( b ) 3DP3 uses a structured generative model of 3D scenes , represented as a probabilistic program . The model uses a prior over object shapes that can be learned from data , a prior over scene structure that is a probability distribution on graphs , a traversal of the scene graph starting at the world node r to compute object poses , and a robust likelihood model for depth images . In the graph at right , the world node r ( not shown ) is the parent of the grey node ( box ) and the red node ( right clamp ) because those objects are not layting flat on other objects . Figure 1 : ( a ) A scene graph inference task and ( b ) the 3DP3 generative model . 3.1 Objects . The most basic element of our generative modeling framework are rigid objects . The first stage in our generative model prior encodes uncertainty about the 3D shape of M types of rigid objects that may or may not be encountered in any given scene . Voxel 3D object shapes We model the coarse 3D shape of rigid objects using a voxel grid with dimensions h , w , l ∈ N and cells indexed by ( i , j , ` ) ∈ [ h ] × [ w ] × [ l ] . Each cell has dimension s×s×s for resolution s ∈ R+ , so that the entire voxel grid represents the cuboid [ 0 , h ·s ] × [ 0 , w ·s ] × [ 0 , l ·s ] . All objects are assumed to fit within the cuboid . An object ’ s shape is defined by a binary assignment O ∈ { 0 , 1 } h×w×l of occupancy states to each cell in the voxel grid , where Oij ` = 1 indicates that cell ( i , j , ` ) is occupied and Oij ` = 0 indicates it is free . Each object also has a finite set of contact planes through which the object may be in flush contact with the contact planes of other objects in physically stable scenes . For example , in Figure 2 , the table has a contact plane for its top surface , the yellow sugar box has six contact planes , one for each of its six faces , and the bottom contact plane of the sugar box is in flush contact with the top contact plane of the table . The pose of a contact plane relative to its object is a function of the object shape O . To simplify notation , we denote the set of contact planes for any object by F . Prior distributions on 3D object shape We assume there are M distinct object types , and each object type m ∈ { 1 , . . . , M } has an a-priori unknown shape , denoted O ( m ) . Let O ( 1 : M ) : = ( O ( 1 ) , . . . , O ( M ) ) . The prior distribution on the shape of each object type m is denoted p ( O ( m ) ) . Although our inference algorithm ( Section 5 ) only requires the ability to sample jointly from p ( O ( 1 : M ) ) , we assume shapes of object types are independent in the prior ( p ( O ( 1 : M ) ) = ∏M m=1 p ( O ( m ) ) ) . Section 4 shows how to learn a specific shape prior p ( O ( m ) ) for an object type from depth images . 3.2 Scenes . Given a collection of M known object types and their shapes , our model generates scenes with N objects by randomly selecting an object type for each object and then sampling a 6DoF object pose for each object . Instead of assuming that object poses are independent , our model encodes an inductive bias about the regularities in real-world scenes : objects are often resting in flush contact with other objects ( e.g . see Figure 2 ) . We jointly sample dependent object poses using a flexible hierarchical scene graph , while maintaining uncertainty over the structure of the graph . Hierarchical scene graphs We model the geometric state of a scene as a scene graph G ( Figure 2 ) , which is a tuple G = ( G , θ ) where G = ( V , E ) is a directed rooted tree and θ are parameters . The vertices V : = { r , v1 , . . . , vN } represent N + 1 3D coordinate frames , with r representing the world coordinate frame . An edge ( u , v ) ∈ E indicates that coordinate frame v is parametrized relative to frame u , with parameters θv . The 6DoF pose of frame v relative to frame u with pose xu ∈ SE ( 3 ) is given by a function ∆xv , where ∆xv ( θv ) ∈ SE ( 3 ) and xv : = xu ·∆xv ( θv ) . Here , · is the SE ( 3 ) group operation , and the world coordinate frame is defined as the identity element ( xr : = I ) . Modeling flush contact between rigid objects While the vertices of scene graphs G can represent arbitrary coordinate frames in a scene ( e.g . the coordinate frames of articulated joints , object poses ) , in the remainder of this paper we assume that each vertex v ∈ V \ { r } corresponds to the pose of a rigid object . We index objects by 1 , . . . , N , with corresponding vertices v1 , . . . , vN . We assume that each object i has an object type ci ∈ { 1 , . . . , M } . For vertices v that are children of the root vertex r , θv ∈ SE ( 3 ) defines the absolute 6DoF pose of the corresponding object ( ∆xv ( θv ) = θv ) . For vertices v that are children of a non-root vertex u , the parameters take the form θv = ( fv , f ′v , av , bv , zv , φv ) and represent a contact relationship between the two objects : fv and f ′v indicate which contact planes of the parent and child objects , respectively , are in contact . ( av , bv ) ∈ R2 is the in-plane offset of the origin of plane fv of object v from the origin of plane f ′v of object u. zv ∈ R is the perpendicular distance of the origin of plane fv of object v from plane f ′v of object u. φv ∈ S2 × S1 represents the deviation of the normal vectors of the two contact planes from anti-parallel ( in S2 ) and a relative in-plane rotation of the two contact planes ( in S1 ) . The relative pose ∆xv ( θv ) of v with respect to u is the composition ( in SE ( 3 ) ) of three relative poses : ( i ) v with respect to its plane fv , ( ii ) v ’ s plane fv with respect to u ’ s plane f ′v , and ( iii ) u ’ s plane f ′ v with respect to u . The 6DoF poses of all objects ( xv for v ∈ V \ { r } ) are computed by traversing the scene graph while taking products of relative poses along paths from the root r. Prior distributions on scene graphs We now describe our prior on scene graphs , given object models O ( 1 : M ) . We assume the number of objects N in the scene is known ( see the supplement for a generalization to unknown N ) . We first sample the types ci ∈ { 1 , . . . , M } of all objects from an exchangeable distribution p ( c ) where c : = ( c1 , . . . , cN ) . This includes as a special case distributions where all types are represented at most once among the objects ( ∑N i=1 1 [ ci = c ] ≤ 1 ) , which is the case in our experiments . Next , we sample the scene graph structure G from p ( G ) . We experiment with two priors p ( G ) : ( i ) a uniform distribution on the set of ( N + 1 ) N−1 directed trees that are rooted at a vertex r , and ( ii ) δG0 ( G ) , where G0 is a graph on N + 1 vertices where ( r , v ) ∈ E for all v ∈ V \ { r } so that each object vertex has an independent 6DoF pose . For objects whose parent is r ( the world coordinate frame ) , we sample the pose θv ∼ punif , which samples the translation component uniformly from a cuboid scene extent , and the orientation uniformly over SO ( 3 ) . For objects whose parent is another object u , we sample the choice of contact planes ( fv , f ′v ) ∈ F × F uniformly , ( av , bv ) ∼ Uniform ( [ −50cm , 50cm ] 2 ) , zv ∼ N ( 0 , 1cm ) , the S2 component of φv from a von Mises–Fisher ( vMF ) distribution concentrated ( κ = 250 ) on anti-parallel plane normals , and the S1 component from Uniform ( S1 ) . We denote this distribution pcont ( θv ) . Note that the parameters of pcont were not tuned or tailored in any detailed way—they were chosen heuristically based on the rough dimensions of table-top objects . The resulting prior over all of the latent variables is : p ( O ( 1 : M ) , c , G , θ ) = ( M∏ m=1 p ( O ( m ) ) ) 1 ( N + 1 ) N−1 p ( c ) ∏ v∈V : ( r , v ) ∈E punif ( θv ) ∏ ( u , v ) ∈E : u 6=r pcont ( θv ) ( 1 )

Summary: This paper proposes a probabilistic model for reconstructing a scene of multiple 3D objects. The key advantage of the approach is that it is able to reason about the shapes of objects, the relations between them (via the scene graph) and their contacts. This ability to use physical constraints for scene parsing is a major limitation of deep learning systems. The results show that the proposed method is significantly more accurate than two recent approaches that are based on deep models (i.e. DenseFusion and Robust6D).  The main limitation of the model seems to be the inference speed, which is currently not realtime.