Generative Scene Graph Networks

1 INTRODUCTION . Learning to discover and represent objects purely from observations is at the core of human cognition ( Spelke & Kinzler , 2007 ) . Recent advances in unsupervised object-centric representation learning have enabled decomposition of scenes into objects ( Greff et al. , 2019 ; Lin et al. , 2020b ; Locatello et al. , 2020 ) , inference and rendering of 3D object models ( Chen et al. , 2020 ) , and object tracking and future generation ( Crawford & Pineau , 2019a ; Jiang et al. , 2020 ; Lin et al. , 2020a ) . These neuro-symbolic approaches , where the discreteness of discovered objects provides the symbolic representation , facilitate various desired abilities such as out-of-distribution generalization , relational reasoning , and causal inference . In this paper , we seek to further discover and represent the structure within objects without supervision . Our motivation is that natural scenes frequently contain compositional objects—objects that are composed of primitive parts . We humans can easily identify the primitive parts and recognize the part-whole relationship . Representing objects as explicit composition of parts is expected to be more efficient , since a vast number of complex objects can often be compositionally explained by a small set of simple primitives . It also allows us to imagine and create meaningful new objects . A well-established representation for part-whole relationships in computer graphics is called the scene graph ( Foley et al. , 1996 ) . It is a tree whose leaf nodes store models of primitive parts , and whose edges specify affine transformations that compose parts into objects . While in computer graphics , the scene graph is manually constructed for rendering , in this paper , we are interested in inferring the scene graph from unlabeled images . To this end , we propose Generative Scene Graph Networks ( GSGNs ) . We formulate this model as a variational autoencoder ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) whose latent representation is a probabilistic scene graph . In the latent tree , each node is associated with an appearance variable that summarizes the composition up to the current level , and each edge with a pose variable that parameterizes the affine transformation from the current level to the upper level . The design of the GSGN decoder follows the rendering process of graphics engines , but with differentiable operations helping the encoder to learn inverse graphics ( Tieleman , 2014 ; Wu et al. , 2017 ; Romaszko et al. , 2017 ; Yao et al. , 2018 ; Deng et al. , 2019 ) . As a result , the pose variables inferred by GSGN are interpretable , and the probabilistic scene graph supports symbolic manipulation by configuring the pose variables . One major challenge is to infer the structure of the scene graph . This involves identifying the parts and grouping them into objects . Notice that unlike objects , parts are often stitched together and thus can have severe occlusion , making it hard to separate them . Existing methods for learning hierarchical scene representations circumvent this challenge by working on single-object scenes ( Kosiorek et al. , 2019 ) and also providing predefined or pre-segmented parts ( Li et al. , 2017 ; Huang et al. , 2020 ) . In contrast , GSGN addresses this challenge directly , and learns to infer the scene graph structure from multi-object scenes without knowledge of individual parts . Our key observation is that the scene graph has a recursive structure—inferring the structure of the tree should be similar to inferring the structure of its subtrees . Hence , we develop a top-down inference process that first decomposes the scene into objects and then further decomposes each object into its parts . This allows us to reuse existing scene decomposition methods such as SPACE ( Lin et al. , 2020b ) as an inference module shared at each level of the scene graph . However , we find that SPACE has difficulty separating parts that have severe occlusion , possibly due to its complete reliance on bottom-up image features . Therefore , simply applying SPACE for decomposition at each level will lead to suboptimal scene graphs . To alleviate this , GSGN learns a prior over plausible scene graphs that captures typical compositions . During inference , this prior provides top-down information which is combined with bottom-up image features to help reduce ambiguity caused by occlusion . For evaluation , we develop two datasets of scenes containing multiple compositional 2D and 3D objects , respectively . These can be regarded as compositional versions of Multi-dSprites ( Greff et al. , 2019 ) and CLEVR ( Johnson et al. , 2017 ) , two commonly used datasets for evaluating unsupervised object-level scene decomposition . For example , the compositional 3D objects in our dataset are made up of shapes similar to those in the CLEVR dataset , with variable sizes , colors , and materials . Hence , we name our 3D dataset the Compositional CLEVR dataset . The contributions of this paper are : ( i ) we propose the probabilistic scene graph representation that enables unsupervised and end-to-end scene graph inference and compositional scene generation , ( ii ) we develop and release the Compositional CLEVR dataset to facilitate future research on object compositionality , and ( iii ) we demonstrate that our model is able to infer the latent scene graph , shows decent generation quality and generalization ability , and improves data efficiency in downstream tasks . 2 RELATED WORK . Object-centric representations . Our model builds upon a line of recent work on unsupervised object-centric representation learning , which aims to eliminate the need for supervision in structured scene understanding . These methods learn a holistic model capable of decomposing scenes into objects , learning appearance representations for each object , and generating novel scenes—all without supervision and in an end-to-end trainable way . We believe such unsupervised and holistic models are more desirable , albeit more challenging to learn . These models can be categorized into scene-mixture models ( Greff et al. , 2017 ; 2019 ; Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Locatello et al. , 2020 ) and spatial-attention models ( Eslami et al. , 2016 ; Crawford & Pineau , 2019b ; Lin et al. , 2020b ; Jiang & Ahn , 2020 ) . Compared to these models , we go a step further by also decomposing objects into parts . We use spatial-attention models as the inference module at each level of the scene graph , because they explicitly provide object positions , unlike scene-mixture models . We combine the inference module with a learned prior to help improve robustness to occlusion . This also allows sampling novel scenes from the prior , which is not possible with spatial-attention models . Hierarchical scene representations . Modeling the part-whole relationship in scenes has attracted growing interest , and has been utilized for improving image classification ( Sabour et al. , 2017 ; Hinton et al. , 2018 ; Kosiorek et al. , 2019 ) , parsing , and segmentation ( Zhu et al. , 2008 ) . However , these models have been applied to scenes with one dominant object only , and can not perform scene generation . Recent work on assembly-based 3D shape modeling also learns the part-whole relationship ( Tulsiani et al. , 2017 ; Li et al. , 2017 ; Zhu et al. , 2018 ; Huang et al. , 2020 ; Kania et al. , 2020 ) , but these methods require predefined or pre-segmented parts as input , and can only model single shapes with no background . By contrast , our model learns the part-whole relationship from multi-object scenes without knowledge of individual parts . There has also been work on 3D part decomposition ( Chen et al. , 2019 ; Deng et al. , 2020 ) , but they require voxels or point clouds as input , and typically focus on geometry ( e.g. , part occupancy ) without learning to represent appearance ( e.g. , color , material ) . Part hierarchies have also been used for shape generation ( Mo et al. , 2019 ) , where the hierarchy is provided as input rather than inferred from the input . Our approach infers compositional structures from static scenes , and is orthogonal to methods that use motion cues for decomposing dynamic scenes ( Xu et al. , 2019 ) and methods that infer physical interactions from dynamic scenes ( Li et al. , 2020 ; Stanić et al. , 2020 ) . Hinton ( 2021 ) recently proposed an iterative procedure that is expected to form the part hierarchy through multiple rounds of message passing among adjacent levels . While our model works without iterative message passing , we believe this is important for parsing more complex scenes . Hierarchical latent variable models . Our model can be regarded as a hierarchical latent variable model , and is inspired by several recent advances ( Bachman , 2016 ; Sønderby et al. , 2016 ; Zhao et al. , 2017 ; Maaløe et al. , 2019 ) that have achieved impressive generation quality . While these methods focus on designing the hierarchical structure and training method that harness the full expressive power of generative models , our goal is to learn the hierarchical structure from unlabeled images that captures the compositional relationship among symbolic entities like objects and parts . 3 GENERATIVE SCENE GRAPH NETWORKS . 3.1 GENERATIVE PROCESS . We assume that the image x is generated by a set of foreground variables collectively denoted zfg and a background variable zbg as follows : p ( x ) = ∫∫ p ( x |zfg , zbg ) p ( zbg |zfg ) p ( zfg ) dzfg dzbg . ( 1 ) To represent the compositional structures within foreground objects , GSGN models zfg as a treestructured probabilistic scene graph , as shown in Figure 1A . Each leaf node represents a primitive entity that is not further decomposed . Each internal node represents an abstract entity that is composed from its children nodes . Similar to graphics engines , the composition is modeled as affine transformations , and is specified by the relative pose ( including rotation , scaling , and translation ) of each child node v with respect to its parent pa ( v ) . We use a pose variable zposev to represent the relative pose , and associate it with the corresponding edge . We also associate an appearance variable zapprv with each node v. It is expected to represent the appearance of entity v in its canonical pose 1 , summarizing all lower-level composition in the subtree rooted at v. In particular , the appearance variable zapprr at the root node r summarizes the full scene . Due to this summarization assumption , given zapprv , we can generate the pose and appearance variables for all children nodes of v in a conditionally independent way . Hence , for a given tree structure with V being the set of all nodes , the prior over foreground variables can be factorized according to the tree structure : p ( zfg ) = p ( z appr r ) ∏ v∈V \ { r } p ( z pose v |z appr pa ( v ) ) p ( z appr v |z appr pa ( v ) ) . ( 2 ) Here we further assume conditional independence between zposev and z appr v , since in graphics engines , one should be able to specify the pose and appearance separately . Representing tree structures . The above factorization only works for a given tree structure . To deal with variable tree structures , we need to include them in the latent representation as well . We start by setting a maximum out-degree for each node so that the total number of possible structures is bounded . To determine the structure , it then suffices to specify the presence of each possible edge . Hence , for an arbitrary edge between node v and its parent , we introduce a Bernoulli variable zpresv to indicate its presence . If zpresv = 0 , meaning the edge is not present , then the pose variable associated with the edge along with all variables in the subtree rooted at v are excluded from the probabilistic scene graph . More precisely , let us define z̄presv to be the product of all the presence variables along the path from root r to node v : z̄presr = 1 , z̄ pres v = z pres v × z̄ pres pa ( v ) for v ∈ V \ { r } . ( 3 ) The foreground variables now become zfg = { zapprr } ∪ { zpresv , zposev , zapprv } v∈V \ { r } , and the prior factorizes as follows : p ( zfg ) = p ( z appr r ) ∏ v∈V \ { r } [ p ( z pres v |z appr pa ( v ) ) ] z̄pres pa ( v ) [ p ( zposev |z appr pa ( v ) ) p ( z appr v |z appr pa ( v ) ) ] z̄presv . ( 4 ) We implement the pose and appearance variables as Gaussian variables , p ( zapprr ) = N ( 0,1 ) , and the parameters of each conditional distribution are output by an MLP . Differentiable decoder . We design the decoder to follow the recursive compositing process in graphics engines , helping the encoder to learn inverse graphics . First , for each leaf node v , we use a neural network g ( · ) to decode its appearance variable into a small image patch x̂v and a ( close to ) binary mask m̂v : ( x̂v , m̂v ) = g ( zapprv ) . Here , g ( · ) is implemented as a spatial broadcast decoder ( Watters et al. , 2019 ) optionally followed by sub-pixel convolutions ( Shi et al. , 2016 ) . We then recursively compose these primitive patches into whole objects and the full scene by applying affine transformations parameterized by the pose variables . Specifically , let u be an internal node , and ch ( u ) be the set of its children . We compose the higher-level image patch x̂u and mask m̂u as follows : x̂u = ∑ v∈ch ( u ) z pres v ·αv ST −1 ( x̂v , z pose v ) , ( 5 ) m̂u = ∑ v∈ch ( u ) z pres v ·αv ST −1 ( m̂v , z pose v ) . ( 6 ) Here , denotes pixel-wise multiplication , and ST −1 is an inverse spatial transformer ( Jaderberg et al. , 2015 ) that differentiably places x̂v and m̂v into the coordinate frame of the parent node u . To deal with occlusion , we include relative depth in zposev , and compute a transparency map αv by the softmax over negative depth values . This ensures that entities with smaller depth will appear in front of entities with larger depth . See Figure 1C for an illustration . When we reach the root node , we obtain an image x̂r and a mask m̂r of all foreground objects . We then use a spatial broadcast decoder ( Watters et al. , 2019 ) to decode zbg into a background image x̂bg . The full scene x can now be modeled as a pixel-wise mixture of foreground and background , where m̂r serves as the mixing weight : p ( x |zfg , zbg ) = m̂r N ( x | x̂r , σ2fg1 ) + ( 1− m̂r ) N ( x | x̂bg , σ2bg1 ) . ( 7 ) Here , σfg and σbg are hyperparameters . 1We use v to refer to both node v in the probabilistic scene graph and the entity that node v represents .

Summary: Generative Scene Graph Networks (GSGN) is a variational auto-encoder with the intermediate representation being tree-like scene graphs. The leaf nodes stand for primitive parts and edges stand for poses to compose parts into objects recursively. The experiments are done in two image datasets of single color, simple shape 2D/3D objects: Multi-dSprites and CLEVR, and the model is able to discover objects without supervision.