Generative Hierarchical Models for Parts, Objects, and Scenes

1 INTRODUCTION . Compositional hierarchies prevail in natural scenes where primitive entities are recursively composed into more abstract entities . Modeling such compositional generative process allows discovery of modular primitives that can be reused across a variety of scenes . Hence , it would bring interpretability and transferability , in which current deep learning models are not quite successful . Due to expensive labeling , such compositional relationships should ideally be learned in an unsupervised manner . Unsupervised approaches can also provide more flexibility and generalization ability since the model is allowed to choose the most appropriate compositional hierarchy for a given scenario . Despite its importance , there has not been much work on unsupervised generative modeling of the compositional hierarchy . Earlier work on hierarchical representation learning ( Lee et al. , 2009 ) obtains a feature hierarchy that captures concepts at different levels of abstraction , with no explicit modeling of composition . Recent researches on deep latent variable models ( Maaløe et al. , 2019 ; Zhao et al. , 2017 ; Sønderby et al. , 2016 ; Bachman , 2016 ) mainly focus on architectural designs and training methods that harness the full expressive power of hierarchical generative models . Although they have shown impressive generation quality and disentanglement of learned representation , the compositional hierarchy is still not captured in a modular and interpretable way . To obtain interpretable scene representation , recent work ( Tieleman , 2014 ; Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Wu et al. , 2017 ; Yao et al. , 2018 ; Romaszko et al. , 2017 ; Deng et al. , 2019 ) has introduced domain-specific decoders that take object pose and appearance information as input and render the object in a similar way to graphics engines . This forces the encoder to invert the rendering process , producing interpretable object-wise pose and appearance representation . In this paper , we extend the interpretable object-wise representation to the hierarchical setting . We propose a deep generative model , called RICH ( Representation of Interpretable Compositional Hierarchies ) , that can use its hierarchy to represent the compositional relationships among interpretable symbolic entities like parts , objects , and scenes . To this end , taking inspiration from capsule networks ( Sabour et al. , 2017 ; Hinton et al. , 2018 ) and the rendering process of computer graphics , we propose a probabilistic scene graph representation that describes the compositional hierarchy as a latent tree . The nodes in the tree correspond to entities in the scene , while the edges indicate the compositional relationships among these entities . We associate an appearance latent with each node to summarize all lower-level composition , and a pose latent with each edge to specify the transformation from the current level to the upper level . To enforce interpretability , the probabilistic scene graph is then paired with a decoder that renders the scene graph by recursively applying the specified transformations . We also introduce learnable templates for the primitive entities . Once learned , RICH is able to generate all lower-level latents and render a partial scene given the latent at a specific level . To infer the scene graph is , however , challenging , since both the tree structure and the latent variables need to be simultaneously inferred . Capsule networks have provided a bottom-up solution to learning the tree structure , but it faces the difficult routing problem caused by the exponentially many possible compositions . Instead , RICH takes a top-down approach that avoids the routing problem . The intuition is that for a given scene , it is natural to first decompose it into high-level objects . If we devote our attention to one of the objects , we can then figure out its constituent parts . In cases where parts are close or have occlusion , we expect the appearance latent of the higher-level object to guide lower-level decomposition , since it summarizes the typical composition for that object . The contributions of this paper are as follows . We propose RICH , the first interpretable representation learning model for compositional hierarchies through probabilistic latent variable modeling . We then implement a three-level prototype of RICH and demonstrate its effectiveness in extensive experiments . RICH is able to learn the hierarchical scene graph representation from images containing multiple compositional objects . Further , it shows decent generation quality and generalization ability to unseen number of objects . 2 RELATED WORK . Interpretable object-wise representation . AIR ( Eslami et al. , 2016 ) is the first generative model that learns interpretable object-wise scene representation . It is able to assign a latent vector ( zpres , zwhere , zwhat ) to each object in the scene , describing the presence , size , center position , and appearance of the object . SPAIR ( Crawford & Pineau , 2019 ) improves the scalability of AIR to images containing a large number of objects . It divides the image into spatially distributed cells , and auto-regressively infers the latent vector for each cell . This crucially reduces the search space for individual cells , since they are each responsible only for explaining objects near themselves . RICH builds upon SPAIR to infer the structure of the probabilistic scene graph . To enable efficient hierarchical inference , we use mean-field approximation for the posterior , allowing inference of all cells to be done in parallel . Hierarchical scene representation . Modeling the part-whole relationship in scenes has attracted growing interest , and it has been utilized for improving image classification , parsing , and segmentation . Two representative models that have inspired our work are hierarchical compositional models ( HCMs ) and capsule networks . In HCMs ( Zhu et al. , 2008 ) , the hierarchical structure is represented as a graph , where leaf nodes interact with image segments , and upper-level nodes store the average position and orientation of lower-level nodes ( with respect to the image coordinates ) . In capsule networks ( Sabour et al. , 2017 ; Hinton et al. , 2018 ) , the part-whole relationship is used for achieving viewpoint invariance . The key insight is that the relative pose of parts with respect to objects is viewpoint invariant , and is thus suitable to be learned as network weights . However , neither of these two approaches uses generative modeling , and they have been applied only to scenes with one dominant object . More recently , the part-whole relationship has been explored in modeling 3D shapes ( Tulsiani et al. , 2017 ; Li et al. , 2017 ; Zhu et al. , 2018 ) and motion decomposition ( Xu et al. , 2019 ) . Although they have employed generative modeling , the latent space is not hierarchically structured , and typically only contains the top-level representation . In contrast , RICH represents the entire scene graph in latent space , modeling both the hierarchical structure and the object pose and appearance at each level as latent variables . This representation is arguably richer , and enables simultaneous learning of a collection of generative models for parts , objects , and scenes . 3 THE PROPOSED MODEL : RICH . RICH ( Representation of Interpretable Compositional Hierarchies ) is a generative model that captures the recursive compositional structure inherent in natural scenes . It builds a tree-structured representation similar to scene graphs in computer graphics ( Foley et al. , 1996 ) . The nodes in the tree describe entities at various levels of abstraction in the scene , and the edges indicate the compo- sitional relationships among these entities . Specifically , each leaf node represents a primitive entity that is not further decomposed . Each internal node represents an abstract entity that is composed from its children nodes . The composition is specified by the relative pose of each child node with respect to the parent node , and this pose information is stored on the corresponding edges . 3.1 GENERATIVE PROCESS . To make a generative model , we associate an appearance vector zapprv with each node v , and a pose vector zposev with the edge between node v and its parent pa ( v ) , as shown in Figure 1A . The intuition is that zapprpa ( v ) represents the entity at pa ( v ) in its canonical pose , summarizing all lower-level composition in the subtree rooted at pa ( v ) . Conditioning on zapprpa ( v ) , we can then sample z appr v and z pose v , the relative pose of v with respect to pa ( v ) . Thus , the latent vectors in any subtree can be recursively generated . Let V be the set of all nodes , r ∈ V be the root node , and L ⊆ V be the set of leaf nodes . The generative model for the entire scene x can be written as follows : p ( x ) = ∫ p ( x | zposeV \ { r } , z appr L ) p ( z appr r ) ∏ v∈V \ { r } p ( zposev | z appr pa ( v ) ) p ( z appr v | z appr pa ( v ) ) dz , ( 1 ) where we assume conditional independence among all latents zposev and z appr v that have the same pa ( v ) . This gives disentangled and interpretable scene representation . We design the decoder p ( x|zposeV \ { r } , z appr L ) to closely follow the rendering process from a given scene graph . First , for each leaf node v ∈ L , we use a neural network to decode its appearance vector into a small image patch x̂v and a ( close to ) binary mask m̂v the same size as x̂v . Here we assume that x̂v has already been masked by m̂v , meaning x̂v ( i , j ) = 0 for all pixel locations ( i , j ) where m̂v ( i , j ) = 0 . We then recursively compose these primitive patches into the entire scene by applying affine transformations level by level . Specifically , let u be an internal node , and ch ( u ) be the set of its children . We compose the higher-level image patch x̂u and mask m̂u as follows : x̂u = ∑ v∈ch ( u ) αv ST −1 ( x̂v , zposev ) , m̂u = ∑ v∈ch ( u ) αv ST −1 ( m̂v , zposev ) , ( 2 ) where a spatial transformer ST ( Jaderberg et al. , 2015 ) is used to properly place x̂v and m̂v into the coordinate frame of the parent node u , according to the scaling , rotation and translation parameters given by zposev . In addition , z pose v also provides relative depth information that helps deal with occlusion . Entities with smaller depth will appear in front of entities with larger depth . This is enforced by a transparency map αv that assigns pixel-wise weights to each transformed patch according to its relative depth . See Figure 1C for an illustration . To ensure that unoccluded part of entities will remain visible , we normalize αv after applying the transformed mask m̃v = ST −1 ( m̂v , zposev ) , namely for all pixel locations ( i , j ) , αv ( i , j ) = 0 if m̃v ( i , j ) = 0 , ∑ v∈ch ( u ) αv ( i , j ) = 1 if ∑ v∈ch ( u ) m̃v ( i , j ) > 0 . ( 3 ) The final decoder output p ( x|zposeV \ { r } , z appr L ) = N ( x̂r , σ2I ) , where σ is a hyperparameter .

Summary: Contributions: this submission proposes a generative framework which employs a hierarchical decomposition of scene, objects and parts. It modifies the SPAIR framework (Crawford & Pineau) and replaces the recurrent generation with parallel generation, by assuming the number of objects is known and fixed. Results are presented on two synthetic datasets with 2D or 3D shapes.