Learning Global Spatial Information for Multi-View Object-Centric Models

1 INTRODUCTION . Extracting representations of individual objects from direct perception ( e.g. , visual input ) is called object-centric representation learning . It is considered to be effective in many aspects such as robotics ( Devin et al. , 2018 ; Veerapaneni et al. , 2019 ; Kulkarni et al. , 2019 ) , reasoning ( Ding et al. , 2021 ) , and sample efficiency ( Watters et al. , 2019a ) . Among them , the deep generative model ( DGM ) -based method is one of the promising directions , as it can infer rich information about objects in a fully unsupervised manner , and some of its models can also generate novel scenes ( Greff et al. , 2017 ; Locatello et al. , 2020 ; Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2020 ; Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Jiang & Ahn , 2020 ) . Recently , several studies have been working on multi-view object-centric models , which predict unobserved views of a scene : novel view synthesis , and infer object-centric representations from several observation views ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) . This can be also regarded as an object-centric expansion of the Generative Query Network ( GQN ) ( Eslami et al. , 2018 ) . In order to represent multi-object scenes , the properties of individual objects and the spatial arrangement of objects should be specified . However , existing multi-view object-centric methods only explicitly model representations of individual objects , and each of them includes their own spatial information separately . Namely , relationship between objects is not represented in this case . This modeling can degrade novel view synthesis and segmentation , because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects ’ spatial relationship . This can be serious especially when the number of available observation views is limited . In addition , this modeling can also lead to inability to generate physically plausible novel scenes , because they need to place objects independently , which often brings about collisions and misplacement of objects . Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables . This variable can be independent of object-level latent variables , or can be modeled as a global latent on top of object-level latent variables . In the former formulation , those latent variables need to be disentangled during training , which can often be challenging . From another point of view , it is common assumption in cognitive science that humans recognize the whole and parts , or objects , separately ( Schmidt , 2009 ) . In feature integration theory ( Treisman & Gelade , 1980 ) , visual feature maps become “ master map of locations ” combined with positions of objects in visual space . Then , attention binds those features to recognize individual objects . Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property . For the above reasons , we believe that introducing a global latent variable is more natural step for improving object-centric generative models . In this paper , we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference : novel view synthesis and segmentation . In addition , our model can also generate physically plausible novel scenes . The problem is that how to execute inference and training with the proposed model is unclear . In regards to this point , we propose several crucial components : Global Encoder , Sequential Encoder and Structured Prior . Firstly , the model needs to infer global representation from variable number of input views , thus we introduced an encoder using Product of Experts ( PoE ) ( Hinton , 2002 ) , which we refer to as Global Encoder . Secondly , the structured , complex posterior of object-level latent variables need to be sampled including spatial relationship between objects in order to generated physically plausible novel scenes . Therefore , we introduced a learnable prior called Structured Prior , which is implemented by Transformer ( Vaswani et al. , 2017 ) . Lastly , we introduced an encoder using auto-regressive model called Sequential Encoder . This stabilize inference and training with the hierarchical probabilistic model of the proposed method . The proposed model has representation about the whole , or spatial information of a scene and has representations about a part of a scene ( i.e. , individual properties of objects ) ; hence , we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and Sampling ( WeLIS ) . Contributions : . • We introduce a multi-view object-centric model with a global latent variable which represents the spatial configuration of objects in a scene . • We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model . • We show that the proposed model performs well in terms of inference quality , and can generate novel scenes and corresponding segmentation masks properly , unlike existing methods . 2 RELATED WORK . Object-Centric Models : The objective of object-centric representation learning is to obtain representations of each object from visual input . It is applicable to many domains such as robotics ( Devin et al. , 2018 ; Florence et al. , 2019 ; Veerapaneni et al. , 2019 ) , reasoning ( Ding et al. , 2021 ) and reinforcement learning ( Watters et al. , 2019a ) . One of the promising directions of object-centric representation learning is variational autoencoder ( VAE ) -based ( Kingma & Welling , 2013 ) methods , as they can obtain rich information about objects that can also be disentangled , and some of them can generate novel scenes outside of the empirical distribution ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2020 ; Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Jiang & Ahn , 2020 ) . VAE-based object-centric models are often referred to as “ scene interpretation models ” due to their inference ability . Some models are capable of object-centric generation ( Dai et al. , 2019 ; NguyenPhuoc et al. , 2020 ; Niemeyer & Geiger , 2021 ) , however , these methods focus on generation and not for inference . Multi-View Object-Centric Models : Lately , several studies have been working on multi-view scene interpretation models ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) . These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way , using several observation views as input . This problem setting can be regarded as an object-centric expansion of GQN family ( Eslami et al. , 2018 ; Tobin et al. , 2019 ; Kosiorek et al. , 2021 ) . The above mentioned single-view setting is referred to as multi-object-single-view ( MOSV ) , and the multi-view setting is defined as multiobject-multi-view ( MOMV ) in ( Nanbo et al. , 2020 ) . Table 1 compares methods related to the MOMV setting . Novel view synthesis stands for prediction of unobserved viewpoints , and novel scene generation stands for random generation of novel scenes . Among these models , ROOTS and MulMON ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) are the only generative models that work on a MOMV setting . In addition to this , O3V ( Henderson & Lampert , 2020 ) have a similar problem setting , but it takes fixed number of viewpoints as input . ObSuRF ( Stelzner et al. , 2021 ) and uORF ( Yu et al. , 2021 ) also work on MOMV setting , but these are not generative models , and are based on Slot Attention ( Locatello et al. , 2020 ) and NeRF ( Mildenhall et al. , 2020 ) . Therefore these methods can not generate novel samples and does not obtain low dimensional representation space , or latent space . In this paper , we introduce a new MOMV model with a global latent variable , which can potentially represents the spatial arrangement of objects in a scene . The proposed model is the only MOMV model that can generate novel scenes . In this paper , we set MulMON as our baseline model due to the following reasons . Firstly , MulMON uses segmentation masks to represent objects ’ regions unlike ROOTS which uses bounding boxes . Bounding box-based methods , which originate from ( Eslami et al. , 2016 ) tend to be more sensitive to object size and shape ( Engelcke et al. , 2021 ) . Secondly , MulMON is more robust in terms of the number of observation views required . Although we built our model based on MulMON , a similar approach can be applied to ROOTS and other methods . Introducing the global latent variable itself has been done in some studies in various areas ( Vasco et al. , 2020 ; Akuzawa et al. , 2021 ) . In particular , GNM ( Jiang & Ahn , 2020 ) introduces a global latent variable to bounding box-based single-view object-centric model for random generation . However , how to execute stable inference and training with the proposed MOMV model is unclear , and is thus the points of investigation in this paper . 3 METHOD . Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability . We achieved this by introducing a global latent variable for global spatial information on top of object latent variables , which is formulated as hierarchical probabilistic model . In this paper , we introduce several crucial components to realize stable inference and training with this MOMV model . An overview of the proposed model is shown in Figure 1 . In this section , we first explain the problem setting in this paper , and then the probabilistic model of our proposed model . Lastly , the training method and additional details of the introduced components are explained . 3.1 PROBLEM SETTING . The target of this research is 3D scenes with multiple objects , and thus a dataset with N data points can be defined as D = { X1 , · · · , XN } , where Xi = { ( x1i , v1i ) , ... , ( xMi , vMi ) } , which corresponds to one scene withM views , and v is a viewpoint vector with arbitrary dimensions , which are mostly three . The task is to predict queried unobserved views of the scene given several observations { ( x1i , v1i ) , · · · , ( x Nobs i , v Nobs i ) } , and infer K representations each of which correspond to an object in the scene Z = { z1 , · · · , zK } . In practice , the M views in Xi is split into Nobs observations and Nqry queries in the training process . Nobs can be varied in every iteration for generalization . 3.2 PROBABILISTIC MODEL . Generative Model : As mentioned above , we introduced the hierarchical probabilistic model with global latent zg on top of object-level latent variables z = { z1 , · · · , zK } . In the following sections , object-level latent variables are abbreviated as z , as required . The marginal likelihood is thus : pγ ( x | v ) = ∫ ∫ pθ ( x|z , v ) pφ ( z | zg ) p ( zg ) dzdzg , ( 1 ) where γ = { θ , φ } . The generative model pθ ( x|z , v ) is a spatial Gaussian mixture model derived from prior works ( Nanbo et al. , 2020 ; Burgess et al. , 2019 ) that can be written as : pθ ( x|z , v ) = D∏ i=1 K∑ k=1 pθ ( Ci = k | zk , v ) ⊗ pθ ( xik|zk , v ) , ( 2 ) where D ∈ N is the number of image pixels , and K is the number of mixture components which are often referred to as “ slots ” . The decoder has parameters θ and implemented by a broadcast decoder ( Watters et al. , 2019b ) . zk and v are first concatenated and projected onto a single vector fk by MLP and then fed into the decoder . The decoder generates K components { x1 , · · · , xK } and corresponding segmentation masks from { f1 , · · · , fK } . The second term in the r.h.s of Equation 1 pφ ( z | zg ) is a learnable prior which we refer to as the Structured Prior , and the third term is a standard Gaussian distribution . The Structured Prior is explained in Section 3.3 . Inference Model : We used amortized variational inference ( Kingma & Welling , 2013 ) and computed variational approximate posteriors because the marginal likelihood in Equation 1 is intractable . We factorized the variational posteriors as : qψ ( z , z g | X ) = qψ ( z1 , · · · , zK | zg ) qψ ( zg | X ) , ( 3 ) where X = { ( x1 , v1 ) , . . . , ( xNobs , vNobs ) } . The first RHS term and second RHS term are approximate posteriors of z and zg respectively . We refer to these encoders as Sequential Encoder and Global Encoder . The Global Encoder adopts PoE ( Hinton , 2002 ) to deal with a variable number of input observation viewsNobs . In particular , we used PoE with prior expert ( Wu & Goodman , 2018 ) . In addition to this , we applied normalizing flow ( NF ) ( Rezende & Mohamed , 2015 ) to this inference model . Although this is optional , the posterior should be expressive in order to represent spatial configuration of complex 3D multi-object scenes . Therefore , the variational approximate posterior is : zg ∼ qψ ( zg | X ) = ( p ( zg1 ) Nobs∏ v=1 qψ ( z g 1 | xv , vv ) ) T∏ t=1 ∣∣∣∣det ∂fψ∂zgt ∣∣∣∣−1 , ( 4 ) where T is the number of transformations of NF , and Nobs is the number of available observation views . We used planar flow for its simplicity , but any methods can be applied and can improve the results potentially . Thus , the transform fψ in the above equation is fψ ( zi+1 ) = zi + ui · h ( wT zi + bi ) . The Sequential Encoder was implemented in auto-regressive manner using LSTM ( Hochreiter & Schmidhuber , 1997 ) . We tried several architectures , but non-auto-regressive models were unstable as long as we have tested . Firstly , this auto-regressive part infers K latent variables z01 , . . . , z 0 K and then , these are updated using iterative amortized inference ( IAI ) ( Marino et al. , 2018 ) as MulMON and IODINE ( Greff et al. , 2019 ) do . However , we omitted image-sized inputs as in ( Emami et al. , 2021 ) . Thus , the Sequential Encoder can be defined as : z01 , . . . , z 0 K ∼ K∏ k=1 qψ ( z 0 k | z01 : k , zg ) , ( 5 ) zi+1k = z i k + fψ ( z i k , z g , ∇zikL i IAI ) , ( 6 ) where ∇zikL i IAI is a gradient of negative log-likelihood in that iteration , and fψ is a refinement network that is implemented by MLP followed by LSTM . The posterior can be updated arbitrary times , and we set L = 5 through this paper . The details ofLiIAI will be explained in the next section . Training : All the parameters in this model can be trained end-to-end by maximizing evidence lower bound ( ELBO ) on the log-marginal likelihood log p ( x1 , · · · , xN ) . The ELBO of the proposed model is LELBO = Eq ( z , zg|X ) [ log pθ ( x|z , v ) pφ ( z | zg ) p ( zg ) qψ ( z | zg ) qψ ( zg | X ) ] ( 7 ) = Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] −KL [ qψ ( zg | X ) || p ( zg ) ] − Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] . Here , KL stands for Kullback–Leibler divergence . Then , introducing two coefficient β1 and β2 for balancing KL terms , the objective can be written as L = Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] − β1KL [ qψ ( zg | X ) || p ( zg ) ] − β2Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] . ( 8 ) Moreover , due to the iterative process of IAI , negative log-likelihood is computed as Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] = 2 L ( L+ 1 ) L−1∑ i=0 ( i+ 1 ) LiIAI , ( 9 ) LiIAI = Ezi∼q ( zi , zg|X ) [ log pθ ( x|zi , v ) ] , ( 10 ) where , L is the number of iterations in IAI . As shown in Equation 9 , log-likelihoods in the later steps of iterative refinement are emphasized as in MulMON and IODINE . In addition to this , we used GECO ( Rezende & Viola , 2018 ) as in some existing methods ( Engelcke et al. , 2020 ; Emami et al. , 2021 ) . GECO reinterprets the maximization of ELBO as a constrained optimization problem that takes KL terms as the main objective and log-likelihood as the constraint . This optimization is done by the method of Lagrange multipliers , namely , the coefficient of KL terms β is regarded as a Lagrange multiplier , and automatically tweaked during training . GECO is not necessary for our model , but proper balancing of β further improves random generation quality . The reinterpreted optimization objective thus becomes argmin ψ , φ KL [ qψ ( z g | xobs , vobs ) || p ( zg ) ] + Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] ( 11 ) such that − Eqψ ( z , zg|X ) [ log pθ ( x | z , v ) ] 5 R , ( 12 ) where R ∈ R is the maximum allowed negative log-likelihood , which is a hyperparameter in GECO instead of β . Although all the parameters can be trained end-to-end , we found that end-to-end training including the parameters of Structured Prior φ is slightly unstable and often degrades novel scene generation . Thus , we introduced a separate training strategy , in which φ is updated after the training of the main model . Detail of this method is explained in the next section .

Summary: This paper presents a hierarchical, object centric view conditional generative model for scenes. The latent structure includes a global latent variable which captures general scene structure/configuration and conditions a set of slotted latents, each one capturing a single object in the scene. An inference method is proposed - a product of expert + flow encoder for the global, and a sequential encoder + iterative refinement for the slots. Everything is view conditional so training requires camera information, but novel views may be synthesized at ease. The model is demonstrated to work on CLEVR and a slightly more complex version of CLEVR with more object structures as well as the Jaco arm dataset.