Learning Global Spatial Information for Multi-View Object-Centric Models

1 INTRODUCTION . Extracting representations of individual objects from direct perception ( e.g. , visual input ) is called object-centric representation learning . It is considered to be effective in many aspects such as robotics ( Devin et al. , 2018 ; Veerapaneni et al. , 2019 ; Kulkarni et al. , 2019 ) , reasoning ( Ding et al. , 2021 ) , and sample efficiency ( Watters et al. , 2019a ) . Among them , the deep generative model ( DGM ) -based method is one of the promising directions , as it can infer rich information about objects in a fully unsupervised manner , and some of its models can also generate novel scenes ( Greff et al. , 2017 ; Locatello et al. , 2020 ; Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2020 ; Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Jiang & Ahn , 2020 ) . Recently , several studies have been working on multi-view object-centric models , which predict unobserved views of a scene : novel view synthesis , and infer object-centric representations from several observation views ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) . This can be also regarded as an object-centric expansion of the Generative Query Network ( GQN ) ( Eslami et al. , 2018 ) . In order to represent multi-object scenes , the properties of individual objects and the spatial arrangement of objects should be specified . However , existing multi-view object-centric methods only explicitly model representations of individual objects , and each of them includes their own spatial information separately . Namely , relationship between objects is not represented in this case . This modeling can degrade novel view synthesis and segmentation , because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects ’ spatial relationship . This can be serious especially when the number of available observation views is limited . In addition , this modeling can also lead to inability to generate physically plausible novel scenes , because they need to place objects independently , which often brings about collisions and misplacement of objects . Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables . This variable can be independent of object-level latent variables , or can be modeled as a global latent on top of object-level latent variables . In the former formulation , those latent variables need to be disentangled during training , which can often be challenging . From another point of view , it is common assumption in cognitive science that humans recognize the whole and parts , or objects , separately ( Schmidt , 2009 ) . In feature integration theory ( Treisman & Gelade , 1980 ) , visual feature maps become “ master map of locations ” combined with positions of objects in visual space . Then , attention binds those features to recognize individual objects . Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property . For the above reasons , we believe that introducing a global latent variable is more natural step for improving object-centric generative models . In this paper , we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference : novel view synthesis and segmentation . In addition , our model can also generate physically plausible novel scenes . The problem is that how to execute inference and training with the proposed model is unclear . In regards to this point , we propose several crucial components : Global Encoder , Sequential Encoder and Structured Prior . Firstly , the model needs to infer global representation from variable number of input views , thus we introduced an encoder using Product of Experts ( PoE ) ( Hinton , 2002 ) , which we refer to as Global Encoder . Secondly , the structured , complex posterior of object-level latent variables need to be sampled including spatial relationship between objects in order to generated physically plausible novel scenes . Therefore , we introduced a learnable prior called Structured Prior , which is implemented by Transformer ( Vaswani et al. , 2017 ) . Lastly , we introduced an encoder using auto-regressive model called Sequential Encoder . This stabilize inference and training with the hierarchical probabilistic model of the proposed method . The proposed model has representation about the whole , or spatial information of a scene and has representations about a part of a scene ( i.e. , individual properties of objects ) ; hence , we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and Sampling ( WeLIS ) . Contributions : . • We introduce a multi-view object-centric model with a global latent variable which represents the spatial configuration of objects in a scene . • We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model . • We show that the proposed model performs well in terms of inference quality , and can generate novel scenes and corresponding segmentation masks properly , unlike existing methods . 2 RELATED WORK . Object-Centric Models : The objective of object-centric representation learning is to obtain representations of each object from visual input . It is applicable to many domains such as robotics ( Devin et al. , 2018 ; Florence et al. , 2019 ; Veerapaneni et al. , 2019 ) , reasoning ( Ding et al. , 2021 ) and reinforcement learning ( Watters et al. , 2019a ) . One of the promising directions of object-centric representation learning is variational autoencoder ( VAE ) -based ( Kingma & Welling , 2013 ) methods , as they can obtain rich information about objects that can also be disentangled , and some of them can generate novel scenes outside of the empirical distribution ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2020 ; Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Jiang & Ahn , 2020 ) . VAE-based object-centric models are often referred to as “ scene interpretation models ” due to their inference ability . Some models are capable of object-centric generation ( Dai et al. , 2019 ; NguyenPhuoc et al. , 2020 ; Niemeyer & Geiger , 2021 ) , however , these methods focus on generation and not for inference . Multi-View Object-Centric Models : Lately , several studies have been working on multi-view scene interpretation models ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) . These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way , using several observation views as input . This problem setting can be regarded as an object-centric expansion of GQN family ( Eslami et al. , 2018 ; Tobin et al. , 2019 ; Kosiorek et al. , 2021 ) . The above mentioned single-view setting is referred to as multi-object-single-view ( MOSV ) , and the multi-view setting is defined as multiobject-multi-view ( MOMV ) in ( Nanbo et al. , 2020 ) . Table 1 compares methods related to the MOMV setting . Novel view synthesis stands for prediction of unobserved viewpoints , and novel scene generation stands for random generation of novel scenes . Among these models , ROOTS and MulMON ( Nanbo et al. , 2020 ; Chen et al. , 2021 ) are the only generative models that work on a MOMV setting . In addition to this , O3V ( Henderson & Lampert , 2020 ) have a similar problem setting , but it takes fixed number of viewpoints as input . ObSuRF ( Stelzner et al. , 2021 ) and uORF ( Yu et al. , 2021 ) also work on MOMV setting , but these are not generative models , and are based on Slot Attention ( Locatello et al. , 2020 ) and NeRF ( Mildenhall et al. , 2020 ) . Therefore these methods can not generate novel samples and does not obtain low dimensional representation space , or latent space . In this paper , we introduce a new MOMV model with a global latent variable , which can potentially represents the spatial arrangement of objects in a scene . The proposed model is the only MOMV model that can generate novel scenes . In this paper , we set MulMON as our baseline model due to the following reasons . Firstly , MulMON uses segmentation masks to represent objects ’ regions unlike ROOTS which uses bounding boxes . Bounding box-based methods , which originate from ( Eslami et al. , 2016 ) tend to be more sensitive to object size and shape ( Engelcke et al. , 2021 ) . Secondly , MulMON is more robust in terms of the number of observation views required . Although we built our model based on MulMON , a similar approach can be applied to ROOTS and other methods . Introducing the global latent variable itself has been done in some studies in various areas ( Vasco et al. , 2020 ; Akuzawa et al. , 2021 ) . In particular , GNM ( Jiang & Ahn , 2020 ) introduces a global latent variable to bounding box-based single-view object-centric model for random generation . However , how to execute stable inference and training with the proposed MOMV model is unclear , and is thus the points of investigation in this paper . 3 METHOD . Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability . We achieved this by introducing a global latent variable for global spatial information on top of object latent variables , which is formulated as hierarchical probabilistic model . In this paper , we introduce several crucial components to realize stable inference and training with this MOMV model . An overview of the proposed model is shown in Figure 1 . In this section , we first explain the problem setting in this paper , and then the probabilistic model of our proposed model . Lastly , the training method and additional details of the introduced components are explained . 3.1 PROBLEM SETTING . The target of this research is 3D scenes with multiple objects , and thus a dataset with N data points can be defined as D = { X1 , · · · , XN } , where Xi = { ( x1i , v1i ) , ... , ( xMi , vMi ) } , which corresponds to one scene withM views , and v is a viewpoint vector with arbitrary dimensions , which are mostly three . The task is to predict queried unobserved views of the scene given several observations { ( x1i , v1i ) , · · · , ( x Nobs i , v Nobs i ) } , and infer K representations each of which correspond to an object in the scene Z = { z1 , · · · , zK } . In practice , the M views in Xi is split into Nobs observations and Nqry queries in the training process . Nobs can be varied in every iteration for generalization . 3.2 PROBABILISTIC MODEL . Generative Model : As mentioned above , we introduced the hierarchical probabilistic model with global latent zg on top of object-level latent variables z = { z1 , · · · , zK } . In the following sections , object-level latent variables are abbreviated as z , as required . The marginal likelihood is thus : pγ ( x | v ) = ∫ ∫ pθ ( x|z , v ) pφ ( z | zg ) p ( zg ) dzdzg , ( 1 ) where γ = { θ , φ } . The generative model pθ ( x|z , v ) is a spatial Gaussian mixture model derived from prior works ( Nanbo et al. , 2020 ; Burgess et al. , 2019 ) that can be written as : pθ ( x|z , v ) = D∏ i=1 K∑ k=1 pθ ( Ci = k | zk , v ) ⊗ pθ ( xik|zk , v ) , ( 2 ) where D ∈ N is the number of image pixels , and K is the number of mixture components which are often referred to as “ slots ” . The decoder has parameters θ and implemented by a broadcast decoder ( Watters et al. , 2019b ) . zk and v are first concatenated and projected onto a single vector fk by MLP and then fed into the decoder . The decoder generates K components { x1 , · · · , xK } and corresponding segmentation masks from { f1 , · · · , fK } . The second term in the r.h.s of Equation 1 pφ ( z | zg ) is a learnable prior which we refer to as the Structured Prior , and the third term is a standard Gaussian distribution . The Structured Prior is explained in Section 3.3 . Inference Model : We used amortized variational inference ( Kingma & Welling , 2013 ) and computed variational approximate posteriors because the marginal likelihood in Equation 1 is intractable . We factorized the variational posteriors as : qψ ( z , z g | X ) = qψ ( z1 , · · · , zK | zg ) qψ ( zg | X ) , ( 3 ) where X = { ( x1 , v1 ) , . . . , ( xNobs , vNobs ) } . The first RHS term and second RHS term are approximate posteriors of z and zg respectively . We refer to these encoders as Sequential Encoder and Global Encoder . The Global Encoder adopts PoE ( Hinton , 2002 ) to deal with a variable number of input observation viewsNobs . In particular , we used PoE with prior expert ( Wu & Goodman , 2018 ) . In addition to this , we applied normalizing flow ( NF ) ( Rezende & Mohamed , 2015 ) to this inference model . Although this is optional , the posterior should be expressive in order to represent spatial configuration of complex 3D multi-object scenes . Therefore , the variational approximate posterior is : zg ∼ qψ ( zg | X ) = ( p ( zg1 ) Nobs∏ v=1 qψ ( z g 1 | xv , vv ) ) T∏ t=1 ∣∣∣∣det ∂fψ∂zgt ∣∣∣∣−1 , ( 4 ) where T is the number of transformations of NF , and Nobs is the number of available observation views . We used planar flow for its simplicity , but any methods can be applied and can improve the results potentially . Thus , the transform fψ in the above equation is fψ ( zi+1 ) = zi + ui · h ( wT zi + bi ) . The Sequential Encoder was implemented in auto-regressive manner using LSTM ( Hochreiter & Schmidhuber , 1997 ) . We tried several architectures , but non-auto-regressive models were unstable as long as we have tested . Firstly , this auto-regressive part infers K latent variables z01 , . . . , z 0 K and then , these are updated using iterative amortized inference ( IAI ) ( Marino et al. , 2018 ) as MulMON and IODINE ( Greff et al. , 2019 ) do . However , we omitted image-sized inputs as in ( Emami et al. , 2021 ) . Thus , the Sequential Encoder can be defined as : z01 , . . . , z 0 K ∼ K∏ k=1 qψ ( z 0 k | z01 : k , zg ) , ( 5 ) zi+1k = z i k + fψ ( z i k , z g , ∇zikL i IAI ) , ( 6 ) where ∇zikL i IAI is a gradient of negative log-likelihood in that iteration , and fψ is a refinement network that is implemented by MLP followed by LSTM . The posterior can be updated arbitrary times , and we set L = 5 through this paper . The details ofLiIAI will be explained in the next section . Training : All the parameters in this model can be trained end-to-end by maximizing evidence lower bound ( ELBO ) on the log-marginal likelihood log p ( x1 , · · · , xN ) . The ELBO of the proposed model is LELBO = Eq ( z , zg|X ) [ log pθ ( x|z , v ) pφ ( z | zg ) p ( zg ) qψ ( z | zg ) qψ ( zg | X ) ] ( 7 ) = Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] −KL [ qψ ( zg | X ) || p ( zg ) ] − Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] . Here , KL stands for Kullback–Leibler divergence . Then , introducing two coefficient β1 and β2 for balancing KL terms , the objective can be written as L = Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] − β1KL [ qψ ( zg | X ) || p ( zg ) ] − β2Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] . ( 8 ) Moreover , due to the iterative process of IAI , negative log-likelihood is computed as Eq ( z , zg|X ) [ log pθ ( x | z , v ) ] = 2 L ( L+ 1 ) L−1∑ i=0 ( i+ 1 ) LiIAI , ( 9 ) LiIAI = Ezi∼q ( zi , zg|X ) [ log pθ ( x|zi , v ) ] , ( 10 ) where , L is the number of iterations in IAI . As shown in Equation 9 , log-likelihoods in the later steps of iterative refinement are emphasized as in MulMON and IODINE . In addition to this , we used GECO ( Rezende & Viola , 2018 ) as in some existing methods ( Engelcke et al. , 2020 ; Emami et al. , 2021 ) . GECO reinterprets the maximization of ELBO as a constrained optimization problem that takes KL terms as the main objective and log-likelihood as the constraint . This optimization is done by the method of Lagrange multipliers , namely , the coefficient of KL terms β is regarded as a Lagrange multiplier , and automatically tweaked during training . GECO is not necessary for our model , but proper balancing of β further improves random generation quality . The reinterpreted optimization objective thus becomes argmin ψ , φ KL [ qψ ( z g | xobs , vobs ) || p ( zg ) ] + Eqψ ( zg|X ) [ KL [ qψ ( z | z g ) || pφ ( z | zg ) ] ] ( 11 ) such that − Eqψ ( z , zg|X ) [ log pθ ( x | z , v ) ] 5 R , ( 12 ) where R ∈ R is the maximum allowed negative log-likelihood , which is a hyperparameter in GECO instead of β . Although all the parameters can be trained end-to-end , we found that end-to-end training including the parameters of Structured Prior φ is slightly unstable and often degrades novel scene generation . Thus , we introduced a separate training strategy , in which φ is updated after the training of the main model . Detail of this method is explained in the next section .

Summary: The paper proposes a new structured generative model of multi-view images, representing their latent decomposition into objects. This model extents MulMON -- a spatial mixture model over multi-view images that is suitable for image decomposition but lacks a prior necessary for synthesis -- by adding such a prior (using an autoregressive model), and making some other architectural changes (e.g. adding a normalising flow posterior instead of gaussian). In the proposed model, a global latent variable conditions the sampling of per-object latents, which in turn parameterise a spatial mixture model for each image. The model is evaluated on novel view synthesis (NVS) and a priori scene synthesis, on simple datasets of renderings (e.g. multi-viewpoint CLEVR); it is shown to out-perform a baseline (MulMON) on several metrics for both tasks quantitatively, and qualitative results are also displayed.