Year: 2021

Abstract:
Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views. In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are specified; however, existing multi-view object-centric models only infer object-level representations and lack spatial information. This insufficient modeling can degrade novel-view synthesis quality and make it difficult to generate novel scenes. We can model both spatial information and object representations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables. However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear. Therefore, we introduce several crucial components which help inference and training with the proposed model. We show that the proposed method achieves good inference quality and can also generate novel scenes.

heading: 1 INTRODUCTION
text: Extracting representations of individual objects from direct perception (e.g., visual input) is called object-centric representation learning. It is considered to be effective in many aspects such as robotics (Devin et al., 2018; Veerapaneni et al., 2019; Kulkarni et al., 2019), reasoning (Ding et al., 2021), and sample efficiency (Watters et al., 2019a). Among them, the deep generative model (DGM)-based method is one of the promising directions, as it can infer rich information about objects in a fully unsupervised manner, and some of its models can also generate novel scenes (Greff et al., 2017; Locatello et al., 2020; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020; Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020).
Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene: novel view synthesis, and infer object-centric representations from several observation views (Nanbo et al., 2020; Chen et al., 2021). This can be also regarded as an object-centric expansion of the Generative Query Network (GQN) (Eslami et al., 2018). In order to represent multi-object scenes, the properties of individual objects and the spatial arrangement of objects should be specified. However, existing multi-view object-centric methods only explicitly model representations of individual objects, and each of them includes their own spatial information separately. Namely, relationship between objects is not represented in this case. This modeling can degrade novel view synthesis and segmentation, because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects’ spatial relationship. This can be serious especially when the number of available observation views is limited. In addition, this modeling can also lead to inability to generate physically plausible novel scenes, because they need to place objects independently, which often brings about collisions and misplacement of objects.
Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables. This variable can be independent of object-level latent variables, or can be modeled as a global latent on top of object-level latent variables. In the former formulation, those latent variables need to be disentangled during training, which can often be challenging. From another point of view, it is common assumption in cognitive science that humans recognize the whole and parts, or objects, separately (Schmidt, 2009). In feature integration theory (Treisman & Gelade, 1980), visual feature maps become “master map of locations” combined with positions of
objects in visual space. Then, attention binds those features to recognize individual objects. Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property. For the above reasons, we believe that introducing a global latent variable is more natural step for improving object-centric generative models.
In this paper, we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference: novel view synthesis and segmentation. In addition, our model can also generate physically plausible novel scenes. The problem is that how to execute inference and training with the proposed model is unclear. In regards to this point, we propose several crucial components: Global Encoder, Sequential Encoder and Structured Prior. Firstly, the model needs to infer global representation from variable number of input views, thus we introduced an encoder using Product of Experts (PoE) (Hinton, 2002), which we refer to as Global Encoder. Secondly, the structured, complex posterior of object-level latent variables need to be sampled including spatial relationship between objects in order to generated physically plausible novel scenes. Therefore, we introduced a learnable prior called Structured Prior, which is implemented by Transformer (Vaswani et al., 2017). Lastly, we introduced an encoder using auto-regressive model called Sequential Encoder. This stabilize inference and training with the hierarchical probabilistic model of the proposed method.
The proposed model has representation about the whole, or spatial information of a scene and has representations about a part of a scene (i.e., individual properties of objects); hence, we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and Sampling (WeLIS).

heading: Contributions:
text: • We introduce a multi-view object-centric model with a global latent variable which represents the spatial configuration of objects in a scene.
• We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model.
• We show that the proposed model performs well in terms of inference quality, and can generate novel scenes and corresponding segmentation masks properly, unlike existing methods.

heading: 2 RELATED WORK
text: Object-Centric Models: The objective of object-centric representation learning is to obtain representations of each object from visual input. It is applicable to many domains such as robotics (Devin et al., 2018; Florence et al., 2019; Veerapaneni et al., 2019), reasoning (Ding et al., 2021) and reinforcement learning (Watters et al., 2019a). One of the promising directions of object-centric representation learning is variational autoencoder (VAE)-based (Kingma & Welling, 2013) methods, as they can obtain rich information about objects that can also be disentangled, and some of them can generate novel scenes outside of the empirical distribution (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020; Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020). VAE-based object-centric models are often referred to as “scene interpretation models” due to their inference ability. Some models are capable of object-centric generation (Dai et al., 2019; NguyenPhuoc et al., 2020; Niemeyer & Geiger, 2021), however, these methods focus on generation and not for inference.
Multi-View Object-Centric Models: Lately, several studies have been working on multi-view scene interpretation models (Nanbo et al., 2020; Chen et al., 2021). These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way, using several observation views as input. This problem setting can be regarded as an object-centric expansion of GQN family (Eslami et al., 2018; Tobin et al., 2019; Kosiorek et al., 2021). The above mentioned single-view setting is referred to as multi-object-single-view (MOSV), and the multi-view setting is defined as multiobject-multi-view (MOMV) in (Nanbo et al., 2020).
Table 1 compares methods related to the MOMV setting. Novel view synthesis stands for prediction of unobserved viewpoints, and novel scene generation stands for random generation of novel scenes. Among these models, ROOTS and MulMON (Nanbo et al., 2020; Chen et al., 2021) are the only generative models that work on a MOMV setting. In addition to this, O3V (Henderson & Lampert, 2020) have a similar problem setting, but it takes fixed number of viewpoints as input. ObSuRF (Stelzner et al., 2021) and uORF(Yu et al., 2021) also work on MOMV setting, but these are not generative models, and are based on Slot Attention (Locatello et al., 2020) and NeRF (Mildenhall et al., 2020). Therefore these methods can not generate novel samples and does not obtain low dimensional representation space, or latent space.
In this paper, we introduce a new MOMV model with a global latent variable, which can potentially represents the spatial arrangement of objects in a scene. The proposed model is the only MOMV model that can generate novel scenes. In this paper, we set MulMON as our baseline model due to the following reasons. Firstly, MulMON uses segmentation masks to represent objects’ regions unlike ROOTS which uses bounding boxes. Bounding box-based methods, which originate from (Eslami et al., 2016) tend to be more sensitive to object size and shape (Engelcke et al., 2021). Secondly, MulMON is more robust in terms of the number of observation views required. Although we built our model based on MulMON, a similar approach can be applied to ROOTS and other methods.
Introducing the global latent variable itself has been done in some studies in various areas (Vasco et al., 2020; Akuzawa et al., 2021). In particular, GNM (Jiang & Ahn, 2020) introduces a global latent variable to bounding box-based single-view object-centric model for random generation. However, how to execute stable inference and training with the proposed MOMV model is unclear, and is thus the points of investigation in this paper.

heading: 3 METHOD
text: Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability.
We achieved this by introducing a global latent variable for global spatial information on top of object latent variables, which is formulated as hierarchical probabilistic model. In this paper, we introduce several crucial components to realize stable inference and training with this MOMV model. An overview of the proposed model is shown in Figure 1.
In this section, we first explain the problem setting in this paper, and then the probabilistic model of our proposed model. Lastly, the training method and additional details of the introduced components are explained.

heading: 3.1 PROBLEM SETTING
text: The target of this research is 3D scenes with multiple objects, and thus a dataset with N data points can be defined as D = {X1, · · · ,XN}, where Xi = {(x1i ,v1i ), ..., (xMi ,vMi )}, which corresponds to one scene withM views, and v is a viewpoint vector with arbitrary dimensions, which are mostly three.
The task is to predict queried unobserved views of the scene given several observations {(x1i ,v1i ), · · · , (x Nobs i ,v Nobs i )}, and infer K representations each of which correspond to an object in the scene Z = {z1, · · · , zK}. In practice, the M views in Xi is split into Nobs observations and Nqry queries in the training process. Nobs can be varied in every iteration for generalization.

heading: 3.2 PROBABILISTIC MODEL
text: Generative Model: As mentioned above, we introduced the hierarchical probabilistic model with global latent zg on top of object-level latent variables z = {z1, · · · , zK}. In the following sections, object-level latent variables are abbreviated as z, as required. The marginal likelihood is thus:
pγ(x | v) = ∫ ∫ pθ(x|z,v) pφ(z | zg) p(zg) dzdzg, (1)
where γ = {θ, φ}. The generative model pθ(x|z,v) is a spatial Gaussian mixture model derived from prior works (Nanbo et al., 2020; Burgess et al., 2019) that can be written as:
pθ(x|z,v) = D∏ i=1 K∑ k=1 pθ(Ci = k | zk,v)⊗ pθ(xik|zk,v), (2)
where D ∈ N is the number of image pixels, and K is the number of mixture components which are often referred to as “slots”. The decoder has parameters θ and implemented by a broadcast decoder (Watters et al., 2019b). zk and v are first concatenated and projected onto a single vector fk by MLP and then fed into the decoder. The decoder generates K components {x1, · · · ,xK} and corresponding segmentation masks from {f1, · · · , fK}. The second term in the r.h.s of Equation 1 pφ(z | zg) is a learnable prior which we refer to as the Structured Prior, and the third term is a standard Gaussian distribution. The Structured Prior is explained in Section 3.3.
Inference Model: We used amortized variational inference (Kingma & Welling, 2013) and computed variational approximate posteriors because the marginal likelihood in Equation 1 is intractable. We factorized the variational posteriors as:
qψ(z, z g | X) = qψ(z1, · · · , zK | zg) qψ(zg | X), (3)
where X = {(x1,v1), . . . , (xNobs ,vNobs)}. The first RHS term and second RHS term are approximate posteriors of z and zg respectively. We refer to these encoders as Sequential Encoder and Global Encoder.
The Global Encoder adopts PoE (Hinton, 2002) to deal with a variable number of input observation viewsNobs. In particular, we used PoE with prior expert (Wu & Goodman, 2018). In addition to this, we applied normalizing flow (NF) (Rezende & Mohamed, 2015) to this inference model. Although this is optional, the posterior should be expressive in order to represent spatial configuration of complex 3D multi-object scenes. Therefore, the variational approximate posterior is:
zg ∼ qψ (zg | X) = ( p(zg1) Nobs∏ v=1 qψ(z g 1 | xv,vv) ) T∏ t=1 ∣∣∣∣det ∂fψ∂zgt ∣∣∣∣−1 , (4)
where T is the number of transformations of NF, and Nobs is the number of available observation views. We used planar flow for its simplicity, but any methods can be applied and can improve the results potentially. Thus, the transform fψ in the above equation is
fψ(zi+1) = zi + ui · h(wT zi + bi).
The Sequential Encoder was implemented in auto-regressive manner using LSTM (Hochreiter & Schmidhuber, 1997). We tried several architectures, but non-auto-regressive models were unstable as long as we have tested. Firstly, this auto-regressive part infers K latent variables z01, . . . , z 0 K and then, these are updated using iterative amortized inference (IAI) (Marino et al., 2018) as MulMON and IODINE (Greff et al., 2019) do. However, we omitted image-sized inputs as in (Emami et al., 2021). Thus, the Sequential Encoder can be defined as:
z01, . . . , z 0 K ∼ K∏ k=1 qψ(z 0 k | z01:k, zg), (5)
zi+1k = z i k + fψ(z i k, z g,∇zikL i IAI), (6)
where ∇zikL i IAI is a gradient of negative log-likelihood in that iteration, and fψ is a refinement network that is implemented by MLP followed by LSTM. The posterior can be updated arbitrary times, and we set L = 5 through this paper. The details ofLiIAI will be explained in the next section. Training: All the parameters in this model can be trained end-to-end by maximizing evidence lower bound (ELBO) on the log-marginal likelihood log p(x1, · · · ,xN ). The ELBO of the proposed model is
LELBO = Eq(z,zg|X) [ log
pθ(x|z,v)pφ(z | zg)p(zg) qψ(z | zg)qψ(zg | X)
] (7)
= Eq(z,zg|X) [log pθ(x | z,v)] −KL [qψ(zg | X) || p(zg)] − Eqψ(zg|X) [KL [qψ(z | z
g) || pφ( z | zg)]] . Here, KL stands for Kullback–Leibler divergence. Then, introducing two coefficient β1 and β2 for balancing KL terms, the objective can be written as
L = Eq(z,zg|X) [log pθ(x | z,v)] − β1KL [qψ(zg | X) || p(zg)]− β2Eqψ(zg|X) [KL [qψ(z | z
g) || pφ( z | zg)]] . (8) Moreover, due to the iterative process of IAI, negative log-likelihood is computed as
Eq(z,zg|X) [log pθ(x | z,v)] = 2
L(L+ 1) L−1∑ i=0 (i+ 1)LiIAI , (9)
LiIAI = Ezi∼q(zi,zg|X) [ log pθ(x|zi,v) ] , (10)
where, L is the number of iterations in IAI. As shown in Equation 9, log-likelihoods in the later steps of iterative refinement are emphasized as in MulMON and IODINE.
In addition to this, we used GECO (Rezende & Viola, 2018) as in some existing methods (Engelcke et al., 2020; Emami et al., 2021). GECO reinterprets the maximization of ELBO as a constrained
optimization problem that takes KL terms as the main objective and log-likelihood as the constraint. This optimization is done by the method of Lagrange multipliers, namely, the coefficient of KL terms β is regarded as a Lagrange multiplier, and automatically tweaked during training. GECO is not necessary for our model, but proper balancing of β further improves random generation quality.
The reinterpreted optimization objective thus becomes
argmin ψ,φ
KL [qψ(z g | xobs,vobs) || p(zg)] + Eqψ(zg|X) [KL [qψ(z | z g) || pφ( z | zg)]] (11)
such that − Eqψ(z,zg|X) [log pθ(x | z,v)] 5 R, (12)
where R ∈ R is the maximum allowed negative log-likelihood, which is a hyperparameter in GECO instead of β. Although all the parameters can be trained end-to-end, we found that end-to-end training including the parameters of Structured Prior φ is slightly unstable and often degrades novel scene generation. Thus, we introduced a separate training strategy, in which φ is updated after the training of the main model. Detail of this method is explained in the next section.

heading: 3.3 STRUCTURED PRIOR
text: In this section, we explain the detail of the Structured Prior and its separate training strategy.
First of all, the Structured Prior is defined as pφ(zL1 , · · · , zLK | zg). As we mentioned in Equation 5, the latent variables of each slots are iteratively updated by IAI in the inference process. However, it makes sampling from the updated posterior zL1 , · · · , zLK difficult. Though IAI updates latent variables using reconstruction error and auxiliary inputs, generative model needs to blindly capture this process without using them.
Therefore, in order to model this process, we implemented Structured Prior as Transformer (Vaswani et al., 2017). Firstly, initial K variables z0k are computed from z
g using the Sequential Encoder qψ(z
0 | zg). Then, Transformer takes the K latent variables as input, and approximates updated ones zLk . Note that the parameter of Sequential Encoder is shared with inference.
Separate Training: As we mentioned above, we introduced a separate training method for the Structured Prior. In a first stage, WeLIS is trained without the parameters of Structured Prior φ.
Namely, φ is frozen and β2 in Equation 7 is set to zero or the second term of Equation 11 is excluded if we use GECO. Then, as a second stage, φ is updated while other parameters are frozen.

heading: 4 EXPERIMENTS
text: We evaluated our model with three different datasets derived from the MulMON paper (Nanbo et al., 2020). CLEVR Multi-View (CLEVR MV) is a multi-view version of the CLEVR dataset (Johnson et al., 2017), and CLEVR Augmented (CLEVR AUG) extends CLEVR MV with more complex and various objects for additional difficulty. GQN Jaco is a simulation based robot arm dataset introduced originally in (Eslami et al., 2018). All the datasets are used in 64× 64 resolution 1, and we trained all the models with Adam optimizer for 300k iterations. We used official implementation of MulMON for reproduction. Up to six views are given as observation during training.
First, we evaluated inference quality quantitatively and show that WeLIS outperforms MulMON in many cases. Then, we show WeLIS can also generate novel scenes, in contrast to baseline models. Lastly, we show ablation results and also investigate how global latent variable zg works. In addition, additional ablation studies, samples, and the results of downstream task are shown in Appendix B.

heading: 4.1 NOVEL VIEW SYNTHESIS AND SEGMENTATION
text: In this section, we evaluate novel view synthesis and segmentation quality. Figure 2 shows quantitative evaluation of three models: WeLIS, WeLIS without NF (ablation) and MulMON. Novel view synthesis is evaluated by Root Mean Squared Error (RMSE), and segmentation is evaluated by mean Intersection over Union(mIoU). Random four seeds are used to calculate these scores. As GQN Jaco does not provide ground truth segmentation, we instead show novel view synthesis results in Figure 3.
We can see that WeLIS outperforms MulMON in CLEVR AUG, GQN Jaco and CLEVR MV with a small number of observation views N . WeLIS is relatively stable in terms of N , which means that WeLIS is good at inferring whole scene from limited, partial observation. Introducing global
1According to the supplemental material, MulMON uses CLEVR AUG in 128 × 128 resolution, but we used in 64× 64 for consistent comparison.

heading: WeLIS w/o Structured Prior WeLIS w/o Normalizing Flow
text: latent in addition to object-level latent variables leads to more precise modeling of the world, thus we consider that it helps inference with small number of observations. As for GQN Jaco (Figure 3), WeLIS captures the balls consistently with clean segmentation.
We also evaluated WeLIS without NF as ablation study. We found that NF is important for stable inference and training. Without it, the model often fall into local minima that does not properly represent one object per slot, as you can see from the relatively large standard error in Figure 2. If the training successfully finished, the model achieved decent performance close to full model.

heading: 4.2 NOVEL SCENE GENERATION
text: In this section, we evaluate novel scene generation results. As shown in Figure 4, WeLIS successfully generates objects and segmentation masks, and as a result, generates physically plausible novel scenes. On the other hand, MulMON generates blurred and spatially ambiguous images. Top row of Figure 5 shows more samples.

heading: 4.2.1 ABLATION STUDIES
text: Bottom row of Figure 5 shows ablation studies of Structured Prior and NF. Firstly, we look into Structured Prior. In this experiment, the results from full model (top left) and ablation study of Structured Prior (bottom left) are generated from same zg . Generated images without Structured Prior are severely blurred, however, we can see that the spatial arrangement of objects itself is similar to full model. This indicates that Structured Prior only contributes to object-level generation quality.
Secondly, bottom right images in Figure 5 shows ablation result of NF. As you can see, the quality is almost same with full model even without NF, which indicates that NF is not essential for generation.
However, as we mentioned in Section 4.1, NF is important for stable inference and training, thus it is also necessary for generation in practice.

heading: 4.2.2 QUANTITATIVE EVALUATIONS
text: We also quantitatively evaluated random generation by Fréchet Inception Distance(FID) score in Table 2. WeLIS outperforms MulMON in every dataset, and as shown in above section, WeLIS without NF achieved similar quality to full model, as long as the training succeeded. Scores without NF in CLEVR AUG and GQN Jaco are degraded because the training fails more frequently as datasets become difficult.
Poor generation quality of MulMON comes from mainly two reasons: insufficient modeling of spatial arrangement and difficulty in sampling actual posterior. In WeLIS, former one is dealt with by introducing global latent zg which represents spatial information of a whole scene, and latter one is dealt with by a Structured Prior which estimates complex posterior inferred from IAI. Effectiveness of introducing zg is evaluated in Section 4.3 and of Structured Prior is already investigated in Figure 5.

heading: 4.3 INSPECTION OF GLOBAL LATENT SPACE
text: To look into the representation obtained by global latent zg , we show random generation results from four different zg (A-D) and four neighborhoods zg + of them in Figure 6. Here, is a small perturbation randomly sampled from standard normal distribution. First column in each group shows images generated from four different zg , and other four columns are generated with different perturbation. Thus, each four group represents a “cluster” around zg .
We can see that generated scenes from same zg have similar spatial structure, while their objects vary. This indicates that the latent space of zg obtains spatial configuration of a scene as we expected.

heading: 5 CONCLUSION
text: We introduced new multi-view object-centric model: WeLIS, which is able to perform multi-view object-centric inference and sampling in 3D scenes. We introduced several components to WeLIS, in order to adapt a global latent variable and to enable inference and training with that model. The global latent variable improved inference quality especially when the observations are limited (Figure 2) and also enabled the model to generate physically plausible novel scenes (Figure 5). We also conducted some ablation studies and investigated the representation obtained by global latent space (Figure 6).
One of the future directions of this research is to sophisticate inference methods: normalizing flow and IAI. There is a room for improvement with optimal choice of normalizing flow. In addition, we used simplified IAI without image-sized input which can reduce computational cost, however there is a possibility that this sacrificed the ability to distinguish similar, ambiguous objects such as rounded cubes and spheres in CLEVR MV dataset.

heading: 6 REPRODUCIBILITY STATEMENT
text: Architecture: Detail of the network architecture is provided in Appendix A. Public version of the source code will be released soon.
Optimization: We derived the training scheme from MulMON in which all models are trained for 300k gradient steps with Adam optimizer. 300k gradient steps is composed of 1000 epochs × 300 scenes. The 300 scenes are randomly picked in each iteration. Learning rate decay in GQN and MulMON is also applied to our model.

heading: A NETWORK ARCHITECTURE

heading: A.1 GLOBAL ENCODER
text: Global Encoder is composed of CNN, viewpoint embedding, feedforward network for reparameterization and PoE. Output from CNN and viewpoint embedding is concatenated and fed into feedforward network.
We used Planar Flow in this encoder. The dimension of the flow is same as the dimension of zg , and the transforms are applied 16 times in this model.

heading: A.2 SEQUENTIAL ENCODER
text: Sequential Encoder is composed of LSTM and linear layer. Input of LSTM is concatenation of zg and z’s parameters inferred in last iteration. Initial input is concatenation of zg and zero vector.

heading: A.3 REFINEMENT NETWORK
text: Refinement Network takes log-likelihood, gradient of the parameters (mean and variance) of zi−1 and the parameters of zg as input, where i means iteration step.

heading: A.4 STRUCTURED PRIOR
text: Structured prior is implemented by Transformer. We used learnable positinal embedding, and activation is GeLU (Hendrycks & Gimpel, 2016) as in BERT (Devlin et al., 2019). We did not use dropout here.

heading: A.5 BROADCAST DECODER
text: We used same decoder as MulMON. v and z are concatenated. Concatenated vector is input to projector (former part of the table), then the output is fed into broadcast decoder (latter part of the table).

heading: B ADDITIONAL RESULTS

heading: B.1 QUANTITATIVE EVALUATION ON OBSERVATION
text: Figure 2 in the main text showed quantitative evaluation for query views. In this section, we show the results for observation views (Figure 7). Namely, this figure is about reconstruction and segmentation quality. The performance of MulMON with small N is relatively not good. This is because MulMON needs to process observation views iteratively, thus the model requires at least several observation views to achieve its best performance. On the other hand, WeLIS is stable against N thanks to the parallel processing of Global Encoder.

heading: B.2 NOVEL VIEW SYNTHESIS AND SEGMENTATION

heading: B.2.1 EVALUATION METRICS AND QUALITATIVE DIFFERENCE
text: WeLIS (RMSE: 0.089 mIoU: 0.58) MulMON (RMSE: 0.093 mIoU: 0.55)

heading: GT Pred. Seg. GT Pred. Seg. GT Pred. Seg. GT Pred. Seg.
text: Figure 8: Qualitative results of novel view synthesis. The number of observation views N is three in this figure. The left most columns in each group show ground truth image that is not shown to the model. The second columns in each group show predicted image. The third columns show segmentation mask.
Figure 8 shows additional quantitative samples of novel view synthesis. This figure helps us to understand how much quantitative difference of RMSE and mIoU actually correspond to the qualitative difference.
In addition, we can see that WeLIS represents background with a fixed slot (yellow), while MulMON represents it with different slots (various colors). This is because the auto-regressive encoder is no longer permutation invariant. It can be beneficial that fixed slot represents background, however, disentanglement might be sacrificed with this. It is difficult to conclude which one is better currently, because this kind of properties should be evaluated by various downstream tasks.

heading: B.3 TRAVERSAL OF GLOBAL LATENT SPACE
text: Figure 9 shows visualization of traversal of global latent space zg . We sample two zg and visualized interpolation of them. The figure shows three different examples. We can see that spatial arrangement is preserved, but individual components changes quickly.

heading: B.4 ABLATION STUDY

heading: B.4.1 NOVEL VIEW SYNTHESIS
text: Figure 10 shows ablation result of Global Encoder (PoE), GECO, separate training of Structured Prior, and Normalizing Flow. Regarding PoE, we replaced PoE with simple average, which is often used in many researches such as NeRF-VAE (Kosiorek et al., 2021).
This result indicates that each component contributes to the scores and training stability.

heading: B.4.2 NOVEL SCENE GENERATION
text: We tested two different architectures for Structured Prior. Both variants use MLP, and one of them updates each slot independently, while the other one considers interaction between slots. As we show in Table 8, there is no significant difference between these variants. This result indicates that zg alone properly models object placement, and Structured Prior does not have to model dependency between slots.

heading: B.5 DOWNSTREAM TASK
text: We evaluated the latent space of zg qualitatively in Figure 6. In this section, we evaluate the obtained representation by downstream task. The task is to predict how many objects are in the observed scenes. We used linear classifier in all the models.
WeLIS can use both z and zg for classification task, thus we tested all three patterns: z only, zg only and both z and zg .

heading: C DATASETS

heading: CLEVR MV CLEVR AUG GQN JACO
text: We show ground truth samples of the three datasets: CLEVR MV, CLEVR AUG and GQN Jaco. These datasets are available from the official implementation of MulMON (https://github. com/NanboLi/MulMON).

Review summaries:
The paper tackles object-centric learning from multiple views. It proposes an extension to MulMON. Concretely, it introduces a global latent variable, aiming at modeling global information like spatial relationships, and predicts object-level latent variables upon the global one by iterative amortized inference. It also uses normalizing flows to stabilize training and inference. The authors compare the proposed model with  MulMON on three simple datasets: CLEVR MV, CLEVR AUG, GQN Jaco.

The paper proposes a new structured generative model of multi-view images, representing their latent decomposition into objects. This model extents MulMON -- a spatial mixture model over multi-view images that is suitable for image decomposition but lacks a prior necessary for synthesis -- by adding such a prior (using an autoregressive model), and making some other architectural changes (e.g. adding a normalising flow posterior instead of gaussian). In the proposed model, a global latent variable conditions the sampling of per-object latents, which in turn parameterise a spatial mixture model for each image. The model is evaluated on novel view synthesis (NVS) and a priori scene synthesis, on simple datasets of renderings (e.g. multi-viewpoint CLEVR); it is shown to out-perform a baseline (MulMON) on several metrics for both tasks quantitatively, and qualitative results are also displayed.

This paper presents a hierarchical, object centric view conditional generative model for scenes. The latent structure includes a global latent variable which captures general scene structure/configuration and conditions a set of slotted latents, each one capturing a single object in the scene. An inference method is proposed - a product of expert + flow encoder for the global, and a sequential encoder + iterative refinement for the slots. Everything is view conditional so training requires camera information, but novel views may be synthesized at ease. The model is demonstrated to work on CLEVR and a slightly more complex version of CLEVR with more object structures as well as the Jaco arm dataset.

The paper proposes a model for inferring object-centric scene representations from multiple views that explicitly factorizes the representation into global scene properties and objects.  The model builds upon previous object-centric VAEs and is capable of generating novel views of a scene. The main contribution of the paper is the global-local factorization of the scene representation together with the architectural changes and optimization tricks required to train a VAE with the proposed latent structure.