Learning Global Spatial Information for Multi-View Object-Centric Models
Authors: Yuya Kobayashi
Masahiro Suzuki
Yutaka Matsuo
Venue: ICLR 2022 Submitted (2022)
Keywords: deep generative models
object-centric representation learning
segmentation

Abstract: Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views. In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are specified; however, existing multi-view object-centric models only infer object-level representations and lack spatial information. This insufficient modeling can degrade novel-view synthesis quality and make it difficult to generate novel scenes. We can model both spatial information and object representations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables. However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear. Therefore, we introduce several crucial components which help inference and training with the proposed model. We show that the proposed method achieves good inference quality and can also generate novel scenes.

Under review as a conference paper at ICLR 2022
LEARNING GLOBAL SPATIAL INFORMATION FOR MULTI-VIES OBJECT-CENTRIC MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, several studies have been working on multi-view object-centrie models, which predict unobserved views of a scene and infer object-centric representations from several observation views. In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are specified; however, existing multi-view object-centric models only infer object-level representations and lack spatial information. This insufficient modeling can degrade novel-view synthesis quality and make it difficult to gen- erate novel scenes. We can model both spatial information and object represen- tations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables. However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear. Therefore, we introduce several crucial components which help infer- ence and training with the proposed model. We show that the proposed method achieves good inference quality and can also generate novel scenes.
INTRODUCTION
Extracting representations of individual objects from direct perception (e.g., visual input) is called object-centric representation learning. It is considered be effective in many aspects such as robotics (Devin et al. 2018 Veerapaneni et al. 2019 Kulkarni et al., 2019), reasoning (Ding et al.
2021). and sample efficiency (Watters al.) 2019a). Among them, the deep generative model (DGM)-baaed method is one of the promising directions, as it can infer rich information about objects in fully unsupervised manner, and some of its models can also generate novel scenes (Gr-
eff et al.
et al. [2200] Burgess et (019 Greff et al. 2019 Engelcke et al.) 2020
Eslami et al. | Crawford & Pineau| 2019 Jiang & Ahn 2020).
Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of scene: novel view synthesis, and infer object-centric representations from several observation views (Nanbo al. 2020 Chen et al. (021). This can be also regarded as an object-centric expansion of the Generative Query Network (GQN) (Eslami et al.] 2018). In order represent multi-object scenes, the properties of individual objects and the spatial arrangement of objects should be specified. However, existing multi-view object-centric methods only explicitly model representations of individual objects, and each of them includes their own spatial information separately. Namely, relationship between objects is not represented in this case. This modeling can degrade novel view synthesis and segmentation, because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects' spatial relationship. This can be serious especially when the number of available observation views is limited. In addition, this modeling can also lead to inability to generate physically plausible novel scenes, because they need to place objects independently, which often brings about collisions and misplacement of objects.
Spatial information of the whole scene can be represented by introducing another latent variable be- sides object-level latent variables. This variable can be independent of object-level latent variables, or can be modeled as global latent on top of object-level latent variables. In the former formula- tion, those latent variables need to be disentangled during training, which can often be challenging. From another point of view, it is common assumption in cognitive science that humans recognize the whole and parts, or objects, separately (Schmidt (009). In feature integration theory (Treisman) & Gelade 1980), visual feature maps become "master map of locations" combined with positions of

---

Under review as a conference paper at ICLR 2022
novel prediction sagmentation
nccodar
Product of Experts (PoE)
shared
global latent variable
Spatial Gaussian Mixture
Gulce 1x)
Structured Prior
x2
20
Zk
query viewpoint
shondd
Iterative Amorüzed Inference
22
22
K slots
Encoder
x3
Sequential Encoder
Global Encoder
Figure 1: Overview of the proposed model. In this example, the model predicted an image and corresponding segmentation masks from unobserved query viewpoint v9, using three observation views x x'-x3. z is global latent variable that represents spatial relationship between objects, and Zy is object-level latent variables that represent each object. Note that the 2 and segmentation shown in this figure are the actual results from our model.
objects in visual space. Then, attention binds those features to recognize individual objects. Model- ing the spatial arrangement of objects as global latent variable closely adheres to this point of view due to its hierarchical property. For the above reasons, we believe that introducing global latent variable more natural step for improving object-centric generative models.
In this paper, we build new multi-view object-centric model which has latent variable for global spatial information to improve inference: novel view synthesis and segmentation. addition, our model also generate physically plausible novel scenes. The problem is that how execute inference and training with the proposed model is unclear. In regards to this point, we propose several crucial components: Global Encoder, Sequential Encoder and Structured Prior. Firstly, the model needs to infer global representation from variable number of input views, thus we introduced an encoder using Product of Experts (PoE) (Hinton 2002), which we refer
Global Encoder.
Secondly, the structured, complex posterior of object- level latent variables need to be sampled in- cluding spatial relationship between objects in order generated physically plausible novel scenes. Therefore, we introduced learnable prior called Structured Prior, which is implemented by Trans- former (Vaswani et al. 2017). Lastly, we introduced an encoder using auto-regressive model called Sequential Encoder. This stabilize inference and training with the hicrarchical probabilistic model of the proposed method.
The proposed model has representation about the whole, spatial information of scene and has representations about a part of scene (i.e., individual properties of objects); hence, we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and Sampling (WeLIS).
Contributions:
We introduce a multi-view object-centric model with a global latent variable which repre- sents the spatial configuration of objects in scene.
We introduce essential components for introducing global latent variable as well as stable learning and inference for our multi-view object-centric model.
We show that the proposed model performs well in terms of inference quality, and can gen- erate novel scenes and corresponding segmentation masks properly, unlike existing meth- ods.
2

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of multi-ohjjc--muliiview (MOMV) related methods.
Novel View Synthesis
Sampling
Model
Object-Centric
Inference
(Novel Scene Generation)
GQN
x
Kosiorek et al.
2021
ObSuRF, uORF
ROOTS
V
MulMON
V
X
WeLIS (Ours)
'These methods are not generative models.
MulMON can generate objects independently, but this leads to ambiguous, blurred image.
RELATED WORK
The objective of object-centric representation learning is to obtain rep- resentations of each object from visual input. applicable to many domains such as robotics (Devin al. 2018 Florence et al.) 2019 Veerapaneni et al., 2019), reasoning (Ding et al. 2021) and reinforcement learning (Watters et al. 2019a). One of the promising directions of object-centric representation learning variational autoencoder (VAE)-based (Kingma & Welling 2013) methods, as they can obtain rich information about objects that can also be disentangled, and some of them generate novel scenes outside of the empirical distribution (Burgess al. 2019 Greff al.,
Object-Centric Models:
2019 Engelcke et al., 2020; Eslami et al., 2016) Crawford & Pineau [2019] Jiang & Ahn] (020). often referred to "scene interpretation models" due their inference ability. Some models are capable of object-centric generation (Dai et al. [0119] Nguyen-
VAE-based object-centric models
Phuoc al. 2020 Niemeyer & Geiger 202T). however, these methods focus on generation and not for inference.
Multi- View Object-Centric Models: Lately, several studies have been working on multi-view scene interpretation models (Nanbo et al. 2020 Chen et al., (021). These methods predict images of a 3D scene from arbitrary viewpoints an object-centric way, using several observation views as input. This problem setting can be regarded as an object-centric expansion of GQN family (Eslami) (ct al. 2018: Tobin et al. 2019: Kosiorek et al. [202I). The above mentioned single-view setting (MOSV), and the multi-view setting is defined as multi- object-multi-view (MOMV) in (Nanbo et al., 2020).
referred to muuli-object-singlee
Table Table]compee compares methods related to the MOMV setting. Novel view synthesis stands for prediction of unobserved viewpoints, and novel scene generation stands for random generation novel scenes. Among these models, ROOTS and MulMON (Nanbo al. 2020 Chen et al. (021) the only generative models that work on MOMV setting. In addition to this, O3V (Henderson & Lampert 2020) have similar problem setting, but it takes fixed number of viewpoints
input. ObSuRF (Stelzner et al. 2021)) and uORF et al. (202I) also work on MOMV setting, but these are not generative models, and based on Slot Attention (Locatello al. 2020) and NeRF (Mildenhall et al., 2020). Therefore these methods can not generate novel samples and does not obtain low dimensional representation space, or latent space.
In this paper, we introduce a new MOMV model with global latent variable, which can potentially represents the spatial arrangement objects in a scene. The proposed model is the only MOMV model that can generate novel scenes. In this paper, we set MulMON as our baseline model due to the following reasons. Firstly, MulMON uses segmentation masks to represent objects' regions unlike ROOTS which uses bounding boxes. Bounding box-based methods, which originate from Eslami et al. 2016) tend to be more sensitive to object size and shape (Engelcke et al., 2021). Secondly, MulMON more robust in terms of the number of observation views required. Although we built our model based on MulMON, similar approach can applied to ROOTS and other methods.

---

Under review as a conference paper at ICLR 2022
Introducing the global latent variable itself has been done in some studies in various areas (Vasco et al.] 2020, Akuzawa et al. 202I). In particular, GNM @Jang & Ahn 2020) introduces a global la- tent variable bounding box-based single-view object-centric model for random generation. How- ever, how to execute stable inference and training with the proposed MOMV model is unclear, and is thus the points of investigation in this paper.
3 METHOD
Our objective was to introduce MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation abil- ity.
We achieved this by introducing a global latent variable for global spatial information on top of object latent variables, which is formulated as hierarchical probabilistic model. In this paper, we in- troduce several crucial components to realize stable inference and training with this MOMV model. An overview of the proposed model is shown in Figure
In this section, we first explain the problem setting this paper, and then the probabilistic model of our proposed model. Lastly, the training method and additional details of the introduced components are explained.
3.1 PROBLEM SETTING
The target of this research is 3D scenes with multiple objects, and thus dataset with N data points can be defined as D = {X1,'.- .NN), J. where X; = {(x], v!), v.......(((.. (x.M, v. ]. which corresponds to one scene with M views, and v is viewpoint vector with arbitrary dimensions, which are mostly three.
The task is to predict queried unobserved views of the scene given several observations {(x), v.),.... (x." (x""oo, vi NNooe)), }}, and infer K representations each of which correspond ob- ject in the scene Z [Zi,". ... 2x} }. In practice, the M views in X; is split into Nots observations and Ngry queries in the training process. Nots can be varied in every iteration for generalization.
3.2 PROBABILISTIC MODEL
Generative Model: As mentioned ahove, we introduced the hierarchical probabilistic model with global latent z top of object-level latent variables {z,,... 2xx] In the following sections, object-level latent variables are abbreviated as z, as required. The marginal likelihood is thus:
/
P (x
pe (x|z v) (z 2²) p(z²) dzdz".
where 7 = {0,0].
Φ). The generative model po (x/2. v) is spatial Gaussian mixture model derived from prior works (Nanbo et al. 2020 Burgess et al. 2019) that can be written as:
D K
po v) = II
po = 1: 2k, v) Ø po (Zix Me, v);
(=1
1==1
where D N is the number of image pixels, and K is the number of mixture components which are often referred to as "slots". The decoder has parameters 0 and implemented by a broadcast decoder (Watters al., (201b)). zž and are first concatenated and projected onto a single vector fr by MLP and then fed into the decoder. The decoder generates K components (x1,". xx) and corresponding segmentation masks from (f.....,xx
The second term in the r.hss of Equation p z²) is learnable prior which we refer to as the Structured Prior, and the third term is standard Gaussian distribution. The Structured Prior is explained in Section 3.3
We used amortized variational inference (Kingma & Welling 2013) and com- puted variational approximate posteriors because the marginal likelihood in Equation is intractable. We factorized the variational posteriors as:
Inference Model:
Tr (2, 28 X) qu (21, ... _K 29) qo (x X),
(3)
4

---

Under review as a conference paper at ICLR 2022
where X { vi)..... (XN VN )}. The first RHS term and second RHS term approx- imate posteriors of z and 2 respectively. We refer to these encoders as Sequential Encoder and Global Encoder.
The Global Encoder adopts PoE (Hinton 2002) to deal with a variable number of input observation views Nots- In particular, used PoE with prior expert (Wu & Goodman 2018). In addition to this, we applied normalizing flow (NF) (Rezende Mohamed] 2015) to this inference model. Although this optional, the posterior should be expressive in order to represent spatial configuration of complex 3D multi-object scenes. Therefore, the variational approximate posterior is:
Nobe
T
a Oz
2 ~ qu (z
X) =
p(zi)
II 1
qol (z)
Xy. Vo)
det II
(4)
where T is the number of transformations of NF, and Nobe is the number of available observation views. We used planar flow for its simplicity, but any methods can be applied and can improve the results potentially. Thus, the transform fø in the above equation is
(0(2)))) 2, u, h(w" 2, b;).
The Sequential Encoder was implemented in auto-regressive manner using LSTM (Hochreiter & Schmidhuber 1997). We tried several architectures, but non-auto-regressive models were unstable as long as we have tested. Firstly, this auto-regressive part infers K latent variables z!..... z&x and then, these are updated using iterative amortized inference (IAI) (Marino et al.] (2018) MulMON and IODINE (Greff et al.) (219)) do. However, we omitted image-sized inputs as in (Emami et al. 2021). Thus, the Sequential Encoder be defined
2.. 2K
II k=
(o 22 zi. x²),
(5)
z/ z + fz 2", Vxj Ciar):
where Vx; Lf is a gradient of negative log-likelihood in that iteration, and So is a refinement network that is implemented by MLP followed by LSTM. The posterior can be updated arbitrary times, and we L 5 through this paper. The details of a will be explained in the next section.
Training: All the parameters in this model can be trained end-to-end by maximizing evidence lower bound (ELBO) on the log-marginal likelihood log p(x1, xN). The ELBO of the proposed model is
CELBO = |)()x))))) log (x(x v v)po(z Zd p(()
(z z z 22 X)
= Ex(2,2211) [log (x z, v)]
KL (z" X) p(z")] Egg (=*|x) [KL (g 12") po(2 zz)]]-
Here, KL stands for Kullback-Leibler divergence. Then, introducing two coefficient 31 and 32 for balancing KL terms, the objective can be written as
L = Eq(x. xx) [log Pa v)]
3 KL (z"
X)
pzz")]
B2 go (wIIx) [KL [z 2²) || Po [z] z²)]].
(8)
Moreover, due to the iterative process of IAI, negative log- likelihhodd is computed
L-1
2
Eg(x,20 x) [log (x v)]
Σ
(i+ (i+1))
(9)
L(L + 1)
C Exinggt Sotiiggooee xx) [log [o v)].
(10)
where, the number of iterations in IAL. As shown in Equation log- ligeliiis in the later steps of iterative refinement emphasized as in MulMON and IODINE.
In addition to this, we used GECO (Rezende & Viola 2018) as in some existing methods (Engelcke
al. 2020: Emami al. 2021). GECO reinterprets the maximization of BLBO constrained
5

---

Under review as a conference paper at ICLR 2022
214
0.32
PASSE
6.0
0.03
040
0:15
3.3
Ours Ours: wi Flow MUMMON (bassline)
045
375
078
the number deservalions
Figure 2: Quantitative evaluation of novel view synthesis. Lower is better for root mean squared eMSor()MSH), and higher is better for mean intersection over union (mloU). There is no mloU for GQN Jaco dataset because does not provide ground truth segmentation masks.
GT
Pred.
Bup.
GT
Pred
Seg.
Pred.
Seg.
QT
Pred.
Seg.
WelLs
MuIMON
Figure 3: Novel view synthesis results on GQN Jaco with four difference scenes. These images are prediction of unobserved query views. The number of observation views is three (N 3).
optimization problem that takes KL terms as the main objective and log- ligeliiel as the constraint. This optimization is done by the method of Lagrange multipliers, namely, the coefficient of KL terms regarded as a Lagrange multiplier, and automatically tweaked during training. GECO necessary for model, but proper balancing of B further improves random generation quality.
The reinterpreted optimization objective thus becomes
argmin KL (z" Xober Vobs) || p|z"]] + (x2 JL (z 2²)) (z z²)]] 19.0
(11)
such that
F. (2.2 px 2, v) S R.
(12)
where R R is the maximum allowed negative log-likelihood, which is a hyperparameter in GECO instead of B. Although all the parameters can trained end-to-end, we found that end-to-end training including the parameters of Structured Prior á is slightly unstable and often degrades novel scene generation. Thus, we introduced separate training strategy, which ø is updated after the training of the main model. Detail of this method is explained in the next section.
3.3 STRUCTURED PRIOR
In this section, explain the detail of the Structured Prior and its separate training strategy.
First all, the Structured Prior defined as (zf ... z 29). As we mentioned Equation the latent variables of each slots are iteratively updated by IAI in the inference process. However, makes sampling from the updated posterior 2f ... 2 difficult. Though LAI updates latent variables using reconstruction error and auxiliary inputs, generative model needs to blindly capture this process without using them.
Therefore, in order model this process, implemented Structured Prior Transformer VVawaani et al. 2017). Firstly, initial K variables z are computed from 2 using the Sequential Encoder Mo zT Then, Transformer takes the K latent variables as input, and approximates updated L. Note that the parameter of Sequential Encoder is shared with inference.
Separate Training: As we mentioned above, we introduced a separate training method for the Structured Prior. In first stage, WeLIS is trained without the parameters of Structured Prior ø

---

Under review as a conference paper at ICLR 2022
Gen.
Slots
Seg.
View
WeLIs (Ours)
View
MuIMON (Baseline)
View
View
==1
2
k 3
5
=
Figure 4: coapneentt-byoooooeenn random generation results of WeLIS and the baseline. Each row corresponds to a different query viewpoint. The first column shows generated images, and the next seven columns show each slot (Xk, k ( K)), while last column shows generated segmentation masks.
Namely, @ frozen and 32 in Equation is set to zero the second of Equation 11 is excluded if we use GECO. Then, as second stage, ø is updated while other parameters are frozen.
4 EXPERIMENTS
We evaluated our model with three different datasets derived from the MulMON paper (Nanbo et al. 2020). CLEVR Multi View (CLEVR MV) is multi-viww version the CLEVR dataset (Johnson al. 2017), and CLEVR Augmented (CLEVR AUG) extends CLEVR MV with more complex and various objects for additional difficulty. GQN Jaco is a simulation based robot arm dataset introduced originally in (Eslami al. 2018). All the datasets are used 64 x 64 resolution and we trained all the models with Adam optimizer for 300k iterations. We used official implementation of MulMON for reproduction. Up six views are given as observation during training.
First, we evaluated inference quality quantitatively and show that WeLIS outperforms MulMON in many cases. Then, we show WeLIS can also generate novel scenes, in contrast baseline models. Lastly, we show ablation results and also investigate how global latent variable z works. In addition, additional ablation studies, samples, and the results of downstream task are shown in Appendix B
4.1 NOVEL VIEW SYNTHESIS AND SEGMENTATION
In this section, we evaluate novel view synthesis and segmentation quality. Figure 2 shows quan- titative evaluation of three models: WeLIS, WeLIS without NF (ablation) and MulMON. Novel view synthesis evaluated by Root Mean Squared Error (RMSE), and segmentation is evaluated by mean Intersection over Union(mloU). Random four seeds are used to calculate these scores. As GQN Jaco does not provide ground truth segmentation, we instead show novel view synthesis results in Figure
We can see that WeL.I outperforms MulMON in CLEVR AUG, GQN Jaco and CLEVR MV with small number of observation views N. WeLIS is relatively stable in terms of N, which means that WeLIS is good at inferring whole scene from limited, partial observation. Introducing global
According to the supplemental material, MulMON uses CLEVR AUG in 128 x 128 resolution, but used 64 64 for consistent comparison.

---

Under review as a conference paper at ICLR 2022
Wel
MuNON (Bacaliro)
Nornaling
Figure 5: Random generation results from WeLIS and MulMON(basellnn) The top row shows results from WeLIS and MulMON. The bottom row shows WeLIS without Structured Prior and without normalizing flow for ablation study. We generated these images by randomly sampling eight by for WeLIS, and by sampling each slot independently for MulMON. Note that the same set of 2g is used in the full model(top left) and in the ablation of the Structured Prior(bottom left).
Table Evaluation of FID score on three models: WeLIS(Ours), WeLIS without Normalizing Flow and MulMON, Lower better.
Datasets
WeLIS (Ours)
WeLIS w/o NF
MulMON
CLEVR MV
125.8 +1.7
117.2 +5.9
157.9 #8.4
CLEVR AUG
83.3 =1.7
134.2 -25.8
168.6 #5.0
GQN Jaco
251.7 42.1
262.1 +11.0
273.7 +10.4
If we exclude the case where training did not converge, the score becomes 90.0 1.3.
latent in addition to object-level latent variables leads to more precise modeling of the world, thus consider that it helps inference with small number of observations. As for GQN Jaco (Figure 3), WeLIS captures the balls consistently with clean segmentation.
We also evaluated WeLIS without NF as ablation study. We found that NF important for stable inference and training. Without it, the model often fall into local minima that does not properly represent one object per slot, as you can from the relatively large standard error in Figure 2 If the training successfully finished, the model achieved decent performance close to full model.
4.2 NOVEL SCENE GENERATION
In this section, we evaluate novel scene generation results. As shown in Figure 4 WeLIS success- fully generates objects and segmentation masks, and as a result, generates physically plausible novel scenes. On the other hand, MulMON generates blurred and spatially ambiguous images. Top row of Figure 5 shows more samples.
4.2. ABLATION STUDIES
Bottom row of Figure shows ablation studies of Structured Prior and NF. Firstly, we look into Structured Prior. In this experiment, the results from full model (top left) and ablation study of Structured Prior (bottom left) are generated from same 2g. Generated images without Structured Prior are severely blurred, however, similar to full model. This indicates that Structured Prior only contributes to object-level generation quality.
that the spatial arrangement of objects itself is
Secondly, bottom right images in Figure 5 shows ablation result of NE. As you can see, the quality is almost same with full model even without NE, which indicates that NF is not essential for generation.
8

---

Under review as a conference paper at ICLR 2022
Iateni space of,
Figure 6: Random generation results from four different 2g (A-D) and four neighborhoods Zg + C, where с Gaussian noise as perturbation. Each row corresponds columns of each group show the original Zg. and other columns show different perturbations e. Therefore, each row represents "cluster" of each zn. We can see that images in the same row share similar spatial structure. The schematic of the latent space the right side is just for illustration purposes.
different zg. The left most
However, as we mentioned in Section 4.1 NF is important for stable inference and training, thus it also necessary for generation in practice.
4.2.2 QUANTITATIVE EVALUATIONS
We also quantitatively evaluated random generation by Fréchet Inception Distance(FID) in Table 2 WeLIS outperforms MulMON in every dataset, and as shown in above section, WeLIS without NF achieved similar quality full model, long the training succeeded. Scores without NF in CLEVR AUG and GQN Jaco are degraded because the training fails more frequently as datasets become difficult.
Poor generation quality of MulMON comes from mainly two reasons: insufficient modeling of spa- tial arrangement and difficulty in sampling actual posterior. In WeLIS, former one dealt with by introducing global latent 2g which represents spatial information of a whole scene, and latter one dealt with by a Structured Prior which estimates complex posterior inferred from IAL. Effective- ness of introducing 2g is evaluated in Section 4.3 and of Structured Prior is already investigated in Figure
4.3 INSPECTION OF GLOBAL LATENT SPACE
To look into the representation obtained by global latent Zg. show random generation results from four different 2, (A-D) and four neighborhoods 2g + 6 of them in Figure 6 Here, e is small perturbation randomly sampled from standard normal distribution. First column in each group shows images generated from four different Zg, and other four columns are generated with different perturbation. Thus, each four group represents "cluster" around Zg.
We can see that generated scenes from same by have similar spatial structure, while their objects vary. This indicates that the latent space of zg obtains spatial configuration of scene as we expected.
CONCLUSION
We introduced new multi-view object-centric model: WeLIS, which is able to perform multi-view object-centric inference and sampling in 3D scenes. We introduced several components to WeLIS, in order to adapt global latent variable and to enable inference and training with that model. The global latent variable improved inference quality especially when the observations are limited (Fig- ure 2 and also enabled the model to generate physically plausible novel scenes (Figure 5). We also conducted some ablation studies and investigated the representation obtained by global latent space (Figure (6).
One of the future directions of this research is to sophisticate inference methods: normalizing flow and IAL. There is a room for improvement with optimal choice of normalizing flow. In addition, used simplified IAI without image-sized input which can reduce computational cost, however there is a possibility that this sacrificed the ability to distinguish similar, ambiguous objects such as rounded cubes and spheres in CLEVR MV dataset.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Architecture:
Detail the network architecture provided in Appendix A Public version of the source code will be released soon.
Optimization:
We derived the training scheme from MulMON in which all models are trained for 300k gradient steps with Adam optimizer. 300k gradient steps is composed of 1000 epochs 300 scenes. The 300 scenes are randomly picked in each iteration. Learning rate decay in GQN and MulMON is also applied to our model.
REFERENCES
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Information-theorete regularization for learn- ing global features by sequential vae. Machine Learming, pp. 1-28, 2021.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv: 1ttp://1111/1111 /3900, 2019.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes, 2021.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 1422-3420, 2019.
Songmin Dai, Xiaoqiang Li, Lu Wang, Pin Wu, Weigin Tong, and Yimin Chen. Learning segmen- tation masks with the independence prior. 33:3429 3436, Jul. 2019. doi: 10.1609/ammi.v33101 33013429. UtLhhtts::////t/ aaai, org http://wwwppp///aaaaatttta//////
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representa- tions for generalizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 71 7111-71 18, 2018. doi: N.1109/ICRA.200 2.020.18000 8.84611999
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 18655//11/119-4225 URL https: www. ac www..wwwb....mmm owg. httpg///athlooo//119999 1423
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. 2021.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized infer- ence for learning symmetric and disentangled multi-object representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, vol- ume 139 of Proceedings of Machine Learning Research, pp. URL http:://proceeeinggs www.hnsss...rr.....mmm...m..mm.. wtw..r.presss..vv.e.eem ema.mm/mmmmmmmmmmmm.mmm 21a html
PMLR, 18-24 Jul 2021.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. International Confer- on Learning Representations, 2020.
Martin Engeleke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2 (0..v:221440099:0 2021.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari. Geoffrey E Hin- ton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
10

---

Under review as a conference paper at ICLR 2022
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 36046394) 2018.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492 499, 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximiza- tion. I. Guyon, U. V. Luxburg. S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL /Rtppttpss//pprreee....gg neeripp http:/pss.cc/pppar//////ELL/L
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.004500 2019.
Paul Henderson and Christoph H Lampert. Unsupervised object-centric video generation and de- composition in 3d. arXiv preprint arXiv:2007.06705, 2020.
Dan Hendrycks and Kevin Gimpel. arXiv:1606.08411 08415, 2016.
Gaussian error linear units (gelus). arXiv preprint
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1771)1177 2002.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, November 1997. ISSN 0899-7667. doi: 10. 22.166/neco.199 199.1999997...7... 1735. URL
nttps://abi.org//1 www./1666cccooo 1997. 9.8.1735
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Informa- fion Processing Systems, 33, 2020.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Diederik Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding
variational
bayes.
arXiv preprint
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokra, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learming Research, pp. 5742-5752. PMLR, 18-24 Jul 2021. URL https::/ppoceeee
mll..press/1139/keeee
html
Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised Learning of Object Keypoints for Perception and Control. Curran Associates Inc., Red Hook, NY, USA, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint urXiv:2006. 1 15055, 2020.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. arXiv preprint arXiv:1807.09356, 2018.
Ben Mildenhall, Pratul P. Srinivasan. Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.

---

Under review as a conference paper at ICLR 2022
Li Nanbo, Cian Eastwood, and Robert B Fisher. Learning object-centric representations of multi- object scenes from multiple views. 34th Conference on Neural Information Processing Sys- tems, 2020.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. In Advances in Neural Information Processing Systems 33, Nov 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL html
mlr wtt.pprre//
Danilo Jimenez Rezende and Fabio Viola. Taming vaes, 2018.
T. Schmidt. Perception: The binding problem and the coherence of perception. In William P. Banks (ed.), Encyclopedia of Consciousness, pp. 147-158. Academic Press, Oxford, 2009. ISBN
10009--. URL https: // htt.://cwiccaccccccooooeooooeeea
978-0-12-373873-8. doi:
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
Joshua Tobin, Wojciech Zaremba, and Pieter Abbeel. Geometry-aware neural rendering. Advances in Neural Information Processing Systems, 32:1 1559-1 11569, 2019.
Anne M. Treisman and Garry Gelade. A feature-integration theory of attention. Cog- nitive Psychology, 12(1):97-136, 1980. ISSN 0010-0285. doi: https://oi.org/) 1r0
101600000-02855800900055555 URL https: httpps://www. /wwww//www///////////////w. sci encedi rect. com/act ence/
Miguel Vasco, Francisco S. Melo, and Ana Paiva. Mhvae: human-inspired deep hierarchical generative model for multimodal representation learning, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 170.0077... 2017.
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B. Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In CoRL, 2019.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. COBRA: data-efficient model-based RL through unsupervised object discovery and curiosity- driven exploration. CoRR, abs/1995.09925, 2019a. URL
09275
Nicholas Watters, Loie Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broad- cast decoder: A simple architecture for learning disentangled representations in vaes. CoRR, abs/1990.07001, 2019b.
Mike Wu and Noah Goodman. Multimodal generative models for scalable supery-s learn- ing. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5580-5590, 2018.
Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields, 2021.
12

---

Under review as a conference paper at ICLR 2022
A NETWORK ARCHITECTURE
A.1
GLOBAL ENCODER
Type
Size/Channels
Stride
Activation
Type
Size
Activation
Input
Input
Conv 3x3
Linear
128
ReLU
Conv 3x3
32
Linear
embed_dim
None
2
ReLU
Conv 3
64
Conv 3x3
Linear Linear
256
Swish None
concatenation of above outputs reparameterization
z
Product of Experts
Table 3: Architecture of Global Encoder. This table contains convolution network (top left), view- point embedding (top right), feedforward network.
Global Encoder composed of CNN, viewpoint embedding, feedforward network reparameter- ization and PoE. Output from CNN and viewpoint embedding is concatenated and fed into feedfor- ward network.
We used Planar Flow in this encoder. The dimension of the flow is same the dimension of 2 and the transforms are applied times in this model.
A.2 SEQUENTIAL ENCODER
Type
input dim
hidden dim (output dim)
Activation
Comment
LSTM
z x + 2
256 x 2
Linear
256
None
parameters of z
Table 4: Architecture of Sequential Encoder.
Sequential Encoder composed of LSTM and linear layer. Input of LSTM is concatenation of z and z parameters inferred in last iteration. Initial input is concatenation of z and zero vector.
A.3
REFINEMENT NETWORK
Type
Size
Activation
Comment
Input
z + 2 x
Linear
256
ELU
Linear
256
None
LSTTM
32 (z x 2)
Table 5: Architecture of Sequential Encoder.
Refinement Network takes log- likeliik,hoodd, gradient of the parameters (mean and variance) of z and the parameters of zd as input, where i means iteration step.
13

---

Under review as a conference paper at ICLR 2022
A.4
STRUCTURED PRIOR
Type
size (model dim)
Activation
Comment
Linear
256
ELU
positional encoding
Feed forward: 1024
Transformer
256
GeLU
Layers: 3
head: 8
Linear
z X 2
None
reparameterization
Table 6: Architecture Structured Prior.
Structured prior is implemented by Transformer. We used learnable positinal embedding, and ac- tivation is GeLU (Hendrycks & Gimpel 2016) in BERT (Devlin al. 2019). We did not use dropout here.
A.5 BROADCAST DECODER
Type
Size/Channels
Activation
Comment
Input
2 + 512
Linear
ReLU
Linear
z
viewpoint projector
Broadcast
Z + 2
Conv 3
32
ReLU
Conv x 3
32
ReLU
Conv 3
32
ReLU
Conv 3 x 3
32
ReLU
Conv 1
4
ReLU
RGB ch. mask logit
Table 7: Architecture of Broadcast Decoder.
used same decoder as MulMON, and are concatenated. Concatenated vector is input to projector (former part of the table), then the output is fed into broadcast decoder (latter part of the table).
14

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL RESULTS
B. QUANTITATIVE EVALUATION ON OBSERVATION
CLEVR
CLEVR AUG
GQN
0.122
0.115
0.115
0.94
0.065
0.5
0.25
Ours Ours: wio Flow MuIMON (baseline)
245
0.52
055
number
Figure 7: Quantitative evaluation of novel view synthesis. This figure shows scores about observa-
tion views, namely, reconstruction and segmentation.
Figure 2 in the main text showed quantitative evaluation for query views. In this section, we show the results for observation views (Figure 7). Namely, this figure is about reconstruction and segmen- tation quality. The performance of MulMON with small N is relatively not good. This is because MulMON needs to process observation views iteratively, thus the model requires at least several observation views to achieve its best performance. On the other hand, WeLIS is stable against N thanks to the parallel processing of Global Encoder.
B.2
NOVEL VIEW SYNTHESIS AND SEGMENTATION
B.2.1
EVALUATION METRICS AND QUALITATIVE DIFFERENCE
Pred.
Seg.
Pred.
Seg.
Pred.
Seg.
GT
Pred.
Seg.
WeLIS (RMSE: 0.089 mloU: 0.58)
MulMON (RMSE: 0.093 mloU: 0.55)
Figure 8: Qualitative results of novel view synthesis. The number observation views is three in this figure. The left most columns in each group show ground truth image that is not shown to the model. The second columns in each group show predicted image. The third columns show segmentation mask.
Figure shows additional quantitative samples of novel view synthesis. This figure helps to derstand how much quantitative difference RMSE and mloU actually correspond to the qualitative difference.
15

---

Under review as a conference paper at ICLR 2022
In addition, that WeLIS represents background with fixed slot (yellow), while MulMON represents it with different slots (various colors). This is because the auto-regressive encoder is no longer permutation invariant. It be beneficial that fixed slot represents background, however, disentanglement might be sacrificed with this. It difficult to conclude which one better currently, because this kind of properties should be evaluated by various downstream tasks.
B.3 TRAVERSAL OF GLOBAL LATENT SPACE
sample A
sample B
Figure 9: Traversal of global latent space
Figure 9 shows visualization of traversal of global latent space 29, We sample two 2 and visu- alized interpolation of them. The figure shows three different examples. We can see that spatial arrangement is preserved, but individual components changes quickly.
B.4
ABLATION STUDY
B.4.1
NOVEL VIEW SYNTHESIS
CLEVR
RMSE
mloUt
0.15
0.1
odservation
03333
0.35
0.39
64
0.35
0.28
03355
03
0.155
089
035
0335
quety
wel tox
5066
04
mode Imean)
0.35
5015
022
Figure 10: Ablation study for each component: Global Encoder (PoE), GECO, separate training of Structured Prior
Figure 10 shows ablation result of Global Encoder (PoE), GECO, separate training of Structured Prior, and Normalizing Flow. Regarding PoE, we replaced PoE with simple average, which is often used in many researches such NARF-VAE (Kosiorek al. 2021).
16

---

Under review as a conference paper at ICLR 2022
This result indicates that each component contributes to the scores and training stability.
B.4.2 NOVEL SCENE GENERATION
We tested two different architectures for Structured Prior. Both variants use MLP, and one of them updates each slot independently, while the other one considers interaction between slots. As we show in Table 8 there is no significant difference between these variants. This result indicates that 2 alone properly models object placement, and Structured Prior does not have to model dependency between slots.
Transformer
MLP (w/o interaction)
MLP (w/ interaction)
FID
83.3 1.7 1.7
84.35 = 2.33
84.18 + 2.61
Table 8: FID score from two different architectures of Structured Prior
B.5 DOWNSTREAM TASK
Models
Datasets
Observations
Accuracy
MulMON
64.6
WeLIS(z*)
73.0
CLEVR MV
3
WeLIS(z)
63.5
WeLIS( + 2))
71.5
MulMON
75.0
WeLIS(z²)
76.5
CLEVR MV
4
(zLIS(z)
53.5
WeLIS(z 2)
76
MulMON
77.5
WeLIS(z")
76.0
CLEVR MV
5
WeLIS(z))
58.0
WeLIS(Z 2)
76.0
MulMON
65.4
WeLIS(z9)
63.9
CLEVR AUG
3
WeLIS(z)
74.5
WeLIS(z" + z)
77.5
MulMON
70.7
WeLIS(z*)
68.8
CLEVR AUG
4
WeLIS(z)
78.7
WeLIS(z + z)
78.3
MulMON
71.8
WeLIS(z9)
69.3
CLEVR AUG
5
WzLIS(z)
79.3
WeLIS(z +z)
79.6
Table 9: Accuracy of downstream task. Observvtvvsss stands for the number of available observation views.
We evaluated the latent space of z qualitatively in Figure 6 In this section, we evaluate the obtained
representation by downstream task. The task is predict how many objects are in the observed scenes. We used linear classifier in all the models.
WeLIS can use both z and z for classification task, thus we tested all three patterns: z only, z only and both z and zz
17

---

Under review as a conference paper at ICLR 2022
C
DATASETS
CLEVR MV
CLEVR AUG
GQN JACO
Figure 11: Ground truth samples of the three datasets used in this paper: CLEVR MV, CLEVR AUG and GQN Jaco.
We show ground truth samples of the three datasets: CLEVR MV. CLEVR AUG and GQN Jaco. These datasets are available from the official implementation of MulMON (https: h/t.tttttttt////// com/Na Com/NNanbbL.J/MuMMMNN
18


Reviews:
[{"review_id": "W_pieDpx3rs", "reviewer": "Reviewer_KiHp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* the global variable seems to help capture global scene structure\n* decent analysis with ablations and latent investigation\n* acceptable results for reconstruction but at the same time nice samples\n\nWeaknesses:\n\n* One thing that is not clear is the interaction between GECO and IAI - it seems to me that this is not trivial to achieve as the lagrange multipliers would need to be maintained both internally and externally in the losses. Can the authors elaborate on that point?\n* What is the purpose of the PoE encoder over the views? it's not immediately clear to me why would that help - there are other ways to handle multiple views which may work as well (see for example NeRF-VAE - Kosoirek et al.).\n* I would be happy to see latent traversals for the global latents - while we can see they capture structure of scenes it would be interesting to see how they change and how the resulting scene changes.\n* What happens when you instantiate the model with different number of slots? does the global latent overfit to the number or does it adapt? what happens when you test on scenes with different number of objects?\n* I feel experimental validation is quite limited - both in dataset use (would be nice to see more complex data with more complex structure globally) and evaluation metrics.\n", "summary_of_the_paper": "This paper presents a hierarchical, object centric view conditional generative model for scenes. The latent structure includes a global latent variable which captures general scene structure/configuration and conditions a set of slotted latents, each one capturing a single object in the scene. An inference method is proposed - a product of expert + flow encoder for the global, and a sequential encoder + iterative refinement for the slots. Everything is view conditional so training requires camera information, but novel views may be synthesized at ease.\nThe model is demonstrated to work on CLEVR and a slightly more complex version of CLEVR with more object structures as well as the Jaco arm dataset.", "summary_of_the_review": "All in all this is a nice paper, with a bit more depth could be a nice contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HU7c7siuwjt", "reviewer": "Reviewer_Pvdr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n\nThe model is a novel extension of MulMON, albeit using fairly standard components and techniques (autoregressive prior; normalising flow posterior). It performs better than MulMON on two tasks -- novel view synthesis (NVS) and a priori scene generation -- both quantitatively and qualitatively.\n\nThe method is described clearly. Components and the overall approach are motivated reasonably, and described precisely. There seems to be enough information to reimplement the method, and code is also provided.\n\n## Concerns:\n\nAdding a global scene prior to MulMON seems a rather small contribution, and the 'auxiliary' modifications (e.g. normalising flow posterior) are of minimal novelty. In particular, global scene priors are well-established in the literature (e.g. GENESIS, GNSM, O3V) and clearly necessary for generative modelling of full scenes, so it's not really surprising that adding one to MulMON is beneficial. The only significant difference of this paper is that it operates in the (easier) multi-view setting.\n\nMoreover, the paper indicates that the global prior is useful for NVS, yet it is not trained jointly with the rest of the model, but post-hoc -- hence has no value in guiding the encoder for NVS. This is in contrast to GNSM and O3V, which both train the scene prior jointly with the full model.\n\nNVS should be evaluated quantitatively and qualitatively vs. GQN and ROOTS and/or other SOTA single-view NVS methods, in the (more challenging) single-view case. It should be evaluated vs. NeRF and Object-Compositional NeRF [Yang, ICCV 2021] and/or other SOTA multi-view methods in the multi-view case.\n\nGeneration should be evaluated vs. GENESIS and/or other SOTA object-centric generation methods, as well as an appropriate baseline that is not object-centric (e.g. NVAE).\n\np7: fig 4 -- why do you show MulMON as a generation baseline, when Tab.1 notes that it is not suitable for generation? This gives a false impression of the benefits of the proposed method. This figure should instead show samples from a SOTA model that is designed for generation, e.g. GENESIS -- even if that model is trained in the more-challenging single-view setting.\n\np8: fig 5 / sec. 4.2.1 -- why does removing the structured prior make the images for WeLIS more blurry? The correct comparison here would be to retrain the model under the assumption of an iid gaussian (or similar) prior for each object and compare results with the structured prior; in that case the individual objects should still look reasonable. And, the purpose of the structured prior is claimed to be to capture global properties of the scene; the statement that it only helps with local appearance, and that the ablated model still places objects correctly, seems to contradict this.\n\nQualitative results are not very impressive, even on simple datasets of rendered shapes. The method uses a spatial-broadcast decoder, which is known to encourage color segmentation, making it particularly tuned to these easy datasets (with single-colored objects). The paper would be significantly strengthened by evaluation on a more challenging dataset.\n\np7: tab. 2 -- the two-std intervals for not-bolded values overlap those for bolded values in most cases; please modify the bolding to indicate which values are better up to statistical significance, as currently it is misleading.\n\n## Minor issues / suggestions:\n\np3: \"...generate novel scenes outside of the training distribution...\" -- no, outside the finite training set, but inside the training distribution\n\np4: \"...mostly three\" -- when is/isn't it three? How is the viewpoint parameterised -- is it a vector in some absolute world-space, or is it relative to one of the camera views?\n\np4: eq. 3 -- explain here why you introduce a variational distribution on the object latents given z_g (and not depending on the image except via z_g), when the prior model already defines the exact distribution of z|z_g. I assume this is due to the fact (sec. 3.3) that the 'prior' is trained post-hoc (and therefore unable to guide the training of the encoder).\n\np6: \"GECO is not necessary for our model ... but improves generation quality\" -- by how much? This should be covered in the ablation study.\n\np6: sec. 3.3 -- as this describes part of the generative process (not just the inference training, except the last four lines), and is supposed to be a significant contribution of the paper, I think this section should be moved before the discussion of the variational distribution used for training the 'lower' level of the model.\n\np7: the paragraph introducing the datasets should cite MulMON, as the supplementary states that is where CLEVR-MV and CLEVR-AUG were introduced\n\nThere is no discussion of computational cost (training / inference time).\n\nThe following should be cited and/or compared against:\n- GIRAFFE [Niemeyer, CVPR 2021] -- focuses on generation, but models multi-object scenes, with semi-explicit 3D\n- BlockGAN [Nguyen-Phuoc, NeurIPS 2020] -- same as previous\n- O3V [Henderson & Lampert, NeurIPS 2020] -- uses multi-view (video) input, and is object-centric, and has a global scene prior\n- Structured Generative Modeling of Images with Object Depths and Locations [Anciukevicius et al, ICML OOL workshop 2020] -- also hierarchical, object-centric, and stage-wise trained (albeit in more challenging single-view setting)\n\nA number of references cite arXiv only, even though the papers are published. Please add the proper conference citations.", "summary_of_the_paper": "The paper proposes a new structured generative model of multi-view images, representing their latent decomposition into objects. This model extents MulMON -- a spatial mixture model over multi-view images that is suitable for image decomposition but lacks a prior necessary for synthesis -- by adding such a prior (using an autoregressive model), and making some other architectural changes (e.g. adding a normalising flow posterior instead of gaussian). In the proposed model, a global latent variable conditions the sampling of per-object latents, which in turn parameterise a spatial mixture model for each image. The model is evaluated on novel view synthesis (NVS) and a priori scene synthesis, on simple datasets of renderings (e.g. multi-viewpoint CLEVR); it is shown to out-perform a baseline (MulMON) on several metrics for both tasks quantitatively, and qualitative results are also displayed.", "summary_of_the_review": "The proposed method is a straightforward but original combination of existing techniques. It improves over MulMON on NVS and generation, but the evaluation needs to be much more comprehensive, in particular adding comparisons vs. appropriate baselines for each task. Moreover, certain important technical points regarding the benefit of the model structure should be clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4OSLYN5cw1", "reviewer": "Reviewer_TFws", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* ***Competitive compared to MulMON on the tested datasets***\nThe model builds upon MulMON, and the empirical results shown in the paper support that the proposed model is competitive and sometimes outpeforms MulMON.\n\nWeaknesses:\n* ***Limited comparison to alternative approaches***\nThe model is mostly compared against MulMON. However there are other models for inferring object-centric scene representations. The authors try to distance their model from approaches that operate using single views such as UORF or ObjSURF on the basis that this model uses multiple views to infer scene representations. While this is technically true, the novel views generated from the proposed approach look quite worse than those of UORF or ObjSURF, while the proposed approach is using more data and thus solving an easier task. Given this, I would encourage the authors to perform a comparison to one of these models on one of the datasets analyzed in those papers and comparing reconstruction quality, to have a sense of whether using multiple views is useful.\n\n* ***Missing a derivation of the training objective***\nThe authors propose a training objective derived from a lower bound on the likelihood (ELBO). However, the authors do not derive this ELBO, and I believe it is not technically correct as explained in the paper. Since the authors propose a hierarchical latent variable model, certain latent variables (in this case, the object variables $\\mathbf{z}$) are conditional and not independent on the global/top latent variable ($\\mathbf{z^g}$). When computing the lower bound, we have terms such as $q(\\mathbf{z}|\\mathbf{z^g})$ which define a gaussian function conditional on $\\mathbf{z^g}$. I believe the denominator cannot be factorized into two KL divergences as shown in the paper due to this dependency, and instead we have a double integral on $\\mathbf{z}$ and $\\mathbf{z^g}$. While in practice this might have small implications since we approximate integrals/KLs with MC sampling with usually a single sample, I would encourage the authors to provide a full derivation of the ELBO that properly justifies the training objective for their hierarchical graphical model.\n\n* ***Key components of the approach are not well-supported, neither theoretically or empirically***\nThe model is complex, featuring normalizing flows, structured priors and an optimization algorithm that updates different parts of the model in alternance.\nThe authors do not provide a theoretical justification for these design decision. Futhermore, they also do not provide a proper empirical justification. In particular, while there are some visualizations of model ablations, the results are pretty inconclusive. It is unclear whether the use of normalizing flows provides a significant improvement, and most of the design choices are left without a proper analysis. This could be improved if the authors provided theoretical justifications as to why certain design decisions are needed or are better than more straightforward choices, or with clear empirical ablations that quantitatively show the difference between using or not using components of the model.", "summary_of_the_paper": "The paper proposes a model for inferring object-centric scene representations from multiple views that explicitly factorizes the representation into global scene properties and objects. \nThe model builds upon previous object-centric VAEs and is capable of generating novel views of a scene.\nThe main contribution of the paper is the global-local factorization of the scene representation together with the architectural changes and optimization tricks required to train a VAE with the proposed latent structure.", "summary_of_the_review": "The proposed approach, while competitive with a previous baseline, has limited theoretical novelty (hirerachical VAEs have been proposed before), and it is unclear what would be the advantages of using the proposed approach compared to alternative approaches. Therefore, I believe the paper is below the acceptance threshold in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2YIaBZehUJ2", "reviewer": "Reviewer_RJ1B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- The paper applies normalizing flows to stable inference and training, which might be empirically useful.\n\n# Weaknesses/Questions\n1. Introducing a global latent variable does not necessarily model spatial information. However, the authors seem to claim a straightforward global latent variable without any explicit modeling can tackle this issue. Even if it might be empirically true, it is better to change the tone, e.g. \"we introduce a global latent variable and empirically find that it can model spatial information, which is shown in Sec xxx\". Otherwise, it is misleading to readers. Or the authors can claim that previous methods may ignore modeling global relationships while this paper takes it into consideration.\nSimilarly, the authors also claim their model can generate physically plausible novel scenes. It seems that the authors mainly rely on the Structured Prior (Transformer) to achieve this, but there is no reasonable analysis about why the Structured Prior is relevant to physical plausibility.\n\n2. The equation (7) looks erroneous. The first KL term seems to be derived from $E_{q(z,z^g|X)} [\\log \\frac{p(z^g)}{q_\\phi (z^g|X)}]$. However, $KL[q_\\phi (z^g|X)||p(z^g)] = E_{q(z^g|X)} [\\log \\frac{q_\\phi (z^g|X)}{p(z^g)}]$. Can the authors check the correctness of the ELBO used in the paper?\n\n3. The baselines of single-view object-centric learning methods, e.g. IODINE, MONET, GENESIS, are missing. Although the setting is multi-view, those baselines are still comparable.", "summary_of_the_paper": "The paper tackles object-centric learning from multiple views. It proposes an extension to MulMON. Concretely, it introduces a global latent variable, aiming at modeling global information like spatial relationships, and predicts object-level latent variables upon the global one by iterative amortized inference. It also uses normalizing flows to stabilize training and inference. The authors compare the proposed model with  MulMON on three simple datasets: CLEVR MV, CLEVR AUG, GQN Jaco.", "summary_of_the_review": "The paper does not fully justify why the introduced hierarchical probabilistic model can lead to their claims: spatial information and physical plausibility, which makes those claims subjective. Besides, the baselines are limited and thus experiments are not convincing. Overall, I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received four high-quality reviews and there are a lot of meaningful discussions during the author response period. After the discussions, all four reviewers agreed that the submission can be strengthened in a number of ways, including more solid experimental results and a justification for the correctness of the ELBO.  The AC agrees. The authors are encouraged to revise the paper based on the reviews for the next venue.
Decision: Reject