Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a hierarchical, object centric view conditional generative model for scenes. The latent structure includes a global latent variable which captures general scene structure/configuration and conditions a set of slotted latents, each one capturing a single object in the scene. An inference method is proposed - a product of expert + flow encoder for the global, and a sequential encoder + iterative refinement for the slots. Everything is view conditional so training requires camera information, but novel views may be synthesized at ease.
The model is demonstrated to work on CLEVR and a slightly more complex version of CLEVR with more object structures as well as the Jaco arm dataset.
main_review: Strengths:

* the global variable seems to help capture global scene structure
* decent analysis with ablations and latent investigation
* acceptable results for reconstruction but at the same time nice samples

Weaknesses:

* One thing that is not clear is the interaction between GECO and IAI - it seems to me that this is not trivial to achieve as the lagrange multipliers would need to be maintained both internally and externally in the losses. Can the authors elaborate on that point?
* What is the purpose of the PoE encoder over the views? it's not immediately clear to me why would that help - there are other ways to handle multiple views which may work as well (see for example NeRF-VAE - Kosoirek et al.).
* I would be happy to see latent traversals for the global latents - while we can see they capture structure of scenes it would be interesting to see how they change and how the resulting scene changes.
* What happens when you instantiate the model with different number of slots? does the global latent overfit to the number or does it adapt? what happens when you test on scenes with different number of objects?
* I feel experimental validation is quite limited - both in dataset use (would be nice to see more complex data with more complex structure globally) and evaluation metrics.

summary_of_the_review: All in all this is a nice paper, with a bit more depth could be a nice contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles object-centric learning from multiple views. It proposes an extension to MulMON. Concretely, it introduces a global latent variable, aiming at modeling global information like spatial relationships, and predicts object-level latent variables upon the global one by iterative amortized inference. It also uses normalizing flows to stabilize training and inference. The authors compare the proposed model with  MulMON on three simple datasets: CLEVR MV, CLEVR AUG, GQN Jaco.
main_review: # Strengths
- The paper applies normalizing flows to stable inference and training, which might be empirically useful.

# Weaknesses/Questions
1. Introducing a global latent variable does not necessarily model spatial information. However, the authors seem to claim a straightforward global latent variable without any explicit modeling can tackle this issue. Even if it might be empirically true, it is better to change the tone, e.g. "we introduce a global latent variable and empirically find that it can model spatial information, which is shown in Sec xxx". Otherwise, it is misleading to readers. Or the authors can claim that previous methods may ignore modeling global relationships while this paper takes it into consideration.
Similarly, the authors also claim their model can generate physically plausible novel scenes. It seems that the authors mainly rely on the Structured Prior (Transformer) to achieve this, but there is no reasonable analysis about why the Structured Prior is relevant to physical plausibility.

2. The equation (7) looks erroneous. The first KL term seems to be derived from $E_{q(z,z^g|X)} [\log \frac{p(z^g)}{q_\phi (z^g|X)}]$. However, $KL[q_\phi (z^g|X)||p(z^g)] = E_{q(z^g|X)} [\log \frac{q_\phi (z^g|X)}{p(z^g)}]$. Can the authors check the correctness of the ELBO used in the paper?

3. The baselines of single-view object-centric learning methods, e.g. IODINE, MONET, GENESIS, are missing. Although the setting is multi-view, those baselines are still comparable.
summary_of_the_review: The paper does not fully justify why the introduced hierarchical probabilistic model can lead to their claims: spatial information and physical plausibility, which makes those claims subjective. Besides, the baselines are limited and thus experiments are not convincing. Overall, I tend to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new structured generative model of multi-view images, representing their latent decomposition into objects. This model extents MulMON -- a spatial mixture model over multi-view images that is suitable for image decomposition but lacks a prior necessary for synthesis -- by adding such a prior (using an autoregressive model), and making some other architectural changes (e.g. adding a normalising flow posterior instead of gaussian). In the proposed model, a global latent variable conditions the sampling of per-object latents, which in turn parameterise a spatial mixture model for each image. The model is evaluated on novel view synthesis (NVS) and a priori scene synthesis, on simple datasets of renderings (e.g. multi-viewpoint CLEVR); it is shown to out-perform a baseline (MulMON) on several metrics for both tasks quantitatively, and qualitative results are also displayed.
main_review: ## Strengths:

The model is a novel extension of MulMON, albeit using fairly standard components and techniques (autoregressive prior; normalising flow posterior). It performs better than MulMON on two tasks -- novel view synthesis (NVS) and a priori scene generation -- both quantitatively and qualitatively.

The method is described clearly. Components and the overall approach are motivated reasonably, and described precisely. There seems to be enough information to reimplement the method, and code is also provided.

## Concerns:

Adding a global scene prior to MulMON seems a rather small contribution, and the 'auxiliary' modifications (e.g. normalising flow posterior) are of minimal novelty. In particular, global scene priors are well-established in the literature (e.g. GENESIS, GNSM, O3V) and clearly necessary for generative modelling of full scenes, so it's not really surprising that adding one to MulMON is beneficial. The only significant difference of this paper is that it operates in the (easier) multi-view setting.

Moreover, the paper indicates that the global prior is useful for NVS, yet it is not trained jointly with the rest of the model, but post-hoc -- hence has no value in guiding the encoder for NVS. This is in contrast to GNSM and O3V, which both train the scene prior jointly with the full model.

NVS should be evaluated quantitatively and qualitatively vs. GQN and ROOTS and/or other SOTA single-view NVS methods, in the (more challenging) single-view case. It should be evaluated vs. NeRF and Object-Compositional NeRF [Yang, ICCV 2021] and/or other SOTA multi-view methods in the multi-view case.

Generation should be evaluated vs. GENESIS and/or other SOTA object-centric generation methods, as well as an appropriate baseline that is not object-centric (e.g. NVAE).

p7: fig 4 -- why do you show MulMON as a generation baseline, when Tab.1 notes that it is not suitable for generation? This gives a false impression of the benefits of the proposed method. This figure should instead show samples from a SOTA model that is designed for generation, e.g. GENESIS -- even if that model is trained in the more-challenging single-view setting.

p8: fig 5 / sec. 4.2.1 -- why does removing the structured prior make the images for WeLIS more blurry? The correct comparison here would be to retrain the model under the assumption of an iid gaussian (or similar) prior for each object and compare results with the structured prior; in that case the individual objects should still look reasonable. And, the purpose of the structured prior is claimed to be to capture global properties of the scene; the statement that it only helps with local appearance, and that the ablated model still places objects correctly, seems to contradict this.

Qualitative results are not very impressive, even on simple datasets of rendered shapes. The method uses a spatial-broadcast decoder, which is known to encourage color segmentation, making it particularly tuned to these easy datasets (with single-colored objects). The paper would be significantly strengthened by evaluation on a more challenging dataset.

p7: tab. 2 -- the two-std intervals for not-bolded values overlap those for bolded values in most cases; please modify the bolding to indicate which values are better up to statistical significance, as currently it is misleading.

## Minor issues / suggestions:

p3: "...generate novel scenes outside of the training distribution..." -- no, outside the finite training set, but inside the training distribution

p4: "...mostly three" -- when is/isn't it three? How is the viewpoint parameterised -- is it a vector in some absolute world-space, or is it relative to one of the camera views?

p4: eq. 3 -- explain here why you introduce a variational distribution on the object latents given z_g (and not depending on the image except via z_g), when the prior model already defines the exact distribution of z|z_g. I assume this is due to the fact (sec. 3.3) that the 'prior' is trained post-hoc (and therefore unable to guide the training of the encoder).

p6: "GECO is not necessary for our model ... but improves generation quality" -- by how much? This should be covered in the ablation study.

p6: sec. 3.3 -- as this describes part of the generative process (not just the inference training, except the last four lines), and is supposed to be a significant contribution of the paper, I think this section should be moved before the discussion of the variational distribution used for training the 'lower' level of the model.

p7: the paragraph introducing the datasets should cite MulMON, as the supplementary states that is where CLEVR-MV and CLEVR-AUG were introduced

There is no discussion of computational cost (training / inference time).

The following should be cited and/or compared against:
- GIRAFFE [Niemeyer, CVPR 2021] -- focuses on generation, but models multi-object scenes, with semi-explicit 3D
- BlockGAN [Nguyen-Phuoc, NeurIPS 2020] -- same as previous
- O3V [Henderson & Lampert, NeurIPS 2020] -- uses multi-view (video) input, and is object-centric, and has a global scene prior
- Structured Generative Modeling of Images with Object Depths and Locations [Anciukevicius et al, ICML OOL workshop 2020] -- also hierarchical, object-centric, and stage-wise trained (albeit in more challenging single-view setting)

A number of references cite arXiv only, even though the papers are published. Please add the proper conference citations.
summary_of_the_review: The proposed method is a straightforward but original combination of existing techniques. It improves over MulMON on NVS and generation, but the evaluation needs to be much more comprehensive, in particular adding comparisons vs. appropriate baselines for each task. Moreover, certain important technical points regarding the benefit of the model structure should be clarified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model for inferring object-centric scene representations from multiple views that explicitly factorizes the representation into global scene properties and objects. 
The model builds upon previous object-centric VAEs and is capable of generating novel views of a scene.
The main contribution of the paper is the global-local factorization of the scene representation together with the architectural changes and optimization tricks required to train a VAE with the proposed latent structure.
main_review: Strengths:
* ***Competitive compared to MulMON on the tested datasets***
The model builds upon MulMON, and the empirical results shown in the paper support that the proposed model is competitive and sometimes outpeforms MulMON.

Weaknesses:
* ***Limited comparison to alternative approaches***
The model is mostly compared against MulMON. However there are other models for inferring object-centric scene representations. The authors try to distance their model from approaches that operate using single views such as UORF or ObjSURF on the basis that this model uses multiple views to infer scene representations. While this is technically true, the novel views generated from the proposed approach look quite worse than those of UORF or ObjSURF, while the proposed approach is using more data and thus solving an easier task. Given this, I would encourage the authors to perform a comparison to one of these models on one of the datasets analyzed in those papers and comparing reconstruction quality, to have a sense of whether using multiple views is useful.

* ***Missing a derivation of the training objective***
The authors propose a training objective derived from a lower bound on the likelihood (ELBO). However, the authors do not derive this ELBO, and I believe it is not technically correct as explained in the paper. Since the authors propose a hierarchical latent variable model, certain latent variables (in this case, the object variables $\mathbf{z}$) are conditional and not independent on the global/top latent variable ($\mathbf{z^g}$). When computing the lower bound, we have terms such as $q(\mathbf{z}|\mathbf{z^g})$ which define a gaussian function conditional on $\mathbf{z^g}$. I believe the denominator cannot be factorized into two KL divergences as shown in the paper due to this dependency, and instead we have a double integral on $\mathbf{z}$ and $\mathbf{z^g}$. While in practice this might have small implications since we approximate integrals/KLs with MC sampling with usually a single sample, I would encourage the authors to provide a full derivation of the ELBO that properly justifies the training objective for their hierarchical graphical model.

* ***Key components of the approach are not well-supported, neither theoretically or empirically***
The model is complex, featuring normalizing flows, structured priors and an optimization algorithm that updates different parts of the model in alternance.
The authors do not provide a theoretical justification for these design decision. Futhermore, they also do not provide a proper empirical justification. In particular, while there are some visualizations of model ablations, the results are pretty inconclusive. It is unclear whether the use of normalizing flows provides a significant improvement, and most of the design choices are left without a proper analysis. This could be improved if the authors provided theoretical justifications as to why certain design decisions are needed or are better than more straightforward choices, or with clear empirical ablations that quantitatively show the difference between using or not using components of the model.
summary_of_the_review: The proposed approach, while competitive with a previous baseline, has limited theoretical novelty (hirerachical VAEs have been proposed before), and it is unclear what would be the advantages of using the proposed approach compared to alternative approaches. Therefore, I believe the paper is below the acceptance threshold in its current form.
review_title: null

Meta-review: This submission received four high-quality reviews and there are a lot of meaningful discussions during the author response period. After the discussions, all four reviewers agreed that the submission can be strengthened in a number of ways, including more solid experimental results and a justification for the correctness of the ELBO.  The AC agrees. The authors are encouraged to revise the paper based on the reviews for the next venue.
Decision: Reject