Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a VAE-based hierarchical generative model (named Latent Object Models ) for scenes that contain multiple objects. This is done by modeling the hierarchical relationship between scenes and objects via a set of latent variables for scenes and objects. The object latent variables are a sample from their corresponding object slots. The object slots are inferred deterministically from the scene latent variable.  Having deterministic object slots makes the inference more tractable by removing the need for sequential decomposition of a scene into object slotsIn the experiments, the authors show results for random sampling of the latent variables and some qualitative results for their method and different versions of the proposed method. The authors also show quantitative comparison with some prior works
main_review: 1- Paper strengths:
- The proposed method does not depend on the ordering of objects in their slots
- The proposed object slots distinctly correspond to masks for each object in a scene
- Thanks to deterministic ordering of the slots, specific object slots correspond to specific semantic concepts (e.g walls, different blocks etc)


2- Paper weaknesses
- The paper needs some polishing in general both for writing and structure. Parts of the paper are written in a casual way and need to be rewritten with a more scientific perspective. Also, some of the sentences are a bit vague and hard to understand 
- The Introduction section is not written well and has vague sentences; parts of it are written more similar to Related Works
- I agree that inference over all orderings of objects in a slot is hard. However, this is not well-motivated in the paper. In other words, it is not clear to me why the authors have tried to tackle the problem of integrating object orderings in a slot. This is because in practice some prior works seem to propose/fill object slots in an autoregressive fashion and produce good results at the end of the day. I think this could probably be because the models proposed in prior works do not really rely on the order of objects. As one of the experiments, the authors need to show that the results obtained from priors models are sensitive to such orderings
- Qualitatively, the produced images are not sharp enough compared to some some prior works (MONet, IODINE, some  of the missing references)
- The authors do not show any experiments for demonstrating minimal generalization of their proposed method for unseen scenes
- The main comparison of the proposed method with prior work is done quantitatively. This is not sufficient. The authors need to show results for qualitative comparison as well instead of different variations of the proposed method.

3- Additional Comments:

Abstract:
- The abstract is too wordy and not terse to the point.
- The abstract is a little bit technical to understand for general readers. It’d be better if the authors do not assume that most future readers are in their field necessarily. Examples of such technical terms: “bag-of-words assumption”, “slots are generated with an autoregressive prior”
- “known limitations”: vague, relies on authors’ prior knowledge


Introduction:
- The introduction starts by quickly talking about this specific work. It lacks a good, high-level motivation on why this problem is important. 
- Pragraph2 (P2): “These models have demonstrated strong generalization and object-level disentanglement” → strong and subjective assessment. I would argue no computational model in the machine learning literature has shown strong generalization capabilities as measured against human generalization ability.
- P2 → Veerapaneni et al. does not seem to be using any of the models cited two sentences before
- “However, we understand little about how to jointly decompose scenes and sample novel
scenes from noise with slot-based models” → vague

Experiments:
- Section 4.1: I do not see an experiment in this section that clearly relate to “EXPLORING THE SCENE-LEVEL LATENT SPACE”. I was expecting some results for latent traversal here, given the section name.

References:
- Many of the references do not have the correct publication venu/information. For instance “Entity abstraction in visual model-based reinforcement learning” is an ICLR 2019 paper but is cited as an arXiv paper. Other examples: “Adam: A method for stochastic optimization”, “Auto-encoding variational bayes”, “Generative neurosymbolic machines.”, “Multi-object representation learning with iterative variational inference” etc

4- Missing References:
- White, Tom (2016): Sampling Generative Networks. Open Access Victoria University of Wellington | Te Herenga Waka. Journal contribution. https://doi.org/10.26686/wgtn.12585362.v1 
- Eslami, SM Ali, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman et al. "Neural scene representation and rendering." Science 360, no. 6394 (2018): 1204-1210..
- Sitzmann, Vincent, Michael Zollhoefer, and Gordon Wetzstein. "Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations." Advances in Neural Information Processing Systems 32 (2019): 1121-1132.
- Nanbo, Li, Cian Eastwood, and Robert Fisher. "Learning object-centric representations of multi-object scenes from multiple views." Advances in Neural Information Processing Systems 33 (2020).
- Crawford, Eric, and Joelle Pineau. "Learning 3D Object-Oriented World Models from Unlabeled Videos." Workshop on Object-Oriented Learning at ICML. 2020.
- Deng, Fei, et al. "Generative scene graph networks." International Conference on Learning Representations. 2020.
- Deng, Fei, Zhuo Zhi, Sungjin Ahn. Hierarchical Decomposition and Generation of Scenes with Compositional Objects,  Workshop on Object-Oriented Learning at ICML. 2020
- Chen, Chang, Fei Deng, and Sungjin Ahn. "Object-Centric Representation and Rendering of 3D Scenes." JMLR (2021).
summary_of_the_review: Overall, I believe the paper needs a lot more work in order to be accepted. The authors reasoning for proposing a generative model that does not depend on ordering of is not well motivated and the contributions of this work are not strong. Also, the results of the experiments the authors have performed are far from convincing, specially because they have not properly compared their method to more relevant prior works. Finally, the paper is not written well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents LOM, a new structured latent variable model for object-centric probabilistic modelling of scenes. It differs from previous models of this type, such as GENESIS, by avoiding the use of an autoregressive prior over slots/objects. In prior work, the use of such priors has meant that either inference needed to introduce an order among slots (e.g., by also occuring autoregressively), or the order of slots had to be implicitly integrated out during training. It is argued that either option is suboptimal. The proposed model instead models slot assignments in the prior deterministically given a scene-level variable. Inference commences largely in an orderless manner; the inferred slots are matched with their counterparts in the prior only after the fact for training.
The model is evaluated experimentally on a suite of synthetic multi-object image datasets, and compared to prior work such as GENESIS.

main_review: Strengths:
 - The paper addresses an important problem, namely how object-centric models can formulate a prior distribution over set-structured latents, while avoiding issues with autoregressive solutions.
 - The proposed model is novel to my knowledge. It follows the existing framework of probabilistic, object-centric modelling, but differs from existing systems in important ways.
 - The paper is generally well written.
 - The construction of the model is sensible given the design goal of eliminating the autoregressive prior from GENESIS-type models.

Weaknesses:
 - The paper lacks a clear theoretical argument as to why the autoregressive prior used in previous systems is a problem. Training autoregressive models while implicitly integrating out the factorization order has been demonstrated to work well at large scale (see e.g. the work deriving from orderless NADE, such as XLNet). It is not a priori clear to me that the solution explored here should necessarily be more effective. It might be better to introduce this issue as an open question, rather than to claim that one solution is clearly better than the other.
 - The quantitative results are mixed. LOM does not outperform previous models in terms of FID score on any of the three datasets. The main positive result is the improved structural accuracy on ShapeStacks, which is a somewhat soft metric due to its manual evaluation.
 - I am unconvinced by the qualitative results as currently presented. The interpolations in Fig. 4 do not strike me as particularly smooth, and contain inconsistent configurations with overlapping objects. Even the top ranked examples in Fig 5 b,c do not share very many commonalities with the given scene. In any case, qualitatively inspecting a single sample scene with no baseline provides too weak of a signal to support the arguments made in the text.
summary_of_the_review: The paper proposes an interesting solution to the issue of formulating priors over set-structured latent variables. However, in its current state, it does not offer clear enough evidence that this solution is actually beneficial compared to previous approaches. In particular, its (potentially) beneficial properties should be more clearly demonstrated using quantitative metrics.

Post rebuttal: I have increased my score to a 6, as the authors have added additional results supporting the argument that LOMs improve the unconditional sample quality of the underlying VAE.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work tackles the problem of learning object-centric generative models of scenes, and in particular it addresses the challenge of handling the orderless nature of objects (represented by latent variables) when learning the model's prior. Most recent models in the literature either assume an arbitrary sequential scene generation (which can hinder the model's performance), or attempt to model the orderless set of object latent variables with an autoregressive prior which can cause discontinuities during training when the allocations of object change.
The proposed method relies on incorporating (a) a scene-level prior that deterministically generates a per-object distribution ordering and (b) an ordering algorithm to match object posterior and prior distributions. By learning to generate an object order from a scene latent variable, it effectively sidesteps the problem of modelling the large equivalent class of sets. Quantitative and qualitative results show that incorporating these two ideas allows learning better generative models that can sample scenes with plausible composition of objects.
main_review: *Strenghts
This paper builds on recent methods in unsupervised object-centric segmentation and scene generation which have seen recent success in a number of synthetic image datasets. The authors give a good explanation of the problems in many of these pieces of work and motivate the need for a method that can better handle the orderless structure of the object in these scenes.  In general this paper is well written and the ideas proposed are clear and technically sound.  To my knowledge, while some of the contributions are inspired by similar ideas, the combined approach is novel and can be a useful contribution to the literature. With some exceptions noted below, the most closely related literature is compared either in the discussion and/or empirically. The experimental methodology focuses on relevant metrics, namely sampling quality, and the structure of the learned representations (e.g. structural accuracy and disentanglement), and results show for the most part that the proposed method leads to clear benefits over related work, and a large set of ablations are carried out to establish the usefulness of the different parts of the method.

*Weaknesses
1) While the key ideas are clear I find the resulting architecture, which relies on having a separately generative model, an odd choice. It makes it a bit more difficult to understand how the two models interact with each other. The new model attempts to address shortcomings of the original model, but depends on the original model to work well as it relies on its approximate posterior -- this could in some cases lead to learning limitations since gradients are not passed through the object latents of the original model. While choosing to stop the gradients is convienent (as otherwise the ordering algorithm's non-differentiability would need to be addressed), the lack of an end-to-end solution might limit the new model's benefits to propagate to learning the original model, or conversely, the limitations of the original model prevents the new model to learn e.g. robust object-centric representations. 

2) The idea to deterministically generate an order to ease the task of modelling a set is not new.  For instance, Vinyals et al. show the performance benefits of finding a canonical ordering, and Zhang et al. learn to optimize these permutations. While these works don't apply it to latent set representations, I think the paper should at least discuss how some of these works relate to the proposed idea.

3) I agree with the authors that introducing a hierarchy (e.g. in the form of a global scene latent) in many of these slot-based models is a great idea. However, I would like to see more evidence of the benefits of having the prior *deterministically* produce the order of slots, as opposed to having an autoregressive slot prior. Conceptually, the burden of reasoning about the correlations between objects (in all its permutations) has now been shifted the the global prior since p(z_{1:k}|s) are now independent. This might still work for some types of scenes but I'm not sure how that would scale to scenes with larger and more complex compositions of objects. Perhaps the authors can provide some examples of what they expect the global latent to represent.

4) It's unclear what architecture is used for the likelihoood model. Is it using/sharing the same parameters as EMORL? This should be clearer in Sec. 3. 

5) Wrt. the main methods compared, GENESIS and GENv2, I did not follow the explanation as to why these methods can handle richer background and textures (leading e.g. to the lower FID in Objects Room) compared to LOM. I would also like to see the resulting metrics using EMORL alone, as that would clearly show the gains of incorporating the ordered prior.

6) I don't think the error bars in Table 2 are described. It would be also interesting to see the ShapeStacks S-Acc performance across the different runs.

Some other minor comments:

- The LOM-u and LOM-s model performance was hard to find. The labelling in the text and the table in the appendix should be consistent.

- In the appendix, it is stated the Transformers have constant time dependency on the sequence length unlike RNN. While this is true if one can fully parallelize the Transformer's SA, they in general have quadratic complexity on the sequence length vs RNN's linear complexity.

**Post-rebuttal update**
While I believe there are interesting ideas in this paper, quantitative results in the paper are still mixed, with LOM-based methods mostly clearly showing its usefulness (vs e.g. GENv2-MoG) on one dataset and one metric (S-Acc). For this reason, I think the paper would benefit from having experiments using carefully designed datasets that more directly tackles the problem of modelling orderless elements using the proposed method. I would like to see the benefits on this problem compared to other ways of modelling sets. In its current form, reliance on an pretrained models complicates understanding the extent of the benefits and limitations of the method. On the point of limitations, I don't think the authors resolved my concern regarding relying on a scene-level Gaussian to capture all the structure between objects.
I appreciate the efforts of the authors to incorporate additional results and overall improving the paper, but for the reasons above  I will keep the current score. 

References.

Vinyals et al: Order Matters: Sequence to sequence for sets. ICLR 2016

Zhang et al. Learning Representations of Sets through Optimized Permutations. ICLR  2019



summary_of_the_review: The overall ideas proposed are novel and a potentially useful contribution to the space of image VAEs. The model proposed is shown to work well on well-known multi-object datasets, and a large set of ablation studie disentangle the contributions of the different parts of their method. I'd like to see more discussions about the conceptual limitations of their ideas, as well as references to related work in finding canonical orders for set-based tasks. The authors should also address the clarity issues in the text and figures mentioned above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a structured latent variable model, called Latent Object Models (LOM), for compositional scene generation. To generate a scene image, LOM first samples a scene-level latent variable, and then samples multiple slot-level latent variables based on a sequence of deterministic variables modeled by an autoregressive model. To infer the ordered slot-level latent variables, LOM first estimates orderless slot posteriors, and then determines the permutation of orderless slot posteriors based on a greedy matching algorithm. After learning, the model is able to generate scene images with reasonable relationships of objects. Experiments are conducted on three datasets, and the proposed method is compared with several existing methods.
main_review: **Strengths**
1. The paper is well-written and the proposed method is clearly described.

2. The proposed method aims to tackle the difficulty of modeling and inferring relationships of objects, which is an important problem in object-centric representation learning of scenes.

**Weaknesses**
1. The novelty is limited as the generative model is similar to GENESIS (Engelcke et al., 2019) in that the relationships of objects are modeled by an RNN, while the inference method is similar to Slot Attention (Locatello et al., 2020).

2. The main claim/contribution that using a sequence of deterministic variables (referred as `causal sequence’ in the paper) to block dependency between scene-level random variable $s$ and slot-level random variables $z_{1:K}$ seems problematic. As shown in the generative process (Eq.1), $z_{1:K}$ still have dependences conditioned on unobserved $s$.

3. Using a causal sequence to model orders of slots is questionable. Although there are dependences between slots, the joint distribution of slot-level latent variables should be invariant under permutation because there are no causal effects among static objects. 

4. Relationships between objects are not considered when estimating orderless slot posteriors. Besides the regularization of the KL divergence term, how to improve the inference of slot-level latent variables by utilizing the learned priors of object relationships is unexploited.

5. Only the scene generation performance is compared in the paper, the scene decomposition performance (e.g. ARI and AMI), which is of vital importance to evaluating object-centric learning performance, is not included. In addition, the reconstruction performance (e.g. Log-Likelihood and MSE) is also missing.

6. The proposed method is not compared with GNM (Jiang & Ahn, 2020), which also has scene latent and considers relationships among objects, and many existing relevant object-centric scene representation methods. The overall scene generation performance of the proposed method is inferior to GENESISv2 (Engelcke et al., 2021).

**Questions**
1. How to select the hyperparameter $K$?
2. How to avoid representing the same object with multiple slots?
3. Is the proposed method able to generate images containing a specific number of objects?
4. Is the proposed method able to generate images containing fewer or more objects than those used for training?

summary_of_the_review: This paper considers an important problem in object-centric scene representation, that is, modeling and inferring relationships among objects in a scene. However, both modeling and inference methods have heavy overlaps to existing works and, the main contribution of the paper, inserting a deterministic layer between scene-level random variable $s$ and slot-level random variables $z_{1:K}$ seems unable to block dependences among slots. The proposed method has not been compared to closely relevant methods, and the evaluation metrics for decomposition and reconstruction performance are also not fully investigated. The overall scene generation performance of the proposed method is inferior to GENESISv2.
review_title: null

Meta-review: This paper proposes a VAE-based hierarchical generative model (Latent Object Model) to model scenes with multiple objects. 
The paper would benefit from a substantial revision to improve text quality and clarity.
The experiments lack proper quantitative baselines and imputations; and the overall results are quite underwhelming relative to existing models.
Decision: Reject