Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn object-centric representation with an energy-based model. This energy model takes a visual scene and a set of object-centric latent variables as input. Latent variables are inferred from visual observations through gradient-based MCMC sampling, where the gradient is derived from the energy function.

To formulate a permutation-invariant energy function over a set of objects, this paper proposes two variants. The first one is to simply sum over individual energy functions from the set of latent variables. Another is to compose them with an attention mechanism.

The proposed model is firstly evaluated on the unsupervised object discovery task, where it demonstrates good segmentation performance. It also exhibits the ability to perform scene manipulation given the compositionality nature of the energy function. Lastly, the proposed model shows robustness to different hyper-parameters and generalization to unseen data.
strength_and_weaknesses: Strength:

(1) The energy-based modelling of object-centric learning minimizes assumptions made on the generative process, relieving neural networks from complicated designs to model visual scenes.

(2) The compositionality nature of the energy function makes it possible to perform scene manipulation with learned object-centric latent variables.

Weakness:
(1) In the Introduction, there is a lack of clear motivation to develop energy-based models compared with existing object-centric learning frameworks such as VAE based (MONet, IODINE, etc) or Attention based (SlotAtt). What are the key advantages of energy-based models over others? 

(2) The proposed method somehow takes a similar formulation with SlotAtt, especially EGO-Attention. In experiments, the performance gain is also marginal compared with SlotAtt.

(3) The datasets used for unsupervised object discovery are quite simple. Previous approaches have already achieved very high ARI scores on them. Actually, the ARI metric seems not informative because it’s very easy to get high scores. More general metrics such as AP are suggested, as also pointed out in a very recent work [1]. 

In addition, evaluation and comparison on more challenging datasets such as ShapeStacks [2], ObjectsRoom [3], or even real-world datasets are very desirable. Otherwise, it’s hard to demonstrate the clear advantages of EGO over other baselines. 

[1] Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images, NeurIPS 2022.
[2] ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking, ECCV 2018.
[3] Neural Scene Representation and Rendering, Science 2018.

(4) In the evaluation of generalization capability on unseen object styles, it is claimed that the created OOD data have altered colours, textures and shapes. However, it is not demonstrated qualitatively or quantitatively how and to what extent colour jitter transformation and neural style transfer can alter the colour, texture and shape distributions.
clarity,_quality,_novelty_and_reproducibility: This paper is of high quality in terms of readability, clarity, and completeness. It also exhibits a new formulation of object-centric learning with an energy based model. The implementation details and experiment settings are very clear, showing good reproducibility from my understanding.

summary_of_the_review: This paper formulates a permutation-invariant energy model to effectively learn object-centric representation from visual scenes.  The proposed method is inspiring by modelling object-centric learning from a different perspective. The major concern is its structural similarity with a previous method and its marginal performance gain over baselines. It is strongly suggested to demonstrate its capacity for more challenging scenarios using more informative metrics.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel encoder that, given an image, learns object-centric slot representations using the EBM framework. The encoder computes an image-conditioned energy function $E(\mathbf{z}_1, \ldots, \mathbf{z}_K, \mathbf{x}; \boldsymbol{\theta})$ that assigns an energy value to slot representations $\mathbf{z}_1, \ldots, \mathbf{z}_K$. It then applies gradient-based MCMC sampling on the energy function to obtain the slots. Crucially, the proposed energy function $E(\mathbf{z}_1, \ldots, \mathbf{z}_K, \mathbf{x}; \boldsymbol{\theta})$ is invariant to the ordering of the slots $\mathbf{z}_1, \ldots, \mathbf{z}_K$. This order-invariance is achieved via two approaches: 1) by summing a per-slot energy function, OR, 2) by adopting a transformer that treats the slots as a set. The learning signal for training the model parameters comes from a reconstruction objective where a pixel-mixture decoder reconstructs the image by decoding the learned slots.
strength_and_weaknesses: ### Pros

1. Energy-based modeling is an important framework and the question of whether this approach can be used to learn slots is an interesting one. In this regard, the experiment results are useful — showing that it can perform comparably or somewhat better than the previous methods — and thus has the potential to spark a new line of exploration in the object-centric learning community.
2. The proposed encoder is almost entirely a transformer. I believe this is a good thing given that the transformer is currently a default model in the community and is perhaps more well-understood than a specifically-tailored architecture. This may also make their encoder applicable more generally beyond the image domain unlike some previous models like IODINE.
3. Useful ablations showing the model to be robust to various hyper-parameters.
4. The scene addition and subtraction via summing/subtracting the energy functions is interesting/surprising.
5. Generalizes well to OOD number of objects and unseen object style.
6. Has a probabilistic interpretation and thus the slots may capture the uncertainty.

### Weaknesses/Questions

1. While I believe in the promise of EGO’s probabilistic interpretation (i.e. modeling uncertainty in the partially-observable environments) there is a lack of some experiment that explores this aspect. Authors may consider showing a qualitative experiment of multi-stability in Tetris dataset similar to that shown in the IODINE paper.
2. Some unclear aspects:
    1. In the conclusion section, there is a line saying “minimal assumptions on the generative process…”. This line is a bit unclear to me. Is this referring to the fact that the model does not apply a prior on the latents like IODINE does? 
    2. Slot-attention performs an explicit spatial attention and mean-pooling to bind low-level information into object slots. The proposed model does this implicitly via the transformer. What are the implications of this design difference?
    3. How does the gradient flow backward via the MCMC sampling steps and to the EGO’s parameters.
3. Another unexplored question is whether the encoder can be adopted to deal with complex scenes in the same way as slot attention could be adopted in SLATE or SAVi.

### Minor Comments/Questions

1. I am wondering why IODINE is not shown in the property prediction experiment and also why GENESIS is omitted from the ARI result in Table 1.
2. The bar colors of various models can be made consistent across Fig. 2 and Fig 5(a).
3. The word ‘controllable’ in the title could be confusing to some because it might imply steerable/conditional slots such as that learned by SAVi. As I understand, ‘controllable’ seems to be about controlling scene generation and not controlling the slot representation itself.
4. In the section on ‘Increasing the number of objects’, it seems that the referred figure should be 4(c) and not 4(b). Also, in Fig. 4(c), the Y-axis range may be set to 0-100 to better highlight the fact that the performance deterioration is not large when testing on more objects.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, of high quality, and novel. The authors provide some implementation details in the appendix. However, I would still like the authors to release the code at some point because some details e.g., how exactly the MCMC sampler should be implemented in code and how the gradient flows to the EGO’s parameters is not obvious to me.
summary_of_the_review: I think the paper is novel and interesting and should be shared with the community. I support the acceptance of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an energy-based model object-centric learning pipeline.
Energy functions are designed to be permutation invariant.
Experiment results show that scenes are decomposed reasonably well.
Energy function algebra leads to meaningful scene editing results.
strength_and_weaknesses:   - Strength
    - The use of EBM is novel.
    - The segmentation performance is better than slot-attention
  - Weakness
    - The paper mainly focuses on segmentation performance leaving the reconstruction quantity untouched.
    - While the generative object-centric learning model can sample objects with various appearances, EBM does not support the generation of any form (at least not trivial).
    - While the beta-VAE-based model encourages feature disentanglement, EBM does not explicitly encourage such behavior.
    - And the above weakness are not discussed.
    - It is not entirely clear to me the meaning of "need for specifically-tailored neural network architecture or excessive generative modeling assumption".
    Could the author explain, for example, how IODINE is specifically-tailored or has excessive generative modeling assumptions?
clarity,_quality,_novelty_and_reproducibility: Paper writing is clear and easy to follow.

summary_of_the_review:   - While the EBM model is not as versatile as the generative object-centric learning model, I agree that it could be a strong object-centric learning candidate.
  - However, I hope the author can explain more about their assertion about their main advantages over previous works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a EGO, a neural module for object-centric representation learning. EGO casts the inference of object-centric latent variables as energy-based modeling. It uses Langevin MCMC to sample latent variables consistent with the given input image. Simplicity of the neural network architecture is emphasized. Empirical evidence of the validity of the approach is provided in the form of segmentation and property prediction on the Multi-Object Datasets environments. Additional experiments evaluating hyperparameter choices, robustness, and generalization are provided as well. Overall, the performance of EGO appears to be comparable to the most similar baseline Slot Attention.

strength_and_weaknesses: Strengths
========
The core idea, which I believe is casting slot inference as sampling slots consistent with an image as scored by an energy function, seems to be a novel and interesting approach to this task.

The presentation of the paper is solid.

Weaknesses
========
I have concerns with various claims made throughout the paper.

- **[EGO is…] “conceptually simple”/“Without the need for specially-tailored neural network architectures”** This is provided as one of the main contributions of the paper. I disagree that EGO is any simpler than the key baseline, Slot Attention. The “variant” of EGO explored in this paper (Algorithm 1) and Slot Attention are fairly similar methods in terms of the neural modules employed and the use of an iterative inference algorithm.

- **“Without the need for … excessive generative modeling” / “Minimal assumptions on the generative process”** When comparing EGO with previous generative object-centric models like GENESIS and GENESIS-v2 [1], it is important to consider that these models support unconditional generation of novel scenes because they have a prior distribution over slots. EGO sacrifices this capability in favor of a simpler approach to slot inference. Also, Slot Attention, the most relevant baseline, does not make any generative modeling assumptions either.

- **“We show that EGO can be easily integrated into existing architectures”** This is also provided as one of the main contributions of EGO, but it is not actually shown outside of the specific encoder-decoder object representation learning context. No other examples of how EGO can be used are provided. In fact, EGO's ability to discover objects seems to hinge on the use of the spatial broadcasting decoder (SBD) as an inductive bias to encourage each latent slot to fixate on a distinct scene object, in the same manner as the baselines (see [2]). Demonstrating how EGO can be used in a variety of diverse contexts to learn slots that fixate, without supervision, on objects would be one way to improve the paper. 

I struggle to see the novel conceptual insights or new capabilities enabled by EGO. One supposedly new capability --- compositional scene editing --- is explored in the experiments, but I believe the baseline object-centric methods can also support this to some degree? It would be good to compare against the performance of at least the key baseline, Slot Attention, at this. Also, I believe IODINE is capable learning a multi-modal distribution over slots (like EGO)? This capability is not empirically explored in the paper, however.

Issues of instability [3] and difficulty with more realistic scenes [4] for object-centric representation learning methods (particularly those based on the SBD) have been demonstrated in the literature, yet these issues are not discussed in this work. I could not find any discussed limitations.

Overall, the segmentation results seem to be comparable to Slot Attention, with only marginal demonstrated improvements in segmentation on Multi-dSprites. Perhaps conducting experiments on more challenging datasets would reveal a larger gap in performance with respect to baselines?

References
========
1. Engelcke, Martin, Oiwi Parker Jones, and Ingmar Posner. "Genesis-v2: Inferring unordered object representations without iterative refinement." Advances in Neural Information Processing Systems 34 (2021): 8085-8094.
2. Singh, Gautam, Fei Deng, and Sungjin Ahn. "Illiterate dall-e learns to compose." International Conference on Learning Representations. 2021.
3. Chang, Michael, Thomas L. Griffiths, and Sergey Levine. "Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation." arXiv preprint arXiv:2207.00787 (2022).
4. Karazija, Laurynas, Iro Laina, and Christian Rupprecht. "Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation." arXiv preprint arXiv:2111.10265 (2021).
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and appears to be reproducible. However, overall the main contribution (EGO) appears to be incremental when taken in the context of the existing body of work. 
summary_of_the_review: Overall, I believe this work is not yet ready for publication due to various incorrect/unjustified claims and limited novelty of the contributions (as described under Weaknesses). More justification and evidence is needed as to A) why this EBM-based approach constitutes a promising new direction for object-centric representation learning and B) whether EGO is truly a generic module for learning object-centric representations in other architectures/contexts as claimed. I believe with these improvements, the work could be a valuable contribution.

---
Update after rebuttal: I have increased my score from 3 --> 6 to reflect my new perspective of this work after discussion and improvements. See comment thread below for more details.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to learn object-centric representation with an energy-based model. This energy model takes a visual scene and a set of object-centric latent variables as input. Latent variables are inferred from visual observations through gradient-based MCMC sampling, where the gradient is derived from a permutation invariant energy function.
The proposed model is evaluated on the unsupervised object discovery task, where it demonstrates good segmentation performance, comparable to previous models of slot attention. It also exhibits the ability to perform scene manipulation given the compositionality nature of the energy function. 
All reviewers agreed on the novelty of the proposed formulation. 
They raised concerns regarding i) the advantages of the energy-based framework over previous methods, such as, slot attention, ii) visualization of alternative samples from the energy-based model iii) additional evaluation metrics to be reported, iv) application of the model in more complex setups, and potentially in the real world, v) discussion of the computational challenges of the proposed framework.

The rebuttal submitted by the authors addressed many of these concerns. The authors are encouraged to add all the additional clarifications  and experiments to the paper. Given the novelty of the work, and its comparable performance with alternatives, we believe this is a valuable read for the ICLR audience. 

justification_for_why_not_higher_score: The method does not perform clearly better than previous alternatives.
justification_for_why_not_lower_score: The proposed formulation is very novel and will help clarify what matters and what not for slot computation.
Decision: Accept: poster