Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides an empirical evaluation of whether object-centric representation pre-training is useful for RL learning. They find that OCR pre-training generally delivers better and more data-efficient model, also allowing generalization to unseen settings (e.g., an unseen number of objects).

strength_and_weaknesses: The paper presents a nice analysis on the environments and tasks it studies. It shows that OCR outperforms end-to-end distributed representations on relational tasks and scenes with many objects but not on simple object goal tasks, which aligns with our expectation of OCR.

However, my main concern is that the experiments only concern two simple synthetic environments. I am not sure if this can be considered as a comprehensive study on whether OCR pre-training is effective for reinforcement learning.

The experiments are als similar to the experiments in COBRA (Watters et al.), which also uses Spriteworld. It would be good if the authors could highlight the difference.

Watters et al. COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration

---

Minor question:

Why would OCR models outperform the GT model in second figure in Figure 3? I would image GT as an upper-bound.

How is the GT state embedded? What does the ground truth state include? 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear. Most implementation details are provided.
summary_of_the_review: Overall, while I think the authors provide nice analysis on the studied environment. However, I am not sure if the experiments can be considered as comprehensive enough to support some of the rather general claims the authors made.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates multiple hypotheses regarding the use of object-centric representations (OCRs) for reinforcement learning. Specifically, they consider object-centric representations that are pre-trained using unlabeled interaction data in the environment before being frozen for use in RL. The hypotheses are that OCR pretraining improves RL sample efficiency (H1), that OCR pretraining improves performance on tasks involving relational learning (H2), and that OCR pretraining helps facilitate out-of-distribution generalization to new objects or combinations of objects (H3). To assess these hypotheses, the authors construct 4 2D interactive navigation tasks in Spriteworld and 1 robotic manipulation task in CausalWorld. The authors deliberately limit the complexity of the involved objects to assess OCR in a best-case setting. The authors consider the SLATE, IODINE, and Slot-Attention OCR encoders, and $\beta$-VAE (using the VAE latent or CNN feature map), and end-to-end learned CNN as non-structured baselines. Finally, ground truth low-dimensional state is also compared to. The authors present results that largely support the 3 hypotheses, and also contribute results tackling 5 other research questions.
strength_and_weaknesses: Strengths

This paper sets out to empirically validate important motivating assumptions underlying the field of object-centric representation learning. The methods are clearly presented and the interpretation of the results is largely sensible.

Weaknesses

The toy nature of the experiments restricts the significance of the study to environments with similarly limited complexity. That is, part of the answer to the "When" in the title must include "when OCRs are essentially perfect". At the very least, the claims made in the paper should be so qualified. And unfortunately, I'm not sure that there existed much doubt about the efficacy of OCRs in this setting. 

The descriptions of the proposed tasks include a large number of plausible-sounding yet unsupported statements about necessary conditions for solving each task. For example, for Object Comparison, the agent does not necessarily need to compare all pairs of objects; it could succeed by simply learning to represent how much different colors appear in the image, and move towards the rarest color. Unfortunately, the inaccuracy in these statements propagates to the interpretation of the results, resulting in some overclaiming.
clarity,_quality,_novelty_and_reproducibility: Clarity

The organization of the paper is logical and facilitates understanding. I would have liked for a bit more detail of the models to be present in the main text, e.g. the precise form of the OCR representations before pooling (presumably a sequence of slots/tokens).

Quality

The execution of the work is largely good. The significance is somewhat limited, as argued above.

Novelty

The main novelty of this work lies in the formulation of the study's research questions and the creation of the Spriteworld and CausalWorld tasks. There is no technical novelty otherwise.

Reproducibility

The methods are described in a reasonable level of detail. Source code is not provided or promised, however.
summary_of_the_review: Overall, this paper is well-written and well-executed, but as an empirical study of prior methods its significance is severely limited by its chosen scope. I currently recommend borderline rejection, but with low confidence since I could be swayed by more positive assessments of impact from other members of the reviewing team with more skin in the field of object-centric representation learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a set of simple RL tasks meant to assess the benefits of using object-centric representations for RL. They compare the performance of 3 recent object-centric representation learning methods (IODINE, Slot Attention and SLATE) as features for a simple PPO agent solving these tasks, assessing details of where they help or not.
strength_and_weaknesses: 1. The paper is clear and very well introduced. It makes its assumptions and targets clear, and the set of proposed tasks are clean and well chosen.
   1. The Spriteworld tasks cover a good extension of capabilities, and if object-centric models fail on this dataset I am not sure how one would expect them to target more complex environments.
   2. The Robotics task is a nice simple increase in complexity, but obviously one would prefer to have more of these (perhaps closer to the scope of MetaWorld [1]?), and more complex visuals.
2. I think the models selected are good, and I found the baselines to be appropriate, but I was expecting the E2E CNN to be used more.
   1. What is the performance of E2E CNN in Table 2b?
   2. I am not entirely sure that CNN(VAE) is that useful, it is not really something that people use?
   3. You could have used other types of “end of convolutional stack as objects” methods, like what’s used in SPACE, RelationNets, or even use the Conv backbone used by Slot Attention directly?
3. It might be important to discuss the effect of the training set used for the object-centric representations.
   1. In the current tasks, a random policy might be sufficient, but this is usually not the case otherwise.
   2. You could introduce more “on-policy” datasets (akin to Dreamer), and see how this changes results.
   3. This would also allow you to “count” or not the samples used for this in your sample efficiency assessment (e.g. consider starting from an empty dataset and training the representation online like Planet/Dreamer. It might fail, but if it works the numbers would be better?)
4. I really liked the presentation of Section 4. It is clear, hypothesis-driven and very well executed.
   1. Figure 3 is clean and easy to follow.
   2. Figure 4 somehow confused me, because I was trying to find the same columns as in Figure 3. Could you reorder/transpose it to match the structure of Figure 3 perhaps?
5. The text in Section 4.2 describing Table 2 does not seem to correspond to the content of Table 2?
   1. It feels like this is a leftover of a draft and should be updated. The table indicates better results than the main text.
6. I also really liked the observations in Appendix B.3, might be nice if this could be brought into the main text more strongly?

[1] https://meta-world.github.io/ 

clarity,_quality,_novelty_and_reproducibility: * Clarity is very high. It’s really easy to follow and the choice of curves and results are great.
* Quality is also high, it is small in scope but everything seemed very well balanced and executed to me.
* Novelty is also good enough. The tasks aren’t dramatically different from existing ones, but they cover a good set, and the exact model comparison performed is clearly novel and will be valuable to refer to in the future.

summary_of_the_review: I am quite torn about this paper, because even though I feel it is well executed and proposes something that would be quite useful to build upon and for the community to use, it is perhaps too limited in scope for ICLR (although there is past evidence that similar papers got accepted [1]). If that was NeurIPS, I would recommend submission to the Dataset and Benchmark track, where it would clearly shine and I would heartily support it.


However, despite these current limitations, as a practitioner I would leverage this benchmark for my own work and I feel it would be a valuable paper to refer to or integrate into existing open-source projects. Hence, I would currently recommend to borderline accept, even though I understand this might not be possible.

[1] https://openreview.net/forum?id=1W0z96MFEoH
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - This paper investigates the qualities of object-centric representations (OCR) in the context of reinforcement learning (RL) tasks. Does so by empirically benchmarking OCR and non-OCR methods (eg: end-to-end learning (E2E) and variational auto-encoding (VAE)) on 5 tasks from 2 domains (see Fig. 2). Each benchmark is either driven by an hypothesis or a question, which are clearly stated up-front.
- The results from these empirical studies can be summarized as follows:
    - Object-centric pretraining improves sample efficiency over VAE or E2E pretraining on object-centric tasks (Section 4.1.1). But it is slower in terms of wall-clock time (Section 4.2-Q2).
    - Object-centric pretraining can learn relational properties of objects (eg: SLATE), which can enable reasoning (Section 4.1.1).
    - OCR and E2E generalize to new objects in the observation but both fail when it comes to compositionality (Sections 4.1.2, 4.2-Q1). They also generalize better to environment that are more visually complex than the ones seen during training (Section 4.2-Q3).
    - SLATE tends to outperform other OCR learning algorithms (Section 4.2-Q4) and MLP pooling degrades performance of SLATE (Section 4.2-Q5).
strength_and_weaknesses: - Strengths:
    - This paper studies an important aspect of transfer in RL: what kind of representations are needed, and when? The empirical evidence makes a strong case for OCR, as they perform best on virtually all experimental testbeds (caveat: those are chosen to be object-centric, see below). I especially enjoyed the evidence from Figure 3 that OCR (ie: SLATE) can ease reasoning on relational tasks.
    - The authors are very clear in their approach (driven by hypotheses and questions), and the experiments look carefully carried out. They also thoroughly describe their results, which makes me trust their conclusions on their experiments. Altogether, I found this paper easy to follow as it is always clear what aspect of OCR is being tested.
- Weaknesses:
    - The main limitation of this work is the scope hypotheses and questions: they all take the form “does method A work better than method B in task C?”. From the title and the introduction, I was hoping to gather new insights on the “how” and the “why” OCR could be useful in RL. But no hypothesis or question ever gets to these types of insights.
        
        In fact, I do not think the reader ever gets an answer to the title question; the first 2 bullet points in the conclusions suggests answers, but I don’t see how those answers are supported by the analysis. For example, which experiments support that:
        
        - “OCR can be slower for tasks where object-wise reasoning is not important”? It looks like all testbeds require some form of object-wise reasoning.
        - “OCR provides disentangled representations for each object”? First, this property is never tested. Second, it is never compellingly shown that it is the reason for the good performance of OCR.
        
        For those reasons, the main message I got from this paper strikes me as somewhat incremental: “object-centric representations work better for object-centric tasks”.
        
    - I also have some concerns over the experimental designs.
        - First, and as mentioned above, what about non object-centric tasks? How far do OCR lag behind non-OCR pretraining? This is important as it starts to provide some ground to answer the “when does OCR work” question.
        - Second, are the E2E and VAE baselines trained on enough data and with enough variations? It would be help the paper if the authors could show that even with more data, VAE and E2E have already “plateaued” and so the benefits of OCR can’t be make up for. Similarly, what if E2E and VAE were trained on much larger datasets (eg: ImageNet) where learning OCR becomes much more challenging (because of the lack of object-centric labels).
        - Third, and a more minor point, the benchmarking testbeds look a little bit simplistic (see Fig. 2). Do the insights in the paper carry to more realistic tasks such as real-world embodied AI (Habitat/AI2Thor)? The most realistic task in this paper (object-reach) is only ever used once (Fig. 6).
    - Some minor points:
        - The background section could include more details for at least one of the OCR method (eg: SLATE) to refresh the reader’s memory. It could also outline the main differences between all OCR methods.
        - Figures are unreadable for color-blind readers or when printed black-and-white.
clarity,_quality,_novelty_and_reproducibility: - Clarity: good, the paper is clear and easy to follow.
- Quality: ok, the presented experiments look sound but I wished the authors dove deeper into the properties of OCR.
- Novelty: incremental, I wish the authors provided more profound insights than “object-centric representations work better for object-centric tasks”.
- Reproducibility: ok, I haven’t look at the Supp. Mat. but believe the experiments are reproducible.
summary_of_the_review: - Strengths:
    - The core question of the paper is important to the ICLR community.
    - The experiments are clearly laid out, and their results discussed in details.
- Weaknesses:
    - The hypotheses tested in the paper are somewhat superficial and do not answer the core question of the paper — why and when are OCR required for RL?
    - Some experimental design choices can be improved (choice of testbeds, training of baselines)?
    - (Minor) Background could be more fleshed out and figures could be more legible.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the utility of object-centric representation (OCR) pretraining for policy learning in multi-object environments using a reinforcement learning (RL) objective. The paper systematically compares OCR pretraining against non-OCR baselines (e.g. CNNs) on a set of synthetic benchmark tasks: multiple tasks using SpriteWorld, specifically designed for this study, and one task on a synthetic robotic reacher task. 

All reviewers agree that this paper is well-written, generally of high quality, and addresses an interesting problem. The reviewers positively highlight the hypothesis-driven experimental investigation in the paper. The paper is generally novel, in that the exact setup chosen in the paper has not been addressed in prior work, but has limited novelty otherwise: related systematic investigations for policy learning exist: prior work [1] has already investigated object-centric representations for control (directly compared to autoencoder and contrastive models) for a related robotics task with very similar conclusions. Although their study uses an implicit behavioral cloning objective instead of a reinforcement learning (RL) objective to learn a policy, this reduces the significance of performing a similar study using RL.

The main (and major) concern shared by all reviewers is that the paper has chosen a very limited scope with potentially low significance for the community. The title of the paper suggests a broader scope than what the paper covers: the environments are of lower complexity than those covered in typical prior work on visual pre-training for control (see e.g. [1,2]): the four SpriteWorld tasks — while being a good test-bed for the hypothesis stated in the paper — consist solely of uni-colored sprites on black background and have unclear relevance for real-world problems. The proposed Object-Reach. task is a good step-up in visual complexity, but is otherwise still very simple with uni-colored cube objects, with the target object being always the same (blue cube). It is unclear how insights gained from these environments would translate beyond these setups which are currently of limited significance. As highlighted by the reviewers, the paper otherwise has limited technical novelty as it primarily consists of an empirical comparison of existing methods on newly designed benchmark tasks.

While the authors have added some clarifications and some additional results for small variations of their existing setup, this main concern remains. I think the direction this paper is taking is interesting and should be further expanded upon: the paper in its current form would heavily benefit from an expanded scope before being considered for publication, but is otherwise of high quality.

To improve the scope/significance of the paper, the authors could for example explore the following directions:
* The authors could consider adding more diverse non-OCR baselines, such as the one considered in [1] or a MAE baseline (e.g. the model from [2])
* Investigate OCR pre-trained representations for tasks with non-obvious object structure in the visual observation
* Investigate tasks where models like SLATE would currently still struggle to understand the limitations and headroom for current approaches


[1] Heravi et al., “Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations” (2022)
[2] Xiao et al., “Masked Visual Pre-training for Motor Control” (2022)
justification_for_why_not_higher_score: The authors have added some clarifications and some additional results for small variations of their existing setup during the rebuttal, but the main concern around the limited scope of the investigation that the authors chose for this paper remains.
Decision: Reject