Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed to learn object representations that can utiltize the category informations for perception and reasoning in reinforcement learning.
Experiments on two RL benchmarks show clear improvements of the proposed method over previous RL methods.
The ablation studies also cover a lot of aspects of the proposed method and show clear improvement of the each part of the proposed method.

strengths_and_weaknesses: 
S1: The idea that object representations should encode category informations is interesting and IMO important for build generlizable RL methods.

S2: In the evaluation of this paper, the proposed method shows a very clear and strong improvements over previous methods.

W1: The tested benchmarks seems to be limited, I am wonderring if there are some larger scale benchmarks to test the proposed method?

W2: The description of the proposed method is not quite clear, I would recommand the paper to at least include some preliminary on SPACE in the main paper or the supplementary instead of asking the reader to refer to the original paper, IMO SPACE is an important component of the proposed method, so it is better to describe it in the main paper.

questions: 
Q1: Some technical details about the SPACE method for unsupervised object discovery is not quite clear, I would 

Q2: The unsupervised object discovery module is trained on images generated from a random policy, it would be interesting to see if using a better policy to generate the images used for training the object discovery module could improve the performance, for example, using the PPO policy to generate images.

Q3: Is there other tasks that are larger than Crafter and Hunter in terms of the image size and action space? I think given the impressive performance of the proposed method, it is reasonable to see if the proposed method can work on larger scale problems.

Q4: There are other object centric representation learning method that could be discussed, such as slot attention[R1].

[R1] Object-Centric Learning with Slot Attention, NeurIPS 2020

limitations: I think the major limitation of this paper is that the evaluated tasks are only two, I would expect to see more tasks given that the proposed method works so well the task tested.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes augmenting Object Oriented RL approaches with additional information in the form of object category. Specifically, they use a module which uses unsupervised clustering to identify object categories from the output of a (previously known) unsupervised object detection algorithm. This is additional info (predicted category) is then incorporated using a convolutional encoder (OCAP). The object-category aware information from this module is passed into the third module which contains category-specific neural networks, which finally returns the action probabilities and values.
strengths_and_weaknesses: Stengths:

1. The motivation is clear and well represented in the paper. The paper does a good job of placing their contributions in the context of existing works.

2. OCARL does generalize better than standard OORL approaches. 

3. The methods section presents approach in detail, and should be reproducible for future researchers.

4. Ablation study is solid and gives a sense of which module helps with the performance bump. 

Weaknesses:

1. Missing literature: The work cited below [A,B] seem closely related to some work on OOD generalization to novel combinations. In fact, experiments in [A] might also explain why OCMR helps generalize when using separate category-specific neural networks. Would be good to include a line or two discussing how OCMR connects to this work. 

A. https://www.nature.com/articles/s42256-021-00437-5.pdf
B. https://www.nature.com/articles/s41593-018-0310-2

2. Might be good to clarify why the two domains mentioned in the paper were chosen. 
questions: 1. Is the number of classes (C) a known, required parameter? Was this known for the games, or not? 

2. Why were Crafter and Hunter domains chosen? Were there any specific properties of these domains that made them harder/easier for OCARL?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes learning object categories in order to enhance object-oriented RL (OORL). While past work on OORL has shown the benefit of extracting objects from a scene before learning with RL, this work has not considered learning to cluster objects into related categories so that the policy can treat similar objects the same. This work uses an existing Unsupervised Object Discovery (UOD) method to discover objects, but then learns a clustering of the resulting objects. It then asks the RL network to not only be able to predict the category of objects (forcing the representation to learn about object categories), but it also uses the object categories to apply independent category-specific networks to each object, which the authors argue improves modularity and therefore generalization. The paper shows results in two gridworld domains which represent complex sequential decision making tasks, and show significant improvements over relevant baselines and ablations. 
strengths_and_weaknesses: ### Strengths
**Originality**: The idea of learning categories of objects, and a policy that treats objects in the same category in a similar way, is both compelling and novel. The ablation studies reveal that the paper essentially makes two novel algorithmic contributions based on object category awareness that improve performance: OCAP (predict object cluster labels), and OCMR (having different modular independent networks for different object categories). 

**Significance:**
The paper benchmarks against two relevant OORL baselines (RRL and SMORL), and a reasonably advanced RL method (PPO), and shows clear and significant performance gains above these methods in complex sequential decision making tasks including Crafter (which is based on MineCraft). Although these environments are gridworlds, crafting is a complex problem and it is difficult for conventional methods to obtain high rewards. The idea of reasoning over object categories is likely to be useful in a broader range of tasks (e.g. object manipulation). The generalization performance is also interesting. 

**Quality**: 
The design of the network architecture proposed in the paper appears to be complex but thoughtful; the authors carefully justify each design decision (e.g. why the network should predict the object category as an objective, rather than use it as input, so it can be robust if the UOD module fails to extract the right category). The ablations also back up the claims that each component is necessary. 

**Clarity**:
The description of related work is clear and gives a good overview of the field while drawing the distinction with this work.

### Weaknesses:
**Significance/Quality:**
In spite of the fact that experiments were conducted in Crafter, most of the analysis for the paper is shown in Hunter. Crafter is a much harder environment of more significance to the community, so it would have been nice to see learning curves and ablations in this environment as well.

Further, although the paper tests generalization, it does so in a pretty limited way, because in no experiment is the agent ever tested on a truly novel object (only on differing combos of zombies and cows, but it has always seen at least one zombie or cow previously). The current paper actually does not test generalization to novel instances of the same category. It would be interesting to extend the experiments to another domain (one example could be simulated robotic manipulation of objects on a tabletop), and test whether the method can truly generalize to new instances of the same category. For example, if the agent had trained to pick up a red cup and a blue cup, could it generalize to picking up a green cup? 

**Clarity**:
The biggest weakness of the paper is clarity. In addition to smaller grammatical errors, there are some places where explanations significantly detract from clearly understanding the paper. The single biggest issue is in the method/results. Line 242-243 states that OCARL with p=1 is exactly RRL (where p is the probability that the UOD module can't detect a category). If this were true, this would significantly detract from the novelty of the paper. However, a few sentences below in lines 253-255 the paper directly contradicts this statement, by pointing out that OCARL uses the OCMR module, while RRL uses a universal network "instead of multiple object-category-specific networks like OCARL does". One of these two statements must be false, and I believe it is the one about the exact equivalence with RRL. This should be clarified, and it would be good to further explain in the related work or methods how the architecture being proposed is different than RRL.

The modularity experiment in Section 4.3.3 seemed interesting, but it was so hard to follow lines 272-276 I was not sure what the experiment actually showed. Is there a table or figure describing the results?

 A non-exhaustive list of other clarity issues: 
- There are so many acronyms in the paper that it is hard to follow what's actually going on. Rather than OCAP and OCMR, it might be nice to use a phrase like "category modules" to cue the reader. 
- In Section 3.1, it is not clear what $z_{ij}^{what}$ actually is... is it a continuous embedding? A categorical label?
- Line 106 appears to state that $z^{fg}$ consists of a fixed number (H x W) of object representations, but the related work section states that this is only a drawback of spacial mixture models, and not SPACE (which uses a spatial attention module for the foreground).
- Lines 100-101 contain an out of place detail (we run a random policy to get data and apply SPACE) that seems premature given the environment has not been introduced at this point. It is hard to evaluate whether a random policy would be able to collect enough data to cover all object categories without knowing about the environment. This also does not seem to be part of the *method*, per se, but rather an implementation detail.
- The explanation in lines 150-157 is unclear. 
- Can Equation 5 be interpreted as applying attention weights based on the category probabilities predicted from Z? The explanation / rationale should be improved here. 
- Is OCARL optimized with PPO as well? This should be clarified.
questions: If you can explain the difference between RRL and OCARL and clarify the issues I have raised above I may be willing to increase my score.

Why were most of the results shown in Hunter rather than Crafter? Crafter is a complex unsolved domain so it is of significant interest to the community. I would suggest moving Figure 8 from the appendix into the main text if possible, since those results are compelling.

I would suggest removing the italics such as "OCARL achieves better performance on *all* tasks" and "is the *only* method". This gives an appearance of a lack of objectivity that is not appropriate for a scientific paper. 

I suggest adding a sentence after line 89 to make it clear how OCARL is distinct from SMORL.

The paper contains several English-language errors that should be proof-read and corrected. A non-exhaustive list:
- Line 23: "To deal with these limitations, OORL is a promising way" -> "OORL is a promising way to deal with these limitations"
- Line 25: invariant -> invariance
- Line 29: "yield in low generality" doesn't make sense. 
- Line 158: concatenate -> concatenated 

I appreciated the cognitive science justification for object awareness that the authors provided.
limitations: The authors acknowledge the limitations of their clustering approach in line 124-126, but did not otherwise acknowledge the limitations of their method. I would suggest acknowledging that the didn't test generalization to novel instances of the same object category, as I explained above.

It does not appear that the authors included a discussion of the societal impact of their work. Perhaps they can talk about the benefit of improved object-oriented RL to the potential for building better robots, and the consequences thereof.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new framework applied to Object-oriented reinforcement learning (OORL) called Object-Category Aware Reinforcement Learning (OCARL), which aims to explicitly model the similarity between different objects of the same category. It consists of three main components: 1) UOD: unsupervised learning method (SPACE+IncrementalPC+KMeans) is used to complete the identification and clustering of objects; 2) OCAP: the clustering results are used as supervised learning signals to guide the encoding of objects; 3) OCMR: a self-attention mechanism is introduced, and different category objects use independent networks to complete the reasoning process. The experiments show that OCARL can improve the sample sampling efficiency and generalization ability of the model.
strengths_and_weaknesses: Strengths：

1. The paper contributes to improving object representation learning in model-free RL which has practical applications in object-oriented RL.
2. Compared with existing methods, OCARL shows outstanding generalization ability.
3. The paper is well written and well organized. 

Weaknesses：

1. The paper gives an implementation that combines unsupervised learning, supervised learning, and reinforcement learning. Each of the sections chooses a classic implementation, but the reasonableness is not well illustrated and there is no experiment showing the advantages compared with other implementable methods.
2. The paper points out that category information improves exploration efficiency and generalization, but there is no essential reason for the improvement (derivation, proof, etc.), nor is the condition and scope of its use stated.
3. The UOD method uses data derived from the random exploration, but random policies often fail to explore the environment effectively and may result in some of the objects not being discovered.
4. This paper uses KMeans for clustering operation, so the number of categories k is the hyperparameter. In practical scenarios, we usually do not know the specific number of categories of objects explicitly, so there may be cases where the preset number of categories is greater or less than the actual number of categories. This paper only implements the "less" situation in the Crafter environment without considering the case where the preset number of categories is greater than the actual number of categories.
5. Generalizability requires that the model be trained and applied to new data or new environments. However, the UOD is trained using the same environment as the test.
6. There may be a formatting error at the top of page 5, where the first two lines have no line number and the formula is not coded, resulting in an error in the formula reference in line 259.
questions: 1. Are there experiments that show it is possible to continue training the UOD in the RL phase?? It would make the results stronger and less reliant on a good random exploration strategy.
2. When the number of preset categories is greater than the actual number of categories (e.g. Hunter preset category number is greater than 4), is there an experimental display?
3. The clustering results of unsupervised learning are shown in Appendix C, but I am more interested in the representation of the latent variable $Z$. Is it possible to show the prediction results of $f_{cat}$ in OCAP?
4. Does the method only apply to environments, which can be divided into grids? In line 105, the image is divided into $H\times W$ cells, and each cell corresponds to an object. if the object is irregular or of a different size, how to handle it?
limitations: The paper doesn't really discuss its own limitations explicitly. Adding a section would be quite helpful. OCARL requires additional information about the environment (the number of categories), which may not work well if the difference between the number of preset categories and the actual number of categories in the environment is large.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a object-oriented reinforcement learning method, wihich consists of three parts: Category-Aware UOD, OCAP and OCMR. Category-Aware UOD mainly provides category supervision information for OCAP, enabling OCAP to perceive objects in the environment more accurately．OCMR learns the interactions between objects in a self-supervised manner to better predict action probabilities. Through the recognition of object categories in the environment and the learning of interrelationships, the model can learn better action strategies. It is verified that the proposed model can better perform reinforcement learning tasks in two experimental environments.
strengths_and_weaknesses: Strengths:
1）It is an interesting strategy to help the agent better perform reinforcement learning tasks by identifying the types of objects in the environment.
2) The experimental results show that the proposed method is effective.




questions: 1) Why do you need to take the maximum value of Z^out and Z in Eq. 6?
2）In the OCAP module, it is surprising that different objects can be segmented unsupervised from the environment image only by a simple convolutional neural network. Can you explain why this works?
limitations: 1) The proposed method does not consider the experimental environment with occlusion between objects, which is unreasonable.
review_title: null

Meta-review: metareview: This paper received three positive reviews and one borderline reject. In the rebuttal, the negative reviewer did not propose a response, but the authors have given detailed responses to the problems. And the other reviewers did not propose further concerns. Thus, taking the comments of the reviewers into account, the AC decides to accept this paper.
confidence: Certain
Decision: Accept