Object-Category Aware Reinforcement Learning
Authors: Qi Yi
Rui Zhang
Shaohui Peng
Jiaming Guo
Xing Hu
Zidong Du
Xishan Zhang
Qi Guo
Yunji Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
object
unsupervised object discovery

Abstract: Object-oriented reinforcement learning (OORL) is a promising way to improve the sample efficiency and generalization ability over standard RL.  Recent works that try to solve OORL tasks without additional feature engineering mainly focus on learning the object representations and then solving tasks via reasoning based on these object representations. However, none of these works tries to explicitly model the inherent similarity between different object instances of the same category.  Objects of the same category should share similar functionalities; therefore, the category is the most critical property of an object. Following this insight, we propose a novel framework named Object-Category Aware Reinforcement Learning (OCARL), which utilizes the category information of objects to facilitate both perception and reasoning. OCARL consists of three parts: (1) Category-Aware Unsupervised Object Discovery (UOD),  which discovers the objects as well as their corresponding categories; (2) Object-Category Aware Perception, which encodes the category information and is also robust to the incompleteness of (1) at the same time; (3) Object-Centric Modular Reasoning, which adopts multiple independent and object-category-specific networks when reasoning based on objects. Our experiments show that OCARL can improve both the sample efficiency and generalization in the OORL domain.

Object-Category Aware Reinforcement Learning
Qi Y2s.33 Rui Zhang Shaohui Peng 2,3, Jiaming Guo
Xing Hu² Zidong D Xishan Zhang Qi Guo² Yunji Chen24
University of Science and Technology of China
SKL of Processors, Institute of Computing Technology, CAS
CCambricon Technologies *University of Chinese Academy of Sciences yiqi Aaqilmail.... ustu. en, fzhangrui, pengshaohui 182, guojiaming18s, huxing, duzidong, zhangxishan, guoqi, eyj}@ict.ac.
Abstract
Object-oriented reinforcement learning (OORL) is promising way improve the sample efficiency and generalization ability over standard RL. Recent works that try to solve OORL tasks without additional feature engineering mainly focus on learning the object representations and then solving tasks via reasoning based on these object representations. However, none these works tries to explicitly model the inherent similarity between different object instances of the same category. Objects of the same category should share similar functionalities; therefore, the category is the most critical property of object. Following this insight, pro- pose novel framework named Object-Category Aware Reinforcement Learning (OCARL), which utilizes the category information of objects to facilitate both perception and reasoning. OCARL consists of three parts: (1) Category-Aware Unsupervised Object Discovery (UOD), which discovers the objects well as their corresponding categories; (2) Object-Category Aware Perception, which encodes the category information and is also robust to the incompleteness of (1) the same time; (3) Object-Centris Modular Reasoning, which adopts multiple inde- pendent and Sejject-ategorryspeeefff networks when reasoning based on objects. Our experiments show that OCARL can improve both the sample efficiency and generalization in the OORL domain.
Introduction
Reinforcement Learning (RL) has achieved impressive progress in recent years, such as results in Atari [24] and Go [28] in which RL agents even perform better than human beings. Despite its successes, conventional RL is also known to be of low sample efficiency [8] and poor generalization ability [2]. Object-oriented RL (OORL) [10; 6; 23; 17; 39] is a promising way to deal with these limitations. Inspired by studies in cognitive science [25] that objects are the basic units of recognizing the world, OORL focuses on learning the invariance of objects' functionalities in different scenarios to achieve better generalization ability.
In OORL, the agent's observation a set of object representations, and the task can be solved via reasoning based on these objects. Most previous works [10; 17; 6] in OORL employ hand-crafted object features given advance, which require human expertise and result in low generality, limiting the applications of these methods. Recent works [39; 38; 37] try to broaden the scope of OORL's applications by avoiding using additional feature enginecring. Their works can be divided into either top-down approaches [39] which purely rely on reward signals to ground representations to objects, or bottom-up approaches [38; 37] which utilize unsupervised object discovery (UOD) technologies to provide structural object representations.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, none of these works [39; 38; 37] tries to explicitly model the inherent similarity between different object instances of the same category when reasoning based on objects. In most cases, objects of the same category should share the same functionalities, such as obeying similar behaviour patterns or having the same effects when interacting with other objects. Therefore, an intelligent agent should recognize an object's category to be aware of its functionalities. Since objects from different categories differ in their functionalities, the objects' functionallities can also be separately modelled (according to their categories), leading to modularity which is beneficial for generalization [4; 5; 36] in general.
Following the above insight, we propose a framework named Cateerry Aware Reinforcement Learning (OCARL), which utilizes the category information of objects to facilitate both perception and reasoning. OCARL consists of three parts: (1) category-aware UOD, which can automatically discover the objects as well as their corresponding categories via UOD and unsupervised clustering. (2) Object- Category A Aware Perception (OCAP), perception module that takes the object category information obtained in (1) as an additional supervision signal. OCAP can encode the category information into representations and is also robust to the incompleteness of UOD at the same time. (3) Object-Centric Modular Reasoning (OCMR), a reasoning module that adopts multiple independent and object-category sepecci networks, each of which processes the object features of the same corresponding category. Such a modular mechanism can enhance further the generalization ability of OCARL. Our experiments show that OCARL can improve the sample efficiency several tasks and also enable the agent to generalize to unseen combinations of objects where other baselines fail.
2 Related Works
Unsupervised Object Discovery Unsupervised object discovery (UOD) methods try to automati- cally discover objects without additional supervision. Roughly speaking, there are two main categories in this area: spatial attention models and spatial mixture models. Spatial attention models such as SCALOR [16] and TBA [14] explicitly factorize the scene into set of object properties such as position, scale and presence. Then these properties are utilized by a spatial transformer network to select small patches in the original image, which constitute a set of object proposals. These methods can deal with a flexible number of objects and thus are prominent in the UOD tasks. Spacial mixture models such as Slot Attention [21], IODINE [9] and MONet [1] decompose scenes via clustering pixels that belong to the same object (often in an iterative manner). However, they often assume a fixed maximum number of objects and thus cannot deal with a large number of objects.
SPACE [20], a UOD method used in this paper, combines both the spatial attention model and spatial mixture model. The spatial attention model extracts objects from the scene, whereas the spacial mixture model is responsible for decomposing the remaining background. Such a combination enables SPACE distinguish salient objects from relative complex backgrounds.
Object-Orientee Reinforcement Learning Object-oriented reinforcement learning (OORL) has been widely studied in RL community. There many different structural assumptions the underlying MDP in OORL, such as relational MDP [10], propositional object-oriented MDPs [6], and deictic object-oriented MDP [23]. Although these assumptions differ in detail, they all share common spirit that the state-space of MDPs can be represented in terms of objects. Following these assumptions, [3] proposes focused Q-learning to improve sample efficiency. [17] learns an bbject-based casual model for regression planning. [31] improves the generalization ability to novel objects by leveraging relation types between objects that are given in advance. Although these works have been demonstrated to be useful in their corresponding domain, they require explicit encodings of object representations, which limits their applications.
There are other lines of work in OORL such as COBRA [33], OODP [40], and OP3 [30] that try to solve OORL tasks in an end-to-end fashion. Most of these works fall into model-based paradigm in which an object-centric model is trained and then utilized to plan. Although these methods are effective in their corresponding domains, in this paper, restrict ourselves the model-free setting, which can generally achieve better asymptotic performance than model-based methods [32; 12).
The most related works should be RRL [39] and SMORL [38]. RRL proposes an attention- based neural network to introduce a relational inductive bias into RI. agents. However, since RRL relies purely this bias, it may fail to capture the objects from the observations. On the other hand,

---

SMORL utilizes an advanced UOD method to extract object representations from raw images, and then these representations are directly used the whole observations. However, SMORL adopts relatively simple reasoning module, which limits its applications in multi-object scenarios.
3 Method
OCARL consists of three parts: (1) category- aware unsupervised object discovery (category-awar) UOD) module, (2) object-category aware perception (OCAP), and (3) object-centric modular reason- ing (OCMR) module. The overall architecture is shown in Figure We will introduce (1) in Section 3.1, Section 3.2, and Section 3.3 respectively.
3.1 Category-Aware Unsupervised Object Discovery
Unsupervised Object Discovery In this work. SPACE [20] is utilized to discover objects from raw images. In SPACE, image decomposed into latent representations: foreground z and background ~ For simplicity, we only introduce the inference model of the foreground latent z/9, which is set of object representations. We encourage the readers refer [20] for more details. To obtain 2 x is treated as if it were divided into H x W cells and each cell is tasked with modelling most one (nearby) object. Therefore, 2/9 consists of a set of H x W object representations (i.e. H 5 5 )==)=ı),, each of which is a 3-tuple² z!
(rer where zighet). pree E {0,,1 } is [ij
1-D variable that indicates the presence of any object in cell (i,j), zijhere encodes z! size and location, and
zij zizhat is latent vector that identifies 2 itself. The whole inference model of z can be written as:
H W
<(2
IIII
pres of
(x)((z)))
biy/ere, x))(z?)) what xxz))
pree
(1)
=1
where Xij is a small patch from the x that is obtained by the proposal bounding box identified by where which should contain exactly one object if prres
Zij =
Unsupervised Clustering After applying SPACE on the randomly collected dataset, we can obtain H W. where N is the size a set of object representations: D {Znij 2nng nij Xnij In tim lim1j=l of the dataset, = zweere whott Xnúj are defined Eq.(1). We first select object patches from Znij
pres where what
D that of high object-presence probability by a threshold т: D, = {xais D p(2 1) > 7). Roughly speaking, each patch Xnzj E D, should contain exactly one object. Therefore, we first project Xnéj E D, into a low-dimensional latent space Rd by running IncrementalPCA [26] on Dr, and then adopt KMeans clustering with given cluster number C' upon the projected latent representations. The object-category predictor q(2 (x,)) = VKMoana o IncrenentaiPCA is the composition of IncrementalPCA and KMeans:
Zij V xHenna Irrreanentarrca (X..))
where 1/ E R.C. is an one-hot latent which indicates the category of Zij-
For simplicity, we regard the background (i.c.
=f = 0) as special 'object', thus gat can be written 2 zgat)). Note that 2 is an one-hot latent E RC+1
as 2 V concatenata ([1 zij :
pres pres
Although the object-category predictor (2) very simple, find it works well in our experiments. In practice, the cluster number C' of KMeans can be set by leveraging prior knowledge about ground- truth category number, or by analysing some clustering quality metrics such silhouette coefficients. In more complex environments, we can also rely on other advanced unsupervised clustering methods such NVISA [19], CC [18], etc.
3.2 OCAP: Object-Category Aware Perception
The OCAP module is designed to robustly incorporate the object knowledge of the (category-aware) UOD model defined in Section 3.1. In word, OCAP is plain convolution encoder that accepts the
depth for simplicity, because we only consider 2-D environment without object occlusions in this 2 we omit Z
paper.
3

---

Category-Aware UOD
Wimg
W
Unsupervised Clustering
2cat
2cat
Hima
UOD
2
2
OCAP
W ong
Z
Predict
2C2E
Z1
Convolution Networhs
Predisst
2 1W
ccat
Homp
Flatten H,W
Seat
Predict
!11
Predict
W
ZHW
ZHW
Observation
Add x-y
OCMR
coord.
Z
Z
Zout
211
2
fi
Self- Attention
f2
Sum
f3
acat
zuw
fc+1
Concatenate with
action_probee value
Figure 1: OCARI. consists of parts: (1) category-aware UOD, (2) OCAP, and (3) OCMR. (1) In category-
aware UOD, the objects firstly discovered by UOD method (SPACE) and then assigned (extended) category label by unsupervised clustering (IcreamentalPCA + KMeans), (2) OCAP, the observation is first encoded into a set of features Z by convolution networks. Each vector in z; Z....j E Z is given a task by the OCAP module to predict the corresponding (extended) object category, which is given by (1). Z is then fed to the OCMR module. (3) In OCMR, x-y coordinate information is added into Z. which gives us 2. Then we apply self-attention Z query related information for each object in Z. Q and are concatenated to form Z. Each entity in Z will be passed through different independent neural networks according to its corresponding object category, which will give us Zost, Finally, we perform max operation along H, dimension of to predict the value function and action probabilities.
nut Z, and the resulting feature
category information from the UOD model as additional supervision signals. Such a design makes OCARL robust to the incompleteness of the UOD model: in extreme cases which the UOD model fails to discover any objects, OCAP degenerates into a plain convolution encoder which still able extract useful information from the raw images with the help of reward signals.
Suppose the convolution encoder in OCAP maps the raw image observation E 3x Hamg x Wany to latent representation Z R. FxFx H XW where H, W are the same with those in the UOD model (see (q)( On the other hand, X is also fed into the UOD model to get the (extended) object category information cat (2 H W. via (2).
b==1=1
The OCAP module forces the latent representation Z to encode the object category information, which is implemented by training additional category predictor fout R feat predicts 2 given Z,..j which is a single channel in Z. Therefore, the additional supervision signal that incorporates the object knowledge of the UOD model is given as:
(atteoricaa(C++:
Leat
CrossEntrropyooss(ffor (Z. 2.9t).

---

Leat is used for training the convolution encoder in OCAP. In practice. Leat can be used as an auxiliary loss to any RL algorithm:
Liotal
LRI. + Aeat Leats
where Acat E R. is coefficient.
3.3
OCMR: Object-Centric Modular Reasoning
OCMR is module that takes Z E R FxxxW from the OCAP module input and outputs a feature vector that summarizes Z. The key design philosophy OCMR is adopt multiple independent and chbject-ategorry-seecifieee networks, each of which focuses processing the object features of the same corresponding category. Compared with using universal category-agnoste network, the processing logic of each independent network in OCMR is much simpler and therefore easier to master, which in turn allows for improved generalization as we will show in Section 4.3.2. This design philosophy also agrees with the recent discovery from [22; 4] which says that neural modules of specialization can lead to better generalization ability.
Given Z E R Fx #W from OCAP, we first encode the x-y coordinates information (corresponding to the H x W grid) to each channel, and then map the resulting tensor into Z E HWW.P. Ž is treated as HW objects with x-y coordinate information encoded. To model the relations between objects, we apply a self-attention [29] module on Z to query information for each object from its related objects:
Q softmax(ZW (2 ŽW, RWWF
where Wg. WR. W. FxF are trainable parameters. Modeling the relations is important, because one object's high-level semantic feature can be derived from other objects. For example, whether door be opened is determined by the existence of the key; therefore, the door should query other objects to check whether it is openable.
After the attention module, Q and Ż are concatenated together to get Ż V concatenate ([,, Z]) R /fW x2F Each Zij.: in Z is then fed into different independent neural networks according to its corresponding object category 2:j² practice, this can be implemented as:
C+1
ZOut Σ Sa(Zes.) seat
Rtjee
(5)
c=1
where fe R22F tensor zou HWW xF.
R.F,
seot Poll
2
(5) can be computed in parallel, resulting
ij:C+1
Finally, the n is added to Ż (i.e. residual connection). We perform max operation along the H, W dimension, and using the resulting vector R predict the value function and action probabilities:
action_probs, value fac (mmxmmXZZ oZ Z)).
(6)
H.W
4 Experiment
4.1 Task Description
In this work, we consider two task domains: Crafter and Hunter. The observations on both tasks are images of shape 64 x 64.
Oafler
Hunle 74C4
Cratter [13] is complex 2-D MineCraft- style RL task, where complex behaviors are nec- essary for the agent' survival. The environment is procedurally generated by arranging various resources, terrain types, and objects (18 in to- tal). The agent is rewarded if it can craft new items and accomplish achievements (22 possible achievements in total).
Figure 2: The Crafter (left) and Hunter (right) envi- ronment.

---

Hunter Z1C0/201
Hunter 21C1
Hunter Z4C0/20C4
Hunter 24C4
key
My
12y Pro Smort
SPORL
Eplook Return
Epoc0s Retum
Splasdk Rutum
Cphonde
Envionment Stepa (x0F))
Enviromment Staps (:1F))
Envirnment Steps (x122))
Environment Staps 1:10
Figure 3:
The mean returns over 12 seeds are plotted with 95% confidence interval. OCARL achieves better performance on all tasks; RRL is better than other baselines on Hunter-ZAC4 22-44/44442 whereas SMORL is better Hunter - 2wz0///CC..... Hunter- -ZC1 and Hunter- -4C0.
The results on Hunter.
Alg.
Return
Score
RRL
3.58 + 0.80
4.22 = 1.24
SMORL
3.48 0.26
3.94 0.50
OCARL((urr)
8.14 + 0.35
12.31 + 0.99
Table 1:
The results (averaged across 12 seeds) on Crafter after training for 25M environment steps. The 'Score' is a metric proposed in [13] which takes the difficultiy of cach achievement into account, thus is more proper than 'Retunn' to benchmark agent's behaviours.
Hunter is much simpler than Crafter. It is also procedurally generated but only contains 4 types of objects: Hunter, Cow, Zombie and Wall 3. The agent can control Hunter and can get positive reward (=1) if Hunter catches a Cow or kill a Zombie. The agent will be given a high reward (=5) if it can accomplish this task by catching/killing all Cows and Zombies. However, once Hunter is caught by a Zombie, the agent will receive a negative reward (=-1), and the episode ends. In a word, the agent in Hunter should master two different behavior patterns: chase & catch the Cow, and avoid & shoot at Zombie. Although Hunter is simple, is a typical OORL task. We can derive different environment instances from Hunter by setting the number of Zombies es and Cows. We use Hunter-Zane ZmCn to denote an environment that spawns (m Zombies + n Cows) at the beginning of each episode, and Hunter-Zmin ZnCm an environment that spawns (m Zombies n Cows) or (n Zombies + Cows).
4.2 Evaluation
In this section, we evaluate OCARL's effect both on sample efficiency and generalization ability. We consider the following baselines: PPO [27], RRL [39], and SMORL [38]. PPO a general-purpose on-policy RL algorithm with a plain neural network (i.e. Convolution + MLP). RRL proposes an attention-basse based neural network to introduce relational inductive biases and iterated relational reasoning into RL agent, which aims to solve object- object-oriented RL tasks without any other external supervision signals. SMORL utilizes the UOD model to decompose the observation into set of object representations which are directly used as the agent's 'ss new observation. All algorithms (RRL, SMORL, and OCARL. are (epllmmmeeeetted upon PPO to ensure a fair comparison. Note that OCARI. is orthogonal to the backbone RI. algorithm, therefore other advanced RL methods such as [15; 11] are also applicable. For more implementation details, please refer to the Appendix.
4.2.1 Sample Efficiency
The results on Hunter domain are shown in Figure 3. We can conclude three facts from this figure: (1) OCARL performs better and is more stable than other baselines all tasks. (2) RRL is competitive with OCARL on Hunter-Zer RRL relies solely on the reward signals to ground the visual images into objects; therefore, it requires enough (rewarded) interactions between objects, which simpler environments such as Hunter- fail to supply. (3) SMORL performs better than other baselines (PPO, RRL) simple environments such as Munto /2011, Montter-1CC and Munter-24CC0 2OC4, but unable to make progress in Hunter-Zer Z4C4 which consists of more objects. This is because SMORL adopts simple reasoning module that may not be well-suited to multi-object reasoning.
but significantly worse on other simper environments. This is because
'The images to render these objects come from Crafter [13], which is under MIT license.

---

RRL
SMORL
OCARL(ours)
100
Success Rate (%)
0.1
0.01
Collect Coal
Collect Diamond
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
Collect Wood
Defeat Skeleton
Defeat Zembie Zombie
Eat Cow
Make Iron Pickaxe
Eat Plant
Make Iron Sword
Make Stone Pickaxe Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
Place Plant
Place Stone
Place Table
Wake Up
Figure 4: Success rates (averaged across seeds) on 22 achievements in Crafter. These success rates are calculated using the 1M environment steps during training. OCARI. presents more meaningful behaviours than other baselines, such as collecting coal, defcating zombies, making wond pickaxes and so on.
Hunter-Z1CO/ZOC1
Hunter- Hunter-Z1C1
Hunter-Z4CO/Z0G
key
key
key
OCAILimund)
OCARLiourI
(CARL
Episode Rtetunn
RRL SMORL
10D
10
RAI
FFO
Episode Return
RAL
PPQ
SNCRL
SMORL
Episode
Environment Steps 1x10
Environment Steps (x10F)
Environment Steps (x10f)
Figure 5: The generalization performance Hunter. Agents are trained different environments Hunter-ZiCO Z1C0/2001, Hunter-ZiC1, and Muuter-Z4C0/2000 respectively, tested Hunter -404. OCARL the only algorithm that is able to generalize to the test environment. The mean returns over 12 seeds are plotted with 95% confidence interval.
The results on Crafter domain are shown in Table I and Figure 4. Crafter is complex envi- ronment in that it features sparse (rewarded) interactions between objects and large object category numbers. Besides, some objects are omitted by the UOD model (see Appendix) because they almost do not appear in the training dataset due to the insufficient exploration of the random policy. Due to these features, both RRL and SMORL fail provide noteworthy results (with scores of 4.22 and 3.94, respectively). On other hand, OCARL is able to present more meaningful behaviours (with score of 12.31), such as collecting coal, defeating zombies, making wood pickaxes and on, as shown in Figure The learning curves and detailed achievement success rates are provided in the Appendix.
4.2.2 Generalization
To evaluate the generalization ability, we train agents on Hunter- -ittterr210//20111. Hunter -IC1 and Minterr-4C0//OCCC separately, and then observe their test performance In Hunter-Zii -ZC1, the agent needs to generalize from a few objects to more objects. In Munter-Z4C0/ZA0 the agent never observes the coincidence of Cow and Zombie. The gener- alization from Hunter-Zico/ ZOC1 to Hunter-Z4Ce is the most difficult. Such a paradigm actually follows the nut-of-distribun never seen in the training environments. Therefore, although such generalization is possible for humans, it is much harder for RL algorithms.
M4Cter-Z444,
(OOD) setting, in that the object combination (4 Zombies + Cows)
As shown in Figure 5, OCARL is the only algorithm that make significant progress on the test environment. Although other basclines can improve their training performance (Figure 3), they
7

---

Hartan 21002200
Hurter 2IC1
Hurter 24C212044
Hurben 24C4
Figure 6: The Robustness of OCARL the incompleteness of UOD methods. We randomly mask out the objects discovered by the UOD method with different confusion probability p (x-axis) then record the final performance (y- (y-axis) over seeds after training for 5M environment steps. The performance of OCARL declines goes larger, but is still better than RRL (which exactly equals OCARL with 1)
Hunter- Z1ce/z01
Hunter-Z1C1
Muuter-z4co/Z00
key
key
key
OCCRL we OCAP OCCRL No OCHR UCAILIONS))
OCARL wio OCAP
OCARL we OCAP OCARL wi OCHR OLAnL/oural
Episode itetun
Episode Return
10
OCARL wio OCYR
ULAIILEonrs
Episode Return
Environment Steps (x)1P))
Environment Steps (x(10f)
Environment Steps (x:18))
Figure The generalization performance of OCARL, OCARL w/o OCMR, OCARL w/o OCAP. mean returns over 12 seeds are plotted with 95% confidence interval. Without the help of OCMR, the generalization ability of OCARL will decline significantly. On the other hand, OCMR alone can bring benefits without the help of OCAP (although not consistently as OCMR+OCAP).
unable to generalize to the test environment. The generalization performance of OCARL is most impressive Hunter-ZiCh. because agent trained on Hunter-Z1e even achieves slightly better than that on Hunter-Ze -444 (i.e. directly trained on the test environment, see the last figure in Figure 3.).
4.3 Ablation study
4.3.1 Robustness to the Incompleteness Object Discovery Methods
OCARL relies on UOD methods to discover different objects from given observations, which are not guaranteed to discover all objects, especially in complex environments [34]. Therefore, one may naturally ask to what extent OCARL's performance is affected by such incompleteness.
To answer this question, we randomly mask out each object (by setting ppres to 0) discovered by the UOD model with given probability p e [0, 1], and evaluate OCARL's performance after training 5M environment steps on Hunter. Such random confusion of the UOD model will affect both the OCAP and OCMR modules of OCARL. Note that OCARL(P 0) is OCARL itself, and OCARL(p 1) exactly RRL The results are shown in Figure 6. Although increasing p does hinder OCARL's performance, OCARL can still benefit from the imperfect UOD model in that its performance is still better than RRL in general. Such robustness should attributed to fact that OCARL does not directly use the output of the UOD model as observation but instead treats it as additional supervision signal that help the encoder to capture object-category information, which gives OCARI. second chance to find other useful information from the image.

---

Test Environment
No Modification
Disable fz
Disable fc
Hunter- -440
8.27 = 1.77
1.17 = 1.16
8.32 = 1.46
Hunter-2OCk
8.29 1.41
8.13 + 1.78
4.90 + 1.77
Table 2: The Modularity of OCMR. We first train agent Mlutter-4Cce and then disable the module fz (corresponding to Zombie) or fe (corresponding to Cow) in OCMR (see Eq.(5). Each agent is evaluated on both Hunter-Z4CH Z400 and Hunter- and the mean and standard deviation of 12 seeds are reported. . The results show that the behaviour patterns (chase & catch the Cow) and (avoid & shoot at Zombie) are encoded in fo and fz respectively.
4.3.2 OCMR Improves Generalization
One key advantage of OCARL is that it can generalize to unseen combinations of objects by leveraging the invariant relations between objects learnt during training. We argue that such an advantage is established via OCMR, which adopts object-category-speeee sppeciffc networks to deal with different objects.
To confirm our argument, we consider a variant of OCARL (i.e. OCARL w/o OCMR) in which the OCMR module is removed and replaced by an RRI-like reasoning module which adopts universal network (instead of multiple chject-category--peciff networks as OCARL does). The comparison between OCARL w/ and w/o OCMR is shown Figure 7. OCARL w/o OCMR (almost completely) fails to generalize from Hunter-Z ZZC00/0010, Hunter 24C00/ /204 to Hunter- -2444, while OCARL w/ OCMR does. In Figure 7, we also consider variant of OCARL (i.e. OCARL w/o OCAP) in which the OCAP module is disabled by setting Acat = 0 in (3) In this paradigm, only OCMR is in effect. As shown in Figure 7, OCMR alone still bring benefits without the help of OCAP (although not as consistently as OCMR+OCAP) ). According to these results, we can conclude that OCMR is crucial in OCARL's success.
4.3.3 The Modularity of OCMR
OCMR adopts different modules to deal with object features from different categories. In this section, will show that each module in OCMR actually encodes object-category -erccciirr--cccfiice behaviour pattern.
As stated in Section 4.1, the agent should master two useful behaviour patterns in Hunter: (1) chase & catch the Cow, and (2) avoid & shoot at Zombie. The existence of these patterns can be observed by evaluating the agent in Hunter-ZOC4 and Hunter-Z4CO respectively, because agents that master pattern (1) (pattern (2)) should achieve better performance on Hunter-ZOck (Hunter-Z4CO), In Table 2, we first train agent on Hunter-Z4C4 and then disable the module fx (corresponding to Zombie) or fc (corresponding to Cow) by replacing fc or fE with fog (corresponding to the "background") ) in (5). The results shows that disabling fz does not affect the performance Hunter-22 -OCt (8.29 > 8.13), which means that fz is not relative to behaviour pattern (I). Instead, fz is corresponding to behaviour pattern (2) because can observe significant performance decline (8.27 + 1.17) on Hunter-24C0 when we disable fz. Similar phenomenon can also be observed fc-
5 Conclusion
In this paper, we propose OCARL, which utilize the category information of objects to facilitate both perception. For the perception, we propose the OCAP, which enables the encoder distinguish the categories of objects. For the reasoning, we propose the OCMR, which adopts object-category- specific networks deal with different objects. Our experiments are carried out on Crafter and Hunter. Experiments show that OCARL outperforms other baselines both on sample efficiency and generalization ability. We also perform several ablation studies to show that (1) OCARL is robust to the incompleteness of UOD methods, (2) OCMR is critical to improving generalization, and (3) each module in OCMR actually encodes an object-category--peeeeeee speciifc behaviour pattern in Hunter.
The main limitation is that we only test OCARL's generalization ability to unseen object combinations, but not novel object instances. In order generalize to novel object instances that looks very different from old ones, the agent should interact with the novel objects first and then
Limitation
'This is because (3) takes no effect and (5) also degenerates into single universal network because all object features passed through the same network fog which is corresponding to the 'background".

---

infer the their underlying categories according to these interactions. Such a problem is likely to be in the meta-RL. RL domain and needs more efforts solve, which we would like leave for future work.
Acknowledgements
This work is partially supported by the National Key Research and Development Program of China(under Grant 2018AAA0103300), the NSF of China(under Grants 61925208, 62102399, 62002338, U19B2019, 61906179, 61732020), Strategic Priority Research Program of Chinese Academy of Science (XDB32050200), Beijing Academy of Artificial Intelligence (BAAI) and Bei- jing Nova Program of Science and Technology (Z191 1000011 19093), CAS Project for Young Scientists Basic Research(YSBR--299, Youth Innovation Promotion Association CAS and Xplore Prize.
10

---

References
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishahh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. ArXiv, 2019.
[2] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. ArXiv, 2020.
[3] Luis C. Cobo, Charles Lee Ishell, and Andrea Lockerd Thomaz. Object focused q-learning for autonomous agents, In AAMAS, 2013.
[4] Vanessa D' Amario, Tomotake Sasaki, and Xavier Boix, How modular should neural module networks be for systematic generalization? In NIPS, 2021.
[5] Harm de Vries, Dzmitry Bahdanau, Shikhar Murty, Aaron C. Courville, and Philippe Beaudoin. Closure: Assessing systematic generalization of clevr models. In NeurlPS Workshop, 2019.
[6] Carlos Diuk, Andre Cohen, Michael Littman. An object-oriented representation for efficient reinforcement learning. In ICML,, 2008
[7] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg. and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ICML, 2018.
[8] Vincent François-Lavee Peter Henderson, Riashat Islam, Mare G. Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. Found. Trends Mach. Learn., 2018.
[9] Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nicholas Watters, Christopher P. Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
[10] Carlos Guestrin, Daphne Koller, Chris Gearhart, and Neal Kanodia. Generalizing plans to new environments in relational mdps. In IJCAI, 2003.
[11] Jiaming Guo, Rui Zhang, Xishan Zhang, Shaohui Peng, Qi Yi, Zidong Du, Xing Hu, Qi Guo, and Yunji Chen. Hindsight value function for variance reduction in stochastic dynamic environment. In ICAI, 2021.
[12] Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors. NIPS, 2017.
[13] Danijar Hafner. Benchmarking the spectrum agent capahilities. ICIRR, 2022.
[14] Zhen He, Jian Li, Daxue Liu, Hangen He. and David Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. CVPR, 2019.
[15] Mattco Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. AAAI, 2018.
[16] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjia Ahn. Scalor: Generative world models with scalable object representations. In ICLR, 2020.
[17] Ken Kansky, Tom Silver, David A. Mély, Mohamed Eldawy, Miguel Lázaro-Gredill, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, D. Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. ArXiv, 2017.
[18] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. AAAI, 2021.
[19] Zhihan Li, Youjian Zhao, Haowen Xu, Wenxiao Chen, Shangqing Xu, Yilin Li, and Dan Pei. Unsupervised clustering through gaussian mixture variational autoencoder with non-reparrmeterrize variational inference and annealing. In 2020 International Joint Conference on Neural Networks (IJCNN), 2020.
[20] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. ArXiv, 2020.

---

[21] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold. Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS. 2020.
[22] Spandan Madan. Timothy Henry. Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how convolutional neural networks generalize out-of-distribution category-viewpont combinations. Nature Machine Intelligence, 2022.
[23] Ofir Marom and Benjamin Rosman. Zero-shot transfer with deictic object-orientedd representation in reinforcement learning. In NewrlPS, 2018.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland. Georg Ostrovski. Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[25] Jean Piaget. Piaget's theory. 1976.
[26] David A. Ross, Jongwoo Lim, Ruci-Sung Lin, and Ming- Hiing- Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 2007.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 2017.
[28] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, L.. Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018.
[29] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
[30] Rishi Voerapaneni. John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B. Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In CoRL, 2019.
[31] V. Vijay, Abhinav Ganesh, Hanlin Tang, and Arjun K. Bansal, Generalization to novel objects using prior relational knowledge. ArXiv, 2019.
[32] Haonan Wang, Ning Liu, Yi yun Zhang, Dawei Feng, Feng Huang, Dongsheng Li, and Yiming Zhang. Deep reinforcement learning: a survey. Frontiers Inf. Technol. Electron. Eng., 2020.
[33] Nicholas Watters, Loie Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerehner. Cobra: Data-efficient model-based ri through unsupervised object discovery and curiosity-driven exploration. ArXiv, 0ttss/995.09975. 2019.
[34] Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Benchmarking unsupervised object representations for video sequences. Mach. Learn. Res., 2021.
[35] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: highly modularized deep reinforcement learning library, arXiv preprint arXiv:2107. 14171, 2021.
[36] Guangyu Robert Yang. Madhura R. Joglekar, H. Francis Song, William T. Newsome, and Xiao-Jing Wang- Task representations in neural networks trained perform many cognitive tasks. Nature Neuroscience, 2019.
[37] Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-supervised reinforcement learning with independently controllable subgoals. ArXiv, 2021.
[38] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with object-centric representations. In ICLR, 2021.
[39] Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert. Timothy P. Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston. Razvan Pascanu, Matthew M. Botvinick, Oriel Vinyals, and Peter W. Battaglia. Deep reinforcement. learning with relational inductive hiases. In ICLR 2019, 2019.
[40] Guangxiang Zhu and Chongjie Zhang. Object-oriented dynamics predictor. In NewrlPS, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Partially.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "hC15WMEZm8P", "reviewer": "Reviewer_ZsUk", "summary": "This paper proposed to learn object representations that can utiltize the category informations for perception and reasoning in reinforcement learning.\nExperiments on two RL benchmarks show clear improvements of the proposed method over previous RL methods.\nThe ablation studies also cover a lot of aspects of the proposed method and show clear improvement of the each part of the proposed method.\n", "questions": "\nQ1: Some technical details about the SPACE method for unsupervised object discovery is not quite clear, I would \n\nQ2: The unsupervised object discovery module is trained on images generated from a random policy, it would be interesting to see if using a better policy to generate the images used for training the object discovery module could improve the performance, for example, using the PPO policy to generate images.\n\nQ3: Is there other tasks that are larger than Crafter and Hunter in terms of the image size and action space? I think given the impressive performance of the proposed method, it is reasonable to see if the proposed method can work on larger scale problems.\n\nQ4: There are other object centric representation learning method that could be discussed, such as slot attention[R1].\n\n[R1] Object-Centric Learning with Slot Attention, NeurIPS 2020\n", "limitations": "I think the major limitation of this paper is that the evaluated tasks are only two, I would expect to see more tasks given that the proposed method works so well the task tested.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nS1: The idea that object representations should encode category informations is interesting and IMO important for build generlizable RL methods.\n\nS2: In the evaluation of this paper, the proposed method shows a very clear and strong improvements over previous methods.\n\nW1: The tested benchmarks seems to be limited, I am wonderring if there are some larger scale benchmarks to test the proposed method?\n\nW2: The description of the proposed method is not quite clear, I would recommand the paper to at least include some preliminary on SPACE in the main paper or the supplementary instead of asking the reader to refer to the original paper, IMO SPACE is an important component of the proposed method, so it is better to describe it in the main paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XtBfK4oQ3o2", "reviewer": "Reviewer_B8LM", "summary": "The paper proposes augmenting Object Oriented RL approaches with additional information in the form of object category. Specifically, they use a module which uses unsupervised clustering to identify object categories from the output of a (previously known) unsupervised object detection algorithm. This is additional info (predicted category) is then incorporated using a convolutional encoder (OCAP). The object-category aware information from this module is passed into the third module which contains category-specific neural networks, which finally returns the action probabilities and values.", "questions": "1. Is the number of classes (C) a known, required parameter? Was this known for the games, or not? \n\n2. Why were Crafter and Hunter domains chosen? Were there any specific properties of these domains that made them harder/easier for OCARL?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\n\n1. The motivation is clear and well represented in the paper. The paper does a good job of placing their contributions in the context of existing works.\n\n2. OCARL does generalize better than standard OORL approaches. \n\n3. The methods section presents approach in detail, and should be reproducible for future researchers.\n\n4. Ablation study is solid and gives a sense of which module helps with the performance bump. \n\nWeaknesses:\n\n1. Missing literature: The work cited below [A,B] seem closely related to some work on OOD generalization to novel combinations. In fact, experiments in [A] might also explain why OCMR helps generalize when using separate category-specific neural networks. Would be good to include a line or two discussing how OCMR connects to this work. \n\nA. https://www.nature.com/articles/s42256-021-00437-5.pdf\nB. https://www.nature.com/articles/s41593-018-0310-2\n\n2. Might be good to clarify why the two domains mentioned in the paper were chosen. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V3wlNUiUyq4", "reviewer": "Reviewer_dKY1", "summary": "This paper proposed a object-oriented reinforcement learning method, wihich consists of three parts: Category-Aware UOD, OCAP and OCMR. Category-Aware UOD mainly provides category supervision information for OCAP, enabling OCAP to perceive objects in the environment more accurately．OCMR learns the interactions between objects in a self-supervised manner to better predict action probabilities. Through the recognition of object categories in the environment and the learning of interrelationships, the model can learn better action strategies. It is verified that the proposed model can better perform reinforcement learning tasks in two experimental environments.", "questions": "1) Why do you need to take the maximum value of Z^out and Z in Eq. 6?\n2）In the OCAP module, it is surprising that different objects can be segmented unsupervised from the environment image only by a simple convolutional neural network. Can you explain why this works?", "limitations": "1) The proposed method does not consider the experimental environment with occlusion between objects, which is unreasonable.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1）It is an interesting strategy to help the agent better perform reinforcement learning tasks by identifying the types of objects in the environment.\n2) The experimental results show that the proposed method is effective.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LxMBHufC0-j", "reviewer": "Reviewer_2Frs", "summary": "The paper proposes learning object categories in order to enhance object-oriented RL (OORL). While past work on OORL has shown the benefit of extracting objects from a scene before learning with RL, this work has not considered learning to cluster objects into related categories so that the policy can treat similar objects the same. This work uses an existing Unsupervised Object Discovery (UOD) method to discover objects, but then learns a clustering of the resulting objects. It then asks the RL network to not only be able to predict the category of objects (forcing the representation to learn about object categories), but it also uses the object categories to apply independent category-specific networks to each object, which the authors argue improves modularity and therefore generalization. The paper shows results in two gridworld domains which represent complex sequential decision making tasks, and show significant improvements over relevant baselines and ablations. ", "questions": "If you can explain the difference between RRL and OCARL and clarify the issues I have raised above I may be willing to increase my score.\n\nWhy were most of the results shown in Hunter rather than Crafter? Crafter is a complex unsolved domain so it is of significant interest to the community. I would suggest moving Figure 8 from the appendix into the main text if possible, since those results are compelling.\n\nI would suggest removing the italics such as \"OCARL achieves better performance on *all* tasks\" and \"is the *only* method\". This gives an appearance of a lack of objectivity that is not appropriate for a scientific paper. \n\nI suggest adding a sentence after line 89 to make it clear how OCARL is distinct from SMORL.\n\nThe paper contains several English-language errors that should be proof-read and corrected. A non-exhaustive list:\n- Line 23: \"To deal with these limitations, OORL is a promising way\" -> \"OORL is a promising way to deal with these limitations\"\n- Line 25: invariant -> invariance\n- Line 29: \"yield in low generality\" doesn't make sense. \n- Line 158: concatenate -> concatenated \n\nI appreciated the cognitive science justification for object awareness that the authors provided.", "limitations": "The authors acknowledge the limitations of their clustering approach in line 124-126, but did not otherwise acknowledge the limitations of their method. I would suggest acknowledging that the didn't test generalization to novel instances of the same object category, as I explained above.\n\nIt does not appear that the authors included a discussion of the societal impact of their work. Perhaps they can talk about the benefit of improved object-oriented RL to the potential for building better robots, and the consequences thereof. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n**Originality**: The idea of learning categories of objects, and a policy that treats objects in the same category in a similar way, is both compelling and novel. The ablation studies reveal that the paper essentially makes two novel algorithmic contributions based on object category awareness that improve performance: OCAP (predict object cluster labels), and OCMR (having different modular independent networks for different object categories). \n\n**Significance:**\nThe paper benchmarks against two relevant OORL baselines (RRL and SMORL), and a reasonably advanced RL method (PPO), and shows clear and significant performance gains above these methods in complex sequential decision making tasks including Crafter (which is based on MineCraft). Although these environments are gridworlds, crafting is a complex problem and it is difficult for conventional methods to obtain high rewards. The idea of reasoning over object categories is likely to be useful in a broader range of tasks (e.g. object manipulation). The generalization performance is also interesting. \n\n**Quality**: \nThe design of the network architecture proposed in the paper appears to be complex but thoughtful; the authors carefully justify each design decision (e.g. why the network should predict the object category as an objective, rather than use it as input, so it can be robust if the UOD module fails to extract the right category). The ablations also back up the claims that each component is necessary. \n\n**Clarity**:\nThe description of related work is clear and gives a good overview of the field while drawing the distinction with this work.\n\n### Weaknesses:\n**Significance/Quality:**\nIn spite of the fact that experiments were conducted in Crafter, most of the analysis for the paper is shown in Hunter. Crafter is a much harder environment of more significance to the community, so it would have been nice to see learning curves and ablations in this environment as well.\n\nFurther, although the paper tests generalization, it does so in a pretty limited way, because in no experiment is the agent ever tested on a truly novel object (only on differing combos of zombies and cows, but it has always seen at least one zombie or cow previously). The current paper actually does not test generalization to novel instances of the same category. It would be interesting to extend the experiments to another domain (one example could be simulated robotic manipulation of objects on a tabletop), and test whether the method can truly generalize to new instances of the same category. For example, if the agent had trained to pick up a red cup and a blue cup, could it generalize to picking up a green cup? \n\n**Clarity**:\nThe biggest weakness of the paper is clarity. In addition to smaller grammatical errors, there are some places where explanations significantly detract from clearly understanding the paper. The single biggest issue is in the method/results. Line 242-243 states that OCARL with p=1 is exactly RRL (where p is the probability that the UOD module can't detect a category). If this were true, this would significantly detract from the novelty of the paper. However, a few sentences below in lines 253-255 the paper directly contradicts this statement, by pointing out that OCARL uses the OCMR module, while RRL uses a universal network \"instead of multiple object-category-specific networks like OCARL does\". One of these two statements must be false, and I believe it is the one about the exact equivalence with RRL. This should be clarified, and it would be good to further explain in the related work or methods how the architecture being proposed is different than RRL.\n\nThe modularity experiment in Section 4.3.3 seemed interesting, but it was so hard to follow lines 272-276 I was not sure what the experiment actually showed. Is there a table or figure describing the results?\n\n A non-exhaustive list of other clarity issues: \n- There are so many acronyms in the paper that it is hard to follow what's actually going on. Rather than OCAP and OCMR, it might be nice to use a phrase like \"category modules\" to cue the reader. \n- In Section 3.1, it is not clear what $z_{ij}^{what}$ actually is... is it a continuous embedding? A categorical label?\n- Line 106 appears to state that $z^{fg}$ consists of a fixed number (H x W) of object representations, but the related work section states that this is only a drawback of spacial mixture models, and not SPACE (which uses a spatial attention module for the foreground).\n- Lines 100-101 contain an out of place detail (we run a random policy to get data and apply SPACE) that seems premature given the environment has not been introduced at this point. It is hard to evaluate whether a random policy would be able to collect enough data to cover all object categories without knowing about the environment. This also does not seem to be part of the *method*, per se, but rather an implementation detail.\n- The explanation in lines 150-157 is unclear. \n- Can Equation 5 be interpreted as applying attention weights based on the category probabilities predicted from Z? The explanation / rationale should be improved here. \n- Is OCARL optimized with PPO as well? This should be clarified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LHTZMomMW0K", "reviewer": "Reviewer_1n95", "summary": "This paper proposes a new framework applied to Object-oriented reinforcement learning (OORL) called Object-Category Aware Reinforcement Learning (OCARL), which aims to explicitly model the similarity between different objects of the same category. It consists of three main components: 1) UOD: unsupervised learning method (SPACE+IncrementalPC+KMeans) is used to complete the identification and clustering of objects; 2) OCAP: the clustering results are used as supervised learning signals to guide the encoding of objects; 3) OCMR: a self-attention mechanism is introduced, and different category objects use independent networks to complete the reasoning process. The experiments show that OCARL can improve the sample sampling efficiency and generalization ability of the model.", "questions": "1. Are there experiments that show it is possible to continue training the UOD in the RL phase?? It would make the results stronger and less reliant on a good random exploration strategy.\n2. When the number of preset categories is greater than the actual number of categories (e.g. Hunter preset category number is greater than 4), is there an experimental display?\n3. The clustering results of unsupervised learning are shown in Appendix C, but I am more interested in the representation of the latent variable $Z$. Is it possible to show the prediction results of $f_{cat}$ in OCAP?\n4. Does the method only apply to environments, which can be divided into grids? In line 105, the image is divided into $H\\times W$ cells, and each cell corresponds to an object. if the object is irregular or of a different size, how to handle it?", "limitations": "The paper doesn't really discuss its own limitations explicitly. Adding a section would be quite helpful. OCARL requires additional information about the environment (the number of categories), which may not work well if the difference between the number of preset categories and the actual number of categories in the environment is large.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n\n1. The paper contributes to improving object representation learning in model-free RL which has practical applications in object-oriented RL.\n2. Compared with existing methods, OCARL shows outstanding generalization ability.\n3. The paper is well written and well organized. \n\nWeaknesses：\n\n1. The paper gives an implementation that combines unsupervised learning, supervised learning, and reinforcement learning. Each of the sections chooses a classic implementation, but the reasonableness is not well illustrated and there is no experiment showing the advantages compared with other implementable methods.\n2. The paper points out that category information improves exploration efficiency and generalization, but there is no essential reason for the improvement (derivation, proof, etc.), nor is the condition and scope of its use stated.\n3. The UOD method uses data derived from the random exploration, but random policies often fail to explore the environment effectively and may result in some of the objects not being discovered.\n4. This paper uses KMeans for clustering operation, so the number of categories k is the hyperparameter. In practical scenarios, we usually do not know the specific number of categories of objects explicitly, so there may be cases where the preset number of categories is greater or less than the actual number of categories. This paper only implements the \"less\" situation in the Crafter environment without considering the case where the preset number of categories is greater than the actual number of categories.\n5. Generalizability requires that the model be trained and applied to new data or new environments. However, the UOD is trained using the same environment as the test.\n6. There may be a formatting error at the top of page 5, where the first two lines have no line number and the formula is not coded, resulting in an error in the formula reference in line 259.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept