Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, a new representation learning that considers the relations between environmental objects is proposed, then a relation graph of objects is used to estimate the dependencies. Finally, the agent selects subgoals and learns to achieve them and also minimizes the influence on other objects. Experiments on locomotion tasks show the proposed method achieves higher performance compared with several previous HRL methods and representation learning methods.
strengths_and_weaknesses: Strengths:

The relations of objects have been investigated in MBRL, object-oriented RL, but from the perspective of HRL, it seems to have few works currently, so this paper is interesting and novel to some extent.

Experiments show some promising results.

Weaknesses:

The authors claim previous works' strong assumption of fully independent objects, while the experiments do not well support this claim.

The literature review covers many related works, but some recent works are missing, such as [1-3], where [1,2] performs better than baselines in this work.

Some formulations are not explained well. Such that the function d_{rel-eff} and d_{rel-press}, what objective ensures these functions have the effect?

I am not sure whether this kind of interaction graph restricts the scenarios to locomotion tasks. Since many atari games, some objects are not task-irrelated, so learning a subgoal as well as minimizing the effects on other objects seems not that important.

1. I2HRL: Interactive Influence-based Hierarchical Reinforcement Learning.
2. Learning Subgoal Representations with Slow Dynamics.
3. DisTop: Discovering a Topological representation to learn diverse and rewarding skills
summary_of_recommendation: Based on the review above, I give a weak accept to this paper, I think some improvements should be further made, such as the clarity of some formulations, and more analysis to support the claim and motivation, more recent baseline comparisons.
issues: See the weaknesses.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an object-centric self-supervised reinforcement learning (RL) method in multi-goal RL settings. The major problem in prior works is that they assume each object in the environment is independently controllable without interference, which prevents the skills from being combined to solve compositional goals in realistic environments. To lift this limitation, the authors propose to learn the interaction graph between objects by recurrent graph neural network (GNN). Based on this graph, they derive a sequence of subgoals that can be controlled independently without interference, for the agent to execute. Also, they add selectivity rewards that encourage the agent to maximize the influence on the goal object while minimizing the interference to the other objects. The experimental results show that the proposed method outperforms several baselines including the recent object-centric RL and typical RL methods in multi-goal RL settings. Furthermore, the paper shows that the proposed method can generalize to unseen compositional goals.
strengths_and_weaknesses: Pros:
- The insight in this paper is important. Typical multi-goal RL and hierarchical RL approaches overlook the potential compositional structure and interference between objects, which limits the generalization and data efficiency. The learned interaction graph is useful for decomposing a compositional goal into several pieces of independently controllable subgoals for expediting the learning process.
- The selectivity reward is an interesting design from the self-supervised RL point of view.
Cons:
- Ordering the subgoals by the interaction graph is a sound approach to derive a sequence of independent subgoals.

Cons:
- The major issue of this paper is notation. There are too many notations in this paper without a systematically or smooth introduction to them. For example, in Section 3.1, there are many similar notations (e.g., d_oo, d_o, d_pred, d_rel-eff), but it's hard to see their relation unless you read this section multiple times. I would suggest either having a list of notations with brief explanation or plotting a figure to depict their relationship. I believe that the readers can digest these materials better.

- Lack of intuitive explanation or high-level overview before explaining the details. For example, I have no idea why do you need to estimate set $P^i$ in all paths from the action variable to object $i$? What is action variable to object $i$? Besides, I'm confused about the difference between $g^i$ and $(s^{goal,i}, P^i)$. I agree $s^{goal,i}$ is a subgoal, but why do we need $P^i$ for defining the "independently controllable subgoal"?

- Lack of baseline explanation. As SMORL is an important baseline, I would like to know more about how is SMORL different from the proposed method? For now, the paper just said SMORL uses object-centric representations and reward constructions. 

summary_of_recommendation: The proposed idea is interesting and technically sound. Though the clarity and readability need lots of works, I think this idea is worth presenting at the conference. I would adjust my ratings if the author can improve their writing.

---

Post rebuttal:

The author's response addressed my questions. I've raised the score.
issues: - How is selectivity compared with empowerment rewards? 
Forrester, Russ. "Empowerment: Rejuvenating a potent idea." Academy of Management Perspectives 14.3 (2000): 67-80.

- In Equation 2, what's the purpose of $p(. | s_t, a_t) = \mathcal{N}(s^{j, where}_{t+1}, \sigma^2I)$? I didn't see it in other parts.

- In Equation 5, $s^{j, where}_{t}$ should be $s^{j, where}_{t+1}$?

- What is the goal space in all experiments? Is it a joint space of all objects? I suppose the baselines should use the joint space; otherwise, they cannot achieve the compositional goal. What I mean by joint space is a vector that concatenates all objects' information (e.g., positions) to a 1-D vector. I ask this question because the proposed method seems to be able to use single object information as the goal representation because the agent just needs to know a subgoal that corresponds to a single object. For a fair comparison and comprehensive study, one should use the same goal space or test both goal spaces.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an approach that can utilize object-centric representations for solving complex goals by decomposing them into sub-goals. In particular, first the dynamics of an environment are captured in an interaction graph between different objects (using their object-centric representation such as their identity and positions). After that, various sub-graphs of the interaction graph are considered to be sub-goals. The sub-goals are ordered by the depth of their sub-graph to ensure that objects that have a lot of dependencies are attempted first, such that the corresponding sub-goals are not destroyed while attempting other sub-goals in the future. In addition, the robot is incentivized to only manipulate objects within the current sub-goal through a selectivity reward. Recurrent Graphical Neural Networks (GNN) with GRUs are used to represent the interaction graph. The goal-conditioned policy used to reach the sub-goals is trained using SAC. The method is evaluated on a simulated multi-object environment with up to 4 objects and some objects having relations between each other. It is shown to outperform a previous method SMORL, which also uses object-centric representations, a goal-conditioned baseline based on SAC with hindsight experience replay (HER), and a hierarchical actor-critic baseline.
strengths_and_weaknesses: Strengths:
- Decomposing difficult problems into sub-goals is important for scaling up robot learning. Using object-centric representations for representing sub-goals enables a significant reduction of the problem dimensionality. 
- Encouraging interaction of objects only within sub-goals and ordering sub-goals by the depth of their dependency graph is important for a structured exploration without undoing previously achieved sub-goals.
- The paper is well-written and easy to understand and follow.
- The method is shown to outperform baselines in a simulated multi-object environment.

Weaknesses:
- Right now experimental evaluation only includes manipulation of a small number of simple objects. It would be interesting to see more experiments on more complicated objects and higher-dimensional environments.
- Presented method requires collecting robot interactions in order to pre-train the object interaction graph. It would be useful to know the amount of data used for pre-training to better compare it to other methods, as other methods such as SAC with HER do not require pre-training.
- The objects in the experiments are represented by their identity and position in the environment. It would be interesting to see how the method works with more examples of object representations, for example the ones that include velocity for more dynamic tasks.
summary_of_recommendation: The presented method is a great step towards a better structured learning of sub-goals. However, the scope of experiments would need to be extended in order to better understand the impact of the method.
issues: Please refer to "Strengths And Weaknesses" for a detailed list of issues.
review_title: null

Meta-review: The reviewers are generally positive about the idea and contribution, and agree that it is important to scaling up robot learning. Post-rebuttal, they are all weakly positive. 
Decision: Accept (Poster)