Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary
The paper proposes to use object-centric representations for RL, which can efficiently handle multiple objects in the scene. To learn a policy that can take a variable number of object observations, the paper proposes the goal-conditioned attention policy, which can focus on objects of interests to achieve each sub-goal, and thus reduce the combinatorial complexity of multiple objects. The goal-conditioned attention policy can be efficiently trained with hindsight experience replay on the object-centric goal representations. The experiments demonstrate the superior performance of the goal-conditioned attention policy on dealing with multiple objects.

### Strengths
- The idea of learning composable object-centric visual representations and goal-conditioned attention policy is an intuitive and plausible way to tackle combinatorial challenges in multi-object manipulation tasks.
- The experiments with ground truth states show that the proposed goal-conditioned attention policy can effectively handle multiple object manipulation tasks. 

### Weaknesses

- Based on Figure 4, any of the methods without ground truth states solve the tasks. Although the proposed method shows better learning performance in the Visual Rearranging task, the improvement is marginal to claim the proposed method can solve the tasks.
- Why only up to 2 objects are considered in Figure 4? The proposed method has the advantage of dealing with multiple objects but did not show the benefit. It would be more convincing if the proposed method can reasonably deal with more than 2 objects.
- The baseline could include recent visual policy learning methods using data augmentation [1,2].
- The paper claims "Self-supervised RL" but it is not clear which part of the method is trained with self-supervised learning. The proposed method seems to consist of unsupervised representation learning and reinforcement learning.
- The proposed method assumes that the sub-goals are independent of each other but it is not true in many cases, e.g., collisions between objects. 
- One of the claims in the paper is that the proposed representations and policy can work with a variable number of objects but the experiments do not cover this setup.

### Questions and additional feedback
- It would be better to include the architecture of the visual encoders for baselines and the proposed method.
- It could be good to show the quality of learned representations, such as object pose prediction and classification.
- The training time might be too short. The proposed method can be trained more (e.g., 1e6 environment steps).
- The website link is provided but nothing is there.
- How does the policy decide when to switch to the next sub-goal? Including how to rollout an episode toward sequential sub-goals with the proposed model would be helpful.

### Overall assessment
The proposed method is intuitive and tackles an important problem of multi-object manipulation. However, the experiments and results are not yet convincing to claim the advantage in dealing with multiple objects. Overall, the reviewer thinks the paper requires more thorough experiments and is not ready to be published.


[1] Laskin et al., Reinforcement Learning with Augmented Data

[2] Kostrikov et al., Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
review_title: Interesting combination of object-centric representations and RL but insufficient experimental results

final_score: rating: 7: Good paper, accept
review_content: This work proposes to use object-centric unsupervised representation learning for self-supervised goal-conditioned RL, as opposed to prior work that assumes no particular structure on the learned representations (eg. VAEs). The proposed method, self-supervised multi-object RL (SMORL), uses the SCALOR architecture from prior work, then modifies the policy representation with single-object attention and also the reward function in RL with imagined goals (RIG). The results show that the method can learn simulated pushing and rearranging tasks in a self-supervised way with up to 4 objects in the scene, and outperforms RIG and Skew-Fit on pushing tasks. The proposed method is sufficiently novel, explores an important direction for self-supervised learning, and the results are quite strong.

The main motivation argued is that in more complex environments, it is difficult for fully unstructured reconstruction-based representation learning methods such as VAEs to recover a disentangled representation. This then causes difficulties running self-supervised RL algorithms such as RIG, which use the representation to compress the input, to set meaningful exploration goals, and for evaluating the reward. Using object-centric representations makes the representation more disentangled and improves RL, as demonstrated by the results. The paper would be strengthened by direct analysis of this hypothesis: for instance, correlations or measures of disentanglement between ground truth state and learned representation for SCALOR vs. VAE in the environments tested, particularly as number of objects increase.

The key novel contributions lie in how SCALOR is integrated with self-supervised learning. First, after learning the SCALOR representation from data, the proposed policy uses an attention mechanism to pay attention to reaching the goal for a single object at a time. One detail I did not understand was how the policy operates at test time. At training time, exactly one z^where is changed and the policy attempts to match that object. At test time, potentially many z^where could be different, so do you cycle between all the objects? Also, the paper would benefit from an ablation where you explore the choice of attention architecture; you could imagine that simply learning the object-centric representation and treating it as a flat representation like a VAE could also provide gains, so it is important to disentangle that effect from the novel policy architecture. The policy contribution is evaluated in Figure 3, where the success of SMORL+GT shows that the architecture makes manipulating a large number of objects possible.

The other differences to RIG have to do with goals and rewards. During self-supervised training, goals are sampled by sampling a new z^where for a single object, encouraging manipulation of exactly one object. This proposal seems logical, although in the long run it could potentially be an assumption that would not scale beyond object repositioning tasks. The reward is also modified to use the SCALOR latent, to penalize distance to the closest z^what object as the current goal with a threshold alpha for detecting the matching object. Again, the proposed reward function would be better evaluated with an ablation where you use the original reward from prior work -||z - z_g||.

The experiments show that the proposed method SMORL outperforms RIG and Skew-Fit on visual pushing tasks with many objects (and “rearranging”, which is pushing with random initial positions of objects - having both sets of experiments potentially seems a bit redundant since it seems rearranging is strictly more difficult). SMORL is worse than an oracle (SAC+GT) which uses ground truth state information, but it seems to tend towards the oracle performance on even the more difficult tasks (seeing longer versions of the learning curve that show asymptotic performance would help understand this better).

Generally, the results on multi-object manipulation and self-supervised learning are strong. Further experiments as mentioned above would better allow the contributions to understood independently.

Minor comments

The related work is a bit thin and narrow - it addresses the nearest-neighbor works well but does not address self-supervised methods and robotics methods more broadly; it would be best to use the related work to make the paper more understandable to the broader community who are not embedded in goal-conditioned RL (and potentially put it in at section 2 instead of at the end).

Page 6: “Out code” -> “our code”

“In general, we expect that it is beneficial for the policy to not always attend to entities conditional on the goal; we thus allow some heads to only attend to additional learned parametric queries (left out above for notational clarity).” - did not understand this, it would be good to explain further what type of information this would include, and perhaps describe formally in the appendix.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper combines object-centric representations and self-supervised HER goal-conditioned policy learning to learn efficient RL policies for a robot manipulation task.

They use SCALOR as an object-based state representation, and use it to propose semantically meaningful goals for a SAC policy to achieve. They can then leverage this on new tasks to solve them efficiently..

Overall, I found this paper very interesting, clearly written, well executed (with very sensible decisions throughout) and presenting several good ideas, especially in how to leverage the additional object structure effectively. It demonstrates good early results in the novel field of Object-oriented RL.

I have a few comments/questions:

1. The explanation of how the goal-conditioned policies were trained was very clear, and I especially like how you use z_what and z_where to construct novel meaningful goals (which will tend to just force the policy to move objects around, but that is a good prior for your environment!). However how the “evaluation on a novel task” is done wasn’t as clear (i.e. when we try to implement a given goal or sequence of goals). More precisely, it is said in several places that the goal z_g is decomposed in sub-tasks where only one of the slots is used as the target. 
   1. Could you provide more details on exactly how that is done? Do you learn p(z^where) on task later?
   2. What happens when some of the “objects” aren’t achievable or controllable? E.g. I’d expect that you see a slot which represents the robot arm, is this treated differently?
2. How good are SCALOR representations in your environment? 
   1. It would be very helpful to show samples / traversals in the Appendix.
   2. Similarly, comparing to the GT information you provide would be interesting (e.g. try to decode it? I understand you’d have to match the slots up unfortunately)
   3. Did you try continuing training SCALOR in the RL phase? It would make the results stronger and less reliant on a good random exploration strategy.
3. Did the hard matching cause issues while learning? I’d guess that the argmin is not too problematic because it is used in the reward computation only, but if you’d consider extending this setting to learning the goal proposal function z_g, this seems like a limitation?
   1. The explanation about the issue of using tracking as part of the model directly wasn’t especially clear to me. It might deserve a bit more expansion, especially in the Appendix?
4. How complex are the observations of the environment?
   1. Could you add more samples of the environment’s observations?
   2. It seems like the environment chosen is extremely similar to the standard Gym Fetch environment, did you try using it instead? https://gym.openai.com/envs/FetchPickAndPlace-v0/ 
5. It is not entirely true that MONet/IODINE “do not contain disentangled and interpretable features like position and scale”. 
It is true that they are not explicitly enforced (like done in SCALOR), but they do arise quite easily purely unsupervised. 
Especially, in my experience with both of these models, obtaining (and identifying) “position” latents is rather easy. See for example Figure 5 in [1] and this animation [2].

So in summary, I believe this is a strong paper in a budding field, which deserves publication at ICLR and may interest many people there.

* [1] https://arxiv.org/abs/1901.11390
* [2] https://twitter.com/cpburgess_/status/1091220207941701632 

review_title: Official Blind Review #2

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

The paper combines an existing generative world model (SCALOR, Jiang et al. 2019) with goal-conditioned attention policy. The method is evaluated on object manipulation environments based on MuJoCo (Todorov et al., 2012), Multiworld (Nair et al. 2018) and a Sawyer arm. The paper is clearly written; the authors discuss challenges and motivate their design choices well throughout the paper.

Score justification:

The paper is mostly incremental but it provides enough contributions for acceptance. SMORL (Algorithm 1; the proposed method) is well-defined and motivated. The method outperforms a strong baseline (Soft Actor-Critic with Hindsight Experience Replay) in an object manipulation task with available ground-truth state representations. In the visual rearranging task, the proposed method performs better than existing self-supervised RL algorithms: RIG (Nair et al., 2019) and Skew-Fit (Pong et al., 2020) when using 1 and 2 objects. Experiments with a larger set of objects which demonstrate the compositional benefit of SMORL would strengthen the paper. I would also be interested in a visualization of the latent object representations learned by SMORL.

Pros:

The paper contributes to improving scene decomposition and object representation learning in model-free RL which has practical applications in robotics and object-oriented RL.

There is a discussion of existing limitations and challenges (limitations of VAEs in visual RL, defining reward functions in goal-conditioned RL), and how SMORL is meant to address them (goal-conditioned attention policy to handle set inputs, incorporating goal and object representations in the reward). 

Experiments in the multi-object environments showing that SMORL might be promising with and without ground-truth representations.

Cons:

There is no discussion of the computation cost of SMORL in comparison to the baselines (SAC with HER, RIG, Skew-Fit).

The “compositional” aspect is unclear in the Experiments section. How does the “compositional generative world model” translate to productivity, substitivity or other forms of compositional generalization with respect to the objects in the image?

Experiments with a larger set of objects would help in highlighting the advantage of SMORL in a general multi-object visual RL setting.

Questions during rebuttal period: 

Please address and clarify the cons above  

Typos and structure:

The link in the abstract leads to an empty page

Equation 1: G (distribution over the set of goals) is once bolded, once in italic

It seems there are two different citation styles used in the paper: (Jiang et al., 2019) and Jiang et al. (2019)

Algorithm 1, line 1: typo “sequences data”

Section header Experiments can be moved to the next page

Typos: “our method scale challenging tasks”, “out code”, “objects identities”, “2 object”, “3 object”, “4 object”
review_title: Object centric learning using SCALOR in goal-conditioned, model-free RL 

Meta-review: This paper proposes a self supervised learning algorithm to compute object-centric representations for efficient RL in the context of robot manipulation tasks.

The key idea is to learn an object-centric representation (using prior work on SCALOR) and use this to intrinsically generate goals for a SAC policy to achieve. The policy is a goal-conditioned attention policy. The evaluation metric is a set of tasks to manipulate objects for a visual rearrangement task. 

${\bf Pros}: $
1. The baselines are reasonable and consist of other unsupervised RL algorithms in recent literature. 

2. Object-oriented RL is a growing area of interest and this paper proposes a reasonably novel and validated set of ideas in this domain. I believe it will be of significant interest and potentially make an impact on research in robotics and deep reinforcement learning.

3. The goal-conditioned attention policy can handle realistic scenarios, namely -- multi-object manipulation tasks

4. The attention mechanism also provides a reasonable solution to mitigate combinatorial hardness in multi-object environments

${\bf Cons}$: 

1. Some of the reviewers felt that the experimental results from pixel inputs could have been pushed further. However, since the setup and algorithm is relatively novel, there are already many moving parts and this paper seems like a step in that direction

2. Experiments with larger set of objects would have been interesting to investigate and report. 

   
Decision: Accept (Spotlight)