Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
This paper uses asymmetric self-play to train a goal-conditioned policy for robot manipulation tasks, which can also generate curriculum automatically and generalize to unseen goals and objects. The experiments contain various challenging manipulation tasks, where the proposed method outperforms all the manually designed curriculum baselines.

Pros:
+ The paper is well-written and easy to follow. 
+ The goals and skills discovered by Alice interesting. The proposed method can generate some meaningful goals which work better than explicit curricula.
+ The novel shape/geometry manipulation results are promising.

Questions & Concerns:
- Lack of important baselines. The author uses PPO, but this framework will still support off-policy learning. This adversarial training idea is very similar to CER which is not cited. 
  1. (HER) Hindsight Experience Replay
  2. (CER) Competitive Experience Replay (not cited): [1]
- This work is also related to HRL methods, so it needs to include more HRL works in section 4, e.g. HAC[2], HIRO[3]
- (minor) In the "Flip" environment, success is also related to the orientation of the box. What's your rotation representation and the corresponding distance to measure "success"?
- (minor) Are there any exploration strategies in Alice?

In general, this paper proposes to use asymmetric self-play to generate a curriculum for the agent, which is better than the explicit one. However, since there is no comparison with HER-based methods (or HRL), it's hard to justify whether asymmetric self-play is necessary for these kinds of problems.

[1] Liu, Hao, et al. "Competitive experience replay." ICLR 2019

[2] Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." ICLR 2019

[3] Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in Neural Information Processing Systems. 2018.

======== POST REBUTTAL RESPONSE========

After reading the feedback and revision of the paper, most of my above concerns are addressed. I agree that the comparison with HER/CER is not fair. I also notice that the author added more references and details based on all 4 reviews.  Thus, I decided to improve my score on this paper.

review_title: Review for paper "Asymmetric self-play for automatic goal discovery in robotic manipulation"

final_score: rating: 7: Good paper, accept
review_content: This paper presents an approach to learn goal conditioned policies by relying on self-play which sets the goals and discovers a curriculum of tasks for learning. Alice and Bob are the agents. Alice's task is to set a goal by following a number of steps in the environment and she is rewarded when the goal is too challenging for Bob to solve. Bob's task is to solve the task by trying to reproduce the end state of Alice's demonstration. As a result, the learned policy performs various tasks and can work in zero-shot settings.

Strong points: 
- This paper proposes a novel way to generate the goal distribution for training on multiple goals that is rich and scalable. The authors describe the properties of the goals as being achievable and not labor intensive.
- The experimental results are very encouraging.
- The analysis of the method helps to understand which components are important.
- The evaluation on the hold-out tasks is very impressive and pushes the state of the art.
- The paper is well written and very easy to follow, the illustrations are informative and appealing. 
- Although this approach is based on previous work on asymmetric self-play, the authors clearly describe the contributions of this work (training clearly from self play, zero-shot generalisation).

Weak points:
- Alice and Bob agents need to perform the task from the same initial state. This might pose some problems when training with a real (not simulated) robot environment.
- The way how goal-conditioning reward is assigned assumes that what matters is the absolute positions of all the objects in the environment. This means that such relative concepts like, for example, "objects being close to each other", cannot be handled in the current framework.
- Alice Behavioural Cloning is currently incorporated into learning by combining the RL and BC losses. Another common way to incorporate demonstrations is to include them in the replay buffer for training a policy. Would it be possible in this method? As this component is crucial for the success of the method, it might be worth investigating the alternatives.
- The baselines of using generative modelling for goal generation are mentioned, but there is no comparison with them. Would it be possible to apply those methods to the studied environments and settings? 
- It seems that every policy includes the state as part of its observation. What does this state include? Why can't the policy be trained directly from vision? Does including the state mean that the learning procedure is not applicable to real environments?
 
I am leaning toward the acceptance of this paper because I find 1) the method interesting and well motivated, 2) experimental results very encouraging and zero-shot generalisation quite impressive. While I still have some concerns (learning from states, starting from the same initial condition), I believe that this paper advances the knowledge and would be beneficial for the research community.

Questions:
- "Multi-goal" game: the experiments show that it improves the results, but I still cannot understand why it is crucial. Does it help because of several goals or because of longer episodes? Currently, it is repeated 5 times, what happens if it is less/more and each game is shorter/longer?
- What happens with multi-goal during testing? Are the tasks split into stages, or the agent solves only for one goal?
- I am a bit confused by step 3 where Bob's reward is computed. It sounds to me now that if Bob fails in the first game, it fails in the whole episode, is it so?
- The policies are not trained directly on the hold-out tasks, but I am wondering, how many such goals (for example, for stacking tasks) appear by accident when Alice sets the goal.
- What about the data complexity of the proposed approach? How many goals did Alice generate, how many steps in each?
- In Fig. 6, why are the results different with a single block and two blocks?


Additional comments:
- I am a bit confused about the content of footnote 1.
- I couldn't see some of the strategies in Fig.4, is there a way to make them more visible?

===Post rebuttal===

I would like to thank the authors for the detailed response and clarification of my questions. I believe that this paper will be valuable for the ML community.
review_title: Goal setting and problem setting agents are brought together to achieve zero-shot generalisation

final_score: rating: 7: Good paper, accept
review_content: This paper presents a self-play approach to learn a goal-conditioned policy for robot manipulation. The Alice and Bob approach is able to generalise to unseen objects and results in a natural curriculum.
The paper is well written and well structured, the proposed approach is framed within the relevant literature and claims are supported by experimental results.
Experimental evaluation shows that the proposed approach outperforms other methods on several manipulation task. Ablation studies complete the analysis of the proposed approach and help understanding the contribution of each part of the approach.

Comments:
- How does the proposed method compare with other goal-based approaches (e.g. multi-task approaches or goal-based intrinsic motivation exploration)?
- What is the main cause of the low success rate with increasing number of objects? What is the main limitation in the generalisation capability of the approach (Fig. 8)?
- How much does memory (used in policy networks) impact the learned behaviours (especially with several objects to manipulate)?
- Can you discuss the main limitations of your approach with respect to other methods?
- What is the expected transferability of the proposed approach to a real world scenario (e.g. with a real manipulator and real objects)?
- With a self-play approach the exploration is "limited" to what the agent can generate during self-play. How does this affect generalisation in your specific case study?
- Videos are interesting and helpful

review_title: Interesting application of self-play for robot manipulation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a variation of asymmetric self play (Sukhbaatar et al. 2018b) where:
- Bob gets to use Alice's trajectories for Behaviour Cloning (ABC) training (with some logic around when best to employ this).
- There may be several goals (reward-related) per episode and extra care is taken to avoid training on bad episodes (e.g. items didn't move or items fell off table)
- Tasks make use of a simulated robotic arm environment (however the observations include ground truth data that make the sim far from anything real)
- Some study-specific details of training methods (PPO-like) and agent network architectures (in suppl) which may be of interest.

At first glance, the movies that accompany this paper show a surprisingly high level of performance for the given task complexity.  This surprise was reduces when by the supplementary material where it is clearly explained that the observations include the full ground-truth information on the robot state and the manipulated objects: "object state observation contains each object’s position, rotation, velocity, rotational velocity, the distance between the object and the gripper, as well as whether this object has contacts with the gripper." In my view, this renders the experiments deep within the realm of toy simulations. I also feel that showing movies of vision inputs is a bit misleading because, in my view, it coveys a message that this work is on vision-based manipulation learning while in reality there is nearly full knowledge of all the items' true states.

There are multiple figures showing the merit of the approach within the designed task framework in the form of comparison with baseline methods that do not use the asymmetric-self play framework and an ablation study. ~~The one comparison with (Sukhbaatar et al. 2018b), which this study is a variation of, appears at the very end of the supplementary material and seems to show that the two methods do equally well on tasks presented in this study. This last point leads me to conclude that the variations from (Sukhbaatar et al. 2018b) in this study do not harm but also do not improve on the original asymmetric-self-play model.~~

**EDIT: per the explanation in the author feedback (not a direct comparison with ukhbaatar et al. 2018b), that conclusion is unwarranted.**

I found the paper lacking in clarity regarding both method and evaluation:
The method: 
Section 3 describes the method. It reads more like a summary pointing out the important bits than an ordered description that would allow someone to replicate the framework and model updates (which seem to all be tightly coupled in that the running of Alice and Bob, the generation of rewards and the rules around when to do what type of model update step all contribute to the framework whose product is two agents and the tasks that they set up for each-other). 
~~If the process could be described as a set of agents and interfaces and pseudocode, perhaps the reader would get a clearer understanding of how they might replicate the experiments. Note also that code is not provided.~~

**EDIT: pseudocode (and potential future link to code) added**


Evaluation:
It is clear that evaluation was performed on a large number of tasks and task variations and there is information regarding the tasks and models and parameters in the supplementary material. What I found to be unclear is - how were the experiments on hold-out tasks different from the other tasks. More specifically, in what ways is the framework used differently during hold out tasks. Is asymmetric self play used? Is Alice Behavioural Cloning applied?  If so, then in what sense are these tasks held out? If not, then how is the Bob agent trained for these tasks? The unclarity in the description of the method may have contributed to the unclarity of the evaluation section (in that it is not clear to me how the framework is modified between these task groups).

**EDIT: Thanks to clarifications in author feedback and some modifications to the manuscript I find the paper clearer w.r.t. evaluation (though still feel that reading the paper only may leave the reader confused).**

The reliance on behavioural cloning (shown to be crucial to achieving non-zero success in this study's modelling) limits the proposed method to cases where the environment can be reset which would make it much harder to implement outside of a simulation environment.

Overall, due to the issues mentioned above, I find this study to add little to the methodology and understanding of asymmetric self play as an RL method for robotic manipulation.

**EDIT: with author feedback and changes to manuscript (and supplementary) I think that the study is more interesting than expressed in my original review, however the sim-real gap with respect to applicability to robotics is still a major concern IMO. I have updated my score accordingly.**

review_title: slight variation on asymmetric self-play extended to robotics tasks (only in sim) and evaluation on hold-out tasks 

Meta-review: The paper extends previous work on asymmetric self-play by introducing a novel behavior cloning loss (referred to as ABC). The zero-shot results are impressive and demonstrate that the proposed curriculum learning approach pushes the state-of-the-art. The reviewers acknowledge these contributions. The pros of the paper are well summarized by R2, 

- The experimental results are very encouraging.
- The analysis of the method helps to understand which components are important.
- The evaluation on the hold-out tasks is very impressive and pushes the state of the art.
- The paper is well written and very easy to follow, the illustrations are informative and appealing.
- Although this approach is based on previous work on asymmetric self-play, the authors clearly describe the contributions of this work (training clearly from self-play, zero-shot generalization).

R1, R2, R5 recommend accepting the paper with scores of 7, 7, 6. R1 expressed that he is not confident about the paper. R4 recommends acceptance with a score of 6. However, R4 also expresses the concern for real-world applicability, "the sim-real gap with respect to applicability to robotics is still a major concern IMO. I have updated my score accordingly." The sim-to-real gap is a concern due to knowledge of perfect state information and the assumption of resets. 

Based on confident reviews of R2, R4, and R5, and the impressive zero-shot results, ordinarily, I would recommend the paper to be accepted. However, unfortunately, both the authors and reviewers missed a comparison to prior work, which I detail below. While the current paper makes a good case for zero-shot generalization, it does not compare to previous approaches that also exhibits zero-shot generalization. For instance, Li et al. ICRA 2020 (https://arxiv.org/abs/1912.11032) show that using a simple curriculum that depends on the number of manipulated objects + Graph Neural Networks can generalize very well to unseen tasks. E.g., the results reported in the paper demonstrate that a policy trained on 2/3 blocks generalizes and can stack many more blocks. Their policy learns to stack 6-7 blocks, whereas the paper under review can only stack up to 3 blocks (Figure 8). 

The authors mention in Section 5.2 of their paper, "The curriculum:distribution baseline, which slowly increases the proportion of pick-and-place and stacking goals in the training distribution, fails to acquire any skill. The curriculum:full baseline incorporates all hand-designed curricula yet still cannot learn how to pick up or stack blocks. We have spent a decent amount of time iterating and improving these baselines but found it especially difficult to develop a scheme good enough to compete with asymmetric self-play." 
This is at odds with results in Li et al. 

This reason for dissonance is that good generalization can be achieved by improving two separate components -- the neural network architecture or the learning curriculum. This paper shows good generalization with weak neural net architectures + a good curriculum learning method. It is unclear to me how critical the self-play method would be with a stronger architecture such as a graph network which is arguable more appropriate for the set of tasks presented in the paper. I would like to see if the curriculum is necessary (i.e., complements a stronger architecture) or is it just a replacement for alternate neural network architecture. Without such a study, this paper should not be accepted, because it will add to more noise rather than advancing the field of robotic manipulation. 
Decision: Reject