Asymmetric self-play for automatic goal discovery in robotic manipulation
Authors: OpenAI OpenAI
Matthias Plappert
Raul Sampedro
Tao Xu
Ilge Akkaya
Vineet Kosaraju
Peter Welinder
Ruben D'Sa
Arthur Petron
Henrique Ponde de Oliveira Pinto
Alex Paino
Hyeonwoo Noh
Lilian Weng
Qiming Yuan
Casey Chu
Wojciech Zaremba
Venue:  (2021)
Keywords: self-play
asymmetric self-play
automatic curriculum
automatic goal generation
robotic learning
robotic manipulation
reinforcement learning

Abstract: We train a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with previously unseen goals and objects. To do so, we rely on asymmetric self-play for goal discovery, where two agents, Alice and Bob, play a game.  Alice is asked to propose challenging goals and Bob aims to solve them.  We show that this method is able to discover highly diverse and complex goals without any human priors.  We further show that Bob can be trained with only sparse rewards, because the interaction between Alice and Bob results in a natural curriculum and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned demonstration.  Finally, we show that our method scales, resulting in a single policy that can transfer to many unseen hold-out tasks such as setting a table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at https://robotics-self-play.github.io.

Under review as a conference paper at ICLR 2021
ASYMMETRIC SELF-PLAY FOR AUTOMATIC GOAL DIS- COVERY IN ROBOTIC MANIPULATION
Anonymous authors Paper under double-bling review
ABSTRACT
We train a single, goal-conditioned policy that can solve many robotic manipula- tion tasks, including tasks with previously unseen goals and objects. We rely on asymmetric self-play for goal discovery, where two agents, Alice and Bob, play a game. Alice is asked to propose challenging goals and Bob aims to solve them. We show that this method can discover highly diverse and complex goals without any human priors. Bob can be trained with only sparse rewards, because the interac- tion between Alice and Bob results in natural curriculum and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned demonstration. Finally, method scales, resulting in single policy that can generalize to many unseen tasks such as setting a table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at https://robotics self- playygithhbb.i
INTRODUCTION
1993) that can solve any robotic manipulation task that human may request given environment. In this work, we make progress towards this goal by solving a robotic manipulation problem in table-top setting where the robot's task to change the initial configuration of variable number objects on a table to match given goal configuration. This problem is simple in its formulation but likely to challenge wide variety cognitive abilities of robot as objects become diverse and goals become complex.
We are motivated to train single goal-conditioned policy (Kaelbling
Motivated by the recent success of deep reinforcement learning for robotics (Levine et al. 2016 Gu et al. 2017 Hwangbo et al. 2019: OpenAl et al. 2019a), we tackle this problem using deep reinforcement learning
very large training distribution. An open question in this approach how we can build a training distribution rich enough to achieve generalization to many unseen manipulation tasks. This involves defining both environment's initial state distribution and goal distribution. The initial state distribution determines how we sample a set of objects and their configuration the beginning of episode, and the goal distribution defines how sample target states given an initial state. In this work, we focus on scalable way to define rich goal distribution.
The research community has started explore automated ways of defining goal distributions. For example, previous works have explored learning generative model of goal distributions (Florensa al. 2018 Nair al. 2018b Racaniere al. 2020) and collecting teleoperated robot trajectories
Intial stat
goal state
Intill state
goal
(a) Table-top setting with robot amm (b) Example initial state for training (c) Example holdout tasks Figure 1: (a) We train a policy that controls a robot arm operating a table-top setting. (b) Ran- domly placed ShapeNet (Chang al. 2015) objects constitute initial state distribution for train- ing. (c) We use multiple manually designed holdout tasks to evaluate the learned policy.

---

Under review as a conference paper at ICLR 2021
last state initial state (X5)
initial slate distribution
set last state goal
run alice set goal
O
holdout task
Alice
goal conditioned reward
Training distribution
holdout task
samplo
holdout tack
initial
run bab to solve goal
state
Bob
holdoul losk
last state
inilial state (x5 lerminate on failure)
(a) Overall framework
(b) Training distribution based on asymmetric self-play
Figure 2: (a) We train a goal-conditioned policy on single training distribution and evaluate its performance many unseen holdout tasks. (b) To construct training distribution, sample an initial state from a predefined distribution, and run a goal setting policy (Alice) to generate a goal. In one episode, Alice is asked to generate goals and Bob solves them sequence until it fails.
identify goals (yncch al. 2020: Gupta et al. 2020 In this paper, we extend an alternative approach called asymmetric seli-play (Skhhbataar et al. 2018b a) for automated goal generation. Asymmetric self-play trains two RL agents named Alice and Bob. Alice learns to propose goals that Bob is likely to fail at, and Bob, a gonl-cond policy, learns to solve the proposed goals. Alice proposes goal by manipulating objects and Bob has to solve the goal starting from the same initial state Alice's. s. By embodying these two agents into the same robotic hardware, this setup ensures that all proposed goals are provided with at least one solution: Alice's 's trajectory.
There two main reasons why consider asymmetric self-play play to be promising goal generation and learning method. First, any proposed goal is achievable, meaning that there exists at least one solution trajectory that Bob can follow achieve the goal. Because of this property, Alice's 's trajectory to provide additional learning signal to Bob via behavioral cloning. This additional learning signal alleviates the overhead of heuristically designing curriculum reward shaping for learning. Second, this approach does not require labor intensive data collection.
exploit
In this paper, we show that asymmetric self-play complex object manipulation tasks, and the learned policy can zero-shot generalize to many man- ually designed holdout tasks, which consist of either previously unseen goals, previously unseen objects, or both. To the best of our knowledge, this is the first work that presents zero-shot general- ization to many previously unseen tasks by training purely with asymmetric self-play]
be used to train goal-conditioned policy for
2 PROBLEM FORMULATION
Our training environment for robotic manipulation consists of robot arm with gripper attached and a wide range of objects placed on a table surface (Figure Ia 1b). The goal-conditioned policy learns control the robot to rearrange randomly placed objects (the initial state) into a specified goal configuration FFigure lc). We aim to train a policy on a single training distribution and to evaluate its performance over a suite of holdout tasks which are independently designed and not explicitly present during training (Figure 2ap. In this work, we construct the training distribution via asymmetric self-play Figure 2h) to achieve generalization many unseen holdout tasks (Figure Ic).
Mathematical formulation Formally, we model the interaction between an environment and a conl-conditioned policy as goal- auggeneeeeteed Markov decision process M A, P, 9), where S is the state space, A is the action space, P S x A x S + IR denotes the transition probability, C S specifies the goal space and R S x G > R a goal-specific reward function. goal- augmented trajectory sequence is {(50. 9, ao, ro)..... (st, g. ae r.)). where the goal is provided to the policy as part of the observation at every step. We say a goal is achieved if s; is sufficiently close g (Appendix A.2 With a slightly overloaded notation, define the goal distribution ((g] as the probability of a goal state E 9 conditioned on an initial state Sp E S.
Asymmetric self-play is proposed in Sukhbaatar al. 2018 but supplement training while the majority of training is conducted target tasks. Zero-shot generalization unseen tasks was not evaluated.
2

---

Under review as a conference paper at ICLR 2021
Training goal distribution A naive design of the goal distribution ([g]s)) is to randomly place objects uniformly on the table, but is unlikely to generate interesting goals, such as an object picked up and held above the table surface by robot gripper. Another possible approach, collecting tasks and goals manually, is expensive and hard to scale. We instead sidestep these issues and automati- cally generate goals via training based on asymmetric self-play (Sukhbaatar et al. 2018b ap. Asym- metric self-play involves using policy named Alice A (a(ss) to set goals and a goal-conditioned policy Bob TB (a|s, 9) solve goals proposed by Alice, as illustrated in Figure 2b We run म to generate a trajectory = {(Su. du ro)..... (ST. Qr, Fr)) } and the last state is labelled as goal for Ty solve. The goal distribution G(sr g|so) is fully determined by я and we train Bob only on this goal distribution. We therefore say zerv-shol generalization when Bob generalizes to holdout task which is not explicitly encoded into the training distribution.
To assess zero-shot generalization of TB g) from our training setup, we hand-designed a suite of holdout tasks with goals that are never directly incorporated into the training distribution. Some holdout tasks also feature previously unseen objects. The holdout tasks are designed to either test whether specific skill has been learned, such as the ability to pick up objects (Figure 3), or represent semantically interesting task, such as setting a table (Figure lc). Appendix B.66 describes the list of holdout tasks that we use in our experiments. Note that none of the holdout tasks are used for training T (a)ss,)))
Evaluation on holdout tasks
ASYMMETRIC SELF-PLAY
To train Alice policy T (als) and Bob policy Tg(a)s,g), we run the following multi-goal game within one episode, illustrated in Figure 26
1. An initial state is sampled from an initial state distribution. Alice and Bob are instanti- ated into their own copies of the environment. Alice and Bob alternate turns as follows.
2. Alice's turn. Alice interacts with its environment for a fixed number of T steps and may rearrange the objects. The state at the end of Alice's turn 8T will be used as a goal y for Bob. If the proposed goal is invalid (e.g. if Alice has not moved any objects, or if an object has fallen off the table), the episode terminates.
3. Bob's turn. Bob receives reward if it successfully achieves the goal g in its environment. Bob's turn ends when it succeeds at achieving the goal or reaches timeout. If Bob's turn ends in a failure, its remaining turns are skipped and treated failures, while we let Alice to keep generating goals.
4. Alice receives reward if Bob fails to solve the goal that Alice proposed. Steps 2-3 are repeated until 5 goals are set by Alice or Alice proposes an invalid goal, and then the episode terminates.
The competition created by this game encourages Alice to propose goals that are increasingly chal- lenging to Bob, while Bob is forced to solve increasingly complex goals. The multi-goal setup was chosen allow Bob take advantage of environmental information discovered earlier in the episode to solve its remaining goals, which MaacchOpeAAleee et al. (2019a) found to be important for transfer to physical systems. Note however that in this work we focus on solving goals simulation only. To improve stability and avoid forgetting, we have Alice and Bob play against past versions of their respective opponent in 20% of games. More details about the game structure and pseudocode for training with asymmetric self-play are available in Appendix A
3.1 REWARD STRUCTURE
For Bob, we assign sparse goal-conditioned rewards. We measure the positional and rotational dis- tance between an object and its goal state as the Euclidean distance and the Euler angle rotational distance, respectively. Whenever both distance metrics are below small error (the success thresh- old), this object is deemed to be placed close enough to the goal state and Bob receives reward immediately. But if this object is moved away from the goal state that it has arrived at in past steps, Bob obtains- 1 reward such that the sum of per-object reward is at most I during given turn. When all of the objects are in their goal state, Bob receives 5 additional reward and its turn is over.
3

---

Under review as a conference paper at ICLR 2021
For Alice, assign a reward after Bob has attempted to solve the goal: 5 reward if Bob failed solving the goal, and if Bob succeeded. We shape Alice's s reward slightly by adding 1 reward it has set valid goal, defined to be when object has fallen off the table and any object has been moved more than the success threshold. An additional penalty of -3 reward introduced when Alice sets goal with objects outside of the placement area, defined to be fixed 3D volume within the view of the robot's camera. More details are discussed in Appendix A.2
3.2 ALICE BENAVIORAL CLONING (ABC)
One of the main benefits of using asymmetric self-play is that the generated goals come with at least
one solution achieve it: Alice's 's trajectory. Similarly to Sukhbaatar al. (2018)), we exploit this
property by training Bob with Behavioral Cloning (BC) from Alice's s trajectory. in addition to the
reinforcement learning (RL) objective. We call this learning mechanism Alice Behavioral Cloning (ABC). We propose several improvements over the original formulation in SukKhbaatar et al. (201 8a
Demonstration trajectory filtering Compared to BC from expert demonstrations, using Alice's trajectory needs extra care. Alice's trajectory is likely to be suboptimal for solving the goal, as Alice might arrive at the final state merely by accident. Therefore, we only consider trajectories with goals that Bob failed to solve demonstrations, to avoid distracting Bob with suboptimal examples. Whenever Bob fails, we relabel Alice's trajectory тA to be a goal-augmented version TBC {80: 8T, ; ro), (ST, ST; (r, (r)rr) demonstration for BC, where 8T is the goal.
PPO-style BC loss clipping The objective for training Bob is C CRL + BCabe, where CRL is an RL objective and Cabe the ABC loss. is a hyperparameter controlling the relative importance of the BC loss. We set 3 = 0.5 throughout the whole experiment. A naive BC loss is to minimize the negative log-likellng ligelekllooodd of demonstrated actions, (s D [log ( (st, Ir: 0)] where DBc a mini-batch of demonstration data and is parameterized by 0. We found that overly aggresvaggeessvve policy changes triggered by BC sometimes led to learning instabilities. To prevent the policy from changing too drastically, introduce PPO-style loss clipping (Schulman et al. 2017) the BC loss by setting the advantage Å ==1 in the clipped surrogate objective:
Tn St, Sc; St, 0) (a (st, g6: Bola)
Lahe Be (e)
clip
where (ae (8,, S65 0) is Bob's likelihood on demonstration based the parameters that we optimizing, and Ty (a:]st. 9t; fola) is the likelihood based on Bob's behavior policy (at the time of demonstration collection) evaluated on demonstration. This behavior policy is identical to the policy that we use collect RL trajectories. By setting À = 1, this objective optimizes the naive (=(0e((e.9::0)) is bigger than 1 + e, to prevent the policy from BC loss, but clips the loss whenever Tplaa (1.9:)a..9 changing too much. c is clipping threshold and we use c = 0.2 in all the experiments.
4 RELATED WORK
Training distribution for RL In the context of multi-task RL (Beattie al., 2016 Hausman al.
2018 Yu et al. 2020 multi-gont RL (Kaelbling 1993 Andrychowicz et al.) 2017). and meta RL (Wang et al. 2016 Duan et al. 2016). previous works manually designed distribution of tasks or goals to see better generalization of policy to a new task or goal. Domain randomiza- tion (Sadeghi & Levine 2017bg Tobin et al. 2017: OpenAl et al., 2020) manually defines a distri- bution of simulated environments, but in service of generalizing to the same task in the real world.
There are approaches to grow the training distribution automatically (Srivastava et al. 2013). Self-
play (Tesauro) 1995, Silver et al. 2016 2017: Bansal et al.) 2018 OpenAl et al. 2019b Vinyals
et al. 2019) constructs an ever wver-growing training distribution where multiple agents learn by compet- ing with each other, so that the resulting agent shows strong performance
single game. OpenAl ct al. (2019a) automatically grew a distribution of domain randomization parameters to accomplish better generalization in the task of solving Rubik's cube the physical robot. Waan et al. 20019
4

---

Under review as a conference paper at ICLR 2021
2020) studied automated way to keep discovering challenging 2D terrains and locomotion poli- cies that can solve them in a 2D bipedal walking environment.
We employ asymmetric self-play to construct training distribution for learning a goal-conditioned policy and to achieve generalization to unseen tasks. Florensa et al. (2018); Nair et al. (2018b); Racaniere al. (2020) had the same motivation ours, but trained generative model instead of a ggoal setting policy. Thus, the difficulties of training a generative model were inherited by these methods: difficulty of modeling high dimensional space and generation of unrealistic samples. Lynch et al. 20200; Gupta et al. (2200) used teleoperation to collect arbitrary robot trajectories, and defined goal distribution from the states in the collected trajectories. This approach likely requires a large number of robot trajectories for each environment configuration (e.g. various types of objects on table), and randomization of objects was not studied in this context.
Asymmetric self-play Asymmetric self-plat play was proposed by Sukhbaatar et al. (0188) as way to supplement RL training. Sukhbaatar al. (2018b) mixed asymmetric self training with standard RL training on the target task and measured the performance on the target task. /Sukhbaatar et al. (2018a) used asymmetric self-play to pre-train a hierarchical policy and evaluated the policy after fine-tuning it on a target task. Liu et al. (209)) adopted self-play to encourage efficient learning with sparse reward in the context of an exploration competition between pair of agents. As far as conditiod policy purely based asymmetric self-
know, no previous work has trained play and evaluated generalization to unseen holdout tasks.
Curriculum learning Many previous works showed the difficulty of RL and proposed an automated
curriculum (Andrychowicz et al. 2017; Florensa et al. 2017 Salıann & Chen 2018, Matüisen
al.
2019 Zhang al. (220)) auxiliary exploration objectives (Oudeyer et al.)
2007 Baranes &
Oudeyer 2013) Pathak et al. 2017 Burda et al. 2019 Ecoffet et al. 2019 2020) to learn predefined
tasks. When training goal-conditioned policies, relabeling or reversing trajectories (Andrychow-
icz et al., 2017: Florensa et al., 2017: Salimans & Chen 2018) or imitating successful demonstra-
tions (Oh et al. 2018 Ecoffet al. 2019120024 naturally reduces the task complexity. Our work shares a similarity in that asymmetric self-play alleviates the difficulty of learning goal-conditioned policy via an intrinsic curriculum and imitation from the goal setter's trajectory, but our work does not assume any predefined task or goal distribution.
Hierarchical reinforcement learning (HRL) Some HRL methods jointly trained a goal setting policy (high-level or manager policy) and goal solving policy (low-level or worker policy) (Vezh- nevets et al. 2017; Levy et al. 2019 Nachum et al. 2018). However, the motivation for learning a goal setting policy in HRL is not to challenge the goal solving policy, but to cooperate to tackle task that can be decomposed into a sequence of sub-goals. Hence, this goal setting policy is trained to optimize task reward for the target task, unlike asymmetric self-play where the goal setter is rewarded upon the other agent's 's failure.
Robot learning for object manipulation. It has been reported that training policy for multi- object manipulation is very challenging with sparse rewards (Riedmiller et al. 2018 Vecerik et al. One example is block stacking, which has been studied for long time in roboties as involves complex contact reasoning and long horizon motion planning (Deisenroth et al. 201T). Learning block stacking often requires a hand-designed curriculum (L et al. 2019,, meticulous reward shaping (Popov et al., 2017). fine-tuning (Rusu et al. (017). human demonstrations Nair et al. 2018a| Duan et al. 2017). In this work, we use block stacking as one of the holdout tasks to test zero-shot generalization, but without training on it.
2018
5 EXPERIMENTS
In this section, we first show that asymmetric self-play generates effective training curriculum that enables generalization to unseen hold-out tasks. Then, the experiment scaled up to train in an environment containing multiple random complex objects and evaluate it with a set of holdout tasks containing unseen objects and unseen goal configurations. Finally, we demonstrate how critical ABC is for Bob to make progress in set of ablation studies.
5

---

Under review as a conference paper at ICLR 2021
Figure Holdout tasks in the environment using blocks. The transparent blocks denote the desired goal state, while opaque blocks are the current state. (a) push: The blocks must be moved to their goal locations and orientations. There no differentiation between the six block faces. (b) flip: Each side of the block is labelled with unique letter. The blocks must be moved to make every face correctly positioned what the goal specifies. (c) pick and ppldee: One goal block is in the air. (d) stack: Two blocks must be stacked in the right order the right location.
push
flip
pick-and-place
stack
Push
Flip
Pick-and-place
Stack
Sucpess rate (8)
60
60
20
10
25
10
20 25
15
20
30
0
10
16
20
30
Training stops (x100)
No curricluum
Curriculum: distance
Curriculum: distribution
Curriculum: full
Self-play
mixture of all holdout tasks. The solid lines represent 2-blocks, while the dashed lines are for 1-block. The x-axis denotes the number of training steps via asymmetric self-play. The y-axis is the zero-shot generalization performance of Bob policy at corresponding training checkpoints. Note that success rate curves of completely failed baselines occluded by others.
Figure 4: Generalization unseen holdout tasks for blocks. Baselines trained
5.1 EXPERIMENTAL SETUP
We implement the training environment described Sec. with randomly placed ShapeNet ob- initial state distribution. In addition, environment using one or two blocks of fixed size, used for small-scale comparisons and ablation studies. Figure 3 visualizes four holdout tasks for this environment. Each task is designed to eval- uate whether the robot has acquired certain manipulation skills: pushing, flipping, picking up and stacking blocks. Experiments in Sec. 5.2 and 5.33 on blocks and experimental results based on ShapeNet objects are present on Sec. 5.4 More details on our training setups are in Appendix B
jects (Chang al. 2013)
up another simpler
We implement Alice and Bob as two independent policies of the same network architecture with memory (Appendix B.4), except that Alice has no observation on goal state. The policies take state observations ("state policy")) for experiments with blocks (Sec. 5.2 5.3 and 5.3), and take both vision and state observations ('hybrid policy") for experiments with ShapeNet objects (Sec. 5.4)
Both policies are trained with Proximal Policy Optimization (PPO) (Schulman et al. 2017
5.2 GENERALIZATION TO UNSEEN GOALS WITHOUT MANUAL CURRICULA
One way train single policy to acquire all the skills in Figure 3 is to train goal-conditioned policy directly over mixture of these tasks. However, training directly over these tasks without curriculum turns out be very challenging, contrast, Bob is able to solve all these holdout tasks quickly when learning via asymmetric self-play, without explicitly encoding any prior knowledge of the holdout tasks into the training distribution.
the policy completely fails to make any progress In
R
To gauge the effect of an intrinsic curriculum introduced by self-play, we carefully designed a set of non-self-pla play baselines using explicit curricula controlled by Automatic Domain Randomiza- tion (OpeAAl et al. 2019a). All baselines are trained over a mixture of block holdout tasks as the
*Our training and evaluation environments are publicly available at hide- for anonymous purpose 'The tasks casier when ignored object rotation part of the goal, and used a smaller table.

---

Under review as a conference paper at ICLR 2021
Lift blocks
and blocks balance Alice discovers many goals that are not covered
Figure 5: Goals discovered by asymmetric self-play, by our manually designed holdout tasks on blocks.
1 block
2 blocks
Figure 6: The empirical payoff matrix be- tween Alice and Bob. Average success rate multiple self-play episode is visualized. Alice with more training steps generates more challenging goals that Bob cannot solve yet. Bob with more training steps can achieve more goals against the same Alice.
Alice training steps
jeiminn
8ob training steps
Bob training
goal distribution. We measure the effectiveness of a training setup by tracking the success rate for each holdout task, as shown in Figure 4 The curriculum baseline fails drastically. The curricul lum distance baseline expands the distance between the initial and goal states gradually as training progresses, but only learns push and flip single block. The surriculum www.llm:disttt...o baseline, which slowly increases the proportion of pick-and-place and stacking goals the training distribution, fails to acquire any skill. The curriculum. full bascline incorporates all desdndd curricula yet still cannot learn how to pick up or stack blocks, We have spent decent amount of time iterating and improving these baselines but found it especially difficult develop scheme good enough to compete with asymmetric self-play See Appendix C.I for more details of our baselines.
5.3 DISCOVERY OF NOVEL GOALS AND SOLUTIONS
Asymmetric self- self-play discovers novel goals and solutions that are not covered by our holdout tasks. As illustrated in Figure 5) Alice can lift multiple blocks at the same time, build a tower and then keep it balanced using an arm joint. Although it is a tricky strategy for Bob to learn on its own, with ABC, Bob eventually acquires the skills for solving such complex tasks proposed by Alice. Videos are available at hhttps::/robbtttt
self play. wwwp..iitaa...... github.io
summarizes Alice and Bob's learning progress against each other. For every pair of Alice and Bob, we ran multiple self-play episodes and measured the success rate. We observe an inter- esting trend with 2 blocks. As training proceeds, Alice tends to generate more challenging goals, where Bob shows lower success rate. With past sampling, Bob continues to make progress against versions of Alices from earlier optimization steps. This visualization suggests desired dynamic of asymmetric self-play that could potentially lead to unbounded complexity: Alice continuously generates goals challenge Bob, and Bob keeps making progress on learning to solve new goals.
Figure 6
5.4 GENERALIZATION TO UNSEEN OBJECTS AND GOALS
The experiments above show strong evidence that efficient curricula and novel goals can au- tonomously emerge in asymmetric self-play. To further challenge our approach, we scale it up
to work with many more complex objects using more computational resources for training. We train a hybrid policy in an environment containing up to 10 random ShapeNet (Chang et al. 2015) objects. During training, we randomize the number of objects and the object sizes via Automatic Domain Randomization (OpenAl et al., 2019a). The hybrid policy uses vision observations to ex- tract information about object geometry and size. We evaluate the Bob policy more diverse set of manipulation tasks, including semantically interesting ones. Many tasks contain unseen objects and complex goals, illustrated in Figure 7
The learned Bob policy achieves decent zero-shot generalization performance for many tasks. Suc- cess rates reported in Figure 8 Several tasks are still challenging. For example, ball-capture
7

---

Under review as a conference paper at ICLR 2021
l HOl
(a) Table setting (b) Mini chess
(c)
Domino
(d) Rainbow
(c) captuce
(f) Tangram
Figure 7: Example holdout tasks involving unseen objects and complex goal states. The goal states are illustrated here, and the initial states have randomly placed objects.
Success (%)
100
100
99
97
96 100 99
100
98
93
84
96 100
99
76
80
64
52
60
47
40
23
20
0
number objects
Blocks
YCB objects
Customized objects
Push
Pick&plane
Stacking
Push
Pidrsplaca-
Ball capture
Mini aheds Dominds
Tallle setting
Tangrain
Rainbbow
Figure 8: Success rates of single goal-conditioned policy solving variety of holdout tasks, aver- aged over 100 trials. The error bars indicate the 99% confidence intervals. Yellow, orange and blue bars correspond to success rates of manipulation tasks with blocks, yCeect and other uniquely built objects, respectively. Videos are available at htttt::/robbticcsss self-pllffpppeeee play. github.in io
requires delicate handling of rolling objects and lifting skills. The rainbow tasks call for under- standing of concave shapes. Understanding the ordering of placement actions is crucial for stacking more than 3 blocks in the desired order. The Bob policy learns such ordering some degree, but fails to fully generalize to an arbitrary number of stacked blocks.
5.5 ABLATION STUDIES
We present series of ablation studies designed for measuring the importance of each component in our asymmetric self-play framework, including Alice behavioral cloning (ABC), BC loss clip- ping, demonstration filtering, and the multi-goal game setup. We disable single ingredient in each ablation run and compare with the complete self-play baseline Figure
Push
Pick-and-place
Stack
100
Success rate (%)
80
rate
80
40
20
0
10
15
20
25
30
5
10
15
20
25
30
5
10
15
20
25
30
0
10
15
20
25
30
Training stops (x100)
No ABC
No BC loss clipping
No demonstration filter
Single-goal
Self-play (full setup)
Figure 9: The ablation studies compare four ablation runs each with one component disabled with the full baseline. Solid lines are for 2-blocks, dashed lines for 1-block. The x-axis denotes the number of training steps via asymmetric self-play. The y-axis is the zero-shot generalization performance of Bob policy at corresponding training steps.
The no ABC baseline shows that Bob completely fails to solve any holdout task without ABC, indicat- ing that ABC is critical mechanism in asymmetric self-play. The no BC loss clipping baseline shows slightly slower learning on pick-and -llapllacee and stack, as well as some instabilities in the middle of training. Clipping in the BC loss is expected to help alleviate this instability by
8

---

Under review as a conference paper at ICLR 2021
trolling the rate of policy change per optimizer iteration. The demonstration filter baseline shows noticeable instability on flip, suggesting the importance of excluding suboptimal demon- strations from behavioral cloning. Finally, the single-gon -gaal baseline uses single goal instead of goals per episode during training. The evaluation tasks are also updated to require single success per episode. Generalization of this baseline to holdout tasks turns out to be much slower and less stable. It signifies some advantages of using multiple goals per episode, perhaps due to the policy memory internalizing environmental information during multiple trials of goal solving.
The results of the ablation studies suggest that ABC with proper configuration and multi-goal game- play are critical components of asymmetric self-play, alleviating the importance of manual curricula and facilitating efficient learning.
6 CONCLUSION
One limitation of our asymmetric self-play approach is that it depends on resettable simulation environment as Bob needs to start from the same initial state as Alice's. Therefore asymmetric self-play training has to happen in simulator which can be easily updated to a desired state. In order run the goal-solving policy on physical robots, we plan to adopt sim-to-real techniques in future work. Sim-to-real has been shown to achieve great performance real world (Sadeghi & Levine 201 7a Tobin et al. 2017: James et al. 2019, OpenAl et al., 2020). pre-train
many robotic tasks in the
One potential approach is to pre- train two agents asymmetric self play simulation, and then fine-tune the Bob policy with domain randomization or data collected on physical robots.
In conclusion, we studied asymmetric self-play framework for defining single training distribu- tion to learn many arbitrary object manipulation tasks. Even without any prior knowledge about the target tasks, asymmetric self-play is able to train strong goal-conditioned policy that can generalize to many unseen holdout tasks. We found that asymmetric self-play not only generates a wide range of interesting goals but also alleviates the necessity of designing manual curricula for learning such goals. We provided evidence that using the goal setting trajectory as a demonstration for training a goal solving policy essential enable efficient learning. We further scaled up our approach to work with various complex objects using more computation, and achieved zero-shot generalization collection of challenging manipulation tasks involving unseen objects and unseen goals.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, 2017.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, llya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In International Conference on Learning Representations, 2018.
Adrien Baranes and Pierre-Yver Yves Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73. 2013.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancg. Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab, arXiv preprint arXiv: 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint /5/2.030 2015.
Mare Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning control low-cost manipulator using data-efficient reinforcement learning. Robotics: Science and Systems VII, pp- 57-64, 2011.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbeel. RL3: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, llya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for eard-exploration problems. arXiv preprint arXiv:1901 10995, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return then explore. arXiv preprint arXiv:2004. 12919, 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference on Machine Learning, 2018.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Picter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference Robot Learning, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference machine learning, 2018.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE international conference on robotics and automation (ICRA). 2017.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning. 2020.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang. Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Leam- ing Representations, 2018
10

---

Under review as a conference paper at ICLR 2021
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via calloomized-t-ccnonnii adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Leslie Pack Kaelbling. Learning to achieve goals. In Proceedings of the 13th International Joint Conference on Artificial Intelligence, 1993.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.660 2014.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1333, 2016.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference Learning Representutions, 2019.
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manip-
ulation using relational reinforcement learning. arXiv preprint arXiv: 2019.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. In International Conference Learning Representations, 2019.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132, 2020.
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE transactions on neural networks and learning systems, 2019.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems, 2018
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In IEEE International Con- ference on Robotics and Automation (ICRA), 2018a.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, 2018b.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint 1eXiv:1806.055 05635, 2018
OpenAI, 1lge Akkaya, Marcin Andrychowiez, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng. Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's s cube with robot hand. arXiv preprint 1911::1110 2019a.
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1rXiv:1912.06660 2019b.
OpenAL, Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20 2020.
11

---

Under review as a conference paper at ICLR 2021
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-266, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In Proceedings of the 34th International Conference Machine Learning-Volle Volume 70, 2017.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073, 2017.
Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated curriculum generation through setter-solver interactions. In International
Conference on Learning Representations, 2020.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch, 2018.
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. In Conference on Robot Learning. 2017.
Fereshteh Sadeghi and Sergey Levine. CAD2RL: real single-image flight without single real image. Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16, 2017, 2017a. URL http://www.rooottet httt:://www..bbot.ccporceeeeeeeo
rss13/p34.htmm
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. In Proceedings of Robotics: Science and Systems, 2017b.
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from a single demonstration. arXiv preprint arXiv: 1812.0338 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- with a general reinforcement learning algorithm. arXiv preprint 1IXiv:1712.01800 2017.
Rupesh Kumar Srivastava, Bas Steunebrink, and Jürgen Schmidhuber. First experiments with powerplay. Neural Networks, 41:130-136, 2013.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Leamiing goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint 1rXi::1811.0908. 201 2018a.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018b.
Gerald Tesauro. Temporal difference learning and td dd-gammon. Communications of the ACM. 38(3): 58-68, 1995.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IEFF/RSJ International Conference Intelligent Robots and Systems (IROS), 2017.
12

---

Under review as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model- control. In Intelligent Robots and Systems (IROS), 2012 JEEE/RSJ International Conference on, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference Machine Learning, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-3555 2019.
Jane Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint /61 1.05763, 2016.
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint vrXiv:190.007333 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth O Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. arXiv preprint Sttiv:2003.085366 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2020.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. arXiv preprint arXiv:2006.096: 2020.
13


Reviews:
[{"review_id": "h9aNAVvz1_v", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a self-play approach to learn a goal-conditioned policy for robot manipulation. The Alice and Bob approach is able to generalise to unseen objects and results in a natural curriculum.\nThe paper is well written and well structured, the proposed approach is framed within the relevant literature and claims are supported by experimental results.\nExperimental evaluation shows that the proposed approach outperforms other methods on several manipulation task. Ablation studies complete the analysis of the proposed approach and help understanding the contribution of each part of the approach.\n\nComments:\n- How does the proposed method compare with other goal-based approaches (e.g. multi-task approaches or goal-based intrinsic motivation exploration)?\n- What is the main cause of the low success rate with increasing number of objects? What is the main limitation in the generalisation capability of the approach (Fig. 8)?\n- How much does memory (used in policy networks) impact the learned behaviours (especially with several objects to manipulate)?\n- Can you discuss the main limitations of your approach with respect to other methods?\n- What is the expected transferability of the proposed approach to a real world scenario (e.g. with a real manipulator and real objects)?\n- With a self-play approach the exploration is \"limited\" to what the agent can generate during self-play. How does this affect generalisation in your specific case study?\n- Videos are interesting and helpful\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h0bScCildPS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a variation of asymmetric self play (Sukhbaatar et al. 2018b) where:\n- Bob gets to use Alice's trajectories for Behaviour Cloning (ABC) training (with some logic around when best to employ this).\n- There may be several goals (reward-related) per episode and extra care is taken to avoid training on bad episodes (e.g. items didn't move or items fell off table)\n- Tasks make use of a simulated robotic arm environment (however the observations include ground truth data that make the sim far from anything real)\n- Some study-specific details of training methods (PPO-like) and agent network architectures (in suppl) which may be of interest.\n\nAt first glance, the movies that accompany this paper show a surprisingly high level of performance for the given task complexity.  This surprise was reduces when by the supplementary material where it is clearly explained that the observations include the full ground-truth information on the robot state and the manipulated objects: \"object state observation contains each object’s position, rotation, velocity, rotational velocity, the distance between the object and the gripper, as well as whether this object has contacts with the gripper.\" In my view, this renders the experiments deep within the realm of toy simulations. I also feel that showing movies of vision inputs is a bit misleading because, in my view, it coveys a message that this work is on vision-based manipulation learning while in reality there is nearly full knowledge of all the items' true states.\n\nThere are multiple figures showing the merit of the approach within the designed task framework in the form of comparison with baseline methods that do not use the asymmetric-self play framework and an ablation study. ~~The one comparison with (Sukhbaatar et al. 2018b), which this study is a variation of, appears at the very end of the supplementary material and seems to show that the two methods do equally well on tasks presented in this study. This last point leads me to conclude that the variations from (Sukhbaatar et al. 2018b) in this study do not harm but also do not improve on the original asymmetric-self-play model.~~\n\n**EDIT: per the explanation in the author feedback (not a direct comparison with ukhbaatar et al. 2018b), that conclusion is unwarranted.**\n\nI found the paper lacking in clarity regarding both method and evaluation:\nThe method: \nSection 3 describes the method. It reads more like a summary pointing out the important bits than an ordered description that would allow someone to replicate the framework and model updates (which seem to all be tightly coupled in that the running of Alice and Bob, the generation of rewards and the rules around when to do what type of model update step all contribute to the framework whose product is two agents and the tasks that they set up for each-other). \n~~If the process could be described as a set of agents and interfaces and pseudocode, perhaps the reader would get a clearer understanding of how they might replicate the experiments. Note also that code is not provided.~~\n\n**EDIT: pseudocode (and potential future link to code) added**\n\n\nEvaluation:\nIt is clear that evaluation was performed on a large number of tasks and task variations and there is information regarding the tasks and models and parameters in the supplementary material. What I found to be unclear is - how were the experiments on hold-out tasks different from the other tasks. More specifically, in what ways is the framework used differently during hold out tasks. Is asymmetric self play used? Is Alice Behavioural Cloning applied?  If so, then in what sense are these tasks held out? If not, then how is the Bob agent trained for these tasks? The unclarity in the description of the method may have contributed to the unclarity of the evaluation section (in that it is not clear to me how the framework is modified between these task groups).\n\n**EDIT: Thanks to clarifications in author feedback and some modifications to the manuscript I find the paper clearer w.r.t. evaluation (though still feel that reading the paper only may leave the reader confused).**\n\nThe reliance on behavioural cloning (shown to be crucial to achieving non-zero success in this study's modelling) limits the proposed method to cases where the environment can be reset which would make it much harder to implement outside of a simulation environment.\n\nOverall, due to the issues mentioned above, I find this study to add little to the methodology and understanding of asymmetric self play as an RL method for robotic manipulation.\n\n**EDIT: with author feedback and changes to manuscript (and supplementary) I think that the study is more interesting than expressed in my original review, however the sim-real gap with respect to applicability to robotics is still a major concern IMO. I have updated my score accordingly.**\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YLZZiEG2WNl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to learn goal conditioned policies by relying on self-play which sets the goals and discovers a curriculum of tasks for learning. Alice and Bob are the agents. Alice's task is to set a goal by following a number of steps in the environment and she is rewarded when the goal is too challenging for Bob to solve. Bob's task is to solve the task by trying to reproduce the end state of Alice's demonstration. As a result, the learned policy performs various tasks and can work in zero-shot settings.\n\nStrong points: \n- This paper proposes a novel way to generate the goal distribution for training on multiple goals that is rich and scalable. The authors describe the properties of the goals as being achievable and not labor intensive.\n- The experimental results are very encouraging.\n- The analysis of the method helps to understand which components are important.\n- The evaluation on the hold-out tasks is very impressive and pushes the state of the art.\n- The paper is well written and very easy to follow, the illustrations are informative and appealing. \n- Although this approach is based on previous work on asymmetric self-play, the authors clearly describe the contributions of this work (training clearly from self play, zero-shot generalisation).\n\nWeak points:\n- Alice and Bob agents need to perform the task from the same initial state. This might pose some problems when training with a real (not simulated) robot environment.\n- The way how goal-conditioning reward is assigned assumes that what matters is the absolute positions of all the objects in the environment. This means that such relative concepts like, for example, \"objects being close to each other\", cannot be handled in the current framework.\n- Alice Behavioural Cloning is currently incorporated into learning by combining the RL and BC losses. Another common way to incorporate demonstrations is to include them in the replay buffer for training a policy. Would it be possible in this method? As this component is crucial for the success of the method, it might be worth investigating the alternatives.\n- The baselines of using generative modelling for goal generation are mentioned, but there is no comparison with them. Would it be possible to apply those methods to the studied environments and settings? \n- It seems that every policy includes the state as part of its observation. What does this state include? Why can't the policy be trained directly from vision? Does including the state mean that the learning procedure is not applicable to real environments?\n \nI am leaning toward the acceptance of this paper because I find 1) the method interesting and well motivated, 2) experimental results very encouraging and zero-shot generalisation quite impressive. While I still have some concerns (learning from states, starting from the same initial condition), I believe that this paper advances the knowledge and would be beneficial for the research community.\n\nQuestions:\n- \"Multi-goal\" game: the experiments show that it improves the results, but I still cannot understand why it is crucial. Does it help because of several goals or because of longer episodes? Currently, it is repeated 5 times, what happens if it is less/more and each game is shorter/longer?\n- What happens with multi-goal during testing? Are the tasks split into stages, or the agent solves only for one goal?\n- I am a bit confused by step 3 where Bob's reward is computed. It sounds to me now that if Bob fails in the first game, it fails in the whole episode, is it so?\n- The policies are not trained directly on the hold-out tasks, but I am wondering, how many such goals (for example, for stacking tasks) appear by accident when Alice sets the goal.\n- What about the data complexity of the proposed approach? How many goals did Alice generate, how many steps in each?\n- In Fig. 6, why are the results different with a single block and two blocks?\n\n\nAdditional comments:\n- I am a bit confused about the content of footnote 1.\n- I couldn't see some of the strategies in Fig.4, is there a way to make them more visible?\n\n===Post rebuttal===\n\nI would like to thank the authors for the detailed response and clarification of my questions. I believe that this paper will be valuable for the ML community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9MMJmoHm8KI", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper uses asymmetric self-play to train a goal-conditioned policy for robot manipulation tasks, which can also generate curriculum automatically and generalize to unseen goals and objects. The experiments contain various challenging manipulation tasks, where the proposed method outperforms all the manually designed curriculum baselines.\n\nPros:\n+ The paper is well-written and easy to follow. \n+ The goals and skills discovered by Alice interesting. The proposed method can generate some meaningful goals which work better than explicit curricula.\n+ The novel shape/geometry manipulation results are promising.\n\nQuestions & Concerns:\n- Lack of important baselines. The author uses PPO, but this framework will still support off-policy learning. This adversarial training idea is very similar to CER which is not cited. \n  1. (HER) Hindsight Experience Replay\n  2. (CER) Competitive Experience Replay (not cited): [1]\n- This work is also related to HRL methods, so it needs to include more HRL works in section 4, e.g. HAC[2], HIRO[3]\n- (minor) In the \"Flip\" environment, success is also related to the orientation of the box. What's your rotation representation and the corresponding distance to measure \"success\"?\n- (minor) Are there any exploration strategies in Alice?\n\nIn general, this paper proposes to use asymmetric self-play to generate a curriculum for the agent, which is better than the explicit one. However, since there is no comparison with HER-based methods (or HRL), it's hard to justify whether asymmetric self-play is necessary for these kinds of problems.\n\n[1] Liu, Hao, et al. \"Competitive experience replay.\" ICLR 2019\n\n[2] Levy, Andrew, et al. \"Learning multi-level hierarchies with hindsight.\" ICLR 2019\n\n[3] Nachum, Ofir, et al. \"Data-efficient hierarchical reinforcement learning.\" Advances in Neural Information Processing Systems. 2018.\n\n======== POST REBUTTAL RESPONSE========\n\nAfter reading the feedback and revision of the paper, most of my above concerns are addressed. I agree that the comparison with HER/CER is not fair. I also notice that the author added more references and details based on all 4 reviews.  Thus, I decided to improve my score on this paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends previous work on asymmetric self-play by introducing a novel behavior cloning loss (referred to as ABC). The zero-shot results are impressive and demonstrate that the proposed curriculum learning approach pushes the state-of-the-art. The reviewers acknowledge these contributions. The pros of the paper are well summarized by R2, 

- The experimental results are very encouraging.
- The analysis of the method helps to understand which components are important.
- The evaluation on the hold-out tasks is very impressive and pushes the state of the art.
- The paper is well written and very easy to follow, the illustrations are informative and appealing.
- Although this approach is based on previous work on asymmetric self-play, the authors clearly describe the contributions of this work (training clearly from self-play, zero-shot generalization).

R1, R2, R5 recommend accepting the paper with scores of 7, 7, 6. R1 expressed that he is not confident about the paper. R4 recommends acceptance with a score of 6. However, R4 also expresses the concern for real-world applicability, "the sim-real gap with respect to applicability to robotics is still a major concern IMO. I have updated my score accordingly." The sim-to-real gap is a concern due to knowledge of perfect state information and the assumption of resets. 

Based on confident reviews of R2, R4, and R5, and the impressive zero-shot results, ordinarily, I would recommend the paper to be accepted. However, unfortunately, both the authors and reviewers missed a comparison to prior work, which I detail below. While the current paper makes a good case for zero-shot generalization, it does not compare to previous approaches that also exhibits zero-shot generalization. For instance, Li et al. ICRA 2020 (https://arxiv.org/abs/1912.11032) show that using a simple curriculum that depends on the number of manipulated objects + Graph Neural Networks can generalize very well to unseen tasks. E.g., the results reported in the paper demonstrate that a policy trained on 2/3 blocks generalizes and can stack many more blocks. Their policy learns to stack 6-7 blocks, whereas the paper under review can only stack up to 3 blocks (Figure 8). 

The authors mention in Section 5.2 of their paper, "The curriculum:distribution baseline, which slowly increases the proportion of pick-and-place and stacking goals in the training distribution, fails to acquire any skill. The curriculum:full baseline incorporates all hand-designed curricula yet still cannot learn how to pick up or stack blocks. We have spent a decent amount of time iterating and improving these baselines but found it especially difficult to develop a scheme good enough to compete with asymmetric self-play." 
This is at odds with results in Li et al. 

This reason for dissonance is that good generalization can be achieved by improving two separate components -- the neural network architecture or the learning curriculum. This paper shows good generalization with weak neural net architectures + a good curriculum learning method. It is unclear to me how critical the self-play method would be with a stronger architecture such as a graph network which is arguable more appropriate for the set of tasks presented in the paper. I would like to see if the curriculum is necessary (i.e., complements a stronger architecture) or is it just a replacement for alternate neural network architecture. Without such a study, this paper should not be accepted, because it will add to more noise rather than advancing the field of robotic manipulation. 
Decision: Reject