Asymmetric self-play for automatic goal discovery in robotic manipulation

1 INTRODUCTION . We are motivated to train a single goal-conditioned policy ( Kaelbling , 1993 ) that can solve any robotic manipulation task that a human may request in a given environment . In this work , we make progress towards this goal by solving a robotic manipulation problem in a table-top setting where the robot ’ s task is to change the initial configuration of a variable number of objects on a table to match a given goal configuration . This problem is simple in its formulation but likely to challenge a wide variety of cognitive abilities of a robot as objects become diverse and goals become complex . Motivated by the recent success of deep reinforcement learning for robotics ( Levine et al. , 2016 ; Gu et al. , 2017 ; Hwangbo et al. , 2019 ; OpenAI et al. , 2019a ) , we tackle this problem using deep reinforcement learning on a very large training distribution . An open question in this approach is how we can build a training distribution rich enough to achieve generalization to many unseen manipulation tasks . This involves defining both an environment ’ s initial state distribution and a goal distribution . The initial state distribution determines how we sample a set of objects and their configuration at the beginning of an episode , and the goal distribution defines how we sample target states given an initial state . In this work , we focus on a scalable way to define a rich goal distribution . The research community has started to explore automated ways of defining goal distributions . For example , previous works have explored learning a generative model of goal distributions ( Florensa et al. , 2018 ; Nair et al. , 2018b ; Racaniere et al. , 2020 ) and collecting teleoperated robot trajectories to identify goals ( Lynch et al. , 2020 ; Gupta et al. , 2020 ) . In this paper , we extend an alternative approach called asymmetric self-play ( Sukhbaatar et al. , 2018b ; a ) for automated goal generation . Asymmetric self-play trains two RL agents named Alice and Bob . Alice learns to propose goals that Bob is likely to fail at , and Bob , a goal-conditioned policy , learns to solve the proposed goals . Alice proposes a goal by manipulating objects and Bob has to solve the goal starting from the same initial state as Alice ’ s . By embodying these two agents into the same robotic hardware , this setup ensures that all proposed goals are provided with at least one solution : Alice ’ s trajectory . There are two main reasons why we consider asymmetric self-play to be a promising goal generation and learning method . First , any proposed goal is achievable , meaning that there exists at least one solution trajectory that Bob can follow to achieve the goal . Because of this property , we can exploit Alice ’ s trajectory to provide additional learning signal to Bob via behavioral cloning . This additional learning signal alleviates the overhead of heuristically designing a curriculum or reward shaping for learning . Second , this approach does not require labor intensive data collection . In this paper , we show that asymmetric self-play can be used to train a goal-conditioned policy for complex object manipulation tasks , and the learned policy can zero-shot generalize to many manually designed holdout tasks , which consist of either previously unseen goals , previously unseen objects , or both . To the best of our knowledge , this is the first work that presents zero-shot generalization to many previously unseen tasks by training purely with asymmetric self-play.1 2 PROBLEM FORMULATION . Our training environment for robotic manipulation consists of a robot arm with a gripper attached and a wide range of objects placed on a table surface ( Figure 1a,1b ) . The goal-conditioned policy learns to control the robot to rearrange randomly placed objects ( the initial state ) into a specified goal configuration ( Figure 1c ) . We aim to train a policy on a single training distribution and to evaluate its performance over a suite of holdout tasks which are independently designed and not explicitly present during training ( Figure 2a ) . In this work , we construct the training distribution via asymmetric self-play ( Figure 2b ) to achieve generalization to many unseen holdout tasks ( Figure 1c ) . Mathematical formulation Formally , we model the interaction between an environment and a goal-conditioned policy as a goal-augmented Markov decision process M = hS , A , P , R , Gi , where S is the state space , A is the action space , P : S ⇥ A ⇥ S 7 ! R denotes the transition probability , G ✓ S specifies the goal space and R : S ⇥ G 7 ! R is a goal-specific reward function . A goalaugmented trajectory sequence is { ( s0 , g , a0 , r0 ) , . . . , ( st , g , at , rt ) } , where the goal is provided to the policy as part of the observation at every step . We say a goal is achieved if st is sufficiently close to g ( Appendix A.2 ) . With a slightly overloaded notation , we define the goal distribution G ( g|s0 ) as the probability of a goal state g 2 G conditioned on an initial state s0 2 S . 1Asymmetric self-play is proposed in Sukhbaatar et al . ( 2018b ; a ) , but to supplement training while the majority of training is conducted on target tasks . Zero-shot generalization to unseen tasks was not evaluated . Training goal distribution A naive design of the goal distribution G ( g|s0 ) is to randomly place objects uniformly on the table , but it is unlikely to generate interesting goals , such as an object picked up and held above the table surface by a robot gripper . Another possible approach , collecting tasks and goals manually , is expensive and hard to scale . We instead sidestep these issues and automatically generate goals via training based on asymmetric self-play ( Sukhbaatar et al. , 2018b ; a ) . Asymmetric self-play involves using a policy named Alice ⇡A ( a|s ) to set goals and a goal-conditioned policy Bob ⇡B ( a|s , g ) to solve goals proposed by Alice , as illustrated in Figure 2b . We run ⇡A to generate a trajectory ⌧A = { ( s0 , a0 , r0 ) , . . . , ( sT , aT , rT ) } and the last state is labelled as a goal g for ⇡B to solve . The goal distribution G ( sT = g|s0 ) is fully determined by ⇡A and we train Bob only on this goal distribution . We therefore say zero-shot generalization when Bob generalizes to a holdout task which is not explicitly encoded into the training distribution . Evaluation on holdout tasks To assess zero-shot generalization of ⇡B ( a|s , g ) from our training setup , we hand-designed a suite of holdout tasks with goals that are never directly incorporated into the training distribution . Some holdout tasks also feature previously unseen objects . The holdout tasks are designed to either test whether a specific skill has been learned , such as the ability to pick up objects ( Figure 3 ) , or represent a semantically interesting task , such as setting a table ( Figure 1c ) . Appendix B.6 describes the list of holdout tasks that we use in our experiments . Note that none of the holdout tasks are used for training ⇡B ( a|s , g ) . 3 ASYMMETRIC SELF-PLAY . To train Alice policy ⇡A ( a|s ) and Bob policy ⇡B ( a|s , g ) , we run the following multi-goal game within one episode , as illustrated in Figure 2b : 1 . An initial state s0 is sampled from an initial state distribution . Alice and Bob are instantiated into their own copies of the environment . Alice and Bob alternate turns as follows . 2 . Alice ’ s turn . Alice interacts with its environment for a fixed number of T steps and may rearrange the objects . The state at the end of Alice ’ s turn sT will be used as a goal g for Bob . If the proposed goal is invalid ( e.g . if Alice has not moved any objects , or if an object has fallen off the table ) , the episode terminates . 3 . Bob ’ s turn . Bob receives reward if it successfully achieves the goal g in its environment . Bob ’ s turn ends when it succeeds at achieving the goal or reaches a timeout . If Bob ’ s turn ends in a failure , its remaining turns are skipped and treated as failures , while we let Alice to keep generating goals . 4 . Alice receives reward if Bob fails to solve the goal that Alice proposed . Steps 2–3 are repeated until 5 goals are set by Alice or Alice proposes an invalid goal , and then the episode terminates . The competition created by this game encourages Alice to propose goals that are increasingly challenging to Bob , while Bob is forced to solve increasingly complex goals . The multi-goal setup was chosen to allow Bob to take advantage of environmental information discovered earlier in the episode to solve its remaining goals , which OpenAI et al . ( 2019a ) found to be important for transfer to physical systems . Note however that in this work we focus on solving goals in simulation only . To improve stability and avoid forgetting , we have Alice and Bob play against past versions of their respective opponent in 20 % of games . More details about the game structure and pseudocode for training with asymmetric self-play are available in Appendix A . 3.1 REWARD STRUCTURE . For Bob , we assign sparse goal-conditioned rewards . We measure the positional and rotational distance between an object and its goal state as the Euclidean distance and the Euler angle rotational distance , respectively . Whenever both distance metrics are below a small error ( the success threshold ) , this object is deemed to be placed close enough to the goal state and Bob receives 1 reward immediately . But if this object is moved away from the goal state that it has arrived at in past steps , Bob obtains -1 reward such that the sum of per-object reward is at most 1 during a given turn . When all of the objects are in their goal state , Bob receives 5 additional reward and its turn is over . For Alice , we assign a reward after Bob has attempted to solve the goal : 5 reward if Bob failed at solving the goal , and 0 if Bob succeeded . We shape Alice ’ s reward slightly by adding 1 reward if it has set a valid goal , defined to be when no object has fallen off the table and any object has been moved more than the success threshold . An additional penalty of 3 reward is introduced when Alice sets a goal with objects outside of the placement area , defined to be a fixed 3D volume within the view of the robot ’ s camera . More details are discussed in Appendix A.2 . 3.2 ALICE BEHAVIORAL CLONING ( ABC ) . One of the main benefits of using asymmetric self-play is that the generated goals come with at least one solution to achieve it : Alice ’ s trajectory . Similarly to Sukhbaatar et al . ( 2018a ) , we exploit this property by training Bob with Behavioral Cloning ( BC ) from Alice ’ s trajectory , in addition to the reinforcement learning ( RL ) objective . We call this learning mechanism Alice Behavioral Cloning ( ABC ) . We propose several improvements over the original formulation in Sukhbaatar et al . ( 2018a ) . Demonstration trajectory filtering Compared to BC from expert demonstrations , using Alice ’ s trajectory needs extra care . Alice ’ s trajectory is likely to be suboptimal for solving the goal , as Alice might arrive at the final state merely by accident . Therefore , we only consider trajectories with goals that Bob failed to solve as demonstrations , to avoid distracting Bob with suboptimal examples . Whenever Bob fails , we relabel Alice ’ s trajectory ⌧A to be a goal-augmented version ⌧BC = { ( s0 , sT , a0 , r0 ) , . . . , ( sT , sT , aT , rT ) } as a demonstration for BC , where sT is the goal . PPO-style BC loss clipping The objective for training Bob is L = LRL + Labc , where LRL is an RL objective and Labc is the ABC loss . is a hyperparameter controlling the relative importance of the BC loss . We set = 0.5 throughout the whole experiment . A naive BC loss is to minimize the negative log-likelihood of demonstrated actions , E ( st , gt , at ) 2DBC ⇥ log ⇡B ( at|st , gt ; ✓ ) ⇤ where DBC is a mini-batch of demonstration data and ⇡B is parameterized by ✓ . We found that overly-aggressive policy changes triggered by BC sometimes led to learning instabilities . To prevent the policy from changing too drastically , we introduce PPO-style loss clipping ( Schulman et al. , 2017 ) on the BC loss by setting the advantage Â = 1 in the clipped surrogate objective : Labc = E ( st , gt , at ) 2DBC clip ⇣ ⇡B ( at|st , gt ; ✓ ) ⇡B ( at|st , gt ; ✓old ) , 1 ✏ , 1 + ✏ ⌘ where ⇡B ( at|st , gt ; ✓ ) is Bob ’ s likelihood on a demonstration based on the parameters that we are optimizing , and ⇡B ( at|st , gt ; ✓old ) is the likelihood based on Bob ’ s behavior policy ( at the time of demonstration collection ) evaluated on a demonstration . This behavior policy is identical to the policy that we use to collect RL trajectories . By setting Â = 1 , this objective optimizes the naive BC loss , but clips the loss whenever ⇡B ( at|st , gt ; ✓ ) ⇡B ( at|st , gt ; ✓old ) is bigger than 1 + ✏ , to prevent the policy from changing too much . ✏ is a clipping threshold and we use ✏ = 0.2 in all the experiments .

Summary: This paper presents an approach to learn goal conditioned policies by relying on self-play which sets the goals and discovers a curriculum of tasks for learning. Alice and Bob are the agents. Alice's task is to set a goal by following a number of steps in the environment and she is rewarded when the goal is too challenging for Bob to solve. Bob's task is to solve the task by trying to reproduce the end state of Alice's demonstration. As a result, the learned policy performs various tasks and can work in zero-shot settings.