SPLID: Self-Imitation Policy Learning through Iterative Distillation

1 INTRODUCTION . Goal-Conditioned Sparse Reward ( GCSR ) tasks are one of the most challenging tasks in reinforcement learning . In these tasks , the goal is combined with the current state as the input of a policy , and only when the agent successfully achieves the desired goal will the agent be assigned with a nontrivial reward , making the task extremely challenging for random-exploration-based algorithms . In many cases , the GCSR task is closely related to the Multi-Goal task , where the goal is not fixed and can be set anywhere in the state space . In order to solve a GCSR task , an agent has to learn a generalizable solution that can be applied to a variety of similar tasks . For example , robotic object grasping is such a GCSR task : the target object can be placed at any position on the table , henceforth , the robot needs to control its arm to get closer to the position of the object and then grasps it . The objective of learning such a policy is to find a feasible path from the current state to the goal ( Tamar et al. , 2016 ) . The work of Plappert et al . ( 2018 ) provides a series of these Goal-Conditioned tasks in robotics control . In previous works , reward shaping ( Ng et al. , 1999 ) , hierarchical reinforcement learning ( Dietterich , 2000 ; Barto & Mahadevan , 2003 ) , curriculum learning ( Bengio et al. , 2009 ) , and learning from demonstrations ( Schaal , 1997 ; Atkeson & Schaal , 1997 ; Argall et al. , 2009 ; Hester et al. , 2018 ; Nair et al. , 2018 ) is proposed to tackle the challenges of learning through sparse rewards . All of those approaches heavily rely on prior knowledge or expert demonstrations for a given task . The work of Hindsight Experience Replay ( HER ) ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ) is proposed to relabel failed trajectories and assigns hindsight credits as complementary to the primal sparse rewards , to address the difficulty in exploration through the way of Temporal Difference ( TD ) learning . Differently , the recent work of GCSL ( Ghosh et al. , 2019b ) proposes to solve the GCSR tasks , extending HER through iterative supervised learning , i.e. , there are no value networks that are optimized based on TD learning . However , in both HER and GCSL , there is no guarantee on the quality of the relabeled data , nor there is a guarantee on the policy improvement based on those relabeled data . On the other hand , the work of PCHID ( Sun et al. , 2019 ) proposes an intuitive approach for GCSR tasks in a supervised learning manner , but the sample efficiency is limited by the explicit curriculum setting . In this work , we aim at improving sample efficiency in GCSR tasks , by improving the quality of relabeled data . Notice that GCSL performs behavior cloning on all the successful experience after relabeled without considering improving the quality of the relabeled data , where quality can be measured by the timesteps taken to achieve the goal or other means . Since the quality of the expert demonstration that behavior cloning relies on determines the performance and the sample efficiency of the training , it is natural to ask if we can also improve the quality in the target of the behavior cloning . To this end , we propose a meta-algorithm Evolutionary Policy Distillation ( SPLID ) , motivated by the theoretical insight , and instantiate it with a practical algorithm in the GCSR tasks to further improve learning efficiency . We summarize our contributions as follows : • To evaluate the quality of the policy in the Multi-Goal RL , we introduce the concept of δ-distilled policy . Motivated by leveling up the quality of the target policy of the behavior cloning , we propose a novel meta-algorithm called SPLID , which iteratively distills the target policy through pursuing δ-distilled policy and conducting self-imitation learning . • From the theoretical aspect , we prove that under certain assumptions , SPLID has a monotonic performance improvement guarantee and a local convergence guarantee . • We specialize such meta-algorithm on a practical implementation in the GCSR tasks with the deterministic environment , where SPLID searches δ-distilled policy by distinguishing First Hit Time ( FHT ) and enhances exploration by parameter noise . • We demonstrate the proposed practical implementations on the goal-oriented tasks benchmark and show our method SPLID can work in isolation to solve GCSR tasks with improved sample efficiency than previous methods involving HER , PCHID , GCSL , and ES . 2 PRELIMINARIES . Multi-Goal Markov Decision Process . We first consider a Markov Decision Process ( MDP ) denoted by a tupleM containing ( S , A , r , T , ρ ( s0 ) , G , γ ) , where S is a finite discrete state space , A denotes the finite discrete action space , r ( s , g ) : S×G 7→ R is the reward function . In this paper we focus on a special case where the reward function is binary , i.e. , tasks with sparse reward . Without loss of generalization , we also assume that reward is bounded in [ 0 , 1 ] . γ ∈ ( 0 , 1 ) is the discount factor . T : S × A 7→ ∆ ( S ) is the transition kenrel and T ( st+1 | st , at ) represents the probability of the state transition from st to st+1 given the action at . Here we denote by ∆ ( X ) distribution set defined on any measurable set X . ρ ( s0 ) is the start state distribution . G is the finite goal space and we assume the goal obeys the fixed distribution pG ( · ) ∈ ∆ ( G ) at the beginning of each episode . Policy of the agent takes the form π ( a | s , g ) : S × G 7→ ∆ ( S ) and π induces the state distribution ρtπ ( · | g ) = P ( st = · | π , g ) . We denote ρπ ( · | g ) = 11−γ ∑∞ t=1 γ tρtπ ( · ) as the discounted average state distribution induced by π . For convenience , we assume that the environment transits to the null state with recieving reward 1 each time step once the agent reaches the goal . For the introduction of nerual network , we parameterize the policy π by πθ , where parameter lies in a certain region θ ∈ Θ . We denote that ΠΘ = { πθ : θ ∈ Θ } the parameterized policy class and assume that optimal policy falls into it , i.e. , π ? ∈ ΠΘ . Universal Value Function Approximator The Universal Value Function Approximator ( UVFA ) Schaul et al . ( 2015 ) extends the state space of Deep Q-Networks ( DQN ) Mnih et al . ( 2015 ) to include goal state as part of the input , which is useful in the setting where there are multiple goals to achieve . Moreover , Schaul et al . ( 2015 ) shows that in the UVFA setting , the learned policy can generalize to previously unseen state-goal pairs . Specifically , let S ( S , G ) = S×G denote the extended state space ofM where G is a finite goal space . Normally , a representation mapping m ( · ) : S → G is assumed to be known in such multi-goal RL frameworks ( Plappert et al. , 2018 ) . Hindsight Experience Replay . Learning with sparse rewards in RL problems remains challenging due to the difficulty to reach the reward through random explorations . Hindsight Experience Replay ( HER ) proposes to relabel the failed rollouts as successful ones ( Andrychowicz et al. , 2017 ) as a method to deal with the goal-oriented sparse reward problem . The agent in HER receives a reward when reaching either the original goal or the relabeled goal in each episode by storing both original transition pairs ( ( st , g ) , at , ( st+1 , g ) , r ) and relabeled transitions ( ( st , g′ ) , at , ( st+1 , g′ ) , r′ ) in the replay buffer , where g′ is a certain hindsight goal that will be visited in the following steps , i.e. , there exist k such that g′ = m ( st+k ) , and r′ is the hindsight reward , which will be 1 when the hindsight goal is achieved . 3 SELF-IMITATION POLICY LEARNING THROUGH ITERATIVE DISTILLATION . 3.1 THEORETICAL FRAMEWORK . First , we consider a multi-goal MDP with infinite horizon , where the discounted goal-specified return V ( π , g ) takes the form V ( π , g ) = ∞∑ t=0 Est∼ρtπ ( ·|g ) [ γ tr ( st , g ) ] , and ρtπ is the visitation measure induced by π . Then the target of the agent is to maximize the discounted return J ( π ) , i.e. , J ( π ) = Eg∼pGV ( π , g ) . This problem can be extremely hard when the goal specified reward r ( · , g ) is sparse or binary . To address this problem with sample efficiency , we are motivated by behavior cloning that mimics a policy from an oracle in a supervised learning manner , e.g . human experts . To implement such a supervised learning method , one issue faced is the access to the oracle data , which is not available in our problem setting , and the quality of the oracle data also determines the performance of the mimicked policy . Although HER ( Rauber et al. , 2017 ) proposes to relabel failure trajectory to successful one with a different goal and GCSL ( Ghosh et al. , 2019b ) applies HER in the behavior cloning method , it is inadequate for GCSL to achieve sample efficiency . Because it considers all the policy reaching the goal as the target in the supervised learning and the quality of the target policy it mimics may be a burden in improving the sample efficiency . To this end , besides the combination of HER and behavior cloning as GCSL do , we desire to improve the target policy dynamically , that is assured by an oracle to be qualified to lead the present policy . Such a process corresponds to the policy distillation , where it is oriented with an evolutionary target policy selected by an oracle . We will introduce how we implement such oracle-guided policy distillation by the paradigm of behavior cloning to boost sample efficiency from the theoretical perspective . Similar to previous work ( Ghosh et al. , 2019b ; Xu et al. , 2019 ) , we only study the discrete case from the perspective of theory . To utilize the relabeling strategy ( Rauber et al. , 2017 ) to address GCSR , we first analyze the formal definition of relabeling strategy . Definition 1 ( Relabeling Strategy ) . We define the relabeling strategy of policy π with initial goal g in the t-timestep as ηtπ ( g ′ | g ) : G × G × N×ΠΘ 7→ ∆ ( G ) . By definition 1 , relabeling strategy is a time varying distribution , reflecting the relabeling preference w.r.t . the time step in one trajectory . We remark that in HER , such relabelling distribution matches the average state visitation distribution induced by the model policy ηtπ ( · | g ) ∝ ∑∞ i=t ρπ ( · | g ) , and in GCSL it matches the last step visitation distribution ( finite horizon T ) i.e. , ηtπ ( · | g ) = ρTπ ( · | g ) , for all g ∈ G and π ∈ ΠΘ . The next step is to formally evaluate the quality of the target policy that acts as the target of behavior cloning . We introduce the definition of δ-distilled policy as follows . Definition 2 ( δ-Distilled Policy ) . We define the δ-distilled policy π̃ ∈ ΠΘ of a policy π ∈ ΠΘ , if the following condition is satisfied : G ( π̃‖π ) : = ∑ g′∈G Eg∼ρπ̃ ( m−1 ( · ) |g′ ) [ pG ( g ) · ( V ( π̃ , g ) − V ( π , g ) ) ] ≥ δ , where δ is a nonnegative constant and ρπ̃ ( m−1 ( · ) | g′ ) is defined as ∑ { s∈S : m ( s ) =· } ρπ̃ ( m ( s ) | g′ ) . Intuitively , if π̃ is the δ-distilled policy of π , it means that π̃ performs nearly strictly better than π for the goals that π̃ reached . Here the initial goal distribution pG ( g ) acts as the weight in the weighted sum , reflecting the prior favor of such performance advantage . We remark that δ-distilled policy does not directly give the relationship of the return J ( π̃ ) and J ( π ) because of the covirate shift in the goal distribution . Nevertheless , such δ-distilled policy is easier to search than the policy with higher return , since we only need to focus on the goals reached by the δ-distilled policy itself rather than the entire goal space G. Before we move on , we first need to avoid the meaningless case when δ keeps zero , which is achieved by the following assumption . Assumption 1 ( Avoid δ Keeping Zero ) . We assume that supp ( pG ( · ) ) ∩ supp ( ρπ ( m−1 ( · ) | g ) ) 6= Φ , for all goal g ∈ G and policy π ∈ ΠΘ , where Φ is the empty set . Assumption 1 excludes the environments where the goal-oriented regions under pG are blocked away from the agents . This assumption can be easily satisfied by a wide variety of tasks in RL and multi-goal RL , e.g . goal-reaching tasks . Based on the assumption and previous intuitive insights , we present a meta-algorithm in Algorithm 1 to solve GCSR tasks iteratively , which is named Self-Imitation Policy Learning through Iterative Distillation ( SPLID ) and involves three main stages for each episode . Stage 1 : Exploration To search for such δ-distilled policy , we need produce new behavior policy π̃ ( k ) from old model policy πθ ( k−1 ) with the enhancement of exploration . To ensure robustness , we control such exploration span by constrainting the total variation between π̃ ( k ) and πθ ( k−1 ) as sup g∈G , s∈S DTV ( π̃ ( k ) ( · | s , g ) ‖πθ ( k−1 ) ( · | s , g ) ) ≤ , ( 1 ) where is an absolute constant . Stage 2 : Distill the Target We owe an oracle to judge whether this new behavior policy π̃ ( k ) is our desired δ-distilled policy . If it is , then we mix π̃ ( k ) with old target policy π ( k−1 ) with a certain proportion , otherwise we skip this stage without changing target policy , i.e. , π ( k ) = π ( k−1 ) . The mixture process satisfies that ρπ ( k ) = ( 1− αk ) ρπ ( k−1 ) + αkρπ̃ ( k ) , ( 2 ) where αk ∈ ( 0 , 1 ) is the mixture proportion at the k-th episode . Such a mixture process is also used for ensure the smoothness of the update . Stage 3 : Behavior Cloning To update parameter θ ( k ) , we conduct behavior cloning towards the π ( k ) after relabeling , where the relabeled strategy is η . If we define the surrogate function Fη ( πθ‖π̄ ) as Fη ( πθ‖π̄ ) : = Eg∼pG ( · ) ∞∑ t=0 Est∼ρtπ̄ ( ·|g ) , g′∼ηtπ̄ ( ·|g ) γ tDTV ( π̄ ( · | s , g′ ) ‖πθ ( · | s , g′ ) ) , ( 3 ) for all policy π , π̄ ∈ ΠΘ , and goal g ∈ G , where DTV ( p‖q ) : = ∑ x |p ( x ) − q ( x ) |/2 is the total variation . Although in behavior cloning , KL divergence is usually adopted as the distance of polices , minimizing KL divergence is also equivalent to minimize total variation by Pinker ’ s Inequality . Then the behavior cloning process corresponds to the following optimization w.r.t . θ. θ ( k ) = argmin θ∈Θ Fη ( πθ‖π ( k ) ) . ( 4 ) Algorithm 1 Self-Imitation Policy Learning through Iterative Distillation ( SPLID ) : Meta Algorithm 1 : Initialize parameter θ ( 0 ) and target policy π ( 0 ) with uniform distribution . 2 : for episode k = 1 , ... , K do 3 : Enhance exploration from πθ ( k−1 ) to generate a new behavior policy π̃ ( k ) with constraint Equation 1 . 4 : Sample trajectories from the environment following π̃ ( k ) ( · | · , g ) . //Exploration 5 : if π̃ ( k ) is a δ-distilled policy ( Definition 2 ) of πθ ( k−1 ) then 6 : Mix π̃ ( k ) and π ( k−1 ) to obtain π ( k ) via Equation 2 . //Distill the target 7 : else 8 : Keep target policy unchanged π ( k ) = π ( k−1 ) . 9 : end if 10 : Relabel the target policy π ( k ) and conduct behavior cloning to update θ ( k ) via Equation 4 . 11 : //Behavior Cloning 12 : end for

Summary: The authors consider the Goal-Conditioned continuous control RL task, which is difficult due to reward sparsity. The authors propose a method for training deep RL agents in a more sample efficient manner through self imitation on policy traces that have been "distilled", i.e., randomly perturbed and argmax'd according to some metric like "First Hit Time", then using hindsight experience replay to relabel traces (handling reward sparsity). This improves of over the previous state of the art which did not use the policy distillation step, simply relabeling with the most recent policy, prone to error propagation. The authors demonstrate that their policy distillation step improves the state of the art on 3 RL baseline settings, and also give theoretical backing to their approach and conditions for convergence.