SPLID: Self-Imitation Policy Learning through Iterative Distillation

1 INTRODUCTION . Goal-Conditioned Sparse Reward ( GCSR ) tasks are one of the most challenging tasks in reinforcement learning . In these tasks , the goal is combined with the current state as the input of a policy , and only when the agent successfully achieves the desired goal will the agent be assigned with a nontrivial reward , making the task extremely challenging for random-exploration-based algorithms . In many cases , the GCSR task is closely related to the Multi-Goal task , where the goal is not fixed and can be set anywhere in the state space . In order to solve a GCSR task , an agent has to learn a generalizable solution that can be applied to a variety of similar tasks . For example , robotic object grasping is such a GCSR task : the target object can be placed at any position on the table , henceforth , the robot needs to control its arm to get closer to the position of the object and then grasps it . The objective of learning such a policy is to find a feasible path from the current state to the goal ( Tamar et al. , 2016 ) . The work of Plappert et al . ( 2018 ) provides a series of these Goal-Conditioned tasks in robotics control . In previous works , reward shaping ( Ng et al. , 1999 ) , hierarchical reinforcement learning ( Dietterich , 2000 ; Barto & Mahadevan , 2003 ) , curriculum learning ( Bengio et al. , 2009 ) , and learning from demonstrations ( Schaal , 1997 ; Atkeson & Schaal , 1997 ; Argall et al. , 2009 ; Hester et al. , 2018 ; Nair et al. , 2018 ) is proposed to tackle the challenges of learning through sparse rewards . All of those approaches heavily rely on prior knowledge or expert demonstrations for a given task . The work of Hindsight Experience Replay ( HER ) ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ) is proposed to relabel failed trajectories and assigns hindsight credits as complementary to the primal sparse rewards , to address the difficulty in exploration through the way of Temporal Difference ( TD ) learning . Differently , the recent work of GCSL ( Ghosh et al. , 2019b ) proposes to solve the GCSR tasks , extending HER through iterative supervised learning , i.e. , there are no value networks that are optimized based on TD learning . However , in both HER and GCSL , there is no guarantee on the quality of the relabeled data , nor there is a guarantee on the policy improvement based on those relabeled data . On the other hand , the work of PCHID ( Sun et al. , 2019 ) proposes an intuitive approach for GCSR tasks in a supervised learning manner , but the sample efficiency is limited by the explicit curriculum setting . In this work , we aim at improving sample efficiency in GCSR tasks , by improving the quality of relabeled data . Notice that GCSL performs behavior cloning on all the successful experience after relabeled without considering improving the quality of the relabeled data , where quality can be measured by the timesteps taken to achieve the goal or other means . Since the quality of the expert demonstration that behavior cloning relies on determines the performance and the sample efficiency of the training , it is natural to ask if we can also improve the quality in the target of the behavior cloning . To this end , we propose a meta-algorithm Evolutionary Policy Distillation ( SPLID ) , motivated by the theoretical insight , and instantiate it with a practical algorithm in the GCSR tasks to further improve learning efficiency . We summarize our contributions as follows : • To evaluate the quality of the policy in the Multi-Goal RL , we introduce the concept of δ-distilled policy . Motivated by leveling up the quality of the target policy of the behavior cloning , we propose a novel meta-algorithm called SPLID , which iteratively distills the target policy through pursuing δ-distilled policy and conducting self-imitation learning . • From the theoretical aspect , we prove that under certain assumptions , SPLID has a monotonic performance improvement guarantee and a local convergence guarantee . • We specialize such meta-algorithm on a practical implementation in the GCSR tasks with the deterministic environment , where SPLID searches δ-distilled policy by distinguishing First Hit Time ( FHT ) and enhances exploration by parameter noise . • We demonstrate the proposed practical implementations on the goal-oriented tasks benchmark and show our method SPLID can work in isolation to solve GCSR tasks with improved sample efficiency than previous methods involving HER , PCHID , GCSL , and ES . 2 PRELIMINARIES . Multi-Goal Markov Decision Process . We first consider a Markov Decision Process ( MDP ) denoted by a tupleM containing ( S , A , r , T , ρ ( s0 ) , G , γ ) , where S is a finite discrete state space , A denotes the finite discrete action space , r ( s , g ) : S×G 7→ R is the reward function . In this paper we focus on a special case where the reward function is binary , i.e. , tasks with sparse reward . Without loss of generalization , we also assume that reward is bounded in [ 0 , 1 ] . γ ∈ ( 0 , 1 ) is the discount factor . T : S × A 7→ ∆ ( S ) is the transition kenrel and T ( st+1 | st , at ) represents the probability of the state transition from st to st+1 given the action at . Here we denote by ∆ ( X ) distribution set defined on any measurable set X . ρ ( s0 ) is the start state distribution . G is the finite goal space and we assume the goal obeys the fixed distribution pG ( · ) ∈ ∆ ( G ) at the beginning of each episode . Policy of the agent takes the form π ( a | s , g ) : S × G 7→ ∆ ( S ) and π induces the state distribution ρtπ ( · | g ) = P ( st = · | π , g ) . We denote ρπ ( · | g ) = 11−γ ∑∞ t=1 γ tρtπ ( · ) as the discounted average state distribution induced by π . For convenience , we assume that the environment transits to the null state with recieving reward 1 each time step once the agent reaches the goal . For the introduction of nerual network , we parameterize the policy π by πθ , where parameter lies in a certain region θ ∈ Θ . We denote that ΠΘ = { πθ : θ ∈ Θ } the parameterized policy class and assume that optimal policy falls into it , i.e. , π ? ∈ ΠΘ . Universal Value Function Approximator The Universal Value Function Approximator ( UVFA ) Schaul et al . ( 2015 ) extends the state space of Deep Q-Networks ( DQN ) Mnih et al . ( 2015 ) to include goal state as part of the input , which is useful in the setting where there are multiple goals to achieve . Moreover , Schaul et al . ( 2015 ) shows that in the UVFA setting , the learned policy can generalize to previously unseen state-goal pairs . Specifically , let S ( S , G ) = S×G denote the extended state space ofM where G is a finite goal space . Normally , a representation mapping m ( · ) : S → G is assumed to be known in such multi-goal RL frameworks ( Plappert et al. , 2018 ) . Hindsight Experience Replay . Learning with sparse rewards in RL problems remains challenging due to the difficulty to reach the reward through random explorations . Hindsight Experience Replay ( HER ) proposes to relabel the failed rollouts as successful ones ( Andrychowicz et al. , 2017 ) as a method to deal with the goal-oriented sparse reward problem . The agent in HER receives a reward when reaching either the original goal or the relabeled goal in each episode by storing both original transition pairs ( ( st , g ) , at , ( st+1 , g ) , r ) and relabeled transitions ( ( st , g′ ) , at , ( st+1 , g′ ) , r′ ) in the replay buffer , where g′ is a certain hindsight goal that will be visited in the following steps , i.e. , there exist k such that g′ = m ( st+k ) , and r′ is the hindsight reward , which will be 1 when the hindsight goal is achieved . 3 SELF-IMITATION POLICY LEARNING THROUGH ITERATIVE DISTILLATION . 3.1 THEORETICAL FRAMEWORK . First , we consider a multi-goal MDP with infinite horizon , where the discounted goal-specified return V ( π , g ) takes the form V ( π , g ) = ∞∑ t=0 Est∼ρtπ ( ·|g ) [ γ tr ( st , g ) ] , and ρtπ is the visitation measure induced by π . Then the target of the agent is to maximize the discounted return J ( π ) , i.e. , J ( π ) = Eg∼pGV ( π , g ) . This problem can be extremely hard when the goal specified reward r ( · , g ) is sparse or binary . To address this problem with sample efficiency , we are motivated by behavior cloning that mimics a policy from an oracle in a supervised learning manner , e.g . human experts . To implement such a supervised learning method , one issue faced is the access to the oracle data , which is not available in our problem setting , and the quality of the oracle data also determines the performance of the mimicked policy . Although HER ( Rauber et al. , 2017 ) proposes to relabel failure trajectory to successful one with a different goal and GCSL ( Ghosh et al. , 2019b ) applies HER in the behavior cloning method , it is inadequate for GCSL to achieve sample efficiency . Because it considers all the policy reaching the goal as the target in the supervised learning and the quality of the target policy it mimics may be a burden in improving the sample efficiency . To this end , besides the combination of HER and behavior cloning as GCSL do , we desire to improve the target policy dynamically , that is assured by an oracle to be qualified to lead the present policy . Such a process corresponds to the policy distillation , where it is oriented with an evolutionary target policy selected by an oracle . We will introduce how we implement such oracle-guided policy distillation by the paradigm of behavior cloning to boost sample efficiency from the theoretical perspective . Similar to previous work ( Ghosh et al. , 2019b ; Xu et al. , 2019 ) , we only study the discrete case from the perspective of theory . To utilize the relabeling strategy ( Rauber et al. , 2017 ) to address GCSR , we first analyze the formal definition of relabeling strategy . Definition 1 ( Relabeling Strategy ) . We define the relabeling strategy of policy π with initial goal g in the t-timestep as ηtπ ( g ′ | g ) : G × G × N×ΠΘ 7→ ∆ ( G ) . By definition 1 , relabeling strategy is a time varying distribution , reflecting the relabeling preference w.r.t . the time step in one trajectory . We remark that in HER , such relabelling distribution matches the average state visitation distribution induced by the model policy ηtπ ( · | g ) ∝ ∑∞ i=t ρπ ( · | g ) , and in GCSL it matches the last step visitation distribution ( finite horizon T ) i.e. , ηtπ ( · | g ) = ρTπ ( · | g ) , for all g ∈ G and π ∈ ΠΘ . The next step is to formally evaluate the quality of the target policy that acts as the target of behavior cloning . We introduce the definition of δ-distilled policy as follows . Definition 2 ( δ-Distilled Policy ) . We define the δ-distilled policy π̃ ∈ ΠΘ of a policy π ∈ ΠΘ , if the following condition is satisfied : G ( π̃‖π ) : = ∑ g′∈G Eg∼ρπ̃ ( m−1 ( · ) |g′ ) [ pG ( g ) · ( V ( π̃ , g ) − V ( π , g ) ) ] ≥ δ , where δ is a nonnegative constant and ρπ̃ ( m−1 ( · ) | g′ ) is defined as ∑ { s∈S : m ( s ) =· } ρπ̃ ( m ( s ) | g′ ) . Intuitively , if π̃ is the δ-distilled policy of π , it means that π̃ performs nearly strictly better than π for the goals that π̃ reached . Here the initial goal distribution pG ( g ) acts as the weight in the weighted sum , reflecting the prior favor of such performance advantage . We remark that δ-distilled policy does not directly give the relationship of the return J ( π̃ ) and J ( π ) because of the covirate shift in the goal distribution . Nevertheless , such δ-distilled policy is easier to search than the policy with higher return , since we only need to focus on the goals reached by the δ-distilled policy itself rather than the entire goal space G. Before we move on , we first need to avoid the meaningless case when δ keeps zero , which is achieved by the following assumption . Assumption 1 ( Avoid δ Keeping Zero ) . We assume that supp ( pG ( · ) ) ∩ supp ( ρπ ( m−1 ( · ) | g ) ) 6= Φ , for all goal g ∈ G and policy π ∈ ΠΘ , where Φ is the empty set . Assumption 1 excludes the environments where the goal-oriented regions under pG are blocked away from the agents . This assumption can be easily satisfied by a wide variety of tasks in RL and multi-goal RL , e.g . goal-reaching tasks . Based on the assumption and previous intuitive insights , we present a meta-algorithm in Algorithm 1 to solve GCSR tasks iteratively , which is named Self-Imitation Policy Learning through Iterative Distillation ( SPLID ) and involves three main stages for each episode . Stage 1 : Exploration To search for such δ-distilled policy , we need produce new behavior policy π̃ ( k ) from old model policy πθ ( k−1 ) with the enhancement of exploration . To ensure robustness , we control such exploration span by constrainting the total variation between π̃ ( k ) and πθ ( k−1 ) as sup g∈G , s∈S DTV ( π̃ ( k ) ( · | s , g ) ‖πθ ( k−1 ) ( · | s , g ) ) ≤ , ( 1 ) where is an absolute constant . Stage 2 : Distill the Target We owe an oracle to judge whether this new behavior policy π̃ ( k ) is our desired δ-distilled policy . If it is , then we mix π̃ ( k ) with old target policy π ( k−1 ) with a certain proportion , otherwise we skip this stage without changing target policy , i.e. , π ( k ) = π ( k−1 ) . The mixture process satisfies that ρπ ( k ) = ( 1− αk ) ρπ ( k−1 ) + αkρπ̃ ( k ) , ( 2 ) where αk ∈ ( 0 , 1 ) is the mixture proportion at the k-th episode . Such a mixture process is also used for ensure the smoothness of the update . Stage 3 : Behavior Cloning To update parameter θ ( k ) , we conduct behavior cloning towards the π ( k ) after relabeling , where the relabeled strategy is η . If we define the surrogate function Fη ( πθ‖π̄ ) as Fη ( πθ‖π̄ ) : = Eg∼pG ( · ) ∞∑ t=0 Est∼ρtπ̄ ( ·|g ) , g′∼ηtπ̄ ( ·|g ) γ tDTV ( π̄ ( · | s , g′ ) ‖πθ ( · | s , g′ ) ) , ( 3 ) for all policy π , π̄ ∈ ΠΘ , and goal g ∈ G , where DTV ( p‖q ) : = ∑ x |p ( x ) − q ( x ) |/2 is the total variation . Although in behavior cloning , KL divergence is usually adopted as the distance of polices , minimizing KL divergence is also equivalent to minimize total variation by Pinker ’ s Inequality . Then the behavior cloning process corresponds to the following optimization w.r.t . θ. θ ( k ) = argmin θ∈Θ Fη ( πθ‖π ( k ) ) . ( 4 ) Algorithm 1 Self-Imitation Policy Learning through Iterative Distillation ( SPLID ) : Meta Algorithm 1 : Initialize parameter θ ( 0 ) and target policy π ( 0 ) with uniform distribution . 2 : for episode k = 1 , ... , K do 3 : Enhance exploration from πθ ( k−1 ) to generate a new behavior policy π̃ ( k ) with constraint Equation 1 . 4 : Sample trajectories from the environment following π̃ ( k ) ( · | · , g ) . //Exploration 5 : if π̃ ( k ) is a δ-distilled policy ( Definition 2 ) of πθ ( k−1 ) then 6 : Mix π̃ ( k ) and π ( k−1 ) to obtain π ( k ) via Equation 2 . //Distill the target 7 : else 8 : Keep target policy unchanged π ( k ) = π ( k−1 ) . 9 : end if 10 : Relabel the target policy π ( k ) and conduct behavior cloning to update θ ( k ) via Equation 4 . 11 : //Behavior Cloning 12 : end for

Summary: This paper presents a self-imitation learning method for goal-conditioned continuous control tasks. The key idea is to identify a $\delta$-distilled policy that performs better than the current policy. Then a mixture policy is generated if a $\delta$-distilled policy is identified. Finally, the parameters are updated with behavior cloning to the relabeled data. Experiments on three goal-oriented tasks demonstrate the efficiency of the proposed method.