REPAINT: Knowledge Transfer in Deep Actor-Critic Reinforcement Learning

1 INTRODUCTION . Most reinforcement learning methods train an agent from scratch , typically requiring a huge amount of time and computing resources . Accelerating the learning processes for complex tasks has been one of the most challenging problems in reinforcement learning ( Kaelbling et al. , 1996 ; Sutton & Barto , 2018 ) . In the past few years , deep reinforcement learning has become more ubiquitous to solve sequential decision-making problems in many real-world applications , such as game playing ( OpenAI et al. , 2019 ; Silver et al. , 2016 ) , robotics ( Kober et al. , 2013 ; OpenAI et al. , 2018 ) , and autonomous driving ( Sallab et al. , 2017 ) . The computational cost of learning grows as the task complexity increases in the real-world applications . Therefore , it is desirable for a learning algorithm to leverage knowledge acquired in one task to improve performance on other tasks . Transfer learning has achieved significant success in computer vision , natural language processing , and other knowledge engineering areas ( Pan & Yang , 2009 ) . In transfer learning , the teacher ( source ) and student ( target ) tasks are not necessarily drawn from the same distribution ( Taylor et al. , 2008a ) . The unseen student task may be a simple task which is similar to the previously trained tasks , or a complex task with traits borrowed from significantly different teacher tasks . Despite the prevalence of direct weight transfer , knowledge transfer from previously trained agents for reinforcement learning tasks has not been gaining much attention until recently ( Barreto et al. , 2019 ; Ma et al. , 2018 ; Schmitt et al. , 2018 ; Lazaric , 2012 ; Taylor & Stone , 2009 ) . In this work , we propose a knowledge transfer algorithm for deep actor-critic reinforcement learning , i.e. , REPresentation And INstance Transfer ( REPAINT ) . The algorithm can be categorized as a representation-instance transfer approach . Specifically , in representation transfer , we adopt a kickstarted training method ( Schmitt et al. , 2018 ) using a previously trained teacher policy , where the teacher policy is used for computing the auxiliary loss during training . In instance transfer , we develop a new sampling algorithm for the replay buffer collected from the teacher policy , where we only keep the transitions that have advantage estimates greater than a threshold . The experimental results across several transfer learning tasks show that , regardless of the similarity between source and target tasks , by introducing knowledge transfer with REPAINT , the number of training iterations needed by the agent to achieve some reward target can be significantly reduced when compared to training from scratch and training with only representation transfer or instance transfer . Additionally , the agent ’ s asymptotic performance is also improved by REPAINT in comparison with the baselines . 2 RELATED WORK : TRANSFER REINFORCEMENT LEARNING . Transfer learning algorithms in reinforcement learning can be characterized by the definition of transferred knowledge , which contains the parameters of the reinforcement learning algorithm , the representation of the trained policy , and the instances collected from the environment ( Lazaric , 2012 ) . When the teacher and student tasks share the same state-action space and they are similar enough ( Ferns et al. , 2004 ; Phillips , 2006 ) , parameter transfer is the most straightforward approach , namely , one can initialize the policy or value network in the student tasks by that from teacher tasks ( Mehta et al. , 2008 ; Rajendran et al. , 2015 ) . Parameter transfer with different state-action variables is more complex , where the crucial aspect is to find a suitable mapping from the teacher state-action space to the student state-action space ( Gupta et al. , 2017 ; Talvitie & Singh , 2007 ; Taylor et al. , 2008b ) . Most of the transfer learning algorithms fall into the category of representation transfer , where the reinforcement learning algorithm learns a specific representation of the task or the solution , and the transfer algorithm performs an abstraction process to fit it into the student task . Konidaris et al . ( 2012 ) uses the reward shaping approach to learn a portable shaping function for knowledge transfer , while some other works use neural networks for feature abstraction ( Duan et al. , 2016 ; Parisotto et al. , 2015 ; Zhang et al. , 2018 ) . Policy distillation ( Rusu et al. , 2015 ) or its variants is another popular choice for learning the teacher task representation , where the student policy aims to mimic the behavior of pre-trained teacher policies during its own learning process ( Schmitt et al. , 2018 ; Yin & Pan , 2017 ) . Recently , successor representation has been widely used in transfer reinforcement learning , in which the rewards are assumed to share some common features , so that the value function can be simply written as a linear combination of the successor features ( SF ) ( Barreto et al. , 2017 ; Madarasz & Behrens , 2019 ) . Barreto et al . ( 2019 ) extends the method of using SF and generalised policy improvement in Q-learning ( Sutton & Barto , 2018 ) to more general environments . Borsa et al . ( 2018 ) , Ma et al . ( 2018 ) , and Schaul et al . ( 2015a ) learn a universal SF approximator for transfer . The basic idea of instance transfer algorithms is that the transfer of teacher samples may improve the learning on student tasks . Lazaric et al . ( 2008 ) and Tirinzoni et al . ( 2018 ) selectively transfer samples on the basis of the compliance between tasks in a model-free algorithm , while Taylor et al . ( 2008a ) studies how a model-based algorithm can benefit from samples coming from the teacher task . However , most of the aforementioned algorithms either assume specific forms of reward functions or perform well only when the teacher and student tasks are similar . Additionally , very few algorithms are designated to actor-critic reinforcement learning . In this work , we propose a representationinstance transfer algorithm to handle the generic cases of task similarity , which is also naturally fitted for actor-critic algorithms and can be easily extended to other policy gradient based algorithms . 3 BACKGROUND : ACTOR-CRITIC REINFORCEMENT LEARNING . A general reinforcement learning ( RL ) agent interacting with environment can be modeled in a Markov decision process ( MDP ) , which is defined by a tuple M = ( S , A , p , r , γ ) , where S and A are sets of states and actions , respectively . The state transfer function p ( ·|s , a ) maps a state and action pair to a probability distribution over states . r : S × A× S → R denotes the reward function that determines a reward received by the agent for a transition from ( s , a ) to s′ . The discount factor , γ ∈ [ 0 , 1 ] , provides means to obtain a long-term objective . Specifically , the goal of an RL agent is to learn a policy π that maps a state to a probability distribution over actions at each time step t , so that at ∼ π ( ·|st ) maximizes the accumulated discounted return ∑ t≥0 γ tr ( st , at , st+1 ) . To address this problem , a popular choice to adopt is the model-free actor-critic architecture , e.g. , Konda & Tsitsiklis ( 2000 ) ; Degris et al . ( 2012 ) ; Mnih et al . ( 2016 ) ; Schulman et al . ( 2015a ; 2017 ) , where the critic estimates the value function and the actor updates the policy distribution in the direction suggested by the critic . The state value function at time t is defined as V π ( s ) = Eai∼π ( ·|si ) ∑ i≥t γi−tr ( si , ai , si+1 ) ∣∣∣∣st = s . ( 3.1 ) Similarly , the action value function ( also called Q function ) is defined by Qπ ( s , a ) = Eai∼π ( ·|si ) ∑ i≥t γi−tr ( si , ai , si+1 ) ∣∣∣∣st = s , at = a . ( 3.2 ) The actor-critic methods usually rely on the advantage function , which is computed as Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . ( 3.3 ) Intuitively , the advantage can be taken as the extra reward that could be obtained by taking a particular action a . The advantage is usually approximated by generalized advantage estimator ( GAE ) ( Schulman et al. , 2015b ) , which is defined as the exponentially-weighted average of the k-step discounted advantage , namely , Ât = Â GAE ( γ , λ ) t : = ∞∑ l=0 ( γλ ) l ( rt+l + γV π ( st+l+1 ) − V π ( st+l ) ) , ( 3.4 ) where the parameter 0 ≤ λ ≤ 1 allows a trade-off of the bias and variance . In deep RL , the critic and actor functions are usually parameterized by neural networks . Then the policy gradient methods can be used to update the actor network . For example , in the clipped proximal policy optimization ( Clipped PPO ) ( Schulman et al. , 2017 ) , the policy ’ s objective function is defined to be the minimum between the standard surrogate objective and an clipped objective : Lclip ( θ ) = Êt [ min ( rt ( θ ) · Ât , clip ( rt ( θ ) , 1− , 1 + ) · Ât ) ] , ( 3.5 ) where the policy π is parameterized by θ , and rt ( θ ) is the likelihood ratio that rt ( θ ) = πθ ( at|st ) πθold ( at|st ) . ( 3.6 ) Moreover , the clip function truncates rt ( θ ) to the range of ( 1− , 1 + ) . 4 REPRESENTATION AND INSTANCE TRANSFER IN ACTOR-CRITIC RL . In this section , we describe our knowledge transfer algorithm , REPAINT , for actor-critic RL framework . There are two core concepts underlying our approach , i.e. , representation transfer and instance transfer . In the representation transfer , we employ a kickstarted training approach ( Schmitt et al. , 2018 ) based on the policy distillation . Then , in the next iteration following the kickstarting , we update the student policy using an advantage-based experience replay . We do not assume that the teacher and student tasks are similar or drawn from the same distribution . The REPAINT algorithm is provided in Algorithm 1 , where the value function and policy function are parameterized by ν and θ , respectively . Without loss of generality , we demonstrate the policy update using the Clipped PPO loss stated in equation 3.5 , and use a single teacher policy in both representation and instance transfers . In practice , it can be directly applied to any advantage-based policy gradient RL algorithms , and it is straightforward to have different and multiple teacher policies in each transfer step . The more general algorithm and several variants are presented in Section A . 4.1 REPRESENTATION TRANSFER : KICKSTARTED TRAINING . In representation transfer , we use a kickstarting training pipeline ( Schmitt et al. , 2018 ) , which can be viewed as a combination of policy distillation ( Rusu et al. , 2015 ) and population based training ( Jaderberg et al. , 2017 ) . The main idea is to employ an auxiliary loss function which encourages the student policy to be close to the teacher policy on the trajectories sampled by the student . Given a teacher policy πteacher , we introduce the auxiliary loss as Ldistill ( θ ) = H ( πteacher ( a|s ) ‖πθ ( a|s ) ) , ( 4.1 ) where H ( ·‖· ) is the cross-entropy . In order for the agent to maximize its own future rewards , kickstarting adds the above loss to the Clipped PPO objective function , i.e. , equation 3.5 , weighted at optimization iteration k by the scaling βk ≥ 0 : LkRL ( θ ) = Lclip ( θ ) − βkLdistill ( θ ) . ( 4.2 ) Algorithm 1 REPAINT algorithm with Clipped PPO for iteration k = 1 , 2 , . . . do if k is odd then Collect trajectories S = { ( s , a , s′ , r ) } following πθold ( s ) Fit state value network Vν using S to update ν Compute advantage estimates Â1 , . . . , ÂT using equation 3.4 Perform gradient optimization on LkRL ( θ ) defined in equation 4.2 // representation transfer else Collect trajectories S ′ = { ( s , a , s′ ) } following πteacher ( s ) // instance transfer Compute r for each transition using current reward function and add to S ′ Compute advantage estimates Â′1 , . . . , Â ′ T ′ using equation 3.4 for t=1 , . . . , T ′ do if Â′t < ζ then Remove Â′t and the corresponding transition ( st , at , st+1 , rt ) from S ′ Perform gradient optimization on Lclip ( θ ) defined in equation 3.5 In our experiments , the weighting parameter βk is relatively large at early iterations , and vanishes as k increases , which is expected to improve the initial performance of the agent while keeping it focused on the current task in later iterations .

Summary: This paper deals with transfer learning in RL. The problem broadly defined is to improve performance of an agent on a new task, given an agent (teacher) trained on a previous (different) task. There are multiple approaches to this from fine-tuning, distillation to instance transfer. A central question in this field is how to transfer learned useful behavior even when task similarity is low. While most methods would work well on similar tasks, naively applying, for example distillation, would hurt when training on a very different target task.