Multi-batch Reinforcement Learning via Sample Transfer and Imitation Learning

1 INTRODUCTION . Reinforcement learning aims to learn an optimal control policy through interactions with the environment ( Sutton & Barto , 2018 ) . Deep reinforcement learning ( Deep RL or DRL ) combines neural networks with reinforcement learning and further enables RL agents to deal with more complex environments . DRL has achieved impressive successes in different areas including the game of Go ( Silver et al. , 2017 ) , Atari games ( Mnih et al. , 2015 ) , and continuous control tasks ( Lillicrap et al. , 2015 ) . However , deploying RL algorithms for real-world problems can be very challenging . Compared with supervised learning algorithms ( e.g. , classification or regression ) , most reinforcement learning algorithms need to interact with the environment many times to learn a reliable control policy . This process can be very costly or even dangerous for some real-world applications , e.g. , safety-critical applications . The current success of deep RL algorithms heavily depends on a large number of interactions with the environment . Thus the practical application of reinforcement learning algorithms in the real world is critically limited by its poor data efficiency and its inflexibility of learning in an offline fashion . Batch reinforcement learning ( also known as offline reinforcement learning ) algorithms have been developed to solve this issue . Batch RL aims to learn a control policy from a previously collected dataset without further interactions with the environment . Batch reinforcement learning has attracted a considerable amount of attention due to its potential in dealing with real-world problems . There have been many efforts in developing batch RL methods : early approaches include fitted Q iteration method ( Ernst et al. , 2005 ) which uses a tree-based model to estimate the state-action value function ( Q function ) in an iterative way , the neural fitted Q method ( Riedmiller , 2005 ) , which leverages a multilayer perceptron ( MLP ) to approximate the Q function . Since these earlier efforts , a number of batch reinforcement learning algorithms have been developed that further improve the learning performance . These approaches can be generally categorized as Q function based methods ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) and imitation learning based methods ( Wang et al. , 2018 ; Peng et al. , 2019 ) . Examples of Q function based methods include ( Fujimoto et al. , 2019 ) , in which the authors constrain updates of Q values to deal with distribution drift , and ( Kumar et al. , 2020 ) , where an additional penalty term was added to constrain the update for the Q function . Best Action Imitation Learning ( BAIL ) ( Chen et al. , 2019 ) , on the other hand , leveraging the so-called upper envelope , directly select a sub-batch of the dataset and then execute imitation learning on the selected data to learn the control policy . Even with these advances , most existing batch RL algorithms assume that we have a large number of data points in the batch . In the real world , this may be an unrealistic assumption . For example , when learning energy management control policies , we may only have a very limited amount of collected data for newly built houses and buildings . It is hard for most current batch RL algorithms to learn a reliable policy with a limited amount of data points . In this work , we use transfer learning ( Torrey & Shavlik , 2010 ) to address this issue . Transfer learning aims to use the knowledge from source domains ( domains for which we have a large amount of data ) to improve the learning performance in the target domain ( the domain we are interested in , but for which we only have a limited amount of data ) . Depending on the manner in which knowledge is reused from the source domains , there are three main categories of transfer learning : sample transfer , model transfer , and representation transfer . In this work , we use sample transfer , i.e. , we transfer some related data points from the source tasks to improve the learning performance in the target control task . The proposed algorithm is referred to as BAIL+ as it is a direct extension of the BAIL algorithm proposed by ( Chen et al. , 2019 ) . Multitask learning ( MTL ) ( Caruana , 1997 ) aims to learn a set of tasks jointly instead of learning them separately to achieve a better overall learning performance . In general , multitask learning can help to improve the data efficiency via reusing the shared representations and training on data from multiple sources . Multitask learning has shown its effectiveness on different applications such as image classification ( Li et al. , 2015 ; 2018 ) , nature language processing ( Collobert & Weston , 2008 ; Liu et al. , 2019 ) , and speech recognition ( Siohan & Rybach , 2015 ) . It has shown to be useful in deep reinforcement learning field in recent years ( Li et al. , 2019 ) . The main objective is to learn one ( or more ) policy that can perform well on multiple tasks . In ( Rusu et al. , 2015 ) , the authors proposed to learn a multitask policy from a set of DQN experts that are pre-trained on a set of tasks separately . Lately , ( Hessel et al. , 2019 ) investigated how to balance the learning over multiple tasks with one deep neural network based policy network . The benefit of representation sharing is also investigated in ( D ’ Eramo et al. , 2019 ) for learning deep RL based control agent . Most of existing multitask RL works are based on online RL . There are some preliminary investigations on batch RL . As shown in ( Li et al. , 2019 ) , the authors proposed to learn distill pretarined BCQ models into one model . However , despite a few attempts , how to further improve the task-level generalization for batch RL is still an open question . In this work , different from previous works , we propose to first improve the learning performance on single tasks and then utilize the policy distillation to combine the learned policies into one single policy . Batch RL algorithms are typically designed to deal with single task scenario ( a single batch setting ) . In the real world , it is more common to have batches collected from a set of tasks that have similar Markov Decision Process ( MDP ) settings . For example , we may have collected datasets from a set of houses/buildings in the same area . Thus , it will very helpful if one general policy that can be learned from different bacthes that perform well on these different tasks even including unseen tasks without further adaption . In ( Li et al. , 2019 ) , the authors implemented preliminary investigations on multitask batch reinforcement learning . In our work , to improve the task-level generalization of the policy learned with batch RL , we further extend BAIL+ for multi-batch settings via policy distillation . The resulting algorithm is referred to as MBAIL . Specifically , we aims to learn a general policy without the need to inferring the task identity which can make the batch RL more applicable in real world . The remainder of this paper is organized as follows . The preliminaries of this work including batch reinforcement learning , multi-batch reinforcement learning , and BAIL algorithm are presented in Section II . The details for BAIL+ and MBAIL are presented in Section III . The effectiveness of the proposed methods are showcased in Section IV . Finally , conclusions and future work are presented in Section V . 2 PRELIMINARIES . This section reviews the main technical background and notation used in this paper including reinforcement learning , batch reinforcement learning , BAIL , and multi-batch reinforcement learning . Reinforcement Learning Reinforcement learning ( Sutton & Barto , 2018 ) is a learning paradigm in which a learning agent aims to learn an optimal control policy by interacting with the environment . Reinforcement learning has been successfully applied in many areas including transportation ( Haydari & Yilmaz , 2020 ) , smart grids ( Yang et al. , 2020 ) , and recommendation systems ( Zou et al. , 2019 ) . Formally , an RL problem is typically formulated as a Markov Decision Process ( MDP ) , i.e. , a tuple 〈S , A , p , r , µ , γ〉 , where S is the state space , A is the action space , p : S ⊗ A → S is the state transition function , r : S ⊗A → R is the reward function , µ is the initial state distribution , and γ is the discount factor . The solution to an RL problem ( control policy ) is a function : π : S → A . To obtain this solution , the agent needs to achieve the maximum expected cumulative reward , that is , the so-called value function . Given a state s ∈ S and a policy π , the state value function ( a.k.a. , value function ) under policy π is defined by V π ( s ) = Eπ ( R1 + γR2 + · · · γn−1Rn ) , where Ri denotes the reward obtained at time step i. Batch Reinforcement Learning In batch reinforcement learning ( batch RL ) , the goal is to learn a high performance control policy using an offline dataset without further interactions with the environment . The dataset consists of N data points B = { ( st , at , rt , s ′ t ) |t = 1 , .. , N } . In general , we have no prior requests on the policy used to collect the batch B . B can be obtained while training an RL policy in a episodic fashion or running some other control policy ( e.g. , rule-based methods ) in the same way . The distribution of states and actions in the dataset can be far from those induced by the current policy under consideration . This complicates computations related to evaluating and optimizing behaviors , and this has been the primary consideration of several Batch RL solution methods . To counter this extrapolation problem , ( Fujimoto et al. , 2019 ) proposed an algorithm to learn policies with soft constrain to lie near the batch , which alleviate the extrapolation problem . BAIL : Best Action Imitation Learning Best Action Imitation Learning ( BAIL ) ( Chen et al. , 2019 ) is a simple and computationally efficient imitation learning based batch RL algorithm . The core concept of BAIL is very simple : finding actions that can achieve high return for each state s and then learning a control policy based on these selection state-action pairs . To be more specific , for a particular state-action pair ( s , a ) , let G ( s , a ) denote the return starting in state s and action a , under the policy π. Denote the optimal value function by V ∗ ( s ) . Then if the action a∗ satisfies G ( s , a∗ ) = V ∗ ( s ) , a∗ is an optimal action for state s. The problem now becomes how to obtain V ∗ in a batch setting . Since there is no further interaction with the environment , it is impossible to find V ∗ . Therefore we seek to eliminate as many useless state-action pairs in the batch as possible , to avoid the algorithm inferring bad actions . To do this , we estimate a supremum of the optimal value function V ∗ , which is referred to as the upper envelope . Given φ = ( w , b ) , a neural network parameterized Vφ : S → R , a regularization weight λ and a dataset D of size m , where Di = ( si , Gi ) and Gi is the accumulated return of the state si computed within the given batch , then the upper envelope function V ∗ : S → R is estimated by minimizing the following loss function : min φ m∑ i=1 [ Vφ ( si ) −Gi ] 2 + λ||w||2 s.t . Vφ ( si ) > Gi where i = 1 , 2 , · · ·m ( 1 ) Once the upper envelope function Vφ is estimated , the best state-action pairs can be selected from the batch data B based on the estimated Vφ . One way of selecting such pair is that for a fixed β > 0 , we choose all ( si , ai ) pairs from the batch data set B such that : Gi > βVφ ( si ) ( 2 ) Typically , one can set β such that p % of the data points are selected , where p is a hyper-parameter . In this work , we follow the same setting as ( Chen et al. , 2019 ) , in which β is set to ensure that approximately 25 % of all the data points are selected for each batch . Algorithm 1 BAIL+ : Best Action Imitation Learning with Multi-source Sample Transfer Input : A target task batch Bt andN source task batches B1 , B2 , · · · , BN and the pre-defined sample selection ratio threshold α̃ 1 : Learn the upper envelope function Vt and the reward function r̂t for batch Bt . 2 : for j = 1 , · · · , N do 3 : for d = 1 , · · · , M do 4 : Denote the current state action pair by ( sjd , a j d ) . 5 : Following equation 5 , estimate return of sample ( sjd , a j d ) , denote by Ĝ j d. 6 : Compute the sample selection ratio α ( sjd , a j d ) via equation 4 . 7 : if α ( sjd , a j d ) > α̃ then 8 : Append ( sjd , a j d ) to dataset Bt 9 : end if 10 : end for 11 : Learn the final policy πt on Bt via imitation learning 12 : end for Output : the final policy of the target task πt Multi-Batch Reinforcement Learning To make batch RL more suitable for real-world applications , it is desirable that the learned control policy performs well in multiple situations . In this work , we aim to learn one RL agent from a set of batches sampled from a set of tasks { T1 , · · · , TN } . Then the multi-task ( multi-batch ) batch reinforcement learning can be formulated as : arg max θ J ( θ ) = ETi∼p ( T ) [ JTi ( πθ ) ] ( 3 ) where JTi ( πθ ) is referred to as the performance of control policy πθ on task i . Here p ( T ) defines the task distribution and for each task i , we have a corresponding dataset consisting of K tuples B = { ( sti , ati , rti , st ′ i ) |t = 1 , .. , K } .

Summary: This paper studies the problem of multi-task offline RL and propose 2 methods, BAIL+ and MBAIL, each is an algorithmic variant based on the BAIL algorithm. The overall goal is to improve performance on multiple tasks. This setting concerns with a number of tasks coming from a MDP distribution where transition function is the same and reward function is different.  The authors propose a 2-stage method: first use the BAIL+ algorithm to boost the performance of a particular task with data from other tasks. Then, distillation is used to distill policies from these tasks into one policy.