Multi-batch Reinforcement Learning via Sample Transfer and Imitation Learning

1 INTRODUCTION . Reinforcement learning aims to learn an optimal control policy through interactions with the environment ( Sutton & Barto , 2018 ) . Deep reinforcement learning ( Deep RL or DRL ) combines neural networks with reinforcement learning and further enables RL agents to deal with more complex environments . DRL has achieved impressive successes in different areas including the game of Go ( Silver et al. , 2017 ) , Atari games ( Mnih et al. , 2015 ) , and continuous control tasks ( Lillicrap et al. , 2015 ) . However , deploying RL algorithms for real-world problems can be very challenging . Compared with supervised learning algorithms ( e.g. , classification or regression ) , most reinforcement learning algorithms need to interact with the environment many times to learn a reliable control policy . This process can be very costly or even dangerous for some real-world applications , e.g. , safety-critical applications . The current success of deep RL algorithms heavily depends on a large number of interactions with the environment . Thus the practical application of reinforcement learning algorithms in the real world is critically limited by its poor data efficiency and its inflexibility of learning in an offline fashion . Batch reinforcement learning ( also known as offline reinforcement learning ) algorithms have been developed to solve this issue . Batch RL aims to learn a control policy from a previously collected dataset without further interactions with the environment . Batch reinforcement learning has attracted a considerable amount of attention due to its potential in dealing with real-world problems . There have been many efforts in developing batch RL methods : early approaches include fitted Q iteration method ( Ernst et al. , 2005 ) which uses a tree-based model to estimate the state-action value function ( Q function ) in an iterative way , the neural fitted Q method ( Riedmiller , 2005 ) , which leverages a multilayer perceptron ( MLP ) to approximate the Q function . Since these earlier efforts , a number of batch reinforcement learning algorithms have been developed that further improve the learning performance . These approaches can be generally categorized as Q function based methods ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) and imitation learning based methods ( Wang et al. , 2018 ; Peng et al. , 2019 ) . Examples of Q function based methods include ( Fujimoto et al. , 2019 ) , in which the authors constrain updates of Q values to deal with distribution drift , and ( Kumar et al. , 2020 ) , where an additional penalty term was added to constrain the update for the Q function . Best Action Imitation Learning ( BAIL ) ( Chen et al. , 2019 ) , on the other hand , leveraging the so-called upper envelope , directly select a sub-batch of the dataset and then execute imitation learning on the selected data to learn the control policy . Even with these advances , most existing batch RL algorithms assume that we have a large number of data points in the batch . In the real world , this may be an unrealistic assumption . For example , when learning energy management control policies , we may only have a very limited amount of collected data for newly built houses and buildings . It is hard for most current batch RL algorithms to learn a reliable policy with a limited amount of data points . In this work , we use transfer learning ( Torrey & Shavlik , 2010 ) to address this issue . Transfer learning aims to use the knowledge from source domains ( domains for which we have a large amount of data ) to improve the learning performance in the target domain ( the domain we are interested in , but for which we only have a limited amount of data ) . Depending on the manner in which knowledge is reused from the source domains , there are three main categories of transfer learning : sample transfer , model transfer , and representation transfer . In this work , we use sample transfer , i.e. , we transfer some related data points from the source tasks to improve the learning performance in the target control task . The proposed algorithm is referred to as BAIL+ as it is a direct extension of the BAIL algorithm proposed by ( Chen et al. , 2019 ) . Multitask learning ( MTL ) ( Caruana , 1997 ) aims to learn a set of tasks jointly instead of learning them separately to achieve a better overall learning performance . In general , multitask learning can help to improve the data efficiency via reusing the shared representations and training on data from multiple sources . Multitask learning has shown its effectiveness on different applications such as image classification ( Li et al. , 2015 ; 2018 ) , nature language processing ( Collobert & Weston , 2008 ; Liu et al. , 2019 ) , and speech recognition ( Siohan & Rybach , 2015 ) . It has shown to be useful in deep reinforcement learning field in recent years ( Li et al. , 2019 ) . The main objective is to learn one ( or more ) policy that can perform well on multiple tasks . In ( Rusu et al. , 2015 ) , the authors proposed to learn a multitask policy from a set of DQN experts that are pre-trained on a set of tasks separately . Lately , ( Hessel et al. , 2019 ) investigated how to balance the learning over multiple tasks with one deep neural network based policy network . The benefit of representation sharing is also investigated in ( D ’ Eramo et al. , 2019 ) for learning deep RL based control agent . Most of existing multitask RL works are based on online RL . There are some preliminary investigations on batch RL . As shown in ( Li et al. , 2019 ) , the authors proposed to learn distill pretarined BCQ models into one model . However , despite a few attempts , how to further improve the task-level generalization for batch RL is still an open question . In this work , different from previous works , we propose to first improve the learning performance on single tasks and then utilize the policy distillation to combine the learned policies into one single policy . Batch RL algorithms are typically designed to deal with single task scenario ( a single batch setting ) . In the real world , it is more common to have batches collected from a set of tasks that have similar Markov Decision Process ( MDP ) settings . For example , we may have collected datasets from a set of houses/buildings in the same area . Thus , it will very helpful if one general policy that can be learned from different bacthes that perform well on these different tasks even including unseen tasks without further adaption . In ( Li et al. , 2019 ) , the authors implemented preliminary investigations on multitask batch reinforcement learning . In our work , to improve the task-level generalization of the policy learned with batch RL , we further extend BAIL+ for multi-batch settings via policy distillation . The resulting algorithm is referred to as MBAIL . Specifically , we aims to learn a general policy without the need to inferring the task identity which can make the batch RL more applicable in real world . The remainder of this paper is organized as follows . The preliminaries of this work including batch reinforcement learning , multi-batch reinforcement learning , and BAIL algorithm are presented in Section II . The details for BAIL+ and MBAIL are presented in Section III . The effectiveness of the proposed methods are showcased in Section IV . Finally , conclusions and future work are presented in Section V . 2 PRELIMINARIES . This section reviews the main technical background and notation used in this paper including reinforcement learning , batch reinforcement learning , BAIL , and multi-batch reinforcement learning . Reinforcement Learning Reinforcement learning ( Sutton & Barto , 2018 ) is a learning paradigm in which a learning agent aims to learn an optimal control policy by interacting with the environment . Reinforcement learning has been successfully applied in many areas including transportation ( Haydari & Yilmaz , 2020 ) , smart grids ( Yang et al. , 2020 ) , and recommendation systems ( Zou et al. , 2019 ) . Formally , an RL problem is typically formulated as a Markov Decision Process ( MDP ) , i.e. , a tuple 〈S , A , p , r , µ , γ〉 , where S is the state space , A is the action space , p : S ⊗ A → S is the state transition function , r : S ⊗A → R is the reward function , µ is the initial state distribution , and γ is the discount factor . The solution to an RL problem ( control policy ) is a function : π : S → A . To obtain this solution , the agent needs to achieve the maximum expected cumulative reward , that is , the so-called value function . Given a state s ∈ S and a policy π , the state value function ( a.k.a. , value function ) under policy π is defined by V π ( s ) = Eπ ( R1 + γR2 + · · · γn−1Rn ) , where Ri denotes the reward obtained at time step i. Batch Reinforcement Learning In batch reinforcement learning ( batch RL ) , the goal is to learn a high performance control policy using an offline dataset without further interactions with the environment . The dataset consists of N data points B = { ( st , at , rt , s ′ t ) |t = 1 , .. , N } . In general , we have no prior requests on the policy used to collect the batch B . B can be obtained while training an RL policy in a episodic fashion or running some other control policy ( e.g. , rule-based methods ) in the same way . The distribution of states and actions in the dataset can be far from those induced by the current policy under consideration . This complicates computations related to evaluating and optimizing behaviors , and this has been the primary consideration of several Batch RL solution methods . To counter this extrapolation problem , ( Fujimoto et al. , 2019 ) proposed an algorithm to learn policies with soft constrain to lie near the batch , which alleviate the extrapolation problem . BAIL : Best Action Imitation Learning Best Action Imitation Learning ( BAIL ) ( Chen et al. , 2019 ) is a simple and computationally efficient imitation learning based batch RL algorithm . The core concept of BAIL is very simple : finding actions that can achieve high return for each state s and then learning a control policy based on these selection state-action pairs . To be more specific , for a particular state-action pair ( s , a ) , let G ( s , a ) denote the return starting in state s and action a , under the policy π. Denote the optimal value function by V ∗ ( s ) . Then if the action a∗ satisfies G ( s , a∗ ) = V ∗ ( s ) , a∗ is an optimal action for state s. The problem now becomes how to obtain V ∗ in a batch setting . Since there is no further interaction with the environment , it is impossible to find V ∗ . Therefore we seek to eliminate as many useless state-action pairs in the batch as possible , to avoid the algorithm inferring bad actions . To do this , we estimate a supremum of the optimal value function V ∗ , which is referred to as the upper envelope . Given φ = ( w , b ) , a neural network parameterized Vφ : S → R , a regularization weight λ and a dataset D of size m , where Di = ( si , Gi ) and Gi is the accumulated return of the state si computed within the given batch , then the upper envelope function V ∗ : S → R is estimated by minimizing the following loss function : min φ m∑ i=1 [ Vφ ( si ) −Gi ] 2 + λ||w||2 s.t . Vφ ( si ) > Gi where i = 1 , 2 , · · ·m ( 1 ) Once the upper envelope function Vφ is estimated , the best state-action pairs can be selected from the batch data B based on the estimated Vφ . One way of selecting such pair is that for a fixed β > 0 , we choose all ( si , ai ) pairs from the batch data set B such that : Gi > βVφ ( si ) ( 2 ) Typically , one can set β such that p % of the data points are selected , where p is a hyper-parameter . In this work , we follow the same setting as ( Chen et al. , 2019 ) , in which β is set to ensure that approximately 25 % of all the data points are selected for each batch . Algorithm 1 BAIL+ : Best Action Imitation Learning with Multi-source Sample Transfer Input : A target task batch Bt andN source task batches B1 , B2 , · · · , BN and the pre-defined sample selection ratio threshold α̃ 1 : Learn the upper envelope function Vt and the reward function r̂t for batch Bt . 2 : for j = 1 , · · · , N do 3 : for d = 1 , · · · , M do 4 : Denote the current state action pair by ( sjd , a j d ) . 5 : Following equation 5 , estimate return of sample ( sjd , a j d ) , denote by Ĝ j d. 6 : Compute the sample selection ratio α ( sjd , a j d ) via equation 4 . 7 : if α ( sjd , a j d ) > α̃ then 8 : Append ( sjd , a j d ) to dataset Bt 9 : end if 10 : end for 11 : Learn the final policy πt on Bt via imitation learning 12 : end for Output : the final policy of the target task πt Multi-Batch Reinforcement Learning To make batch RL more suitable for real-world applications , it is desirable that the learned control policy performs well in multiple situations . In this work , we aim to learn one RL agent from a set of batches sampled from a set of tasks { T1 , · · · , TN } . Then the multi-task ( multi-batch ) batch reinforcement learning can be formulated as : arg max θ J ( θ ) = ETi∼p ( T ) [ JTi ( πθ ) ] ( 3 ) where JTi ( πθ ) is referred to as the performance of control policy πθ on task i . Here p ( T ) defines the task distribution and for each task i , we have a corresponding dataset consisting of K tuples B = { ( sti , ati , rti , st ′ i ) |t = 1 , .. , K } .

Summary: (Note that the rest of the review will use Offline RL and Batch RL interchangeably)   Building on top of BAIL (Chen et al. 2019), this paper provides two algorithms for the multi-task offline RL setting. The BAIL+ algorithm assumes that we know the identity of the target task and that all tasks share the same transition function. It uses sample transfer from non-target task data to boost its performance on the target task. The MBAIL algorithm builds on top of BAIL+ and MBML (Li et al 2019), where it distills the BAIL+ policies for each individual task into one master policy that performs decently on all tasks. The authors evaluated the proposed methods on the Ant-Dir, Ant-Goal, and HalfCheetahVel environments from Mujoco. The authors compared MBAIL, BAIL+, BAIL, and contextual BCQ. MBAIL consistently performed the best out of the first three, while contextual BCQ performed better on the Ant-Goal env.