Adaptive Q-learning for Interaction-Limited Reinforcement Learning

1 INTRODUCTION . Conventional online reinforcement learning ( RL ) methods ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) usually learn from experiences generated by interactions with the online environment . They are impractical in some real-world applications , e.g. , dialog ( Jaques et al. , 2019 ) and education ( Mandel et al. , 2014 ) , where interactions are costly . Recently , offline RL ( Levine et al. , 2020 ) has aroused much attention . It targets the above challenge by making the agent learn from an offline dataset collected by other policies in a purely data-driven manner . The difference between online RL and offline RL is shown in Figure 1 . Existing offline RL studies try to target the distribution mismatch or out-of-distribution actions issue by employing a pessimistic update scheme ( Kumar et al. , 2019 ; 2020 ) or in combination with imitation learning ( Fujimoto et al. , 2019 ) . However , when the dataset is fixed , offline RL can not learn the optimal policy ( Kidambi et al. , 2020 ) , and even worse , when the dataset ’ s quality is poor , offline RL usually gains a relatively bad performance ( Kumar et al. , 2020 ; Fu et al. , 2020 ; Levine et al. , 2020 ) . On the other hand , it is challenging to evaluate the learned policy when learning totally from the offline dataset . Even though some research topics , e.g. , off-policy evaluation ( Dann et al. , 2014 ) , study how to evaluate the learned policy without the interaction with the environment , it is still not ideal for the practical purpose . Some recent works try to address the above issues by employing an offline-online setting . Such methods ( Lee et al. , 2021 ; Nair et al. , 2020 ) focus on pre-training a policy using the offline dataset and fine-tuning the policy through further online interactions . Even though their methods alleviate the above issues to some extent , their main bottleneck is that they do not consider the different characteristics of offline and online data . For instance , pre-existing offline data can prevent agents from converging prematurely due to the potential diverse offline dataset , while online data can improve stability and accelerate convergence . Generally , these different data are mixed and used by a pessimistic strategy to update the policy in their methods , which may be problematic since using a pessimistic strategy for online data may harm the policy performance ( Nair et al. , 2020 ) . Moreover , with sufficiently large and diverse offline data , a high-performing policy can be learned just using a pure online RL algorithm ( Agarwal et al. , 2020 ) . And the online near-on-policy data also play a key role in improving the RL algorithm ’ s stability ( Fujimoto et al. , 2019 ) . Hence , we should take full advantage of both offline and online dataset . To tackle the above problems , in this paper , we emphasize that online and offline RL should be coupled organically . First , a separate updating strategy should be employed for online and offline data , respectively , considering their different characteristics . To do so , we present a framework called adaptive Q-learning that integrates the advantage of offline learning and online learning effectively . When learning from the offline dataset , we conduct a pessimistic update strategy . In contrast , we use a greedy or non-pessimistic update strategy when learning from the online dataset . Second , we design a novel replay buffer to distinguish the offline from online datasets in a simple way . By utilizing such a novel framework and buffer design , the agent can achieve an expert policy using limited online interaction steps regardless of the quality of the offline dataset . In the experiments , our proposed framework can achieve better performance by using only one fifth number of interactions compared with the previous method ( Nair et al. , 2020 ) . Our contributions can be summarized as below : • We propose a unified framework called Adaptive Q-learning that can effectively benefit from both the offline dataset and limited number of online interaction data . • Based on the general framework , we initialize a practical algorithm , called Greedyconservative Q-ensemble learning ( GCQL ) that builds on top of State-of-the-Art offline RL and online RL method . • We empirically verify the effectiveness of our method by comprehensive experiments on the popular continuous control tasks MuJoCo ( Todorov et al. , 2012 ) with offline dataset coming from D4RL ( Fu et al. , 2020 ) . 2 RELATED WORK . Online RL In general , online RL algorithms can be divided into two categories , i.e. , on-policy and off-policy algorithms . On-policy methods ( Schulman et al. , 2015 ; 2017 ) update the policy using data collected by its current behavior policy . As ignoring the logged data collected by its history behaviour policies , they usually have a lower sample efficiency than the off-policy RL . On the other hand , off-policy methods ( Fujimoto et al. , 2018 ; Chen et al. , 2021 ) enable the policy to learn from experience collected by history behavior polices , however , they can not learn well from history trajectories collected by other agents ’ behavior policies ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) . Consequently , the need for huge online interaction makes online RL impractical for some real-world applications , such as dialog agents ( Jaques et al. , 2019 ) or education system ( Mandel et al. , 2014 ) . Offline RL Offline RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset . As the value estimation error can not be corrected using online interactions here , these methods tend to utilize a pessimistic updating strategy to relieve the distribution mismatch problem ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . To implement such a strategy , model-free offline RL methods generally employ value or policy penalties to constrain the updated policy close to the data collecting policy ( Wu et al. , 2019 ; Kumar et al. , 2020 ; Fujimoto et al. , 2019 ; He & Hou , 2020 ) . And model-based methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them ( Kidambi et al. , 2020 ; Yu et al. , 2020 ) . Those offline RL methods can not guarantee a good performance , especially when the data quality is poor ( Kumar et al. , 2020 ) . Besides , policy evaluation when the online environment is unavailable is also challenging . Even though off-policy evaluation ( OPE ) methods ( Dann et al. , 2014 ) present alternative solutions , they are still far from perfect . Above issues of online and offline RL motivate us to investigate the offline-online setting . Offline-online RL Lee et al . ( 2021 ) and Nair et al . ( 2020 ) focus on the mixed setting where the agent is first learned from the offline dataset , and then trained online . Nair et al . ( 2020 ) propose an advantage-weighted actor-critic ( AWAC ) method that restricts the policy to select actions close to those in the offline data by an implicit constraint . When online interactions are available , such conservative constraint may have adverse effects on the performance . Lee et al . ( 2021 ) employ a balanced replay scheme to address the distribution shift issue . It uses the offline data by only selecting near-on-policy samples . Unlike these two works , our method utilizes all online and offline data , and explicitly considers the difference between them by adaptively applying non-conservative or conservative updating schemes , respectively . Matsushima et al . ( 2021 ) focuses on optimizing deployment efficiency , i.e. , the number of distinct data-collection policies used during learning , by employing a behavior-regularized policy updating strategy . Although in terms of deployment efficiency , their work is between online and offline RL , it ignores existing offline dataset , and dose not focusing on improving sample efficiency , while both are addressed in our paper . Some works ( Zhu et al. , 2019 ; Vecerik et al. , 2017 ; Rajeswaran et al. , 2018 ; Kim et al. , 2013 ) can also learn from online interactions and offline data . However , they need expert demonstrations instead of any dataset , and this limits their applicability . 3 PRELIMINARIES . In RL , the interaction between the agent and environment is usually modelled using Markov decision process ( MDP ) ( S , A , pM , r , γ ) , with state space S ( state s ∈ S ) , action space A ( action a ∈ A ) . At each discrete time step , the agent takes an action a based on the current state s , and the state changes into s′ according to the transition dynamics pM ( s′ | s , a ) , and the agent receives a reward r ( s , a , s′ ) ∈ R. The agent ’ s objective is to maximize the return , which is defined as Rt = ∑∞ i=t+1 γ ir ( si , ai , si+1 ) , where t is the time step , and γ ∈ [ 0 , 1 ) is the discounted factor . The mapping from s to a is denoted by the stochastic policy π : a ∼ π ( ·|s ) . Policy can be stochastic or deterministic , and we use the stochastic from in this paper for generality . Each policy π have a corresponding action-value function Qπ ( s , a ) = Eπ [ Rt | s , a ] , which is the expected return following the policy after taking action a in state s. The policy π ’ s action-value function can be updated by the Bellman operator T π : T πQ ( s , a ) = Es′ [ r + γQ ( s′ , π ( s′ ) ) ] ( 1 ) Q-learning ( Sutton & Barto , 2011 ) directly learns the optimal action-value function Q∗ ( s , a ) = maxπ Q π ( s , a ) , and such Q-function can be modelled using neural networks ( Mnih et al. , 2015 ) . In principle , off-policy methods , such as Q-learning , can utilize experiences collected by any policies , and thus they usually maintain a replay buffer B to store and repeatedly learn from experiences collected by behavior policies ( Agarwal et al. , 2020 ) . Such capability also enables off-policy methods to be used in the offline setting , by storing offline data into the buffer B , and not updating the buffer during learning since no further interactions are available here ( Levine et al. , 2020 ) . But this simple adjusting can not guarantee the agent to have a reasonable performance , especially when the dataset is not diverse ( Kumar et al. , 2020 ; Agarwal et al. , 2020 ; Fujimoto et al. , 2019 ) , and this is also the problem tackled in most offline RL works . In this paper , we focus on the offline-online setting , where the agent is first learned from the offline dataset , and then trained via online interactions . And without additional remarks , online RL methods refer to off-policy algorithms in the rest of this paper . We only use off-policy methods because they can make more use of offline data than on-policy ones for gaining high sample efficiency , and on-policy methods are not compatible with our proposed framework introduced next .

Summary: The authors propose a mixed offline-online RL approach for which they design an algorithm. They propose to maintain 2 separate replay buffers, one for online data and one for offline data, to allow them to sample either an online or offline batch of data when doing an update step, and tailor the loss function based on the batch's provenance. In addition, the authors propose using a CQL variant which uses an ensemble of action-value function as done by REDQ to learn in this setting. They conclude by showing some empirical results on the D4RL Mujoco benchmark domains.