Adaptive Q-learning for Interaction-Limited Reinforcement Learning

1 INTRODUCTION . Conventional online reinforcement learning ( RL ) methods ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) usually learn from experiences generated by interactions with the online environment . They are impractical in some real-world applications , e.g. , dialog ( Jaques et al. , 2019 ) and education ( Mandel et al. , 2014 ) , where interactions are costly . Recently , offline RL ( Levine et al. , 2020 ) has aroused much attention . It targets the above challenge by making the agent learn from an offline dataset collected by other policies in a purely data-driven manner . The difference between online RL and offline RL is shown in Figure 1 . Existing offline RL studies try to target the distribution mismatch or out-of-distribution actions issue by employing a pessimistic update scheme ( Kumar et al. , 2019 ; 2020 ) or in combination with imitation learning ( Fujimoto et al. , 2019 ) . However , when the dataset is fixed , offline RL can not learn the optimal policy ( Kidambi et al. , 2020 ) , and even worse , when the dataset ’ s quality is poor , offline RL usually gains a relatively bad performance ( Kumar et al. , 2020 ; Fu et al. , 2020 ; Levine et al. , 2020 ) . On the other hand , it is challenging to evaluate the learned policy when learning totally from the offline dataset . Even though some research topics , e.g. , off-policy evaluation ( Dann et al. , 2014 ) , study how to evaluate the learned policy without the interaction with the environment , it is still not ideal for the practical purpose . Some recent works try to address the above issues by employing an offline-online setting . Such methods ( Lee et al. , 2021 ; Nair et al. , 2020 ) focus on pre-training a policy using the offline dataset and fine-tuning the policy through further online interactions . Even though their methods alleviate the above issues to some extent , their main bottleneck is that they do not consider the different characteristics of offline and online data . For instance , pre-existing offline data can prevent agents from converging prematurely due to the potential diverse offline dataset , while online data can improve stability and accelerate convergence . Generally , these different data are mixed and used by a pessimistic strategy to update the policy in their methods , which may be problematic since using a pessimistic strategy for online data may harm the policy performance ( Nair et al. , 2020 ) . Moreover , with sufficiently large and diverse offline data , a high-performing policy can be learned just using a pure online RL algorithm ( Agarwal et al. , 2020 ) . And the online near-on-policy data also play a key role in improving the RL algorithm ’ s stability ( Fujimoto et al. , 2019 ) . Hence , we should take full advantage of both offline and online dataset . To tackle the above problems , in this paper , we emphasize that online and offline RL should be coupled organically . First , a separate updating strategy should be employed for online and offline data , respectively , considering their different characteristics . To do so , we present a framework called adaptive Q-learning that integrates the advantage of offline learning and online learning effectively . When learning from the offline dataset , we conduct a pessimistic update strategy . In contrast , we use a greedy or non-pessimistic update strategy when learning from the online dataset . Second , we design a novel replay buffer to distinguish the offline from online datasets in a simple way . By utilizing such a novel framework and buffer design , the agent can achieve an expert policy using limited online interaction steps regardless of the quality of the offline dataset . In the experiments , our proposed framework can achieve better performance by using only one fifth number of interactions compared with the previous method ( Nair et al. , 2020 ) . Our contributions can be summarized as below : • We propose a unified framework called Adaptive Q-learning that can effectively benefit from both the offline dataset and limited number of online interaction data . • Based on the general framework , we initialize a practical algorithm , called Greedyconservative Q-ensemble learning ( GCQL ) that builds on top of State-of-the-Art offline RL and online RL method . • We empirically verify the effectiveness of our method by comprehensive experiments on the popular continuous control tasks MuJoCo ( Todorov et al. , 2012 ) with offline dataset coming from D4RL ( Fu et al. , 2020 ) . 2 RELATED WORK . Online RL In general , online RL algorithms can be divided into two categories , i.e. , on-policy and off-policy algorithms . On-policy methods ( Schulman et al. , 2015 ; 2017 ) update the policy using data collected by its current behavior policy . As ignoring the logged data collected by its history behaviour policies , they usually have a lower sample efficiency than the off-policy RL . On the other hand , off-policy methods ( Fujimoto et al. , 2018 ; Chen et al. , 2021 ) enable the policy to learn from experience collected by history behavior polices , however , they can not learn well from history trajectories collected by other agents ’ behavior policies ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) . Consequently , the need for huge online interaction makes online RL impractical for some real-world applications , such as dialog agents ( Jaques et al. , 2019 ) or education system ( Mandel et al. , 2014 ) . Offline RL Offline RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset . As the value estimation error can not be corrected using online interactions here , these methods tend to utilize a pessimistic updating strategy to relieve the distribution mismatch problem ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . To implement such a strategy , model-free offline RL methods generally employ value or policy penalties to constrain the updated policy close to the data collecting policy ( Wu et al. , 2019 ; Kumar et al. , 2020 ; Fujimoto et al. , 2019 ; He & Hou , 2020 ) . And model-based methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them ( Kidambi et al. , 2020 ; Yu et al. , 2020 ) . Those offline RL methods can not guarantee a good performance , especially when the data quality is poor ( Kumar et al. , 2020 ) . Besides , policy evaluation when the online environment is unavailable is also challenging . Even though off-policy evaluation ( OPE ) methods ( Dann et al. , 2014 ) present alternative solutions , they are still far from perfect . Above issues of online and offline RL motivate us to investigate the offline-online setting . Offline-online RL Lee et al . ( 2021 ) and Nair et al . ( 2020 ) focus on the mixed setting where the agent is first learned from the offline dataset , and then trained online . Nair et al . ( 2020 ) propose an advantage-weighted actor-critic ( AWAC ) method that restricts the policy to select actions close to those in the offline data by an implicit constraint . When online interactions are available , such conservative constraint may have adverse effects on the performance . Lee et al . ( 2021 ) employ a balanced replay scheme to address the distribution shift issue . It uses the offline data by only selecting near-on-policy samples . Unlike these two works , our method utilizes all online and offline data , and explicitly considers the difference between them by adaptively applying non-conservative or conservative updating schemes , respectively . Matsushima et al . ( 2021 ) focuses on optimizing deployment efficiency , i.e. , the number of distinct data-collection policies used during learning , by employing a behavior-regularized policy updating strategy . Although in terms of deployment efficiency , their work is between online and offline RL , it ignores existing offline dataset , and dose not focusing on improving sample efficiency , while both are addressed in our paper . Some works ( Zhu et al. , 2019 ; Vecerik et al. , 2017 ; Rajeswaran et al. , 2018 ; Kim et al. , 2013 ) can also learn from online interactions and offline data . However , they need expert demonstrations instead of any dataset , and this limits their applicability . 3 PRELIMINARIES . In RL , the interaction between the agent and environment is usually modelled using Markov decision process ( MDP ) ( S , A , pM , r , γ ) , with state space S ( state s ∈ S ) , action space A ( action a ∈ A ) . At each discrete time step , the agent takes an action a based on the current state s , and the state changes into s′ according to the transition dynamics pM ( s′ | s , a ) , and the agent receives a reward r ( s , a , s′ ) ∈ R. The agent ’ s objective is to maximize the return , which is defined as Rt = ∑∞ i=t+1 γ ir ( si , ai , si+1 ) , where t is the time step , and γ ∈ [ 0 , 1 ) is the discounted factor . The mapping from s to a is denoted by the stochastic policy π : a ∼ π ( ·|s ) . Policy can be stochastic or deterministic , and we use the stochastic from in this paper for generality . Each policy π have a corresponding action-value function Qπ ( s , a ) = Eπ [ Rt | s , a ] , which is the expected return following the policy after taking action a in state s. The policy π ’ s action-value function can be updated by the Bellman operator T π : T πQ ( s , a ) = Es′ [ r + γQ ( s′ , π ( s′ ) ) ] ( 1 ) Q-learning ( Sutton & Barto , 2011 ) directly learns the optimal action-value function Q∗ ( s , a ) = maxπ Q π ( s , a ) , and such Q-function can be modelled using neural networks ( Mnih et al. , 2015 ) . In principle , off-policy methods , such as Q-learning , can utilize experiences collected by any policies , and thus they usually maintain a replay buffer B to store and repeatedly learn from experiences collected by behavior policies ( Agarwal et al. , 2020 ) . Such capability also enables off-policy methods to be used in the offline setting , by storing offline data into the buffer B , and not updating the buffer during learning since no further interactions are available here ( Levine et al. , 2020 ) . But this simple adjusting can not guarantee the agent to have a reasonable performance , especially when the dataset is not diverse ( Kumar et al. , 2020 ; Agarwal et al. , 2020 ; Fujimoto et al. , 2019 ) , and this is also the problem tackled in most offline RL works . In this paper , we focus on the offline-online setting , where the agent is first learned from the offline dataset , and then trained via online interactions . And without additional remarks , online RL methods refer to off-policy algorithms in the rest of this paper . We only use off-policy methods because they can make more use of offline data than on-policy ones for gaining high sample efficiency , and on-policy methods are not compatible with our proposed framework introduced next .

Summary: The paper proposes GCQL, a new RL algorithm that is trained on a mixture of offline data and online interactions. The novel component of the algorithm is a reweighting that balances acting pessimistically on offline data and greedily on online interactions. The authors propose a mixture replay buffer that consists of both offline and online samples. For online samples, the policy is trained as in the REDQ algorithm; meanwhile, for offline samples, the policy is additionally trained on a CQL-like value penalty. Finally, the authors show that GCQL outperforms existing SOTA offline RL algorithms that simply fine-tune on online data.