Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

1 INTRODUCTION . Offline or batch reinforcement learning ( RL ) deals with the training of RL agents from fixed datasets generated by possibly unknown behavior policies , without any interactions with the environment . This is important in problems like robotics , autonomous driving , and healthcare where data collection can be expensive or dangerous . Offline RL has been challenging for model-free RL methods due to extrapolation error where the Q networks predict unrealistic values upon evaluations on outof-distribution state-action pairs ( Fujimoto et al. , 2019 ) . Recent methods overcome this issue by constraining the policy to stay close to the behavior policy that generated the offline data distribution ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ; Kostrikov et al. , 2021 ; Fujimoto & Gu , 2021 ) , to demonstrate even better performance than the behavior policy on several simulated and real-world tasks ( Siegel et al. , 2020 ; Singh et al. , 2020 ; Nair et al. , 2020 ) . However , the performance of pre-trained policies will be limited by the quality of the offline dataset and it is often necessary or desirable to fine-tune them by interacting with the environment . Also , offline-to-online learning reduces the risks in online interaction as the offline pre-training results in reasonable policies that could be tested before deployment . In practice , offline RL methods often fail during online fine-tuning by interacting with the environment . This offline-to-online RL setting is challenging due to : ( i ) the sudden distribution shift from offline data to online data . This could lead to severe bootstrapping errors which completely distorts the pre-trained policy leading to a sudden performance drop from the very beginning of online fine-tuning , and ( ii ) constraints enforced by offline RL methods on the policy to stay close to the behavior policy . While these constraints help in dealing with the sudden distribution shift they significantly slow down online fine-tuning from newly collected samples . We propose to adaptively weigh the offline RL constraints such as behavior cloning loss during online fine-tuning . This could prevent sudden performance collapses due to the distribution shift while also enabling sample-efficient learning from the newly collected samples . We propose to perform this adaptive weighing according to the agent ’ s performance and the training stability . We start with TD3+BC , a simple offline RL algorithm recently proposed by Fujimoto & Gu ( 2021 ) which combines TD3 ( Fujimoto et al. , 2018 ) with a simple behavior cloning loss , weighted by an α hyperparameter . We adaptively weigh this α hyperparameter using a control mechanism similar to the proportional–derivative ( PD ) controller . The α value is decided based on two components : the difference between the current return and the target return ( proportional term ) as well as the change of return between current episode and the last episode ( derivative term ) . We demonstrate that these simple modifications lead to stable online fine-tuning after offline pre-training on datasets of different quality . We also use a randomized ensemble of Q functions ( Chen et al. , 2021 ) to further improve the sample-efficiency . We attain state-of-the-art online fine-tuning performance on locomotion tasks from the popular D4RL benchmark . 2 RELATED WORK . Offline RL . Offline RL aims to learn a policy from pre-collected fixed datasets without interacting with the environment ( Lange et al. , 2012 ; Agarwal et al. , 2020 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Nachum et al. , 2019 ; Siegel et al. , 2020 ; Levine et al. , 2020 ; Peng et al. , 2019 ) . Off-policy RL algorithms allow for reuse of off-policy data ( Konda & Tsitsiklis , 2000 ; Degris et al. , 2012 ; Haarnoja et al. , 2018 ; Silver et al. , 2014 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Mnih et al. , 2015 ) but they typically fail when trained offline on a fixed dataset , even if it ’ s collected by a policy trained using the same algorithm ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . In actor-critic methods , this is due to extrapolation error of the critic network on out-of-distribution state-action pairs Levine et al . ( 2020 ) . Offline RL methods deal with this by constraining the policy to stay close to the behavioral policy that collected the offline dataset . BRAC ( Wu et al. , 2019 ) achieves this by minimizing the Kullback-Leibler divergence between the behavior policy and the learned policy . BEAR ( Kumar et al. , 2019 ) minimizes the MMD distance between the two policies . TD3+BC ( Fujimoto & Gu , 2021 ) proposes a simple yet efficient offline RL algorithm by adding an additional behavior cloning loss to the actor update . Another class of offline RL methods learns conservative Q functions , which prevents the policy network from exploiting out-of-distribution actions and forces them to stay close to the behavior policy . CQL ( Kumar et al. , 2020 ) changes the critic objective to also minimize the Q function on unseen actions . Fisher-BRC ( Kostrikov et al. , 2021 ) achieves conservative Q learning by constraining the gradient of the Q function on unseen data . Model-based offline RL methods ( Yu et al. , 2020 ; Kidambi et al. , 2020 ) train policies based on the data generated by ensembles of dynamics models learned from offline data , while constraining the policy to stay within samples where the dynamics model is certain . In this paper , we focus on offline-to-online RL with the goal of stable and sample-efficient online fine-tuning from policies pre-trained on offline datasets of different quality . Offline pre-training in RL . Pre-training has been vastly investigated in the machine learning community from computer vision ( Sharif Razavian et al. , 2014 ; Donahue et al. , 2014 ; Yosinski et al. , 2014 ) to natural language processing ( Devlin et al. , 2018 ; Turian et al. , 2010 ) . Offline pre-training in RL could enable deployment of RL methods in domains where data collection can be expensive or dangerous . ( Silver et al. , 2016 ; Gupta et al. , 2019 ; Rajeswaran et al. , 2017 ) pre-train the policy network with imitation learning to speed up RL . QT-opt ( Kalashnikov et al. , 2018 ) studies visionbased object manipulation using a diverse and large dataset collected by seven robots over several months and fine-tune the policy with 27K samples of online data . However , these methods pre-train using diverse , large , or expert datasets and it is also important to investigate the possibility of pretraining from offline datasets of different quality . Yang & Nachum ( 2021 ) ; Ajay et al . ( 2020 ) use offline pre-training to accelerate downstream tasks . AWAC ( Nair et al. , 2020 ) and Balanced Replay Lee et al . ( 2021 ) are recent works that also focus on offline-to-online RL from datasets of different quality . AWAC updates the policy network such that it is constrained during offline training while not too conservative during fine-tuning . Balanced Replay trains an additional neural network to prioritize samples in order to effectively use new data as well as near-on-policy samples in the offline dataset . We compare with AWAC and Balanced Replay to attain state-of-the-art offline-to-online RL performance on the popular D4RL benchmark . Ensembles in RL . Ensemble methods are widely used for better performance in RL ( Faußer & Schwenker , 2015 ; Osband et al. , 2016 ; Chua et al. , 2018 ; Janner et al. , 2019 ) . In model-based RL , PETS ( Chua et al. , 2018 ) and MBPO ( Janner et al. , 2019 ) use probabilistic ensembles to effectively model the dynamics of the environment . In model-free RL , ensembles of Q functions have been shown to improve performance ( Anschel et al. , 2017 ; Lan et al. , 2020 ) . REDQ ( Chen et al. , 2021 ) learns a randomized ensemble of Q functions to achieve similar sample efficiency as model-based methods without learning a dynamic model . We utilize REDQ in this work for improved sampleefficiency during online fine-tuning . Specific to offline RL , REM ( Agarwal et al. , 2020 ) uses random convex combinations of multiple Q-value estimates to calculate the Q targets for effective offline RL on Atari games . MOPO ( Yu et al. , 2020 ) uses probabilistic ensembles from PETS to learn policies from offline data using uncertainty estimates based on model disagreement . MBOP ( Argenson & Dulac-Arnold , 2020 ) uses ensembles of dynamic models , Q functions , and policy networks to get better performance on locomotion tasks . Balanced Replay ( Lee et al. , 2021 ) uses ensembles of pessimistic Q functions to mitigate instability caused by distribution shift in offline-to-online RL . While ensembling of Q functions has been studied by several prior works ( Lan et al. , 2020 ; Chen et al. , 2021 ) , we combine it with behavioral cloning loss for the purpose of robust and sampleefficient offline-to-online RL . Adaptive balancing of multiple objectives in RL . Ball et al . ( 2020 ) train policies using learned dynamics models with the objective of visiting states that most likely lead to subsequent improvement in the dynamics model , using active online learning . They adaptively weigh the maximization of cumulative rewards and minimization of model uncertainty using an online learning mechanism based on exponential weights algorithm . In this paper , we focus on offline-to-online RL using model-free methods and propose to adaptively weigh the maximization of cumulative rewards and a behavioral cloning loss . Exploration of other online learning algorithms such as exponential weights algorithm is a line of future work . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) deals with sequential decision making to maximize cumulative rewards . RL problems are often formalized as Markov decision processes ( MDPs ) . An MDP consists of a set of states S , a set of actions A , a transition dynamics st+1 ∼ p ( ·|st , at ) that represents the probability of transitioning to a state st+1 by taking action at in state st at timestep t , a scalar reward function rt = R ( st , at ) , and a discount factor γ ∈ [ 0 , 1 ] . A policy function π of an RL agent is a mapping from states to actions and defines the behavior of the agent . The value function Vπ ( s ) of a policy π is defined as the expected cumulative rewards from state s : V π ( s ) = E [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] , where the expectation is taken over state transitions st+1 ∼ p ( ·|st , at ) and policy function at ∼ π ( st ) . Similarly , the state-action value function Qπ ( s , a ) is defined as the expected cumulative rewards after taking action a in state s : Qπ ( s , a ) = E [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . The goal of RL is to learn an optimal policy function πθ with parameters θ , that maximizes the expected cumulative rewards : πθ = arg max θ Es∼S [ V πθ ( s ) ] = arg max θ Es∼S [ Qπθ ( s , πθ ( s ) ) ] . We use the TD3 algorithm for reinforcement learning ( Fujimoto et al. , 2018 ) . TD3 is an actor-critic method that alternatingly trains : ( i ) the critic network Qφ to estimate the Qπθ ( s , a ) values of the policy network πθ , and ( ii ) the policy network to produce actions that maximize the Q function : ∇θQφ ( s , πθ ( s ) ) .

Summary: This paper studies the fine-tuning problem from offline to online RL. While the naive approach to fine-tune offline policy suffers from a sudden distributional shift by online samples and too much behavior constraint in offline algorithms. The proposed method leverage (1) the adaptive coefficient tuning in TD3+BC loss, (2) randomized ensembles of Q-functions (proposed by Chen et al. 2021), and (3) down-sampling of offline data. The experiments seem to show better or competitive results to other approaches.