Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

1 INTRODUCTION . Offline or batch reinforcement learning ( RL ) deals with the training of RL agents from fixed datasets generated by possibly unknown behavior policies , without any interactions with the environment . This is important in problems like robotics , autonomous driving , and healthcare where data collection can be expensive or dangerous . Offline RL has been challenging for model-free RL methods due to extrapolation error where the Q networks predict unrealistic values upon evaluations on outof-distribution state-action pairs ( Fujimoto et al. , 2019 ) . Recent methods overcome this issue by constraining the policy to stay close to the behavior policy that generated the offline data distribution ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ; Kostrikov et al. , 2021 ; Fujimoto & Gu , 2021 ) , to demonstrate even better performance than the behavior policy on several simulated and real-world tasks ( Siegel et al. , 2020 ; Singh et al. , 2020 ; Nair et al. , 2020 ) . However , the performance of pre-trained policies will be limited by the quality of the offline dataset and it is often necessary or desirable to fine-tune them by interacting with the environment . Also , offline-to-online learning reduces the risks in online interaction as the offline pre-training results in reasonable policies that could be tested before deployment . In practice , offline RL methods often fail during online fine-tuning by interacting with the environment . This offline-to-online RL setting is challenging due to : ( i ) the sudden distribution shift from offline data to online data . This could lead to severe bootstrapping errors which completely distorts the pre-trained policy leading to a sudden performance drop from the very beginning of online fine-tuning , and ( ii ) constraints enforced by offline RL methods on the policy to stay close to the behavior policy . While these constraints help in dealing with the sudden distribution shift they significantly slow down online fine-tuning from newly collected samples . We propose to adaptively weigh the offline RL constraints such as behavior cloning loss during online fine-tuning . This could prevent sudden performance collapses due to the distribution shift while also enabling sample-efficient learning from the newly collected samples . We propose to perform this adaptive weighing according to the agent ’ s performance and the training stability . We start with TD3+BC , a simple offline RL algorithm recently proposed by Fujimoto & Gu ( 2021 ) which combines TD3 ( Fujimoto et al. , 2018 ) with a simple behavior cloning loss , weighted by an α hyperparameter . We adaptively weigh this α hyperparameter using a control mechanism similar to the proportional–derivative ( PD ) controller . The α value is decided based on two components : the difference between the current return and the target return ( proportional term ) as well as the change of return between current episode and the last episode ( derivative term ) . We demonstrate that these simple modifications lead to stable online fine-tuning after offline pre-training on datasets of different quality . We also use a randomized ensemble of Q functions ( Chen et al. , 2021 ) to further improve the sample-efficiency . We attain state-of-the-art online fine-tuning performance on locomotion tasks from the popular D4RL benchmark . 2 RELATED WORK . Offline RL . Offline RL aims to learn a policy from pre-collected fixed datasets without interacting with the environment ( Lange et al. , 2012 ; Agarwal et al. , 2020 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Nachum et al. , 2019 ; Siegel et al. , 2020 ; Levine et al. , 2020 ; Peng et al. , 2019 ) . Off-policy RL algorithms allow for reuse of off-policy data ( Konda & Tsitsiklis , 2000 ; Degris et al. , 2012 ; Haarnoja et al. , 2018 ; Silver et al. , 2014 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Mnih et al. , 2015 ) but they typically fail when trained offline on a fixed dataset , even if it ’ s collected by a policy trained using the same algorithm ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . In actor-critic methods , this is due to extrapolation error of the critic network on out-of-distribution state-action pairs Levine et al . ( 2020 ) . Offline RL methods deal with this by constraining the policy to stay close to the behavioral policy that collected the offline dataset . BRAC ( Wu et al. , 2019 ) achieves this by minimizing the Kullback-Leibler divergence between the behavior policy and the learned policy . BEAR ( Kumar et al. , 2019 ) minimizes the MMD distance between the two policies . TD3+BC ( Fujimoto & Gu , 2021 ) proposes a simple yet efficient offline RL algorithm by adding an additional behavior cloning loss to the actor update . Another class of offline RL methods learns conservative Q functions , which prevents the policy network from exploiting out-of-distribution actions and forces them to stay close to the behavior policy . CQL ( Kumar et al. , 2020 ) changes the critic objective to also minimize the Q function on unseen actions . Fisher-BRC ( Kostrikov et al. , 2021 ) achieves conservative Q learning by constraining the gradient of the Q function on unseen data . Model-based offline RL methods ( Yu et al. , 2020 ; Kidambi et al. , 2020 ) train policies based on the data generated by ensembles of dynamics models learned from offline data , while constraining the policy to stay within samples where the dynamics model is certain . In this paper , we focus on offline-to-online RL with the goal of stable and sample-efficient online fine-tuning from policies pre-trained on offline datasets of different quality . Offline pre-training in RL . Pre-training has been vastly investigated in the machine learning community from computer vision ( Sharif Razavian et al. , 2014 ; Donahue et al. , 2014 ; Yosinski et al. , 2014 ) to natural language processing ( Devlin et al. , 2018 ; Turian et al. , 2010 ) . Offline pre-training in RL could enable deployment of RL methods in domains where data collection can be expensive or dangerous . ( Silver et al. , 2016 ; Gupta et al. , 2019 ; Rajeswaran et al. , 2017 ) pre-train the policy network with imitation learning to speed up RL . QT-opt ( Kalashnikov et al. , 2018 ) studies visionbased object manipulation using a diverse and large dataset collected by seven robots over several months and fine-tune the policy with 27K samples of online data . However , these methods pre-train using diverse , large , or expert datasets and it is also important to investigate the possibility of pretraining from offline datasets of different quality . Yang & Nachum ( 2021 ) ; Ajay et al . ( 2020 ) use offline pre-training to accelerate downstream tasks . AWAC ( Nair et al. , 2020 ) and Balanced Replay Lee et al . ( 2021 ) are recent works that also focus on offline-to-online RL from datasets of different quality . AWAC updates the policy network such that it is constrained during offline training while not too conservative during fine-tuning . Balanced Replay trains an additional neural network to prioritize samples in order to effectively use new data as well as near-on-policy samples in the offline dataset . We compare with AWAC and Balanced Replay to attain state-of-the-art offline-to-online RL performance on the popular D4RL benchmark . Ensembles in RL . Ensemble methods are widely used for better performance in RL ( Faußer & Schwenker , 2015 ; Osband et al. , 2016 ; Chua et al. , 2018 ; Janner et al. , 2019 ) . In model-based RL , PETS ( Chua et al. , 2018 ) and MBPO ( Janner et al. , 2019 ) use probabilistic ensembles to effectively model the dynamics of the environment . In model-free RL , ensembles of Q functions have been shown to improve performance ( Anschel et al. , 2017 ; Lan et al. , 2020 ) . REDQ ( Chen et al. , 2021 ) learns a randomized ensemble of Q functions to achieve similar sample efficiency as model-based methods without learning a dynamic model . We utilize REDQ in this work for improved sampleefficiency during online fine-tuning . Specific to offline RL , REM ( Agarwal et al. , 2020 ) uses random convex combinations of multiple Q-value estimates to calculate the Q targets for effective offline RL on Atari games . MOPO ( Yu et al. , 2020 ) uses probabilistic ensembles from PETS to learn policies from offline data using uncertainty estimates based on model disagreement . MBOP ( Argenson & Dulac-Arnold , 2020 ) uses ensembles of dynamic models , Q functions , and policy networks to get better performance on locomotion tasks . Balanced Replay ( Lee et al. , 2021 ) uses ensembles of pessimistic Q functions to mitigate instability caused by distribution shift in offline-to-online RL . While ensembling of Q functions has been studied by several prior works ( Lan et al. , 2020 ; Chen et al. , 2021 ) , we combine it with behavioral cloning loss for the purpose of robust and sampleefficient offline-to-online RL . Adaptive balancing of multiple objectives in RL . Ball et al . ( 2020 ) train policies using learned dynamics models with the objective of visiting states that most likely lead to subsequent improvement in the dynamics model , using active online learning . They adaptively weigh the maximization of cumulative rewards and minimization of model uncertainty using an online learning mechanism based on exponential weights algorithm . In this paper , we focus on offline-to-online RL using model-free methods and propose to adaptively weigh the maximization of cumulative rewards and a behavioral cloning loss . Exploration of other online learning algorithms such as exponential weights algorithm is a line of future work . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) deals with sequential decision making to maximize cumulative rewards . RL problems are often formalized as Markov decision processes ( MDPs ) . An MDP consists of a set of states S , a set of actions A , a transition dynamics st+1 ∼ p ( ·|st , at ) that represents the probability of transitioning to a state st+1 by taking action at in state st at timestep t , a scalar reward function rt = R ( st , at ) , and a discount factor γ ∈ [ 0 , 1 ] . A policy function π of an RL agent is a mapping from states to actions and defines the behavior of the agent . The value function Vπ ( s ) of a policy π is defined as the expected cumulative rewards from state s : V π ( s ) = E [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] , where the expectation is taken over state transitions st+1 ∼ p ( ·|st , at ) and policy function at ∼ π ( st ) . Similarly , the state-action value function Qπ ( s , a ) is defined as the expected cumulative rewards after taking action a in state s : Qπ ( s , a ) = E [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . The goal of RL is to learn an optimal policy function πθ with parameters θ , that maximizes the expected cumulative rewards : πθ = arg max θ Es∼S [ V πθ ( s ) ] = arg max θ Es∼S [ Qπθ ( s , πθ ( s ) ) ] . We use the TD3 algorithm for reinforcement learning ( Fujimoto et al. , 2018 ) . TD3 is an actor-critic method that alternatingly trains : ( i ) the critic network Qφ to estimate the Qπθ ( s , a ) values of the policy network πθ , and ( ii ) the policy network to produce actions that maximize the Q function : ∇θQφ ( s , πθ ( s ) ) .

Summary: This paper proposes a new offline RL with online fine-tuning method. The authors first pretrain the policy using recent offline RL method TD3+BC with offline data and then collect on-policy data to further improve the pretrained policy. To prevent the policy from either degrading performance or failing to improve at the fine-tuning stage, the authors propose an automatic scheme to adjust the $\alpha$ term that controls the BC loss in TD3+BC to ensure that the policy does not continue to contain itself to the behavior policy if there's room for improvement and also is able to stick to the previous policy if the performance is already near optimal. The authors conduct evalutions in D4RL mujoco environments and show that the approach is able to outperform prior methods in halfcheetah.