A Minimalist Approach to Offline Reinforcement Learning

1 Introduction . Traditionally , reinforcement learning ( RL ) is thought of as a paradigm for online learning , where the interaction between the RL agent and its environment is of fundamental concern for how the agent learns . In offline RL ( historically known as batch RL ) , the agent learns from a fixed-sized dataset , collected by some arbitrary and possibly unknown process [ Lange et al. , 2012 ] . Eliminating the need to interact with the environment is noteworthy as data collection can often be expensive , risky , or otherwise challenging , particularly in real-world applications . Consequently , offline RL enables the use of previously logged data or leveraging an expert , such as a human operator , without any of the risk associated with an untrained RL agent . Unfortunately , the main benefit of offline RL , the lack of environment interaction , is also what makes it a challenging task . While most off-policy RL algorithms are applicable in the offline setting , they tend to under-perform due to “ extrapolation error ” : an error in policy evaluation , where agents tend to poorly estimate the value of state-action pairs not contained in the dataset . This in turn affects policy improvement , where agents learn to prefer out-of-distribution actions whose value has been overestimated , resulting in poor performance [ Fujimoto et al. , 2019b ] . The solution class for this problem revolves around the idea that the learned policy should be kept close to the data-generating process ( or behavior policy ) , and has been given a variety of names ( such as batchconstrained [ Fujimoto et al. , 2019b ] , KL-control [ Jaques et al. , 2019 ] , behavior-regularized [ Wu et al. , 2019 ] , or policy constraint [ Levine et al. , 2020 ] ) depending on how this “ closeness ” is chosen to be implemented . While there are many proposed approaches to offline RL , we remark that few are truly “ simple ” , and even the algorithms which claim to work with minor additions to an underlying online RL 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithm make a significant number of implementation-level adjustments . In other cases , there are unmentioned hyperparameters , or secondary components , such as generative models , which make offline RL algorithms difficult to reproduce , and even more challenging to tune . Additionally , such mixture of details slow down the run times of the algorithms , and make causal attributions of performance gains and transfers of techniques across algorithms difficult , as in the case for many online RL algorithms [ Henderson et al. , 2017 , Tucker et al. , 2018 , Engstrom et al. , 2020 , Andrychowicz et al. , 2021 , Furuta et al. , 2021 ] . This motivates the need for more minimalist approaches in offline RL . In this paper , we ask : can we make a deep RL algorithm work offline with minimal changes ? We find that we can match the performance of state-of-the-art offline RL algorithms with a single adjustment to the policy update step of the TD3 algorithm [ Fujimoto et al. , 2018 ] . TD3 ’ s policy π is updated with the deterministic policy gradient [ Silver et al. , 2014 ] : π = argmax π E ( s , a ) ∼D [ Q ( s , π ( s ) ) ] . ( 1 ) Our proposed change , TD3+BC , is to simply add a behavior cloning term to regularize the policy : π = argmax π E ( s , a ) ∼D [ λQ ( s , π ( s ) ) − ( π ( s ) − a ) 2 ] , ( 2 ) with a single hyperparameter λ to control the strength of the regularizer . This modification can be made by adjusting only a single line of code . Additionally , we remark that normalizing the states over the dataset , such that they have mean 0 and standard deviation 1 , improves the stability of the learned policy . Importantly , these are the only changes made to the underlying deep RL algorithm . To accommodate reproduciblity , all of our code is open-sourced1 . We evaluate our minimal changes to the TD3 algorithm on the D4RL benchmark of continuous control tasks [ Fu et al. , 2020 ] . We find that our algorithm compares favorably against many offline RL algorithms , while being significantly easier to implement and more than halving the required computation cost . The surprising effectiveness of our minimalist approach suggests that in the context of offline RL , simpler approaches have been left underexplored in favor of more elaborate algorithmic contributions . 2 Related Work . Although to the best of our knowledge , we are the first to use TD3 with behavior cloning ( BC ) for the purpose of offline RL , we remark that combining RL with BC , and other imitation learning approaches , has been previously considered by many authors . RL + BC . With the aim of accelerating reinforcement learning from examples ( known as learning from demonstrations [ Atkeson and Schaal , 1997 ] ) , BC has been used as a regularization for policy optimization with DDPG [ Lillicrap et al. , 2015 , Nair et al. , 2018 , Goecks et al. , 2020 ] and the natural policy gradient [ Kakade , 2001 , Rajeswaran et al. , 2017 ] , but with additional sophistication through modified replay buffers and pre-training . The most similar work to our own is a SAC+BC baseline [ Haarnoja et al. , 2018 ] from Nair et al . [ 2020 ] and an unpublished course project [ Booher ] combining PPO [ Schulman et al. , 2017 ] with BC . RL + Imitation . Other than directly using BC with RL , imitation learning has been combined with RL in a variety of manners , such as mixed with adversarial methods [ Zhu et al. , 2018 , Kang et al. , 2018 ] , used for pre-training [ Pfeiffer et al. , 2018 ] , modifying the replay buffer [ Večerík et al. , 2017 , Gulcehre et al. , 2020 ] , adjusting the value function [ Kim et al. , 2013 , Hester et al. , 2017 ] , or reward shaping [ Judah et al. , 2014 , Wu et al. , 2021 ] . In all cases , these methods use demonstrations as a method for overcoming challenges in exploration or improving the learning speed of the RL agent . Offline RL . As aforementioned , offline RL methods generally rely on some approach for “ staying close ” to the data . This may be implemented using an estimate of the behavior policy and then defining an explicit policy parameterization [ Fujimoto et al. , 2019b , Ghasemipour et al. , 2020 ] or by using divergence regularization [ Jaques et al. , 2019 , Kumar et al. , 2019 , Wu et al. , 2019 , Siegel et al. , 2020 , Guo et al. , 2021 , Kostrikov et al. , 2021 ] . Other approaches use a weighted version of BC to favor high advantage actions [ Wang et al. , 2018 , Peng et al. , 2019 , Siegel et al. , 2020 , Wang et al. , 1https : //github.com/sfujim/TD3_BC 2020 , Nair et al. , 2020 ] or perform BC over an explicit subset of the data [ Chen et al. , 2020 ] . Some methods have modified the set of valid actions based on counts [ Laroche et al. , 2019 ] or the learned behavior policy [ Fujimoto et al. , 2019a ] . Another direction is to implement divergence regularization as a form of pessimism into the value estimate [ Nachum et al. , 2019 , Kumar et al. , 2020 , Buckman et al. , 2020 ] . Meta Analyses of RL Algorithms . There are a substantial amount of meta analysis works on online RL algorithms . While some focus on inadequacies in the experimental protocols [ Henderson et al. , 2017 , Osband et al. , 2019 ] , others study the roles of subtle implementation details in algorithms [ Tucker et al. , 2018 , Engstrom et al. , 2020 , Andrychowicz et al. , 2021 , Furuta et al. , 2021 ] . For example , Tucker et al . [ 2018 ] , Engstrom et al . [ 2020 ] identified that superior performances of certain algorithms were more dependent on , or even accidentally due to , minor implementation rather than algorithmic differences . Furuta et al . [ 2021 ] study two broad families of off-policy algorithms , which most offline algorithms are based on , and find that a few subtle implementation details are strongly co-adapted and critical to specific algorithms , making attributions of performance gains difficult . Recent offline research also follows a similar trend , where a number of implementation modifications are necessary for high algorithmic performances ( see Table 1 ) . In contrast , we derive our algorithm by modifying the existing TD3 with only a few lines of codes . Our results suggest the community could also learn from careful explorations of simpler alternatives , besides emphasizing algorithmic novelties and complexities . 3 Background . RL . Reinforcement learning ( RL ) is a framework aimed to deal with tasks of sequential nature . Typically , the problem is defined by a Markov decision process ( MDP ) ( S , A , R , p , γ ) , with state space S , action space A , scalar reward function R , transition dynamics p , and discount factor γ [ Sutton and Barto , 1998 ] . The behavior of an RL agent is determined by a policy π which maps states to actions ( deterministic policy ) , or states to a probability distribution over actions ( stochastic policy ) . The objective of an RL agent is to maximize the expected discounted return Eπ [ ∑∞ t=0 γ trt+1 ] , which is the expected cumulative sum of rewards when following the policy in the MDP , where the importance of the horizon is determined by the discount factor . We measure this objective by a value function , which measures the expected discounted return after taking the action a in state s : Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ trt+1|s0 = s , a0 = a ] . BC . Another approach for training policies is through imitation of an expert or behavior policy . Behavior cloning ( BC ) is an approach for imitation learning [ Pomerleau , 1991 ] , where the policy is trained with supervised learning to directly imitate the actions of a provided dataset . Unlike RL , this process is highly dependent on the performance of the data-collecting process . Offline RL . Offline RL breaks the assumption that the agent can interact with the environment . Instead , the agent is provided a fixed dataset which has been collected by some unknown datagenerating process ( such as a collection of behavior policies ) . This setting may be considered more challenging as the agent loses the opportunity to explore the MDP according to its current beliefs , and instead from infer good behavior from only the provided data . One challenge for offline RL is the problem of extrapolation error [ Fujimoto et al. , 2019b ] , which is generalization error in the approximate value function , induced by selecting actions not contained in the dataset . Simply put , it is difficult to evaluate the expected value of a policy which is sufficiently different from the behavior policy . Consequently , algorithms have taken the approach of constraining or regularizing the policy to stay near to the actions in the dataset [ Levine et al. , 2020 ] .

Summary: This paper presents a simple offline RL method that only requires a few lines of changes in code. The authors add a BC loss as the regularization to the standard policy update objective while also normalizing the states. The authors also normalize the coefficient of the BC loss with the average Q values in the dataset to balance the policy update objective and the BC loss. Through experiments, the proposed method appears to perform comparably to the best-performing prior offline RL approaches despite being simple and requiring lower tuning complexity.