A Minimalist Approach to Offline Reinforcement Learning

1 Introduction . Traditionally , reinforcement learning ( RL ) is thought of as a paradigm for online learning , where the interaction between the RL agent and its environment is of fundamental concern for how the agent learns . In offline RL ( historically known as batch RL ) , the agent learns from a fixed-sized dataset , collected by some arbitrary and possibly unknown process [ Lange et al. , 2012 ] . Eliminating the need to interact with the environment is noteworthy as data collection can often be expensive , risky , or otherwise challenging , particularly in real-world applications . Consequently , offline RL enables the use of previously logged data or leveraging an expert , such as a human operator , without any of the risk associated with an untrained RL agent . Unfortunately , the main benefit of offline RL , the lack of environment interaction , is also what makes it a challenging task . While most off-policy RL algorithms are applicable in the offline setting , they tend to under-perform due to “ extrapolation error ” : an error in policy evaluation , where agents tend to poorly estimate the value of state-action pairs not contained in the dataset . This in turn affects policy improvement , where agents learn to prefer out-of-distribution actions whose value has been overestimated , resulting in poor performance [ Fujimoto et al. , 2019b ] . The solution class for this problem revolves around the idea that the learned policy should be kept close to the data-generating process ( or behavior policy ) , and has been given a variety of names ( such as batchconstrained [ Fujimoto et al. , 2019b ] , KL-control [ Jaques et al. , 2019 ] , behavior-regularized [ Wu et al. , 2019 ] , or policy constraint [ Levine et al. , 2020 ] ) depending on how this “ closeness ” is chosen to be implemented . While there are many proposed approaches to offline RL , we remark that few are truly “ simple ” , and even the algorithms which claim to work with minor additions to an underlying online RL 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithm make a significant number of implementation-level adjustments . In other cases , there are unmentioned hyperparameters , or secondary components , such as generative models , which make offline RL algorithms difficult to reproduce , and even more challenging to tune . Additionally , such mixture of details slow down the run times of the algorithms , and make causal attributions of performance gains and transfers of techniques across algorithms difficult , as in the case for many online RL algorithms [ Henderson et al. , 2017 , Tucker et al. , 2018 , Engstrom et al. , 2020 , Andrychowicz et al. , 2021 , Furuta et al. , 2021 ] . This motivates the need for more minimalist approaches in offline RL . In this paper , we ask : can we make a deep RL algorithm work offline with minimal changes ? We find that we can match the performance of state-of-the-art offline RL algorithms with a single adjustment to the policy update step of the TD3 algorithm [ Fujimoto et al. , 2018 ] . TD3 ’ s policy π is updated with the deterministic policy gradient [ Silver et al. , 2014 ] : π = argmax π E ( s , a ) ∼D [ Q ( s , π ( s ) ) ] . ( 1 ) Our proposed change , TD3+BC , is to simply add a behavior cloning term to regularize the policy : π = argmax π E ( s , a ) ∼D [ λQ ( s , π ( s ) ) − ( π ( s ) − a ) 2 ] , ( 2 ) with a single hyperparameter λ to control the strength of the regularizer . This modification can be made by adjusting only a single line of code . Additionally , we remark that normalizing the states over the dataset , such that they have mean 0 and standard deviation 1 , improves the stability of the learned policy . Importantly , these are the only changes made to the underlying deep RL algorithm . To accommodate reproduciblity , all of our code is open-sourced1 . We evaluate our minimal changes to the TD3 algorithm on the D4RL benchmark of continuous control tasks [ Fu et al. , 2020 ] . We find that our algorithm compares favorably against many offline RL algorithms , while being significantly easier to implement and more than halving the required computation cost . The surprising effectiveness of our minimalist approach suggests that in the context of offline RL , simpler approaches have been left underexplored in favor of more elaborate algorithmic contributions . 2 Related Work . Although to the best of our knowledge , we are the first to use TD3 with behavior cloning ( BC ) for the purpose of offline RL , we remark that combining RL with BC , and other imitation learning approaches , has been previously considered by many authors . RL + BC . With the aim of accelerating reinforcement learning from examples ( known as learning from demonstrations [ Atkeson and Schaal , 1997 ] ) , BC has been used as a regularization for policy optimization with DDPG [ Lillicrap et al. , 2015 , Nair et al. , 2018 , Goecks et al. , 2020 ] and the natural policy gradient [ Kakade , 2001 , Rajeswaran et al. , 2017 ] , but with additional sophistication through modified replay buffers and pre-training . The most similar work to our own is a SAC+BC baseline [ Haarnoja et al. , 2018 ] from Nair et al . [ 2020 ] and an unpublished course project [ Booher ] combining PPO [ Schulman et al. , 2017 ] with BC . RL + Imitation . Other than directly using BC with RL , imitation learning has been combined with RL in a variety of manners , such as mixed with adversarial methods [ Zhu et al. , 2018 , Kang et al. , 2018 ] , used for pre-training [ Pfeiffer et al. , 2018 ] , modifying the replay buffer [ Večerík et al. , 2017 , Gulcehre et al. , 2020 ] , adjusting the value function [ Kim et al. , 2013 , Hester et al. , 2017 ] , or reward shaping [ Judah et al. , 2014 , Wu et al. , 2021 ] . In all cases , these methods use demonstrations as a method for overcoming challenges in exploration or improving the learning speed of the RL agent . Offline RL . As aforementioned , offline RL methods generally rely on some approach for “ staying close ” to the data . This may be implemented using an estimate of the behavior policy and then defining an explicit policy parameterization [ Fujimoto et al. , 2019b , Ghasemipour et al. , 2020 ] or by using divergence regularization [ Jaques et al. , 2019 , Kumar et al. , 2019 , Wu et al. , 2019 , Siegel et al. , 2020 , Guo et al. , 2021 , Kostrikov et al. , 2021 ] . Other approaches use a weighted version of BC to favor high advantage actions [ Wang et al. , 2018 , Peng et al. , 2019 , Siegel et al. , 2020 , Wang et al. , 1https : //github.com/sfujim/TD3_BC 2020 , Nair et al. , 2020 ] or perform BC over an explicit subset of the data [ Chen et al. , 2020 ] . Some methods have modified the set of valid actions based on counts [ Laroche et al. , 2019 ] or the learned behavior policy [ Fujimoto et al. , 2019a ] . Another direction is to implement divergence regularization as a form of pessimism into the value estimate [ Nachum et al. , 2019 , Kumar et al. , 2020 , Buckman et al. , 2020 ] . Meta Analyses of RL Algorithms . There are a substantial amount of meta analysis works on online RL algorithms . While some focus on inadequacies in the experimental protocols [ Henderson et al. , 2017 , Osband et al. , 2019 ] , others study the roles of subtle implementation details in algorithms [ Tucker et al. , 2018 , Engstrom et al. , 2020 , Andrychowicz et al. , 2021 , Furuta et al. , 2021 ] . For example , Tucker et al . [ 2018 ] , Engstrom et al . [ 2020 ] identified that superior performances of certain algorithms were more dependent on , or even accidentally due to , minor implementation rather than algorithmic differences . Furuta et al . [ 2021 ] study two broad families of off-policy algorithms , which most offline algorithms are based on , and find that a few subtle implementation details are strongly co-adapted and critical to specific algorithms , making attributions of performance gains difficult . Recent offline research also follows a similar trend , where a number of implementation modifications are necessary for high algorithmic performances ( see Table 1 ) . In contrast , we derive our algorithm by modifying the existing TD3 with only a few lines of codes . Our results suggest the community could also learn from careful explorations of simpler alternatives , besides emphasizing algorithmic novelties and complexities . 3 Background . RL . Reinforcement learning ( RL ) is a framework aimed to deal with tasks of sequential nature . Typically , the problem is defined by a Markov decision process ( MDP ) ( S , A , R , p , γ ) , with state space S , action space A , scalar reward function R , transition dynamics p , and discount factor γ [ Sutton and Barto , 1998 ] . The behavior of an RL agent is determined by a policy π which maps states to actions ( deterministic policy ) , or states to a probability distribution over actions ( stochastic policy ) . The objective of an RL agent is to maximize the expected discounted return Eπ [ ∑∞ t=0 γ trt+1 ] , which is the expected cumulative sum of rewards when following the policy in the MDP , where the importance of the horizon is determined by the discount factor . We measure this objective by a value function , which measures the expected discounted return after taking the action a in state s : Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ trt+1|s0 = s , a0 = a ] . BC . Another approach for training policies is through imitation of an expert or behavior policy . Behavior cloning ( BC ) is an approach for imitation learning [ Pomerleau , 1991 ] , where the policy is trained with supervised learning to directly imitate the actions of a provided dataset . Unlike RL , this process is highly dependent on the performance of the data-collecting process . Offline RL . Offline RL breaks the assumption that the agent can interact with the environment . Instead , the agent is provided a fixed dataset which has been collected by some unknown datagenerating process ( such as a collection of behavior policies ) . This setting may be considered more challenging as the agent loses the opportunity to explore the MDP according to its current beliefs , and instead from infer good behavior from only the provided data . One challenge for offline RL is the problem of extrapolation error [ Fujimoto et al. , 2019b ] , which is generalization error in the approximate value function , induced by selecting actions not contained in the dataset . Simply put , it is difficult to evaluate the expected value of a policy which is sufficiently different from the behavior policy . Consequently , algorithms have taken the approach of constraining or regularizing the policy to stay near to the actions in the dataset [ Levine et al. , 2020 ] .

Summary: The authors revisit addition of a behavior regularization term for offline reinforcement learning but with minimal change to existing RL approach, namely TD3. They showcase their approach in D4RL gym environments. The results are surprisingly good for such a small change and  are competitive with SOTA for the most part.