Offline RL Without Off-Policy Evaluation

1 Introduction . An important step towards effective real-world RL is to improve sample efficiency . One avenue towards this goal is offline RL ( also known as batch RL ) where we attempt to learn a new policy from data collected by some other behavior policy without interacting with the environment . Recent work in offline RL is well summarized by Levine et al . [ 2020 ] . In this paper , we challenge the dominant paradigm in the deep offline RL literature that primarily relies on actor-critic style algorithms that alternate between policy evaluation and policy improvement [ Fujimoto et al. , 2018a , 2019 , Peng et al. , 2019 , Kumar et al. , 2019 , 2020 , Wang et al. , 2020b , Wu et al. , 2019 , Kostrikov et al. , 2021 , Jaques et al. , 2019 , Siegel et al. , 2020 , Nachum et al. , 2019 ] . All these algorithms rely heavily on off-policy evaluation to learn the critic . Instead , we find that a simple baseline which only performs one step of policy improvement using the behavior Q function often outperforms the more complicated iterative algorithms . Explicitly , we find that our one-step algorithm beats prior results of iterative algorithms on most of the gym-mujoco [ Brockman et al. , 2016 ] and Adroit [ Rajeswaran et al. , 2017 ] tasks in the the D4RL benchmark suite [ Fu et al. , 2020 ] . We then dive deeper to understand why such a simple baseline is effective . First , we examine what goes wrong for the iterative algorithms . When these algorithms struggle , it is often due to poor off-policy evaluation leading to inaccurate Q values . We attribute this to two causes : ( 1 ) distribution shift between the behavior policy and the policy to be evaluated , and ( 2 ) iterative error exploitation whereby policy optimization introduces bias and dynamic programming propagates this bias across the state space . We show that empirically both issues exist in the benchmark tasks and that one way to avoid these issues is to simply avoid off-policy evaluation entirely . Finally , we recognize that while the the one-step algorithm is a strong baseline , it is not always the best choice . In the final section we provide some guidance about when iterative algorithms can perform better than the simple one-step baseline . Namely , when the dataset is large and behavior policy has good coverage of the state-action space , then off-policy evaluation can succeed and iterative 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithms can be effective . In contrast , if the behavior policy is already fairly good , but as a result does not have full coverage , then one-step algorithms are often preferable . Our main contributions are : • A demonstration that a simple baseline of one step of policy improvement outperforms more complicated iterative algorithms on a broad set of offline RL problems . • An examination of failure modes of off-policy evaluation in iterative offline RL algorithms . • A description of when one-step algorithms are likely to outperform iterative approaches . 2 Setting and notation . We will consider an offline RL setup as follows . LetM = { S , A , ρ , P , R , γ } be a discounted infinitehorizon MDP . In this work we focus on applications in continuous control , so we will generally assume that both S andA are continuous and bounded . We consider the offline setting where rather than interacting withM , we only have access to a dataset DN of N tuples of ( si , ai , ri ) collected by some behavior policy β with initial state distribution ρ . Let r ( s , a ) = Er|s , a [ r ] be the expected reward . Define the state-action value function for any policy π by Qπ ( s , a ) : = EP , π|s0=s , a0=a [ ∑∞ t=0 γ tr ( st , at ) ] . The objective is to maximize the expected return J of the learned policy : J ( π ) : = E ρ , P , π [ ∞∑ t=0 γtr ( st , at ) ] = E s∼ρ a∼π|s [ Qπ ( s , a ) ] . ( 1 ) Following Fu et al . [ 2020 ] and others in this line of work , we allow access to the environment to tune a small ( < 10 ) set of hyperparameters . See Paine et al . [ 2020 ] for a discussion of the active area of research on hyperparameter tuning for offline RL . We also discuss this further in Appendix C . 3 Related work . Iterative algorithms . Most prior work on deep offline RL consists of iterative actor-critic algorithms . The primary innovation of each paper is to propose a different mechanism to ensure that the learned policy does not stray too far from the data generated by the behavior policy . Broadly , we group these methods into three camps : policy constraints/regularization , modifications of imitation learning , and Q regularization : 1 . The majority of prior work acts directly on the policy . Some authors have proposed explicit constraints on the learned policy to only select actions where ( s , a ) has sufficient support under the data generating distribution [ Fujimoto et al. , 2018a , 2019 , Laroche et al. , 2019 ] . Another proposal is to regularize the learned policy towards the behavior policy [ Wu et al. , 2019 ] usually either with a KL divergence [ Jaques et al. , 2019 ] or MMD [ Kumar et al. , 2019 ] . This is a very straighforward way to stay close to the behavior with a hyperparameter that determines just how close . All of these algorithms are iterative and rely on off-policy evaluation . 2 . Siegel et al . [ 2020 ] , Wang et al . [ 2020b ] , Chen et al . [ 2020 ] all use algorithms that filter out datapoints with low Q values and then perform imitation learning . Wang et al . [ 2018 ] , Peng et al . [ 2019 ] use a weighted imitation learning algorithm where the weights are determined by exponentiated Q values . These algorithms are iterative . 3 . Another way to prevent the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state , action pairs [ Wu et al. , 2019 , Nachum et al. , 2019 , Kumar et al. , 2020 , Kostrikov et al. , 2021 , Gulcehre et al. , 2021 ] . However , being able to properly quantify uncertainty about unknown states is notoriously difficult when dealing with neural network value functions [ Buckman et al. , 2020 ] . One-step algorithms . Some recent work has also noted that optimizing policies based on the behavior value function can perform surprisingly well . As we do , Goo and Niekum [ 2020 ] studies the continuous control tasks from the D4RL benchmark , but they examine a complicated algorithm involving ensembles , distributional Q functions , and a novel regularization technique . In contrast , we analyze a substantially simpler algorithm and get better performance on the D4RL tasks . We also focus more of our contribution on understanding and explaining this performance . Gulcehre et al . [ 2021 ] studies the discrete action setting and finds that a one-step algorithm ( which they call “ behavior value estimation ” ) outperforms prior work on Atari games and other discrete action tasks from the RL Unplugged benchmark [ Gulcehre et al. , 2020 ] . They also introduce a novel regularizer for the evaluation step . In contrast , we consider the continuous control setting . This is a substantial difference in setting since continuous control requires actor-critic algorithms with parametric policies while in the discrete setting the policy improvement step can be computed exactly from the Q function . Moreover , while Gulcehre et al . [ 2021 ] attribute the poor performance of iterative algorithms to “ overestimation ” , we define and separate the issues of distribution shift and iterative error exploitation which can combine to cause overestimation . This separation helps to expose the difference between the fundamental limits of off-policy evaluation from the specific problems induced by iterative algorithms , and will hopefully be a useful distinction to inspire future work . Finally , a one-step variant is also briefly discussed in Nadjahi et al . [ 2019 ] , but is not the focus of that work . There are also important connections between the one-step algorithm and the literature on conservative policy improvement [ Kakade and Langford , 2002 , Schulman et al. , 2015 , Achiam et al. , 2017 ] , which we discuss in more detail in Appendix B . 4 Defining the algorithms . In this section we provide a unified algorithmic template for model-free offline RL algorithms as offline approximate modified policy iteration . We show how this template captures our one-step algorithm as well as a multi-step policy iteration algorithm and an iterative actor-critic algorithm . Then any choice of policy evaluation and policy improvement operators can be used to define one-step , multi-step , and iterative algorithms . 4.1 Algorithmic template . Algorithm 1 : OAMPI input : K , dataset DN , estimated behavior β̂ Set π0 = β̂ . Initialize Q̂π−1 randomly . for k = 1 , . . . , K do Policy evaluation : Q̂πk−1 = E ( πk−1 , DN , Q̂πk−2 ) Policy improvement : πk = I ( Q̂πk−1 , β̂ , DN , πk−1 ) end We consider a generic offline approximate modified policy iteration ( OAMPI ) scheme , shown in Algorithm 1 ( and based off of Puterman and Shin [ 1978 ] , Scherrer et al . [ 2012 ] ) . Essentially the algorithm alternates between two steps . First , there is a policy evaluation step where we estimate the Q function of the current policy πk−1 by Q̂πk−1 using only the dataset DN . Implementations also often use the prior Q estimate Q̂πk−2 to warm-start the approximation process . Second , there is a policy improvement step . This step takes in the estimated Q function Q̂πk−1 , the estimated behavior β̂ , and the dataset DN and produces a new policy πk . Again an algorithm may use πk−1 to warm-start the optimization . Moreover , we expect this improvement step to be regularized or constrained to ensure that πk remains in the support of β and DN . Choices for this step are discussed below . Now we discuss a few ways to instantiate the template . One-step . The simplest algorithm sets the number of iterations K = 1 . We learn β̂ by maximum likelihood and train the policy evaluation step to estimate Qβ . Then we use any one of the policy improvement operators discussed below to learn π1 . Importantly , this algorithm completely avoids off-policy evaluation . Multi-step . The multi-step algorithm now sets K > 1 . The evaluation operator must evaluate off-policy since DN is collected by β , but evaluation steps for K ≥ 2 require evaluating policies πk−1 6= β . Each iteration is trained to convergence in both the estimation and improvement steps . Iterative actor-critic . An actor critic approach looks somewhat like the multi-step algorithm , but does not attempt to train to convergence at each iteration and uses a much larger K. Here each iteration consists of one gradient step to update the Q estimate and one gradient step to improve the policy . Since all of the evaluation and improvement operators that we consider are gradient-based , this algorithm can adapt the same evaluation and improvement operators used by the multi-step algorithm . Most algorithms from the literature fall into this category [ Fujimoto et al. , 2018a , Kumar et al. , 2019 , 2020 , Wu et al. , 2019 , Wang et al. , 2020b , Siegel et al. , 2020 ] .

Summary: The authors study the notion of iterative policy improvement in offline reinforcement learning, notably finding that a single policy improvement step can outperform standard iterative benchmarks on the D4RL benchmark. To provide more insight on this, the authors also compare to an intermediate version, “multi-step”, and analyze the performance of the three variants (one-step, multi-step, and iterative) with different regularization hyperparameters. They also show results for the MSE of the estimated Q function, as well as illustrate multi-step overestimation on Gridworld environments. Finally, the authors discuss when multi-step algorithms are better, namely when noise signal is low and more propagation of signal can help.