Offline RL Without Off-Policy Evaluation

1 Introduction . An important step towards effective real-world RL is to improve sample efficiency . One avenue towards this goal is offline RL ( also known as batch RL ) where we attempt to learn a new policy from data collected by some other behavior policy without interacting with the environment . Recent work in offline RL is well summarized by Levine et al . [ 2020 ] . In this paper , we challenge the dominant paradigm in the deep offline RL literature that primarily relies on actor-critic style algorithms that alternate between policy evaluation and policy improvement [ Fujimoto et al. , 2018a , 2019 , Peng et al. , 2019 , Kumar et al. , 2019 , 2020 , Wang et al. , 2020b , Wu et al. , 2019 , Kostrikov et al. , 2021 , Jaques et al. , 2019 , Siegel et al. , 2020 , Nachum et al. , 2019 ] . All these algorithms rely heavily on off-policy evaluation to learn the critic . Instead , we find that a simple baseline which only performs one step of policy improvement using the behavior Q function often outperforms the more complicated iterative algorithms . Explicitly , we find that our one-step algorithm beats prior results of iterative algorithms on most of the gym-mujoco [ Brockman et al. , 2016 ] and Adroit [ Rajeswaran et al. , 2017 ] tasks in the the D4RL benchmark suite [ Fu et al. , 2020 ] . We then dive deeper to understand why such a simple baseline is effective . First , we examine what goes wrong for the iterative algorithms . When these algorithms struggle , it is often due to poor off-policy evaluation leading to inaccurate Q values . We attribute this to two causes : ( 1 ) distribution shift between the behavior policy and the policy to be evaluated , and ( 2 ) iterative error exploitation whereby policy optimization introduces bias and dynamic programming propagates this bias across the state space . We show that empirically both issues exist in the benchmark tasks and that one way to avoid these issues is to simply avoid off-policy evaluation entirely . Finally , we recognize that while the the one-step algorithm is a strong baseline , it is not always the best choice . In the final section we provide some guidance about when iterative algorithms can perform better than the simple one-step baseline . Namely , when the dataset is large and behavior policy has good coverage of the state-action space , then off-policy evaluation can succeed and iterative 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithms can be effective . In contrast , if the behavior policy is already fairly good , but as a result does not have full coverage , then one-step algorithms are often preferable . Our main contributions are : • A demonstration that a simple baseline of one step of policy improvement outperforms more complicated iterative algorithms on a broad set of offline RL problems . • An examination of failure modes of off-policy evaluation in iterative offline RL algorithms . • A description of when one-step algorithms are likely to outperform iterative approaches . 2 Setting and notation . We will consider an offline RL setup as follows . LetM = { S , A , ρ , P , R , γ } be a discounted infinitehorizon MDP . In this work we focus on applications in continuous control , so we will generally assume that both S andA are continuous and bounded . We consider the offline setting where rather than interacting withM , we only have access to a dataset DN of N tuples of ( si , ai , ri ) collected by some behavior policy β with initial state distribution ρ . Let r ( s , a ) = Er|s , a [ r ] be the expected reward . Define the state-action value function for any policy π by Qπ ( s , a ) : = EP , π|s0=s , a0=a [ ∑∞ t=0 γ tr ( st , at ) ] . The objective is to maximize the expected return J of the learned policy : J ( π ) : = E ρ , P , π [ ∞∑ t=0 γtr ( st , at ) ] = E s∼ρ a∼π|s [ Qπ ( s , a ) ] . ( 1 ) Following Fu et al . [ 2020 ] and others in this line of work , we allow access to the environment to tune a small ( < 10 ) set of hyperparameters . See Paine et al . [ 2020 ] for a discussion of the active area of research on hyperparameter tuning for offline RL . We also discuss this further in Appendix C . 3 Related work . Iterative algorithms . Most prior work on deep offline RL consists of iterative actor-critic algorithms . The primary innovation of each paper is to propose a different mechanism to ensure that the learned policy does not stray too far from the data generated by the behavior policy . Broadly , we group these methods into three camps : policy constraints/regularization , modifications of imitation learning , and Q regularization : 1 . The majority of prior work acts directly on the policy . Some authors have proposed explicit constraints on the learned policy to only select actions where ( s , a ) has sufficient support under the data generating distribution [ Fujimoto et al. , 2018a , 2019 , Laroche et al. , 2019 ] . Another proposal is to regularize the learned policy towards the behavior policy [ Wu et al. , 2019 ] usually either with a KL divergence [ Jaques et al. , 2019 ] or MMD [ Kumar et al. , 2019 ] . This is a very straighforward way to stay close to the behavior with a hyperparameter that determines just how close . All of these algorithms are iterative and rely on off-policy evaluation . 2 . Siegel et al . [ 2020 ] , Wang et al . [ 2020b ] , Chen et al . [ 2020 ] all use algorithms that filter out datapoints with low Q values and then perform imitation learning . Wang et al . [ 2018 ] , Peng et al . [ 2019 ] use a weighted imitation learning algorithm where the weights are determined by exponentiated Q values . These algorithms are iterative . 3 . Another way to prevent the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state , action pairs [ Wu et al. , 2019 , Nachum et al. , 2019 , Kumar et al. , 2020 , Kostrikov et al. , 2021 , Gulcehre et al. , 2021 ] . However , being able to properly quantify uncertainty about unknown states is notoriously difficult when dealing with neural network value functions [ Buckman et al. , 2020 ] . One-step algorithms . Some recent work has also noted that optimizing policies based on the behavior value function can perform surprisingly well . As we do , Goo and Niekum [ 2020 ] studies the continuous control tasks from the D4RL benchmark , but they examine a complicated algorithm involving ensembles , distributional Q functions , and a novel regularization technique . In contrast , we analyze a substantially simpler algorithm and get better performance on the D4RL tasks . We also focus more of our contribution on understanding and explaining this performance . Gulcehre et al . [ 2021 ] studies the discrete action setting and finds that a one-step algorithm ( which they call “ behavior value estimation ” ) outperforms prior work on Atari games and other discrete action tasks from the RL Unplugged benchmark [ Gulcehre et al. , 2020 ] . They also introduce a novel regularizer for the evaluation step . In contrast , we consider the continuous control setting . This is a substantial difference in setting since continuous control requires actor-critic algorithms with parametric policies while in the discrete setting the policy improvement step can be computed exactly from the Q function . Moreover , while Gulcehre et al . [ 2021 ] attribute the poor performance of iterative algorithms to “ overestimation ” , we define and separate the issues of distribution shift and iterative error exploitation which can combine to cause overestimation . This separation helps to expose the difference between the fundamental limits of off-policy evaluation from the specific problems induced by iterative algorithms , and will hopefully be a useful distinction to inspire future work . Finally , a one-step variant is also briefly discussed in Nadjahi et al . [ 2019 ] , but is not the focus of that work . There are also important connections between the one-step algorithm and the literature on conservative policy improvement [ Kakade and Langford , 2002 , Schulman et al. , 2015 , Achiam et al. , 2017 ] , which we discuss in more detail in Appendix B . 4 Defining the algorithms . In this section we provide a unified algorithmic template for model-free offline RL algorithms as offline approximate modified policy iteration . We show how this template captures our one-step algorithm as well as a multi-step policy iteration algorithm and an iterative actor-critic algorithm . Then any choice of policy evaluation and policy improvement operators can be used to define one-step , multi-step , and iterative algorithms . 4.1 Algorithmic template . Algorithm 1 : OAMPI input : K , dataset DN , estimated behavior β̂ Set π0 = β̂ . Initialize Q̂π−1 randomly . for k = 1 , . . . , K do Policy evaluation : Q̂πk−1 = E ( πk−1 , DN , Q̂πk−2 ) Policy improvement : πk = I ( Q̂πk−1 , β̂ , DN , πk−1 ) end We consider a generic offline approximate modified policy iteration ( OAMPI ) scheme , shown in Algorithm 1 ( and based off of Puterman and Shin [ 1978 ] , Scherrer et al . [ 2012 ] ) . Essentially the algorithm alternates between two steps . First , there is a policy evaluation step where we estimate the Q function of the current policy πk−1 by Q̂πk−1 using only the dataset DN . Implementations also often use the prior Q estimate Q̂πk−2 to warm-start the approximation process . Second , there is a policy improvement step . This step takes in the estimated Q function Q̂πk−1 , the estimated behavior β̂ , and the dataset DN and produces a new policy πk . Again an algorithm may use πk−1 to warm-start the optimization . Moreover , we expect this improvement step to be regularized or constrained to ensure that πk remains in the support of β and DN . Choices for this step are discussed below . Now we discuss a few ways to instantiate the template . One-step . The simplest algorithm sets the number of iterations K = 1 . We learn β̂ by maximum likelihood and train the policy evaluation step to estimate Qβ . Then we use any one of the policy improvement operators discussed below to learn π1 . Importantly , this algorithm completely avoids off-policy evaluation . Multi-step . The multi-step algorithm now sets K > 1 . The evaluation operator must evaluate off-policy since DN is collected by β , but evaluation steps for K ≥ 2 require evaluating policies πk−1 6= β . Each iteration is trained to convergence in both the estimation and improvement steps . Iterative actor-critic . An actor critic approach looks somewhat like the multi-step algorithm , but does not attempt to train to convergence at each iteration and uses a much larger K. Here each iteration consists of one gradient step to update the Q estimate and one gradient step to improve the policy . Since all of the evaluation and improvement operators that we consider are gradient-based , this algorithm can adapt the same evaluation and improvement operators used by the multi-step algorithm . Most algorithms from the literature fall into this category [ Fujimoto et al. , 2018a , Kumar et al. , 2019 , 2020 , Wu et al. , 2019 , Wang et al. , 2020b , Siegel et al. , 2020 ] .

Summary: Popular RL (both online and offline) algorithms iterate between two stages: policy evaluation where a Q-function: $S \times A \rightarrow \mathbb{E} \sum r$ is estimated and policy improvement where the policy $\pi: S \rightarrow A$ which maximizes the rewards is constructed. Typically, the stages are interleaved through the training process. The paper evaluates and argues for the idea of using, in the case of offline RL, only a single step of policy improvement after training Q-function to convergence.