Offline RL Without Off-Policy Evaluation
Authors: David Brandfonbrener
William F Whitney
Rajesh Ranganath
Joan Bruna
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Offline reinforcement learning
reinforcement learning

Abstract: Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

Offline RL Without Off-Policy Evaluation
David Brandfonbrener
Rajesh Ranganath Department of Computer Science, Center for Data Science New York University
William F. Whitney
Joan Bruna
david.
Abstract
Most prior approaches to offinn reinforcement learning (RL. have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained//egullrizze policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one- step algorithm beats the previously reported results of iterative algorithms large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, the one-step algorithm is due to a combination of favorable structure the environment and behavior policy,
hypothesize that the strong performance
Introduction
An important step towards effective real-world RL is to improve sample efficiency. One avenue towards this goal is offline RL (also known batch RL) where attempt learn new policy from data collected by some other behavior policy without interacting with the environment. Recent work in offline RL is well summarized by Levine al. (2020).
In this paper, we challenge the dominant paradigm in the deep offline RL literature that primarily relies on actor-critic style algorithms that alternate between policy evaluation and policy improvement
[Fujimoto et al. 201 Sa, 2019 Peng et al. [019 Kumar et al. 2019 2020. Wang et al. 2020b Wu al.) 2019] Kostrikov et al. | Jaques al.) 2019 (Siegel al.) 2020 Nachum et al. 2019]. All these algorithms rely heavily on off-policy evaluation to learn the critic. Instead, we find that simple baseline which only performs one step of policy improvement using the behavior Q function often outperforms the more complicated iterative algorithms. Explicitly, we find that one-step algorithm beats prior results of iterative algorithms on most of the gym-mujoco [Brockman et al. 2016) and Adroit RRajeswaran et al. 2017 tasks in the the D4RL benchmark suite [Fu et al. 2020
We then dive deeper to understand why such simple baseline is effective. First, we examine what goes wrong for the iterative algorithms. When these algorithms struggle, it is often due to poor off-policy evaluation leading to inaccurate Q values. We attribute this to two causes: (1) distribution shift between the behavior policy and the policy to be evaluated, and (2) iterative error exploitation whereby policy optimization introduces bias and dynamic programming propagates this bias across the state space. We show that empirically both issues exist the benchmark tasks and that one way to avoid these issues is to simply avoid off-polic evaluation entirely.
Finally, we recognize that while the the one-step algorithm is strong baseline, it is not always the best choice. In the final section provide some guidance about when iterative algorithms can perform better than the simple one-step baseline. Namely, when the dataset is large and behavior policy has good coverage of the state-action space, then off- policy evaluation can succeed and iterative
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Ope-step
Mult-step
x
B
Safe policies Unsaf's policies
Figure 1: A cartoon illustration of the difference between one-step and multi-step methods. All algorithms constrain themselves neighborhood of "safe" policies around B. A one-step approach (left) only uses the on-policy Q&, while multi-step approach (right) repeatedly uses off off-policy @
algorithms can be effective. contrast, if the behavior policy is already fairly good, but as result does not have full coverage, then one-step algorithms are often preferable.
Our main contributions are:
demonstration that a simple baseline of one step of policy improvement outperforms more complicated iterative algorithms
broad set of offline RL problems.
An examination of failure modes of off-policy evaluation in iterative offline RL algorithms.
description of when one-step algorithms are likely to outperform iterative approaches.
2 Setting and notation
We will consider offinn RL setup follows. Let M (S, R, be discounted infinite- horizon MDP. In this work we focus on applications in continuous control, so we will generally assume that both S and A continuous and bounded. We consider the offline setting where rather than inter- acting with M, we only have access to a dataset DN of N tuples of (st, a,, ri) collected by some be- havior policy 3 with initial state distribution p. Let a) Er|s,a[r] be the expected reward. Define the state-action value function for any policy x by (= a) == Eppesso= ag=a (s 0z)]. The objective is to maximize the expected return J of the learned policy:
J(T) =
E p.P.st
Σ
? (r at)
E snup
(s)
amo
Following Fu et al. (020) and others in this line of work, we allow access to the environment to tune a small (< 10) set of hyperparameters. See Paine et al. [2020] for discussion of the active area of
research on hyperparameter tuning for offline RI. We also discuss this further in Appendix C
Related work
Iterative algorithms. Most prior work on deep offline RL consists of iterative actor-critic algo- rithms. The primary innovation of each paper is to propose different mechanism to ensure that the learned policy does not stray too far from the data generated by the behavior policy. Broadly, we group these methods into three camps: policy constraints/regularizztiin modifications of imitation learning, and Q regularization:
1.
The majority of prior work acts directly on the policy. Some authors have proposed explicit constraints on the learned policy to only select actions where (s, a) has sufficient support under the data generating distribution [Fujimoto et al.) 2018a 2019 Laroche et al., 2019]. Another proposal to regularize the learned policy towards the behavior policy [Wu et al. 2019) usually
either with KL. divergence [Jaques et al. 2019 or MMD [umarr et al. 2019] This is a very
straighforward way to stay close to the behavior with hyperparameter that determines just how close. All of these algorithms are iterative and rely on off-policy evaluation.

---

2. Siegel et al. 2020), Wang et al. [2020b]. Chen et al. [2020] all use algorithms that filter out datapoints with low Q values and then perform imitation learning. [Wang et al. (2018), Peng et al. [2019] use weighted imitation learning algorithm where the weights are determined by exponentiated Q values. These algorithms are iterative.
Another way to prevent the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state, action pairs (Wu al.) 2019 Nachum al. 2019 Kumar al., 2020 Kostrikov et al., 2021 Gulcehre et al. 2021] However, being able to properly quantity uncertainty about unknown states is notoriously difficult when dealing with neural network value functions [Buckman et al. 2020
One-step algorithms. Some recent work has also noted that optimizing policies based on the behavior value function can perform surprisingly well. As we do, Goo and Niekum (2020) studies the continuous control tasks from the D4RL benchmark, but they examine a complicated algorithm involving ensembles, distributional Q functions, and novel regularization technique. In contrast, we analyze a substantially simpler algorithm and get better performance on the D4RI, tasks. We also focus more of our contribution on understanding and explaining this performance. Gulcehree ict al. [202]] studies the discrete action setting and finds that a one-step algorithm (which they call "behavior value estimation") outperforms prior work on Atari games and other discrete action tasks from the RL Unplugged benchmark [Gulcehre et al.) 2020]. They also introduce a novel regularizer for the evaluation step. In contrast, we consider the continuous control setting. This is a substantial difference in setting since continuous control requires actor-critic algorithms with parametric policies while in the discrete setting the policy improvement step can be computed exactly from the Q function. Moreover, while Gulcehre et al. [202I] attribute the poor performance of iterative algorithms to "overestimation" we define and separate the issues of distribution shift and iterative error exploitation which can combine to cause overestimation. This separation helps to expose the difference between the fundamental limits of off-policy evaluation from the specifie problems induced by iterative algorithms, and will hopefully be a useful distinction to inspire future work. Finally, one-step variant is also briefly discussed in Nadjahi et al. 2019 but is not the focus of that work.
There are also important connections between the one-step algorithm and the literature conservative policy improvement Kakade and Langford| 2002 Schulman et al. 2015 Achiam et al. 2017), which
we discuss in more detail in Appendix B
4 Defining the algorithms
In this section we provide unified algorithmic template for model-free offline RL algorithms as offline approximate modified policy iteration. We show how this template captures our one-step algorithm as well as multi-step policy iteration algorithm and an iterative actor-critic algorithm. Then any choice of policy evaluation and policy improvement operators can be used to define one-step, multi-step, and iterative algorithms.
4.1 Algorithmic template
We consider generic offline ap- proximate modified policy iteration (OAMPI) scheme, shown in Algo- rithm I (and based off of [Puter man and Shin 1978), [Scherrer al. (2012)) Essentially the algorithm alternates between two steps. First, there is a policy evaluation step where we estimate the Q function of the cur-
Algorithm 1: OAMPI
input: K, dataset DN estimated behavior ,3
Set æ = 8. Initialize QT- randomly.
for k = I..., K do
Policy evaluation:
Q=L E(T DN. QTh 2) DN, 1k-1)
Policy improvement: T =(@ 1,,
end
rent policy - by TTL-1 using only the dataset Dx Implementations also often use the prior Q estimate == to warm-start the approximation process. Second, there is policy improvement step. This step takes in the estimated Q function Q 1, the estimated behavior 3 and the dataset D and produces a new policy Ti- Again an algorithm may use - to warm-start the optimization. Moreover, we expect this improvement step to be regularized or constrained to ensure that गa remains in the support of and Dx Choices for this step are discussed below. Now we discuss a few ways to instantiate the template.
3

---

One-step.
The simplest algorithm sets the number of iterations K = 1. We learn 3 by maximum likelihood and train the policy evaluation step to estimate Q Then we use any one of the policy improvement operators discussed below to learn Ti. Importantly, this algorithm completely avoids off-policy evaluation.
Multi-step.
The multi-step algorithm now sets K > 1. The evaluation operator must evaluate off-policy since Dy is collected by B. but evaluation steps for K 2 2 require evaluating policies Tk-1 + B. Each iteration is trained to convergence in both the estimation and improvement steps.
Iterative actor-critic. An actor critic approach looks somewhat like the multi-step algorithm, but does not attempt to train to convergence at each iteration and uses a much larger K. Here each iteration consists of one gradient step update the Q estimate and one gradient step to improve the policy. Since all of the evaluation and improvement operators that we consider are gradient-based this algorithm can adapt the same evaluation and improvement operators used by the multi-step algorithm. Most algorithms from the literature fall into this category [Fujimoto et al., 2018a Kumar
al.
2019 2020
Wu et al. 2019 Wang et al. 20206 Siegel al. 2020
4.2 Policy evaluation operator
Following prior work on continuous state and action problems, we always evaluate by simple fitted Q
evaluation [Fujimoto et al. 2018a Kumar et al. 2019 Sieeel et al. 2020 Wang et al. 22200b Paine
et al. 2020 Wang et al. [021]] J. In practice this is optimized by TD-style learning with the use of a
target network Mnih et al. 2015) in DDPG [Lillierap al.. 2015). We do not use any double
Q learning or Q ensembles FFujimoto et al., 2018b). For the one-step and multi-step algorithms we
train the evaluation procedure convergence on each iteration and for the iterative algorithm each iteration takes a single stochastic gradient step. See Voloshin et al. [2019), Wang et al.) [202] I] for more comprehensive examinations of policy evaluation and some evidence that this simple fitted Q iteration approach is reasonable. It is an interesting direction for future work to consider other operators that use things like importance weighting [Munos al. 2016 or pessimism [Kumar et al.) 2020 Buckman et al. 2020).
4.3 Policy improvement operators
To instantiate the template, we also need to choose specific policy improvement operator I. We consider the following improvement operators selected from those discussed in the related work section. Each operator has a hyperparameter controlling deviation from the behavior policy.
Behavior cloning. The simplest baseline worth including is to just return 3 as the new policy n. Any policy improvement operator ought to perform at least well this baseline.
Constrained policy updates. Algorithms like BCQ Fujimoto et al. 201 Sa and SPIBB Laroche al. 2019] constrain the policy updates to within the support of the data/behavior. In favor of simplicity we implement simplified version of the BCQ algorithm that removes the "perturbation network" which call Easy BCQ. We define new policy #M by drawing M samples from B and then executing the one with the highest value according to Q². Explicitly:
M t (als) 1aa arg max (s, az) aj m--1(-1s), is aj
(2)
Regularized policy updates. Another common idea proposed in the literature is to regularize towards the behavior policy [Wu et al., 2019 Jaques et al. 2019 Kumar et al. 2019). For general divergence we can define an algorithm that maximizes regularized objective:
#
arg max Σ
1QTH-1 a)) aD (..((((1ss)
(3)
a~a
comprehensive review of different variants of this method can be found in Wu et al. (2019 which does not find dramatic differences across regularization techniques. In practice, will use reverse To compute the reverse KL, we draw samples from π T-((s:)
KL divergence, i.e. K (=< and use the density estimate 3 to compute the divergence. Intuitively, this regularization forces n to remain within the support of 8 rather than incentivizing to cover 3.

---

Variants of imitation learning.
Another idea, proposed by [Wang et al. 2018 Siegel et al. 2020
Wang al. 2020b Chen al. 2020 modify an imitation learning algorithm either by filtering
or weighting the observed actions to incentivize policy improvement. The weighted version that we implement uses exponentiated advantage estimates to weight the observed actions:
# arg max Σ exp(((((*~11s,))) (s:))) log (aals:)).
(4)
With these definitions, now move to testing various combinations of algorithmic template (one-step, multi-step, or iterative) and improvement operator (Easy BCQ, reverse KL regularization, exponentially weighted imitation).
Benchmark Results
Our main empirical finding is that one step of policy improvement is sufficient to beat state of the art results much of the D4RL benchmark suite (Fu al. 2020]. This is striking since prior work focuses on iteratively estimating the Q function the current policy iterate, but we only use one step derived from Q Results are shown in Table | Full experimental details are in Appendix C and code can be found at httpp:://githbb com hwtp///wdddbaandffoooommo www.ennnnnnnnnneeeeeeaoeee onestep-rl
Table 1:
Results of one-step algorithms the D4RL benchmark. The first column gives the best results across several iterative algorithms considered in Fu et al. [2020). Each algorithm is tuned over 6 values of their respective hyperparameter. We report the mean and standard error over 10 seeds of the training process and using 100 evaluation episodes per seed. We bold the best result on each dataset and blue any result where one-step algorithm beat the best reported iterative result from Fu et al.] (2020). We use m for medium, m-e for medium-expert, m-re for medium-replay, for random, and for cloned.
Iterative
One-step
Fu al. 2020
BC
Easy BCQ
Rev. KL Reg
Exp. Weight
halfcheetah-m walker2d-m
46.3
42.1 = 0.1
52.6 + 0.1
55.6 + 0.2
48.6 = 0.0
81.1
70.2 = 1.3
86.9 + 0.4
85.6 + 0.4
80.3 = 1.1
hopper-m
58.8
49.8 = 0.6
69.7 +
83.3 + 1.4
56.7 = 0.8
halfcheetah-m--e
64.7
-
77.0 0.9
93.5 0.1
91.7
walker2d-m-e
111.0
93.6 - 5.6
111.8 1 0.2
110,9 0.1
112,9 1 0,2
hopper-m-e
111.9
48.1 = 1.5
81.4 1.9
102.1 1.3
83.1 = 7.0
halfcheetah-mm-re
47.7
34.9 = 0.3
38.4 0.3
42.4 + 0.1
38.6 + 0.5
walker2d-m-re
26.7
23.9 = 1.6
6..4
71.6 3.1
49.3 3.5
hopper-m-re
48.6
21.2 1.3
77.3 2.7
71.0 +.1
94.1 + 2.4
halfcheetah-r
35.4
2.2 0.0
5.1
6.9 1.0
3,7 + 0.2
walker2d-r
7.3
0.7
4.2
6.1
5,2 0.2
hopper-r
12.2
2.6 0.4
6.1
7.8
5.6 0.6
pen-c
56.9
39.3222
67.0 1.1
1.9
54.7 - 2.3
hammer-c
2.1
0.1
2.5
0.2
1.2 0.2
relocate-c
-0.1
0.0
0.3
0.1
0.1 0.0
door-c
0.4
0.0
0.4 0.2
0.0
0.1 0.1
As we can see the table, all the one-step algorithms usually outperform the best iterative algorithms tested by Iu et al. [2020] The one notable exception is the case of random data (especially on halfcheetah), where iterative algorithms have a clear advantage. We will discuss potential causes of this further in Section
To give more direct comparison that controls for any potential implementation details, we use our implementation reverse KI regularization to create multi-step and iterative algorithms. not using algorithmic modifications like Q ensembles, regularized Q values, or early stopping that have been used in prior work. But, our iterative algorithm recovers similar performance to prior regularized actor-critic approaches. These results shown in Table 2
are
5

---

Table 2: Results of reverse KL regularization on the D4RL benchmark across one-step, multi-step. and iterative algorithms. Again we run 6 hyperparameters and report the mean and standard error across 10 seeds using 100 evaluation episodes.
One-step
Multi-step
Iterative
halfcheetah-m walker2d-m
55.6 + 0.2
40.8 + 8.6
47.4 + 3.5
85.6 + 0.4
75.9 + 0.5
75.4 + 0.8
hopper-m
83.3 + 1,4
53.0 + 1.0
54.2 + 0.6
halfcheetah-me m-e
93.5 + 0.1
93.6 + 0.3
93.6 + 0.2
walker2d-m-4
110.9 0.1
76.3 + 15,9
108.2
hopper-me m-e
102.1 = 1.3
101.3
82.7 7.4
halfcheetah-r
6.9 1.0
13.7 1.7
16.3 + 1.6
walker2d-r
1 0.3
5.0 1.3 0.3
hopper-r
7.8 1 0.3
15.4 1 2.9
9.7 0.1
Put together, these results immediately suggest some guidance to the practitioner: it worthwhile to run the one-step algorithm as baseline before trying something more elaborate. The one-step algorithm is substantially simpler than prior work, but frequently achieves better performance.
What goes wrong for iterative algorithms?
The benchmark experiments show that one step of policy improvement often beats iterative and multi-step algorithms. In this section dive deeper understand why this happens. First, by examining the learning curves of each of the algorithms we note that iterative algorithms require stronger regularization to avoid instability. Then identify two causes of this instability: distribution shift and iterative error exploitation.
Distribution shift causes evaluation error by reducing the effective sample size in the fixed dataset for evaluating the current policy and has been extensively considered in prior work discussed below. Iterative error exploitation occurs when we repeatedly optimize policies against our Q estimates and exploit their errors. This introduces bias towards overestimation each step (much like the training error in supervised learning is biased to be lower than the test error). Moreover, by iteratively re-using the data and using prior Q estimates warmstart training at each step, the errors from one step amplified at the next. This type of error is particular to multi-step and iterative algorithms.
6.1 Learning curves and hyperparameter sensitivity
To begin understand why iterative and multi-step algorithms can fail instructive to look the learning curves. As shown in Figure Z we often observe that the iterative algorithm will begin to learn and then crash. Regularization can help to prevent this crash since strong enough regularization towards the behavior policy ensures that the evaluation is nearly on-policy.
Mobistep stop
One
stop
Fiuil horformnce
V
12
12
18
10
12
10
62300
22800 43200 53000
B9CC 102000
Figure 2: Learning curves and final performance on halfcheetah-mmciumm across different algorithms and regularization hyperparameters (all using the reverse KL regularized improvement operator). Error bars show min and max over 3 seeds. Similar figures for other datasets from D4RL can be found in Appendix

---

In contrast, the one-step algorithm is more robust to the regularization hyperparameter. The rightmost panel of the figure shows this clearly. While iterative and multi-step algorithms can have their performance degrade very rapidly with the wrong setting of the hyperparameter, the one-step approach more stable. Moreover, we usually find that the optimal setting of the regularization hyperparameter is lower for the one-step algorithm than the iterative multi-step approaches.
6.2 Distribution shift
Any algorithm that relies on off-policy evaluation will struggle with distribution shift in the evaluation step. Trying to evaluate policy that is substantially different from the behavior reduces the effective sample size and increases the variance of the estimates. Explicitly, by distribution shift we mean shift between the behavior distribution (the distribution state-action pairs in the dataset) and the evaluation distribution (the distribution that would induced by the policy π we want to evaluate).
Prior work. There is substantial body of prior theoretical work that suggests that off-policy evaluation can be difficult and this difficulty scales with some measure of distribution shift. Wang et al. 2020al [Amortila et al. 2020)_ Zanette [202]] give exponential (in horizon) lower bounds on sample complexity in the linear setting even with good feature representations that can represent the desired Q function and assuming good data coverage. Upper bounds generally require very strong assumptions on both the representation and limits on the distribution shift [Wang et al. [202]] Duan al. 2020 Chen and Jiang 2019]. Moreover, the assumed bounds on distribution shift can be exponential in horizon in the worst case. On the empirical side, Waan et al. [2021] demonstrates issues with distribution shift when learning from pre-trained features and provides nice discussion of why distribution shift causes error amplification. Fujimoto et al. [2018a) 80]8a raises a similar issue under the name "extrapolation error". Regularization and constraints are meant reduce issues stemming from distribution shift, but also reduce the potential for improvement over the behavior.
Empirical evidence. Both the multi-step and iterative algorithms in our experiments rely on off- policy evaluation as a key subroutine. We examine how easy it is to evaluate the policies encountered along the learning trajectory. To control for issues of iterative error exploitation (discussed the next subsection), we train Q estimators from scratch on heldout evaluation dataset sampled from the behavior policy. We then evaluate these trained Q function on rollouts from 1000 datapoints sampled from the replay buffer. Results are shown in Figure 3
The results show a correlation betweed KL and MSE. Moreover, we see that the MSE generally increases over training. One way mitigate this, as seen in the figure, use large value of We just cannot take a very large step before running into problems with distribution shift. But, when we take such a small step, the information from the on-policy Q3 is about as useful as the newly estimated QT. This is seen, for example, in Figure where get very similar performance across algorithms at high levels of regularization.
39'
30'
1."
22
20
32
10'
10000
Figure Results of running the iterative algorithm on halfcheetah-mediun Each checkpointed policy evaluated by Q function trained from scratch on heldout data. MSE refers to Es.a~ [((T (s, QT a] and KL refers to Es~ol [ [ ([-+s)]]] Left: 90 policies taken from various points in training with various hyperaparmeters and random seeds. Center: MSE learning curves. Right: KL. learning curves. Error bars show min and max over 3 random seeds.
7

---

6.3 Iterative error exploitation
The previous subsection identifies how any algorithm that uses off- oiflpplicy evaluation is fundamentally limited by distribution shift, even we were given fresh data and trained Q functions from scratch at every iteration. But, in practice, iterative algorithms repeatedly iterate between optimizing policies against estimated Q functions and re-estimating the Q functions using the same data and using the Q function from the previous step to warm-start the re-estimation. This induces dependence between steps that causes a problem that we call iterative error exploitation.
Intuition about the problem. In short, iterative error exploitation happens because т, tends to choose overestimated actions in the policy improvement step, and then this overestimation propagates via dynamic programming in the policy evaluation step. To illustrate this issue more formally, consider the following: at each s, we suffer some Bellman error eā(s,a) based on fixed dataset collected by 3. Formally,
= r(s,a)+
E 18.0
-
(5)
Intuitively, 53 will be larger at state-actions with less coverage in the dataset collected by B. Note that e/ can absorb all error whether it is caused by the finite sample size or function approximation error.
All that is needed to cause iterative error exploitation is that the eg are highly correlated across different æ,, but for simplicity, we will assume that eg is the same for all policies π estimated from our fixed offline dataset and instead write Eg. Now that the errors do not depend on the policy we can treat the errors as auxiliary rewards that obscure the true rewards and see that
(7((s)))
a)) ==
E
Σ (e)
(6)
This assumption is somewhat reasonable since we expect the error to primarily depend on the data. And, when the prior Q function is used to warm-start the current one (as is generally the case in practice), the approximation errors are automatically passed between steps.
Now we can explain the problem. Recall that under our assumption the Eg are fixed once we have a dataset and likely to have larger magnitude the further we go from the support of the dataset. So, with each step म; is able to better maximize Eg, thus moving further from /3 and increasing the magnitude of Q7 relative to Q*.. Even though Q* may provide better signal than Qo it can easily be drowned out by Q In contrast, Q has small magnitude, so the one-step algorithm is robust to errors
An example. Now we consider simple gridworld example to illustrate iterative error exploitation. This example fits exactly into the setup outlined above since all errors are due to reward estimation so the Eg is indeed constant over all æ. The gridworld we consider has one deterministic good state with reward 1 and many stochastic bad states that have rewards distributed as (-.5., 1). We collect a dataset of 100 trajectories, each of length 100. One run of the multi- steli-step offline regularized policy iteration algorithm is illustrated in Figure 4
In the example we see that one step often outperforms multiple steps of improvement, Intuitively. when there so many noisy states, it is likely that few of them will be overestimated. Since the data is re-used for each step. these overestimations persist and propagate across the state space due to iterative error exploitation. This property of having many bad, but poorly estimated states likely also exists in the high-dimensional control problems encountered in the benchmark where there are many ways for the robots to fall down that are not observed in the data for non-random behavior. Moreover, both settings have larger errors in areas where we have less data. So even though the errors in the gridworld are caused by noise in the rewards, while errors in D4RL are caused by function approximation, we think this is a useful mental model of the problem.
'We should note that iterative error exploitation is similar to the overestimation addressed by double learning (Van Hasselt al. 2016 Fujimoto et al., 20018b) but distinct. Since we are in the offline setting, the errors due to our finite dataset can be iteratively exploited more and more, while in the online setting considered by double Q learning, fresh data prevents this issue. We are also considering an algorithm based on policy iteration rather than value iteration.
8

---

433
Figure 4: An illustration of multi-step offline regularized policy iteration. The leftmost panel in each row shows the true reward (top) or error EB (bottom). Then each subsequent panel plots n, (with arrow size proportional to (a(s)) over either Q (top) or Q (bottom), averaged over actions at each state. The one-step policy (TT) has the highest value. The behavior policy here a mixture of optimal n¹ and uniform with coefficient 0.2 so that 33 0.2 x² + 0.8 We set 0.1 the regularization parameter for reverse KL regularization.
Training
150
150
Step
75-
75
50-
50
-500
500
1000
1500
-500
500
1000
1500
Ovansdiimation
Onrrsdmmttio
Figure 5: Histograms of overestimation error (Q (= a)) on weafihhhetah-mediii with the iterative algorithm. Left: errors from the training Q funccionn Right: errors from an independently trained Q function.
Empirical evidence. In practice we cannot easily visualize the progression of errors. However, the dependence between steps still arises as overestimation of the Q values. We can track the overestimation of the Q values over training optimizing against our dependent Q estimators. As a control we can also train Q estimators from scratch on independently sampled evaluation data. These independently trained functions do have the same overestimation bias even though the squared error does tend to increase as the policy further from the behavior (as seen in Figure 3) Explicitly, track 1000 state, action pairs from the replay buffer over training. For each checkpointed policy we perform 3 rollouts at each state get an estimate of the true Q value and compare this the estimated Q value. Results are shown in Figure 5
way measure how much bias is being induced by
When are multiple steps useful?
So far we have focused on why the one-step algorithm often works better than the multi-step and iterative algorithms. However, we do not want give the impression that one-step is always better. Indeed, our own experiments in Section 5 showow a clear advantage for the multi-step and iterative approaches when have randomly collected data. While we cannot offer a precise delineation of when one-step will outperform multi-step, in this section we offer some intuition as to when we can expect to see benefits from multiple steps of policy improvement.
As seen in Section 6, multi-step and iterative algorithms have problems when they propagate estima- tion errors. This is especially problematic noisy and/or high dimensional environments. While the multi-step algorithms propagate this noise more widely than the one-step algorithm, they also

---

30
She
Metun
Rancom
0.0
02
0.6
0.1
Medim
Figure 6: Performance of all three algorithms with reverse KL regularization across mixtures between salfcheetah-randoon and halffheetthhmmedum. Error bars indicate min and max over seeds.
propagate the signal. So, when we have sufficient coverage reduce the magnitude of the noise, this increased propagation of signal can be beneficial. The D4RL experiments suggest that we are usually on the side of the tradcoff where the errors are large enough make one-step preferable.
In Appendix A we illustrate simple gridworld example where slight modification of the behavior policy from
makess multi-step dramatically outperform one-step. This modified behavior policy (1) has better coverage of the noisy states (which reduces error, helping multi-step), and (2) does a worse job propagating the reward from the good state (hurting one-step).
We can also test empirically how the behavior policy effects the tradeoff between error and signal propagation. To do this we construct simple experiment where we mix data from the random behavior policy with data from the medium behavior policy. Explicitly we construct dataset D out of the datasets D, for random and D. for medium such that each trajectory in D comes from the medium dataset with probability Pm- So for Pm = we have the random dataset and Pm = 1 we have the medium dataset, and in between
have various mixtures. Results are shown in Figure 6. It takes surprisingly little data from the medium policy for one-step to outperform the iterative algorithm.
Discussion, limitations, and future work
This paper presents the surprising effectiveness of simple one-step baseline for offline RL. We examine the failure modes of iterative algorithms and the conditions where we might expect them to outperform the simple one-step baseline. This provides guidance to a practitioner that the simple one-step baseline is good place to start when approaching an offline RL problem.
But, we leave many questions unanswered. One main limitation is that we lack a clear theoretical characterization of which environments and behaviors can guarantee that one-step outperforms multi-step visa versa. Such results will likely require strong assumptions, but could provide useful insight. We don't expect this to be easy as requires understanding policy iteration which has been notoriously difficult to analyze, often converging much faster than the theory would suggest [Sutton and Barto, 2018 Agarwal et al. 2019). Another limitation is that while only using one step perhaps the simplest way to avoid the problems of off-policy evaluation, there are possibly other more elaborate algorithmic solutions that we did not consider here. However, our strong empirical results suggest that the one-step algorithm is at least strong baseline.
Broader impact. Our paper studies a simple and effective baseline approach to the offline RL problem. The effectiveness of this baseline raises some serious questions about the utility of prior work proposing substantially more complicated methods. By making this observation of prior shortcomings, our paper has the potential to encourage researchers derive new and better methods for offline RL. This has many potential impacts on fields as diverse as robotics and healthcare where better offline decision making can lead better real-world performance. As always, we note that machine learning improvements come in the form of "building machines to do X better". For sufficiently malicious or ill-informed choice X, almost any progress in machine learning might indirectly lead to negative outcome, and our work not excluded from that.
10

---

Acknowledgements
This work is partially supported by the Alfred P. Sloan Foundation, NSF RI-1816755, NSF CAREER CIF 1845360, NSF CHS-190109) 1901091, Samsung Electronics, and the Institute for Advanced Study. DB supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program.
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
Alekh Agarwal, Nan Jiang, and S. Kakade. Reinforcement learning: Theory and algorithms. 2019.
P. Amortila, Nan Jiang, and Tengyang Xie. variant of the wang-foster-kakade lower bound for the discounted setting. ArXiv, abs/2011. 101075, 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1660.011400, 2016. URL httpp:/arxxi wwaiiiivorrg/e org/ abs/
Jacob Buckman, Carles Gelada, and Mare G. Bellemare. The importance of pessimism in fixed-dataset policy optimization, 2020.
John Burkhardt. The truncated normal distribution, 2014.
Jinglin Chen and Nan Jiang. Information-theoree considerations batch reinforcement learning. In Proceedings of the 36th International Conference Machine Learning. PMLR, 2019.
Xinyue Chen, Zijian Zhou, Zheng Wang. Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pages 2701-2709, PMLR, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint /812. 2018a.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.00920 2018b.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pincau. Benchmarking batch 2019.
deep reinforcement learning algorithms. arXiv preprint
Wonjoon Goo and Scott Niekum. You only evaluate once simple baseline algorithm for offline In Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2020.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006. 2020.
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang. Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. arXiv preprint 103 2021.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 2019.

---

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In ICML, volume 2. pages 267-274, 2002.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2 08050, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy
q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 2tti:20000007779, 2020.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. International Conference on Machine Learning, pages 3652-3661. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.0/66 0/643,, 2020.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 00.55099099777 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- aff-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054-1062, 2016.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint 19Xiv:19/2.0000 2019.
Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. 2019.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weiggte regressionn Simple and scalable off-policy reinforcement learning, arXiv preprint
2019.
Martin I. Puterman and Moon Chirl Shin. Modified policy iteration algorithms for discounted markov decision problems. Management Science, 24(11): 12731777333 1978.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint urXiv: 10087, 2017.
Bruno Scherrer, Victor Gabillon, Mohammad Ghavamzadeh, and Matthieu Geist. Approximate modified policy iteration. arXiv preprint 1rXiv:1205.305:. 2012.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
Noah Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
12

---

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study off- policy evaluation for reinforcement learning. arXiv preprint arXiv:191..0685 2019.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, al. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems, pages 6288-622 0288-6297, 2018.
Ruosong Wang, Dean P. Foster, and Sham M. Kakade, What are the statistical limits of offline rl with linear function approximation?, 2020a.
Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham M Kakade. Instabilities of offline rl with pre-trained neural representation. arXiv preprint arXiv:2103.04947, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh $ Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020b.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019.
Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 8 and Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 8.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] In all relevant figures.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix [
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Data from Fu et al.
2020).
(b) Did you mention the license of the assets? [Yes] The license is Apache 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
Code in supplement.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A] Data is simulated.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Data is simulated.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "cQjBl68Ll-5", "reviewer": "Reviewer_T3oZ", "summary": "The paper examines offline RL algorithms, and finds that iterative algorithms (ones that alternate gradient updates between actor and critic) tends to perform worse than one-step algorithms (ones that fit Q to convergence then pi to convergence, with exactly 1 epoch of training and no updates to Q against the updated pi). The one-step results appear better across many of the benchmarks in D4RL.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper examines a number of recent methods for offline RL - learning policies through reinforcement learning from static datasets. It draws a distinction between three classes of methods:\n\n* Single step - train a Q-function estimate to convergence to evaluate a policy pi, then train the policy to converge against that Q\n* Multi-step - the same as above, except with a few iterations (N = 5)\n* Iterative - instead of training to convergence, do alternating gradient steps between the Q-function update and policy update. Most baselines they compare against (the ones evaluated in the D4RL paper) fall under this category.\n\nFor every class of method, it is assumed we used imitation learning to learn an initial behavior estimate $\\hat{\\beta}$ from the dataset (which I believe is only used in the policy update step for the experiments that regularize with $KL(\\pi || \\hat{\\beta})$.\n\nThe authors find that their single step experiments outperform the reported numbers from the iterative algorithms in the D4RL paper in most setups, with the exception of the setting where all data is from a random policy. They additionally find that iterative versions of their one-step algorithm tend to perform slightly worse. Based on this, they conclude that the problems of iterative algorithms tend to outweigh their advantages in an offline RL setting. Specifically, they argue that in iterative settings, Q-evaluations become noisier because distribution shift reduces the effective number of samples available to estimate $Q$ over the state-action distribution that $\\pi$ generates, and this error compounds iteratively as $\\pi$ and $Q$ update against each other. Neither of these claims is particularly interesting in my opinion (they are both fairly well known), so to me the main significance is that empirically it is better to avoid these iterative updates entirely, rather than try to regularize them with conservative objectives. They conclude by suggesting that iterative algorithms perform better on random data because the increased state coverage allows for better evaluation and has less clear reward propagation from good states, requiring more steps of iterating.\n\nI overall think this is an interesting result, and agree with the authors that these one-step methods are simpler to implement and appear to be more robust to hyperparameter settings. However, the claim that it outperforms iterative methods in general seems strong. For example, the authors use 2 layers of 1024 hidden units each + a truncated normal representation, whereas the numbers reported by the D4RL paper use a tanh of the normal and I believe used layers of 256 hidden units instead. To me it seems possible that the difference traces to policy representation rather than the algorithm used. (Although the iterative results in Table 2 do suggest that multi-step is harmful.) I would be interested to see if these results still hold when the architecture is controlled to be closer to the settings used in the D4RL baselines.\n\nI also have some disagreements with the abstract's wording, specifically that the one-step baselines perform better without using many of the tricks from the iterative methods. The best one-step results use essentially the same update rules as the iterative methods (reverse KL regularization, weighting by exp(A), etc.), the main difference is just in the ordering of updates (to convergence for Q / pi rather than alternating).\n\nBut in general I think this paper is good.\n\nNote: for the claim in Appendix B based on Achiam 2017, there is no need to argue that KL can be substituted for TV to get overall soundness, as later in the paper there is a citation for a 1983 lemma that bounds TV distance by sqrt(KL) if the authors want a bound using KL instead.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "b3uq8EZXuZz", "reviewer": "Reviewer_oZUe", "summary": "The authors study the notion of iterative policy improvement in offline reinforcement learning, notably finding that a single policy improvement step can outperform standard iterative benchmarks on the D4RL benchmark. To provide more insight on this, the authors also compare to an intermediate version, “multi-step”, and analyze the performance of the three variants (one-step, multi-step, and iterative) with different regularization hyperparameters. They also show results for the MSE of the estimated Q function, as well as illustrate multi-step overestimation on Gridworld environments. Finally, the authors discuss when multi-step algorithms are better, namely when noise signal is low and more propagation of signal can help.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1. Thorough analysis of various algorithms, analyzing both raw performance and value estimation, as well as when multi-step can improve on one-step algorithms\n2. Proposes a simple baseline that should be beneficial for the offline RL community\n3. Paper is written well and is clear\n\nCons\n1. I think a possible undiscussed issue that could limit the applicability of one-step/\"without off-policy estimation\" approaches is that frequently we might think of the offline dataset as coming from several sources/policies, in which case it makes less sense to say the initial Q function estimate is on-policy.\n2. For Figure 6, I think using a different environment (such as Hopper or Walker) might be more illustrative, as many algorithms/approaches (that don't fail) tend to get a fairly similar range of performances on the HalfCheetah environment\n\nThis paper presents an interesting and important finding, that one-step policy improvement can yield state-of-the-art results on the D4RL benchmark, and follows this result with a nice analysis and discussion of single vs multi-step/iterative approaches, using a few different update rules, to provide more intuition on the benefits and drawbacks of single-step approaches. The analysis looks technically sound and the writing is well organized. In particular, I think the subsequent analyses in Sections 6 and 7 are well done. I think this finding will be significant and important for offline RL practitioners to consider.\n\nAs stated above in Cons, I think one minor idea that is not discussed in the paper is the case of multiple behavior policies/sources of data, which seems to hold much of the promise in offline RL (and supervised learning inspirations). One thing to note is that both the \"Medium-Replay\" and \"Medium-Expert\" experiments, as well as Figure 6, show results on such datasets, so it does not seem like a major issue, but rather more of a conceptual one. In this case, it feels unavoidable to perform off-policy evaluation if these different behaviors are truly distinct, and the authors do discuss this a bit in the experiments discussing the utility of a multi-step approach when the data is very poor. Regardless, even if the one-step approach doesn't scale beyond D4RL, I also think the multi-step approach -- or at least the notion that fewer iterative policy/value updates could serve as regularization -- will be very useful for the community.\n\nMiscellaneous (no bearing on score)\n1. I think it would improve clarity in Table 2 if the labels for Multi-Step and Iterative were Multi-Step (K=5) and Iterative (K=10^5)\n\n=====\n\nUpdates\n\nGiven the responses to the other reviewers (new experiments and discussion), I have increased my score from 7 to 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "OpLMpx-8tqK", "reviewer": "Reviewer_BBLZ", "summary": "This paper applies a simple baseline to the offline-RL paradigm. Rather than iterating through steps of policy evaluation and improvement as done by most current offline-RL algorithms, in this work the authors investigate the performance of one-step of constrained policy improvement on the D4RL benchmark. This simple baseline is shown to perform considerably well when compared to existing methods. The paper then claims that the reason for this is likely due to the deleterious effects of off-policy evaluation. A simple experiment of different algorithms shows that strong regularization against the behavior policy during learning can be crucial in the offline setting. The authors then analyze the potential pitfalls of off policy evaluation through simple analysis on an existing task and a convincing gridworld experiment.\nFinally the paper tries to shed some intuition as to when multi-step approaches may perform better than the one-step methods. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is quite a good paper. The idea being studied is simple, the experiments and analysis are sound and well described and the writing is clear, effective and persuasive. While it has been known for some time that off policy evaluation is notoriously hard to get right, the simple one-step policy improvement baseline presented in the paper presents convincing insight into how bad this problem is in practice. I also particularly enjoyed the gridworld experiments to study the effect of iterative error exploitation in Figure 4 and its counterpart showing when multi-step algorithms perform better in the Appendix. I appreciate that the paper does not make any grandiose claims and assumptions - in my opinion, the claims are modest and well validated. \n\nMy only suggestion to improve the paper would be to potentially have some experiments on another domain, like RL Unplugged, that has been studied in some other works. Although my intuition is that these experiments will show a similar result, having more empirical evidence can only be helpful ultimately.\n\nWhile there isn’t much I would change in this paper there are a few things that could help improve it overall I think. One thing that could be improved is in the description of the one-step algorithm. I was initially quite confused about how Beta was being trained until I looked at the Appendix. I understand that there is usually a tradeoff between what gets included in the main text given space restrictions but I think this is an important enough point to be highlighted more. \n\nApart from that I have a few minor points of correction which did not significantly impact my score:\n 1. Line 66 it should be ‘The’ instead of ‘he’ \n 2. Line 68 should be 'modifications' instead of modified\n 3. Line 89 should be Parameterizations (m missing)\n 4. Another minor point is to include [1] in the related work section. The intuition given in that work is different so it would be interesting to discuss it through the lens of the narrative described in this paper. \n\nReferences:\n[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint\narXiv:2010.13611, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the limitations were adequately discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "F-iofoPNyp_", "reviewer": "Reviewer_vwkm", "summary": "The paper investigates the well-known 1-step heuristic for Q-function based offline RL methods.\nThe method is evaluated for a model-free, Q-function based offline RL method on various benchmarks and the overall good performance of the approach is shown.\nThe paper differs from results published in [1] mainly by the benchmarks used and the analysis of the reasons for the good performance. \n\n[1] Gulcehre et al, Addressing Extrapolation Error in Deep Offline Reinforcement Learning, preprint 2021.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\\\nThe paper is nicely presented, it is very well written and well structured.\nThe topic is interesting, in particular, the research can provide a piece to the puzzle to better understand the causes of the problems of iterated Q-function based, model-free, offline RL method causes a contribution.\n\nPreliminary remark: \\\nThe weaknesses of the paper are easily remedied in my opinion, making it a valuable contribution.\n\nWeaknesses:\\\nIt is not made sufficiently clear in the text that the one-step heuristic has been studied already several times (e.g., [1], [2]) and this is not a novelty of this paper.\nIt is ignored in the text that the previous investigations of the one-step heuristic [1], [2] have also reported the good results that can be obtained with it. It should be noted that the studies in [1] were performed with similar algorithms. Since [1] is a preprint, the results presented there must be considered preliminary, but this does not affect the fact that the novelty value of the present paper can only be seen in differences to [1].\n\nIn measuring performance, only three repetitions (three random seeds) are used, without any justification for this undesirably small number.\nStandard deviations are calculated using only three values (three random seeds) and this procedure, which does not meet scientific standards, is not commented on.\n\nWhen these weaknesses are corrected, it is an interesting, very well written paper. As it stands, it is unsuitable for publication.\n\nSpecific requirements for rework:\\\n•\tIt must be clear from the text that neither the use of the one-step heuristic in Q-function-based, (model-free) offline RL is new, nor the observation that this simple heuristic is performing well, is new. The formulation \"In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well.\" is mentioned here to illustrate the problem. This has been shown before and it is therefore questionable whether the good performance is still surprising.\n\n•\tIt must be mentioned in the text that in [1] and [2] it has already be shown that \"simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well.\"\n\n•\tThe contribution of the present paper needs to be justified based on the differences to [1] and [2].\n\n•\tThe presentation of the results in Tables 1 and 2 needs to be modified. This should involve at least 10 trials per reported mean.\nWhen using the expression A +/- B to report the measured value A (here the mean of the performance) together with the uncertainty of the measurement, B should not be called standard deviation, but uncertainty (or error). \n Either calculate B as the standard error and call it so, or use an estimate of uncertainty for B and call it like that.\n\nIt would be simplest and most advantageous to perform at least 10 repetitions of the experiments, calculate B as standard error, and designate it as standard error in the text.\n\nIf this is not possible because of the cost of the calculations, then do at least 5 iterations and write a comment, why more has not been possible.\n\nIf even 5 is impossible, then either the uncertainty must be omitted (because determining the uncertainty on the basis of only three experiments can lead to wrong results) and the significance of the results must be demonstrated otherwise, e.g. with a rank test, or the uncertainty must be estimated appropriately (this is the responsibility and authority of the experimenter who performs the measurements). This could be done, unless I am missing something, e.g. by using the experiments of all 6 hyperparameters to determine the uncertainty. E.g. for each hyperparameter the standard error is calculated using the three trials. Since this value is very uncertain in each case but is also unbiased, the mean value of the 6 standard errors is calculated over all 6 hyperparameters and used for the measurements given in Tables 1 and 2. This procedure implicitly makes the assumption that the variability of the measurements is the same for all hyperparameters studied, which is certainly not entirely true. However, it is to be expected that the variability for the respective sub-optimal hyperparameters is larger, not smaller, than for the best hyperparameter, so this approach seems justified because it overestimates rather than underestimates the uncertainty.\nThe particular choice must be justified in the text (or caption).\n\nFurther comments:\\\nPage 1, line 35: \"the the\" -> \"the\"\\\nPlease make sure that in the published version the capitalization in the bibliography is correct, e.g.\nwang-foster-kakade, fisher\n\n[1] Gulcehre et al., Addressing Extrapolation Error in Deep Offline Reinforcement Learning, preprint 2021\\\n[2] K. Nadjahi, R. Laroche, R. Tachet des Combes. Safe Policy Improvement with Soft Baseline Bootstrapping. Proceedings of the 2019 European Conference on Machine  Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD). 2019.\n\n**Update after feedback** \\\nSince the authors now have 10 repetitions and want to provide the missing references I am clearly in favor of accepting the paper and increased the score to 7.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In my opinion, sufficient.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are positive, and a consensus is reached for acceptance (spotlight) through rebuttal + internal discussions. While the method is very simple, the message is clear and the authors have done excellent job on concise and thorough writing and experimentation. As offline RL + D4RL benchmark are becoming mainstream, such work can likely guide the community to explore more impactful research directions.  
Decision: Accept (Spotlight)