Offline RL Without Off-Policy Evaluation

Abstract: Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.


Reviews and author responses:

The paper investigates the well-known 1-step heuristic for Q-function based offline RL methods.
The method is evaluated for a model-free, Q-function based offline RL method on various benchmarks and the overall good performance of the approach is shown.
The paper differs from results published in [1] mainly by the benchmarks used and the analysis of the reasons for the good performance. 

[1] Gulcehre et al, Addressing Extrapolation Error in Deep Offline Reinforcement Learning, preprint 2021.
  
Strengths:\
The paper is nicely presented, it is very well written and well structured.
The topic is interesting, in particular, the research can provide a piece to the puzzle to better understand the causes of the problems of iterated Q-function based, model-free, offline RL method causes a contribution.

Preliminary remark: \
The weaknesses of the paper are easily remedied in my opinion, making it a valuable contribution.

Weaknesses:\
It is not made sufficiently clear in the text that the one-step heuristic has been studied already several times (e.g., [1], [2]) and this is not a novelty of this paper.
It is ignored in the text that the previous investigations of the one-step heuristic [1], [2] have also reported the good results that can be obtained with it. It should be noted that the studies in [1] were performed with similar algorithms. Since [1] is a preprint, the results presented there must be considered preliminary, but this does not affect the fact that the novelty value of the present paper can only be seen in differences to [1].

In measuring performance, only three repetitions (three random seeds) are used, without any justification for this undesirably small number.
Standard deviations are calculated using only three values (three random seeds) and this procedure, which does not meet scientific standards, is not commented on.

When these weaknesses are corrected, it is an interesting, very well written paper. As it stands, it is unsuitable for publication.

Specific requirements for rework:\
•	It must be clear from the text that neither the use of the one-step heuristic in Q-function-based, (model-free) offline RL is new, nor the observation that this simple heuristic is performing well, is new. The formulation "In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well." is mentioned here to illustrate the problem. This has been shown before and it is therefore questionable whether the good performance is still surprising.

•	It must be mentioned in the text that in [1] and [2] it has already be shown that "simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well."

•	The contribution of the present paper needs to be justified based on the differences to [1] and [2].

•	The presentation of the results in Tables 1 and 2 needs to be modified. This should involve at least 10 trials per reported mean.
When using the expression A +/- B to report the measured value A (here the mean of the performance) together with the uncertainty of the measurement, B should not be called standard deviation, but uncertainty (or error). 
 Either calculate B as the standard error and call it so, or use an estimate of uncertainty for B and call it like that.

It would be simplest and most advantageous to perform at least 10 repetitions of the experiments, calculate B as standard error, and designate it as standard error in the text.

If this is not possible because of the cost of the calculations, then do at least 5 iterations and write a comment, why more has not been possible.

If even 5 is impossible, then either the uncertainty must be omitted (because determining the uncertainty on the basis of only three experiments can lead to wrong results) and the significance of the results must be demonstrated otherwise, e.g. with a rank test, or the uncertainty must be estimated appropriately (this is the responsibility and authority of the experimenter who performs the measurements). This could be done, unless I am missing something, e.g. by using the experiments of all 6 hyperparameters to determine the uncertainty. E.g. for each hyperparameter the standard error is calculated using the three trials. Since this value is very uncertain in each case but is also unbiased, the mean value of the 6 standard errors is calculated over all 6 hyperparameters and used for the measurements given in Tables 1 and 2. This procedure implicitly makes the assumption that the variability of the measurements is the same for all hyperparameters studied, which is certainly not entirely true. However, it is to be expected that the variability for the respective sub-optimal hyperparameters is larger, not smaller, than for the best hyperparameter, so this approach seems justified because it overestimates rather than underestimates the uncertainty.
The particular choice must be justified in the text (or caption).

Further comments:\
Page 1, line 35: "the the" -> "the"\
Please make sure that in the published version the capitalization in the bibliography is correct, e.g.
wang-foster-kakade, fisher

[1] Gulcehre et al., Addressing Extrapolation Error in Deep Offline Reinforcement Learning, preprint 2021\
[2] K. Nadjahi, R. Laroche, R. Tachet des Combes. Safe Policy Improvement with Soft Baseline Bootstrapping. Proceedings of the 2019 European Conference on Machine  Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD). 2019.

**Update after feedback** \
Since the authors now have 10 repetitions and want to provide the missing references I am clearly in favor of accepting the paper and increased the score to 7.

 In my opinion, sufficient.

The authors study the notion of iterative policy improvement in offline reinforcement learning, notably finding that a single policy improvement step can outperform standard iterative benchmarks on the D4RL benchmark. To provide more insight on this, the authors also compare to an intermediate version, “multi-step”, and analyze the performance of the three variants (one-step, multi-step, and iterative) with different regularization hyperparameters. They also show results for the MSE of the estimated Q function, as well as illustrate multi-step overestimation on Gridworld environments. Finally, the authors discuss when multi-step algorithms are better, namely when noise signal is low and more propagation of signal can help.  Pros
1. Thorough analysis of various algorithms, analyzing both raw performance and value estimation, as well as when multi-step can improve on one-step algorithms
2. Proposes a simple baseline that should be beneficial for the offline RL community
3. Paper is written well and is clear

Cons
1. I think a possible undiscussed issue that could limit the applicability of one-step/"without off-policy estimation" approaches is that frequently we might think of the offline dataset as coming from several sources/policies, in which case it makes less sense to say the initial Q function estimate is on-policy.
2. For Figure 6, I think using a different environment (such as Hopper or Walker) might be more illustrative, as many algorithms/approaches (that don't fail) tend to get a fairly similar range of performances on the HalfCheetah environment

This paper presents an interesting and important finding, that one-step policy improvement can yield state-of-the-art results on the D4RL benchmark, and follows this result with a nice analysis and discussion of single vs multi-step/iterative approaches, using a few different update rules, to provide more intuition on the benefits and drawbacks of single-step approaches. The analysis looks technically sound and the writing is well organized. In particular, I think the subsequent analyses in Sections 6 and 7 are well done. I think this finding will be significant and important for offline RL practitioners to consider.

As stated above in Cons, I think one minor idea that is not discussed in the paper is the case of multiple behavior policies/sources of data, which seems to hold much of the promise in offline RL (and supervised learning inspirations). One thing to note is that both the "Medium-Replay" and "Medium-Expert" experiments, as well as Figure 6, show results on such datasets, so it does not seem like a major issue, but rather more of a conceptual one. In this case, it feels unavoidable to perform off-policy evaluation if these different behaviors are truly distinct, and the authors do discuss this a bit in the experiments discussing the utility of a multi-step approach when the data is very poor. Regardless, even if the one-step approach doesn't scale beyond D4RL, I also think the multi-step approach -- or at least the notion that fewer iterative policy/value updates could serve as regularization -- will be very useful for the community.

Miscellaneous (no bearing on score)
1. I think it would improve clarity in Table 2 if the labels for Multi-Step and Iterative were Multi-Step (K=5) and Iterative (K=10^5)

=====

Updates

Given the responses to the other reviewers (new experiments and discussion), I have increased my score from 7 to 8. Yes, the authors adequately addressed the limitations and potential negative societal impact.

 Thank you for your response! Given the responses to the other reviewers, I have raised my score from 7 to 8.

(On datasets of multiple policies) I think conceptually, it is not an issue if you consider the implicit stochastic policy, however I think ideally you'd want to "mimic" the best quality data, so to speak.

(On figure 6) I do still think this would benefit the paper, as I think Cheetah is the most uninformative out of the standard environments in D4RL, but this is a minor point in general.

 I am glad that the authors are now basing their results on the statistics of 10 trials and want to give appropriate credit and reference to the previous work on the "one step policy improvement".

I liked the paper from the beginning. 

For me all the reasons for the objections have been resolved.


 We would like to thank the reviewer for their time and their positive feedback about the thoroughness, simplicity, and clarity of our paper as well as their opinion that our findings will be significant and important for practitioners.

**On datasets of multiple policies**: The reviewer raises an interesting and somewhat nuanced concern about what happens when the data is generated by multiple policies. As the reviewer notes, we do evaluate on several datasets that are constructed from these mixed behavior policies and get fairly effective results. To rationalize this theoretically, it is useful to note that even when the data may have been generated by multiple policies, there exists a single policy which could have generated the data (and which behavior cloning could learn). Specifically, the dataset defines some distribution over state-action pairs and this distribution can be generated by some *stochastic* policy. So, data generated by multiple policies might create more diverse data or induce other interesting properties in the dataset, but it is not fundamentally problematic. Also, on a semantic point, note that we are doing Q estimation with SARSA-style updates, so we are essentially evaluating this implicit data-defined policy (making the evaluation "on-policy" in the sense that we are updating based on actions in the data rather than sampling fictitious actions).

**On figure 6**: The reviewer would like to see Figure 6 replicated on different environments. We can definitely run the same experiment on Walker2d and can report the two plots alongside each other in the paper. We ran the experiments requested by the other reviewers first, but are working on running this now.

Please let us know if this comment addresses your concerns and if not what we can do to address any lingering issues.


 We would like to thank the reviewer for their time and their very positive feedback! We especially appreciate the positive evaluation of the simplicity, soundness, and clarity of our approach. 

We agree that more evidence can always be better, but due to time/compute constraints during the discussion period we won't be able to run these experiments now. We can potentially do this before the camera ready.

With respect to the description of the one-step algorithm, we agree that we could be more clear when introducing the algorithm and that it is worth the space to make sure that this important aspect of the paper is as clear as possible. We will add more explanation in Section 4, moving in some of the explanation from the appendix that the reviewer referenced.

Finally, we would also like to thank the reviewer for catching our typos with their close reading. We will make all those changes.

 We would like to thank the reviewer for their time and their positive comments regarding the clarity of our writing and their interest in the topic of the paper.

We would also like to thank the reviewer for their constructive criticism and seeming openness to revising their opinion of the paper if we make the right changes. To that end, we propose to make the following improvements to the paper to reflect the criticism from the reviewer: 

**On related work**: The reviewer is concerned that the idea of only using one step of policy improvement has been raised by [1] and [2] and that we did not make this clear enough in the paper. We agree that we should have been more clear about our positioning relative to prior work and will revise the paper to reflect the following:

- We agree that we are not the first ones to use a one-step algorithm and will make that more clear in the paper. Instead, our main contributions are (1) to provide rigorous empirical evidence on benchmark continuous control problems of the efficacy *and* robustness of the simple one step baseline approach, (2) to provide a deeper empirical and theoretical evaluation of this phenomena to help explain *why* the one step approach works, and (3) to raise the profile of this phenomena as it sheds light on issues with prior work. Specifically on (3), the clear and focused presentation in our paper will help to raise the salience of this important issue since a large amount of prior work has attempted to come up with substantially more complicated approaches that do not work significantly better than such a simple baseline.
- Specific comparison to [1]:
  - As the reviewer notes, [1] only considers discrete action environments while we consider continuous control, so the results are not directly comparable as the algorithms are substantially different. Namely, they do not have to learn policy networks since they can just act greedily with respect to the learned Q function.
  - We also want to clarify that we meant to cite [1] in Line 87, but accidentally cited a different paper by the same authors (Gulcehre et al.). As explained briefly there in the paper, while [1] considers a variant of its algorithm that uses only one step of improvement, it instead focuses on proposing and analyzing (1) a novel regularizer and (2) a novel parameterization. Instead we focus on the simplest possible version of the algorithm and analyze it directly. 
  - As the reviewer also briefly notes in the summary section, we provide a different analysis of the one step algorithm. We separate the issues of distribution shift and iterative error exploitation and provide both theory and empirical evidence related to both issues (Sections 6.2 and 6.3). We also provide a deeper comparison about when multiple steps outperforms one step based on the dataset (Section 7).  
- Specific comparison to [2]:
  - We were not aware of the use of the one step algorithm in [2] and thank the reviewer for bringing it to our attention. We will add a reference to the paper.
  - However, we still think that our contribution is substantially different. In [2], the use of the one step algorithm is nowhere near the main point of the paper. The paper considers a substantially different set of algorithms (based on SPIBB and designed primarily for tabular and discrete-action settings) and primarily focuses on theory related to those algorithms with only small-scale experiments on toy domains. In contrast, we conduct empirical analysis in continuous control problems and provide a much more detailed analysis of the one step approach. 

**On random seeds**: The reviewer is concerned that we only run 3 random seeds and report standard deviation rather than standard error.

- We are in the process of running more seeds and preliminary results can be found below. However, we would also like to briefly rationalize why we initially used 3 seeds. Briefly, 3 seeds is fairly standard in the offline RL literature (e.g. Fu et al. sited in the paper). We attribute this to the fact that the offline RL pipeline uses a fixed dataset so the variance (especially for the one step methods) is lower than in online RL.
- Here are the results for 10 seeds on all of the "medium" datasets with standard error instead of (note that the means are all near the numbers reported in the paper and the errors have all decreased by switching from standard deviation to standard error):

| Dataset               | BC               | Easy BCQ         | Rev. KL Reg.     | Exp. Weight      |
| --------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
| halfcheetah-medium-v2 | 42.1  $\pm$  0.1 | 52.6  $\pm$  0.1 | 55.5  $\pm$  0.2 | 48.6  $\pm$  0.1 |
| walker2d-medium-v2    | 70.2  $\pm$  1.3 | 87.0  $\pm$  0.3 | 85.6  $\pm$  0.4 | 78.6  $\pm$  2.0 |
| hopper-medium-v2      | 49.8  $\pm$  0.6 | 70.2  $\pm$  2.2 | 83.6  $\pm$  1.4 | 57.4  $\pm$  0.8 |

- It is somewhat computationally intensive, but we will run the rest of the datasets in the coming weeks and update the paper accordingly.

We will fix the minor issues with typos and the bibliography and thank the reviewer for pointing these out.

If this comment addresses your concerns, we ask you to please increase your score of the paper. If the comment is not sufficient, please let us know what else we can do to address any lingering issues.

 We would like to thank reviewer for their time, their generally positive assessment of our paper, and their constructive criticism. Here we offer a brief explanation of how we will improve the paper based on the feedback from the reviewer. 

**Clarifying our main argument**: We want to emphasize that we don't intend to claim that the one-step algorithm "outperforms iterative methods in general" as the reviewer says. We recognize that we were not as clear as we should have been in the abstract. We meant to claim that the one-step algorithm performs comparably to prior SOTA iterative approaches (sometimes better and sometimes worse) while being *much* simpler and somewhat more robust to hyperparameters. 

**On network architecture**: The reviewer raises some concern about the network architecture we use. We thank the reviewer for raising this subtle issue and offer two responses:
1. We re-ran the results for the "medium" datasets from Table 1 using the different architecture that the reviewer requests (we can finish running a full sweep with this setting for the camera-ready and add this table to the appendix for completeness). Results can be found in the following table, note that they are comparable to the reported results in the paper. So it seems the architecture is not a major factor in the results. With more time we can run this experiment across all datasets and add the results to the Appendix:

| Dataset               | BC               | Easy BCQ         | Rev. KL Reg.     | 
| --------------------- | ---------------- | ---------------- | ---------------- | 
| halfcheetah-medium-v2 | 42.6  $\pm$  0.2 | 52.1  $\pm$  0.2  | 53.9  $\pm$  0.3  |
| walker2d-medium-v2    | 69.9  $\pm$  4.5  | 85.6  $\pm$  1.4 | 81.0  $\pm$  0.9 |
| hopper-medium-v2      | 60.2  $\pm$  6.6 | 64.0  $\pm$  7.4 | 80.6  $\pm$  13.8  | 

2. We run the iterative algorithms in Table 2 using the same architecture as the one-step algorithms. Our iterative implementation gets comparable results to those reported in Fu et al. More importantly, this isolates the difference between one-step and iterative by using the same exact regularizer and controlling for all other implementation-level details because it uses our same code base. Thus it is a more direct comparison in our minds.  



In response to the minor note about Appendix B: Yes, we agree this was probably poorly worded and we should reference Pinsker's inequality there.

Please let us know if this comment addresses your concerns and if not what we can do to address any lingering issues. 

This paper applies a simple baseline to the offline-RL paradigm. Rather than iterating through steps of policy evaluation and improvement as done by most current offline-RL algorithms, in this work the authors investigate the performance of one-step of constrained policy improvement on the D4RL benchmark. This simple baseline is shown to perform considerably well when compared to existing methods. The paper then claims that the reason for this is likely due to the deleterious effects of off-policy evaluation. A simple experiment of different algorithms shows that strong regularization against the behavior policy during learning can be crucial in the offline setting. The authors then analyze the potential pitfalls of off policy evaluation through simple analysis on an existing task and a convincing gridworld experiment.
Finally the paper tries to shed some intuition as to when multi-step approaches may perform better than the one-step methods. 
  Overall I think this is quite a good paper. The idea being studied is simple, the experiments and analysis are sound and well described and the writing is clear, effective and persuasive. While it has been known for some time that off policy evaluation is notoriously hard to get right, the simple one-step policy improvement baseline presented in the paper presents convincing insight into how bad this problem is in practice. I also particularly enjoyed the gridworld experiments to study the effect of iterative error exploitation in Figure 4 and its counterpart showing when multi-step algorithms perform better in the Appendix. I appreciate that the paper does not make any grandiose claims and assumptions - in my opinion, the claims are modest and well validated. 

My only suggestion to improve the paper would be to potentially have some experiments on another domain, like RL Unplugged, that has been studied in some other works. Although my intuition is that these experiments will show a similar result, having more empirical evidence can only be helpful ultimately.

While there isn’t much I would change in this paper there are a few things that could help improve it overall I think. One thing that could be improved is in the description of the one-step algorithm. I was initially quite confused about how Beta was being trained until I looked at the Appendix. I understand that there is usually a tradeoff between what gets included in the main text given space restrictions but I think this is an important enough point to be highlighted more. 

Apart from that I have a few minor points of correction which did not significantly impact my score:
 1. Line 66 it should be ‘The’ instead of ‘he’ 
 2. Line 68 should be 'modifications' instead of modified
 3. Line 89 should be Parameterizations (m missing)
 4. Another minor point is to include [1] in the related work section. The intuition given in that work is different so it would be interesting to discuss it through the lens of the narrative described in this paper. 

References:
[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint
arXiv:2010.13611, 2020. Yes the limitations were adequately discussed. 

The paper examines offline RL algorithms, and finds that iterative algorithms (ones that alternate gradient updates between actor and critic) tends to perform worse than one-step algorithms (ones that fit Q to convergence then pi to convergence, with exactly 1 epoch of training and no updates to Q against the updated pi). The one-step results appear better across many of the benchmarks in D4RL.  The paper examines a number of recent methods for offline RL - learning policies through reinforcement learning from static datasets. It draws a distinction between three classes of methods:

* Single step - train a Q-function estimate to convergence to evaluate a policy pi, then train the policy to converge against that Q
* Multi-step - the same as above, except with a few iterations (N = 5)
* Iterative - instead of training to convergence, do alternating gradient steps between the Q-function update and policy update. Most baselines they compare against (the ones evaluated in the D4RL paper) fall under this category.

For every class of method, it is assumed we used imitation learning to learn an initial behavior estimate $\hat{\beta}$ from the dataset (which I believe is only used in the policy update step for the experiments that regularize with $KL(\pi || \hat{\beta})$.

The authors find that their single step experiments outperform the reported numbers from the iterative algorithms in the D4RL paper in most setups, with the exception of the setting where all data is from a random policy. They additionally find that iterative versions of their one-step algorithm tend to perform slightly worse. Based on this, they conclude that the problems of iterative algorithms tend to outweigh their advantages in an offline RL setting. Specifically, they argue that in iterative settings, Q-evaluations become noisier because distribution shift reduces the effective number of samples available to estimate $Q$ over the state-action distribution that $\pi$ generates, and this error compounds iteratively as $\pi$ and $Q$ update against each other. Neither of these claims is particularly interesting in my opinion (they are both fairly well known), so to me the main significance is that empirically it is better to avoid these iterative updates entirely, rather than try to regularize them with conservative objectives. They conclude by suggesting that iterative algorithms perform better on random data because the increased state coverage allows for better evaluation and has less clear reward propagation from good states, requiring more steps of iterating.

I overall think this is an interesting result, and agree with the authors that these one-step methods are simpler to implement and appear to be more robust to hyperparameter settings. However, the claim that it outperforms iterative methods in general seems strong. For example, the authors use 2 layers of 1024 hidden units each + a truncated normal representation, whereas the numbers reported by the D4RL paper use a tanh of the normal and I believe used layers of 256 hidden units instead. To me it seems possible that the difference traces to policy representation rather than the algorithm used. (Although the iterative results in Table 2 do suggest that multi-step is harmful.) I would be interested to see if these results still hold when the architecture is controlled to be closer to the settings used in the D4RL baselines.

I also have some disagreements with the abstract's wording, specifically that the one-step baselines perform better without using many of the tricks from the iterative methods. The best one-step results use essentially the same update rules as the iterative methods (reverse KL regularization, weighting by exp(A), etc.), the main difference is just in the ordering of updates (to convergence for Q / pi rather than alternating).

But in general I think this paper is good.

Note: for the claim in Appendix B based on Achiam 2017, there is no need to argue that KL can be substituted for TV to get overall soundness, as later in the paper there is a citation for a 1983 lemma that bounds TV distance by sqrt(KL) if the authors want a bound using KL instead. Addressed adequately.

Meta-review: Reviewers are positive, and a consensus is reached for acceptance (spotlight) through rebuttal + internal discussions. While the method is very simple, the message is clear and the authors have done excellent job on concise and thorough writing and experimentation. As offline RL + D4RL benchmark are becoming mainstream, such work can likely guide the community to explore more impactful research directions.  

Decision: accept