Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: # Offline RL Without Off-Policy Evaluation

## Summary

The paper examines an issue with existing offline RL methods and proposes a way to prevent the issue that leads to a strong family of baselines. The key idea is not to perform multiple steps of off-policy evaluation and improvement in offline RL.

In addition to providing evidence to the claim that one-step methods outperform multistep and iterative ones, the paper also provides explantion and a toy example that shed light on the issues faced by multistep and iterative offline RL methods. I believe these examples can be useful in the future as "unit tests" for new offline RL methods.

## For the response

Could you please comment on Gulcehre et al. (https://arxiv.org/pdf/2103.09575.pdf) in relation to your work? 

Please include a discussion of the broader societal impact of your work.
main_review: ## Strengths

The paper is well written and the claims and empirical evidence are adequate and clearly marked, which makes it easier to assess its contributions. The results are also interesting and add to the current understanding of offline RL.

The notion of proposing simple, improved baselines for offline RL is also interesting and helps increase applicability of offline RL methods.

## Weaknesses

A more detailed assessment of the related work can help clarify the novelty of the contributions.

The paper does not contain a discussion of the broader societal impact of its results.

## Correctness

The paper seems correct, though I think it could be more emphatic that the issue identified in Sec 6.3 is one potential issue with iterative and multi-step methods, but perhaps not the only one.

## Clarity

The paper is clearly written.


## Relation to prior work

In a recent paper, Gulcehre et al. (https://arxiv.org/pdf/2103.09575.pdf) seem to arrive at similar conclusions about one-step methods, but they additionally prescribe a regularization term to discourage the improved policy from selecting actions (at respective states) absent from the training data. The example you proposed for Figure 4 and Appendix A can be useful for validating the claim from Gulcehre et al. as to why they prescribe the ranking regularization.

I do not agree with the assessment in lines 86-95. YOEO fits the bill of a one-step method (and in fact the authors state that they want to propose an alternative to multi-step and iterative algorithms) and has some improvements over multistep and iterative methods. Moreover, the Niekum et al. make some of the core points this paper makes as well (see their Sec. 4), so I think it is important to acknowledge that. This will help the reader understand the contributions better in light of existing work.

## Reproducibility

I find there is enough information for reproducing the results.

## Additional Feedback

Thank you for your submission. I have a few miscellaneous comments.

OAMPI is similar to MPI, so maybe cite https://www.jstor.org/stable/2630487 and possibly also GPI in Sutton & Barto's book. I would also suggest calling the evaluation step $\mathcal{E}$. Some of this taxonomy is also used by Levine et al. (https://arxiv.org/pdf/2005.01643.pdf) so perhaps point to their work as well.

To me the Q "without a stop gradient" in eq. 2 is confusing, even though later you say that in practice a target network is used. Fitted Q evaluation/iteration (https://www.jmlr.org/papers/volume6/ernst05a/ernst05a.pdf) aims to solve an optimization problem with fixed next-state value estimates, not to optimize your eq. 2.

I find the interpretation and example in Section 6.3 enlightening. In fact, it made me wonder whether one can leverage methods such as generalized policy evaluation (https://www.pnas.org/content/pnas/117/48/30079.full.pdf) and generalized policy improvement as an extension to one-step baselines.

### Typos

l. 66 he learned -> the learned

l. 89 paraterizations -> parametrizations

l. 167 suite [Fu et al., 2020]

l. 323 effects -> affects (maybe)
limitations_and_societal_impact: The authors have not mentioned the limitations and potential negative societal impact of their work ([N/A in the checklist]).

Authors, please include in your text a balanced perspective of the broader impact of your work, including a discussion of its ethical aspects and future societal consequences. Also see https://neuripsconf.medium.com/introducing-the-neurips-2021-paper-checklist-3220d6df500b:
> we have extended the page limit from 8 pages to 9 pages (with an additional page allowed after acceptance) and encourage authors to prioritize using this space to address societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper discusses a new class of simplified, so called one step, algorithms for offline reinforcement learning, where the fitting of the Q function on offline data is followed by optimisation of policy with respect to the obtained Q-function estimated. The authors suggest that this class of methods is a surprisingly strong baseline, and that it also gives the insight into when and how the iterative methods failed.
main_review: The paper presents a very interesting finding that doing one step improvement of policy using on-policy Q-estimate is a surprisingly strong baseline of offline reinforcement learning(RL). Offline RL is a promising approach for the tasks when, despite having plenty of recorded data, there is no way of evaluation of the RL method in control setting. Additionally, it could be useful to improve sample efficiency. 

*Pros: *
Simplicity of the approach and new insights into the dynamics of learning for offline RL: the main point of the paper is that there exists a baseline  which is simpler than the existing methods and still performs competitively on *much of the D4RL benchmark suite* The proposed method could be of impact as it defines a new benchmark method for offline RL. 

*Cons: *
While the study has multiple limitations, such as that the analysis is mostly empirical and does not strictly answer when exactly this improvements happen, the reviewer has the opinion, that these limitations of the study are clearly defined in the paper, and the reported results are novel and valuable enough as they show the previously overlooked simpler benchmark method. 

Comments:

[1] *"Results of one-step algorithms on the D4RL benchmark. The first column gives the best results across several iterative algorithms considered in Fu et al. [2020].”*
Would it be possible to give results on separate state-of-the-art algorithms considered in Fu et al; also, is it possible to report confidence intervals for Fu et al’s experiments?

P.S. There was a problem opening this file in Acrobat Reader; so the reviewer needed to use an alternative PDF reader tool to open it

== Update: the score is updated reflecting the contribution of the paper in providing simpler and better performing baselines

  

limitations_and_societal_impact: The authors accurately discussed limitations of theoretical justification of the proposed method
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Popular RL (both online and offline) algorithms iterate between two stages: policy evaluation where a Q-function: $S \times A \rightarrow \mathbb{E} \sum r$ is estimated and policy improvement where the policy $\pi: S \rightarrow A$ which maximizes the rewards is constructed. Typically, the stages are interleaved through the training process. The paper evaluates and argues for the idea of using, in the case of offline RL, only a single step of policy improvement after training Q-function to convergence.
main_review: As arguments for the thesis of the paper, the authors:
1. Evaluate 3 different, typically iterative algorithms while performing only a single step of policy improvement on D4RL, observing improvements (over present in D4RL iterative algorithms) in around half of the tasks
2. In case of one of these algorithms (reverse KL regularization), implement 3 versions: one-step, multi-step, and iterative, observing best results in one-step version ~half of the times again
3. Provide some intuition (based on representing errors as auxiliary, hurtful rewards) about the reason for multi-step/iterative algorithms to do poorly in offline rl
4. Show examples of MDPs where multi-step is bound to fail and one where it's necessary to solve the MDP (in the appendix).
5. Argue for some theoretical guarantees of performance of single-step methods (in the appendix).
6. Show some empirical evidence relating overestimation with regularization in their algorithm.

I share the authors' intuition regarding the statement of the paper: doing multiple steps of improvement can be hurtful due to accumulation of errors and a single step can be a useful baseline/practice. Having said that, I cast a weak vote to reject the paper due to perceived lack of novelty.

11. The paper doesn't on its own introduce a novel algorithm, the (simple) idea it tests is literally just doing a single step of known improvement operators (as opposed to multi-step). As authors correctly point out, this notion was proposed last year by Goo and Niekum (GN) and Gulcehre et al. (CG) (nota bene I think the citation for CG should be ["Addressing Extrapolation Error in Deep Offline Reinforcement Learning"](https://openreview.net/forum?id=OCRKCul3eKN)).
12.Authors mention that these papers "propose complicated variants of the one-step approach". While I can agree the methods in either of these papers are more complex, the results are also better (CG evaluates on different dataset, so hard to say) and they both provide ablation studies showing the behavior of the one-step approach with a simple, single improvement step (I believe BVE in CG and BC w/ Filter in GN). If we consider these two papers as prior work, the proposed method is basically a simplified version of CG/GN that doesn't work as well. 
13. After having read these two papers, I don't think I gained much new insights (basically only point 5 above) from reading the paper under review. At the same time, I consider the presentation of this particular idea as a useful contribution to the community.

Minor notes:
- I'm not sure what is meant by "removes the policy correction network" in BCQ; I didn't find it in Fujimoto et al.
- I don't find the abstraction of OAMPI particularly useful, as it doesn't capture well all 3 types of algorithms that you have (one-step, multi-step, and iterative).
- You state that "$\varepsilon_\beta$ are likely to have larger magnitude the further we go from the support of the dataset." This matches my intuition, but I think it should be possible to give more formal reasoning why this would be so.
- You say "In the example, like in the D4RL benchmark, we see that one step outperforms multiple steps of improvement." This is not substantiated by the data, where the results were half-half.
- In sec. 6.1/2, it's not clear what algorithms are tested. You write "As shown in Figure 2, we often observe that the iterative algorithm": which iterative algorithm? It looks like one particular algorithm (with different alpha) was used and it's not clear whether the results would transfer to other ones.
- In sec. 4.3 "incentivising $\pi$ to cover $\beta$"

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper is concerned with offline RL, and discusses the problem of error propagation of multi-step iterative algorithms in this domain. They propose to resolve this problem through one-step updates of the critic and regularization to the behavior policy.
main_review: - Originality

The paper is very simple. This is a virtue of the paper, I don't expect papers to be complex to be good.

The paper has little novelty in it. This would not bother me in general, but I also feel that the authors did not delve deep into the phenomenon described in the paper, neither theoretically nor empirically. 

The paper was not informative. The paper felt more like a survey to me. I did not feel I learned any new ideas after reading it.


- Quality

The submission is not technically sound. There is little to no theoretical analysis in the paper. The work seems rushed and incomplete. Empirical evaluation is lacking. Either the authors add strong empirical evidence, or they provide new theoretical insights. The fact that overestimation of Q-values affects performance has already been proven theoretically as well as empirically. 
Are the authors providing another method for overcoming overestimation? (i.e., non iterative evaluation) If so, why are they not comparing their work to the vast array of current work, such as the following simple paper that uses BC with TD3 (without any special tricks) to achieve sota performance:
https://arxiv.org/pdf/2106.06860.pdf

- Clarity

While this work is attempting to deliver a very simple idea, the overall organization and flow of the paper make the paper very hard to follow.


- Significance

I think that the solution of one step gradients could be useful as a regularization technique, and in that sense the paper could help practitioners. Nevertheless, I do not believe a "trick for practitioners" is enough to pass the bar to a top-tier conference.


limitations_and_societal_impact: The authors have addressed some of the limitations in their paper.
review_title: null

Meta-review: The submission provides  surprising empirical results for a simple, yet overlooked, approach for off-policy continuous RL. As such this work can be of  interest of  AI/NeurIPS community.  
Decision: Accept (Spotlight)