You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
Authors: Wonjoon Goo
Scott Niekum
Venue: CoRL2021 Poster (2021)
Keywords: offline reinforcement learning
conservative policy evaluation

Abstract: The goal of offline reinforcement learning (RL) is to find an optimal policy given prerecorded trajectories. Many current approaches customize existing off-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated. However, the convergence of such approaches is not guaranteed due to the use of complex non-linear function approximation and an intertwined optimization process.
By contrast, we propose a simple baseline algorithm for offline RL that only performs the policy evaluation step once so that the algorithm does not require complex stabilization schemes. Since the proposed algorithm is not likely to converge to an optimal policy, it is an appropriate baseline for actor-critic algorithms that ought to be outperformed if there is indeed value in iterative optimization in the offline setting. Surprisingly, we empirically find that the proposed algorithm exhibits competitive and sometimes even state-of-the-art performance in a subset of the D4RL offline RL benchmark. This result suggests that future work is needed to fully exploit the potential advantages of iterative optimization in order to justify the reduced stability of such methods.


Reviews:
[{"review_id": "qQq-_wokbsR", "reviewer": "Reviewer_HJhz", "summary": "The paper introduces a simple baseline for Offline RL that attempts to recover the best action of the behavior policy. The authors fit state value function with Implicit Quantile Regression, while the state-action value function is trained using the median value of the distributional critic and a pessimistic constraint to get lower values for out-of-distribution actions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n* Fitting value-critic using behavior data and then impossible pessimism only on Q is a simple to implement and test baseline for Offline RL. It avoids action extrapolation since it fits a value function using behavior policy and fits Q with a pessimistic constraint.\n* The approach is interesting and theoretically sound. \n* This simple baseline demonstrates strong empirical performance comparable to the state-of-the-art methods for Offline RL.\n\n### Weaknesses\n* I do not see any major weaknesses", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe that the paper introduces an important baseline for Offline RL that will impact the design of algorithms and research benchmarks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I would encourage the authors to discuss failure cases of this approach.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "qNQlvJOzxfK", "reviewer": "Reviewer_q44u", "summary": "This paper proposes an offline RL method thas is supposed to be the well-designed baseline for other offline policy-learning algorithms.  It's goal is to effectively recover an optimal policy relative to the value function describing the behavior policy, that is to /not/ perform any explicit policy improvement.  The reasoning is that it is primarily the policy improvement step that induces instability by aggressively sampling out-of-distribution updates.\n\nThe general method is to learn a state-value function Y from sarsa updates, and then to learn a Q-funciton with both on-policy pairs and off-policy pairs, with the off-policy pairs having an extra regularization function.\n\nA policy is then found using this Q-function (not sure how).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I really enjoy this paper's idea, and its questioning of many offline RL algorithms.   I think it should be accepted, and the results are important for the community.  \n\n....however, I found it often confusing to read, there are a lot of things going on and it's not always clear /why/.  For example, the naive reader would assume that running Q-learning SARSA would be sufficient to have a stable Q-function that one could then find an optimal policy for either through CEM (such as QT-opt), or by learning a stochastic BC agent and then using that as the support of \\Beta(s) (this might actually reduce back to Critic-Reguralized Regression, but I've run out of time to check).\n\nInstead of doing this you decide to go the pessimism route, which is one of the criticisms later on of similar offline PI algorithms.  This was a bit surprising turn of events that I think could be more justified.  I also am not clear why you then choose to use a <= instead of < or even <-epsilon to enforce your pessimistic constraint.  Clarification here would be welcome.\n\nEq. 8 is basically the core of the paper, IIUC the \\lambda is zero if the pair is on-policy and some value >0 if it is off-policy.  If this is true it sholud be explicit in Eq. 8 (I'm not currently sure it's true).  \n\nAs for R I clearly don't understand how it works, it includes \\ph_\\phi and \\u(s), I don't fully understand how these are produced.  Are you using direct gradient updates from the critic to learn your actor?  Please make this more clear.  I also think the Y(s,tao) notation needs to be clarified,  it probably only takes one sentence but would make the reader's life much easier.  Additionally, I mention it below, but pleas call this function V.  I have to rewrite Y->V in my head for things to make sense and it took me a while to understand things the first pass because of this (FYI, this was the paper that took me the longest to review and I had to come back to it because it was hard to parse).\n\nOnce all this comes together, it would be nice to clearly explain how you actuall produce the final policy, ideall ywith a clear equation.  When reading the paper for the first pass, I'm clearly looking for \"\\pi = ________\" to understand what's going on.  Explicit is your friend :)\n\nOverall the experiments look reasonable, but I was a bit dissapointed to see that the figure I was primarily looking for (Performance of Q^B\\R) is i the appendix.  This figure is quite important because it justifies why I need to put it with all this regularization stuff and not just run SARSA, I would suggest trying to fit it in if possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I put my recommendations in strength & weaknesses.  My main suggestion would be to rewrite Sec 4., especially Sec 4.1, and don't move so quickly, justify your choices, clean up the notation, explain a couple things more explicitly and say less on the unimportant things.  I don't know what the 1-wasserstein metric is, and I don't really care in this context, but because you wrote that you're doing 1-wasserstein metrics now my head is thinking about OT and I'm wondering if I totally missed something in the paper and then I'm lost.  I would replace lines 171-173 with something much more important: what in the world does the second parameter of your value function mean and do!?  Now that would have made me a happier and less confused reader :)\n\nOverall I will gladly move to a strong accept, but I think it's important to clear up Sec 4 and 4.1 so that this is a solid and clear paper.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "l.29 Please define the deadly triad.  I know it's a thing, I had to look it up to get the exact definition.\nEq 6. I found this equation a bit confusing.  If \\Beta(s) is your behavior policy that you are learning, I would have much preferred a \\pi notation.  I am still a bit confused which is the learnt policy and which is the demonstrated policy.  Also please define supp someplace, it's not a super common notation (I'm assuming it's the set of possible actions that the behavior policy has taken in state s).\nl145 in the stated theorem, shouldn't Q^hat be strictly less than the max? Otherwise Q-values for \\tilde{a} could be equal to the max, and the argmax would but undefined (or multiple), which would not guarantee the reproduction of \\Beta^*.  Perhaps having a margin would guarantee that the non-expert values stay below the max.\nl.160 Could you consider using V vor your state value distribution? It took me a good 5 minutes to understand what you meant it to be, it could very well be called V^\\Beta, this would greatly increase legibility.\nl.189 Can you clarify Y(s; t_{1,2})'s meaning? Does a smaller tao push towards a less probable distribution? I should not have to read the IQN paper to understand what you are doing here ideally.\n"}}, {"review_id": "XN4Qn-4pSOM", "reviewer": "Reviewer_LDRr", "summary": "This paper proposes a simple baseline for offline RL where the authors only perform the policy evaluation step once without performing policy optimization in an iterative loop like traditional actor-critic algorithms. The authors also conduct pessimistic regularization on the Q function of the behavior policy to ensure that it does not suffer from OOD actions. Empirically, the proposed method achieves competitive results and even best performances in certain cases in standard offline RL benchmarks despite being just a policy evaluation approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents an interesting and strong baseline for offline RL, which is somewhat novel. The insight on the challenges of iterative off-policy actor-critic learning in offline RL is also intriguing and is overlooked by the community in the past. The authors also present strong empirical evidence that removing the iterative process and only keeping the policy evaluation step also yield strong performances in certain cases, which collaborating the claim that iterative actor-critic learning is hard in offline RL. The authors also provide a theoretical analysis of the proposed approach, which makes the paper more theoretically grounded.\n\nHowever, I do have a few concerns over the paper, which I will discuss as follows.\n\nFirst, despite that the results show that the method can achieve competitive results compared to prior offline RL methods, the domains where the authors evaluated on are either hard for all of the methods or generated by a single behavior policy except the medium-replay, medium-expert and random datasets in D4RL where the proposed method is underperforming. Therefore, I think the results are somewhat less surprising and a bit uninteresting since the proposed method (YOEO ($\\pi^\\tilde$)) can be viewed as a strong behavior cloning approach and should perform well in settings where data distribution is narrow and data quality is high. Besides, I believe the CQL results in the paper are not up-to-date. The CQL results in the NeurIPS camera-ready (https://proceedings.neurips.cc/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf) are significantly better than the proposed method in almost all the domains. Therefore, I think the benefits of the method over SOTA offline RL methods are somewhat unclear. \n\nMoreover, I think it would also be important to evaluate the method in settings such as antmaze and kitchen where the dataset is diverse and composed of undirected data. I think the proposed method could struggle there since simply learning the optimal behavior policy is not sufficient and also challenging in those settings.\n\n============================\n\nAfter reading the rebuttal, I still think that the YOEO does not perform well on wide datasets such as random and medium-replay, suggesting it could just be a strong BC method with some additional tricks. The variants of YOEO also doesn't perform well uniformly against CQL on all settings of kitchen. It is also unclear how the authors would pick the two variants of YOEO, which could just introduce additional hyperparameters for tuning. Therefore, I would keep my current score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper presents an interesting baseline for offline RL, given the two points on the insufficient empirical evidence mentioned in the Strengths And Weaknesses section, I would vote for a weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. Compare the latest CQL results.\n2. Evaluate the method on more complex domains such as antmaze and kitchen.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all like the conceptual idea of the algorithm presented in the paper. The authors' have provided substantial revision to placate Reviewer q44u's concerns. Reviewer LDRr still has some concerns about the baseline performing well only on some datasets and not others. I suggest for the final version, the authors add a paragraph or two discuss this. Indeed, I personally believe that one of the main reasons such a simple baseline performs so well may be because of inadequate benchmarks in the offline RL community.
Decision: Accept (Poster)