Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper surveys recent offline RL algorithms and seeks to analyze different factors contributing to their performance. Novel evaluation protocols are designed to analyze their representation and behavior. Based on this analysis, the authors propose a well-motivated modification to IQL which achieves strong results on several D4RL datasets.
strengths_and_weaknesses: Strengths:
- Well-motivated range of analysis on the representation learned by an offline RL algorithm.
Proposed algorithms (RIQL, USS variants) show strong performance and are motivated by earlier findings in the paper.
- Novel and insightful analysis of how to integrate model-free methods into a model-based framework, and how to address the failures of a naive approach.

Weaknesses:
- On line 208 and 269, the uncertainty estimate used is the max mean-discrepancy, this measure is unusual as we would use the standard ensemble variance in supervised learning [1].
- Evaluation is solely on med/exp datasets, a good understanding of the strengths of each algorithm on mixed/random data where we may expect a higher extrapolation gap would be useful.

Minor:
- TD3BC should TD3+BC
- Typo on line 182: dose -> does
- Line 204, 262: the learned probabilistic model and model-based training are more accurately attributed to MOPO rather than COMBO.
In all tables and results: total random seeds should be included.

[1] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell.
questions: - It would be nice to disentangle the different contributions of model-based RL algorithms, could the analysis of model-based algorithms be better served by using MOPO, a more minimal algorithm?
- On line 157, the assumption is that an online learned “optimal” Q* would appropriately assign higher Q values to better true performing actions. Would this be flawed due to approximation? We may expect the Q functions’ values on OOD actions to be inaccurate. Could there be further issues from using a TD3-learnt policies which is different from the SAC policy used in many algorithms and used to generate the data?

limitations: The paper provides good insights on the datasets it tests, however:

- Limited evaluation on med/exp datasets, this could be expanded to include more D4RL-suite datasets.
- Incomplete information on seeds used during evaluation.
- Possible inaccuracy of using online Q-functions as ground-truth. Tabular settings where precise Q-function can be determined could elucidate this better.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a comprehensive analysis of the state-of-the-art offline reinforcement learning algorithms. In particular, the paper evaluates the critics and policies from offline RL algorithms using several metrics, including representation probing, when the learned representations are used to predict various quantities, effective rank, action ranking, etc. The authors observe a mismatch between the quality of critics' predictions and the performance of the policy for some methods. Based on this insight, the authors propose a modification of IQL that achieves significant improvements over the original version.
strengths_and_weaknesses: # Strengths
* The paper is well written and easy to follow.
* I believe that the analysis of offline reinforcement learning methods is very valuable to the community and has received only limited attention so far. 
* The analysis covers a variety of metrics for evaluating the critic and policy separately.
* The metrics introduced in the paper can help tune the components of offline reinforcement learning methods separately. The analysis has been applied in practice to the improvement of IQL.

# Weaknesses
* The paper focuses only on a subset of tasks considered in CQL and IQL.
* The paper focuses only on improving IQL. 

To sum up, I believe this paper is relevant to the offline reinforcement learning community, and the pros outweigh the cons. Therefore, I recommend this paper for acceptance.
questions: * Does the analysis help to improve the other methods as well?
* Does the analysis extends to other D4RL datasets? In particular, how have you trying running the metrics on antmaze tasks?
limitations: The paper can be improved by considering a wider variety of datasets from D4RL and applying the same analysis to enhance the other methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Introduces a set of experiments to evaluate & diagnose the bottleneck of offline RL algorithms along three axes:
(1) Representations
- Representation probing: Use the second last layer of the critic network to predict the next state, reward, and action via linear regression. Similarly, use the actor embedding to predict the optimal action, value function, and Q-function (which is approximated by a trained TD3 policy).
- Representation metric: Feature dot-product and effective rank of critic representations.

(2) Value functions
- Rank actions using the learned Q(s, a)

Observation: TD3BC and IQL learn more accurate value functions (policy evaluation might be more effective), but achieve worse performance (policy improvement might be limited). More generally,  an offline RL algorithm can be higher-performing, but learn poor representations and value functions.

(3) Policies
- Policy ranking experiment: Average MSE of the selected action vs. optimal action.
- How often does the policy take OOD actions?

Observations:
- COMBO selects most optimal & worst actions.
- Performant policy is good at selecting better actions, even if the actions are sub-optimal.

The paper also introduces a new offline RL algorithm called RIQL (Relaxed In-Sample Q-learning), based on a simple modification of IQL.
- The method is motivated by empirical, heuristic-based observations: that AWR-based policy improvement is effective to avoid taking OOD actions, but is sometimes over-conservative. It add extra policy constraints and use a less conservative actor loss to enable learning from OOD actions.
- Note that there is no theoretical justification or provable guarantees for the method.

The paper also investigates when a learned dynamics model helps model-free offline RL. Introduces an uncertainty-based sample selection method that is more robust to model noises.

strengths_and_weaknesses: Strengths:
The paper presents thoughtful experiments analyzing different offline RL algorithms. It is an interesting result that performant offline RL algorithms often exhibit poor representations and inaccurate value functions. (However, I wonder if this is due to the environments/tasks, which are all similar, state-based locomotion tasks.) Overall, I think the empirical experiments & analyses are useful for better understanding offline RL algorithms on simple classic control tasks, although it is unclear if these analyses still hold for more challenging tasks.

Weaknesses:
1. My main concern is that all evaluations are done on the toy classic control locomotion tasks in the D4RL benchmark, which are limited due to the fact that simple filtered behavioral cloning outperforms SOTA offline RL methods on D4RL; and therefore, good performance on these simple D4RL tasks does not necessarily translate to good offline RL performance. The paper's contribution would be greatly improved if the authors add a more challenging and different task other than toy locomotion, such as D4RL AntMaze navigation, manipulation tasks, or image-based tasks such as Atari.

2. The proposed method (RIQL) performs similarly as other baseline methods on the toy locomotion tasks (Table 10), which shows that RIQL is a reasonable offline RL algorithm to use. On the other hand, RIQL is a heuristic-based modification of existing offline RL objectives, and there is no theoretical justification (e.g. provable guarantees about policy improvement) for the proposed method RIQL.
questions: 1. Is it possible to add a more different task that is known to be more challenging than the standard locomotion tasks (halfcheetah, hopper, walker2d)?

2. Description of the policy ranking metric in L89/L637 is unclear: “Average percentage of policy pi_j that ranks the first/last (with largest/smallest Q* value) across states.” Did you instead mean: “Percentage of states where the Q*(pi_j(s), s) is largest/smallest out of the baseline and BC agents”?


--Minor typos that did not affect this review in any way--

L35: “When does a learned dynamics model is helpful to a model-free offline RL agent” -> “When is a learned dynamics model helpful to a model-free offline RL agent?”

L162: “rank the M action” -> “rank the M actions” ?

L228: “IQL uses expectile regression (ER) to for policy evaluation” -> “IQL uses expectile regression (ER) for policy evaluation”

There are many duplicate citations, e.g.:
- [18] and [19]
- [33] and [34]
- [38] and [39]
- [43] and [44]
- [54] and [55]

limitations: Yes
review_title: null

Meta-review: metareview: The main strengths of this paper are that (1) it provides some interesting analysis that leads to some somewhat surprising findings, and (2) it presents and evaluates some new technical algorithmic ideas based on this analysis that lead to improved performance.

After the author discussion, the main weaknesses is that the new ant-maze results are somewhat disappointing, showing that the algorithmic ideas don't improve over IQL on a more complex problem setting. The ant maze tasks are a lot more interesting and complex than the standard locomotion tasks, and so this as a fairly major weakness. Of lesser importance, the title is not particularly descriptive, and could be used to describe a lot of papers. So, I would like to suggest to the authors to make the title more specific to the contributions of this paper.

Overall, the reviewers and AC think the strengths outweigh the weaknesses, especially since the analysis is interesting on its own and since there is some new analysis on more complex image-based settings, irrespective of the technical ideas only providing benefits on simplistic locomotion tasks. Nonetheless, we encourage the authors to use our feedback to further improve the paper.
confidence: Less certain
Decision: Accept