Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper points out the previous offline reinforcement learning methods are too conservative about the out-of-distribution (OOD) actions and instead propose a mild conservative algorithm. It introduces an auxiliary loss term to properly train the value function for OOD actions. The proposed method, MCQ, is shown to outperform previous methods empirically and theoretically proved to behave at least as well as the behavior policy, and has no erroneous overestimation.
strengths_and_weaknesses: Pros:
This paper is clearly written, and the whole structure is organized and easy to follow. The method is well-motivated and the claims in the paper are all supported by either theoretical analysis or experimental results. Although over-pessimism is not a new problem in offline RL, this method is very innovative and elegant in its processing of OOD actions. In comparison with the baselines, MCQ archives a remarkable improvement on random or medium datasets. The authors also make a careful analysis of the sensitivity of hyperparameters.

Cons:
I want to draw the authors' attention to a recent paper that also addresses the over-pessimism of offline RL algorithms (https://arxiv.org/pdf/2207.02200.pdf), which uses an adaptive method. Besides, I am also confused about the results in Figure 2. It seems that decreasing $\lambda$ is always beneficial to the final performance, and there is no trend that introducing the auxiliary loss can help the final performance. I think the authors should use a small scale of $\lambda$ (0-0.3) to explicitly show the benefits of the auxiliary loss.

questions: Please see the main review.
limitations: The authors have addresses the limitations in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Offline RL is a topic of significant interest. One common class of approaches is to learn an action-value function but to enforce that the function is `conservative' so that it does not result in a policy which takes actions that were not in the training data (and therefore of unknown value).

This work introduces a ``mildly'' conservative Bellman operator. In particular, for actions in the support of the behavior policy the operator behaves like standard Bellman operator, but for actions outside the behavior policy support it assumes that value is delta less than an action in the support.

They show that this operator will always result in a conservative Q estimate (that is, it will not over-estimate the value of any action). They then introduce a practical approximation (where the behavior policy is estimated by a CVAE) and test on a set of offline RL control tasks. It performs notably better than prior work on poor demonstrates, but does not consistently outperform TD3+BC when expert demonstrations are available.

strengths_and_weaknesses: Strengths:
- Well communicated.

- Principled explanation of an algorithm that empirically performs well on a set of benchmark tasks.

- Offline RL is a topic of significant interest to the community and active research.

Weaknesses:

- Ideally, the algorithm would be tested on a different style of tasks as well (e.g. perhaps Atari), rather than only MuJoCo control tasks.

- This approach does not seem to perform as well when expert level demonstrations are available.

Minor:

I found definition 2 (line 123) confusing since it refers to $\mu(a|s)$ which is stated above you are trying to avoid. It is explained further what the actual "practical" solution is when $\mu$ is not known, but I found this a bit confusing on first read.

questions: Why do you think TD3+BC seems to be better for expert-level demonstrations (for most tasks)?

The checklist for this paper indicates the code is available for reproducing the experiments but I didn't see a link anywhere to the code?

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a method for offline reinforcement learning based that involves assigning pseudo-Q-values to OOD values called MCQ. The method's main idea is to modify the Q-targets by detecting the out-of-distribution actions via a density model and assign the Q values to these actions similarly to BCQ by taking a maximum of a Q function over N samples from a density model. The main difference to BCQ is that the authors propose to use this backup operator only for OOD actions instead of all actions and use an actor to recover optimal actions from the modified Q-function. The method is evaluated on severals datasets from D4RL where it outperforms the baselines.
strengths_and_weaknesses: # Strengths
* The method is theoretically sound and practical. Even though all method components have been explored in prior work, the idea of not penalizing the values of OOD actions but using a BCQ-style value estimate to impute the values for these actions is novel.
* The practical version of the algorithm (Eqn. 11) is easy to implement. 
* MCB demonstrates good empirical results on a subset of D4RL tasks.
* The paper is well written and easy to follow.

# Weaknesses
* The method can be seen as an extension of BCQ. However, the comparison to BCQ is missing.
* The practical implementation of the method diverges from the theory. In particular, the practical implementation of the method omits OOD evaluation and instead regularizes all actions sampled from the training policy, which are not necessary OOD and, throughout training, will certainly become less OOD which can result in over-penalizing the optimal actions due to value underestimation caused by the BCQ-style operator.
* The method is evaluated only on the locomotion tasks from D4RL, which do not require stitching [1] (dynamical programming).
* Also, the method is evaluated using only 4 seeds which might be insufficient.
* The offline to online experiments miss essential details. In particular, it is unclear how the authors obtained results for other methods. Also, the authors pick a different subset of tasks for offline to online finetuning than considered in the original papers (AWAC, IQL).

[1] RvS: What is Essential for Offline RL via Supervised Learning?
S Emmons, B Eysenbach, I Kostrikov, S Levine
questions: * Did you try implementing the version of the method that regularizes only OOD actions?
* How well does the method perform on other D4RL datasets (for example, antmaze, adroit and kitchen tasks)?
* What implementations of baselines did you use for offline to online experiments?
limitations: Right now, the main limitation of this work is limited experimental evaluation. In particular, the method is evaluated only on locomotion tasks using an insufficient number of runs. The paper considers a set of tasks different from the standard tasks used in prior work for offline to online experiments. I will raise my score if these concerns are addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes using mild conservatism in offline RL to benefit generalization and to avoid overly conservative on OOD actions.
Specifically, this paper develops a Mildly Conservative Bellman (MCB) operator for offline RL, where OOD actions are actively trained and their Q values are actively queried.
Theoretical results under the tabular setting and a practical MCB operator are provided.
Empirically, combining the practical MCB operator with SAC performs well on the D4RL MuJoCo locomotion tasks and when transferring from offline learning to online.
strengths_and_weaknesses: ### Strengths
1. The proposed method is well-motivated by theory.
2. The idea of actively train the Q-values of OOD actions is interesting, though in some sense similar to the CQL paper [1].
3. Experiments are extensive and the proposed method generally performs well.

### Weaknesses
1. Since the proposed method requires behavior cloning (BC), it is doubtful if the proposed method can work well on higher-dimensional and/or non-Markovian datasets where BC can be difficult. Theoretical results such as Proportion 5 requires sufficiently accurate BC, which may not be possible on harder datasets. \
In fact, from Table 8, the proposed method performs less favorable on maze2d datasets compared with other stronger baseline such as OptiDICE [2], and is slightly worse than one-step RL [3] on Adroit dataset.
2. There seems to be discrepancies between the impractical theoretical algorithm and the theoretically less-supported practical algorithm, in particular the practical Loss functions (L167-193). 
3. From Table 4 & 9, the proposed method requires per-dataset tuning of the weighting coefficient $\lambda$ on a relatively fine grid, which muds the empirical significance since many of the compared baselines actually *unify* hyperparameters across the MuJoCo datasets, *e.g.*, CQL, IQL and TD3+BC.


\
[1] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 1179-1191.
\
[2] Lee, Jongmin, et al. "Optidice: Offline policy optimization via stationary distribution correction estimation." International Conference on Machine Learning. PMLR, 2021.\
[3] Brandfonbrener, David, et al. "Offline rl without off-policy evaluation." Advances in Neural Information Processing Systems 34 (2021): 4933-4946.
questions: Below are my questions and concerns on this paper.

1. From Proportion 5, even under the rather strong assumption on the discrepancy between $\hat \mu$ and $\mu$, is it still possible that the OOD actions will Q-values higher than the supremum of the in-distribution Q-values? If so, how the proposed Q-learning-based method avoids the detrimental impact of those "overestimated" OOD actions?   
2. Why in the proof of Proportion 4 (Below L72 of Appendix), we have $\\{a_i'\\}^N \sim \mathrm{support}(\mu)$? And the $\hat T_1$ here seems inconsistent with Eq. (9).
3. Why in L183-184 "in-distribution actions are still trained to approximate the optimal batch-constraint Q value"? Is it possible that the target value of such in-distribution actions are inflated by the psedo-label? \
Since Eq. (13) is independent of $a^{ood}$ and $\pi$, will the psedo-target value Eq. (13) collapese $Q(s^{in}, a^{ood})$ at different $a^{ood}$ into a same value, even those in-distribution? If so, how could the proposed the method select out good in-distribution actions?
limitations: The authors briefly addressed the limitations.
No potential negative societal impacts is discussed.
review_title: null

Meta-review: metareview: All reviewers are generally positive or borderline about this paper. Reviewer's note that the method is theoretically sound and practical to implement. Even though all of the components have been explored previously, the authors combine them in a novel approach that convincingly improves over prior works. 

Major concerns have been addressed by the author's response, however, I agree with reviewer fVHB that per dataset tuning of $\lambda$ muddies the comparison with previous approaches that do not do similar. I would encourage the authors to additionally report the best performance with a single setting across datasets to make the comparison clearer.


confidence: Certain
Decision: Accept