Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on the problem of offline reinforcement learning, where the authors consider the limitation of the conservative Q learning (CQL) method and propose a new way of applying a fine-grained control of the conservatism level. The authors argue that setting the level of conservatism to be a global constant as done in CQL is suboptimal, because the dataset often contains both good and poor actions and hence it is intuitive to put conservatism on good actions compared to the poor actions. Instead, the paper introduces adaptive coefficients to weight the conservatism terms.


Building on top of the formulation of CQL, the authors introduce two state-action dependent coefficients to weight the push up and push down loss instead of using a single global conservatism coefficient. The authors show theoretically that a properly chosen set of coefficients can still produce value estimates that lower bounds the true Q function, but with less conservatism than having a globally constant coefficient. The authors then propose a concrete formulation of the coefficient d based on the quality of the transition, which is measured by its relative reward and cumulative return magnitudes.

The authors evaluate the proposed method in the D4RL suite of tasks, and show that the proposed method improves upon the baseline CQL method in some environments.

strength_and_weaknesses: I reviewed this paper for NeurIPS 2022 before it was withdrawn by the authors. Overall I think this paper presents an interesting direction towards improving value conservatism methods, where they often suffer from the problem of being overly conservative. However I do still have some concerns about the particular approach and empirical results.

 

### Strength

This paper presents a very interesting and potentially very effective direction for improving the use of value conservatism in Q-learning style offline RL algorithms. Conservative algorithms in offline RL are known to be difficult to tune, and often suffer the problem of either being overly conservative where the policy does not improve beyond the behavior policy, or being insufficiently conservative where the Q functions suffer from overestimation. Having adaptive conservatism coefficients that depend on the transition could be a promising way of solving this problem.



### Weaknesses

First of all, I want to emphasize (again) that certain baseline evaluations for CQL in this paper seem flawed, where a properly tuned CQL can achieve much better performance than reported in this paper. For example, in one open source CQL implementation I used, CQL can achieve 95 in hopper-medium-replay-v2 and the reported performance is 48, and in walker-medium-replay-v2, CQL can achieve 82 while the reported performance is only 26. Hence I believe that some of the CQL results are not valid, and this presents a major problem for this paper because the proposed algorithm directly improves upon CQL. I’ve brought up this point in my NeurIPS 2022 review of this paper and pointed out the open source CQL implementation suggested by the original CQL authors to reproduce these numbers. Unfortunately, the authors have not addressed this problem in this submission.


Besides the flaw in certain baseline evaluations, the empirical performance of the proposed method is also not very strong. Following the numbers on the D4RL suite, it seems that ACQL does not really outperform significantly, despite introducing many more degrees of freedom and complexity. It makes me question whether the marginal improvement is really due to the proposed method, or because of the extra tunable components introduced.

The specific choices of the loss functions to learn the conservative coefficients are not well justified. It is intuitive to see why the agent should be less conservative on good actions and more conservative on bad actions, but I’m not convinced that the proposed transition quality function is a good way of measuring this. Moreover, the overall loss formulation is quite complicated with many moving parts, so it is difficult to understand the necessity and choices of each part without good justifications.

clarity,_quality,_novelty_and_reproducibility: Regarding the presentation quality, the paper is well written and is easy to follow. The novelty of the proposed method is rather limited due to the fact that it is a small improvement on CQL.
summary_of_the_review: Overall, while the general direction of research is interesting, due to the concerns about the empirical evaluations and justifications for the proposed method, I still cannot recommend acceptance of the paper in its current state.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces ACQL, adaptive conservative Q-learning for offline reinforcement learning. Different from the standard Q learning framework, ACQL learns a less conservative policy by controlling the weight of each sample with a learnable parameter. The weight is directly associated with the statistics of the dataset and the quality of the transitions. Theoretically, ACQL gives monotonic improvement on the conservative level of CQL constraints. Empirically, the authors have presented improved results on the D4RL benchmark across multiple tasks.
strength_and_weaknesses: Strengths:

Overall, the paper is well-written, well-motivated, and easy to follow. The paper has studied an important and interesting problem of the famous conservative learning methods for offline reinforcement learning, i.e., the over-conservative value estimation. Theoretically, the proposed ACQL addresses the issue of over-conservative value estimation.

Weaknesses:

The main weakness of this paper is the insufficient empirical results.

1/ The results presented in Table 1 are susceptible. According to the authors, the results are directly taken from the original paper of baselines or the D4RL whitepaper [1]. However, the reported results of CQL are significantly worse than the ones discussed in IQL [2]. For example, in the submitted manuscript, CQL has a normalized return of 26.7 on walker2d-medium-replay, while in IQL paper, CQL has a normalized return of 77.2, which is already higher than the reported ACQL result (45.2). There are 4 such environments where the reported CQL performs worse than the CQL results in the IQL paper, including halfcheetah-m-e, hopper-m-e, hopper-m-r, and walker2d-m-r. The performance gap is caused by the misuse of environments. In gym-locomotion-v0 environments, the performance is generally worse than in the v2 environments. It seems that the reported results of CQL are directly taken from the D4RL whitepaper, where v0 environments are used, while the reported IQL results are run on v2 environments. As a result, the results reported in Table 1 are not comparable. In addition, the submitted manuscript mentions no information about the version of its environment, which further makes it hard to justify the soundness of its empirical results.

2/ No results on antmaze are reported. Antmaze is considered one of the most difficult environments due to its sparse and delayed reward. In addition, low-quality transitions often exist in such environments, which I believe should be an ideal environment to test ACQL against CQL. It would be good to add such results.

3/ In addition to the ablation studies, some intuition should be provided to better understand the influences of these two weights. Although theoretically, ACQL guarantees a monotonic improvement, it remains unclear how such behavior reflects on the learned weights. It would be great to visualize the relationship between the learned weight and the quality of transitions, i.e., m(s, a).

4/ According to the ablation studies, it seems that ACQL is fairly unstable. Without L_{mono}, ACQL would fail completely and even perform worse than the reported CQL result. How about training only with L_{mono} without other losses?

Besides the above-mentioned issues, another issue is that ACQL has multiple losses to ensure the learned weight is reasonable and stable. How does this compare with directly weighting the samples with a fixed weight? For example, m(s, a)?

References:

[1] Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020). D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.
[2] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. International Conference on Learning Representations (ICLR)
[3] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS)

clarity,_quality,_novelty_and_reproducibility: The paper is overall clear, well-written, and original. The paper has a relatively high theoretical quality but low in its empirical studies.
summary_of_the_review: I think the paper is interesting and novel. However, as discussed above, the empirical study needs significant improvements. Thus, I think it is not ready for publication yet.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ACQL, which weights the regularization term used in CQL to improve performance. By using a weight function that is based on heuristics, ACQL regularizes each state action differently according to the weight. In the experiment, the paper shows the results on various benchmarks that ACQL can improve on CQL.
strength_and_weaknesses: Strength
- performance improvement over CQL
- paper is clearly written and easy to follow
- extensive experiments on many different benchmarks including ablation study

Weakness
- weak support on "monotonicity"
While it is interesting to have proposition 4.1~4.2 and propose weight functions having such characteristics, the definition 4.1 is not well supported. Even in the case where we have $Q^*$, using the weight functions following eq 6, 7 will not recover an optimal policy. If we have some other choice, e.g. weight that is proportional to exponential of $Q$s, we may get some other intuitions on the algorithm by relating it to AWAC or etc.

- not well-motivated transition quality measurements
While the authors suggested using $Q^*$ as a weights, the suggested transition quality measurements are too irrelevant to $Q^*$, even considering that $Q^*$ cannot be computed. There are number of choices that are not explained, e.g.
1. averaging normalized reward and return: it is not a normal choice to average between reward and the return; if we wanted to make a bit more myopic measure than the return, I may just increase the discount factor for this. This choice is too ad-hoc and will be very suboptimal in some specific cases.
2. using the behavior $Q$ instead of $Q^*$: using a simple return $g$ corresponds to using a behavior $Q$ function. It does not matches the above motivation of using $Q^*$ as a weight function.

- about the positivity
I think using the positivity objective (19) won't ensure the model has positive output. What is the reason of not using a positive transformation at the output of the neural network? e.g. softplus
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very clearly written and easy to follow.
Quality: There does not seem to be a wrong argument in the paper, and extensive experiments support the algorithm. However, the algorithm is somewhat ad-hoc and not well supported by the theoretical results.
Originality: This is a paper about souped-up CQL, so it is not that original. The theoretical results also use the proof methods used in CQL.
summary_of_the_review: The paper is overall clearly written, and there may be some readers interested in reading the paper. However, in my opinion, the theoretical results are weak, and the algorithm is not well supported by the theory. Despite the extensive evaluation of the algorithm, considering the low originality of ACQL, I am more leaning toward rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: There are definitely some interesting ideas in this paper, but that there is not enough reviewer support to warrant acceptance. In particular, there are weaknesses in the empirical performance of the proposed method (e.g. only small improvements over CQL, instability, poor performance on antmaze) and the theoretical analysis. None of these weaknesses are a dealbreaker on their own; e.g. I agree that gaps between theory and practice are expected, but the weaknesses seem to outweigh the strengths in the current version.
justification_for_why_not_higher_score: There are weaknesses in the empirical performance of the proposed method (e.g. only small improvements over CQL, instability, poor performance on antmaze) and the theoretical analysis. None of these weaknesses are a dealbreaker on their own; e.g. I agree that gaps between theory and practice are expected, but the weaknesses seem to outweigh the strengths in the current version.
Decision: Reject