Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To solve the distribution shift problem in offline reinforcement learning of conservative methods, the paper let value functions learn on different degrees of conservatism (optimism/pessimism) instead of a fixed degree of conservatism. The paper proposes an algorithm, called CCVL, to achieve its goal in discrete-action environments.

The major contribution of the paper shows the shift from fixed conservatism to flexible conservatism improves performance on some discrete control domains.

strength_and_weaknesses: This paper has the idea to learn confidence-conditioned value fundction for adaptive policy optimization. The idea is the strength of the paper. However, there are flaws in the experimental design and experiment discussion, which make it impossible to thoroughly access the new idea.

Problems (ordered by appearance in the paper)
1. Two important terms, epistemic and aleatoric uncertainty, are unexplained in Section 2. It would be better for the paper to include the definition especially when the paper tries to emphasize its work on epistemic uncertainty.

    1.1 The last sentence of Section 2 gives a false impression that studying epistemic uncertainty is better than aleatoric uncertainty. Focusing purely on epistemic is no better or worse than focusing purely on aleatoric uncertainty. These are two different areas. The paper should address the difference of epistemic (parametric) vs. aleatoric (intrinsic) uncertainties.

    1.2 Could the paper also include a comparison between uncertainty-based work and confidence-conditioned work (this paper)?

2. Last sentence ("existing Markovian policies that can only act according to a fixed level of pessimism") of Section 4.2 is a false claim, which shows the paper does not cover a thorough literature review. One NeurIPS paper (Tactical Optimism and Pessimism for Deep Reinforcement Learning, https://arxiv.org/abs/2102.03765) uses the idea of dynamically apply the level of conservatism (optimism/pessimism).

3. The Gridworld example in Section 7.1 is unconvincing. The idea of the setup is similar to the Cliff Walking example in Rich Sutton's RL book, but the reward dynamics is weird: entering a lava state (a dangerous state) only receives 0 reward for the remaining trajectory and without further penalty.

    3.1 Should the paper use a large negative reward for dangerous states instead of 0?

    3.2 The paper only compares the optimal path between CQL and CCVL. However, a shorter path does not always mean a good path (like the SARSA vs. Q-learning performance in the Cliff Walking experiment). It would be more convincing if the paper includes a comparison plot of CQL and CCVL on cumulative reward.

4. Missing study on weight alpha. The paper changes from tuning conservatism "delta" to tuning weight "alpha" from my understanding. I believe the alpha is important in CCVL. As in Section 4.1, why tuning hyperparameter \alpha in CCVL is easier (or better in any aspects) than tuning an "opaque" hyperparameter (degree of conservatism)? Could the paper clearly explain or elaborate its statements?

    4.1 Also missing alpha in Table 3: Hyperparameters setup

5. As in Table 4 - 6 in Appendix B., the CCVL shows a level of minor to moderate improvement in the majority of the selected Atari games. However, the variance of CCVL is way larger than CQL in all games in Table 4 settings, and way larger than CQL in some games in Table 5-6 settings. Given the level of improvement of CCVL, I believe the introduction of large variance is unacceptable.

    5.1 The paper should include an explanation of the large variance.

Minors:
1. repeated article "a" in the first paragraph of Section 1
2. "delta" -> "\delta" in the first paragraph of Section 4 
3. mixed use of symbols of "Pr" vs. "\mathbb{P}" in Section 4
4. References order is confusing, making it hard to follow. It's neither alphabetically ordered nor first-appearance ordered. May I ask what is the reference setting?

Questions:
1. For Theorem 6.1, what is the drawback of only showing lower-bound on states than state-action pairs?
2. In Appendix A.2, could the paper include more detailed steps of solving inner-minimization over Q while proofing Theorem 6.2?



clarity,_quality,_novelty_and_reproducibility: Quality: The idea is interesting. However, it currently feels like a stretch to call the paper complete due to the aforementioned problems.

Clarity: I like that all notations are clearly defined. The paper reads clearly overall. The flow of content is reasonable as well.

Novelty: The idea of flexible conservatism in conservative methods under offline RL is novel.

Reproducibility: The pseudocode is clear, but it might still take some effort to reproduce the results as the equation 4/5 is somewhat complicated.
summary_of_the_review: I really like the idea of using flexible or dynamic level of conservatism/optimism/pessimism in RL. I feel there is an ongoing trend in this direction, in both online and offline RL. However, because of the flaws in the empirical study of the paper, I recommend the paper to be rejected at this point.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel approach to estimating lower bounds of $Q$-functions. Based on the approach, the authors propose CCVL and empirically justify that CCVL outperforms several offline RL SOTAs.
strength_and_weaknesses: $\textbf{Strength:}$

The topic of offline RL is essential to the field. The quantile estimation and confidence adaptation proposed in this paper are novel to me and perform well empirically.


$\textbf{Weaknesses and Questions:}$

There are a couple of unclear points to me. 
$\textbf{Q1:}$
The authors may want to specify the randomness source in equation (3), as $Q^*(s, a)$ is itself deterministic, so the probability that $Q^*(s, a)>q$ is either $0$ or $1$. I assume that the probability bound means the bound on some estimated $Q^*$ given the dataset, or is there a typo in (3)?

$\textbf{Q2:}$
Which objective ((4) or (5)) does the authors adopt in the empirical comparisons? Is the policy in (9) adopted in empirical comparisons? Does the uniformly sampled $\delta$, $\delta_1$, and $\delta_2$ affect the lower bound estimation in Algorithm 1, as the right-hand side does not align with (4)?

$\textbf{Q3:}$
In addition, the theorem feels insufficient to me. What is the benefit of utilizing different confidence levels in the target estimation of (4)? Can the authors justify their benefit compared with standard pessimistic methods in (1)? 

$\textbf{Q4:}$
In the grid world experiments, how does CCVL learns the optimal trajectory given that there seems to be no coverage of such trajectory in the dataset (illustrated in the leftmost figure of Figure 1)? Is the performance of CCVL desirable, as offline RL is supposed to rely more on confident regions from the dataset?

clarity,_quality,_novelty_and_reproducibility: See questions above. I might have missed it but I did not find the code for experiments.
summary_of_the_review: The paper seems interesting but there are a few unclear points to me. I will raise my score if the authors can address my questions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a new offline RL algorithm, by learning the value function conditioned on the confidence level. The proposed algorithm learns a Q function parameterized by the confidence level of future backup and executes an adaptive policy during test time. The paper shows the learned Q function as a high-confidence bound of the optimal Q function. Empirical results in Atari games show the benefit of the proposed algorithm over CQL and REM.
strength_and_weaknesses: Strength:
1. The idea of confidence-conditioned value function learning is a very novel way of encoding confidence levels into offline reinforcement learning algorithms. This is different from all previous papers using explicit or implicit confidence-based penalty in Q learning, as CCVL realize a more ambitious learning goal of learning Q at all confidence level.
2. At the test time, CCVL executes a non-Markovian policy which adaptively changes the confidence level \delta. This gives an additional capacity for test time learning/tuning for the policy. It is also a novel contribution of this paper.

Weakness:
1. Additional ablation study and more description of methods in existing ablation studies.
 - The implementation use Eq (5) instead of Eq (4), what will be the performance of CCVL with the update rule from Eq (4)?
 - A more detailed description of how AEVL and Fixed-CCVL are implemented should be included in the appendix.
2. The proposed algorithm requires additional sampling or computation. It would be better to conduct an analysis of how much these parts cost additional computation compared with baselines.
 - Eq (5) need to hold for all \delta. So the CCVL needs to perform the learning over all \delta or sample \delta enough times (as in Alg 1) for convergence. 
 - Confidence adaptive policy requires more computation at the test time as well.


Minor comments:
1. In section 5, it is unclear why n(s) is approximated by phi(s)^T phi(s). According to the argument in O’Donoghue et al. (2018), ignoring the difference between actions, it should still be phi(s)^T \Lambda^{-1} phi(s), where \Lambda is the empirical feature covariance matrix.
2. Eq (4) and Eq(5) have ambiguity around the operator max_{\delta_1, \delta_2}. There should be a parenthesis to clarify the scope.
3. \hat{Q} is not well defined without showing the limit exists or contraction, as the Bellman operator is modified.
clarity,_quality,_novelty_and_reproducibility: Clarity: There are some clarity issues with the paper. See minor comments in the previous section.

Quality: This paper gives theoretical guarantees for finite state-action MDP and empirical study in Atari games. This is solid work, though the empirical study can be improved by more benchmarks and baselines. 

Novelty: The proposed algorithm has two novel ideas: confidence-conditioned value learning and adaptive policy. 

Reproducibility: No code provided. Hyperparameters and implementation details are provided.

summary_of_the_review: A good paper with several novel ideas. Some clarity issues and additional studies need to be settled before acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes CCVL (confidence conditioned value learning), an offline RL method that learns value functions parameterized by the "confidence level". The main algorithm is adapted from CQL and anti-exploration bonus, with explicit modifications (supported by theory) to guarantee that the learned Q-values are lower (or upper) bounds with the probability at least being the selected "confidence level". Experiments were conducted in a toy grid-world problem as well as Atari benchmark and shows favorable performance compared to CQL and other baselines. 
strength_and_weaknesses: **Strengths:**
- Problem is well-motivated, mathematical descriptions and derivations are detailed. 
- I really like the toy gridworld problem, which is illustrative of the expected effect of using the proposed algorithm. 

**Weaknesses:**
- Some theoretical details need clarification:
  - Could you clarify the proof for Lemma 6.1, I don't think I follow why $\delta_1,\delta_2 \le \delta'$ "implies $Q(s, a, δ) ≤ Q(s, a, δ' )$, as desired". 
  - How does setting $\delta_1 = \delta$ in Eqn (4) reduce it to Eqn (1), the numerator of the fraction under the sqrt is $\log(1/\delta)$ instead of 1
- Is $\alpha$ still a hyperparameter, how should it be chosen, and what is its interpretation now that $\delta$ "confidence" is introduced. Since in CQL $\alpha$ is used to implicitly control the confidence level in the pessimistic Q-values. 
- It's best to make clearer how much of the observed performance gains in experiments is from the proposed learning confidence-conditioned values, vs from some of the empirical/engineering decisions in Sec 5 practical algorithms (IQN, approx inverse visitation). Perhaps if there's a simpler (maybe tabular?) problem that doesn't have these deep RL complexities that would strengthen the results and help the reader understand the contribution better - and help to show what the learned Q-values are actually doing. 
- There are some typos throughout, "hyperparamters" and "suprenum"
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is largely well-written, with mathematical details and experiment descriptions. 

**Quality:** Experiments adopted best practices (IQM) in reporting RL benchmark results. 

**Novelty:** As far as this reviewer is aware, the proposed method is new and not seen in previous work. 

**Reproducibility:** Implementation is currently not provided with the submission. Since RL results tend to be highly variable depending on the implementation, and there are a lot of moving pieces in the proposed approach (confidence sampling, hyperparameters, quantile nets, inverse visitation), I highly encourage the authors to share code after publication. 
summary_of_the_review: Overall the paper is well-written with good methodological contribution and theoretical analysis, supported by benchmarking results. Some theoretical details can be clarified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Claims: The paper introduces an algorithm that learns a confidence-conditioned lower bound for $Q$ values from an offline dataset.  This allows the algorithm to adaptively change its degree of conservatism online/at test time based, rather than having to choose a conservatism hyperparameter in advance.  One reviewer described this as allowing for simultaneously learning with many confidence levels at once.

(b) Strengths: The main strength of the paper is the novelty of learning an approximation that allows for online tuning based on how different the target distribution of trajectories generated by the algorithm appears to be from the distribution of trajectories in the offline dataset.  The empirical evaluations give evidence that this approach is fruitful.

(c) Weaknesses: Most reviewers noted some clarity issues, that seem to have largely been fixed in an intermediate draft.  One reviewer had serious concerns about the empirical evaluation, especially the dynamics of the gridworld environment.
justification_for_why_not_higher_score: n/a
justification_for_why_not_lower_score: I agree with the reviewers that this paper contributes novel ideas in the important area of offline reinforcement learning.  Reviewer gE4Y's concerns about the empirical setup are well-taken, but I don't consider them to be a fatal flaw.
Decision: Accept: notable-top-5%