Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper revisits the design of ensemble critics in offline RL. The authors argue that the common design where critics in the ensemble sharing the same, pessimistic target function in learning can lead to actually optimistic critics. The authors analyze this phenomenon theoretically under the NTK assumption, and present toy simulation examples. The authors further use this insight to design an offline RL algorithm MSG, which gives SoTA results on common offline RL benchmarks. In these experiments, they show that the separating the targets is a key to the algorithm's superior performance.



********** Comments for Rebuttal *************


Thanks for the rebuttal and the clarification. While they address some of my concerns, my main concern stay the same. As stated in the original review, the main issue I have is "whether the proposal here is sufficient to design a full offline RL algorithm or just provide an important note on implementation choice". The rebuttal also points out "it is the objective of our work to advocate for relying on uncertainty estimation as the main source of pessimism for offline RL".

Let's examine this question from two aspects based on the paper and rebuttal . 

From the theoretical side, the paper provides that Theorem 3.1, which compares the iterates of $Q_{LCB}$ of Independent Targets and the Shared Targets. However, it does not show "how"  good pessimistic the estimate of Independent Targets is. In the review, I mentioned "in general optimizing a pessimistic critic or being more pessimistic does not imply good performance in offline RL." because whether a pessimistic critic is useful or not depends on how "tight" the under estimation is and where it is pessimistic. Being more pessimistic is not always good, e.g., estimating all values as $V_{min}$ is pessimistic but it's obviously useless. The current theory does not provide enough insights to how good the pessimistic estimate is, or how good the learned policy based on such value estimate will be. This is why I said "the significance of the theoretical results are rather limited" in the review. 

For empirical side, I think to demonstrate the authors claim, it is necessary to show that SoTA can be achieved with $\alpha=0$. However, the current results do not support that fully. While I agree that in Figure 3, $\alpha=0$ is among the best performing results. I also think that Figure 3 does not provide a conclusive answer, as it is missing results of larger $\alpha$ value for $\beta=0$, as there is an increasing trend. This was pointed out in the review. It's also hard to compare Fig 3 and Fig 2 directly

I think the failure of using $\alpha=0$ in simpler mujoco domains is actually showing that the proposed approach "alone" might not be sufficient to provide enough pessimism "broadly". I do not agree with the rebuttal's statement that these environments can be solved by behavior cloning. All the methods the authors listed there are not pure behavior cloning, which mimics all actions, as all of them perform some reasoning what actions are better based on the rewards. Therefore I don't think this is a good excuse of using the CQL term here. Yes, I agree that the proposed method works with $\alpha=0$ with Antmaze which is considered as a harder domain, but is it because of the reason that the authors mentioned? if this is the case, why does not perform well in simpler domain. Or is it because of something that is related to the structure of Antmaze environment and dataset? Currently, we don't have sufficient evidence to tell.

Thus, I think that currently the paper provides insufficient results to show that the proposed uncertainty estimation "alone" can achieve SoTA offline RL results. Nonetheless, I also think that this paper provides more than sufficient reasons to show that it would be a good design choice to improve an existing value-based offline RL algorithm. Therefore, I keep my original recommendation.


strengths_and_weaknesses: Strengths: 
1. This paper presents an overlooked finding. 
2. It provides some theoretical reasons to back it up. It also provides empirical validation. 
2. The writing of this paper is clear and the experimental results are thorough. 

Weakness: 
1. While the authors provide explanations of why the common usage of Shared Targets may lead to optimism, the current results are not conclusive. In particular, while existing offline RL implementations use Shared Targets, the pessimism of Shared Targets is not the main source of pessimism but rather an implementation detail. Therefore, it is unclear whether the proposal here is sufficient to design a full offline RL algorithm or just provide an important note on implementation choice. This factor limits the significance of the paper. 

2. It is unclear what data assumptions are needed for the proposed method to work properly as an offline RL method.
questions: My main concern is whether the independent ensemble idea here is indeed sufficient to induce adequate pessimism for offline RL as the author claim. Here're few reasons why it does not seem to be the case. 

1.  Theorem 3.1 compare the results of a pessimistic policy evaluation procedure. The authors argue that using Shared Targets may lead to optimism, compared with the Independent Targets version. However, significance of such a comparison is quite limited for the following reasons. First, to my knowledge, I do not know of existing offline RL methods introducing pessimism purely based on what the authors describe in the Shared Targets update rule. While that update rule are used in some implementations, that is usually not the main source of pessimism that the offline RL algorithm uses but just a detail. So the theoretical comparison here may not be realistic, even if we acknowledge the NTK approximation part. Second, the authors study a version based on E[Q] - sqrt{V[Q]}. A more common implementation I know is pointwise pessimism, i.e. min_i Q_i(s,a), which e.g. is used in CQL and TD3+BC implementation based on double Q networks. I doubt the comparison would carry through to this more realistic case, because using pointwise pessimism would be strictly more pessimistic than the Independent Targets.

2. The above theoretical analysis is limited to offline policy evaluation. It's unclear how it can be translated into policy optimization, as in general optimizing a pessimistic critic or being more pessimistic does not imply good performance in offline RL. Therefore, the significance of the theoretical results are rather limited in my mind, unless the authors discuss in details how the proposed procedure affect the performance of the learned policy.

3. Another reason why the pessimism induced by Independent Training is not sufficient is that in MSG, the authors introduce the difference term H(\thteta^i) as part of the objective. The authors write "Empirically, as evidenced by our results in Appendix A.2, we have observed that such a regularizer can be necessary" This seems to imply that  the pessimism of the Independent Target can be serve as the main source of pessimism. In particular, the authors suggests using this extra regularizer H(\theta^i) is necessary, when the data distribution is narrow. Since addressing the lack of coverage is the main focus of offline RL (say compared with off-policy RL), this shows again the proposed pessimism is not sufficient for offline RL. 

4. The above insufficiency is partially shown in the experimental results as well. For instance in Figure 3, MSG does not perform the best when alpha=0. In the experiments of beta=0, the authors should also try even larger alpha to get a fuller picture of the performance and how hyperparamters interact. From these results, it is unclear whether the good performance of MSG is due to using the ensemble uncertainty as the primary source of uncertainty.

In summary, while this paper presents an interesting observation, showing that using Independent Targets can be more pessimistic than using Shared Targets, such results do not directly address the main research question of offline RL for the reasons above. (But the title "Why So Pessimistic?" seems to want to convey that we should be less pessimistic?) It is therefore hard for me to position this paper. This paper presents a good and thorough empirical study on how a certain design knob in implementation can affect the performance. However, the results here do not provide enough insights to design offline RL methods more broadly outside the tested implementation here.



limitations: The authors well discuss the limitation such as the extra complexity needed by the proposed method. However, in my view, the results here are more limited than what the authors claim for the reasons above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies ensemble-based pessimism in offline RL from both theoretical and empirical aspects,
- Giving an analysis mathematically through NTK to show shared pessimistic targets can paradoxically lead to Q-estimates which are in fact optimistic.
- Proposing MSG that trains each Q-network independently, and conducts experiments in D4RL and RL Unplugged tasks.


strengths_and_weaknesses: Strengths
- Formally analyze the offline RL methods based on Q-ensemble pessimism on infinite-width neural networks, and shows the pessimism term may become positive in shared targets. 
- Empirically verifies the effectiveness of the algorithm by combining Q-ensemble with CQL. 

Weaknesses
- The NTK assumptions in section 3.1 and the Gaussian assumptions in section 3.2 seems limited in broader value iteration.
- The Q-ensemble needs to combine with CQL to obtain reasonable performance. The use of CQL makes it difficult to analyze the source of performance improvement. However, there exist several methods (e.g., EDAC in NeurIPS 2021 and PBRL in ICLR 2022) that perform Q-ensemble for offline-RL with purely uncertainty. 
- The experiment results are not complete for D4RL benchmark.
questions: - In line 191, why the pessimism term of method 2 may become positive, while it cannot be positive in method 1?
- ln line 781, how to get Eq. 20, 21, 22 from Eq. 19?
- ln line 791, how to get Eq. 31, 32 from Eq. 30?

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper discusses the uncertainty estimation in RL, which is an alternative to induce pessimism in offline RL. Though uncertainty estimation through ensembles has been proposed in the offline RL literature, this paper points out a critical flaw in how to incorporate the LCB in the actor-critic based algorithm. It shows theoretically that the previous algorithms, which regress different Q functions to the shared  pessimistic target values, and does policy evaluation based on the LCB could sometime leads to over-estimation of the Q function. To address this, the paper proposes a simple fix, that is in the Bellman backup stage, instead of regressing the different Q values to the shared LCB estimate, just regressing them to independent Q target. Empirically, it shows better performance in challenging tasks that require stitching. Beyond this, the paper also examines how different efficient ensemble method work in RL setting, and it seems there still exist a large gap in the performance compared with deep ensemble, which opens up more interesting questions in efficient ensemble methods in the RL setting. 
strengths_and_weaknesses: Strength:
[1]. The paper is well-written and very easy to follow.
[2]. It discusses a major flaw in how to incorporate LCB estimate in offline RL algorithms, which seems being overlooked in the literature, but empirically seems make a big difference. The theoretical claim is well-supported. 
[3]. It has a comprehensive set of empirical studies, which covers various aspects about the applicability of the method, such as the ensemble size, the hyper-parameter sensitivity. I do appreciate the authors' effort in discussing how to transfer efficient ensemble method in supervised learning setting to the RL setup, to make it more computationally efficient, though some negative results there.

Weakness:
[1]. Maybe I am misunderstanding something, but i do feel some of the claims and findings are not explained in a super clear way, see Questions section for details for this.
[2]. The experiment section does show that incorporating the independent target leads to the better performance in the challenging tasks, it would be great to see that this is result from better LCB estimate. We see the overestimation issues in the toy task, and it would be really helpful to see in the challenging tasks, that shared target does lead to over-estimation, which is the reason that the method helps. 
[3]. Section 4.2 seems a little bit out of picture. As it seems alpha=0 works great in most cases, the authors state that it might help in some narrow data regime, is any empirical study supporting this?

questions: [1]. The comparison of independent and shared target comparison is interesting. Could the authors summarize the literature in a clearer way in how they do ensembles, and mark the subtle difference, as this is missing in Section 2, but these subtle differences seem make a big difference, as what is discussed in this paper.
[2]. From L154-155, it seems the shared targets are doing this "doubling pessimism" in both the policy evaluation and learning step, which seems actually a little weird to me, could the authors point out the exact literature that is doing this. As I can imagine either (1). we are doing the independent target, or (2). we do the evaluation pessimistically, but in learning we take the mean of the Q, compared to LCB. Just curious that do the authors try (2)?
[3]. Followed by [2], it seems the shared target do seem doing more pessimism, but theoretically it could lead to over-estimation. Any comments in bringing this intuition to the theoretical results? it seems related with the correlation of the transition (s,a) paris and properties of the C matrix. Could the authors comment more on when the shared target leads to over-pessimism or optimism, and the practical guidelines?
[4]. The toy example in Section 3.2 is a little bit confusing to me, what is the rationale of using the policy that play the same a in the next state? 
[5].The discussion in L327-337 shows that the shared target has worse performance as ensemble sizes decreases, any explanation on this? 
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper observes a problem in existing pessimism estimation in offline RL using ensembles: using shared targets for all ensembles updates. The paper instead proposes to update each ensemble individually and apply the pessimism at policy updates. The paper derives the update form of both methods in the NTK setting and shows that the update method with shared target could even result in optimism, which is also shown with some synthetic simulation data. Finally the paper evaluates the proposed method in several offline RL benchmarks and show its empirical competitiveness. 
strengths_and_weaknesses: ## Strength

1. Overall the paper is well written, easy to follow, and the technical part seems correct. 

2. The paper makes a good observation of the existing methods for offline RL when they update the Q-values for ensembles: from hindsight one really does not need to incorporate the pessimism into the function update procedure, but instead just apply pessimism during policy update. This also seems to agree to the theory RL algorithms: one can just perform the regular bellman updates (or perform elimination in version space algorithms) and define policy with LCB or take minimum over the remaining set of functions (for pessimism) (for example, [1]). 

3. Although it may not be obvious under what kind of conditions such that in the NTK setting, using the shared target could result in optimism, the following subsection provides good evidence that that indeed could happen. It could be better to provide some more intuitive scenario or even a closed-form construction. 

4. The paper provides extensive and convincing experiments, including a) good ablation experiments which contains the different kinds of shared target updating methods (such as shared-LCB Ens., Shared-Min Deep Ens, and with a different number of ensembles)
. b) The paper tries many different hyperparameters for the baselines, so the baselines seem to be fine-tuned for the final presentation of the results. c) The experiments are performed on extensive benchmarks. 

## Weakness

1. The theoretical results provide very good intuition into the problems of the previous pessimism estimation in offline deep RL methods, but since the result is based on the NTK setting, it still has some gap between the practical situations. 

2. The result presented in table 1 has different hyperparameter for different tasks, which likely undermines the empirical merits of the proposed algorithm. 

### references

[1] Xie, Tengyang, et al. "Bellman-consistent pessimism for offline reinforcement learning." Advances in neural information processing systems 34 (2021): 6683-6694.
questions: ## Major Questions

1. This question is less about the proposed algorithm itself: for the shared target update methods, the target itself contains a pessimism term: it's already a LCB estimation. Why, for policy update, another LCB term is added into the $Q_{LCB}$ estimation, why not just directly using $\mathbb{E}[Q_{\theta^i}]$? It seems like it's doing pessimism twice? Although it looks like removing the LCB term in $Q_{LCB}$ won't affect the theoretical result by much (you just subtract the backup term by 1? which of course leads to optimism more easily).

2. In addition to the existing baselines, it may seem like the following baselines could also be of interest? For example, a) we take the min over the ensemble with individual targets? b) every ensemble is bootstrapped from their own target but we also subtract with a common lcb term? 

### some minor problems

- The formula in the algorithm box in section 3.1 overflows. 

- Why the $\gamma$ in section 3.2 is chosen to be very small (instead of 0.9)?

- The y-axis are inconsistent in Fig.2 and Fig.3. Also in Fig.3, what does each dot mean? It seems a little bit hard to interpret the plot.

- Why is line 763 true, for $\theta_{lin}^{(0)} = 0$?
limitations: 1. The overall algorithm has good intuition and motivation, but the introduction to the additional term in section 4.2 looks irrelevant to the rest of the paper. From the experiments, this term seems crucial to a good performance of the algorithm and thus unavoidable in the current version. Although it makes sense that some kind of regularization may be needed for unseen action, this additional term indeed undermines the overall message a little bit. 

2. The ablation of using a more condensed surrogate for ensemble is a good experiment, and as the paper already suggests, it would be better if a more efficient way of pessimism could be derived, which seems beyond the scope of this paper.
review_title: null

Meta-review: metareview: The paper identifies a common flaw in pessimistic algorithms related to the use of shared targets, and propose an alternative based on independent targets that mitigate the overly-optimistic estimates. The rebuttal has addressed a number of concerns raised by the reviewers, and in particular, the negative reviewer qbbN acknowledged that 

> ... the proposed idea here would make an existing algorithm that uses e.g. double Q networks (which is quite common) and also other main pessimism (like value penalty or closeness to behavior policy) to perform better. Thus, the insight here can be quite useful in practice.

That said, the reviewer is still concerned about the framing of the work

> the paper does not provide sufficient evidence (theoretically or empirically) that the proposed pessimistic estimate based on Independent Training "alone" is sufficient to design a SoTA offline RL algorithm [which the paper claims to]... I think that the paper needs to provide stronger evidences or changes the framing.

Given the strong support from other reviewers, the AC is leaning towards acceptance, but strongly recommend that the authors change the framing of the paper to honestly reflect the contributions of the work. 
confidence: Less certain
Decision: Accept