Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper agues that methods, which they call "support constraint", consisting in assuming that any action outside the training yields negative outcomes, lead the agent's training to be overly pessimistic. In contrast, they propose MSG an algorithm where a lower bound of the value function is estimate from ensembles of networks. They provide some theoretical arguments on infinite-width neural networks and validate their methods on many applications, ranging from illustrative toy examples to classic continuous state-action spaces offline RL benchmarks.
main_review: Offline RL is certainly an important area of research and there is still plenty to do, since the existing algorithms have all shown their limits. However I believe that the submission falls short on several crucial points that I enumerate below:
* There is a terminology confusion with the word pessimistic. 
   * Offline RL inductive bias: Offline RL algorithms rely on the assumption that the control policy(ies) is(are) much better than average, and therefore that omitted actions are likely to be underperforming. This is a desirable feature, and the ex machina argument made by the authors that "it would be preferable if we could place more trust into the predictions of value networks beyond the training dataset" is not supported. And this is not what the authors propose to do: they simply propose a way to estimate uncertainty and to enforce this assumption: LCB means that the training is going to be conservative.
   * Pessimistic meaning in the offline RL literature: Usually, pessimistic refers to the opposite of optimistic exploration: methods that distort the reward/value function to induce pessimism in the face of uncertainty. It is generally used in opposition to methods that conservatively constraint the policy search [Petrik2016], [Fujimoto2019], [Laroche2019]. Most (all?) RL algorithm can be viewed as policy iteration: loop over policy improvement and policy evaluation. Pessimism refers to pressure on the policy evaluation step, and constraint of the policy search refers to pressure on the policy improvement step. The method developed in the paper applies pressure on the value function and is therefore pessimistic, hence the confusion for the readers. There is even a third approach consisting on early stopping the loop [Brandfronbrener2021].
* The novelty is limited and the positioning incomplete. As pointed out by the authors, the use of deep ensembles for offline RL is not novel. The authors claim that the networks share target. Usually yes, but not always: Algorithm 2 of [Osband2018] is training each network independently. And when they train them all together, they share a form of pessimistic target (reusing the paper terminology: independent double Q, shared LCB, shared min), not the expectation (shared mean), otherwise it does not make sense!
* The theoretical analysis is not convincing. Following my previous point, it would have made the analysis much stronger if the other forms of pessimistic updates were accounted for and analyzed. It is straightforward that using a common target will provide a collapse of the inter-network variance. The use of infinite width NN is a big hammer for such a small nail. Figure 1 reports an experiment but its result is inconclusive since all pessimistic updates deliver the same lower bound.
* The authors claim that shared LCB, shared min, CQL were unable to train effective policies, although these have been reported in the past. This, combined with the lack of compelling arguments on the theory side, cast doubt in the empirical results. Also, "the other data setting could not adequately differentiate between competitive methods" is not an argument for not reporting results.

[Petrik2016] Petrik, M., Ghavamzadeh, M., & Chow, Y. (2016). Safe policy improvement by minimizing robust baseline regret. Advances in Neural Information Processing Systems, 29, 2298-2306.

[Osband2018] Osband, I., Aslanides, J., & Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv:1806.03335.

[Fujimoto2019] Fujimoto, S., Meger, D., & Precup, D. (2019, May). Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (pp. 2052-2062). PMLR.

[Laroche2019] Laroche, R., Trichelair, P., & Des Combes, R. T. (2019, May). Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (pp. 3652-3661). PMLR.

[Brandfronbrener2021] Brandfonbrener, D., Whitney, W. F., Ranganath, R., & Bruna, J. (2021). Offline RL Without Off-Policy Evaluation. arXiv preprint arXiv:2106.08909.
summary_of_the_review: I recommend reject for the reasons detailed in the full review.

If the authors could clarify the following things, I would consider raising up my score:
* How does their algorithm differs from Algorithm 2 of [Osband2018]?
* Why would the LCB of independent networks would have a better uncertainty estimate than an ensemble network learnt on common LCB targets?
* Why were the authors unable to train effective policies of well established algorithms?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: There is a good literature for off-line RL. However, this  is reviewed at all. For example, LSPI (model free), LAMAPI (model-based), pseudo-MDPs (model-based) and RKHS embedding (model-based). The paper went straight to discussing the reduction of the value estimation error constraint. However, most of the algorithms use the error to directly learn the parameters. It is not necessarily formalized as constraint. 
main_review: The title of the paper: why so pessimistic? Is a bit little misleading. Actually most RL exploration methods are being the opposite: optimistic. 

One major motivation of the reviewed paper is to “some of the unique challenges of uncertainty estimation in reinforcement learning ”. Could be explicit on this? The message/answer is not super clear after reading the paper. The DLTV paper has an observation that there are both intrinsic and parametric uncertainties going on with DRL. This shows the exploration for DRL is very challenging.  The DLTV paper uses distributional RL to estimate the upper confidence and lower confidence bound from the distribution of Q values, also using neural networks. 

The presentation of the paper can be improved. Some times buzz words just come out without defining, like “deep ensembles”, "efficient approximation to ensembles"....

The results are evaluated on two ant mazes. The CQL paper performed pretty bad from Table 2. Did you try the other baseline? With only baseline and it’s poor, it is hard to see the proposed algorithm is strong. 

Questions:
What is the connection of using ensembles and distributional RL for uncertainty estimation? In a superficial sense, both use a number of agents. The only difference is how they form an uncertainty estimator. However, identifying their connections and differences are interesting, isn’t it? For distributional RL, e.g., see the DLTV algorithm:
https://arxiv.org/abs/1905.06125

Actually the method proposed in this paper (estimating LCB, see Section 4.1) is very similar to DLTV. The formula is almost the same. Compare the equation above Sec. 4.2 with eq 4 of the DLTV paper. The only difference is that this is implemented using ensembles instead of distributional agents.  

The statement “update of each member’s update is completely independent from each other”. Here some clarification is necessary. By independence, do you mean the learning rule sense? Do you have shared parameters between the members? 

Beyond Method 1 and Method 2, have you thought about using shared maximum of the ensemble? See eq 8 of the ACE paper for example:
https://arxiv.org/abs/1811.02696 
When coming to ensembles, the crucial question is what operator to choose to combine these estimate. The use of mean or max is a fundamental question to investigate. 



Minor:
Some reference links are missing. 

Refs:
LSPI
https://www.jmlr.org/papers/volume4/lagoudakis03a/lagoudakis03a.pdf

LAMapi:
https://sites.ualberta.ca/~szepesva/papers/lamapi.pdf

RKHS:
https://arxiv.org/abs/1206.4655

Pseudo-MDPs:
https://sites.ualberta.ca/~szepesva/papers/ieee_adprl2014.pdf
summary_of_the_review: The paper addresses exploration with DRL algorithms, and explores estimating confidence interval of Q values using ensembles. Some discussion on the use of ensembles, the connection and difference from distributional RL, and comparisons to a good baseline (set) can make the paper much better and well supported.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provided a model-free algorithm for offline-RL. It provided Theorem 5.1 to justify its uncertainty quantifiers. It also conducted experiments to validate its algorithm.
main_review: Strengths. The algorithm in this paper is model-free, which means it can be applied to realistic settings.
It used 

Weakness. The theory part of this paper seems unfinished. The detailed derivation of the std should be included in the proof, and there should be more details, like descriptions of the network initialization,  in the description of the theory part 

Typo in Line 8 of page 4. "This an ..."
summary_of_the_review: Although this paper proposed an algorithm that can be applied to real-life applications without imposing strong structural assumption to allow uncertainty quantification, the theory part of this paper should be improved to be more complete, rigorous and self-contained.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper propose to study ensemble methods for offline RL in the NTK framework. They derive 
main_review: The paper looks interesting, so are the approach, the theory, and observations. 

# Revisions/questions/remarks
  * Roughly half of the papers refer to non peered reviewed papers (arXiv)
  * Hasn't the D4RL dataset been updated since Kumar et al. publication (CQL) which could explain the differences observed by the authors?
  * Lack of rigor in the theorems: the authors should have a formal theorem followed by its interpretation on which they base their intuitions
  * Lack of rigor in the results presentation: what bold format means? Maybe add a statistical significance test.
  * Too many claims not well supported
  * Section 5.2: wouldn't it have more impact if you used the git attached to Lee et al. 2019 in Section 5.2 to match the framework the authors are working on?

# Some typos
Lot of typos have to be corrected, here are some
  * p.1: "confident", citation missing
  * p.2: "data(Guo et al., 2017)" (missing space), "only has", "success"
  * p.5: Eq. 4 missing "i", "in the offline dataset. from the dataset."
  * p.8: footnote of p.7, "otherwis The"
  * p.9: "computationl efficieny"
summary_of_the_review: I found the paper quite hard to follow and overall there is a lack of rigor. However, looking at the experiments, there might be something to explore.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel method to utilize ensemble in quantification of uncertainty in offline RL, where the particular structure of dynamic programming and cumulative error are unique and interesting. Some theoretical justification emphasizes the ability of independent TD targets to capture the variance of the prediction, hence provide uncertainty quantification. Simulations show the proposed method gets larger variance in low-confidence regions. In experiments in benchmark environments, the proposed method outperforms those in the literature. 
main_review: Strength:
The work proposes a novel method to use independent ensembles, which might be interesting towards a better understanding of uncertainty quantification in offline RL. 
The experiment results are encouraging, showing superior performance of the proposed methods. 
The theoretical intuitions are backed up with simulations, which helps readers better understand the ideas.

Weakness: I have several concerns that are listed below. 
Intuitively, the method of using independent initialization captures the randomness in the training procedure, i.e., conditional on the observed data. I was a bit concerned how such training uncertainty relates to the uncertainty of data with respect to the true value. Intuitively, the uncertainty often refers to the data being unable to capture the truth because of lack of samples. One possible relation is that latter uncertainty is large in regions with scarce data, where the former training uncertainty will also be large. I Since this is related to the core theme of the paper, could you explain more on such relations, or on the uncertainty this work targets at?
The writing of the paper might be improved. I think it would be better to have fewer emphasized bold-font sentences, so that readers can focus on really important points. There are also some missing citations and references as well as typos in the paper which hurts the reading. I would expect the authors to have a careful check and make the presentation more organized. 

There are some further questions:
What is the \pi in equation (3) in Section 4.2?
In theorem 5.2, the difference in ensemble variances is only discussed very vaguely. I don’t quite understand the meaning - it would be better if a direct comparison with equations is presented.
I am not sure whether the toy example and Figure 1 captures the advantage of MSG. Figure 1 shows the mean value and the standard deviations of the predictions, where the randomness can be viewed as from the training process, i.e., the random initialization, instead of the randomness in data. The crucial point should be that the estimations can provide a valid uncertainty quantification of the prediction with respect to the *true value*, so the shared LCB and shared min approach seems to directly target at the lower uncertainty bound (as observed in the paper). I would expect directly using the result of shared LCB/min can somehow provide a valid uncertainty quantification, without referring to the standard deviation induced by the training process. The approximation of the lower confidence bound should be more essential than evaluating the randomness in training process. Do you have more discussions on this issue?
Choice of hyper-parameters: In the experiment part, it would be better to add discussions on how the parameter beta and alpha are chosen. They seem to change across different domains and I think readers would expect some justifications on such choices (I only see ‘hyperparameter searching’, but not sure whether it’s cherry picking). Are the performances of both methods stable across different choice of hyper-parameters? 

Minor points: 
A missing citation in the introduction (page 1). 
Missing “.” in Section 4.2. 
Is there a missing superscript i for y(r,s’,pi) in equation (4)?
Missing reference in the caption of Figure 1.
summary_of_the_review: The proposed method to use independent initializations to capture the uncertainty is interesting, validated in simulations and performs well in experiments. However, the credibility of the work is compromised by the confusion in the type of uncertainty it targets at, which is the key theme. The writing could also be improved by another check to fix typos and missing references.
review_title: null

Meta-review: The paper attacks an interesting problem: accurately estimating uncertainties in action-value estimates in offline RL. It proposes a method based on ensembles of Q functions, where we alternately train an ensemble to estimate Q(s,a) for the current policy, and then adjust our policy based on the mean and uncertainty in this ensemble. By choosing mean + \beta * [standard deviation] as the basis for our policy updates, we can be either conservative (\beta < 0) or optimistic (beta > 0). The paper analyzes the ensemble training using the Gaussian process (NTK) view of deep nets.

The largest weakness of the paper is a lack of rigor in its analysis. While its main topic is uncertainty in Q estimates, the paper does not specify a valid probabilistic model on which such uncertainty estimates could be based. The theorems analyze only a part of the algorithm (policy evaluation), and don't take into account the interplay between this evaluation and any policy updates. The theorems also do not show that the computed output distribution is relevant to the actual uncertainty of the algorithm; e.g., they do not describe a prior for which the ensemble approximates the correct posterior (nor any other similar notion). Despite these omissions, the theorems are nonetheless presented as providing a reason to trust the output of the algorithm.

On the other hand, there definitely is valuable material in the paper; the experiments are interesting (and would be even more interesting if we could compare to some notion of a correct answer for at least the small ones), and the intuition and analysis could be enlightening if presented more clearly and formally, with a better description of the connection between theory and practice. Unfortunately, the paper as written doesn't enable the reader to accurately understand and assess the contributions.
Decision: Reject