Why so pessimistic? Estimating uncertainties for offline RL through ensembles, and why their independence matters.
Authors: Seyed Kamyar Seyed Ghasemipour
Shixiang Shane Gu
Ofir Nachum
Venue: ICLR 2022 Submitted (2022)
Keywords: offline reinforcement learning
batch reinforcement learning
ensembles
uncertainty estimation.

Abstract: In order to achieve strong performance in offline reinforcement learning (RL),  it is necessary to act conservatively with respect to confident lower-bounds on anticipated values of actions. Thus, a valuable approach would be to obtain high quality uncertainty estimates on action values. In current supervised learning literature, state-of-the-art approaches to uncertainty estimation and calibration rely on ensembling methods. In this work, we aim to transfer the success of ensembles from supervised learning to the setting of batch RL. We propose, MSG, a model-free dynamic programming based offline RL method that trains an ensemble of independent Q-functions, and updates a policy to act conservatively with respect to the uncertainties derived from the ensemble. Theoretically, by referring to the literature on infinite-width neural networks, we demonstrate the crucial dependence of the quality of uncertainty on the manner in which ensembling is performed, a phenomenon that arises due to the dynamic programming nature of RL and overlooked by existing offline RL methods. Our theoretical predictions are corroborated by pedagogical examples on toy MDPs, as well as empirical comparisons in benchmark continuous control domains. In the more challenging domains of the D4RL offline RL benchmark, MSG significantly surpasses highly well-tuned state-of-the-art methods in batch RL. Motivated by the success of MSG, we investigate whether efficient approximations to ensembles can be as effective. We demonstrate that while efficient variants outperform current state-of-the-art, they do not match MSG with deep ensembles. We hope our work engenders increased focus into deep network uncertainty estimation techniques directed for reinforcement learning.

Under review as a conference paper at ICLR 2022
WHY SO PESSIMISTIC? ESTIMATING UNCERTAINTIES FOR OFFLINE RL THROUGH ENSEMBLES, AND WHY THEIR INDEPEN- DENCE MATTERS
Anonymous authors Paper under double-blind review
ABSTRACT
In offline/batch reinforcement learning (RL), the predominant class of approaches with most success have been "support constraint" methods, where trained poli- cies are encouraged to remain within the support of the provided offline dataset. However, support constraints correspond to an overly pessimistic assumption that actions outside the provided data may lead worst-case outcomes. In this work, we aim to relax this assumption by obtaining uncertainty estimates for predicted action values, and acting conservatively with respect lower-confidence bound (LCB) on these estimates. Motivated by the success of ensembles for uncertainty estimation in supervised learning, we propose MSG, an offline RL. method that employs an ensemble of independently updated Q-functions. First, theoretically, by referring to the literature on infinite-width neural networks, we demonstrate the crucial dependence of the quality of derived uncertainties which ensembling is performed, a phenomenon that arises due to the dynamic programming nature of RL and overlooked by existing offline RL methods. Our theoretical predictions are corroborated by pedagogical examples on toy MDPs, as well empirical comparisons in benchmark continuous control domains. In the significantly more challenging antmaze domains of the D4RL benchmark, MSG with deep ensembles by a wide margin surpasses highly well-tuned state-of-the- art methods. Consequently, we investigate whether efficient approximations can be similarly effective. We demonstrate that while some very efficient variants also outperform current state-of-the- they do not match the performance and robustness of MSG with deep ensembles. We hope that the significant impact of our less pessimistic approach engenders increased focus into uncertainty estima- tion techniques directed at RI., and engenders new efforts from the community of deep network uncertainty estimation researchers whom thus far have not em- ployed offline reinforcement learning domains as a testbed for validating modern uncertainty estimation techniques.
the manner in
INTRODUCTION
Offline reinforcement learning (RL), also referred as batch RL, is the setting where we are pro- vided with dataset of interactions with a Markov Decision Process (MDP), and the goal is to learn an effective policy without further interactions with the MDP. Offline RL holds the promise of data- efficiency through data reuse and improved safety due to minimizing the need for policy rollouts. As a result, offline RL has been subject of significant renewed interest the machine learning literature (Levine et al. 2020).
One common approach to offline RL, known as model-free, uses value estimation through approxi- mate dynamic programming (ADP). The predominant algorithmic philosophy with most success in bDs-based offline RL is to limit obtained policies to the support set of the available offline data, with the intuition being that such constraints would reduce inaccurate value estimates since the ac- tions chosen by the policy are close to the observed data. A large variety of methods have been developed for enforcing such constraints, examples of which include regularizing policies with be- havior cloning objectives (Kumar et al. 2019 Fujimoto & Gu (02I), only performing updates on actions observed in (Peng et al. al. 2019 Nair air et al. 2020 Wang et al. 2020 Ghasemipour al.) 2021)

---

Under review as a conference paper at ICLR 2022
or close to (Fujimoto et al., 2019) the offline dataset, and regularizing to lower the estimated value of actions not seen in the dataset (Wu et al. 2019; Kumar et al. 2020 Kostrikov et al. 2021).
Support constraint forms of regularization correspond to an overly pessimistic assumption that any action outside the provided data leads to highly negative outcomes (Buckman et al., 2020). Instead, it would be preferable if we could place more trust into the predictions of value networks beyond the training dataset. Indeed in adjacent fields of A.I., for different tasks of interest, researchers have spent significant effort developing architectures with built-in inductive biases help neural networks not only obtain strong accuracy on the training set, but perform well beyond the data they were trained on: convolutional and residual networks for computer vision (He et 2 recurrent networks and transformers for NLP (Vaswani et al. (2017), graph neural networks for graph-based data (Li et al. 2015), and Nerf and Siren for effective signal modelling (Sitzmann al., 2020 Mildenhall et al., 2020).
However, trusting the predictions of neural networks beyond the training data is challenging ordeal. Many works have demonstrated that neural networks are prone to making high-confidence incorrect predictions outside the training set, even on i.i.d. test data (Guo et al. 2017). This is a particularly major issue for value estimation, since errors rapidly accumulate through dynamic programming and lead to catastrophic results. Thus, in order to trust our models beyond the training set, a potentially promising approach would be to obtain high quality uncertainty estimates on predicted action values.
In current supervised learning literature, "Deep Ensembles" and their more efficient variants have been shown to be the most effective method for uncertainty estimation (Ovadia et al.] 2019). In this work, we aim to transfer the success of ensembles for uncertainty estimation to the setting of offline RL. We begin by presenting MSG, an actor-critic offline RL algorithm leveraging an ensemble of Q-value functions. MSG trains N Q-value functions completely independent of one-another, and updates an actor network with respect to lower confidence bound (LCB) on action values obtained from the ensemble. By referring to the literature on infinite-widde neural networks, we theoretically demonstrate the critical importance of independence in Q-functions, a deviation from standard practice in RL which more often uses the full ensemble to compute identical target values for training each ensemble member. Our theoretical predictions are corroborated by experiments in a toy MDP, and their relevance to practical settings is also verified through benchmark experiments.
In established benchmarks for offline RL, we demonstrate that MSG matches, and in the more challenging domains, significantly exceeds the prior state-of-the-art. Inspired by this success, we investigate whether the performance of MSG can be recovered through modern efficient approaches to ensembling. While we demonstrate that efficient ensembles continue to outperform current state- or-the-art batch RL. algorithms, they cannot recover the performance of MSG using full deep ensem- bles.
We hope that our work highlights some of the unique challenges of uncertainty estimation in reinforcement learning, and the significant impact it can have the performance of algo- rithms. We also hope that our work encourages stronger engagement from researchers specializing in uncertainty estimation techniques- who typically use computer vision and NLP tasks bench- marks- - to use offline RL as an additional testbed. Our results reveal that offline RL presents unique challenges not seen in standard supervised learning, due the accumulation of errors over multiple back ups, and can be a valuable domain for testing novel uncertainty estimation techniques.
RELATED WORK
Uncertainty estimation a core component of RL, since an agent only has a limited view into the mechanics of the environment through its available experience data. Traditionally, uncertainty estimation has been key to developing proper exploration strategies such as upper confidence bound (UCB) and Thompson sampling (Lattimore & Szepesvári] (020). in which an agent is encouraged to seek out paths where its uncertainty is high. Offline RL presents an alternative paradigm, where the agent must act conservatively and is thus encouraged to seek out paths where its uncertainty is low (Buckman et al.) 2020). In either case, proper and accurate estimation of uncertainties is paramount. To this end. much research has been produced towards the end of devising provably correct uncertainty estimates (Thomas et al. 2015) Feng et al.) 22020: Dai et al. 2020), or, at least, bounds uncertainty that are good enough for acting either exploratory (Strehl et al. 2009) or
2

---

Under review as a conference paper at ICLR 2022
conservatively (Kuzborskij et al.) 2021). However, these approaches require exceedingly simple environment structure, typically either a finite discrete state and action space or linear spaces with linear dynamics and rewards.
While theoretical guarantees for uncertainty estimation are more limited in practical situations with deep neural network function approximators, number of works have been able to achieve practical success, for example using deep network analogues for count-based uncertainty (Ostrovski et al.) 2017). Bayesian uncertainty (Ghavamzadeh et 2016 Yang al. 2020)), and bootstrapping (Os- band et al. 2019 Kostrikov & Nachum, 2020). Many of these methods employ ensembles. In fact, in continuous control RL, common use an ensemble of two value functions and use their minimum for computing a target value during Bellman error minimization (Fujimoto et al.) (0118). number of works in offline RI. have extended this to propose backing up minimums or lower confidence bound estimates over larger ensembles (Kumar et al. 2019 Wu et al., 2019; Agarwal al., 2020). In our work, we continue find that ensembles are extremely useful for acting con- servatively, but the manner in which ensembles are used is critical. Specifically our proposed MSG algorithm advocates for using independently learned ensembles, without sharing of target values, and this import design decision is supported by empirical evidence.
The widespread success of ensembles for uncertainty estimation in RL echoes similar findings in supervised deep learning. While there exist proposals for more technical approaches to uncertainty estimation (Li et al. 2007 Neal 2012 Pawlowski et al. 2017), ensembles have repeatedly been found to perform best empirically (Lee al. 2015] Lakshminarayanan al.. 2016). Much of the active literature on ensembles in supervised learning is concerned with computational efficiency, with various proposals for reducing the compute or memory footprint of training and inference on
large ensembles (Wen et al. 2020 Zhu et al.] 2019; Havasi et al. 2020). While these approaches have been able to achieve impressive results in supervised learning, our empirical results suggest that their performance suffers significantly in an offline RL setting compared to deep ensembles, and even naive Multi-Head ensembles (Lee et al., 2015] Oshand et al., 2016 Tran et al. 2020) which are not considered to be as effective in the supervised learning celting(Havasii et al. 2020).
METHODOLOGY
Notation Throughout this work, we represent Markov Decision Process (MDP) M (S,A, r, P,7), with state space S, action space A, reward function r S x A + R, transition dynamics P. and discount y. In offline RL, we assume access to dataset of interactions with the MDP, which we will represent as collection of tuples D = {(s, a, ,,,,, term)} where is an indicator variable that is set to True when s' is terminal state.
Two neural networks with an identical architecture, trained in an identical manner, will make differ- predictions outside the training set when their weights are initialized with different random draws from the initial weight distribution. So natural question is, "Which network's predictions can we trust?" An answer this question can be motivated by referring to the literature on infinite-widue networks. When performing mean-squared error regression in the infinite-width regime, informally speaking, an intriguing property is that the distribution of predictions on unseen data is given by a Gaussian Process whose kernel function is solely defined by. 1) the architecture, and 2) the choice of initial weight distribution (Lee al. 2019). Hence, to leverage the built-in architectural in- ductive biases of value networks for offline RI, we can train policies to act with respect to the lower-confidence bound (LCB) of the derived Gaussian Process.
In what follows, we present our proposed algorithm, MSG, which leverages ensembles to approx- imate the LCB. Additionally, using the literature on infinite-width networks, we demonstrate the theoretical advantage of forming ensembles in the manner we propose, which deviates from the current use of ensembles in the reinforcement learning literature.
3.1 MODEL STANDARD- GRADIENTS (MSG)
MSG follows an actor critic setup where in each iteration, first estimate the Q-values of the current policy, and subsequently optimize the policy through gradient ascent on the lower confidence bound of action value estimates.
3

---

Under review as a conference paper at ICLR 2022
Policy Evaluation the beginning of training, create ensemble of N Q-functions by taking N samples from the initial weight distribution. Throughout training, the loss for the Q-functions is the standard least-square Q-evaluation loss,
[()') E (s,0,,, (Qo (s, a) (r term,, rt)²
y² = (1- term) - (x)) (5',
where 0', 0h denote the parameters and target network parameters for the 1th Q-function, and term = [s' is terminal]. In practice, the expectation in equation I is estimated by a minibatch, and the expectation in equation 2 is estimated with a single action sample from the policy. After every update to the Q-function parameters, their corresponding target parameters are updated to be an exponential moving average of the parameters in the standard fashion. key factor to note is that, in contrast to the typical usage of ensembles in actor critic algorithms, each ensemble member's update (and y²) is completely independent from other ensemble members. As we will present below, this an important algorithmic choice, both theoretically (Theorem 4.1 and empirically (section 5.3.2).
Policy Optimization As described above, the choice of architecture and weight initialization in- duces distribution on predictions. Having approximated this distribution using an ensemble of value networks, we can optimize the policy with respect to lower confidence bound on action values. Specifically, our proposed policy optimization objective in MSG is,
max (.. ((s,a) + as) where uuss, a) mean (co (s. a))] a(s,a) =std [ (s.
where B E R is a hyperparameter that trades off conservatism and optimism. As our problem setting is that of offline RL requiring conservatism, we use 3 << O.
3.2 THE TFA BETWEEN TRUST AND PESSIMISM
While our hope is to leverage the distributions induced by the architecture choice, is not always feasible to do so; designed architectures can still be fundamentally biased some manner, or we can simply be in a setting with insufficient data coverage. Thus we need to trade-off trusting the gen- eralization ability of our models with the pessimistic approach of support constraints. In this work, inspired by CQL (Kumar et al., 2020), we add the following regularizer to our policy evaluation objective (equation [].
R(0') E.-D Ea (o a)] Qo (s, ap)
(3)
where ap denotes the action taken in state in the dataset. We control the contribution of ((0)) by weighting this term with weight parameter Practically, estimate the outer expectation using the states in the mini-batch, and we approximate the inner expectation using a single sample from the policy. An example scenario where we have observed such a regularizer helpful is when the offline dataset only contains a narrow (e.g., expert) data distribution. We believe this is because the power of ensembles comes from predicting a value distribution for unseen (s, a) based on the available training data. Thus, if no data for sub-optimal actions is present, ensembles cannot make accurate predictions and increased pessimism for unseen actions becomes necessary.
INDEPENDENCE IN ENSEMBLES MATTERS
4.1 THE STRUCTURE OF UNCERTAINTIES DEPENDS ON HOW ENSEMBLING IS DONE
In the reinforcement learning literature, ensembles are widely used for combatting over-estimation bias (Haarnoja et al., 2018 Fujimoto et al. 2018: 2019 Kumar et al. 2020 Agarwal et al.) 2020 Ghasemipour et al./ 2021). However, the typical usage of ensembles is to compute a target value using an ensemble of target networks, and subsequently update all the Q-functions with respect to the same target, typically the minimum over targets. Hence, the objectives for all the Q-funciions share the same target value. In this section, drawing upon the literature of infinitely wide networks, demonstrate that having the updates for ensemble members be independent of one another as
4

---

Under review as a conference paper at ICLR 2022
done in MSG results in uncertainty estimates that align more closely with intuitive expectations, compared to when their targets are shared.
We study this difference in the setting of policy evaluation using infinite ensemble of infinite- width Q-function networks (i.e. an ensemble of infinitely many Q-functions, each of which is an infinite- unftth neural network, with the only difference being that their weights are initialized through independent random draws from the initial weight distribution). For sake of simplicity of derivations we assume that the policy are trying evaluate is deterministic and that do not have terminal states in the MDP, The policy evaluation routine we consider is Fitted Q-Evaluatinn (Fonteneau et al. 2013). which can be described as repeatedly performing the following steps.
Compute TD Targets For each Fit the Q-functions For each ensemble member, optimize the following objective until conver- gence using full batch gradient descent.
E D compute the TD targets s',,,',,
Σ (s.a.,:s)ED
-',,,,))))
(4)
D
First we establish some notation. Let X denote matrix where the rows are the state-action pairs (s, a) in the offline dataset. Let R be the [D] x matrix containing the rewards observed after each (s, a) in X. Let X" denote a matrix where the rows are the next state and policy action (s', (s')) Additionally let,
Oo(A, B) = (aQu(A) VoQo(B)
6 6
C C= 6, x) θ,²
(o B3 above is referred to as the tangent kernel (Jacot et al. 2018). which is defined as the outerproduct of gradients of the Q-function, initialization (iteration / = 0). The definition of θo does not contain the variable indexing the ensemble members because, at infinite width, 6, converges to a deterministic kernel and hence is the same for all ensemble members.
Intuitively, C' is |D] x |D] matrix where Gp.g (the element at row p. column g) captures notion of similarity between (s', π (s))) in the ydh row of X", and (s, a) in the 9th row of X. Let ) (with shape D x 1) denote the targets used for fitting Qo at iteration t.
Here, will study the difference between the following two methods for computing targets: one where each ensemble member uses its own predictions to compute the TD targets (analogous to MSG), and another where all ensemble members share same target (analogous to typical use of ensembles in offline RL):
Method (MSG, Independent Targets): (s +r y
Method 2 (Shared LCB): Vi,y'(s,
r
E ensemble
Std (f) ensemble
As can be seen in Appendix F for the two methods considered above, under the described policy evaluation procedure, the values Qo: (s, a)le (at iteration can be computed in closed form for all (s, E S x A. This enables us to compare the final distribution of Q(s,, a) after policy evaluation under two methods.
Theorem 4.1. After t iterations, for both Method 1 and Method we have,
V (1 (AC -1C0011001100))000000)))))) CQu(1))) ?
+ Independent: LCB ((+1(2²)) so (1+.. +C
Shared LCB: LCB (111(21) (1+... +C++...+ +C) cQu(x)))
where LCB refers to mean- std, and the square and square-root operations are applied element-wise.
Proof. Please refer to Appendix F
5

---

Under review as a conference paper at ICLR 2022
As can be seen, the equations for the lower-confidence bound (LCB) in both settings are very similar, with the main difference being in the second terms which correspond to the "pessimism" terms. In ensembles, the only source of randomness is in the initialization of the networks. In the infinite- width setting this presents itself in the two equations above, where the random variables Qo(2") (Q)(X) produce the uncertainty in the ensemble of networks: For both Independent and Shared- LCB, at iteration + 1 we have,
Q++((2") = = +00
(5)
= =C + (Q6(") - -Q6(2") CQu(2))
(6)
Thus, Q0(X")-CQo(2" represents the random value accumulated in each iteration. The accumula- tion of the uncertainty is captured by the geometric term (1+.... (1+..++yC"). Here is where we observe the key difference between Independent and Shared-LCB: whether the term (1 +.... + Y'C') is applied inside or outside the expectation. In Independent ensembles, the randomness/uncertainyy is first backed-up by the geometric term and afterward the standard-deviation is computed. In Shared- LCB however, first the Htandard-deviation of the Fundommesssuncertainny is computed, and after- wards this value is backed up. Not only do we believe that the former (Independent) makes more sense intuitively, but the case of Shared-LCB, the pessimism term may contain negative values which would actually result in an optimism bonus! As will be discussed below, we also empirically investigate this question in section 5.3.2] (results in Figure? 2 and find that while Shared-LCB can perform decently in D4RL gym locmotion (Fu et al., 2020), Shared-LCB (and Shared-Min) com- pletely fail on the more challenging domains. This line with the observation that no prior offline RL methods rely solely Shared- or Shared-Min as the source of pessimism/cconserattii
(Fujimoto et al. 2019 Kumar et al. 2019 Ghasemipour et al. 2021 An et al. 2021).
4.2
VALIDATING THEORETICAL PREDICTIONS
In this section we aim to evaluate the validity of our theoretical discussion above in a simple toy MDP that allows allows for visualization of uncertainties obtained through different ensembling approaches.
follow the idcalized setting of the presented theorems more closely, and
1]. action space A E R. deterministic transition dynamics + clipped to remain inside S, and reward function r(s,a) = 1[s' E [0.75, 1]]-
Continuous Chain MDP The MDP we consider has state-space S [|1,
Data Collection & Evaluation Policy The offline dataset we generate consists of 40 episodes, each of length 30. At the beginning of each episode we initialize at a random state E S. In each step take random action sampled from Unif(-0.3, 0.3), and record all transitions (s, a, s'). For evaluating the uncertainties obtained from different approaches, we create regions of missing data by removing all transitions such that s or s' are in the range [-0.33, 0.33]. The policy we choose to evaluate with the different approaches is Vs, (s)) = 0.1.
Optimal Desired Form of Uncertainty Note that the evaluation policy (s)) = 0.1 is always moving towards the positive direction, and there is lack of data for states in the interval [-0.33, 0.33]. Hence, what we would expect is that in the region [0.33, 1] there should not be a significant amount of uncertainty, while in the region [-1, -0.33] there should be significantly more uncertainty about the Q-values of æ because the policy will be passing through [-0.33, 0.33] where there is no data.
Results We visualize and compare the uncertainties obtained when the targets in the policy evalu- ation procedure are computed as:
Independent (MSG): of = Qo: ((('))
Independent Double-Q: r min (8', (8,)),
Shared Mean: = + mean Qo (5', (()))
Shared LCB: y = r + mean Qo (s', ((,, stal (oo m((])
Shared Min: y r + min (s', (6]))

---

Under review as a conference paper at ICLR 2022
Independent
Ind. Double-Q
Shared Mean
Shared LCB
Shared
130
130
130
130
130
122
Ols. Qls, ((s))
130-
110
110
110
110-
100
100
92
50
80
86
82
60
50
65
60
-140
1:00 X-...
1.0
-0.33
1.00
-0.33
-1.00
0.33
State
state
state
State
state
Figure 1: Verifying theoretical predictions the toy Continuous Chain MDP. The marked interval [-0.33, 0.33] denotes the region of state-space with no data. As anticipated by Theorem 4.1 when the value functions are trained independently, the derived uncertainties capture the interaction between available data, the structure of the MDP, and the policy being evaluated. When the targets are shared, the networks behave similarly to performing regression for oracle-given target values, i.e. there is randomness amongst ensemble members only between [-0.33, 0.33] because there is no data in that region.
We include Independent Double-0, as using Q-functions of the form Q(s.a) a)
min (s) a) has become common practice in recent deep RL literature (Fujimoto al,
2018
Haarnoja al. 2018
In Figure I we plot the mean and two standard deviations of the predicted values for the policy we evaluated, T(s) 0.1 (additional experimental details presented in Appendix G.I). The first thing to note is that, "Independent" ensembles effectively match our desired form of uncertainty: states that under the evaluation policy lead to regions with little data have wider uncertainties than states that do not. A second observation is that rea LCB and Shared Mi n provide seemingly good approximation to the lower-bound of Independent predictions. Nonetheless, our theoretical considerations suggest that these lower bounds may have important failure cases. Furthermore, empirically - as we discuss below (Figure we were unable to train effective policies on offline RL benchmarks using Shared LCB and Shared Min, despite the implementation differing in only 2 lines of code.
Appendix G.2 presents additional very interesting empirical observations in this toy setting which due space limitations we were unable to include in the main manuscript. We highly encourage readers interested in the intersection of aalinite-width networks and RL to take a look at our observations as they may be hinting at intriguing avenues for future theoretical and practically important work.
EXPERIMENTS
In this section we seek to empirically answer the following questions: 1) How well does MSG perform compared to current state-of-the-an in offline RL? 2) Can we match the performance of MSG through efficient ensemble approaches popular in supervised learning literature? 3) Are the theoretical differences in ensembling approaches elaborated on in the previous section practically relevant?
5.1 D4RL BENCHMARK
We begin by evaluating MSG the Gym (Brockman et al.) 2016) subset of the D4RL offline RL benchmark (Fu et al. 2020). Amongst the different data settings we focus our experiments on the medi um and medi um replay (sometimes referred to mixed) settings as the other data setting could not adequately differentiate between competitive methods. Experimental details such as hyperparameter search procedure are described in Appendix C
In addition to validating MSG, a secondary objective of ours is to gain a sense for the upper bound of performance for various algorithms on D4RL Gym. For this reason with an equal hyperparameter tuning budget- we tune the main hyperparameter for each algorithm. As can be seen in Table (Appendix A), across the board in D4RL. Gym, MSG is competitive with very well-tuned current state-of-the-at algorithms, CQL (umarr et al. 2020) and F-BRC (Kostrikov et al.) 2021). We
Note that Independent Double-9 is still independent ensemble, where each ensemble member has architecture containing min-pooling on top of subnetworks.
7

---

Under review as a conference paper at ICLR 2022
Domain
CQL (Reported)
MSG (N 64)
a
maze2d-umaze
5.7
100.2:33..
-8
0
mazz2d-medium
5.0
87.4 10.6
0
maze2d-large
12.5
147.8 55.8
4 0.1
antmazee-maaze
74.0
96.8 2.0
8 0.1
untmaze-umaze-diven
84.0
7.1
-8 0.5
untmaze-medium play
61.2
8.4
-4 0.1
antmaze-medium- diverrse
53.7
78.8 5.5
-4 0.1
untmaze-large-play antmaze-large-divrre
15.8
64.8 - 10.7
-8 0
14.9
68.8 + 11.7
-8 D
Table 2: D4RL antmaze tasks. Figure
Result D4RL maze2d and antmaze domains. we were unable to reproduce CQL antmaze results, we present the numbers reported by the original paper which uses the same net- work architectures.
Table 1:
taken from Fu et al. 2020).
also note that our results for baseline algorithms exceed prior reported results (often significantly), providing confidence in their implementation.
Thus far we established the validity of MSG as an offline RL algorithm. However, on the gym domains considered above, MSG's performance on par with CQL and F-BRC, and does not mo- tivate the use of ensembles in lieu support constraints. To enable a better comparison amongst competing algorithms, we experiment with the significantly more challenging antmaze tasks.
The antmaze tasks in D4RL, and in particular the two antmaze-large settings, are considered to be extremely challenging. The data for antmaze tasks consists of many episodes of an Ant agent (Brockman et al., (216)) running along arbitrary paths a maze. The data from these trajectories is relabeled with reward of when near a particular location in the maze (at which point the episode is terminated), and 0 otherwise. The undirected, extremely sparse reward nature of antmaze tasks make them very challenging, especially for the large maze sizes.
To the best of our knowledge, the antmaze-large domains are considered unsolved, unless specialized technique such as hierarchical policies which significantly simplify the problem are used (e.g. Ajay el al. ).020)). The current state-of-the-are is CQL (Kumar et al. 2019), and is the method we compare to. Table presents our empirical results. As we were unable reproduce the reported results for CQL, for fairness we include the numbers reported by the original work which uses the same network architectures. As can be seen, MSG achieves unprecedented results on the antmaze domains, clearly demonstrating the significant advantage of employing ensembles for batch RL.
5.2 RL UNPLUGGED
Figure presents additional results using the RL Unplugged benchmark (Gulcehre et al. 2020). We compare to results for Behavioral Cloning (BC) and two -fate-of-the-an methods, Critic-Rees Regularized Regression (CRR) (Wang al. 2020) and MuZero Unplugged (Schrittwieser al. 2021). Despite the relatively very small architectures we used (σ) number of parameters), we observe that MSG on par with the current state-of-the-ar these tasks with the exception of humanoid d. run which appears to require the larger architectures used by (Gulcehre et al., (222)). Additional experimental details as well as numerical presentation of results can be found appendix
5.3 EFFICIENT ENSEMBLES & ENSEMBLE ABLATIONS
In this section we dissect what aspects of MSG contribute to its superior results, and investigate whether the advantages of MSG can be realized through efficient ensemble approximations popular in supervised learning.
5.3.1 EFFICIENT ENSMEBLES
Thus far we have demonstrated the significant performance gains attainable through MSG. An im- portant concern however, is that of parameter and computational efficiency: "Deep Ensembles" re- sult in an N-fold increase in memory and compute usage for the Q-networks. While this might not be a significant problem for D4RL benchmark domains due to small model footprints it becomes major bottleneck with larger architectures such those used in language and vision domains. To
8

---

Under review as a conference paper at ICLR 2022
100
30
haffcheetah-medlum
hepper-medium
walker20-medium
halfcheetah-miixd
hopper-mixed
walker2d-mbxed
artmaze-large-dive
antmaze-lerge-play
CQL
Deep Ensembles (default) Multi-Head
MIMO-0.5
MIMO-1
Batch Ensembles
N=1
Shared LCB
Shared
Figure 2:
Results for efficient ensembles and ensemble ablations. Numerical values can be found in Tahle
this end, we evaluate whether recent advances efficient ensemble approaches also transfer well to the problem of batch RL. Specifically, the efficient ensemble approaches consider are:
Multi-Head (Lee al., 2015; Osband et al. 2016: Tran al., 2020 Multi-Head refers to en- sembles that share a "trunk" network and have separate "head" networks for each ensemble member. In this work, we modify the last layer of value network to output N predictions instead of single Q-value, making the computational cost of this ensemble on par with single network.
Multi- Inpti Multi-Output (MIMO) (Havasi et al. 2020) MIMO is ensembling approach that approximately has the same parameter and computational footprint as a single network. The MIMO approach only modifies the input and output layers of given network. In MIMO, compute predictions for data-points Ii, ... N under an ensemble of size N, the data-points are concatenated and passes to the network. The output of the network then split into N predictions 1/1....: V/.. For added clarification, we include Figure 3a depicting how MIMO ensemble network functions.
Batch Ensembles (Wen et al.) 2020) Batch Ensembles incorporate rank-1 1 modulations to the weights of fully-connected layers. More specifically, let be the weight matrix given fully- connected layer and let x be the input to the layer. The output of the layer for ensemble member i computed as a (((WT(z (I: r")) o s*) + b'), where o the element-wiss product, parameters with superscript i separate for each ensemble member, and is the activation function. While Batch Ensemble is efficient in terms of number of parameters, in our actor-critic setup its computational cost is on the same order as deep ensembles since for policy updates we need evaluate each ensemble member separately.
Results As can be seen in Table 4 the performance gains of MSG can to some extent also be realized by efficient ensemble variants. Interestingly, in our experiments the most effective efficient ensemble approach is Multi-Head ensembles, which is not considered to be the most competitive uncertainty estimation technique in the supervised learning literature (Havasi et al.] 2020). Com- pared to CQL, Multi-Hean ensembles continue to be competitive outperform CQL on antmaze-large tasks. Additionally, training Multi-Head ensembles is the most efficient method, par with training without ensembling (N 1).
D4RL gym, and noticeably
Nonetheless, compared to MSG using deep ensembles, there is a significant performance gap. We believe this observation very clearly motivates future work in developing efficient uncertainty esti- mation approaches that are better suited to the domain of reinforcement learning. To facilitate this direction of research, in our codebase which will be open-sourced, we also include a complete boil- erplate example amenable to drop-in implementation of novel uncertainty-estiman techniques.
5.3.2 ENSEMBLE ABLATIONS
Finally, through the ablations in Table 4 we seek to create a better sense for the various components of MSG.
Comparing MSG with deep ensembles (N 64), and Multi-Head ensembles (N = 64), to ensembling (N = 1) we see very clearly the massive advantage ensembling.
Comparing MSG, which uses Independent targets, to Shared LCB and Shared Min we that the latter non-independent approaches significantly underperform.
Comparing CQL to N 1 (no ensembling, only CQL-inspired regularizer), we observe that the CQL regularizer tends to better on Gym domains, and our regularizer may be better on antmaze domains but results are inconclusive. The advantage of the regularizer used is significant computationl efficieny due to not using importance sampling.

---

Under review as a conference paper at ICLR 2022
DISCUSSION
Our work has highlighted the significant power of ensembling as a mechanism for uncertainty es- timation for offline RI. Theoretically, and practically through benchmark experiments, we have studied the critical importance of the manner in which ensembling is done. An important out- standing direction is how can we design improved efficient ensemble approximations, as we have demonstrated that current approaches used in supervised learning some of which do lead to state- of-the-art results offline RL results - are not nearly as effective as deep ensembles. hope that this work engenders new efforts from the community of deep network uncertainty estimation researchers whom thus far have not employed offline reinforcement learning domains as a testbed for validating modern uncertainty estimation techniques.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.1300 2020.
Gaon An, Seungyong Moon, lang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01. 011160000150,, 2016.
Jacob Buckman, Carles Gelada, and Mare Bellemare. The importance of pessimism in fixed- dataset policy optimization. arXiv preprini arXiv:2009.06799 2020.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. arXiv preprint 1rXiv:2010.11652, 2020.
Yihao Feng, Tongzheng Ren, Ziyang Tang, and Qiang Liu. Accountable off- evaluation with kernel bellman statistics. In International Conference Machine Learning, pp. 3102-3111. PMLR, 2020.
Raphael Fonteneau, Susan A Murphy, Louis Wehenkel, and Damien Ernst. Batch mode reinforce- ment learning based on the synthesis of artificial trajectories. Annals of operations research, 208 (1):383-416, 2013.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv: 106.0066660 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard Zemel. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. arXiv preprint arXiv: (w....69904333 2016.
Caglar Gulcehre. Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A collection of benchmarks for offline reinforcement learning. Advances in Newral Information Processing Systems, 33, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- 2018
cations. arXiv preprint arXiv:

---

Under review as a conference paper at ICLR 2022
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint ar arXhv:2010.066100 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645, Springer. 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. arXiv preprint arXiv:1806.07522 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.,6940 2014.
Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off- off-pollcy evaluation. arXiv preprint arXiv:2007.1360, 2020.
Iyya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774--7883. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint w:Xiv:1906.00944 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477) 2020.
Il Kuzborskij, Claire Vernade, Andras Gyorgy, and Csaba Szepesvári. Confident off-policy eval- uation and selection through self- neff-normalized importance weighting. In International Conference on Artificial Intelligence and Statistics, pp. 640-648. PMLR, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv: 1612.01444 2016.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1711:01111000060: 2017.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32:8572-88 8583,, 2019.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06311 2015.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.006633 2020.
Ping Li, Jinde Cao, and Zidong Wang. Robust impulsive synchronization of coupled delayed neural networks with uncertainties. Physica A: Statistical Mechanics and its Applications, 373:261-272, 2007.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint
2015.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: 1271, 2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2022
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A Alemi, Jascha bink-Dic and Samuel S Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. arXiv preprint arXiv: 1912.0280,, 2019.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026 2016.
Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20(124):1-62, 2019.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv:190.002530, 2019.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1711.01297 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 19X10019100000177,, 2019.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. arXiv preprint arXiv:2104.06299 2021.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Alexander Strehl, Lihong Li, and Michael I. Littman. Reinforcement learning in finite mdps: Pac analysis. Journal of Machine Learning Research, 10(11), 2009.
P. Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence off-policy evaluation. In Proceedings of the 29th Conference on Artificial Intelligence, 2015.
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint arXiv:2001.0465 004694, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008 2017.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006.11 /5134, 2020.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rX1v:191..1136. 2019.
Greg Yang and Edward J Hu. Feature learning in indinite-width neural networks. arXiv preprint arXiv:201 14522, 2020.
13

---

Under review as a conference paper at ICLR 2022
Mengjiao Yang. Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selec- tion under uncertainty. arXiv preprint arXiw:2012.06919, 2020.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. arXiv preprint arXiv:7999.09314, 2019.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4923 4932, 2019.
14

---

Under review as a conference paper at ICLR 2022
A
D4RL GYM LOCOMOTION BENCHMARK TABLE
Domain
BC
CQL
F-BRC (no bonus)
C
MSG (N = 64)
Data Top 25
halfcheetah-meddum
36.1
50.1
1.2
45.7 + 0.5
0.01
50.6 1.5
37.0 + 2.0
hopper-medium
29.0
90.2
10.5
2.58
99.0 4.1
0.012
88.2 11.6
-1 0.5
101.1 1.1
walker2d-mcdium
6.6
82.8
1 0.6
3.38
81.6 0.2
0.016
83.6 = 2.0
-1 0.1
77.6 + 1.4
balfchectah- mixed
38.4
4..6
0.67
0.1
0.01
52.1 1.7
38.1
hopper-mixed
11.8
61.1
36.8
0.67
29.3
0.019
82.1 24.2
-2
101.8 1.5
walker2d-mixed
11.3
20.5 + 3.8 3.38
28.4 5.7
0.038
24.0 - 9.4
- 0.1
50.6 = 3.9
Table 3: Results Gym subset of the D4RI. benchmark. For MSG use an ensemble size of N 64. For fairness comparison, F-BRC ran without adding survival reward honus. For each method we also report the hyperparameter value used. Full experimental details are presented in Appendix
B EFFICIENT ENSEMBLES & ENSEMBLE ABLATIONS TABLE
Domain
CQL
Deep Ens.
MIMO-0.5
MIMO-1
Multi Head
Batch Ens
N 1
Shared LCB
Shared Min
halfcheetah-medium
50.1
50.6
40.0
44.4
49.8
40.2
48.6
47.9
36.0
hopper- medium
90.2
88.2
60.2
65.1
94.8
41.6
50.2
30.0
43.6
walker2d
welker2d-medium medium
82.8
83,6
80.7
83.6
84.5
13.2
84.9
85.5
0.0
halfcheetah mixed
47.6
52.1
0.0
43.0
49.6
51.0
2.3
hopper-mixed
61.1
82.1
22.7
32.5
35.0
30.5
32.5
6.7
walker2d-mixed
20.5
24.0
14.5
16.8
15.9
12.0
11.5
26.3
0.0
antmaze-larg large-iveereeeeeeeeee
15.8
64.8
11.0
26.0
33.5
noisy 30.5
latmaze-large-lay
14.9
68.8
7.5
20.5
27.5
0.0
noisy 11.0
0.0
0.0
Table 4: Results for efficient ensembles and ensemble ablations.
C D4RL GYM DETAILS
All policies and Q-functions are a 3 layer neural network with relu activations and hidden layer size 256. The policy output a normal distribution that squashed to [-1, 1] using the tanh function. All methods were trained for 1M steps. CQL and MSG are trained with behavioral cloning (BC) for the first 50K steps. F-BRC pretrains with 1M steps of BC.
CQL, F-BRC, and MSG tuned with equal hyperparameter search budget. Running the best found hyperparameter using 5 new random seeds. Each run evaluated for 10 episodes, and the mean and standard deviation across the 5 runs is reported in the table. For faimesss of comparison, F-BRC ran without adding a survival reward bonus. MSG and CQL are implemented our code, and for F-BRC use the opensourced codebase.
C.I HYPERPARAMETER SEARCII
For all methods we performed hyperparameter search using 2 seeds. Based the results of the seeds, we chose the hyperparameter to use, and using this choice hyperparameter we ran experi- ments with 5 new random seed.
MSG B (0., 10,,-1,,22 -2, -4. 8.).
CQL a
+ h1l.--Hww.ppppppppppppooooolaoa0 steps = 23))
F-BRC E (0.) wwwp://wwwaaannocccoooooolloollm steps 24))
15

---

Under review as a conference paper at ICLR 2022
3/1
AY
Network
N
VN
(b) D4RL antmaze tasks. Figure taken from
(a) Visual depiction of MIMO Ensemble
Fu et al. 2020
130
manipudater Jull maripudator soadd
Walerrwnl
shweteh-sun
BC MSG
MuZero Unplugged
Figure 4: Results for DM Control Suite subset the RL Unplugged benchmark (Gulcehre et al. 2020p. note that: 1) the architecture we used are smaller by factor of approximately 60x, 2) CRR results are reported by their best checkpoint throughout training which differs from ours, BC, and MuZero Unplugged which report performance the end of training. Baseline results taken from Schrittwieser et al. 2021
D ANTMAZE DETAILS
We use the same hyperparameter search procedure Gym results, with same architectures. The only difference is that models are trained for 2M steps and at evaluation time they are rolled out for 100 episodes instead of 10.
In prior work, the rewards in the offline dataset are converted using the formula 4(r 4(r- ((r-0.5) 0.5). We also use the same reward transformation.
RL UNPLUGGED
DM CONTROL SUITE TASKS
The networks used in Gulcehre al. 2020) for DM Control Suite Tasks are very large relative to the networks we used in the D4RL benchmark: roughly the networks contain 60x more parame- ters. Using large ensemble size with such architectures requires training using a large number of devices. Furthermore, since in our experiments with efficient ensemble approximations we did not find a suitable alternative to deep ensembles (section [.3.I]. we decided use the same network architectures and N 64 as in the D4RL setting (enabling single-GPU training before).
Our hyperparameter search procedure was similar to before, where we first performed a coarse search using 2 random seeds and hyperparameters 3 {-1., -2, -4. 8.), {0., 0.5), and for the best found hyperparameter, ran final experiments with new random seeds.
F THEORY
For notation match Lee al. (2019), throughout this section the f to denote the network Q, and we use x instead of (s, a).
In Lee et al. 2019) (section 2.4) it is shown that when training infinitely wide neural network to perform regression using mean squared error, subject to technical conditions on the learning rate used, the predictions of the trained network equivalent to if we had linearized (Taylor expanded)
16

---

Under review as a conference paper at ICLR 2022
the network at its initialization, and trained the linearized network instead. This means that after t iterations of our policy evaluation procedure, Va, t, ft (z) = fx)), where
(x) == fo(x) + Vofo(2>)oo=n we
W/ == 8, 0o
Hence we only need to study the evolution of the linearized network f!n across iterations. The theorems in the main manuscript are direct corollaries of the following derivations.
Below, we will overload some notation. When fln is applied to a matrix we mean that we apply flin to each row of the matrix and stack the results into vector. By Oo(r, X) we mean to treat as row matrix.
F.1 CLOSED FORM WHEN ENSEMBLE MEMBERS USE THEIR OWN TARGETS
For single infinitely wide ensemble member, using the equations in Lee et al. (019)) (sec- tion 2.2, equations 9-10-11) we can write the following recursive updates for our policy evaluation procedure
6,1 = (
C 0o(X", X)0,1
4 = R + T (X')
S (X) = )1
(10)
Vx, flh (I) fo(x) + ()) - f((-)
(11)
f (X') =o((²') 6o(2" 10) So(H))
(12)
=o(2²" x)) (R+ 1 (X') fo(2))
(13)
=o(('' CR +Rf (x') - -fo(2)
(14)
fo(x") -CR -f((X) -xf"
(15)
(16)
= +1+ + ,C (F6(2') + CR- - + +170)++56(2))
(17)
(Xf)) (1 + /C'CR
(18)
(erff(((((')) = E (11
- + + (10)))))(())+ (6(2)))
(19)
+ ~ + +00)((((') -C(((()))))((((() ast +
(20)
F.2 CLOSED FORM WHEN ENSEMBLE MEMBERS USE SHARED MEAN TARGETS
Let consider the setting where all ensemble memebers their mean the target.
17

---

Under review as a conference paper at ICLR 2022
6-² x)
(21)
C: (2)0'
(22)
D: = + +
(23)
[41 (2) = 34
(24)
Vz, f (z) fu(x) + Ou(z, X (3) fo(x))
(25)
V: ++ E
[f!
(26)
ensemble
= R +
E
[fo(2")] + +000111 - -C [fo(X)
(27)
ensemble
ensemble
= R -C+
(28)
(29)
+1. +
(30)
F.3 CLOSED FORM WHEN ENSEMBLE MEMBERS USE SHARED LCB TARGETS
Let us consider the setting where all ensemble memebers use shared LCB the target.
6 6)(X)
(31)
C ())
(32)
}1 LCB R + +x("t
(33)
= R xLCB (LCB(((((2)) (x*))
(34)
flin (X) = =Y,
(35)
Vx, th (x) = fo(x) + 20, 2)) So(2))
(36)
E (xfe+1(X') = E(fo(X') -. -u(2))]
(37)
C. V/.
(38)
Varlfe+1(X")) = E (fo(2") - -fo(2)) = constant
(39)
A cconstant
(40)
LCB 110 (2')
-C V/. A
(41)
y +R+ OR+LLCB(p (CCB(A(((2)) (x))
(42)
= =R+10J -7A
(43)
=(1 +1+ + CCC. (fo(2²)
(44)
2 (1+ +
as
(45)
LCB (1211))1))) (1+ - +CC)
(46)
F.4 WHY INDEPENDENT PREFERABLE TO SHARED-LCB
An important question consider is why Independent ensembles should be preferred over Shared- LCB ensembles? Here we present our reasoning for why we Independent ensembles would be preferable Shared-LCB ensembles.
With the derivations in the above sections, we can compare the difference amongst uncertainty estimation techniques. The key comparison needed is to understand the difference between
18

---

Under review as a conference paper at ICLR 2022
LCB 1 under Independent vs. Shared-LCB settings. As a reminder, X' is a matrix where each row contains (s, T()), and fe+1 - under the infinite-width regime. From the above equations have:
Independent: I.CB (ft (2')) (1 + + -C) - ( - 37 I
Shared-LCB: LCB (1 (2'))
+r ( -(f()))((((())))))
where the square and square-root operations are applied element- wise the vector values.
As can be seen, the equations for the lower-confidence bound (LCB) in both settings are very similar, with the main difference being in the second terms which correspond the "pessimism" terms. In
the infinite-width setting, the only source of randomness is in the initialization of the networks. This fact presents itself the two equations above, where the random variables fo the uncertainty in the ensemble of networks; regardless of using Independent or Shared-LCB, after any iteration / we have,
produce
Vi- R ++
(47)
Se+1(x") fo(2²) + CO, (((A))
(48)
= =CV, + (fo(2*) - Cfo(2))
(49)
E[f++1(X')] = CED1
(50)
Thus, Jo(X") - (fo( represents the random value accumulated in each iteration, and they are accumulated through backups by the geometric term (1 +.... CC).
Here is where we observe the key difference between Independent and Shared-I.CB, whether the term (1+... is applied inside or outside the expectation. In Independent ensembles, the ran- dommess/uncerrainties is first backed-up by the geometric term and afterward the Handard-deviation computed. In Shared-LCB however, first the standard-deviatinn of the randomness/uncertainti computed, and afterwards this value is backed up. Not only do believe that the former (Inde- pendent) makes more sense intuitively, but in the case Shared-LCB, the second term may contain negative values which would actually result in optimism bonus!
F.5 COMPARING THE STRUCTURE OF UNCERTAINTIES UNDER INDEPENDENT AND SHARED-MEAN
The above results enable us compute in closed form the predictions of the infinite-wide networks under different training regimes Va.
The equations above present the closed form expressions for the predictions each ensemble mem- ber after t iterations of the policy evaluation procedure. Since the ensemble members only differ their weight initialization (random draws from the initial weight distribution), the random vari- ables are ( (x))), fo (x").. As mentioned the main text, the neural tangent Ô is identical across ensemble members due being in the infinite-width regime (Jacot et al. 2018).
[fo[=]] = Lee et al. 2019 2017: Matthews et al. 2018). the expected values of ftt1 (x: ) is identical in for both methods of computing TD targets,
Since Va, E ensemble
E D + .... +...,, ensembbe
(51)
Var ensembble
E
( Ox( X70 (1 (1+...+
(52)
However, the expression for variances is very different. When the targets used are independent we have,
Ya, Var [f]+ ensemble
(folee) @o(o,
(0 23) (01''((11.... +000 (156(2)
E ensemble
(53)
19

---

Under review as a conference paper at ICLR 2022
In contrast, when the targets shared mean of targets, we have,
(u(, (-fo(2)) )
Vx, Var ensemble
[f++(()]]= =
E ensemble
(54)
The matrix C' capturess notion of similarity between the (s, a) in X, and the (s', iT (T(s)) ) in X'. Thus, the term C2 has the interpretation of where the policy x (8) would find itself steps into the future, and (1 ... C'C* can be interpreted the policy's discounted state-action visitation, but in the feature-space given by the neural network architecture. Since in ensembles the standard deviation of predictions quantifies the amount of uncertainty, the expression in equation (3 tells us that when the targets are independent, the ensemble of Q-functions "backs up the uncertainties through dynamic programming with respect to the policy being evaluated".
In contrast, when the targets are shared, the closed form expression for f(+1(z) is equivalent to an oracle presenting us with targets (1 +... + C²CC) R for training examples X, and training the ensemble members using mean squared error regression to regress these values.
G ADDITIONAL TOY EXPERIMENTS & DETAILS
G.1 ADDITIONAL IMPLEMENTATION DETAILS FOR FIGURE
To evaluate the quality of uncertainties obtained from different Q-function ensembling approaches, we create N = 64 Q-function networks, each being a one hidden layer neural network with hidden dimension 512 and tanh activation. The initial weight distribution a fan-in truncated normal dis- tribution with scale 10.0, and the initial bias distribution is fan-in truncated normal distribution with scale 0.05. We did not find results with other activation functions and choices of initial weight and bias distribution be qualitatively different. We use discount mized using the Adam (Kingma & Ba 2014) optimizer with learning rate 1e-4. In each iteration, first compute the TD targets using the desired approach (e.g. independent vs. shared targets) and then fit the Q-functions to their respective targets with 2000 steps of full batch gradient descent. We train the networks for 1000 such iterations (for a total of 2000 x 1000 gradient steps). Note that we do not use target networks. Given the small size of networks and data, these experiments can be done within few minutes using a single GPU in Google Colaboratory which we will also opensource.
0.99 and the networks
opti-
G.2 ADDITIONAL TOY EXPERIMENTS
The toy experiment presented in section 4.2 uses a single-hidden layer finite-width neural network architecture with tanh activations, uses the "standard weight parameterization" (i.e. the weight parameterization used in practice) as opposed to the NTK parameterization (Novak et al. 2019). and optimizes the networks using the Adam optimizer (Kingma & Ba 2014). While this setup is close to the practical setting and demonstrates the relevance of our proposal for independent ensembles for the practical setting, important question posed by our reviewers is how close these results too the theoretical predictions present in 4.1 To answer this question, we present the following set of results.
Using the identical MDP and offline data as before, we implement I hidden layer neural networks with erf non-lineartiy. The networks are implemented using the Neural Tangents library (Novak et al. 2019), and use the NTK parameterization. The networks in the ensemble are optimized using bull- gradient descent with learning rate 1 for 500 steps of FQE Fonteneau al. (2013), where each in each step the networks are updated for 1000 gradient steps. We vary the width of the networks from 32 to 32768 in increments of a factor of 4, plotting the mean and standard deviation of the network predictions. The ensemble size is set to N 16, except for width 32768 where N 4.
We compare the results from finite-width networks to computing the mean and standard deviation in closed for using 4.1 Using the Neural Tangents library (Novak et al., 2019) we obtained the NTK for the architecture described in the previous paragraph (1 hidden layer with erf -incaricrr We found that the matrix inversion required in our equations results in numerical errors. Hence, we make the modification O(X' X) @(X'X0 X) + I.
20

---

Under review as a conference paper at ICLR 2022
NTK Parameterization
Width 32
Width 1228
Width 512
170
140
00
125
62-
22
-1.00
-0.35
-10
+0.31
0.3
1.00
2.00
-0.13
0.33
1
Width 2048
Width 8192
Width 32768
Infinite Width
129
105
109
16c
176-
339
30
135
-100
-033
-122
0.3
1.0
032
1.5
1.00
-333
122
Maximal Parameterization
Width 32
Width 128
Width 512
130
90
20
-1.00
-0.55
0.55
1.00
-1.90
-2.45
0.55
1.00
-1.00
-0
0:55
100
Width 2048
Width 8192
Width 32768
300
300
30
-1.00
-0.35
-1.00 -0.30
-3.00 -0.25
Figure 5: Comparing results of finite-width networks to closed form equations derived in Theorem 4.1 the NTK parameterization, as width -> 30, the structure the variances collapse and resem- ble the wndtnite-width closed-form results. We believe this is due infinite-width networks under the NTK regime not being able to learn features (Yang & Hu 2020). Supporting this hypothesis, observe that networks parameterized by the Maximal Parameterization of Yang & Hu (2020) maintain the desired uncertainty structure as the width the networks grows larger.
FigureF|presents our results. As the width of the networks grow larger, the shape of the uncertainties closed-form equations (i.e. the variances become very small). While
becomes more similar explanation for why finite-width networks exhibit intuitively more de- we do not have rigorous sirable behaviors, we present below strong hypothesis backed by empirical evidence. We believe rigorously answering this question is an incredibly interesting avenue for future work.
Infinite-width networks in the NTK Darameterization/regine do not learn data- dependent features (Yang & Hul 2020).. Furthermore, as can be seen in the equations of Theorem 4.1 the variances depend the function values and features (kernel. matrix, etc.) at initialization. Yang & Hu (2020) present different approach for parameterizing anfinite-width networks called the "Maximal Parameterization". which enables inifinite- width networks to learn data-dependent features. We perform the same experiment as above, by replacing the TIK-pparameterized networks with Maximal Parameterizations. Figure 5 presents our empirical results for network widths from 32 to 32768. Excitingly, we observe that with Maximial Parameterization, even our widest networks recover the intuitively desired form of uncertainty described section 4.2] The so- lutions of these networks also appear much more accurate, particularly the right hand side of the plot where with the stepped structure; each step appears to approximately 0.1 width, which is the action of the policy being evaluated.
Hypothesis:
21

---

Under review as a conference paper at ICLR 2022
π
D
π
D
alg
arch
alg
arch
prior
(a) Q-function generative process: the graphical model above represents the induced distribution over Q- functions when conditioning particular policy evaluation algorithm, policy, offline dataset, and Q- function network architecture.
(b) An example of an interesting extension
H STATISTICAL MODEL
An interesting question posed by reviewers of our work was "[W]hatever formal reasoning system we'd d like use, what the ideal answer, given access to arbitrary computational resources, so that approximations are unnecessary? L.e., how do we quantify our uncertainty about the MDP and value function before seeing data, and how do we quantify it after seeing data?"
It is important to begin by clarifying what is the mathematical object we are trying to obtain un- certainties over. this work, we not quantify uncertainties about any aspects of the MDP itself (although this is an interesting question which comes up in model-based methods as well other settings such as Meta-Learning (Yu et al. 2019; Ghasemipour et al. 2019)). Our goal in this work is to directly estimate (= r(s Y ED mm Q(5 a')], for (s)), and obtain uncertainties about (s)
Let Q(s,a) a) be predictor S x A R that needs be evaluated and hopefully generalize well to - (s, a) & D (D being the offline dataset). When we choose to represent Q(s, a) using neural networks. Gaussian Processes, or K-nearest-neighhouu computational reasons, but are actually choosing a function class which we believe will generalize well to unseen a).
are not just making approximations for
One practical example of learning Q-functions is to use Fitted Q iteration (Fonteneau et al., 2013) on the provided data using gradient descent with a particular neural network architecutre. Due to the random weight initialization, this procedure induces a distribution on the Q-functions which is captured by the probabilistic graphical model (PGM) in Figure 6a In other words, by conditionining on the policy, data, architecture, and policy evaluation algorithm, we are imposing a belief over Q- functions. Note that this is essentially the same justification as using ensembles supervised deep learning, where ensembles are state-of-the-arr for accuracy and calibration (Ovadia et al. 2019). For the sake of theoretical analysis (Section(4). we studied this belief distribution under the udhnite-width NTK network setting, in which case the distribution over Q-functions is Gaussian Process.
The focus of this work to ask the question: "Under this imposed belief, what should the policy update be?". Our proposed answer is to optimize the policy with respect to the lower-confidence bound of our beliefs: In an actor-critic setup, the policy optimization objective takes a form like max, Ea(s) [Q(s) ([(ss((()))))))) m(s)], where d(s) is some distribution over states (e.g. initial state dis- tribution, or the states in the offline dataset D, etc.). Thus, our proposed policy objective takes the form max, LCB [Eas) (()))) and for practical reasons, in MSG we convert this to max, Ed(s) [CB (Q(s, (s))) (which a lower-bound of the first).
The graphical model in Figure 6a also highlights example of interesting future directions: Con- sider an offline RL setup where we keep track of the various policies generating the data, and their Q-functions. Then, by imposing prior the architecture tion over architectures, then learn the Q-function of a new policy under the posterior architecture distribution.
first infer posterior distribu-
22

---

Under review as a conference paper at ICLR 2022
and chome Mooiii setaed D94 200 2041 200-200000 Hnsiiratio wwu.e. Pil
234 Jo0t preriin aoceco cantaitr aariaio Feee SoAA Fullala
432
(8C, rartia radur-ddddddd dww.edeeeeeeeeeeeeeeeeee wigearreeeaaarraaa.. rigeraencl, PESVOLL Jbe
2ad rame arrmaae wwggetteeeeeeeee weennng GLatt Mobort, JOME
Tal. IoL-/eno ceate anate diere--t Surgprthe sumsemble sesttatii Jetverk 3ele siosa
(od, porrrneee weare Aulle www.erarnnennk 22us222 wew.aaiti wet.eeeeeeiinne www.mnaeeeeeee.
2201
recirtation
InerDDECDS enfeee
Figure 7: Results on the six antmaze domains. Each color represents the mean and standard devia- tion of results across 5 seeds for a particular hyperparameter setting (3 E -4, -8 and a E 0, 0.1.). As can be seen, results quite robust across wide range of hyperparameter values.
I PRACTICAL HYPERPARAMETER TUNING ADVICE
In reporting our results, we preferred to also report the hyperparameter values that we used, which may given impression of significant hyperparameter tuning. We emphasize that our results generally robust across a range of hyperparameter values. As an example, in Figure 7 we present results on the six antmaze domains. Each color represents the mean and standard deviation of results across 5 seeds for a particular hyperparameter setting (3 E -4, -8 and a E O, 0.1). As can be seen, results quite robust across wide range of hyperparameter values.
Here, we include practical advice on hyperparameter tuning when faced with a new domain. For given new domain, would first low 3 values: 3 E (-4, -8). If B -4 is clearly better than 3 -8 then we would guess that this new domain high pessimism may not be necessary and explore the use of 3 E 10,-1, 2). For the hyperparameter, we found that {0, 0.1, 1.0) is wide enough range to explore. If a = 1.0 was clearly better than the other values, then this would indicate us that maybe the offline dataset is narrow (lacks diversity, e.g. imitation learning datasets) and then we could increase the value of a. Generally, we would prefer to lower 3 before attempting to increase a, but we never tried 88
23


Reviews:
[{"review_id": "nlpOoKrTE-3", "reviewer": "Reviewer_dnQ3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The title of the paper: why so pessimistic? Is a bit little misleading. Actually most RL exploration methods are being the opposite: optimistic. \n\nOne major motivation of the reviewed paper is to “some of the unique challenges of uncertainty estimation in reinforcement learning ”. Could be explicit on this? The message/answer is not super clear after reading the paper. The DLTV paper has an observation that there are both intrinsic and parametric uncertainties going on with DRL. This shows the exploration for DRL is very challenging.  The DLTV paper uses distributional RL to estimate the upper confidence and lower confidence bound from the distribution of Q values, also using neural networks. \n\nThe presentation of the paper can be improved. Some times buzz words just come out without defining, like “deep ensembles”, \"efficient approximation to ensembles\"....\n\nThe results are evaluated on two ant mazes. The CQL paper performed pretty bad from Table 2. Did you try the other baseline? With only baseline and it’s poor, it is hard to see the proposed algorithm is strong. \n\nQuestions:\nWhat is the connection of using ensembles and distributional RL for uncertainty estimation? In a superficial sense, both use a number of agents. The only difference is how they form an uncertainty estimator. However, identifying their connections and differences are interesting, isn’t it? For distributional RL, e.g., see the DLTV algorithm:\nhttps://arxiv.org/abs/1905.06125\n\nActually the method proposed in this paper (estimating LCB, see Section 4.1) is very similar to DLTV. The formula is almost the same. Compare the equation above Sec. 4.2 with eq 4 of the DLTV paper. The only difference is that this is implemented using ensembles instead of distributional agents.  \n\nThe statement “update of each member’s update is completely independent from each other”. Here some clarification is necessary. By independence, do you mean the learning rule sense? Do you have shared parameters between the members? \n\nBeyond Method 1 and Method 2, have you thought about using shared maximum of the ensemble? See eq 8 of the ACE paper for example:\nhttps://arxiv.org/abs/1811.02696 \nWhen coming to ensembles, the crucial question is what operator to choose to combine these estimate. The use of mean or max is a fundamental question to investigate. \n\n\n\nMinor:\nSome reference links are missing. \n\nRefs:\nLSPI\nhttps://www.jmlr.org/papers/volume4/lagoudakis03a/lagoudakis03a.pdf\n\nLAMapi:\nhttps://sites.ualberta.ca/~szepesva/papers/lamapi.pdf\n\nRKHS:\nhttps://arxiv.org/abs/1206.4655\n\nPseudo-MDPs:\nhttps://sites.ualberta.ca/~szepesva/papers/ieee_adprl2014.pdf", "summary_of_the_paper": "There is a good literature for off-line RL. However, this  is reviewed at all. For example, LSPI (model free), LAMAPI (model-based), pseudo-MDPs (model-based) and RKHS embedding (model-based). The paper went straight to discussing the reduction of the value estimation error constraint. However, most of the algorithms use the error to directly learn the parameters. It is not necessarily formalized as constraint. ", "summary_of_the_review": "The paper addresses exploration with DRL algorithms, and explores estimating confidence interval of Q values using ensembles. Some discussion on the use of ensembles, the connection and difference from distributional RL, and comparisons to a good baseline (set) can make the paper much better and well supported. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "k7FGPflmub6", "reviewer": "Reviewer_QFka", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths. The algorithm in this paper is model-free, which means it can be applied to realistic settings.\nIt used \n\nWeakness. The theory part of this paper seems unfinished. The detailed derivation of the std should be included in the proof, and there should be more details, like descriptions of the network initialization,  in the description of the theory part \n\nTypo in Line 8 of page 4. \"This an ...\"", "summary_of_the_paper": "This paper provided a model-free algorithm for offline-RL. It provided Theorem 5.1 to justify its uncertainty quantifiers. It also conducted experiments to validate its algorithm.", "summary_of_the_review": "Although this paper proposed an algorithm that can be applied to real-life applications without imposing strong structural assumption to allow uncertainty quantification, the theory part of this paper should be improved to be more complete, rigorous and self-contained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hShcXYqIySh", "reviewer": "Reviewer_FzHF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper looks interesting, so are the approach, the theory, and observations. \n\n# Revisions/questions/remarks\n  * Roughly half of the papers refer to non peered reviewed papers (arXiv)\n  * Hasn't the D4RL dataset been updated since Kumar et al. publication (CQL) which could explain the differences observed by the authors?\n  * Lack of rigor in the theorems: the authors should have a formal theorem followed by its interpretation on which they base their intuitions\n  * Lack of rigor in the results presentation: what bold format means? Maybe add a statistical significance test.\n  * Too many claims not well supported\n  * Section 5.2: wouldn't it have more impact if you used the git attached to Lee et al. 2019 in Section 5.2 to match the framework the authors are working on?\n\n# Some typos\nLot of typos have to be corrected, here are some\n  * p.1: \"confident\", citation missing\n  * p.2: \"data(Guo et al., 2017)\" (missing space), \"only has\", \"success\"\n  * p.5: Eq. 4 missing \"i\", \"in the offline dataset. from the dataset.\"\n  * p.8: footnote of p.7, \"otherwis The\"\n  * p.9: \"computationl efficieny\"", "summary_of_the_paper": "The paper propose to study ensemble methods for offline RL in the NTK framework. They derive ", "summary_of_the_review": "I found the paper quite hard to follow and overall there is a lack of rigor. However, looking at the experiments, there might be something to explore.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8qx2gsnF0Ao", "reviewer": "Reviewer_s2kZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe work proposes a novel method to use independent ensembles, which might be interesting towards a better understanding of uncertainty quantification in offline RL. \nThe experiment results are encouraging, showing superior performance of the proposed methods. \nThe theoretical intuitions are backed up with simulations, which helps readers better understand the ideas.\n\nWeakness: I have several concerns that are listed below. \nIntuitively, the method of using independent initialization captures the randomness in the training procedure, i.e., conditional on the observed data. I was a bit concerned how such training uncertainty relates to the uncertainty of data with respect to the true value. Intuitively, the uncertainty often refers to the data being unable to capture the truth because of lack of samples. One possible relation is that latter uncertainty is large in regions with scarce data, where the former training uncertainty will also be large. I Since this is related to the core theme of the paper, could you explain more on such relations, or on the uncertainty this work targets at?\nThe writing of the paper might be improved. I think it would be better to have fewer emphasized bold-font sentences, so that readers can focus on really important points. There are also some missing citations and references as well as typos in the paper which hurts the reading. I would expect the authors to have a careful check and make the presentation more organized. \n\nThere are some further questions:\nWhat is the \\pi in equation (3) in Section 4.2?\nIn theorem 5.2, the difference in ensemble variances is only discussed very vaguely. I don’t quite understand the meaning - it would be better if a direct comparison with equations is presented.\nI am not sure whether the toy example and Figure 1 captures the advantage of MSG. Figure 1 shows the mean value and the standard deviations of the predictions, where the randomness can be viewed as from the training process, i.e., the random initialization, instead of the randomness in data. The crucial point should be that the estimations can provide a valid uncertainty quantification of the prediction with respect to the *true value*, so the shared LCB and shared min approach seems to directly target at the lower uncertainty bound (as observed in the paper). I would expect directly using the result of shared LCB/min can somehow provide a valid uncertainty quantification, without referring to the standard deviation induced by the training process. The approximation of the lower confidence bound should be more essential than evaluating the randomness in training process. Do you have more discussions on this issue?\nChoice of hyper-parameters: In the experiment part, it would be better to add discussions on how the parameter beta and alpha are chosen. They seem to change across different domains and I think readers would expect some justifications on such choices (I only see ‘hyperparameter searching’, but not sure whether it’s cherry picking). Are the performances of both methods stable across different choice of hyper-parameters? \n\nMinor points: \nA missing citation in the introduction (page 1). \nMissing “.” in Section 4.2. \nIs there a missing superscript i for y(r,s’,pi) in equation (4)?\nMissing reference in the caption of Figure 1.", "summary_of_the_paper": "This paper proposes a novel method to utilize ensemble in quantification of uncertainty in offline RL, where the particular structure of dynamic programming and cumulative error are unique and interesting. Some theoretical justification emphasizes the ability of independent TD targets to capture the variance of the prediction, hence provide uncertainty quantification. Simulations show the proposed method gets larger variance in low-confidence regions. In experiments in benchmark environments, the proposed method outperforms those in the literature. ", "summary_of_the_review": "The proposed method to use independent initializations to capture the uncertainty is interesting, validated in simulations and performs well in experiments. However, the credibility of the work is compromised by the confusion in the type of uncertainty it targets at, which is the key theme. The writing could also be improved by another check to fix typos and missing references.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1d6vnOUfRD", "reviewer": "Reviewer_XyHA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Offline RL is certainly an important area of research and there is still plenty to do, since the existing algorithms have all shown their limits. However I believe that the submission falls short on several crucial points that I enumerate below:\n* There is a terminology confusion with the word pessimistic. \n   * Offline RL inductive bias: Offline RL algorithms rely on the assumption that the control policy(ies) is(are) much better than average, and therefore that omitted actions are likely to be underperforming. This is a desirable feature, and the ex machina argument made by the authors that \"it would be preferable if we could place more trust into the predictions of value networks beyond the training dataset\" is not supported. And this is not what the authors propose to do: they simply propose a way to estimate uncertainty and to enforce this assumption: LCB means that the training is going to be conservative.\n   * Pessimistic meaning in the offline RL literature: Usually, pessimistic refers to the opposite of optimistic exploration: methods that distort the reward/value function to induce pessimism in the face of uncertainty. It is generally used in opposition to methods that conservatively constraint the policy search [Petrik2016], [Fujimoto2019], [Laroche2019]. Most (all?) RL algorithm can be viewed as policy iteration: loop over policy improvement and policy evaluation. Pessimism refers to pressure on the policy evaluation step, and constraint of the policy search refers to pressure on the policy improvement step. The method developed in the paper applies pressure on the value function and is therefore pessimistic, hence the confusion for the readers. There is even a third approach consisting on early stopping the loop [Brandfronbrener2021].\n* The novelty is limited and the positioning incomplete. As pointed out by the authors, the use of deep ensembles for offline RL is not novel. The authors claim that the networks share target. Usually yes, but not always: Algorithm 2 of [Osband2018] is training each network independently. And when they train them all together, they share a form of pessimistic target (reusing the paper terminology: independent double Q, shared LCB, shared min), not the expectation (shared mean), otherwise it does not make sense!\n* The theoretical analysis is not convincing. Following my previous point, it would have made the analysis much stronger if the other forms of pessimistic updates were accounted for and analyzed. It is straightforward that using a common target will provide a collapse of the inter-network variance. The use of infinite width NN is a big hammer for such a small nail. Figure 1 reports an experiment but its result is inconclusive since all pessimistic updates deliver the same lower bound.\n* The authors claim that shared LCB, shared min, CQL were unable to train effective policies, although these have been reported in the past. This, combined with the lack of compelling arguments on the theory side, cast doubt in the empirical results. Also, \"the other data setting could not adequately differentiate between competitive methods\" is not an argument for not reporting results.\n\n[Petrik2016] Petrik, M., Ghavamzadeh, M., & Chow, Y. (2016). Safe policy improvement by minimizing robust baseline regret. Advances in Neural Information Processing Systems, 29, 2298-2306.\n\n[Osband2018] Osband, I., Aslanides, J., & Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv:1806.03335.\n\n[Fujimoto2019] Fujimoto, S., Meger, D., & Precup, D. (2019, May). Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (pp. 2052-2062). PMLR.\n\n[Laroche2019] Laroche, R., Trichelair, P., & Des Combes, R. T. (2019, May). Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (pp. 3652-3661). PMLR.\n\n[Brandfronbrener2021] Brandfonbrener, D., Whitney, W. F., Ranganath, R., & Bruna, J. (2021). Offline RL Without Off-Policy Evaluation. arXiv preprint arXiv:2106.08909.", "summary_of_the_paper": "The paper agues that methods, which they call \"support constraint\", consisting in assuming that any action outside the training yields negative outcomes, lead the agent's training to be overly pessimistic. In contrast, they propose MSG an algorithm where a lower bound of the value function is estimate from ensembles of networks. They provide some theoretical arguments on infinite-width neural networks and validate their methods on many applications, ranging from illustrative toy examples to classic continuous state-action spaces offline RL benchmarks.", "summary_of_the_review": "I recommend reject for the reasons detailed in the full review.\n\nIf the authors could clarify the following things, I would consider raising up my score:\n* How does their algorithm differs from Algorithm 2 of [Osband2018]?\n* Why would the LCB of independent networks would have a better uncertainty estimate than an ensemble network learnt on common LCB targets?\n* Why were the authors unable to train effective policies of well established algorithms?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attacks an interesting problem: accurately estimating uncertainties in action-value estimates in offline RL. It proposes a method based on ensembles of Q functions, where we alternately train an ensemble to estimate Q(s,a) for the current policy, and then adjust our policy based on the mean and uncertainty in this ensemble. By choosing mean + \beta * [standard deviation] as the basis for our policy updates, we can be either conservative (\beta < 0) or optimistic (beta > 0). The paper analyzes the ensemble training using the Gaussian process (NTK) view of deep nets.

The largest weakness of the paper is a lack of rigor in its analysis. While its main topic is uncertainty in Q estimates, the paper does not specify a valid probabilistic model on which such uncertainty estimates could be based. The theorems analyze only a part of the algorithm (policy evaluation), and don't take into account the interplay between this evaluation and any policy updates. The theorems also do not show that the computed output distribution is relevant to the actual uncertainty of the algorithm; e.g., they do not describe a prior for which the ensemble approximates the correct posterior (nor any other similar notion). Despite these omissions, the theorems are nonetheless presented as providing a reason to trust the output of the algorithm.

On the other hand, there definitely is valuable material in the paper; the experiments are interesting (and would be even more interesting if we could compare to some notion of a correct answer for at least the small ones), and the intuition and analysis could be enlightening if presented more clearly and formally, with a better description of the connection between theory and practice. Unfortunately, the paper as written doesn't enable the reader to accurately understand and assess the contributions.
Decision: Reject