Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
Authors: Seyed Kamyar Seyed Ghasemipour
Shixiang Shane Gu
Ofir Nachum
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline reinforcement learning
batch reinforcement learning
ensembles
uncertainty estimation

Abstract: Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of $Q$-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member's Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of $Q$-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL benchmarks demonstrate that on challenging domains such as antmazes, MSG with deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide margin. Additionally, through ablations on benchmarks domains, we verify the critical significance of using independently trained $Q$-functions, and study the role of ensemble size. Finally, as using separate networks per ensemble member can become computationally costly with larger neural network architectures, we investigate whether efficient ensemble approximations developed for supervised learning can be similarly effective, and demonstrate that they do not match the performance and robustness of MSG with separate networks, highlighting the need for new efforts into efficient uncertainty estimation directed at RL.

Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
Seyed Kamyar Seyed Ghasemipour Roboties @ Google kamyar@google.. com
Shixiang Shane Gu Google Research, Brain Team shanegu@google. com
Ofir Nachum Google Research, Brain Team of rrirnachum@google.com
Abstract
Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member's 's Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of Q-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL benchmarks demonstrate that on challenging domains such as antmazes, MSG with deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide margin. Additionally, through ablations on benchmarks domains, we verify the critical significance of using independently trained Q-functions, and study the role ensemble size. Finally, as using separate networks per ensemble member can become computationally costly with larger neural network architectures, we investigate whether efficient ensemble approxima- tions developed for supervised learning can be similarly effective, and demonstrate that they do not match the performance and robustness MSG with separate networks, highlighting the need for new efforts into efficient uncertainty estimation directed at RL.
Introduction
Offline reinforcement learning (RL), also referred batch RL [1] is problem setting in which one is provided a dataset of interactions with an environment in the form of a Markov decision process (MDP), and the goal is learn effective policy exclusively from this fixed dataset. Offline RL holds the promise of data-efficiency through data reuse, and improved safety due to minimizing the need for policy rollouts. As a result, offline RL has been the subject of significant renewed interest in the machine learning literature 2
One common approach offline RL in the model-free setting is approximate dynamic programming (ADP) to learn Q-value function via iterative regression to backed-up target values. The predominant algorithmic philosophy with most success in ADP-based offline RL is encourage
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

obtained policies to remain close to the support set of the available offline data. A large variety of methods have been developed for enforcing such constraints, examples of which include regularizing policies with behavior cloning objectives 3 4 performing updates only on actions observed inside 5
or close to [9] the offline dataset, and regularizing value functions to underestimate the
value of actions not seen in the dataset [10] [2].
The need for such regularizers arises from inevitable inaccuracies in value estimation when function approximation, bootstrapping, and off-policy learning- i.c. The Deadly Triad 13 are involved. In offline RL in particular, such inaccuracies cannot be resolved through additional interactions with the MDP. Thus. remaining close to the offline dataset limits opportunities for catastrophic inaccuracies to arise. However, recent works have argued that the aforementioned constraints can be overly pessimistic, and instead opt for approaches that take into consideration the incertainty about the value function [4 15 [6], thus re-focusing the offline RI. problem to that of deriving accurate lower confidence bounds (LCC) of Q-values.
In the empirical supervised learning literature, deep network ensembles (definition in Appendix L) and their more efficient variants have been shown to be the most effective approaches for uncertainty estimation, towards learning calibrated estimates and confidence bounds with modern neural network function approximators [17] Motivated by this, in our work we take a renewed look into Q-ensembles, and study how to leverage them as the primary source of pessimism for offline RL.
In deep RL, a very popular algorithmic choice is to use an ensemble of Q- O-fuuctions to obtain pes- simistic value estimates and combat overestimation bias [88. Specifically, in the policy evaluation procedure, all Q-networks are updated towards a shared pessimistic temporal difference target. Simi- larly in offline RL, in addition the main offline RL objective that they propose, several existing methods use such Q-ensembles [1O 3 20 21 23 88. We begin by mathematically character- izing critical flaw in the aforementioned ensembling procedure. Specifically, we demonstrate that using shared pessimistic targets can paradoxically lead to Q-estimates which are in fact optimistic! We verify our finding by constructing pedagogical toy MDPs. These results demonstrate that the formulation of using shared pessimistic targets is fundamentally ill-formed.
To resolve this problem, propose Model Standard- deviaaion Gradients (MSG), an ensemble- based offline RI. algorithm. In MSG, each Q-network is trained independently, without sharing targets. Crucially, ensembles trained with independent target values will always provide pessimistic value estimates. The pessimistic lower-confidence bound (LCB) value estimate - computed as the mean minus standard deviation of the Q-value ensemble- is then used to update the policy being trained. Evaluating MSG on the established D4RL (24) and RL Unplugged (25) benchmarks for offline RL, we demonstrate that MSG matches, and in the more challenging domains such antmazes, significantly exceeds the prior state-of-the-art. Additionally, through a series of ablation experiments on benchmark domains, we verify the significance of our theoretical findings, study the role of ensemble size, and highlight the settings in which ensembles provide the most benefit.
The use of ensembles will inevitably be computational bottleneck when applying offline RL to domains requiring large neural network models. Hence, as a final analysis, we investigate whether the favorable performance of MSG can be obtained through the of modern efficient ensemble approaches which have been successful in the supervised learning literature [26] (77 28, [7]. We demonstrate that while efficient ensembles are competitive with the state-of-hheane on simpler offline RL benchmark domains, similar to many popular offline RL methods they fail on more challenging tasks, and cannot recover the performance and robustness of MSG using full ensembles with separate neural networks.
Our work highlights some of the unique and often overlooked challenges of ensemble-basee uncer- tainty estimation in offline RL. Given the strong performance of MSG, we hope our work motivates increased focus into efficient and stable ensembling techniques directed at RL, and that it highlights intriguing research questions for the community of neural network uncertainty estimation researchers whom thus far have not employed sequential domains such as offline RL. testbed for validating modern uncertainty estimation techniques.
2 Related Work
Uncertainty estimation is a core component of RI., since an agent only has limited view into the mechanics of the environment through its available experience data. Traditionally, uncertainty

---

estimation has been key to developing proper exploration strategies such as upper confidence bound (UCB) and Thompson sampling (29]. in which an agent is encouraged to seek out paths where its uncertainty is high. Offline RI presents an alternative paradigm, where the agent must act conservatively and is thus encouraged to seek out paths where its uncertainty is low [14] In either case, proper and accurate estimation of uncertainties is paramount. To this end, much research has been produced with the aim of devising provably correct uncertainty estimates [30 31 [2], or at least bounds uncertainty that are good enough for acting exploratorily [33 conservatively [34]. However, these approaches require exceedingly simple environment structure, typically either a finite discrete state and action space linear spaces with linear dynamics and rewards.
While theoretical guarantees for uncertainty estimation are more limited practical situations with deep neural network function approximators, number of works have been able to achieve practical success, for example using deep network analogues for count-based uncertainty 35), Bayesian uncertainty 136 37. and bootstrapping 38 [9].
Many of these methods employ ensembles. In fact, in continuous control RI., common to use an ensemble of two value functions and use their minimum for computing target value during Bellman enror minimization [8]. A number of works in offline RL have extended this to propose backing up minimums or lower confidence bound estimates over larger ensembles B 10 19
23 21I. In our work, we continue to find that ensembles are extremely useful for acting conservatively, but the manner in which ensembles used is critical. Specifically proposed MSG algorithm advocates for using independently learned ensembles, without sharing of target values, and this important design decision is supported by empirical evidence.
20
22
The widespread success of ensembles for uncertainty estimation in RL echoes similar findings in supervised deep learning. While there exist proposals for more technical approaches to uncertainty estimation [40] 41 42), ensembles have repeatedly been found to perform best empirically 126 43]. Much of the active literature ensembles in supervised learning is concerned with computational efficiency, with various proposals for reducing the compute or memory footprint of training and inference large ensembles [28 27 While these approaches have been able to achieve impressive results in supervised learning. our empirical results suggest that their performance suffers significantly in challenging offline RL settings compared to deep ensembles.
Pessimistic Q-Ensembles: Independent or Shared Targets?
In this section we identify a critical flaw in how ensembles commonly employed in offline as well as online RL- for obtaining pessimistic value estimates [0]
19
20
21
22
23
g
21],
which can paradoxically lead an optimism bonus! We begin by mathematically characterizing this problem and presenting a simple fix. Subsequently, we leverage our results to construct pedagogical toy MDPs demonstrating the practical importance of the identified problem and solution.
3.1 Mathematical Characterization
We assume access to a dataset D com- posed of (s, a, s') transition tuples from a Markov Decision Process (MDP) deter- mined by a tuple M A, R. P, P. corresponding to state space, action space, reward function, transitions dynamics, and discount, respectively. As is standard in RL, we do not assume any knowledge R, P. other than that implicitly provided by the dataset D. this section, for clarity of exposition, we assume that the policies we consider are deterministic, and that our MDPs do not have terminal states.
1. Initialize 0 for all i Z.
2. For 1.2....:
For each (s, a, r, s') E D and i E Z com- pute target values y (r, 8r)
For each i E Z, update Ø to optimize the regression objective
1
Σ
(s)
D
(N.D,,,)))D
3. Return a pessimistic Q-value function Qpessimistic based on the trained ensemble.
We consider Q-value ensemble members given by parameterization Qø:, where i indexes into some set Z, which finite in practice but may be infinite or uncountable in theory. We assume Z has associated probability space allowing to make expectation E or variance V
3

---

computations over the ensemble members. Given fixed policy म, general dynamic programming based procedure for obtaining pessimistic value estimates is outlined by the iterative regression described in the box above.
A key algorithmic choice in this recipe is where pessimism should be introduced. This can be done by either (a) pessimistically aggregating Q-values after training, i.c. inside Step 3, or (b) also incorporating pessimism during Step 2, by using shared pessimistic target value y. Through our review of the offline RL (as well as online RL) literature, we have observed that the most common approach the latter, where the targets are pessimistic, shared, and identical across ensemble members [10 19 20 21 22 23 8) Specifically, they are computed as, y' s', T) PO([r 7Qp (s', " (s))). E 2 with PO being desired pessimism operator aggregating the TD target values of the ensemble members (e.g. "mean minus standard deviation", ". or "minimum") "),
In this section, our goal is compare these two alternative approaches. For analysis, we will use "mean minus standard deviation" (a lower confidence bound (LCB)) as our pessimism operator, and use the notation QLCB in place of pessimistic (defined in the box above). Under the LCB pessimism operator we will have:
Independent Targets (Method I): (r(,, 8',)) + Qa ((s))
Shared Targets (Method 2): ('(,,''=z) = =r (Fen (Qa: (T V
Vens (Q
TT( (
For both we have: Qr.cn(s, Eens (Q a)) / Vens (s,
To characterize the form of QieBR when using complex neural networks, we refer to the work on wndth neural networks, namely the Neural Tangent Kernel (NTK) 45. We consider Q-value ensemble members, Qas, which all share the same infinite-width neural network architecture (and thus the same NTK parameterization). As noted in the algorithm box above, and as is the case in deep ensembles 43). the only difference amongst ensemble members Qo is in their initial weights 0 sampled from the neural network's initial weight distribution.
Before presenting our results, we establish some notation relevant to the infinite-width and NTK regime. Let X. R, X' denote data matrices containing (s, a), r, and (s', xx')) appearing in the offline dataset D: i... the k-th transition (s, o, r, s') in is represented by the k--th rows X, R, X'. Let A, B denote two data matrices, where similar (s, a) E S x A. The NTK, which governs the training dynamics of the infinitely-wide neural network, then given by the outer product of gradients of the neural network at initialization: (()) B) VaQu- (A) Qo (B)T lt=0- where we overload notation Qa (A) to represent the column vector containing Q-values. At infinite-width in the NTK regime, 6 (A, B) converges to a deterministic kernel (i.e. does not depend on the random weight sample 0³), and hence the same for all ensemble members. Thus, hereafter we will remove the index i from the notation of the NTK kernel and simply write, ((0) (A, B). With our notation in place, we define, C == ( (x', X) ( (') Intuitively, C' is DD] x DI matrix where the element at column q, similarity between (s, a) in the 9h row of X, and (s', ((s') in the pth row of X".
X", each
contains atate-cciion tuple
p. captures notion of
We now have all the necessary machinery to characterize the form of Qt.cn:
Theorem 3.1. For given (s) SxA, let Q (s, a) denote Qo (s. a),=o (value at initialization). with 9 sampled from the initial weight distribution. After t + iterationn of pessimistic policy evaluation. the LCB value estimate for (s', mss))) E X' is given by.
Independent Targets (Method I):
QUi (2') O(1]]())))) (1+ +
Fems
( "CC" .0 (x))) 2
backopp term
box Kaep term
Shared Targets (Method 2):
(2)
Qitt (x') =(>''((1))) + +1+ +C
27 + Eeas (0000 (X') - CQ220 (x))
buckap term
buchhup term

---

where the square and square-root operations are applied element-wise Please refer to Appendix for the proof.
As can be seen, the equations for the pessimistic LCB value estimates in both settings are similar, only differing in the third term. The first term is negligible and tends towards zero as the number of iterations of policy evaluation increases. The second term shared by both variants corresponds to the expected result of the policy evaluation procedure without any pessimism (as before, we mean expectation under 0 sampled from the initial weight distribution). Accordingly, the differing third term in each variant exactly corresponds to the "pessimism" or "penalty" induced by that variant.
Considering the available offline RI. dataset D as restricted MDP in itself, we see that the use of Independent Targets (Method 1) leads to pessimism term that performs "backups" along the trajectories that the policy would experience in this restricted MDP (using the geometric term I ++'.)+"CC)) before computing a variance estimate. Meanwhile the use of Shared Targets (Method 2) does the reverse it first computes a variance term and then performs the "backups".
While this difference may seem inconsequential, becomes critical when one realizes that in Equation for Shared Targets (Method 2), the pessimism term (third term) may become positive, i.e. negative penalty, yielding an effectively optimistic LCB estimate. Critically, with Independent Targets (Method
1), this problem cannot occur.
3.2 Validating Theoretical Predictions
In this section we demonstrate that our analysis not solely a theoretical result concerning the idiosyncracies of infinite-width neural networks, but that it is rather straightforward to construct combinations of an MDP, offline data, and a pol- icy, that lead to the critical flaw of an optimistic LCB estimate.
1. Initialize empty X, R, X
For N episodes:
sample (0,
For T steps:
sample a ~ 10,
sample s' T((,T)
Let dz denote the dimensionality of state and action vectors respectively. We consider an MDP whose initial state distribution a spher- ical multivariate normal distribution (N. I), and whose transition function is given by p(s'|s,a) = (0,, I). Consider the procedure for generating our offline data matrices, de- scribed in the box to the right. This procedure returns data matrices X, R, X' by generating N episodes of length T, using a behavior policy (O, 1). In this generation process, we set the policy we seek to pessimistically evaluate, x, to always apply the behavior policy's action in state s to the next state s'.
set T(s') ta a
Add (s, a) to X
Add r (0, 1) to R
Add (s', (s')) to x"
Set s + s'
3. Return the offline dataset X, R, X"
To construct our examples, we consider the setting where we use linear models to represent Qg., with the initial weight distribution being a spherical multivariate normal distribution, (0. With linear models, the equations for QI.C takes an identical form to those in Theorem 3.1
Given the described data generating process and our choice of linear function approximation, we can compute the pessimism term for the Shared Targets (Method 2) (i.e. the third term in Theorem 3.1 Equation 2). We implement this computation in a simple Python script, which we include in the supplementary material. We choose, d, = 30, d, = 30, = 0.5, N = 5, T = 5, and t = 1000 is the exponent in the geometric term above). We run this simulation 1000 times, each with a different random seed. After filtering simulation runs to ensure 7 7/|C|| < 1 (as discussed in an earlier footnote). we observe that 221 of the simulation runs result in an optimistic LCB bonus, meaning that in those experiments, the pessimism term was in fact positive for some (s', (ss)) E X'. We have made the python notebook implementing this experiment available in our supplementary material. For further intriguing investigations in pedagogical toy MDPs regarding the structure of uncertainties, we strongly encourage the interested reader refer to Appendix G
1. dynamic programming is liable to diverge in either setting. In our discussions, we avoid this degenerate case and assume >||C7|| 1.
Note that if >||||| 2
5

---

4 Model Standard-deviation Gradients (MSG)
important to note that even if the pessimism term does not become positive for a particular combination of MDPs, offline datasets, and policies, the fact that it can occur highlights that the formulation of Shared Targets fundamentally ill-formed. To resolve this problem we propose Model Standard-deviatin Gradients (MSG), an offline RL algorithm which leverages ensembles to approximate the LCB using the approach of Independent Targets.
4.1 Policy Evaluation and Optimization in MSG
MSG follows an actor-critic setup. At the beginning of training, we create an ensemble of N Q- functions by taking N samples from the initial weight distribution. During training, in each iteration, we first perform policy evaluation by estimating the QLCB for the current policy, and subsequently optimize the policy through gradient ascent on QLCB.
Policy Evaluation As motivated by our analysis in Section 3 we train the ensemble Q-functions independently using the standard least-squares Q-evaluation loss,
((0') -EGanna (Qa (s. (()))) y'=r+ EErrt--eratte (o (5 a')
(3)
where 8¹, H denote the parameters and target network parameters for the 7thh Q-function.
In each iteration, as is common practice, we do not update the Q-functions until convergence, and instead update the networks using single gradient step. In practice, the expectation in C(O') is estimated by a minihatch, and the expectation in y² is estimated with a single action sample from the policy. After every update the Q-function parameters, their corresponding target parameters are updated to be an exponential moving average of the parameters in the standard fashion.
Policy Optimization As in standard deep actor-critic algorithms, policy evaluation steps (learning Q) are interleaved with policy optimization steps (learning т). In MSG, we optimize the policy through gradient ascent on QLCB. Specifically, our proposed policy optimization objective in MSG is,
((m) Fgg.D.axx(e) (QLCB a)] = E [FenlQu (s, (8)a)) Vom (4)
where 3 < is hyperparameter that determines the amount of pessimism.
4.2 The Trade-Off Between Trust and Pessimism
While our hope is to leverage the implicit generalization capabilities of neural networks to estimate proper LCBs beyond states and actions in the finite dataset D, neural network architectures can be fundamentally biased, or we can simply be in a setting with insufficient data coverage, such that the generalization capability of those networks limited. To this end, augment the policy evaluation objective of MSG (L(0", equation 3 with a support constraint regularizer inspired by CQL 11 H(('') P.a-D.a>stt [Qar(s, (s.(s.a) a)] ((,,a)=-D Qu (s)a)) This regularizer encourages the Q-functions to increase the values for actions seen in the dataset D, while decreasing the values of the actions of the current policy. Practically, we estimate the latter expectation of H using the states in the mini-batch, and we approximate the former expectation using single sample from the policy. We control the contribution of H(O)) by weighting this term with weight parameter a. The full critic loss is thus given by,
N
0 1O......NN) = Σ (C(00)+H(((1)))
(5)
Empirically, as evidenced by our results in Appendix A.2 we have observed that such a regularizer can be necessary in two situations: 1) The first scenario where the offline dataset only contains narrow data distribution (e.g., imitation learning datasets only containing expert data). We believe
Instead of a CQL-style value regularizer, other forms of support constraints such as a behavioral cloning regularizer on the policy could potentially be used.

---

200
Sr SATAAAR
Unplugged
(N=64)
Results for DM Control Suite subset of the RL Unplugged benchmark [25). We note that: 1) the architecture used for MSG is smaller by factor of approximately 60x which contributes poor performance humanoid.. run, CRR results are reported by their best checkpoint throughout training which differs from MSG, BC, and MuZero Unplugged which report performance at the end of training. Baseline results taken from 47). Despite MSG's disadvantage on the humanoid. run task, it still edges out the bascline methods in mean performance.
Figure 1:
Domain
CQL
IQL
MSG (N 64)
3
MSG (N 4)
8
maze2d-umaze-vl
5.7
101.1 + 26.3
-8
0
68.8 + 20.2
-4
0.1
maze2d-medium-
5.0
57.0 17.2
4
0.1
53.2 + 26.8
4
0.1
maze2d-large-vl
12.5
159.3 49.4
4
0.1
59.2 59.1
antmaze-umazz-vv)
74.0
87.5
0.5
98.6 1.4
1.0
satiaaze-umazeddiveer
84.0
62.2
81.8 3.0
4
76.6
untmaze-medium play-v0
61.2
71.2
89.6 -
4
0.5
83.0 1.1
4
0.1
(ldmaaze-mediun-divereeee.
53.7
70.0
88.6 2.6
-4
0.5
83.0 - 6.2
4
0.5
antmaze-large-play-V antmazz-large-dvvers
15.8
39.6
72.6 - 7.0
-8
46.8 14.7
-4
0.5
14.9
47.5
71.4
-8
0.1
0.1
Table 1: Results on D4RL maze2d and antmaze domains. In MSG, 9 is the hyperparameter controlling the amount of pessimism in QLEB (Equation A and is the hyperparameter controlling the contribution of the unable reproduce CQL antmaze results despite extensive hyperparameter tuning (see also 12] we present numbers reported by the original paper which uses same network architectures as MSG. We also present reported results for the current state-of-the-art, IQL. 48
CQL-style regularizer (Equation As
this is because the power of ensembles comes from predicting value distribution for unseen a) based on the available training data. Thus, if no data for sub-optimal actions is present, ensembles cannot make accurate predictions and increased pessimism via 7. becomes necessary. The second scenario is where environment dynamics can be chaotic (e.g. Gym [46] hopper and walker2d). In such domains it would beneficial to remain close to the observed data the offline dataset. Pseudo-code for our proposed MSG algorithm can be viewed in Algorithm Box
5 Experiments
In this section we seek to empirically answer following questions: 1) How well does MSG perform compared to current state-of-the-art in offline RL? 2) Are the theoretical differences in ensembling approaches (Section3) practically relevant? 3) When and how does ensemble size affect perfomance? 4) Can match the performance of MSG through efficient ensemble approximations developed the supervised learning literature?
5.1
Offline RL Benchmarks
D4RL Gym Domains We begin by evaluating MSG on the Gym domains cheetah hopper, walker2d) of the D4RL offline RL benchmark [24], using the medium, medium-reph -eellar and expert data settings. Our results presented in Appendix A.2 (summarized in Figure demonstrates that MSG competitive with well-tuned state-of-the-at methods CQL.
and F-BRC [12]
D4RL Antmaze Domains Due to the narrow range of behaviors Gym environments, offline datasets for these domains tend to be very similar to imitation learning datasets. As result, many prior offline RL approaches that perform well on D4RL Gym fail on harder tasks that require stitching trajectories through dynamic programming (c.f.
48) An example of such tasks are the D4RL. antmaze settings, particular those in the antmaze med um and an antmaze-large environments. The data for antmaze tasks consists of many episodes of an Ant agent [66] running along arbitrary
7

---

paths in a maze. The agent is tasked with using this data to learn point-to to--tont navigation policy from one corner of the maze to the opposite corner. where rewards are given by a sparse signal that is when near the desired end location in the maze at which point the episode is terminated and O otherwise. The undirected, extremely sparse reward nature of antmaze tasks make them very challenging, especially for the large maze sizes.
Table 1 and Appendix B.2 present our results. To the best of our knowledge, the antmaze domains are considered unsolved, with few prior works reporting non-zero results on the large mazes M As can be seen, MSG obtains results that far exceed the prior state-of-the-an results reported by 48 While some works that use specialized hierarchical approaches have reported strong results as well [49], is notable that MSG is able solve these challenging tasks with standard architectures and training procedures, and this shows the power that ensembling can provide as long as the ensembling is performed properly!
48
RL Unplugged In addition to the D4RL benchmark, we evaluate MSG on the RL Unplugged benchmark [25]. Our results are presented in Figure We compare results for Behavioral Cloning (BC) and two state-of-the-at methods in these domains, Regularieed Regression (CRR) Z and MuZero Unplugged [47]. Due computational constraints when using deep ensembles, we use the same network architectures as we used for D4RL experiments. The networks we use are approximately 60 -th the size of those used by the BC, CRR, and MuZero Unplugged baselines in terms of number of parameters. We observe that MSG is on par with or exceeds the current state-of- the-art on all tasks with the exception of humanoid, run, which appears require the larger architectures used by the baseline methods. Experimental details can be found in Appendix C
Benchmark Conclusion Prior work has demonstrated that many offline RL approaches that per- form well on Gym domains, fail to succeed on much more challenging domains [48]. Our results demonstrate that through uncertainty estimation with deep ensembles, MSG is able to very signifi- cantly outperform prior work on very challenging benchmark domains such as the D4RL antmazes.
5.2 Ensemble Ablations
Independence in Ensembles Ablation In Section through theoretical arguments and toy experiments we demonstrated the importance of training using "Independent" ensembles. Here, we seek to validate the significance of our theo-
antmaze-medium-die
Encemsie Fbw
Treamble Siz d
Iexembble Gie
treamble Siee
antimaze medium play.vO
retical findings using offline RL benchmarks, by
comparing Independent targets (as in MSG), to Shared-LCE and Shared-Min targets. Our results are presented in Appendices A.3 and B.3 with a summary in Figures and
Creemoia Sbn
frasmble Stze
traemble
treamtte Size
antmaae- antmaze-large-din becpeeellrro-divveese
In the Gym domains (Appendix A.3 with en- semble size N 4, Shared-LCB significantly underperforms MSG. In fact, not using ensem- bles at all (N = 1) outperforms Shared-LCB. With ensemble size N = 4, Shared-Min is on par with MSG. When the ensemble size is increased to N 64 (Figure (7) we observe the performance of Shared-Min drops signif- icantly on 7/12 D4RL Gym settings. In con- strast, the performance of MSG stable and does not change.
5+0
-4
antrmazo-targe-play
Z+0
p-
2-
2- 8
Figure 2: Ensemble size ablation on the antmaze
medium and large domains with varying 8 E (-4, 88 and E {0, 0.1, 0.5, 1} (colored blue, red, yellow, and green respectively). We observe the general trend that bigger ensembles lead to better performance.
In the challenging antmaze domains (Appendix B.3). for both ensemble sizes N = 4 and N = 64, Shared-LCB and Shared -Min targets completely fail solve the tasks, while for both ensemble sizes MSG exceeds the prior state-of state-of-the-art (Table T]
IQL [48).
8

---

srensee-aaae-aaaeo Now Wast insit
HGdA NOLA 15-41
Viarocii Deep una 16-41 Skand Hir cetooriooooo Lie, 20-41
Figure 3: Summary of D4RL antmazze benchmark results (full results presented in Appendix B For each method, we report the experiments are available the supplementary material.
random seeds for the best hyperparameter. Numerical results for all
Independence in Ensembles Conclusion Our experiments corroborate the theoretical results in Section 3 demonstrating that Independent targets are critical to the success of MSG. These results are particularly striking when one considers that the implementations for MSG, Shared-LCB, and Shared-Mar differ by only 2 lines of code.
Ensemble Size Ablation
An important ablation is to understand the role ensemble size in MSG.
In the Gym domains, does not result in a noticeable change in performance.
demonstrates that increasing the number of ensembles from 4 to 64
In the antmaze domains, we evaluate MSG under ensemble sizes {1, 4, 16, 64). Figure presents our results. Our key takeaways are as follows:
For the harder antmaze-large tasks, there is clear upward trend as ensemble size increases.
Using small ensemble size (e.g. NN = 4) is already quite good, but more sensitive to hyperparam- eter choice especially on the harder tasks.
Very small ensemble sizes benefit more from using a 0 However, across the board, using 0 preferable to using too large of value for with the exception of N 1 which cannot take advantange of the benefits of ensembling.
When using lower values of 8, lower values of a should be used.
Ensemble Size Conclusion In domains such as D4RL Gym where offline datasets are qualitatively similar imitation learning datasets, larger ensembles do result in noticeable gains. In domains such as D4RL antmaze which contain more data diversity, larger ensembles significantly improve the performance of agents.
5.3 Efficient Ensembles
Thus far have demonstrated the significant performance gains attainable through MSG. An important concern however, is that of parameter and computational efficiency: Deep ensembles of Q-networks result in an --fold increase in memory and compute usage, both in the policy evaluation and policy optimization phases of actor-critic training. While this might not be a significant problem in offline RL benchmark domains due to small model footprints" becomes major bottleneck with larger architectures such as those used in language and vision domains. To this end, we evaluate whether recent advances in "Efficient Ensemble" approaches from the supervised learning literature transfer well to the problem of offline RL. Specifically, the efficient ensemble approaches we
consider are: Multi-Heas Ensembles (26 50 51, MIMO Ensembles (27,, and Batch Ensembles (28 For description of these efficient ensembling approaches please refer to Appendix F A runtime comparison of different ensembling approaches can be viewed in Table
D4RL Gym Domains Appendix A.4 presents our results in the D4RL Gym domains with ensemble
size N = 4 (summary in Figure 4 Amongst the considered efficient ensemble approaches, Batch Ensembles (28 result the best performance, which follows findings from the supervised learning
literature 17
3 As reminder, a is weight of -tl--ttye regularizer loss discussed Section All our experiments
single Nvidia P100 GPU.

---

D4RL Antmaze Domains Appendix B.4
presents our results in the D4RL antmaze domains for both ensemble sizes of N and 64 (summary in Figure As can be seen, compared to MSG with deep ensembles (separate networks), the efficient ensemble approaches we consider are very unreliable, and fail for most hyperparameter choices.
Efficient Ensembles Conclusion We believe the observations in this section very clearly motivate future work in developing efficient uncertainty estimation approaches that are better suited to the domain reinforcement learning. To facilitate this direction of research, in our codebase we have included a complete boilerplate example of offline RI. agent, amenable to drop-in implementation of novel encertainty-estimatie techniques.
6 Discussion & Future Work
Our work has highlighted the significant power of ensembling as a mechanism for uncertainty estimation for offline RL. In this work took a renewed look into Q-ensembles, and studied how to leverage them as the primary source of pessimism for offline RL. Through theoretical analyses and toy constructions, for obtaining pessimistic Q-values, and demonstrated that it can in fact lead to optimistic estimates. Using simple fix, we developed a practical deep offline RL algorithm, MSG, which resulted in large performance gains on established offline RL benchmarks.
demonstrated critical flaw in the popular approach of using shared targets
As demonstrated by experimental results, important outstanding direction is study how we can design improved efficient ensemble approximations, as we have demonstrated that current approaches used in supervised learning are not nearly as effective MSG with ensembles that use separate networks. We hope that this work engenders new efforts from the community of neural network uncertainty estimation researchers towards developing efficient uncertainty estimation techniques directed at reinforcement learning.
Acknowledgments and Disclosure of Funding
We would like to thank Yasaman Bahri for insightful discussions regarding infinite-width neural networks. We would like to thank Laura Graesser for providing a detailed review of our work. We would like to thank conference reviewers for posing important questions that helped clarify the organization of this manuscript.
References
[1] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[2] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.00643 2020.
[3] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:/906.00 2019.
[4] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06866 2021.
[5] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiighte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00/77 2019.
[6] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09335 09309, 2020.
[7] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression, arXiv preprint arXiv:2006. 15134, 2020.
10

---

[8] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max g-leariig operator for simple yet effective offline and online rl. International Conference Machine Learning, pages 3682-3691 3691. PMLR, 2021.
[9] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[10] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXtv:1911.11361, 2019.
[11] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
[12] Hya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offlie reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 3774-5784 PMLR, 2021.
[13] Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 1xXiv:8112006666 2018.
[14] Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.061 06799, 2020.
[15] Ying Jin, Zhuoran Yang. and Zhaoran Wang. pessimism provably efficient for offline rI? In International Conference Machine Learning, pages 5084-5096. PMLR, 2021.
[16] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. arXiv preprint arXiv:2106.0692 2021.
[17] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sehastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncer- tainty? evaluating predictive uncertainty under dataset shift. arXiv preprint 190i:029000023300 2019.
[18] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018
[19] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, pages (04-114 PMLR, 2020.
[20] Jordi Smit, Canmanie T Ponnambalam, Matthijs TJ Spaan, and Frans A Oliehoek. Pebl: Pessimistic ensembles for offline deep reinforcement learning. In Robust and Reliable Autonomy in the Wild Workshop the 30th International Joint Conference of Artificial Intelligence, 2021.
[21] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline reinforcement learning with diversified gennemble.. Advances in Neural Information P Processing Systems, 34, 2021.
[22] Kimin Lee, Michael Laskin. Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages 6131-6141. PMLR, 2021.
[23] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[24] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXivv2004.072119, 2020.
11

---

[25] Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: collection of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[26] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 151 1.06310 14, 2015.
[27] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610. 2020.
[28] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
[29] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[30] P. Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence off-policy evaluation. In Proceedings of the 29th Conference on Artificial Intelligence, 2015.
[31] Yihao Feng, Tongzheng Ren, Ziyang Tang, and Qiang Liu. Accountable off-policy evaluation with kernel bellman statistics. In International Conference on Machine Learning, pages 3102-
3111. PMLR, 2020.
[32] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. arXiv preprint urXiv:2010.11633 2020.
[33]
Alexander Strehl, Lihong Li, and Michael Littman. Reinforcement learning finite mdps: Pac analysis. Journal of Machine Learning Research, 10(1)), 2009.
[34] Ilja Kuzborskij, Claire Vernade, Andras Gyorgy, and Csaba Szepesvári. Confident off-policy evaluation and selection through self-normalized importance weighting. In International Conference on Artificial Intelligence and Statistics, pages 640-648. PMLR, 2021.
[35] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721 27300. PMLR, 2017.
[36] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pincau, and Aviv Tamar. Bayesian reinforce- ment learning: A survey. arXiv preprint 160iv:1609.0444 2016.
[37] Mengjiao Yang. Bo Dai, Ofir Nachum. George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint arXiv:2012.069199 2020.
[38] Ian Osband, Benjamin Van Roy, Daniel Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 106, 2019.
[39] Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off- policy evaluation. arXiv preprint arXiv:2007.13609 20200
[40] Ping Li, Jinde Cao, and Zidong Wang. Robust impulsive synchronization of coupled delayed neural networks with uncertainties. Physica A: Statistical Mechanics and its Applications, 373:261-272 2007.
[41] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[42] Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1711.01290 2017.
[43] Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint 16120:161,.01474, 2016.
12

---

[44] Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4923-4932, 2019.
[45] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:/806.0755/7 2018 8.
[46] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0154000 2016.
[47] Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with 2021.
learned model. arXiv preprint arXiv:2
[48] Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 2021.
[49] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.1360 2020.
[50] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026 4034, 2016.
[51] Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint wxXiv:2001.046900 2020.
[52] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32:8572-8583, 2019.
[53] Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A Alemi, Jascha Sohl-Dickstein. and Samuel S Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. 2019.
arXiv preprini arXiv:
[54] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[55] Raphael Fonteneau, Susan Murphy, Louis Wehenkel, and Damien Ernst. Batch mode reinforcement learning based on the synthesis of artificial trajectories. Annals of operations research, 208(1):383-414 2013.
(56) Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:200 14522, 2020.
[57] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. arXiv preprint arXiv:1909.09314. 2019.
[58] Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Richard Zemel. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. Advances Neural Information Processing Systems, 32, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] The abstract and introduction summarize the organization and contents of the paper.
13

---

(b) Did you describe the limitations of your work? [Yes] The most important limitation of using ensembles is their N-fold increase in computational cost. Section [5.3] frames this problem and investigates a variety of approaches used in the supervised Tearning literature. We demonstrate that while "Efficient Ensembles" perform well on the simpler D4RL Gym domains, they fail on the D4RL antmaze domains. We believe this section very clearly motivates future work in developing efficient uncertainty estimation approaches that are better suited to the domain of reinforcement learning.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work is focused on algorithmic advances in the field of offline reinforcement learning (RL). The potential positive and negative impacts of offline RL methods are directly function of the domain to which they are applied. In this work, we only apply our methods to open-soureed, simulated benchmark domains which do not have societal implications. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section sets up the stage for the introduction of Theorem[3.]]
3.1
(b) Did you include complete proofs of all theoretical results? [Yes] Please refer to Appendix F
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Please refer to the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The offline datasets provided by the benchmarks we used. We have detailed all experimental details, with pointers to details in the respective sections.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In Table we report mean and standard deviation across random seeds. For the summarized bar plots in Figures and 4 This is because for each bar in this summary plot, we report the results for the best hyperparameter value (which for many settings, due to significant computational cost, had 2 random seeds). However, this work we have shown exact detail the dense hyperparameter searches we have done, and furthermore, in we have reported the result for every single experiment individually (i.e. every combination of data setting, method, hyperparameter, and random seed) (Figure [ Appendix A Appendix [B). Reporting results for each individual experiment demonstrate the trends, even in the settings where each hyperparameter was ran with 2 seed. We have also included all of our numerical results as pk1 files in our codebase.
we only plot the mean.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No] We mentioned that experiments were done on Nvidia P100 GPU devices. We did not include the total amount of compute we used.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite the D4RL 24 and RL Unplugged 25 benchmarks.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include out codebase, python notebooks, as well as .k11 files of experimental results.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] D4RL (44 and RL Unplugged 25 established and open-sourced offline RL datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The D4RL. [24 and RI. Unplugged 25 do not contain such sensitive materials.
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "oYIO1OIzbb", "reviewer": "Reviewer_JRTT", "summary": "This paper discusses the uncertainty estimation in RL, which is an alternative to induce pessimism in offline RL. Though uncertainty estimation through ensembles has been proposed in the offline RL literature, this paper points out a critical flaw in how to incorporate the LCB in the actor-critic based algorithm. It shows theoretically that the previous algorithms, which regress different Q functions to the shared  pessimistic target values, and does policy evaluation based on the LCB could sometime leads to over-estimation of the Q function. To address this, the paper proposes a simple fix, that is in the Bellman backup stage, instead of regressing the different Q values to the shared LCB estimate, just regressing them to independent Q target. Empirically, it shows better performance in challenging tasks that require stitching. Beyond this, the paper also examines how different efficient ensemble method work in RL setting, and it seems there still exist a large gap in the performance compared with deep ensemble, which opens up more interesting questions in efficient ensemble methods in the RL setting. ", "questions": "[1]. The comparison of independent and shared target comparison is interesting. Could the authors summarize the literature in a clearer way in how they do ensembles, and mark the subtle difference, as this is missing in Section 2, but these subtle differences seem make a big difference, as what is discussed in this paper.\n[2]. From L154-155, it seems the shared targets are doing this \"doubling pessimism\" in both the policy evaluation and learning step, which seems actually a little weird to me, could the authors point out the exact literature that is doing this. As I can imagine either (1). we are doing the independent target, or (2). we do the evaluation pessimistically, but in learning we take the mean of the Q, compared to LCB. Just curious that do the authors try (2)?\n[3]. Followed by [2], it seems the shared target do seem doing more pessimism, but theoretically it could lead to over-estimation. Any comments in bringing this intuition to the theoretical results? it seems related with the correlation of the transition (s,a) paris and properties of the C matrix. Could the authors comment more on when the shared target leads to over-pessimism or optimism, and the practical guidelines?\n[4]. The toy example in Section 3.2 is a little bit confusing to me, what is the rationale of using the policy that play the same a in the next state? \n[5].The discussion in L327-337 shows that the shared target has worse performance as ensemble sizes decreases, any explanation on this? ", "limitations": "Yes. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n[1]. The paper is well-written and very easy to follow.\n[2]. It discusses a major flaw in how to incorporate LCB estimate in offline RL algorithms, which seems being overlooked in the literature, but empirically seems make a big difference. The theoretical claim is well-supported. \n[3]. It has a comprehensive set of empirical studies, which covers various aspects about the applicability of the method, such as the ensemble size, the hyper-parameter sensitivity. I do appreciate the authors' effort in discussing how to transfer efficient ensemble method in supervised learning setting to the RL setup, to make it more computationally efficient, though some negative results there.\n\nWeakness:\n[1]. Maybe I am misunderstanding something, but i do feel some of the claims and findings are not explained in a super clear way, see Questions section for details for this.\n[2]. The experiment section does show that incorporating the independent target leads to the better performance in the challenging tasks, it would be great to see that this is result from better LCB estimate. We see the overestimation issues in the toy task, and it would be really helpful to see in the challenging tasks, that shared target does lead to over-estimation, which is the reason that the method helps. \n[3]. Section 4.2 seems a little bit out of picture. As it seems alpha=0 works great in most cases, the authors state that it might help in some narrow data regime, is any empirical study supporting this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "hH7DWQUSSTQ", "reviewer": "Reviewer_nkxP", "summary": "The paper observes a problem in existing pessimism estimation in offline RL using ensembles: using shared targets for all ensembles updates. The paper instead proposes to update each ensemble individually and apply the pessimism at policy updates. The paper derives the update form of both methods in the NTK setting and shows that the update method with shared target could even result in optimism, which is also shown with some synthetic simulation data. Finally the paper evaluates the proposed method in several offline RL benchmarks and show its empirical competitiveness. ", "questions": "## Major Questions\n\n1. This question is less about the proposed algorithm itself: for the shared target update methods, the target itself contains a pessimism term: it's already a LCB estimation. Why, for policy update, another LCB term is added into the $Q_{LCB}$ estimation, why not just directly using $\\mathbb{E}[Q_{\\theta^i}]$? It seems like it's doing pessimism twice? Although it looks like removing the LCB term in $Q_{LCB}$ won't affect the theoretical result by much (you just subtract the backup term by 1? which of course leads to optimism more easily).\n\n2. In addition to the existing baselines, it may seem like the following baselines could also be of interest? For example, a) we take the min over the ensemble with individual targets? b) every ensemble is bootstrapped from their own target but we also subtract with a common lcb term? \n\n### some minor problems\n\n- The formula in the algorithm box in section 3.1 overflows. \n\n- Why the $\\gamma$ in section 3.2 is chosen to be very small (instead of 0.9)?\n\n- The y-axis are inconsistent in Fig.2 and Fig.3. Also in Fig.3, what does each dot mean? It seems a little bit hard to interpret the plot.\n\n- Why is line 763 true, for $\\theta_{lin}^{(0)} = 0$?", "limitations": "1. The overall algorithm has good intuition and motivation, but the introduction to the additional term in section 4.2 looks irrelevant to the rest of the paper. From the experiments, this term seems crucial to a good performance of the algorithm and thus unavoidable in the current version. Although it makes sense that some kind of regularization may be needed for unseen action, this additional term indeed undermines the overall message a little bit. \n\n2. The ablation of using a more condensed surrogate for ensemble is a good experiment, and as the paper already suggests, it would be better if a more efficient way of pessimism could be derived, which seems beyond the scope of this paper. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. Overall the paper is well written, easy to follow, and the technical part seems correct. \n\n2. The paper makes a good observation of the existing methods for offline RL when they update the Q-values for ensembles: from hindsight one really does not need to incorporate the pessimism into the function update procedure, but instead just apply pessimism during policy update. This also seems to agree to the theory RL algorithms: one can just perform the regular bellman updates (or perform elimination in version space algorithms) and define policy with LCB or take minimum over the remaining set of functions (for pessimism) (for example, [1]). \n\n3. Although it may not be obvious under what kind of conditions such that in the NTK setting, using the shared target could result in optimism, the following subsection provides good evidence that that indeed could happen. It could be better to provide some more intuitive scenario or even a closed-form construction. \n\n4. The paper provides extensive and convincing experiments, including a) good ablation experiments which contains the different kinds of shared target updating methods (such as shared-LCB Ens., Shared-Min Deep Ens, and with a different number of ensembles)\n. b) The paper tries many different hyperparameters for the baselines, so the baselines seem to be fine-tuned for the final presentation of the results. c) The experiments are performed on extensive benchmarks. \n\n## Weakness\n\n1. The theoretical results provide very good intuition into the problems of the previous pessimism estimation in offline deep RL methods, but since the result is based on the NTK setting, it still has some gap between the practical situations. \n\n2. The result presented in table 1 has different hyperparameter for different tasks, which likely undermines the empirical merits of the proposed algorithm. \n\n### references\n\n[1] Xie, Tengyang, et al. \"Bellman-consistent pessimism for offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 6683-6694.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IGHaWAKM0An", "reviewer": "Reviewer_nPdV", "summary": "This paper studies ensemble-based pessimism in offline RL from both theoretical and empirical aspects,\n- Giving an analysis mathematically through NTK to show shared pessimistic targets can paradoxically lead to Q-estimates which are in fact optimistic.\n- Proposing MSG that trains each Q-network independently, and conducts experiments in D4RL and RL Unplugged tasks.\n\n", "questions": "- In line 191, why the pessimism term of method 2 may become positive, while it cannot be positive in method 1?\n- ln line 781, how to get Eq. 20, 21, 22 from Eq. 19?\n- ln line 791, how to get Eq. 31, 32 from Eq. 30?\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Formally analyze the offline RL methods based on Q-ensemble pessimism on infinite-width neural networks, and shows the pessimism term may become positive in shared targets. \n- Empirically verifies the effectiveness of the algorithm by combining Q-ensemble with CQL. \n\nWeaknesses\n- The NTK assumptions in section 3.1 and the Gaussian assumptions in section 3.2 seems limited in broader value iteration.\n- The Q-ensemble needs to combine with CQL to obtain reasonable performance. The use of CQL makes it difficult to analyze the source of performance improvement. However, there exist several methods (e.g., EDAC in NeurIPS 2021 and PBRL in ICLR 2022) that perform Q-ensemble for offline-RL with purely uncertainty. \n- The experiment results are not complete for D4RL benchmark.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "AMNsJ3ddHe7", "reviewer": "Reviewer_qbbN", "summary": "The paper revisits the design of ensemble critics in offline RL. The authors argue that the common design where critics in the ensemble sharing the same, pessimistic target function in learning can lead to actually optimistic critics. The authors analyze this phenomenon theoretically under the NTK assumption, and present toy simulation examples. The authors further use this insight to design an offline RL algorithm MSG, which gives SoTA results on common offline RL benchmarks. In these experiments, they show that the separating the targets is a key to the algorithm's superior performance.\n\n\n\n********** Comments for Rebuttal *************\n\n\nThanks for the rebuttal and the clarification. While they address some of my concerns, my main concern stay the same. As stated in the original review, the main issue I have is \"whether the proposal here is sufficient to design a full offline RL algorithm or just provide an important note on implementation choice\". The rebuttal also points out \"it is the objective of our work to advocate for relying on uncertainty estimation as the main source of pessimism for offline RL\".\n\nLet's examine this question from two aspects based on the paper and rebuttal . \n\nFrom the theoretical side, the paper provides that Theorem 3.1, which compares the iterates of $Q_{LCB}$ of Independent Targets and the Shared Targets. However, it does not show \"how\"  good pessimistic the estimate of Independent Targets is. In the review, I mentioned \"in general optimizing a pessimistic critic or being more pessimistic does not imply good performance in offline RL.\" because whether a pessimistic critic is useful or not depends on how \"tight\" the under estimation is and where it is pessimistic. Being more pessimistic is not always good, e.g., estimating all values as $V_{min}$ is pessimistic but it's obviously useless. The current theory does not provide enough insights to how good the pessimistic estimate is, or how good the learned policy based on such value estimate will be. This is why I said \"the significance of the theoretical results are rather limited\" in the review. \n\nFor empirical side, I think to demonstrate the authors claim, it is necessary to show that SoTA can be achieved with $\\alpha=0$. However, the current results do not support that fully. While I agree that in Figure 3, $\\alpha=0$ is among the best performing results. I also think that Figure 3 does not provide a conclusive answer, as it is missing results of larger $\\alpha$ value for $\\beta=0$, as there is an increasing trend. This was pointed out in the review. It's also hard to compare Fig 3 and Fig 2 directly\n\nI think the failure of using $\\alpha=0$ in simpler mujoco domains is actually showing that the proposed approach \"alone\" might not be sufficient to provide enough pessimism \"broadly\". I do not agree with the rebuttal's statement that these environments can be solved by behavior cloning. All the methods the authors listed there are not pure behavior cloning, which mimics all actions, as all of them perform some reasoning what actions are better based on the rewards. Therefore I don't think this is a good excuse of using the CQL term here. Yes, I agree that the proposed method works with $\\alpha=0$ with Antmaze which is considered as a harder domain, but is it because of the reason that the authors mentioned? if this is the case, why does not perform well in simpler domain. Or is it because of something that is related to the structure of Antmaze environment and dataset? Currently, we don't have sufficient evidence to tell.\n\nThus, I think that currently the paper provides insufficient results to show that the proposed uncertainty estimation \"alone\" can achieve SoTA offline RL results. Nonetheless, I also think that this paper provides more than sufficient reasons to show that it would be a good design choice to improve an existing value-based offline RL algorithm. Therefore, I keep my original recommendation.\n\n", "questions": "My main concern is whether the independent ensemble idea here is indeed sufficient to induce adequate pessimism for offline RL as the author claim. Here're few reasons why it does not seem to be the case. \n\n1.  Theorem 3.1 compare the results of a pessimistic policy evaluation procedure. The authors argue that using Shared Targets may lead to optimism, compared with the Independent Targets version. However, significance of such a comparison is quite limited for the following reasons. First, to my knowledge, I do not know of existing offline RL methods introducing pessimism purely based on what the authors describe in the Shared Targets update rule. While that update rule are used in some implementations, that is usually not the main source of pessimism that the offline RL algorithm uses but just a detail. So the theoretical comparison here may not be realistic, even if we acknowledge the NTK approximation part. Second, the authors study a version based on E[Q] - sqrt{V[Q]}. A more common implementation I know is pointwise pessimism, i.e. min_i Q_i(s,a), which e.g. is used in CQL and TD3+BC implementation based on double Q networks. I doubt the comparison would carry through to this more realistic case, because using pointwise pessimism would be strictly more pessimistic than the Independent Targets.\n\n2. The above theoretical analysis is limited to offline policy evaluation. It's unclear how it can be translated into policy optimization, as in general optimizing a pessimistic critic or being more pessimistic does not imply good performance in offline RL. Therefore, the significance of the theoretical results are rather limited in my mind, unless the authors discuss in details how the proposed procedure affect the performance of the learned policy.\n\n3. Another reason why the pessimism induced by Independent Training is not sufficient is that in MSG, the authors introduce the difference term H(\\thteta^i) as part of the objective. The authors write \"Empirically, as evidenced by our results in Appendix A.2, we have observed that such a regularizer can be necessary\" This seems to imply that  the pessimism of the Independent Target can be serve as the main source of pessimism. In particular, the authors suggests using this extra regularizer H(\\theta^i) is necessary, when the data distribution is narrow. Since addressing the lack of coverage is the main focus of offline RL (say compared with off-policy RL), this shows again the proposed pessimism is not sufficient for offline RL. \n\n4. The above insufficiency is partially shown in the experimental results as well. For instance in Figure 3, MSG does not perform the best when alpha=0. In the experiments of beta=0, the authors should also try even larger alpha to get a fuller picture of the performance and how hyperparamters interact. From these results, it is unclear whether the good performance of MSG is due to using the ensemble uncertainty as the primary source of uncertainty.\n\nIn summary, while this paper presents an interesting observation, showing that using Independent Targets can be more pessimistic than using Shared Targets, such results do not directly address the main research question of offline RL for the reasons above. (But the title \"Why So Pessimistic?\" seems to want to convey that we should be less pessimistic?) It is therefore hard for me to position this paper. This paper presents a good and thorough empirical study on how a certain design knob in implementation can affect the performance. However, the results here do not provide enough insights to design offline RL methods more broadly outside the tested implementation here.\n\n\n", "limitations": "The authors well discuss the limitation such as the extra complexity needed by the proposed method. However, in my view, the results here are more limited than what the authors claim for the reasons above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper presents an overlooked finding. \n2. It provides some theoretical reasons to back it up. It also provides empirical validation. \n2. The writing of this paper is clear and the experimental results are thorough. \n\nWeakness: \n1. While the authors provide explanations of why the common usage of Shared Targets may lead to optimism, the current results are not conclusive. In particular, while existing offline RL implementations use Shared Targets, the pessimism of Shared Targets is not the main source of pessimism but rather an implementation detail. Therefore, it is unclear whether the proposal here is sufficient to design a full offline RL algorithm or just provide an important note on implementation choice. This factor limits the significance of the paper. \n\n2. It is unclear what data assumptions are needed for the proposed method to work properly as an offline RL method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept