Uncertainty Weighted Offline Reinforcement Learning
Authors: Yue Wu
Shuangfei Zhai
Nitish Srivastava
Joshua M. Susskind
Jian Zhang
Ruslan Salakhutdinov
Hanlin Goh
Venue:  (2021)
Keywords: reinforcement learning
offline
batch reinforcement learning
off-policy
uncertainty estimation
dropout
actor-critic
bootstrap error

Abstract: Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that models the epistemic uncertainty to detect OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.

Under review as a conference paper at ICLR 2021
UNCERTAINTY WEIGHTED OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline Reinforcement Learning promises learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from distiiiiittiittononn (OOD) actions or states. We hypothesize that key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that models the epistemic uncertainty to detect OOD state- action pairs and down-weights their contribution in the training objectives accord- ingly. Implementation-wise certainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-arr baseline on datasets with sparse demonstrations col- lected from human experts.
we adopt practical and effective dropout-base un-
INTRODUCTION
Deep reinforcement learning (RL) has seen surge of interest over the recent years. It has achieved remarkable success in simulated tasks (Silver et al. 2017] Schulman et al., 2017 Haarnoja et al.) 2018), where the cost of data collection is low. However, one of the drawbacks of RL is its difficulty of learning from prior experiences. Therefore, the application of RI to unstructured real-world tasks still its primal stages, due to the high cost of active data collection. It is thus crucial to make full use of previously collected datasets whenever large scale online RI. is infeasible.
Offline batch RL algorithms offer a promising direction to leveraging prior experience (Lange al. 2012). However, most prior off-policy RL algorithms (Haarnoja et al.) 2018: Munos et al. 2016 Kalashnikov et al.- 2018 Espeholt et al.] 2018, |Png et al.] 2019) fail on offline datasets, even on expert demonstrations (Fu et al.) 2020). The sensitivity to the training data distribution is well known issue in practical offline RL algorithms (Fujimoto et al.) 2019 Kumar et 2019 2020 Peng al. 2019 Yu al.. 2020). A large portion of this problem is attributed to actions states not being covered within the training set distribution. Since the value estimate on out-of aat-of-distribution (OOD) actions or states can be arbitrary, OOD value or reward estimates can incur destructive estimation errors that propagates through the Bellman loss and destabilizes training. Prior attempts try to avoid OOD actions states by imposing strong constraints or penalties that force the actor distribution to stay within the training data (Kumar et al. 2019: 2020 Fujimoto et al., 2019 Laroche et al. 2019). While such approaches achieve some degree of experimental success, they suffer from the loss of generalization ability of the Q function. For example, state-action pair that does not appear in the training set can still lie within the training set distribution, but policies trained with strong penalties will avoid the unseen states regardless of whether the Q function can produce accurate estimate of the state-action value. Therefore, strong penalty based solutions often promote a pessimistic and sub-optimal policy. In the extreme case, e.g., in certain benchmarking environments with human demonstrations, the best performing offline algorithms only achieve the same performance as a random agent (Fu et al. 2020), which demonstrates the need of robust offline RL algorithms.
In this paper, we hypothesize that key aspect of a robust offline RL algorithm is proper estimation and usage of uncertainty. On the one hand, one should be able reliably assign uncertainty score

---

Under review as a conference paper at ICLR 2021
to any state-action pair; the other hand, there should be a mechanism that utilizes the estimated uncertainty to prevent the model from learning from data points that induce high uncertainty scores. The first problem relates closely to OOD sample detection, which has been extensively studied in the Bayesian deep learning community, (Gal &
Average Return
2.0 1e12
o Target
-ecate-expert-vv
4000
Ghahramani 2016a| Gal 2016: Osawa et al., (019)) often measured by the
1.5
relocate-expent
3000
epistemic uncertainty of the model. We
ours BEAR
1.0
adopt the dropout based approach Gal
Ghahramani (2016a), due to its sim- plicity and empirical success in prac- tice. For the second problem, we pro-
1000
0.0
vide an intuitive modification to the Bellman updates in actor-critic based algorithms. Our proposed Uncertainty Weighted Actor Critic (UWAC) is to simply down weigh the contribution of target state and action pairs with high uncertainty. By doing so, we pre- vent the Q function from learning from overly optimistic targets that lie far away from training data distribution (high uncertainty).
200 400 600
0
200
400 600 Training Epochs
Training Epochs
Figure 1: Left. Plot of average return training epochs of our proposed method (red) v.s. baseline (brown) (Kumar et al. 2019) on the relocate-expert offline dataset. Right. Correspond- ing plot of Q-Target values v.s. training epochs. Our proposed method achieves much higher average return, with better train- ing stability, and more controlled Q-values.
Empirically, we first verified the effectiveness of dropout uncertainty estimation at detecting OOD samples. We show that the uncertainty estimation makes intuitive sense in a simple environment. With the uncertainty based down weighting scheme, our method significantly improves the training stability over our chosen baseline (Kumar et al. 2019), and achieves state-of-the-art performance in variety of standard benchmarking tasks for offline RL.
Overall, our contribution can be summarized as follows: 1) We propose a simple and efficient tech- nique (UWAC) to counter the effect of OOD samples with no additional loss terms or models. 2) We experimentally demonstrate the effectiveness of dropout uncertainty estimation for RL. 3) UWAC offers novel way for stabilizing offline RL. 4) UWAC achieves SOTA performance common offline RL benchmarks, and obtains significant performance gain on narrow human demonstrations.
RELATED WORK
In this work, we consider offline batch reinforcement learning (RL) under static datasets. Offline RL. algorithms are especially prone to errors from inadequate coverage of the training set distribution, distributional shifts during actor critic training, and the variance induced by deep neural networks. Such error have been extensively studied as "error propagation" in approximate dynamic program- ming (ADP) (Bertsekas & Tsitsiklis 1996. Farahmand et al. 2010 Munos 2003 Scherrer et al.) 2013). Scherrer et al. (2015) obtains a bound on the point-wise Bellman error of approximate mod- ified policy iteration (AMPI) with respect to the supremum of the error in function approximation for an arbitrary iteration. We adopt the theoretical tools from (Kumar et al. 2019) and study the accumulation and propagation of Bellman errors under the offline setting.
One of the most significant problems associated with off-policy and offline RL is the bootstrapping error (Kumar et al. 2019): When training encounters action or state unseen within the training set, the critic value estimate on out-of-distribution (OOD) samples can be arbitrary and incur an error that destabilizes convergence on all other states (Kumar al., 2019 Fujimoto et al. 2019) through the Bellman backup. Yu et al. (2020) trains a model of the environment that captures the epistemic uncertainty. The uncertainty estimate used to penalize reward estimation for uncertain states and actions, promoting a pessimistic policy against OOD actions and states. The main drawback of such a model based approach is the unnecessary introduction of a model of the environment - is often very hard to train good model. On the other hand, model-free approaches either train an agent pessimistic to OOD states and actions (Wu et al.) 2019 Kumar et al.) 2020) or constrain the actor distribution to the training set action distribution (Fujimoto et al. 2019: Kumar et al., 2019 WW et al., 2019; Daques et al., 2019 Fox et al.] 2015) Laroche et al. 2019). However, the pessimistic assumption that all unseen states actions bad may lead sub-optimal agent and
2

---

Under review as a conference paper at ICLR 2021
greatly reduce generalization to online fine-tuning (Nair et al. 2020). Distributional constraints, in addition, rely on approximations since the actor distribution is often implicit. Such approximations cause practical training instability that will study in detail in section 5.4
We propose a model-free actor-critic method that down-weighs the Bellman loss term by inverse uncertainty of the critic target. Uncertainty estimation has been implemented in model-free RL for safety and risk estimation (Clements et al., 2019 Hoel et al.) (020) or exploration (Gal & Ghahra- mani 2016a, Lines & Van Der Wilk), through ensembling (Hoel et al., 2020) or distributional RL (Dabney et al. 2018; Clements et al. 2019). However, distributional RL works best on discrete action spaces (Dabney et al.) 2018) and require additional distributional assumptions when extended to continuous action spaces (Clements et al., (201)). Our approach estimates uncertainty through Monte Carlo dropout (MC-dropout) (Srivastava et al. 2014). MC-dropout uncertainty estimation simple method with minimal overhead and has been thoroughly studied in many traditional super- vised learning tasks in deep learning (Gal & Ghahramani 2016a Hron et al.) 2018 Kingma et al.)
2015
Gal & Ghahramani] 2016b). Moreover, we observe experimentally that MC-dropout uncer- lainly estimation behaves similarly to explicit ensemble models where the prediction is the mean of the ensembles, while being much simpler (Lakshminarayanan et al. 2017 Srivastava et al. 2014).
The most relevant to our work are MOPO (Yu et al. 2020). BEAR (Kumar et al. 2019), and CQL (Kumar et al. 2020). MOPO approaches offline RL from different model-based paradigm, and ob- tains competitive results on some tasks with wide data distribution. However, due to the model-based nature, MOPO achieves limited performance on most other benchmarks due the performance of the model being limited by the data distribution. On the other hand, BEAR and CQL both use actor- critic and do not suffer from the above problem. We use BEAR (discussed in section 3.2) our baseline algorithm and achieve significant performance gain through dropout uncertainty weighted backups. CQL avoids OOD states/actions through direct Q value penalty on actions that leads to OOD unseen states within the training set. However the penalty proposed by CQL 1) risks hurting Q estimates for (action, state) pairs that are not OOD, since samples not seen within the dataset can still lie within the true dataset distribution; 2) limits the policy to be pessimistic, which may be hard to tune once on on-policy data becomes available. Additionally our method is not limited to BEAR and can apply to other actor-critic methods like CQL. We leave such exploration to future works.
PRELIMINARIES
3.1 NOTATIONS
Following Kumar et al. 2019). we represent the environment as a Markov decision process (MDP) comprising of a 6-tuple (S, A, P. R. po.7), where S is the state space. A is the action space, P(s'ls, a) is the transition probability distribution, Po is the initial state distribution, R S x A + IR the reward function, and y (0, 1) is the discount factor. Our goal is to find policy π (s)la) from the set of policy functions IT to maximize the expected cumulative discounted reward.
Standard Q-learning learns an optimal state-action value function (*(s, a), representing the ex- pected cumulative discounted reward starting from with action and then acting optimally there- after. Q-learning is trained on the Bellman equation defined as follows with the Bellman optimal operator T defined by:
TQ(s,a) R(s, a) + (Ep(s)),,)) max Q(s', aa))
In practice, the critic (Q function) is updated through dynamic programming, by projecting the target Q estimate (TQ) into Q (i.e. minimizing Bellman Squared Error E [[Q TQ) Since max,' Q(s', ) in generally intractable in continuous action spaces, actor (Te) function is learned to maximize the critic function (TT()) ;= arg max, Q(s, a)) (Haarnoja et al. 2018 Fujimoto et al,
2018 Sutton & Barto 2018).
In the context of offline reinforcement learning, naively performing max Q(s', aa) in equation may result in a' unseen within the training dataset (OOD). and resulting in Q estimate with very large error that can propagate through the Bellman bootstrapping and destabilize training on other states (Kumar al. 2019.
3

---

Under review as a conference paper at ICLR 2021
3.2 BASELINE ALGORITHM
We use BEAR (Kumar et al. 2019) as our baseline algorithm. BEAR restricts the set of policy functions (IT9) to output actions that lies in the support of the training distribution:
π (-|ss) == arg max Ea- (-1s) [Q(s,
(2)
e'elle
Since the true support of II is intractable. Kumar et al. 2019) instead relies an approximate support constraint through optimizing sampled maximum mean discrepancy (MMD) (Gretton et al. 2012) between the training action distribution and the policy distribution.
However, this constraint eliminates the possibility the Q function to learn to generalize to state- action pairs beyond the training dataset and therefore limits the agent's performance and general- ization. Moreover, the justification behind the sampled MMD approximation as support constraints largely based on empirical evidence, and we observe numeric instability caused by discrepan- cies between Q estimates and average returns on some narrower offline datasets (Figure II Such observations also correspond to Kumar al. (2019) description in section
4 UNCERTAINTY WEIGHTED OFFLINE RL
Our approach (UWAC) is motivated by connecting offline RL with the well-cstablished Bayesian uncertainty estimation methods. This connection enables UWAC to "identify" and "ignore" OOD training samples, with additional models or constraints.
4.1 UNCERTAINTY ESTIMATION THROUGHI DROPOUT
Let X capture all the state-action pairs in the training set: X = (s, and Y capture the true Q value of the states. We follow Bayesian formulation for the Q function in RL parameterized by 0, and maximize p(0) K,Y. Y) p(Y X, ())(()/p(Y|X as our objective. Since p(|[X) is gener- ally intractable, & Ghahramani 2016a). , by training with dropout before every weight layer, and also performing dropout test time (referred to as Monte Carlo dropout). The epistemic uncertainty is captured by the approximate predictive variance with respect to the estimated Q for T stochastic forward passes
approximate the inference process through dropout variational inference (Gal
1 T ()(ss,))
a))) = 0
with o² corresponding the inherent noise in the data, the second term corresponding to how much the model is uncertain about its predictions, and E[Q(s, (s, a)] the predictive mean. We therefore the second-third term to capture model uncertainty for OOD sample detection.
Overall, instead of training Q function on the policy æ, define distribution n' with respect to the original policy distribution π (-Iss) and normalization factor Z(x)
eicertainty-weigghee policy
2(s) = / Var [Q(s, a)] (alssdda
B
n' (als) = Var [Q ((als)
(3)
We show in the appendix A.I that optimizing n' results in theoretically better convergence properties against OOD training samples.
4.2 UNCERTAINTY WEIGHTED ACTOR-CRITIC
Instead of training the Q function on Equation we train Qe on N'. For clarity, we denote the TD Q-target as in (Mnih et al. 2013) Kumar al. 2019 by Qo.
((Qo) = =B(o' a
(Qo(s,a) Ba (-1o")
(R(s, a) + (Qr (s',
Var (Qr(s', (Qo(s, (R(s, (( (4)
=R(olle.a) (T)(e)))
4

---

Under review as a conference paper at ICLR 2021
We absorb the normalization factor Z into B. The resulting training loss down-weighs the Bell- man loss for the Q function by inverse the uncertainty of the Q-target (Qar (s', .d)) that does track gradient. This directly reduces the effect that OOD backups has on the overall training process.
Similarly, we optimize the actor π using samples from x'. Substituting <-11s) by (+Is) in equation 2. arrive the following actor loss
[(T) (E) [Qo(s,a)] (110)(1))) Var (Qo(s, a)) @o(s,a)
(5)
The resulting actor loss intuitively decreases the probability of maximizing the Q function on OOD samples, further discouraging the vicious cycle of Q function explosion. Such loss further stabilizes Q function estimations without constraints on the actor function distribution.
Algorithm summarizes the proposed training curriculum, mostly the same as in the baseline (Ku- mar et al. 2019). Note that we do not propagate gradient through the uncertainty (V (Var(y(s,a)))
Algorithm Pseudo code for UWAC, differences from (uumar et al. 2019) are colored
Input: Dataset D, target network update rate т, mini-batch size N, sampled actions for MMD
(n = 10), sample numbers stochastic forward passes (T' = 100), hyper-parameters: X, a, B
1: Initialize Q networks {Qo, Qo,} with MC Dropout. Initialize actor Td, target networks
(Qu, Qx, } and a target actor Tar with o' Φ, f/,2 01.2
2: for + I to N do
3: Sample mini-batch of transitions (8, r, 8') ~D
4: Q-update:
5: Sample p action samples, {a, ~ To (-1s))}
6: y(s, a) max.; [Qein(Qe; (6', a;), Q03 (5' a;)) + (1- -) max (5', az), Qog (s', a:))
7: Calculate variance of the y(s, a) through variance T stochastic samples from QØ: Qog
8:
Perform one step of SGD to minimize L(Qo,,)) Varlv(e.a]] (Qo, 2 (s, a) (r + rv
9:
Policy-update:
10:
Sample actions fa; ~ Tep (-Is²) } and far ~ =D
11:
Update o, according to equation with MMD penalty with weight a as in section 3.2
12: Update Target Networks: 8122 < r0,,2 S + T
13: end for
5 EXPERIMENTAL RESULTS
Our experiments structured as follows: In section we validate and visualize the effectiveness of dropout uncertainty estimation in RL. In section we present competitive benchmarking results on the widely-used D4RL MuJoCo walkers dataset. We then experiment with the more complex Adroit hand manipulation environment effectiveness against OOD samples by examining the Q target functions in section 5.4 We report the implementation detail and
section 5.3 and analyze the training stability and the
and ablation studies in appendix A.2
5.1 DROPOUT UNCERTAINTY ESTIMATION FOR REINFORCEMENT LEARNING
For the ease of 2D-visualization, we investigate our dropout uncertainty framework on the OpenAI gym LunarLander-v2 environment. The LunarLander-v2 environment features lunar lander agent trying to land a goal location in 2D world (between two yellow flags) with 4 actions (do nothing, fire left engine, fire downward engine, fire right engine).
We generate the expert offline dataset from the final replay buffer (size 100.000) of a fully trained expert AWR (Peng et al. 2019) agent with average reward 270. Note that the state-action distribution has a relatively complete coverage over the observation space (Fig. 2
To simulate the scenario in most offline datasets, where the agent encounters lots of out-of- distribution states and actions, we create two skewed datasets by removing all observations from the
We will release our code at ggituuh.com/anonymous
5

---

Under review as a conference paper at ICLR 2021
action do nuthing
actiion 1: fite left engin
actiion fiee main engine
action 3: firw right engine
15
beitees digisdemet
neraonaa Stobeement
63
arrtan thokasmert
lesurtal
Figure 2: Expert Trajectory Visualization.
maps of expert's action distribution with respect to horizontal/vertical displacement from the goal location. Warmer locations represent more observations.
action 0: 50 nothing
action 1. fire loft engine
action D: do nothing
action fire left enginc
Biglassmenn
nencenua Sxclkmment
Intuental Caplecmmett
Iatuontal diplucmentt
main engine
action dite engine
main engine
right engine
emet
inplocttentt
Figure 3: Left. The training set with horizontal displacements (< 0.1) removed. This makes all states on the left OOD. Right. Our model estimates higher uncertainty (brighter color) the left and lower uncertainty (colder color) the right.
upper-half or the leftmost-half according displacement from objective. We visualize the clipped datasets distribution together with the estimated Q function uncertainty in Figure 3 Our proposed framework reports higher uncertainty estimates at locations where the observations cially where the observations are removed (OOD states). The results demonstrate the effectiveness of our proposed method at estimating the epistemic uncertainty of the Q function.
sparse, espe-
5.2 PERFORMANCE ON STANDARD BENCHMARKING DATASETS FOR OFFLINE RL
We evaluate our method on the MuJoCo datasets in the D4RL benchmarks (Fu al. 2020), includ- ing three environments (halfcheetah, hopper, and walker2d) and five dataset types (random, medium, medium-replay, medium-expert, expert), yielding total of problem settings. The datasets in this benchmark have been generated as follows: random: roll out a randomly initialized policy for 1M steps. expert: 1M samples from a policy trained to completion with SAC. medium: 1M samples from policy trained to approximately 1/3 the performance of the expert. medium-repla replay buffer of a policy trained up to the performance of the medium agent. medium-expert: 50-50 split of medium and expert data.
Results are shown Table Our method is the strongest by a significant margin on all the medium- expert datasets and most of the medium-expert datasets, and also achieves good performance all of the random and medium datasets, where the datasets lack state/action diversity. Our approach per- forms less well the medium-replay datasets compared to model based method (MOPO) because model-based methods typically perform well on datasets with diverse state/action.
5.3 PERFORMANCE ON ADROIT HAND DATASET WITH HUMAN DEMONSTRATIONS
We then experiment with more complex robotic hand manipulation dataset. The Adroit dataset in the D4RL benchmarks (Rajeswaran et al. 2017) involves controlling 24-DoF simulated hand to perform 4 tasks including hammering a nail, opening a door, twirling a pen, and picking/moving ball. This dataset particularly hard for previous state-of-the-art works in that contains of narrow human demonstrations on high- sith-ddmensionaal robotic manipulation task.
The dataset contains three types datasets for each task. human: a small amount of demonstration data from human; expert: a large amount of expert data from fine-tuned RL policy; cloned: the third dataset is generated by imitating the human data, running the policy, and mixing data at 50- 50 ratio with the demonstrations. It is worth noting that mixing (for cloned) is performed because the cloned policies themselves do not successfully complete the task, making the dataset otherwise difficult learn from (Fu al. 2020).

---

Under review as a conference paper at ICLR 2021
Table 1: Normalized Average Returns on the D4RL MuJoCo Gym dataset according (Fu al, 2020). We report the average over random seeds (+ standard deviation). Fu et al. 2020); Kumar al. (2020) do not report standard deviation. We omit cREM, BRAC-p. aDICE, and SAC-off because they do not obtain performance meaningful for comparison. We bold the highest mean.
Task Name
UWAC (OURS)
MOPO
BEAR
BRACY
AWR
BCQ
BC
CQL
halfcheetah-random
14.5 +
31.9 + 3.9
25.1
31.2
2.5
2.1
35,4
walker2d-random
15.5 + 11.7
6.0
7.3
1.9
1.5
4.9
1.6
hopper-random
22,4 + 12.1
13.0
11.4
12.2
10.2
10.6
9.8
10.8
sellihheeta--mediiu
46.5 2.5
=
41.7
46.3
37.4
40.7
36.1
44.4
walker2d-medium
57.5
2.3
17.4
53.1
6.6
79.2
hopper-medee medium
88,9 1.99:12.2
1.6
52.1
31.1
35.9
54.5
29.0
58
malfchectah-med-rplaaa
46.8
12.6
38.6
47.7
40,3
38.2
38.4
46.2
walker2d-med-repep
lece-reppayy
6.3
32.4
19.2
0.9
15.5
15.0
11.3
26.7
hopper med-replay
3.14
42.7
33.7
0.6
28.4
33.1
11.8
48.6
127.4 3.7
$7.9 - 9.5
53.4
41.9
S2.7
64.7
35.8
walker2d-med-exper
99.7 12.2
51.7 = 34.5
40.1
81.6
53.8
57.5
6.4
98.7
hopper- pepper-mdd-exerr
55.0 + 3.7
96.3
0.8
27.1
110.9
111.9
111
halfcheetah expeet
128.6 2.9
108.2
-1.1
107
104.8
walker2d-ewd expert
121.1 + 22.4
106.1
125.7
153.9
hopper-exper
135.0 + 14.1
110.3
3.7
109
109.9
Figure 4: Our learned policies successfully accomplishes manipulation tasks, such as opening a door as shown.
Results are shown in Table 2 (Kumar et al. 2019) on all the "human" demonstration datasets, where the datasets lacks state/action diversity and the agent will encounter lots of OOD backups during training. We also obtain state-of- the art performance all other datasets in Adroit.
UWAC achieves significant improvement on the baseline (BEAR)
5.4 ANALYSIS OF TRAINING DYNAMICS
Although the baseline method BEAR (Kumar al., 2019) already improves offline RL training stability on most of the MuloCo Walkers dataset, we encounter significantly worse training stability when training it the from a narrow policy (i.c. human demonstrations). We show some selected results in Figure 5
complex Adroit hand dataset, especially demonstrations collected
Note that on of the 6 panels shown, the performance of BEAR drops after obtaining peak very early on into training, and sometimes even falls back to initial performance. We also observe similar behavior in all other environments, we observe strong correlation between the training instability and the explosion of Q values. All per- formance drops begin within 5 epochs when Q target estimate greatly exceeds the average return. We attribute the problem of Q function over-estimation and explosion to performing backups from OOD states and actions: As performance improves initially, the OOD Q estimates increases together with the average Q estimates. Since the agent is unable to explore the OOD actions/states, any over-estimation on the OOD samples can further increase average Q estimates through the Bellman backups, causing vicious cycle leading Q value explosion.
full adroit results in Figure 7 in the Appendix. Additionally,
In the initial stages of training, the performance UWAC increases together with the baseline. By weightttg the OOD backups, UWAC breaks the vicious cycle, and maintains meaningful Q estimates throughout training. This allows UWAC to further train on the offline dataset and surpass BEAR after the performance drop and maintain positive performance.

---

Under review as a conference paper at ICLR 2021
Normalized Average Returns on the D4RL Adroit dataset in the same format Table 1 over random seeds (# standard deviation). We omit BRAC-p, BRAC-v, cREM, and aDICE because they do not obtain performance meaningful for comparison. We bold the highest mean.
Table 2:
Task Name
UWAC (OURS)
BEAR
BC
SAC-off
CQL(H)
CQL(p)
AWR
BCQ
SAC-on
pen-human
3.0
-1.0
34.4
6.3
37.5
55.8
12.3
68.9
21.6
hammer-human
7.9
0.3
1.5
0.5
4.4
2.1
1.2
0.5
0.2
door-human
1.57
3.9
9.9
0.0
-0.2
relocate-umman
0.5 0.6
-0.3
0.0
0.0
0.2
0.4
0.0
-0.1
-0.2
pen-cloned
45.1 5.8
26.5
56.9
23.5
39.2
40.3
28.0
44.0
21.6
hammer-cloned
3.4
0.3
0.8
0.2
2.1
5.7
0.4
0.4
0.2
door-cloned
1.2 1.2
-0.1
-0.1
0.0
0.4
3.5
0.0
0.0
-0.2
relocat-cloneed
0.0 0.0 0.2
-0.3
-0.1
-0.2
-0.1
-0.1
-0.2
-0,3
-0.2
pen-expert
119.8 1: 4.1
105.9
85.1
6.1
111.0
114.9
21.6
hammer-expert
128.8 + 4.8
127.3
125.6
25.2
39.0
107.2
0.2
door-expert
105.4 2.1
103.4
34.9
7.5
102.9
99.0
40.2
relocate-expert
108.7 1.7
101.3
-0.3
91.5
41.6
-0.2
2000
Average Return
1e12
Q Target
Average Return
1e12 Targe Q value
ours
4400
burs
2
1000
500
200
200
Training
1e12 Training Epochs
Training Epechs
1e12 Training
100
DuTs BEAR
burs BEAR
2.0
pen-eloneddoo
1000
1.5
750
1.0
500
0.5
-50
0.0
200
200
400
Tra Training Epochs
Training Epochs
Training Epochs
1e12 Training Epachs
4000
4000
20
durs BEAR
2.0
3000
3000
Woocat--uuatt-nn
10
1.5
2000
2000
1.0
1000
1000
curs
0.0
200
200 400
Traiinn Epochs
Training Epachs
Training Epachs
Training Epochs
Figure 5: Plot average return v.S. training epochs, together with corresponding average Q Target over training epochs. Results are averaged across random seeds. Left: Results of different types (human, cloned, expert) on the Adroit pen task. Right: Results on human demos on the 3 remaining tasks. The performance of baseline (BEAR) degrades over time (also noted in (Kumar et al. 2019)), and the Target value explodes.
CONCLUSION AND FUTURE WORK
In this work, we have leveraged uncertainty estimation to detect and down-weight OOD backups in the Bellman squared loss for offline RL.. We show our proposed technique, UWAC, achieves superior performance and improved training stability, without introducing any additional model or losses. Furthermore, we experimentally demonstrate the effectiveness of dropout uncertainty estimation at detecting OOD samples in offline RL. UWAC also can be applied to stabilize other actor-critic methods, and we leave the investigation future works.
In addition, our work demonstrates a valuable application of Bayesian uncertainty estimation in RL. Future works can combine model-based and model-free methods for offline off-policy RL and use uncertainty estimation to decide when to use the model to train the actor. Additionally, uncertainty estimation may used to guide curiosity based RI. agents for on-policy curiosity- learning-
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
William R Clements, Benoit-Marie Robaglia, Bastien Van Delft, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXhv:1905.09638, 2019.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. learning with quantile regression. In AAAI, 2018.
Distributional reinforcement
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In JCML, 2018
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, pp. 568-576, 2010.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv: 15t.:::5/2..866. 2015.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
S Fujimoto, H van Hoof, D Meger, et al. Addressing function approximation error in actor-critic methods. Proceedings of Machine Learning Research, 80, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062, 2019.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, 1(),, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016a.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pp. 1019-1027, 2016b.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870, 2018.
Carl-Johan Hoel, Tommy Tram, and Jonas Sjöberg. Reinforcement learning with uncertainty esti- mation for tactical decision decision-making intersections. arXiv preprint arXiv:2006.0977 2020.
Jiri Hron, Alex Matthews, and Zoubin Ghahramani. Variational bayesian dropout: pitfalls and fixes. In International Conference on Machine Learning, pp. 2019-2028. 2018.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off- off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.0044 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Scalable deep reinforce- ment learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651- 673, 2018.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in neural information processing systems, pp. 2575-2583, 2015.

---

Under review as a conference paper at ICLR 2021
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, pp. 11794, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pp. 6402-6413 2017.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning, pp. 3652-3661. PMLR, 2019.
David Lines and M Van Der Wilk. Disentangling sources of uncertainty for active exploration.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini arXiv:1312.5602,200 2013.
Rémi Munos. Error bounds for approximate policy iteration. In ICML, volume 3, pp. 560-567, 2003.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054-1062, 2016.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093:: 2020.
Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. In Advances in neural information processing systems, pp. 4287-4299 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine Advantage-weighted regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint 1rx1v:1910.001177, 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10088 2017.
Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game of tetris. J. Much. Learn. Res., 16: 629-1166,6 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
David Silver, Julian Schritwiieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 19 1361, 2019.
10

---

Under review as a conference paper at ICLR 2021
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 ANALYSIS FOR CONVERGENCE PROPERTIES
We firstly state Theorem B.1 in (Kumar et al. 2019) as Theorem A.1 for the sake of completeness.
Theorem A.1. Suppose we run approximate setribbution-contraineen
value iteration with a set set of policies II. Let ö(s, a) be the upper-bound for the Bell-
constrained backup T man approximation error for a given state-action pair (s, a) over k training steps: ô(5, a) sup; (Qx(s, a) - -T a)}. Then,
lim 7 [Vaa()) - S 2 [CMM, (1 (1-
1-2 - ?
max Ex [=( xell
1-000
with the suboptimality constant (a(IT) and the concentrability coefficient defined as:
(TI max - C(II) def (1 7) by bnyh-1o(R
The proof of the theorem is a direct modification of the contraction proof in Theorem of Farah- mand et al. 2010) or Theorem of (Munos) 2003).
The suboptimality constant (a(II)) captures how far n² is from II, namely the suboptimality of the actor. The concentrability coefficient quantifies how far the visitation distribution generated by policies from ll is from the training data distribution, namely the degree to which the training may encounter OOD actions and states. Kumarr et al. (2019) note a trade-off between (III) and C(II), and propose bound both terms by constraining to the set of policies with support the same as the training set policy with MMD loss.
However, the most important Bellman approximation error term which the root cause of the bootstrapping problem is still left unbounded. We proceed to show that for aa(ass
bound the Bellman error term max, E,, [8(s, a)] by any constant C' with arbitrarily high probability by opti- mizing on n².
supt (Q..a) Talss)/ Assuming that Z 2 1, and that is bounded,
Note that Theorem A.2 considers woighweigh by inverse the square-root of the variance (stan- dard deviation), which is different from the inverse variance in Equation 3 and Algorithm I Dewn-neig by the variance has the same practical effect since we clip the ratio for numeri- cal stability. We adopt variance for practical implementation for the ease of tracing after multiple max,min, summatunttioon operations in Algorithm
Theorem A.2. Let <'(ass
sups /Var Qu(xa)) T (s)s)/ with the normalization factor
((s) la Variioaa (als) as in equation Assume that 1) Vs Z(s) 2 1 and 2) Q is bounded (Vs, a: : 1Q(s, a)l S Qmhh
Then for any C E R, there exists B, K such that
P max Eg (8(s, (8 [0)
c) S K
Proof. We firstly apply triangle inequality to unwrap the original formulation into a sum of two differences, and bound the two terms respectively.
max ES ([s max E. x²
sup A
Qx a) TT
x²
max Es x²
sup
(2(ss)) + F(Qu(s, a)) - T"
< max E x²
sup (uls,a)) - E(Q + max E.,, sup B(Qa(s, - (")
12

---

Under review as a conference paper at ICLR 2021
Starting with the red term, firstly obtain probabilistic bound the distance term inside the expectation with the Chebyshev's s inequality
1
P[x--ixii oK) S
K
EIQu K KVVar[Qu(ssaa]
- K
3
P
sup /Var(Qu Qu((s,)) - ElQa(s, a)ll 3K
K2
Secondly, note that by assumption |Q] is bounded by Qm- This provides us an upper- bound on the difference [Q(s,a) a) F(Q(s, a)]l S 2Qm. Making use of both the general upper-bound and the tight probabilistic bound, by setting n' ra(as) sups (Var(Qu(s.aa) 7 ( we have
B
max E, sup (Q(ss,) E[Qu(s,a)] x¹
max E, x²'
sup ((l(s,a))ElQuss.lll -l)
supk (ar(Quu()a))
2 3K + K2 2m SB
$
By assumption ((s) 2 1 and can be safely ignored. By picking the appropriate K and 3, we can bound the red term by any constant B R. The same bound holds for the blue term since E[T"Qx-1($ a]] E[Qu(s,a)) We therefore arrive at constant bound for the Bellman error term max, F (s),
Note that in Theorem A.2 Assumption 1) does not change the optimization problem in equation and Assumption 2) can be easily satisfied by imposing Spectral Norm on the function.
Now according to the constant bound on 0(s, a) from Theorem A.2
the convergence of our proposed
framework follows directly from Theorem A.I Kumaar et 2019
Farahmand et 2010 Munos
2003 with the set of policies II' x π (a|s)) sups Var(Qa(aaa) n (clss)/((s) E
9
A.2 IMPLEMENTATION DETAILS
LunarLander: We set our expert to be simple 3-layer actor-critic agent trained completion with (Peng et al. 2019). We take the final replay buffer (size 100,000) with average reward of 269.7. The contains
vertically clipped dataset in Figure contains 76,112 samples, and the horizontally clipped dataset in Figure contains 21,038 samples.
We then train simple 3-layer actor-critic off-policy agent the clipped datasets according to Algorithm (we do not take the MMD loss in line 11 to enlarge the effect of OOD samples).
We ran benchmarks the official GitHub code of BEAR and the updated version provided by the authors. We ran parameter search on all the recommended parameters kernel. type (gaussian, laplacian), mmd sigmaE 10.20), 100 actions sampled for evaluation, and 0.07 being the amnd_target taandtttrgt.threhholee wtt.eshorood.... We are able to reproduce the results reported in (Fu et al. 2020) with both the official GitHub and the updated version.
Baseline (BEAR):
Our method (UWAC): We apply our weighted loss to Algorithm I to the updated BEAR code provided by Kumar et al. 2019). We keep the hyperepassmeters and the network architecture exactly the same as in BEAR. For experiments on the Adroit hand environment, we further enforce Spectral Norm on the Q function for better stability similar to (Yu et al. 2020) and theoretical guarantee as
guthuh.com/mill-eekkceyyyl.t. evaluations
13

---

Under review as a conference paper at ICLR 2021
shown in Appendix A.1 stability. For the choice of in Algorithm[]] We swept over beta values from the set (0.8, 1.6, 2.5), determined by matching the average uncertainty output during training time. We found that the model is quite robust against betas: 0.8, 1.6 gave similarly good performance across all tasks in our experiments. We also note that / can be absorbed into the learning rate since it acts both the actor loss and critic loss. However, since the MMD loss from BEAR is not B-weighted, we make the design choice to tune in stead of the MMD weight a.
We clip the inverse variance to within the range of (0.0, 1.5) for numerical
Ablations:
Our first study isolates the effect of Spectral Norm on agent performance. Although BEAR Spec- tral Norm enforces a bounded Q function and maintains good training stability, Spectral Norm does handle OOD backups on the narrow Adroit datasets. We discover experimentally that BEAR+SN performs much worse than BEAR only, we plot the complete results of BEAR+SN V.S. BEAR in Figure
Our second study isolates the effect of Dropout on agent performance as regularizer, since dropout alone does not handle OOD backups on the narrow Adroit datasets. We observe experimentally that UWAC without uncertainty weighing (BEAR+
Norm) does not change the behavior of BEAR under Spectral Norm (Figure TO) and performs worse than UWAC (Figure TT and the original BEAR (Figure 12
14

---

Under review as a conference paper at ICLR 2021
B FIGURES
action do nothing
action fire left
action nothing
action fire left
-A3
-044 -6. 80
cas
0.0
distacoment
action 2: fire main engine
actionn 3: fire right engine
2159
actiiooon fire main engine
1.50
action 3: fire right engine
8.35
630
0.0
acrtal daslacament
hortaonial Esplecement
52
hottontta disllacmmnn
12
nerizonda duptacceent
Figure 6: Left. The training dataset observations with vertical displacements > 0.8 removed. This makes all states on the top OOD states. Right. Our model estimates higher uncertainty (brighter color) on the top and lower uncertainty (colder color) on the bottom.
15

---

Under review as a conference paper at ICLR 2021
Average Return
le12
Target
Average Retum
1e12
Target
-45
2,0
olar BEAR
outs
-50
1.5
relocate eelecte-ookneee clonee--oddddoo
BEAR
-10
-55
1.0
-60
-15
0.5
-65
0.0
-20
200
400
200
400
200
400
200
400
Training Epochs
1e12 Training Epachs
Training Epochs
1e12 Training Epochs
3000
1250
ours BEAR
1000
ober expe-xppevvv
2000
750
BEAR
500
1
250
0
200
400
200
400
200
400
200
400
Training Epochi
1e12 Tralining Epachs
Training Epochs
1o12 Training Epachs
3000
-45
outS BEAR
00o-exppet-v0
2000
2
door noo--doed-vo..
-50
-55
BEAR
1000
-65
200
400
200
400
200
400
0
200
400
Training Epochs
Training Epochs
Training Epochs
1e12
Training Epochs
4000
4000
20
out BEAR
relocate- humanvo
gen cipootoo
3000
3000
10
1.5
2000
2000
1.0
1000
-10
ours
BEAR
200
400
200
400
200
400
0
200
400
Training Upochs
1e12 Training Epochs
Training Lpochs
1e12 Training Epachs
100
auTs
2.0
hammer- humn-
400
our BEAR
BEAR
50
1.5
200
1.0
door
0.5
-200
0.0
200
400
200
400
200
400
0
200
400
2000
Training Epochs
1e12 Training Epochs
Training Epochs
1e12
Training Epochs
ours BEAR
4000
humatvuu
1500
regcate expertvv
1000
2000
BEAR
500
1
1000
0
200
400
200 400
200
400
200
400
Training Epochs
Training Epochs
Training Epochs
Training Epochs
Figure 7: Plot average return v.s. training epochs, together with the corresponding average Q Target over training epochs on the D4RL Adroit hand offline data set. Results are averaged across 5 random seeds. Note that the performance of baseline (BEAR) degrades over time (also noted in original paper Kumar et al. (209)). and the Target Q value explodes. Our method, UWAC, achieves significantly better overall performance and training stability.
16

---

Under review as a conference paper at ICLR 2021
Average Retur
Target
Average
Target
-45
outs BEAR
outs BEAR
reoccaee doned
-10
-55
20
300
-60
-15
200
100
-20
Training Epochs
Tnaining Epachs
Training
Tra Epochs
1250
ours BEAR
80
3000
1000
503r 503/ eoprrttttt.... 50
2000
outs
60
500
40
1000
BEAR
40
250
20
Training Epochs
ining
Training
Tra iniin Epochs
3000
-05
ours
doof (oppeettt
2000
80
droncloned-o vo
-00
BLAR
ours
60
-55
20
1000
40
10
-55
200
200 Training Epochs
Training Epochs
Training Epochs
Training Epochs
4000
4000
20
ours BEAR
125
e5h oxgertyo
3000
3000
relocte- human-um
10
100
2000
2000
0
50
1000
1000
-10
burs
25
BEAR
-20
200
200
400
200
200
Training Epochs
Training Epochs
Training Epochs
Praining Epochs
100
Boor-hmmm--v
50
BEAR
OUTS
250
200
hammer- www.an--m
400
outs
BEAR
600
150
200
100
0
-50
200
200
400
200
400
200
Training Epochs
Tra Inaining Epochs
Training Epochs
ininigg Epochs
2000
150
pen huren-o
1500
BEAR
oUrs
relicate expertvvo
4000
1000
2000
2000
BEAR
50
1000
1000
200
400
200
400
200
400
200
400
Training Epochs
Training Epochs
Training Epochs
Training Epachs
Figure 8: Plot of average return v.s. training epochs (zoomed-in). The figure the same as except that the second column is zoomed- in on the Q values of the UWAC critic.
17

---

Under review as a conference paper at ICLR 2021
Average Return
le12
Target
4000
Average Return
Target
1250
1000
BEAR
gon Dipet-v0
3000
3000
750
2000
SN
2000
BEAR
500
1
1000
250
200
400
200
400
200
400
200
400
Training Epochs
1e12
Traiiing Epachs
Training Epochs
1e12 Training Epochs
2,0
SN BEAR
hammer-human-i -unne--uuaa-i
-55
1.5
-10
-60
1.0
0.5
-65
BEAR
200
400
200
400
200
400
0
200
400
Training Epochs
1e12 Traiiinn Epachs
Training Epochs
1e12
Training Epachs
SN BEAR
SN BEAR
2.0
door-expert-o
2000
2
door Door-human-g
-20
1.5
1.0
1000
40
0.5
200
400
200
400
200
400
200
400
Training Epochs
1e12 Training Epochs
Training Epochs
1e12 Training Epochs
SN BEAR
2.0
-5
pon hamanumnvv
1.5
-10
1000
BEAR
1.0
-15
0.5
500
1
-20
0.0
0
200
400
200
400
200
400
0
200
400
Training Epochs
1e12 Training Epochs
Training Lpochs
1e12 Training Epachs
2500
harneer Clened-a0
-230
2.0
re0Gate expert- 8
2000
SN BEAR
-240
1.5
1500
-250
1.0
1000
260
SN
0.5
500
270
BEAR
0.0
200
400
200
400
200
400
0
200
400
Training Epochs
1e12 Training Epochs
Training Epochs
1e12
Training Epochs
-5
2.5
12500
SNN BEAR
2.0
hammer expert-vo vo
10000
-10
1.5
7500
BEAR
1.0
5000
0.5
2500
-20
0.0
0
200
400
200
400
200
400
200
400
Training Epochs
Training Epochs
Training Epochs
Training Epochs
Figure 9: Ablation: Plot of average return v.s. training epochs for BEAR v.s. BpARt Norm, together with the corresponding average Q Target over training epochs on the D4RL Adroit hand offline data set. Results are averaged across 5 random seeds. Although BEAR with Spectral Normalized Q function maintains stable Q estimate during training, BEAR with SN achieves significantly worse training performance in terms of average return.
18

---

Under review as a conference paper at ICLR 2021
Average Retur
+ Target
Average Return
Target
-52.5
150
-240
500
hammer--dee Nemime--dendd-00
door par-doondd- 40
-55.0
57.5
-200
300
60.00
50
200
SN- Dropout
-270
SN+ Droprrt
100
62.5
200
400
200
400
200
Training Epochs
Training Epochs
Training Epochs
Training Epochs
150
Dropout
2500
600
125
relecate cloned-v
2000
-6
400
100
75
-8
200
50
500
SN+
25
400
200
400
Training Epochs
Train Training Epochs
Training Epochs
Tra Epochs
-50
-240
1500
hammer-enn www.ee--p--ttt
-250
4000
-55
3000
1000
-260
2000
-270
SN-+Dopout
500
5N+Dropout
1000
-200
-65
Training Epochs
Training Epachs
Training
Tra Epochs
800
Dropout
SN
reloatt--oae rebrate-expertv0
Sen-exourt
600
3000
2000
-8
400
2000
-10
1000
SM+ SN+Dropout
-12
Training Epochs
ining Epachs
Training
Tra Epachs
-2.5
harmar-humann -huann-o
deor Num--umma-e
-55.0
200
250
600
57.5
400
-260
-60.0
100
200
62.5
SN Dropptttt
Dropout
SN
65.0
200
200
Training Epochs
Training Epochs
Training Epochs
Epochs
SN+ Dropout
4000
1000
200
SN
relocate- human-vo
800
3000
600
100
400
Dr-pDroput
200
SN
-12
400
200
200
Training Epochs
Training Epochs
Training Epochs
Training Epochs
Figure 10:
training epochs for BEAR+Spectral Norm BEARR-ropout-SSpectal Norm, together with the corresponding average Q Target over training epochs on the D4RL Adroit hand offline data set. The results are averaged across random seeds. Without the UWAC reweighing loss, performing dropout on the Q function does not lead to improved performance.
Ablation: Plot of average return
19

---

Under review as a conference paper at ICLR 2021
Average Return
o Target
Average Retum
Tanget
-45
-180
ouTS
150
outs
dooncloned-v0
-50
uncertainty
-200
We uncertainty
400
100
-20
-55
Sammer--com
-240
200
-60
-200
200
400
200
400
200
400
200
400
Training Epochs
Training Epochs
Training Epochs
Tra iniinin Epochs
600
1250
2500
-5.0
1000
2000
-7.5
750
OUTS
1500
-10.0
ouT
W/O uncertainty
we uncertainty
pen
500
1000
-12.5
200
500
-15.0
-17.5
200
400
200
400
200
400
0
200
400
Training Epochs
Training Epochs
Training Epochs
Tra Training Epochs
3000
2000
15000
5000
dosp expertvv
1500
4000
2000
10000
ours
1000
ours
1000
wro incertainty
5000
w/o uncertainty
2000
500
1000
200
400
200
400
200
400
200
400
Training Epochs
Tra lning Epochs
Training Epochs
Training Epochs
peh 00hoxppttvvv
3000
4000
relocate expest-vo
4000
3000
3000
2000
2000
ours
ours
w/o uncertainty
2000
2000
w/o uncortainty
1000
1000
1000
400
200
400
200
400
200
Training Epochs
Tra iniin Epochs
Training Epochs
Training Epochs
door-ruman-vd
50
w/o uncertainty
200
www.er-hummn--n
400
w/b uncertaisty
600
200
400
100
-50
-200
0
400
200
400
200
400
200 Tra Training Epochs
Training Epochs
Tra ining Epochs
Training Epochs
2000
1000
4000
hu 40
1500
w/o uncertainty
3000
relccate-hie human-yv
10
we uncertainty
600
1000
2000
400
500
1000
200
-10
400
200
400
200
400
200
Training Epochs
Training Epochs
Training Epochs
Training Epochs
Figure 11: Ablation: Plot of average return v.s. training epochs for UWAC (ours) v.s. ours without uncertainty weighing but with dropout in the Q function, together with the corresponding average Target over training epochs on the D4RL Adroit hand offline data set. The results are averaged across 5 random seeds. Without the weighing loss, performance of the agent drops drastically. Note that low performance on hammer-cloned, door-cloned, and relocated-cloned may be attributed to the bad quality of the datasets caused by data collection (explained in section 5.3
20

---

Under review as a conference paper at ICLR 2021
Average Retur
1+12
Target
Average
1+12
Target
-45
2.0
ours
2.0
1.5
-200
SN Dropous
SN
1.5
-55
-220
1.0
-60
SN+Dropput
ours
0.5
-240
BEAR
0.0
-280
Training Epochs
1e12
Epachs
Training
1+12 Training Epochs
1250
relocate--conn relocste-cloeede -ooe----o-edddd
pen clnee 2
1000
ours
-10
curs
SN-Dropout
SN+ Dropout
500
BEAR
-15
BEAR
250
-20
Training Epochs
1e12 ining
Training
1e12 Tra Traiiing Epochs
3000
15000
hammerr- -otper--evvee
doof pippertet
2000
ours
SK Dropout
10000
S Dropout
curs
1000
BEAR
5000
BEAR
0
200 Training Epochs
200
Training Epochs
Training Epochs
1e12 Training Epochs
4000
4000
relocate expertvve
4000
e5h ougertyo
3000
curs
3000
ours
2000
SN Dropout
2000
2000
i Dropout
BEAR
BEAR
1000
1000
1000
200
400
200 Training Epochs
200
Training Epochs
1e12 Training Epochs
1.12 Tra Training Epochs
100
SK+ Dropout
curs
2.0
hammer--huaan hhw.---uma-
400
purs
SN Dropout
50
1.5
BEAR
200
BEAR
1.0
0
1
0.5
-50
-200
0.0
200
400
200
400
200
400
200
Training Epochs
1e12 Training Epochs
Training Epochs
1e12 Training Epochs
2000
20
curs
purs
pen huren-v)
1500
SN Dropout
relocatee Pelraat--ummnny humaeruuaooo
10
SNI Dropout
BEAR
2
BEAR
1000
1.0
-10
-20
200
400
200
400
200
400
200
400
Training Epochs
Training Epochs
Training Epochs
Training Epochs
Figure 12: Ablation: Figure
10
plotted together. Note that SN+Dropout (purple) also denoted as
ours- mi-rsw/o-unnertaing in Figure
21

---

Under review as a conference paper at ICLR 2021
Figure 13: Sequences of our offline agent trained from expert demonstrations executing learned poli- cies performing on the halfcheetah, walker2d, and hopper tasks in the MuJuCo Gym environment. See the videos attached in the supplementary.
Figure 14: Sequences of the agent trained from human demonstrations executing learned policies performing the Adroit tasks of hammering a nail, twirling pen and picking/moving ball. The task of opening a door is shown in Figure See the videos attached in the supplementary.
22


Reviews:
[{"review_id": "wDEUSe5RkRi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a simple and efficient method for training offline RL agents. They estimate the epistemic uncertainty of their model through dropout  variational inference. While this method is not novel, it has never been applied to offline RL as far as I know. On the basis of this epistemic uncertainty estimate, they regularize the policy search to avoid sensitivity to overestimates in poorly known states. The way this regularization is performed is taken from [Kumar2019]. The authors then experimentally demonstrate the effectiveness of dropout uncertainty estimation for RL, and achieve the best performance on a classic offline RL benchmark, with the best offline RL algorithms for continuous state-action MDPs.\n\nThere is not much to say about the paper. It is well written and positioned. One could that the method is the straigthforward application of dropout variational inference to [Kumar2019] algorithm, but it remains that the empirical results are quite significantly improving the state of the art, and as such, it deserves to be known.\n\nI only have one minor remark to share: Section 4.1, the explanation of the variance decomposition is a bit misleading. Red minus blue is the dropout variational variance, and this is what measures the uncertainty of the model (not singlehandedly the red term). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "vGJ0TQspD2A", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple offline RL algorithm, called UWAC, which down-weights the bellman errors for uncertain state and action pair while the uncertainty is measured by MC-dropout. The algorithm empirically shows competitive performance in various test domains in the D4RL benchmarks, including Adroit hand manipulation tasks. Overall, I agree with the main motivation for the paper that tries to estimate and use uncertainty, and it seems that the method empirically achieves the goal. Still, I have a concern about the theoretic validity of MC-dropout in the Q-learning setup and the clarity of the papers (especially typos and proofs in Appendix). Hopefully, the authors can address my concern in the rebuttal period.\n\nConcerns:\n* It is unclear whether MC-dropout measures epistemic uncertainty precisely even in Q-learning in which bootstrapping is in use; since the target Y for certain input X depends on its own estimation, MC-dropout could become unreliable. In the meantime, the same algorithm could leverage uncertainty measured for transition dynamics with proper scaling as it is done for $Q$. Why can measuring epistemic uncertainty for $Q$ via MC-dropout be valid (or effective)? What is the main advantage (or motivation) over other uncertainty measuring methods (such as deep ensemble) or other domains (such as transition dynamics learning)?\n* Is there a fundamental reason for using BEAR as a base algorithm to work upon? How well will the raw UWAC -- without MMD penalty of BEAR -- perform? As it is mentioned in Section 3.2, the support constraints via MMD is rather empirical and could be wrong in some cases. If raw UWAC fails, then what will be the main cause?\n\nMinor Comments:\n\n* Section 4.1, Variance equation: what is $\\sigma^2$, what is the theoretic impact of this term, and how is this value decided from the implementation perspective? Also, why $\\hat{Q}$ is in a vector notation (transpose)?\n* Algorithm 1: is $Z$ approximated with $\\{a_i\\}$, or is $Z$ simply ignored?\n* How to set a proper $\\beta$, and how sensitive is UWAC to the hyperparameter?\n* Section 5.1: what do you mean by the ‘final’ replay buffer? Does this mean that the last 100,000 transition tuples are saved for the offline RL dataset? Then, why do trajectories gathered by executing nearly expert policy have relatively complete coverage over the observation space? The expert policy won’t execute the action that is suboptimal, so the coverage would be limited. Also, it would be nice if you can include the uncertainty figure of Q which is trained without skewing.\n* Figure 5, pen-human-v0 results: what is the possible reason for the degeneration of UWAC? Wrong OOD detection?\n\nTypos:\n\n* Equations are the components of a sentence, and every sentence ends with proper punctuation; add proper punctuation before or after every equation.\n* Section 3.1 -- $\\gamma \\in (0,1)$ → $\\gamma \\in (0,1]$. \n* The second paragraph of page 5 -- erase parenthesis around $\\pi$.\n* For the completeness of the paper, please include all loss functions used in the algorithm; the exact MMD penalty term is missing in Algorithm 1.\n* Target network update should be $\\theta’ \\leftarrow (1-\\tau) \\theta’ + \\tau \\theta$, if $\\tau$ is a polyak coefficient used for slowly moving target network. Also, just use $\\theta_{1,2}$ instead of $i$. $i$ is used in another context, so this is confusing.\n* In Algorithm 1, $\\phi_i$ in line 12 is a typo, and $\\{ a_j \\sim D\\}_{i=1}^{n=10}$ should be $\\{a_j \\sim D\\}_j^n$.\n* In Tables 1 and 2, when your results are the best, the results that lie in some confidence interval of yours should also be bolded for a fair comparison.\n* The second paragraph from the bottom of page 7, “when Q Target” → “when Q target”, “and actions: As performance” → “and actions: as performance”.\n* It would be nice if Figures 7 and 8 in the appendix can be integrated into a single figure.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mrnxCQOf_Ir", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary :\nThis paper considers the problem of dealing with uncertainty for static datasets in offlineRL. The authors propose a novel algorithm ‘UWAC’, uncertainty weighted actor-critic. UWAC takes a Bayesian perspective of RL, and uses Monte Carlo dropout for detecting, down-weighting OOD samples. Building on BEAR, they estimate the epistemic uncertainty as the Var(Q(s, a)), and modify the update to downweight samples with higher variance. On D4RL datasets, UWAC seems to perform competitively against model-based baselines like MOPO, model-free baselines like BEAR/CQL.\n\nReasons for the score:\nI vote for accepting the paper, with some improvements. The paper suggests a strong improvement in the stability and performance of model-free algorithms for offlineRL. I would strongly encourage the authors to improve the writing and presentation of the algorithm in favour of clarity. \n\n\nStrengths:\n+ UWAC bridges the gap between the performance of model-based and model-free algorithms in offlineRL. \n+ The problem is well motivated, and uses dropout variational inference to estimate the epistemic uncertainty and imposing a soft-penalizing on OOD samples.\n+ Provides extensive experiments and empirical evidence on the D4RL benchmark, where it outperforms MOPO and appears to be more stable than BEAR.\n\nWeaknesses:\n- The baseline for MOPO is missing from the plots. As we are considering the stability of the algorithm as compared to other baselines (like BEAR) it would be critical to compare the same for MOPO (trained for a similar number of epochs).\n- Drop in performance seems to be significantly related to exponential increase of the Q-values. Taking this into account, I would consider an implementation with some variant of clipping (gradient or reward) to be a strong missing baseline. \n- Some sections were hard to follow, particularly Section 3, Section 4 could be substantially improved for clarity. Comparison among related work could probably be represented via a table of tradeoffs to better understand the strengths, nuances of the proposed algorithm. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P0omdO9CwgU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n\nThis paper shows that a easy-to-implement dropout to estimate uncertainty can address the value overestimation problem in offline RL for OOD action or states. The paper baselined against the results from D4RL and showed to outperform those more than half of the times.\n\n \n##########################################################################\nPros: \n\nUncertainty is an important yet underexplored domain in Offline RL. Having a simple method that can be applied to actor-critic based methods is attractive. \n\nAs far as I know, there hasn’t been literature that combines dropout and Offline RL uncertainty estimation in the loss function\n\nThe performance of the algorithm is decent on D4RL, a standard offline RL baseline.\n\n##########################################################################\n\nCons: \n\nSince offline RL is motivated by a potentially costly evaluation and/or data collection process, It is risky to add additional hyperparameters in offline RL. Thus it would be informative to mention the following in the paper: Is your model sensitive to the choice of beta? How many values of beta did you sweep over? \n\n\n##########################################################################\n\nQuestions during rebuttal period: \n1. I could be missing something. Since the beta hyperparameter is a constant in all losses. Can it be absorbed into the learning rate?\n\n2. In the appendix, I found theorem A.2 confusing. It assumes that Q is bounded for any (s,a). this assumption makes the proof almost trivial. The given justification is that Q can be bounded with spectral norm, yet in practice spectral norm performs much worse than no spectral norm. I thus find the theoretical backing shaky, especially you further assumed in the paper that Z can be absorbed into beta. Can you expand more on this?\n\n3. Could you address the Cons above?\n\n##########################################################################\n\nAdditional suggestions:\n\n1. Can you add a graph in the appendix where you zoom in a bit on the Q target? I want to see whether it still overestimates, and if so by how much. Having the y axis = 1e12 hides those information.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7G90ThkiIIS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use an uncertainty-weighted objective for offline RL with BEAR (Kumar et al.) that penalizes the MMD distance between the learned policy and the previous policy. The uncertainty weighted objective weights the policy improvement objective with the variance in the Q-function, where this variance primarily represents aleatoric or intrinsic uncertainty, not the epistemic or belief uncertainty. They show that their method performs reasonably better than prior methods on the D4RL datasets and show that the learned Q-values are better than BEAR.\n\nWhile the work seems promising empirically, I have a number of concerns about the method:\n\n1. The division by variance comes across as a hack -- while standard deviation could be more appropriate, I wonder why there is no need to even center the Q-function before dividing by the variance. Moreover, what does the current weighting do? Does it penalize specific state-action pairs? Can it be compared to a more standard square root inverse-counts style weighting in a tabular setting? The lack of such ablation experiments make it impossible to interpret the method.\n\n2. Typically the notion of uncertainty in batch RL or exploration is epistemic uncertainty, where uncertainty stems from the fact that there is limited data available to the agent. The notion of uncertainty or variance considered here is not this notion. Since there are no time-consisent backups, it is unclear how epistemic uncertainty is being preserved termporally in a trajectory. The method, as it stands, seems to resemble aleatoric uncertainty more, and if this is the case, why not just use distributional critics? What is the specific beneift from dropout-based Q-functions?\n\n3. Ablations on uncertainty: the method uses both a constraint against the behavior policy and a weighting on the policy objective. Which of these helps more and when? The Q-value plots shown in the paper are reasonable, but it is still unclear when the uncertainty still helps and why, and more understanding is needed on this part in my opinion. In fact, the Q-values plots perhaps signify my belief about 2, that their approach works because it performs \"better\" Q-learning similar to Agarwal et al. 2020. \n\nOverall, I do not buy the claim that the real benefits are coming from uncertainty based Q-functions, but perhaps more so like a stronger critic function. The paper doesn't ablate over showing the benefits of uncertainty over a policy constraint or comparing to a method that actually learns Q-value estimates aware of the epistemic uncertainty making it hard to compare to these. Finally, the method does come across as adhoc. Is it possible to test the method on some Atari domains, which have some different properties than the continuous control D4RL benchmarks? This would validate if dividing by the variance is indeed transferable across domains.\n\n\n------------Post Rebuttal----------------\n\nI thank the authors for their response. I disagree about epistemic and aleatoric uncertainty. Bootstrapped DQN (Osband et al. 2016), Randomized prior functions (Osband et al. 2019), and several other works show that to get the variance of different possible Q-functions given the data, $p(Q|D)$ or differnet possible MDPs given the data, $p(M|D)$, you need backups consistent in time, i.e. the same dropout mask is to be used for both the main Q-network and the target Q-network for the backup. This is the uncertainty that we trypically need in offline RL and is also used in Theorem A.2 (for the high probability bound which is given the data), and that's why ensembles with Q-functions consistent with themelves are typically used. By merging the target values across different dropout masks, the uncertainty is not timewise consistent. While the paper that the authors point does actually do dropout masks with Q-function at each step, it is discussed in later works including Osband et al 2016, 2018 that not being consistent over time is leading to wide uncertainty estimates.  I would recommend the authors read the discussion on Posterior sampling for RL vs Optimism and Thmpson sampling for a discussion on this.\n\nI am a bit disappointed with the rebuttal. I expected a comparison to such metods that perform timewise consistent uncertainty estimation and also to distributional RL, since the algorithm that the authors use can also be drawn similar to a set of particles of Q-functions and performing a backup using target values computed using all of the possible particles, which is essentially what, for instance, QR-DQN does in a way or even IQN does in a way. Even REM would have been fine. Without this comparison, I unfortunately cannot increase my score and I am going to retain my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Being able to give confidence intervals or have a robust measure of uncertainty is very important for offline RL methods. In this work, they proposed a dropout based method to have a measure of uncertainty. The authors provide an significant empirical improvements over other baselines. Nevertheless, as it stands right now and as AnonReviewer5 have pointed out, this paper has some important shortcomings. I have noticed that the authors have updated the paper, but still the some of the important points made by AnonReviewer5 are unaddressed as it stands right now. Thus, I am suggesting to reject this paper hoping that the authors will address those issues and resubmit to a different venue.

Firstly, I agree with AnonReviewer5, it is not clear if the dropout and the variance trick used in this paper actually represents epistemic uncertainty that we would like to have for an offline RL algorithm, because the variance do not necessarily need to shrink as you train it with more data, and as opposed to supervised learning setting it is not clear what type of uncertainty the proposed dropout method will induce in the offline RL setting. It would have been nice to have some results showing how calibrated the uncertainty estimates coming from the dropout is... I would recommend the authors not to include any claims regarding the epistemic uncertainty in the camera-ready version of the paper.

Also as AnonReviewer5 pointed out, having distributional baselines and/or ensemble methods like REM or bootstrapped DQN would be a more fair comparison. So, it would be nice to see some of those baselines in a future version of this paper.
Decision: Reject