Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
Authors: David Brandfonbrener
Remi Tachet des Combes
Romain Laroche
Venue: Submitted to ICLR 2023 (2023)

Abstract: Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings where such estimates exist. In this work, we develop a novel method for incorporating scalable uncertainty estimates into an offline reinforcement learning algorithm called deep-SPIBB that extends the SPIBB family of algorithms to environments with larger state and action spaces. We use recent innovations in uncertainty estimation from the deep learning community to get more scalable uncertainty estimates to plug into deep-SPIBB. While these uncertainty estimates do not allow for the same theoretical guarantees as in the tabular case, we argue that the SPIBB mechanism for incorporating uncertainty is more robust and flexible than pessimistic approaches that incorporate the uncertainty as a value function penalty. We bear this out empirically, showing that deep-SPIBB outperforms pessimism based approaches with access to the same uncertainty estimates and performs at least on par with a variety of other strong baselines across several environments and datasets.

Under review as a conference paper at ICLR 2023
INCORPORATING EXPLICIT UNCERTAINTY ESTIMATES INTO DEEP OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings where such estimates exist. In this work, we develop novel method for incorporating scalable uncertainty estimates into an offine reinforcement learning algorithm called deep-SPIBB that extends the SPIBB family of algorithms to environments with larger state and action spaces. We use recent innovations in uncertainty estimation from the deep learning community to get more scalable uncertainty estimates to plug into deep-SPIBB. While these uncertainty estimates do not allow for the same theoretical guarantees as in the tabular case, we argue that the SPIBB mechanism for incorporating uncertainty is more robust and flexible than pessimistic approaches that incorporate the uncertainty as a value function penalty. We bear this out empirically, showing that deep-SPIBB outperforms pessimism based approaches with access the same uncertainty estimates and performs at least on par with a variety of other strong baselines across several environments and datasets.
INTRODUCTION
In the study of offline reinforcement learning (OffRL,, uncertainty plays a key role (Buckman et Levine Levine et al., (2020). This is because, unlike online RL where an agent receives feedback in the form of low rewards after taking bad action, an OffRL agent must learn from fixed dataset without feedback from the environment. As a result, a consistent issue for OffRI. algorithms is the overestimation of states and actions that are not seen in the dataset, leading to poor performance when the agent is deployed and finds that those states and actions in fact have low reward (Fujimoto) ctt al., 2019b). To overcome this issue, OIRLL algorithms often attempt to incorporate some notion of uncertainty to ensure that the learned policy avoids regions of high uncertainty.
There are two main issues with this approach: (I) how to define uncertainty and (2) how to incorporate uncertainty estimates into the OffRL algorithm. In tabular and linear MDPs, issue (1) is resolved by using visitation counts and elliptical confidence regions, respectively (Yin et al.) 2021 Yin Wang
2021 Jin et al. 2021 Laroche et al., 2019). In the large-scale MDPs that consider, neither of these solutions work, but there is a large literature from the deep learning community on uncertainty
quantification that we can leverage for OftL (Ciosek al.. 2019 Osband et al. [0018] 202 Burda
ct al., 2011 Ostrovski et al. 2017 Lakshminarayanan et al. 2017 Blundell et al. 2015) Gal &
Ghahramani 2016). Given these uncertainty estimators, this paper focuses primarily on issue (2),
how to incorporate uncertainty for OffRL.
To understand how to best incorporate uncertainty into an OffRL algorithm, we first provide a high level algorithmic template that captures the majority of related work as instances of modified policy iteration (Scherrer et al., 2012) that alternate between policy evaluation and policy improvement. We then can sort prior work into four categories along two axes: whether the algorithm modifies the evaluation step or the improvement step, and whether the algorithm uses an explicit uncertainty estimator not. One class of algorithms modifies the evaluation step by introducing value penalties based on explicit uncertainty estimates, which we will call pessimism (Petrik et al. 2016 Buckman al. 20200 Jin et al.] 2021). An alternative modifies the value estimation without using uncertainty estimate, like in CQL (Kumar et al., 2020). Another family uses behavior constraints that modify the policy improvement step keep the learned policy near the behavior policy (Fujimoto al.

---

Under review as a conference paper at ICLR 2023
2019b a), but does not explicit uncertainty. Instead, we propose the fourth class of methods that leverages uncertainty- -bssed constraints in the policy improvement step and is inspired by the SPIBB family of algorithms (Laroche et al. 2019 Laroche & Tachet des Combes 2019 Nadjahi et al. 2019 Simão et al. 2020). These algorithms modity the policy improvement step like a behavior constraint, but also reason about state-based uncertainty like the pessimistic algorithms. Explicitly, we define the deep-SPIBB algorithm that effectively incorporates uncertainty estimates into OftRL.
The main contributions of this paper are as follows:
We introduce the deep-SPIBB algorithm which provides a principled way to incorporate scalable uncertainty estimates for OffLL. We instantiate this algorithm using ensemble- based uncertainty estimates inspired by Bayesian inference (Ciosek et al. 2019 Osband ct al. 2021).
We provide a detailed comparison of several different mechanisms to incorporate uncertainty by considering how each mechanism operates at the extreme settings of its hyperparameters. This analysis shows that deep-SPIBI provides flexible and robust mechanism to interpolate between various extremes (greedy RL, behavior cloning, and one-step RL).
Through experiments on classical environments (cartpole and catch) as well at Atari games, we demonstrate the efficacy of deep-SPIBB. In particular, we find that deep-SPIBB con- sistently outperforms pessimism when given access to the same imperfect uncertainty estimators.
When deep-SPIBB has access better uncertainty estimators (as in the easier cartpole environment) it is able to substantially outperform our other baselines of CQL and BCQ well. This suggests that uncertainty estimators improve, deep-SPIBB will provide useful mechanism for incorporating them for OffI..
PRELIMINARIES
We consider an OffRL setup with discrete action space and access dataset D {(s), Ty.Ty: }- consisting of N transitions collected by some behavior policy 3. The goal is to
learn policy from this data to maximize expected discounted returns J(T) Er 12 oc
rrd..
2.1 ALGORITHMIC TEMPLATE
The vast majority of prior work on the OffRL. problem can be seen through a common algorithmic template of modified policy iteration. Each algorithm alternates between policy evaluation and policy improvement steps. The main difference between algorithms comes in how they modify either the evaluation the improvement step. Below we first define the generic version of the OffRL algorithmic template and then explain how different OffRL algorithms modify this template.
Policy improvement by greedy maximization:
(-1s) arg max TEI aEA
Σ T (aas)) (((als)Qui)) ((als)Q())ss,))
(1)
Value estimation by fitted Q evaluation given the dataset D = {[3]] aj Tj s3) Define the Bell-
man operator from datapoint j with x (6+1), Q(1,, as T(j, Q) = r + + EvEA ((d166))((3 a'). Then the evaluation step is:
Q(++1) arg min
- T(i Σ - QEO
Q00)²
In addition to the policy and Q function, some algorithms we consider will also learn an estimated behavior policy B(a]s) and/or an uncertainty function a(s, a). Generally, 3 is learned by maximum likelihood supervised learning. The uncertainty á the other hand can be learned many different ways. We will discuss à in more detail in Section 3 when we describe our method.
With this template can provide characterization of much prior work that is summarized in Table I The essential axes that we consider are (1) whether the algorithm modifies the improvement step or the evaluation step and (2) whether the algorithm uses uncertainty function u(s, a) or not.
2

---

Under review as a conference paper at ICLR 2023
Table 1: characterization of how several baseline methods fit into our template.
Uncertainty- BCQ CQL
Uncertainty- deep-SPIBB (ours)
Modified improvement step Modified evaluation step
Pessimism
2.2 MODIFYING EVALUATION STEP
One approach to incorporating uncertainty is to introduce penalty into the value estimation step that encourages the policy avoid novel states or actions.
The simplest value penalty is to use explicit estimate of uncertainty, we will can this algorithm pessimism. Variants of pessimism have been examined in broad range of prior work (Petrik et al., 2016 Buckman et al. 2020, Jin et al. (022). Given uncertainty estimator u S + R, the pessimism algorithm modifies the evaluation step to be:
Uncertainty-based
(
arg min
Σ [(((s)) (T(i Q10) m((s).a)))
(3)
QEO
The hyperparameter a controls the amount of pessimism
Uncertainty-fres Alternatively, the algorithm can modify the evaluation step without use of an explicit uncertainty function. A popular representation of this approach is the CQL algorithm Kumar et al. (2020). In CQL there is no explicit estimate of uncertainty. Instead, the algorithm makes the following update in the evaluation step:
Q(++1) arg min Σ a log Σ exp(Q(s),a) Q(s),a)) + (Q(s). - ((+)) ))))
QEQ
(4)
The first term encourages the Q estimates to underestimate Q values at unobserved actions (via the log-sum-exp term) while remaining large at observed actions (via the Q(5,, aj) term). Again the hyperparameter controls the penalty.
As explained in the original paper, this version of CQL can viewed as a version of pessimism with entropy regularization term. In analog pessimism, the implicit uncertainty function would take the form of u(s, a) Stais)
where æ is the current policy iterate. Note this function is non-stationary since it depends on и. Moreover, in practice with neural function approximation, the CQL objective may behave differently than trying to implement this function explicitly. The implicit nature of the uncertainty used by CQL makes it different from standard pessimism with explicit uncertainty estimates.
r(als)
2.3 MODIFYING THE IMPROVEMENT STEP
Instead of modifying the evaluation step, we can alternatively modify the improvement step.
The simplest way modify the improvement step without using uncertainty estimate is to constrain the learned policy to choose actions that are well-supported under the estimated behavior. The main example of this algorithm that consider the BCQ algorithm (Fujimoto et al.) 2019b a). Explicitly, the BCQ algorithm with hyperparameter defines
Uncertainty-free.
B(als)
π (a)s) = a arg max (1) a'EA-(s)
Ar(s) a aEA:
max.'EA 3(alss) 2
2 (5)
where 3 is maximum likelihood estimate of the behavior policy. When 1 this is exactly behavior cloning and when the constraint has no effect. Importantly, these methods do not use any notion of uncertainty
states.
Another instance of an uncertainty baseed modification of the evaluation step is from the MBS algorithm of Liu et al. 2020). Instead of using uncertainty penalty, they threshold an uncertainty function and propagate the minimal possible return in the Bellman hackup if the uncertainty is too high.
3

---

Under review as a conference paper at ICLR 2023
Uncertainty-based. The final option is to modify the policy improvement step with the of an explicit estimate of uncertainty. Our propsed deep-SPIBB algorithm falls into this category. In contrast to behavior constraints, that we have in a given state. And in contrast to a value penalty, the uncertainty is not propagated in the evaluation step. The main example of this style of algorithm is the SPIBB family of algorithms (Laocche et al. 22019; Nadjahi et al., [019 Simão et al. 2020) which we will discuss further in Section when we introduce deep-SPIBB.
imcertainty-basee constraint takes into account the confidence
2.4 (UN)RELATED WORK
In this subsection, we promptly acknowledge the existence of algorithm approaches for Offline RL that are not directly related with the uncertainty question that this work is endeavoring to address. A wide range of algorithms rely on actor-critic algorithmic architecture in order to handle MDPs with continuous actions Wang et al. (2020): Wu et al. (2019); [Siegel et al.] (2020)] Kostrikov et al. (202T): the discrete action case and study different ways of incorporating uncertainty into the algorithm. Another group of algorithms take advantage of an explicit MDP model nh, which confers them better outtof-ddittibutiinn generalization capabilities Yu ct al. 2020):: Kidambi et al. 2020); Yu et al. (202)): Janner et al. (202I). In our study, all the considered algorithms are model-free in order to guard ourselves against confounding factors of additional implementation details. Finally, there is the return-condition supervised learning approach
Fujimoto & Gu (021)) We focus exclusively
that has recently been introduced in the Offline RL literature Chen et al. (202T): Emmons et al. (2021). We found the structure of these algorithms to be too distant from our work.
DEEP-SPIBB
We can now explicitly define the deep-SPIBB algorithm. To make the algorithm scale up high dimensional inputs, we use a neural uncertainty estimator described in Section 3.] Using this uncertainty estimator, deep-SPIBB uses a variant of the approximate Soft-SPIBB (Nadjahi al., 2019) mechanism to incorporate uncertainty estimates into the policy improvement step, described in Section 3.2
3.1 NEURAL UNCERTAINTY ESTIMATION
As presented in prior work, soft-SPIBB relies count-based uncertainty estimates i(s, a) that have high probability guarantees. In the tabular case, we can derive these estimates from visitation counts n(s, a) and set u(s a) = c/ v n(ss, a). Unfortunately, this is not a scalable solution in larger domains since it is unknown how to best generalize these precise notions of uncertainty to larger domains that require data-efficient generalization across states. So, we borrow from the literature neural uncertainty estimation (Ciosek et al. 2019 Osband al. 2021 and use uncertainty estimator based on ensembles trained to estimate random targets and regularized by random priors. The objective and estimator are described in full detail in Appendix [ At high level, the uncertainty is proportional to the variance of an ensemble of models, each trained to predict random function with high- -igee-iinniional outputs.
We make one key change relative to prior work. Rather than using both state and action as imput to our uncertainty estimator, we only use the ensemble-based uncertainty estimator estimate state-based uncertainty û(s). We then combine this with the behavior policy estimate (( s) derive u(s, a). In particular, we define our uncertainty estimator as
a(s, a) = i(s)
(6)
BBaass)
The rationale for using / /B(als) comes from the tabular setting, where if we were to use counts to
define û(s)
n(=) and ((a)s)
n(s,a) then our u(s, a) would be exactly the standard count-
n(s)
This decision is supported empirically the experiments section. Intuitively, this helps because it guarantees that the uncertainty estimator is consistent with the estimated behavior used in the SPIBB constraint.
based Vn(s,a)
4

---

Under review as a conference paper at ICLR 2023
3.2 ALGORITHM
The algorithm consists of three models: B(als) an estimate of the behavior policy, ú(s; a) an uncertainty quantification, and Q an estimated Q function. The Q updates are much like those of SPIBB-DQN (Laroche Tachet des Combes, 2019) except we use approximate soft-SPIBB and scalable neural uncertainty estimates. Since we use target networks to approximate policy iteration, we will use parenthetical superscripts to keep track of the policy iteration step (i.e. the number of times the target network has been updated).
Policy improvement step.
At step 1+ the policy x (f+1) approximates the solution to the following constrained optimization problem:
(-|s)) = arg max
Σ (s) aEA
s.t.
Σ u(s a) <(a|s) - M(ass) S Crrón
REAIA
aEA
Since this problem is difficult to solve exactly, the approximation technique described in Nadjahi et al. 2019).
Value estimation step. For the value estimation step, the standard expected SARSA backup from the Equation (2) Since x ((++1) is mucertainty-constraanee it prevents Q values from being propagated from state-action pairs with high uncertainty under ú.
Evaluation policy. One modification that make from prior work soft-SPIBB is generalize the algorithm by separating the hyperparameter étrain that governs the deviation from the behavior during the policy improvement step during training from Cvval that governs this deviation during evaluation. So the evaluation policy using the final estimated Q function Q(1) becomes the solution of the optimization:
(- (-1s) arg max
s.t. E m(als) -(alss))((asss))))))))))))))) S Ceval-
IEAIA
As we will see in Section this choice allows us to capture a richer tradeoff between different methods instead of simply interpolating between behavior cloning and greedy RL.
Algorithmic variants. We will consider two variants of the algorithm:
1. Standard deep-SPIBB where écnal étrain-
2. Generalized deep-SPIBB where we tune Geval independently of Etrain-
These variants will be discussed greater depth the next section where we compare the tradeoffs made by Etrain and leval with those made by the baseline offline RL algorithms introduced above.
4 COMPARING ALGORITHMIC TRADEOFFS
Each algorithm introduced above comes with hyperparameter that governs the tradeoff between acting greedily and restricting the learned policy to be safe (or two hyperparameters in the case of generalized deep-SPIBB). The key difference between the algorithms is how they choose to modulate this tradeoff and which points they choose at the extremes.
Extremal hyperparameter settings. To understand the tradeoff that each hyperparameter governs, it useful to understand what happens at the extremal values. The results of this analysis are summarized in Table There are few key takeaways from this analysis:
All algorithms capture greedy RL one extreme of the hyperparameters.
All algorithms except for pessimism capture variant of BC at another extreme setting of the hyperparameters. Note that due to the greedy nature of the policies defined in BCQ and CQL, they cannot exactly represent BC for stochastic behavior policy and instead choose the action that has maximum probability under the behavior (which we will denote argmax BC).
5

---

Under review as a conference paper at ICLR 2023
Table 2: A summary of what each algorithm reduces at extreme settings of their hyperparameters.
Algorithm
Extreme 1 argmax BC at T = argmax BC as -
Extreme 2
BCQ
Greedy RL at T =
CQL
Greedy RL at a
Pessimism
Minimal uncertainty as a BC at Etrain = tecal
Greedy RL at a 0
Deep-SPIBB
Greedy RL as étrain = cenall - 00
Gen Deep-SPIBB
BC at Eevol 0 One-step RL at Etrain
Greedy RL as Etrain Geval
0
Only generalized deep-SPIBB captures one-step RL (Brandionbrener et al., 2021 Gulcehre et al. 2021) that performs one step of policy improvement another extreme setting of the hyperparameters.
What happens to pessimism at the extreme. It is worth delving deeper into what happens for the pessimism algorithm as a + co. Note that we can view the pessimistic algorithm as optimizing an augmented reward f(s, a) r(s.s))) a)
u(s, a). As we send a + oc the impact of on i tends to zero. As result, we end up learning a policy that approaches T. (als) ) = [a = arg max,' Qu (s. a')] where Q is the Q function of the optimal policy for the reward function -w(8, (8, a). We will call Ty the minimal uncertainty policy. One key difference that the minimal uncertainty policy can prefer policies that remain in states that are often observed in the dataset over actions chosen by the behavior. This difference is not necessarily bad in all cases, but provides a different inductive bias than the other algorithms considered.
The issues with pessimism become more problematic when the uncertainty function is poorly estimated (as it may be in high- -igh-dimensioal state spaces). Since the default is minimize the uncertainty, if the uncertainty estimate has some accidentally underestimated uncertainties there is way to remedy the situation by tuning the hyperparameter a. Sending or - ac will act greedily with respect to the negative uncertainty and thereby exploit the underestimated uncertainty estimate, yielding an unsafe policy. Alternatively, sending a + 0 will be greedy with respect the estimated reward and yield different unsafe policy. This is especially troublesome in realistic datasets where we expect the behavior to already give us reasonable policy. In contrast, by defaulting to the behavior policy, deep-SPIBB can at least recover the performance of the behavior simply by tuning no matter the quality the uncertainty estimates
The benefits of deep-SPIBB. From this perspective, our deep-SPIBB algorithm has few benefits. First, unlike pessimism, SPIBB is able to remain robust to poor quality uncertainty estimates by defaulting the behavior policy. Second, unlike methods like BCQ and CQL that ignore state-based uncertainty, SPIBB can leverage uncertainty estimates when available.
Moreover, our generalized deep-SPIBB algorithm goes one step further by introducing a different default setting of the hyperparameters. When Etrain is set to O, then the evaluation step will learn Qo, as in one-step RI. (Brandfonbrener et al. 2021 Gulcehre al., 2021). Then as we vary Ceval from to infinity while fixing étrain = we interpolate between BC and argmax of Q3, the greedy one-step policy. Capturing this algorithm special case (while the other bascline algorithms do not) provides further illustration that generalized deep-SPIBB capturing a different tradcoff than the baseline algorithms.
EXPERIMENTS
We conduct an empirical analysis to evaluate how well deep-SPIBB incorporates uncertainty estimates and to compare SeeB-SPIBB to four baselines: BC, BCQ (Fujimoto et al. 2019b a pessimism
Note that there may also emrrs with large, figh-dimensionll state spaces, but finite action spaces. As a result, we expect it to be easier to solve the supervised learning problem of predicting action given state than to solve the uncertainty quantification problem of quantifying uncertainty state and action, which implicitly requires learning joint density model over and rather than just the model of conditioned
behavior estimate. However, here we are considering problems

---

Under review as a conference paper at ICLR 2023
(Buckman al., 2020 Jin al., 2021), and CQL (Kumar et al. 2020 We all of these algorithms along with Seep-SPIBB on a diverse variety of datasets generated in the classic Cartpole and Catch environments well as standard Atari benchmark datasets (Gulcehre et al. 2020). These experiments allow us to test how well deep-SPIBB is able to incorporate uncertainty across different domains from low-dimensional observation spaces (Cartpole) image-based observations with simple dynamics (Catch) to image-based observations with complex dynamics (Atari). Our main finding is that deep- SPIBB consistently outperforms pessimism, suggesting that it does better job of incorporating explicit uncertainty estimates. Our secondary finding is that when uncertainty is easier to estimate (as in Cartpole) deep-SPIBB substantially outperforms all the baselines, while in more challenging environments (as in Atari) it performs about the same the strongest baseline method, CQL, suggesting a robustness to poor uncertainty estimates.
Experimental setup. Following CQL (Kumar et al. (020), we build our deep-SPIBB algorithm and each of the baselines on top of QR RR-DON (Dabney et al. 20188) using JAX (Bradbury et al. 2018) and the Acme framework (Hoflman et al. 2020). For all experiments, we tune each algorithm (BCQ. CQL, pessimism, and Aeep-SPIBBB) across 4 values of the hyperparameters controlling the deviation from the behavior policy. All other training hyperparameters are held fixed. All algorithms have access to the same behavior and uncertainty estimates. Full details can be found in Appendix B
5.1 BSUITE ENVIRONMENTS
Datasets.
For our first of experiments we consider simple environments (cartpole and catch) from bsuite (Osband et al., 2019). In each environment we collect 5 different types of datasets with 10 seeds for each type of data (for total of 50 datasets per environment). Caripole has horizon and maximum return of 1000 and catch has horizon of 10 and returns bounded between -1 and 1. Datasets on cartpole have 20k transitions and datasets on eatch have 2k transitions. The five dataset types in each environment are collected by as follows: (1) med is data collected by a DQN agent trained a mixture of different policies each trained to medium performance, (3) uni is uniformly random policy, (4) uni_med is an equal mixture of data from a uniform policy and medium policy, (5) uni uni_exp is an equal mixture of data from uniform policy and an expert policy (trained for 500 episodes cartpole, 1000 episodes catch). We report mean and standard error across seeds. Results are shown in Figure
to medium performance (200 training episodes), (2) med_seed
Cartpole
700
0.0
500
BCQ
COL
400
0.00
Pessimismn
Deep SPIB
0.25
gen Deep- 5P188
100
-0.75
med
medd_ seed
UF1
uni medded
uni_exp
med
medd seee
UF1
151,
Figure 1: Final performance for OffRL agents trained on five dataset types across two environments. Error bars show standard error across ten seeds for dataset generation.
These results show deep-SPIBB to be consistently the top performer across the suite of experiments and to emphasize our two main findings. First, consider the comparison to pessimism. Across all ten datasets, generalized deep-SPIBB outperforms pessimism and standard deep-SPIBB outperforms pessimism on nine out ten datasets. The performance gap particularly large on Catch, where the based inputs and smaller dataset size make uncertainty quantification more challenging. By defaulting to the behavior policy instead the minimal uncertainty policy, deep- SPIBB is more robust poor uncertainty estimates, while still being able to leverage good uncertainty estimates in Cartpole
Results.
Second, consider the comparison to all baselines. Generalized deep-SPIBBBI is the top performer nine out of ten datasets, with the only exception being a slight underperformance relative to BCQ and COL the Catch med seed dataset. On Cartpole where the uncertainty estimation task is easier,
7

---

Under review as a conference paper at ICLR 2023
deep-SPIBB dramatically outperforms the uncertainty- BCQ and CQL methods. On Catch, where uncertainty estimation is more difficult, deep-SPIBB does not outperform BCQ and CQL, but is able to match their performance while pessimism struggles due to the difficulty of uncertainty estimation.
5.2 ATARI ENVIRONMENTS
Datasets. Next we run deep-SPIBB and our four baselines on the atari 1% benchmark (Agar- wal et al. 2020). Specifically, we use the data from RL Unplugged (Gulcehre et al.] 2020) subsampled down to 1% of the trajectories for each run and report mean and standard deviation across three seeds for each environment. Results shown in Figure
BC BCD CDL
Pessimiumm
Deep
Gen Deep-
Pong
Breakout
Seacuest
Qbert
Results. Again these experiments back up our two main findings. First, compared to pessimism, deep-SPIBB is dramatically better. While deep-SPIBB consistently outperforms BC recovers
Figure 2: Final performance for OffRL agents on atari % datasets. Results are normalized so that BC achieves score of and a randomly initial- ized policy achieves a score O. Error bars show standard deviation across three seeds for dataset generation.
substantially, pessimism never the performance of BC. Uncertainty estimation in Atari is very difficult and thus pessimism's
choice to default to the minimal uncertainty policy can cause serious issues seen here. In contrast, deep-SPIBB much more robust to poor uncertainty estimates. Second, compared to all baselines, deep-SPIBB generally performs slightly better than BCQ and is competitive with CQL. Improving performance of deep-SPIBB will likely require improved uncertainty quantification.
5.3 HYPERPARAMETER ABLATIONS
Seaquest
107
10"*
20'*
30t
10tt
10"
10"
20
Brabs opailon
Figure 3: Sweeps across étrain and ceval on Seaquest and Pong. Color indicates performance, darker is better. Generally, we find that lower étrain and higher eeval is beneficial. This setting of hyperparameters places us closer to one-step RL.
To validate the usefulness of generalized deep-SPIBB over deep-SPIBB with ézrain = Eeval. we conduct hyperparameter sweeps over the two different types of epsilon. Results are shown in Figure
Essentially, we find that it is often beneficial to set Ctrin substantially lower than Cevel. especially on more challenging environments. This is consistent with the observations of Nadjahi al. (2019): Brandfonnbener et al. 2021); Gulcehre et al. (202T) that the one-step algorithm that just performs one step of policy improvement often a very strong algorithm (generalized soft-SPIBB captures the one-step algorithm for étrain 0). Moreover, the more challenging environments and datasets likely yield worse uncertainty estimates, making it more risky to propagate values with low uncertainty since the uncertainty may be erroneously low. Thus, setting lower Ctrain learn more robust Q function, while allowing larger Geeal can yield improved performance.
The plots also show that the hyperparameters value of Goval is stable across different values of érrain This observation can allow for more efficient hyperparameter tuning by avoiding complete grid search.
somewhat independent in the sense that the optimal
8

---

Under review as a conference paper at ICLR 2023
5.4 UNCERTAINTY ABLATIONS
To validate our choices about how to parameterize the uncertainty estimates, we run deep-SPIBB wih several different uncertainty estimators. The most important decision in our uncertainty estimate is to use separate estimates of u(8) and {(a]s) to derive u(8, a), as described above. This choice is validated by the results in Figure
Essentially, we see major gains of our factored uncertatinty estimator over an uncertainty esti- mator trained on s and a jointly. Interestingly,
1000
500
just using the behavior component of the un- certainty estimator also outperforms the joint uncertianty estimator suggesting that the state- based part of the uncertainty is either not very important in this task or so poorly estimated in this high dimensional state space that it adds little to the performance.
600
700
Figure 4: Uncertainty ablations on Seaquest. "BC as unc" uses u(s, a)
Vilals) and "unc sa" trains uncertainty estimator that takes as in- put rather than using u(s) combined with ((a]s).
5.5 CONNECTION TO PRIOR RESULTS
This chunk of empirical analysis builds on the re- sults already reported in the SPIBB papers Nad-
jahi et al.] (2019) [Simão et al. (2020) where the pessimistic algorithm RaMDP was found to perform as well Soft-SPIB under two conditions: (i) the uncertainty estimates are well-estimated, either from count-based statistical concentration bounds such as Hoeffding's s inequality, or from handcrafted uncertainty measures based on states' similarity computed from their Euclidean distance in well- adapted environment, and (ii) the intrinsic/extrinsic reward balance is fine tuned, meaning that the pessimism hyperparameter setting appeared as more sensitive than Soft-SPIBB's
Our novel empirical results bring light on the robustness of these two approaches when the uncertainty estimator is more brittle. In Cartpole, with its oiw-dmensional observation space, find that both are able to take a significant advantage over the uncertainty-fi methods. In Catch where the state representation more image-like, we see that pessimism's 's performance immediately crashes, while deep-SPIBB is more robust to these gess-than-perfee uncertainty estimates and remains on-par with BCQ and CQL. Finally, in the Atari environments where the state a complex image, pessimism is consistently and significantly worse than behavior cloning, which it cannot even fall back on.
This set experiments with increasing difficulty the uncertainty estimation (and the RL task) tells us that uncertainty-base -based algorithms are better than uncertainty- free algorithms when the uncertainty estimates are reliable but that out of the uncertainty-base based algorithms, only deep- is robust to bad estimates. While deep-SPIBB does not surpass CQL in hard environments for now, there is hope that advances in neural uncertainty estimation will allow it to so in the future.
DISCUSSION
Here we have introduced the deep-SPIBB algorithm for inclorporating explicit uncertainty estimates into deep offine RI. We have seen that the deep-SPIBB mechanism of incorporating uncertainty into the policy improvement step is more performant and robust than the pessimism mechanism of incorportating uncertainty as a penalty in the evaluation step. When the uncertainty estimates are good, deep-SPIBB also improves over the uncertainty-fre baselines of BCQ and CQL.
Our work does have few limitations that are worth mentioning to inspire future work in these directions. First, it is not clear how to extend the deep-SPIBB algorithm to continuous action spaces. Second, like most algorithmic work in offline RL, deep-SPIBB has important hyperparameters (etrain and feval) that govern the policy constraints. How to practically tune hyperparameters like these offline without interacting with the environment remains an open challenge, although recent work makes some progress (Paine et al. 2020 Zhang & Jiang 2021). Finally, perhaps the most interesting direction for future work based deep-SPIBB is to improve the uncertainty estimators. Our results suggest that access to better uncertainty estimators in challenging domains like Atari could dramatically improve deep- SPIBB over the uncertainty-fret -free baselines.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, mir
pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL wwt.httpp/p sreess/37/blundelle html
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-MMlne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github.
David Brandfonbrener, William Whitney Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. arXiv preprint arXiv:2106.08909 2021.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019. URL https:
pettps://openreeeee http://pppeeeeee/ee//e/eeee wwtpp//wwwww...eaeeoorrraaaaaaaa Hoteee/ooooum...HHUSUT
net wwwp:eeeeeffforeee
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conserva- tive uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2019.
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. arXiv preprint arXiv: 106.06860, 2021.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint 1910.0779 2019a
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062 PMLR, 2019b.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances Neural Information Processing Systems, 33:7248-7258 2020.
Caglar Gulcehre, Sergio Gómez Colmenarejo. Ziyu Wang. Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. arXiv preprint 103.:2203300900 2021.
10

---

Under review as a conference paper at ICLR 2023
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXi::200..00979, 2020. URL mnttps://arxvv..t 020///006.009733
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 11820, 2020.
Ila Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint ar 10.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint Xxix:20066047799 2020.
Balaji Lakshminarayanan A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
Romain Laroche and Rémi Tachet des Combes. SPIBB-DQN: Safe batch reinforcement learning with function approximation. In Proceedings of the 4th Reinforcement Learning and Decision Making (RLDM), 2019.
Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Icarning: Tutorial, review, and perspectives open problems, arXiv preprint arXiv:2005.0 2020.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint arXiv:2007.08200 08200,, 2020.
Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In Proceedings of the 17th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databuses (ECML-PKDD), 2019.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv: 1.0.iv:8006.00 180..0333. 2018
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari. Satinder Singh, al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: / 03568, 2019.
Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, and Benjamin Van Roy. Epistemic neural networks. arXiv preprint arXiv:2107.0899 2021.
Georg Ostrovski, Mare G. Bellemare, Aäron den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 2721-2730, 2017.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arxiv:2007.00055 2020.
Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Proceedings of the 29th Advances in Neural Information Processing Systems (NIPS), 2016.
11

---

Under review as a conference paper at ICLR 2023
Bruno Scherrer, Victor Gabillon, Mohammad Ghavamzadeh, and Matthieu Geist. Approximate modified policy iteration. arXiv preprint arXiv: 1205.30003 2012.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprini arXiv:2002.0883 2020.
Thiago D. Simão, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with estimated baseline bootstrapping. In Proceedings of the 19th International Conference on Autonomous Agents and Mulli-Agent Systems (AAMAS, in review), 2020.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 200::2000.113444 15134, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.11360 2019.
Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34, 2021.
Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems, 34, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information 2020.
Processing Systems,
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
Siyuan Zhang and Nan Jiang. Towards hyperparameter-fiee policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Under review as a conference paper at ICLR 2023
A UNCERTAINTY ESTIMATOR
We take an approach based off Bayesian ensembling with prior network from Ciosek et al. (2019). While that work focused on estimating uncertainty supervised learning problems, use the same technique with no labels to get a measure of state-based uncertainty. Explicitly, we create an ensemble of models that each take a state s and output an M M-dimensional vector. Each component of the ensemble has corresponding random prior function defined by randomly initialized network. The fi networks are then trained to predict difference between the prior and some target (in our case, the target is gaussian noise 6j sampled from N oe) for each datapoint i). State-based uncertainty then calculated as
a(s) V ()(s)) + (s),
(9)
1
1 1/ -1 p.(())12)
2
(s)
[[f(()) (s)),
(5(s)
MB
B
M
B EXPERIMENTAL DETAILS
Hyperparameters. First we will provide all of the hyperparameters used in the various steps of our training algorithms. Every algorithm is trained with the Adam optimizer.
Table Hyperparameters for behavior estimation in cartpole and catch
Hyperparameter
Value
Training steps
le4
Learning rate
le
Batch size
256
ML.P width
64
MLP depth
2
Prior MLP depth
1
Table 4: Hyperparameters for behavior estimation in Atari
Hyperparameter
Value le5
Training steps
Learning rate
le - 4
Batch size
256
Network Architecture
DQN
Table 5: Hyperparameters for uncertainty estimation in cartpole and catch
Hyperparameter
Value
Training steps
lc4
Learning rate
le
Batch size
256
MLP width
256
MLP depth
2
M
64
1.0
a
0.1
13

---

Under review as a conference paper at ICLR 2023
Table 6: Hyperparameters for uncertainty estimation in Atari
Hyperparameter
Value 1e5
Training steps
Learning rate
le
Batch size
256
Network architecture
DQN
Prior network architecture
DQN 1 dense layer
M
64
B
5
a
1.0
o,
0.1
Table 7: Shared hyperparameters for RL in cartpole and catch
Hyperparameter
Value 1e5
Training steps
Learning rate
3 - 5
Batch size
256
Target update period
1000
MLP width
256
MIP depth
2
QR quantiles
201
Discount
0.99
Table 8: Shared hyperparameters for RI. in Atari
Hyperparameter
Value
Training steps
1e6
Learning rate
3
Batch size
256
Target update period
2500
Network architecture
DQN
QR quantiles Discount
201
0.99
Each RL algorithm also has specific hyperparameters. For each algorithm choose four values of the hyperparameter. We should note that for the deep-SPIBB and pessimism that rely on our learned uncertainty estimates,
normalize the values of and respectively to the scale of the uncertainty estimator. We estimate the scale of the uncertainty function by just evaluating the mean of the uncertainty function on batch of data from the training set.
Table 9: Aggortthm-specin hyperparameters for cartpole and catch
Algorithm
Hyperparameter
Value
BCQ
T
[0.01, 0.03, 0.1, 0.3]
Pessimism
a
[0.3, 1.0, 3.0, 10.0]
CQL
a Etrain
10.0]
deep-SPIBB
[0.1, 0.3, 3.0]
gen deep-SPIBB
Geval
[0.001, 0.01, 0.1, 1.0]
14

---

Under review as a conference paper at ICLR 2023
Table 10: Algorithm-specifi hyperparameters for Atari
Algorithm
Hyperparameter
Value
BCQ
[0.01, 0.03, 0.1, 0.3] [0.1, 1.0, 10.0, 100.0] [0.3, 1.0, 10.0] [0.001, 0.01, 0.03, 0.1]
Pessimism
CQL
deep-SPIBB
Etrain
gen decp-SPIBB
Cevol
[0.0001, 0.001, 0.01, 0.1]
Evaluation. For evaluation we run 50 episodes of each trained RL model and take the mean. Plots in the text then report the mean and standard deviation of this mean value across training seeds. We report the results for the best performing hyperparameter out of those in the table for each algorithm.
Compute. All models are trained on various types of GPU on an internal cluster. Each run for cartpole/catch takes less than 15 minutes and each run Atari takes less than day.
For completeness, we also report the licenses of the assets that we used in the paper: JAX Bradbury et (2018): Apache-2.0. Acme Hoffman al. (2020): Apache-2.0, 2.0, RL-unplugged Gulcehre et al. (2020): Apache-2.0. 2.0,, bsuite Osband et al. (2019): Apache-2.0.
Asset licenses.
15


Reviews:
[{"review_id": "uVhyHO6SDth", "reviewer": "Reviewer_hNm1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new algorithm for offline RL,deep-SPIBB, that uses uncertainty estimation instead of pessimism for regularizing the behavior to prevent undesirable behaviors. The method uses an ensemble based uncertainty estimation on the state and MLE behavior policy estimate to estimate the uncertainty for a given state-action pair and using the uncertainty, the algorithm performs standard offline RL procedure with constraints that force the learned policy to not deviate from the MLE policy when there is high uncertainty. The algorithm also introduces different deviation thresholds for the policy improvement and policy evaluation step. Experimentally, the algorithm performs prior work on two simple environments and outperforms pessimism and is competitive with CQL on  Atari.", "summary_of_the_review": "The paper is well written but is limited in significance and novelty. I am leaning towards reject.", "strength_and_weaknesses": "### Strength\n1. The proposed algorithm is very sensible and using uncertainty in offline RL seems very promising\n2. The analysis of how different offline RL algorithms use uncertainty is very nice as it unifies different perspective\n3. The paper overall is relatively easy to follow\n\n### Weakness\n1. The performance of the algorithm is not extremely compelling as it underperforms CQL in most settings.\n2. The previous point would normally not be a big problem but the idea of using uncertainty for offline RL has already been proposed in the literature [1]. This work does not discuss [1] and furthermore, [1] seems to outperform CQL on a range of D4RL tasks. This casts doubt on the additive value of this work.\n3. The ablation and analysis of different types of uncertainty could be improved. Currently, only two short paragraphs are dedicated to them, but it seems like uncertainty is extremely important for the proposed method and the proposed way of computing is new, so I believe more thorough analysis is warranted. For example, if we really believe that the quality of uncertainty estimation is the problem for the performance of the algorithm, then one could do an ablation on increasing the size of the ensemble and check if the performance improves.\n\n**Reference**\n\n[1] Why so pessimistic? Estimating uncertainties for offline RL through ensembles, and why their independence matters. Ghasemipour, et al.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7QAig9u_vZ_", "reviewer": "Reviewer_eF2b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to incorporate uncertainty estimations for offline reinforcement learning to prevent the learner from favoring regions of high uncertainty (which are often over-estimated). The authors extend safe policy improvement with soft baseline bootstrapping (soft-SPIBB) to large state-action space, where count-based uncertainty measures become infeasible, by leveraging a neural uncertainty estimator (using random priors). ", "summary_of_the_review": "Although the proposed algorithm is well-motivated (upon some existing works), its novelty is limited, and the discussions and experiments could use some improvements. I am leaning toward a (weak) reject at this moment.", "strength_and_weaknesses": "Pros:\n\n- The authors extend soft-SPIBB to large state-action space and the empirical performance seemed comparable to CQL. \n\n- The design of the estimator $\\hat{u}(s, a) := \\hat{u}(s) / \\sqrt{\\hat{\\beta}(a|s)}$ is sensible.\n\n- The discussion about constrained improvement step versus penalized evaluation step is interesting. \n\nCons:\n\nLimited novelty:\n\n- The authors modified soft-SPIBB (Nadjahi et al., 2019)  by replacing (count-based) error functions $e(s, a)$ with neural uncertainty estimates $\\hat{u}(s, a)$.\n\n- Estimating state uncertainty via random priors is not new in reinforcement learning, e.g. RND (Burda et al., 2019).\n\nRelated works: prior works should be more explicitly explained \n\n- RND is only mentioned in \"there is a large literature from the deep learning community on uncertainty quantification that we can leverage for OffRL (... Burda et al., 2019; ...)\", which does not emphasize that random priors methods have already been applied in reinforcement learning literature, although in an online setting.\n\n- It is encouraged to briefly explain the essence of SPIBB and soft-SPIBB in e.g. preliminaries, so that it is easier to see which parts in Eqn (7) are proposed by the authors, especially for the audiences who are not familiar with soft-SPIBB.\n\n\nQuestions/additional comments:\n\n- Introducing $\\epsilon_{eval}$ seemed a bit odd to me. Do the authors mean: (a) first train $\\hat{Q}$ until convergence with $\\epsilon_{train}$ and (b) then do one-step policy improvement with $\\epsilon_{test}$?\n\n- Could the authors plot the training curves of experiments in 5.1 and 5.2? As Eqn (7) is now approximated by a greedy heuristic (if my understanding is correct), it is good to see whether it is requiring more iterations to converge.\n\n- The comparison versus Pessimism is not necessarily fair. Deep-SPIBB keeps most essential parts of soft-SPIBB except using a neural quantifier instead of a count-based one. While Pessimism does not keep all key designs of e.g.  (Buckman et al., 2020; Jin et al., 2021), Pessimism should not recover theoretical guarantees of pessimistic approaches while using any count-based uncertainty quantifier because of its simplification. Therefore it does not fully support the claim made in the abstract, \"we argue that the SPIBB mechanism for incorporating uncertainty is more robust and flexible than pessimistic approaches that incorporate the uncertainty as a value function penalty.\"\n\n- I believe pessimistic approaches should be able to recover behavior cloning with proper choice of uncertainty penalty and also proper algorithmic designs. For example, see [1].\n\n[1] Rashidinejad, Paria, et al. \"Bridging offline reinforcement learning and imitation learning: A tale of pessimism.\" Advances in Neural Information Processing Systems 34 (2021): 11702-11716.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "21pDJPvfW9", "reviewer": "Reviewer_uHxA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to enforce $\\pi(a|s)$ close $\\beta(a|s)$ to incorporate uncertainty at each state-action pair $(s,a)$ in enforcing $\\pi(a|s)$ close the estimated behavior $\\hat{\\beta}(a|s)$ where the enforcement weight is controlled the uncertainty estimate $\\hat{u}(s,a)$ (eq. 7). To compute uncertainty estimate $\\hat{u}(s,a)$, they also incorporate estimated behavior policy as $\\hat{u}(s,a) = \\hat{u}(s) / \\hat{\\beta}(a|s)$ where $\\hat{u}(s)$ is the ensemble-based estimator at state $s$. They claim that this proposed method is “more performant and robust than the pessimism mechanism [they meant LCB] of incorportating uncertainty as a penalty in the evaluation step”.", "summary_of_the_review": "See the weaknesses and question sections. ", "strength_and_weaknesses": "**Strength**\n\n- Good classification of recent offline RL methods\n\n- the paper is easy to read \n\n**Weaknesses**\n\n- Incremental methods with weak empirical results: the most interesting part is perhaps using the estimated behavior in the uncertainty quantifier. However, simple experiments in RL Unplugged cannot support the benefit of this modification. \n\n- Justification and comparison with LCB is unfair: the extreme hyperparameter analysis is not useful and the paper missed literature of ensemble-based implementation of LCB. The paper also does not clearly explain how LCB is implemented in their baseline\n\n- Significance of the work is low: Given the above reasons. \n\nSee my questions below for details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject