Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
Authors: Chenjia Bai
Lingxiao Wang
Zhuoran Yang
Zhi-Hong Deng
Animesh Garg
Peng Liu
Zhaoran Wang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Pessimistic Bootstrapping
Bootstrapped Q-functions
Uncertainty Estimation
Offline Reinforcement Learning

Abstract: Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. Previous methods tackle such problem by penalizing the Q-values of OOD actions or constraining the trained policy to be close to the behavior policy. Nevertheless, such methods typically prevent the generalization of value functions beyond the offline data and also lack precise characterization of OOD data. In this paper, we propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints. Specifically, PBRL conducts uncertainty quantification via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty. To tackle the extrapolating error, we further propose a novel OOD sampling method. We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL. Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art algorithms.

Published as a conference paper at ICLR 2022
PESSIMISTIC BOOTSTRAPPING FOR UNCERTAINTY- DRIVEN OFFLINE REINFORCEMENT LEARNING
Chenjia Bai
I Lingxiao Wang Northwestern University
Zhuoran Yang
Harbin Institute of Technology baichen fi2255 gmai comm
Princeton University
Zhihong Deng
Animesh Garg University of Toronto Vector Institute, NVIDIA
Peng Liu Harbin Institute of Technology
University of Technology Sydney
Zhaoran Wang
Northwestern University
ABSTRACT
Offline Reinforcement Learning (RL) aims to learn policies from previously col- lected datasets without exploring the environment. Directly applying off-policy algorithms offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. Previous methods tackle such problems by penalizing the Q-values of OOD actions or constraining the trained policy be close to the behavior policy. Nevertheless, such methods typically prevent the generalization of value functions beyond the offline data and also lack a precise characterization of OOD data. In this paper, we propose Pessimistic Bootstrap- ping for offline RI. (PBRL), purely uncertainty-driven offline algorithm without explicit policy constraints. Specifically, PBRL. conducts uncertainty quantifica- tion via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty. To tackle the extrapolating error, we further propose a novel OOD sampling method. We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL. Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art al algorithms.
INTRODUCTION
Deep Reinforcement Learning (DRL) (Sutton & Barto 2018) achieves remarkable success in a va- riety of tasks. However, in most successful applications, DRL requires millions of interactions with the environment. In real-world applications such as navigation (Mirowski et al. 2018) and health- care et al.) 2019), acquiring a large number of samples by following a possibly suboptimal policy can be costly and dangerous. Alternatively, practitioners seek to develop RL algorithms that learn policy based solely on an offline dataset, where the dataset is typically available. However, di- rectly adopting online DRL algorithms to the offline setting is problematic. On the one hand, policy evaluation becomes challenging since no interaction allowed, which limits the usage of on poloccc algorithms. On the other hand, although it is possible to slightly modify the off-policy value-based algorithms and sample solely from the offline dataset training, such modification typically suffers from significant performance drop compared with their online learning counterpart (Levine et al.) 2020). An important reason for such performance drop is the so-called distributional shift. Specifi- cally, the offline dataset follows the visitation distribution of the behavior policies. Thus, estimating the Q-functions of the corresponding greedy policy with the offline dataset is biased due to the dif- ference in visitation distribution. Such bias typically leads to significant extrapolation error for DRL algorithms since the estimated Q-function tends to overestimate the out-of di-iittribttion (OOD) actions (Fujimoto et al. 2019).

---

Published as a conference paper at ICLR 2022
To tackle the distributional shift issue in offline RL, previous successful approaches typically fall into two categories, namely, policy constraints (Kumar et al. 2019; Fujimoto & Gu Gu|20211 (021] and con- servative methods (Kumar et al.) 2020: Yu et al. 2020; 2021). Policy constraints aim to restrict the learned policy to be close to the behavior policy, thus reducing the extrapolation error in pol- icy evaluation. Conservative methods seek penalize the Q-functions for OOD actions in policy evaluation and hinge on gap-expanding property to regularize the OOD behavior. Nevertheless, since policy constraints explicitly confine the policy to be close to the behavior policy, such method tends to be easily affected by the non-optimal behavior policy. Meanwhile, although the conserva- tive algorithms such as CQL (Kumar et al. (020) do not require policy constraints, CQL equally penalizes the OOD actions and lacks a precise characterization of the OOD data, which can lead to overly conservative value functions. To obtain more refined characterization of the OOD data, uncertainty quantification is shown to be effective when associated with the model-based approach (Yu et al. 2020 Kidambi et al.) 2020), where the dynamics model can be learned in static data thus providing more stable uncertainty in policy evaluation. Nevertheless, model-based methods need additional modules and may fail when the environment becomes high-dimensional and noisy. In addition, the uncertainty quantification for model-free RL is more challenging since the Q-function and uncertainty quantifier need to be learned simultaneously (Yu et al. 2021
To this end, we propose Pessimistic Bootstrapping for offline RL (PBRL), an uncertainty model-free algorithm for offline RL. To acquire reliable Q-function estimates and their correspond- ing uncertainty quantification, two components of PBRL play a central role, namely. bootstrapping and OOD sampling. Specifically, we adopt bootstrapped Q-functions (Osband et al., 2016) for un- certainty quantification. We then perform pessimistic Q-updates by using such quantification as a penalization. Nevertheless, solely adopting such penalization based on uncertainty quantification is neither surprising nor effective. We observe that training the Q-functions based solely on the offline dataset does not regularize the OOD behavior of the Q-functions and suffers from the extrapolation error. To this end, we propose novel OOD sampling technique as regularizer of the learned Q- functions. Specifically, we introduce additional OOD datapoints into the training buffer. The OOD datapoint consists of states sampled from the training buffer, the corresponding OOD actions sam- pled from the eurrent policy, and the corresponding OOD target based the estimated Q-function and uncertainty quantification. We highlight that having such OOD samples in the training buffer plays an important role in both the Q-function estimation and the uncertainty quantification. We remark that, OOD sampling controls the OOD behavior in training, which guarantees the stability of the trained bootstrapped Q-functions. We further show that under some regularity conditions, such OOD sampling is provably efficient under the linear MDP assumptions.
We highlight that PBRL exploits the OOD state-action pairs by casting a more refined penalization over OOD data points, allowing PBRL to acquire better empirical performance than the policy- constraint and conservatism baselines. As an example, if an action lies close to the support of offline data but is not contained in the offline dataset, the conservatism (Kumar et al., (020) and policy constraint (Fujimoto et al. 2019) methods tend to avoid selecting it. In contrast, PBRL tends to assign a small Q-penalty for such an action as the underlying epistemic uncertainty is small. Hence, the agent trained with PBRL has a higher chance consider such actions the corresponding value estimate is high, yielding better performance than the policy constraint and conservatism baselines. Our experiments on the D4RL benchmark (Fu et al. 2020) show that PBRL provides reasonable uncertainty quantification and yields better performance compared to the state-of-the-art algorithms.
2 PRELIMINARIES
We consider an episodic MDP defined by the tuple (S, A, P.). where S is the state space, A is the action space, T E N is the length of episodes. distribution. The goal of RL. is to find policy л that maximizes the expected cumulative rewards E E T-1 7'rz]. where 7 E [0, 1) is the discount factor in episodic settings. The corresponding Q Q-function of the optimal policy satisfies the following Bellman operator,
the reward function, and IP is the transition
TQo(s, a) == r(s, a) + (Es
max Q8- (s', a'
where 0 is the parameters of Q-network. In DRL, the Q-value is updated by minimizing the TD- error, namely B(s.,,,./)) [(Qu.r.a))(QQ l(Q-((Q))))) T. Empirically, the target TQ is typically calculated by separate target-network parameterized by 0 without gradient propagation (Miih al. 2015). In online RI.,
2

---

Published as a conference paper at ICLR 2022
one typically samples the transitions (s, a. s') through iteratively interacting with the environment. The Q-network is then trained by sampling from the collected transitions.
In contrast, in offline RL, the agent is not allowed to interact with the environment. The experiences are sampled from an offline dataset Pm = { ai ri, s 1 Naive off-policy methods such as Q-learning suffer from the distributional shift, which is caused by different visitation distribution of the behavior policy and the learned policy. Specifically, the greedy action a' chosen by the target Q-network in s' can be an OOD-action since a') is scarcely covered by the dateset Din. Thus, the value functions evaluated on such OOD actions typically suffer from significant extrapolation errors. Such errors can be further amplified through propagation and potentially diverges during training. We tackle such a challenge by uncertainty quantification and OOD sampling.
3 PESSIMISTIC BOOTSTRAPPING FOR OFFLINE RL
3.1 UNCERTAINTY QUANTIFICATION WITH BOOTSTRAPPING
In PBRL., we maintain K bootstrapped Q-functions in critic to quantify the epistemic uncertainty. Formally, we denote by Q the k-th Q-function in the ensemble. Q6 is updated by fitting the following target
(Q) = ==((s, + + (0,
(2)
Here we denote the empirical Bellman operator by T. which estimates the expectation E[Q)- a'', 8, a] empirically based on the offline dataset. We adopt such an ensemble technique from Bootstrapped DQN (Osband et al.] (016), which is initially proposed for the online exploration task. Intuitively, the ensemble forms an estimation of the posterior distribution of the estimated Q- functions, which yields similar value on areas with rich data and diversely on areas with scarce data. Thus, the deviation among the bootstrapped Q Q-functions yields epistemic uncertainty estimation, which we aim to utilize as a penalization in estimating the Q- Q-functions. Specifically, we introduce the following uncertainty quantification (((ss) a) based the Q-functions {Q*) ]NE[K]
U(s, a) = Std(("(s, - K Σx K 1 (P(s.))-Q(s.a))
(3)
Here we denote by Q the mean among the ensemble of Q-functions. From the Bayesian perspective, such uncertainty quantification yields an estimation of the standard deviation of the posterior of Q- functions. To better understand the effectiveness of such uncertainty quantification, we illustrate with a simple prediction task. We use 10 neural networks with identical architecture and different initialization as the ensemble. We then train the ensemble with 60 datapoints in R²² plane, where the covariate r is generated from the standard Gaussian distribution, and the response is obtained by feeding x into randomly generated neural network. We plot the datapoints for training and the uncertainty quantification in Fig. I(a) As shown in the figure, the uncertainty quantification rises smoothly from the in-distribution datapoints the OOD datapoints.
In offline RL, we perform regression (s,a) + iQ' a) in Din to train the bootstrapped Q- functions, which is similar to regress + in the illustrative task. The uncertainty quantification allows us to quantify the deviation of datapoint from the offline dataset, which provides more refined conservatism compared to the previous methods (Kumar et al. 2020 Wu et al. 2019)
3.2 PESSIMISTIC LEARNING
We now introduce the pessimistic value iteration based on the bootstrapped uncertainty quantifica- tion. The idea is to penalize the Q-functions based on the uncertainty quantification. To this end, we propose the following target for sate-actionn pairs sampled from Diu.
7'n ( a) == r(s,a) +
-
@2 (s',a') Bm Uo (s', a')
(4)
where Mg- (s', aa) is the uncertainty estimation at (s', a') based on the target network, and 8m, is tuning parameter. In addition, the empirical mean B. P(l..e())) a) is obtained by sampling the
3

---

Published as a conference paper at ICLR 2022
1.0
Target Q-net
r.d
TQ
0.6
Offline Dataset
Target Qunet
a'
0.4
u
%
0.2
Target Qunel
Pessimistic 7
Gradictie
1.0
PBRI
Main Q-aets
Q+
Mibere (n)
offline data
0.8
PARL for
0OD data
OOD compling
Main set
0.4
Main Q-net
Q
Tibemrrr (OOD)
0.2
Lood
Pessimisti frnnd D
Main Q- K
(a) Uncertainty
(b) PBRL: overall architecture
Figure 1: (a) Illustration of the uncertainty estimations in the regression task. The white dots rep- resent data points, and the color scale represents the bootstrapped -aappppddduueeraittyy values in the whole input space. (b) Illustration of the workflow of PBRL. PBRL. splits the loss function into com- ponents. The TD-error (in) represents the regular TD-error for in-distribution data (i.e., from the offline dataset), and pseudo TD-error (ood) represent the loss function for OOD actions. In the update of Q-functions, both losses are computed and summed up for the gradient update.
transition (s, a, s') from Di and further sampling a' ~ T(. s') from the current policy т. We denote by 2 (s, a) the in-distribution target of Q (8, a) and write Fin to distinguish the in-distribution target from that of the OOD larget, which we introduce in the sequel.
We remark that there are two options to penalize the Q-functions through the operator iin In Eq. (4), we penalize the next-Q value Qh (8',') a') with the corresponding uncertainty Ug- (s', a'). Alternatively, we can also penalize the immediate reward by f(s, a) == r(s, a) - Ldss, a) and use f(s, a) in place of rss, a) for the target. Nevertheless, since the datapoint (s, E Dmn lies rich- data areas, the penalization Ug (s, a) on the immediate reward is usually very small thus having less effect in training. In PBRL, we penalize the uncertainty of (s', a') in the next-(? value.
Nevertheless, our empirical findings in $D suggest that solely penalizing the uncertainty for in- distribution target is insufficient to control the OOD performance of the fitted Q-functions. To enforce direct regularization over the OOD actions, we incorporate the OOD datapoints directly in training and sample OOD data of the form (oood a Doodd- Specifically, we sample OOD) states from the in-distribution dataset Din- Correspondingly, we sample OOD actions a ood by following the current policy 7( good) We highlight that such OOD sampling requires only the offline dataset Din and does not require additional generative models or access to the simulator.
It remains to design the target for OOD samples. Since the transition P(- | (ood) and reward r( r(send a add) are unknown, the true target of OOD sample is inapplicable. In PBRL, we propose a novel assuddotarged for the OOD datapoints,
gooo Qu ood "araa,nn god) = ( and aod) - BBod Ug (sood gooo))
(ood)
(5)
which introduces an additional uncertainty penalization (g gody to enforce pessimistic Q- function estimation, and Food is a tuning parameter. For OOD samples that are close to the in- distribution data, such penalization is small and the OOD target close to the Q-function estima- tion. In contrast, for OOD samples that are distant away from the in-distribution dataset, a larger penalization is incorporated into the OOD target. In our implementation. we introduce an addi- tional truncation to stabilize the early stage training max Tood ( ( ood)). In addition, we remark that Bun is important to the empirical performance. Specifically,
At the beginning of training, both the Q-functions and the corresponding uncertainty quantifi- cations are inaccurate. We use a large Bood to enforce a strong regularization on OOD actions.
We then gradually decrease Food in the training process since the value estimation and uncer- tainty quantification becomes more accurate in training. We remark that a smaller Bood requires more accurate uncertainty estimate for the pessimistic target estimation 7ood Mit In addition, a decaying parameter Bood stabilizes the convergence of Q and " gody in the training from the empirical perspective.
4

---

Published as a conference paper at ICLR 2022
Incorporating both the in-distribution target and OOD target, conclude the loss function for critic in PBRL as follows,
(6) where we iteratively minimize the regular TD-erro for the offline data and the pseudo TD-error for the OOD data. Incorporated with OOD sampling, PBRL obtains a smooth and pessimistic value function by reducing the extrapolation error caused by high-uncertain state-action pairs.
Ceitiie ((s.,,))) ( (7 - 24) + E.ond Dudd [TFood Q*)
Based on the pessimistic Q-functions, we obtain the corresponding policy by solving the following maximization problem.
min 2 (s.aa). T max E Ba k=l.....K K
where yp is the policy parameters. Here we follow the previous actor-critic methods (Haamnoj al.) 2018 Fujimoto et al. (2018) and take the minimum among ensemble Q-functions, which stablizes the training of policy network. We illustrate the overall architecture PBRL. in Fig. T(b)
3.3 THEORETICAL CONNECTIONS TO LCB-PENALTY
In this section, show that the pessimistie target in PBRL aligns closely with the recent theoretical investigation on offline RI. (Jin et al. 2021 Xie et al. 2021a). From the theoretical perspective, an appropriate uncertainty quantification is essential to the provable efficiency in offline RI.. Specifi-
cally, the ξ-uncertainty quantifier plays a central role in the analysis of both online and offline RL Jaksch et al. 2010 Azar et al. 2017 Wang al. 2020a Jin al., 2002 2021 Xie al. 2021a ().
Definition (5-Uncertainty Quantifier (Jin et al. 2021)). The set of penalization {I' (er forms
E-Uncertainty Quantifier if it holds with probability at least 1 1 that
(Tv+118, - IV++ S (T(s,a)
for all (s) 8 x A, where T is the Bellman equation and T is the empirical Bellman equation that estimates T based on the offline data.
In linear MDPs (Jin et al. 2020 Wang et al. 2020a Jin et al. (021) where the transition kernel and reward function are assumed to be linear to the state-action representation á S x A + RR The following LCB- Beeenatlyy (Abbasi- Yadkori al. 2011 Jin et al. 2020 is known be a
a É-uncertainty
quantifier for appropriately selected [ptheT]T)
pleb ae) Be [ at) TA, A, "()s,
(8)
where A, = Em dis), (s) TT) X accumulates the features of state-action pairs in Di, and plays the role of pseudo-count intuitively. We remark that under such linear MDP assump- tions, the penalty proposed in PBRL and [ pleb(se, az) in linear MDPs equivalent under a Bayesian perspective. Specifically, we make the following claim.
Claim 1. In linear MDPs, the proposed bootstrapped uncertainty Br -st, ae) is an estimation to the LCB- LCB-penalty Ilob a:) in Eq. (8) for appropriately selected tuning parameter Be.
We refer to | for a detailed explanation and proof. Intuitively, the bootstrapped Q-functions esti- mates non- non-parmmetric Q-posterior (Osband al. 2016 2018a). Correspondingly, the uncertainty quantifier (s;, a,) estimates the standard deviation of the Q Q-posterior, which scales with the LCB- penalty in linear MDPs. As an example, we show that under the tabular setting, rich(s,, az) is approximately proportional to the reciprocal pseudo-count of the corresponding state-action pair in the dataset (See Lemma in { offline RI, such uncertainty quantification measures how trust- worthy the value estimations state-action pairs are. A low LCB- LCB-penalty (or high pseudo-count) indicates that the corresponding state-action pair aligns with the support of offline data.
Under the linear MDP or Bellman-consisten assumptions, penalizing the estimated value function based on the uncertainty quantification is known to yield an efficient offline RL algorithm (Jin et al. 2021 Xie et al. 2021a b). However, due to the large extrapolation error of neural networks, find that solely penalizing the value function of the in-distribution samples insufficient to regularize the fitted value functions of OOD state-action pairs.
5

---

Published as a conference paper at ICLR 2022
A key to the success of linear MDP algorithms (Jin et al. 2020: Wang al.) 2020a Jin et al. 2021 is the extrapolation ability through Lg-regularization in the least-squares value iteration (LSVI), which guarantees that the linear parameterized value functions behave reasonably on OOD state- action pairs. From a Bayesian perspective, such Izz-reuularization enforces a Gaussian prior on the estimated parameter of linear approximations, which regularizes the value function estimation on OOD state-action pairs with limited data available. Nevertheless, our empirical study in $D shows that Lz-regularization not sufficient to regularize the OOD behavior of neural networks.
To this end, PBRL introduces a direct regularization over an OOD dataset. From the theoretical perspective, we observe that adding OOD datapoint (s annd y) into the offline dataset leads to an equivalent regularization to the L2-regularization under the linear MDP assumption. Specifically, in linear MDPs, such additional OOD sampling yields covariate matrix of the following form,
A > i=1 His) (())) a:) a
+ E (sood and D
((5 a ( T
ddo)) ( T plays the role of the X X-I prior where the matrix Anod Essodd D)ED (8 aood)
in LSVI. It remains to design a proper target y in the OOD datapoint (sond and ,y). The following theorem show that setting y TVn+ ( aaud) leads valid Ş-uncertainty quantifier under the linear MDP assumption.
Theorem 1. Let Aood - X 1. Under the linear MDP assumption, for all the OOD datapoint (Sood good y/) E Dood, we set y TV ) it then holds for O vdd (dd-)g((//E)) that (s a.) = (o(),,, T AA," A (s(s, at)] forms valid f-uncertainty quantifier.
We refer to $ for detailed discussion and proof. Theorem shows that if we set TV (sood, a the bootstrapped uncertainty based on disagreement among ensembles is valid f-uncertainty quantifier. However, such an OOD target is impossible to obtain in practice as it re- quires knowing the transition at the OOD datapoint (sand a.mi). ). In practice, if TD error is sufficiently minimized, then ((xood a should well estimate the target TV Thus, in PBRI,, we utilize
y = Q(s _and)) -
(1od aoo))
(10)
as the OOD target, where we introduce additional penalization pleb aood) to enforce pes- simism. In addition, we remark that in theory, we require that the embeddings of the OOD sample are isotropic in the sense that the eigenvalues of the corresponding covariate matrix Aood are lower bounded. Such isotropic property can be guaranteed by randomly generating states and actions. In practice, we find that randomly generating states more expensive than randomly generating ac- tions. Meanwhile, we observe that randomly generating actions alone are sufficient to guarantee reasonable empirical performance since the generated OOD embeddings are sufficiently isotropic. Thus, in our experiments, we randomly generate OOD actions according to our current policy and sample OOD states from the in-distribution dataset.
RELATED WORKS
Previous model-free offline RI. algrrithms typically rely on policy constraints restrict the learned policy from producing the OOD actions. In particular, previous works add behavior cloning (BC)
loss in policy training (Fujimoto et al.] 2019: Fujimoto & Gul [202]] Ghasemipour al. 1021)
measure the divergence between the behavior policy and the learned policy (Kumar et al., 2019
Wu et al. 2019 Kostrikov al.) (021). apply advantage-weightr
constraints balance BC and
advantages (Siegel et al.] 2020; Wang et al. 2020b), penalize the prediction-error of a variational auto-encoder (Rezaeifar et al. 2021). and learn latent actions (or primitives) from the offline data (Zhou et al. 2020)] Ajay et al.) 202T). Nevertheless, such methods may cause overly conservative value functions and are easily affected by the behavior policy (Nair et al. 2020 Lee et al. 2021b). We remark that the OOD actions that align closely with the support of offline data could also be trustworthy. CQL (Kumar et al., 2020) directly minimizes the Q-values of OOD samples and thus casts an implicit policy constraint. Our method is related to CQL in the sense that both PBRL and CQL enforce conservatism in Q-learning. In contrast, we conduct explicit uncertainty quantification for OOD actions, while CQL penalizes the Q-values of all OOD samples equally.
In contrast with model-free algorithms, model-based algorithms learn the dynamics model directly with supervised learning. Similar to work, MOPO YYu et al. 2020) and MOReL. (Kidambil

---

Published as a conference paper at ICLR 2022
et al. 2020) incorporate ensembles of tmnamics-model for uncertainty quantification, and penalize
the value function through pessimistic updates. Other than the uncertainty quantification, previous model-basee methods also attempt constrain the learned policy through BC loss (Matsushimal
et al. 2020), ddlaantage-wiighted prior (Cang et al. 2021), CQI-style penalty (Yu et al. ). and Riemannian submanifold (wwnnfoooodeeeennaaaaannn (Tennenholtz et al. 2021). Decision Transformer (Chen et al. [202]] builds a transformer-style dynamic model and casts the problem of offline RL as conditional sequence modeling. However, such model-based methods may suffer from additional computation costs and may perform suboptimally in complex environments (Chua et al. 2018; Janner et al., 2019). In
contrast, PBRL conducts model-free learning and is less affected by such challenges.
Our method is related to the previous online RL exploration algorithms based on uncertainty quan- tification, including bootstrapped Q-networks (Bai et al., 2021 Lee et al., al.; 2021a), ensemble dynam- ics (Sekar et al., 2020), Bayesian NN (O' O'IDonoghue et al. 2018] Azizzadenesheli et al. 2018), and distributional value functions (Mavrin et al. | Nikolov et al. 2019). Uncertainty quantification is more challenging in offline RL than its online counterpart due to the limited coverage of offline data and the distribution shift of the learned policy. In model-based offline RL, MOPO (Yu et al. 2020) and MOReL (Kidambi et al. 2020) incorporate ensemble dynamics-model for uncertainty quantification. BOPAH (Lee et al. 2020 combines uncertainty penalization and behavior-policc constraints. In model-free offline RL, UWAC (Wu et al. (222T) adopts dropout-based uncertainty (Gal & Ghahramani] 2016) while relying on policy constraints in learning value functions. In con- trast, PBRL does not require additional policy constraints. In addition, according the study in image prediction with data shift (Ovadia et al. 2019). the bootstrapped uncertainty is more robust to data shift than the dropout-based approach. EDAC (An et al. 202T is concurrent work that uses the ensemble Q-network. Specifically, EDAC calculates the gradients of each Q- o-function and diver- sifies such gradients ensure sufficient penalization for OOD actions. In contrast, PBRL penalizes the OOD actions through direct OOD sampling and the associated uncertainty quantification.
Our algorithm is inspired by the recent advances in the theory of both online RL, and offline RL.. Previous works propose provably efficient RI. algorithms under the linear MDP assumption for both the online setting (Jin et al.) 2020) and offline setting (Jin et al. 202I), which we follow for our analysis. In addition, previous works also study the offline RL under the Bellman completeness assumptions (Modi et al.) 2021 | Uehara et al. 2021, Xie et al. 2021a Zanette et al., (021) and the
RL under the kerne lized nonlinear regulator (KNR) setting (Kakade et al. 20200] Mania
et al. 2020 Chang et al. 2021. In contrast, our paper focus on model-free RL.
5 EXPERIMENTS
In experiments, we include an additional algorithm named PBRL-prior, which is slight modi- fication of PBRL by incorporating random prior functions (Osband et al., 2018b). The random prior technique is originally proposed for online exploration in Bootstrapped DQN (Osband et al., 2016). Specifically, each Q-function in PBRL-Prior contains a trainable network Q2 and a prior network Pk, where Pk is randomly initialized and is fixed in training. The prediction of each Q-function is the sum of the trainable network and the fixed prior, Q Q + Pk. where Q and p) shares the same network architecture. The random prior function increases the diver- sity among ensemble members and improves the generalization of bootstrapped functions (Osband al. 2018b). We adopt SAC (Haarnoja et al. 2018) as the basic actor-critic architecture for both PBRL and PBRL-prior. We refer to $B for the implementation details. The code is available at https: /ttp////tttttbb wooo/aaicheeeee jaaaeennnnnaaa j
In D4RL benchmark (Fu et al. 2020) with various continuous-contro tasks and datasets, we com- pare the baseline algorithms on the Gym and Adroit domains, which are more extensively studied in the previous research. We compare PBRL and PBRL-Prior with several -fhe-of-the-rrt algorithms, including (i) BEAR (Kumar et al., 2019) that enforces policy constraints through the MMD distance, (ii) UWAC (Wu et al. 2021) that improves BEAR through dropout incertainty-weighee update, (iii) CQL (Kumar et al. 2020) that learns conservative value functions by minimizing Q-values of OOD actions, (iv) MOPO ( Yu et al. 2020) that quantifies the uncertainty through ensemble dynam- ics in a model-based setting, and (v) TD3-BC (Fujimoto & Gul 202I), which adopts adaptive BC constraint to regularize the policy in training.
7

---

Published as a conference paper at ICLR 2022
BEAR
UWAC
CQL
MOPO
TD3-BC
PBRL
PBRL-Prior
Random
HalfCheetah
2.3 10.0
2.3 10.0
17.5 11.5
35.9 12.9
11.0 11.
11.0 15.8
13.1 #1.2
Hopper
3.9
2.7 #0.3
7.9 =0.4
16.7
#12.2
8.5 10.6
26.8 19.3
31.6 #0.3
Walker2d
12.8
2.04
5.1 #13
4.2 #5.7
1.6
8.1 +4,4
8.8 +6.3
Medium
HalfCheetah
43.0
42.2 10.4
47.0 10.5
73.1
+2.4
48.3 10.3
57.9 11.
58.2 11.
Hopper
51.8 1:400
50,9
+4.4
53.0
#28.5
38.3
+34,9
59.3
+:42
75.3 #31.2
81.6
+ 14.5
Walker2d
1..11
75.4
+3.0
73.3
117.7
41.2
130.8
83.7
12.11
89.6 10..7
90.3
11.2
Medium Replay
HalfCheetah
4:.1
35.9 +3.7
45.5 +0.7 69.2
+1.1
44.6
+0.5
45.1
+8.0
49.5
+-088
Hopper
52.2 19..3
25.3 11.7
88.7
12299
32.7
19.4
60.9
1.18.8
100.6
11.00
100.7
1.0.4
Walker2d
7.0.17.8
23.6 16.9
81.8
-2.7
73.7
19.4
81.8
15.55
77.7
1:4.55
86.2
13.4
Expert Medium
Expert
HalfCheetah
46.0 4:4.
42.7
+0.3
75.6
+22.77
70.3
4:11.9
90.7
+:4.3
92.3 +1.1
93.6
+22.3
Hopper
50.6 1:25.3
44.9
18.1
105.6
11..
60.6
1:2.55
98.0
19.4
110.8
10.8
111.2
10.7
Walker2d
22.1 1:44.9
96.5
19.1
107.9
11.6
77.4
1:27.9
110.1
10.5
110.1
100.3
109.8
10.2
HalfCheetah
92.7 +:0.6
92.9
+0.6
96.3
+1.
81.3
#:21.8
96.7
+1.1
92.4
+1.7
96.2
+:2.3
Hopper
54.6 112100
110.5
10.5
96.5
128.0
62.5
1:290
107.8
177
110.5
10.4
110.4
10.3
Walker2d
106.6 16.8 108.4
10.4
108.5
=0.5
62.4
#3.2
110.2
10.3
108.3
1.0.3
108.8
1.22
Average
38.78 10.0 50.41
+:2.7 67.35 19.1 53.3
16.3 67.55
+3.8
74.37
4:5.3
76.66
4:2.4
Table 1: Average normalized score and the standard deviation of all algorithms over five seeds in Gym. The highest performing scores highlighted. The score of TD3-BC is the reported scores in Table 7 of Fujimoto & Gu (202I) The scores for other baselines are obtained by re-training with the 'v2" dataset D4RL (Fu et al. 2020).
Results in Gym domain. The Gym domain includes three environments (HalfCheetah, Hop- per, and Walker2d) with five dataset types (random, medium, medium-replay, medium-expert, and expert), leading to total of 15 problem setups. We train all the baseline algorithms in the latest released 'v2' version dataset, which is also adopted in TD3-BC (Fujimoto & Gu (021) for evaluation. For methods that are originally evaluated on the 'v0' dataset, we retrain with their respective official implementations on the 'v2' dataset. We refer to $ for the training de- tails. We train each method for one million time steps and report the final evaluation perfor- mance through online interaction. Table I reports the normalized score for each task and the cor- responding average performance. We find CQL and TD3-BC perform the best among all base- lines, and PBRL outperforms the baselines in most of the tasks. In addition, PBRL-Prior slightly outperforms PBRL and is more stable in training with a reduced variance among different seeds.
We observe that compared with the baseline algo- rithms, PBRL has strong advantages in the non- optimal datasets, including medium, medium-replay, and medium-expert In addition, compared with the policy-constraint baselines, PBRL exploits the opti- mal trajectory covered in the dataset a theoreti- cally grounded way and is less affected by the be- havior policy. We report the average training curves in Fig. 2 In addition, we remark that the perfor- mance PBRL and PBRL-Prior are weaker than TD3-BC and CQL in the early stage of training, in- dicating that the uncertainty quantification is inaccu- rate initially. Nevertheless, PBRL and PBRL-Prior converge to better policies that outperform the base- lines in the learning of uncertainty quantifiers, demonstrating the effectiveness of uncertainty penal- ization and OOD sampling.
Average
83
53
BEAR
UWAC
22-
MOPO
10-
TD3-BC
PARI [nurs)
PSRL-Phio (ours]
200
402
600
abc
1058
Gradiert Steps (thoueando
Figure 2: Average training curve in Gym.
Results in Adroit domain. The adroit tasks more challenging than the Gym domain in task complexity. In addition, the use of human demonstration in the dataset makes the task even more challenging in the offline setting. We defer the results $E We observe that CQL and BC have the best average performance in all baselines, and PBRL outperforms baselines in most of the tasks.
8

---

Published as a conference paper at ICLR 2022
wwkgdeeneo//sttplooo. Orcerdattzl
wo (Q
wakcerds med. (Ihoeralmm))
wakedd makerr (f
0-4aliyy
Q Randoondttionn
(-tandomvcttion
uhees:
Orross
Uheosz
Qursbxt
(a) Walker2d Balker2d-MMdium Replaa
(b) Walker2d Meedii
Figure 3: The uncertainty and Q-value for different state-action sets in the training process.
Uncertainty quantification. To verify the effectiveness of the bootstrapped uncertainty quantifi- cation, we record the uncertainty quantification for different sets of state-action pairs in training. In our experiments, we consider sets that have the same states from the offline dataset but with differ- ent types of actions, including (i) Guffline which is drawn from the offline ni-distribution transition; (ii) Apolicy, which is selected by the training policy; (iii) Grand, which is uniformly sampled from the action space of the corresponding task; (iv) Gnoisel = Ooffline + (0, 0.1), which adds a small Gaussian noise onto the offline action to represent state-action pair that is close to in-distribution data; and (v) Qnoise2 = Goffline + (O, 0.5), which adds large noise to represent the OOD action.
We compute the uncertainty and Q-value in 'Walker2d' task with two datasets (' ("medium-replay and 'medium'). The results are shown in Fig. 3 We observe that (i) PBRL yields large uncertainties for (s, Quoise2) and (s, (random), indicating that the uncertainty quantification is high on OOD samples. (ii) The (s. Aoffline) pair has the smallest uncertainty in both settings as it is an in-distribution sample. (iii) The (s, Onoisel) pair has slightly higher uncertainty compared to (s, Goffline), showing that the uncertainty quantification rises smoothly from the in-distribution actions to the OOD) actions. (iv) The Q-function of the learned policy (s, (policy) is reasonable and does not deviate much from the in-distribution actions, which shows that the learned policy does not take actions with high uncertainty due to the penalty with uncertainty quantification. In addition, we observe that there no superior maximum for OOD actions, indicating that by incorporating the uncertainty quantification and OOD sampling, the Q- f-ffucctions obtained by PBRL does not suffer from the extrapolation error.
Ablation study. In the following, briefly report the result of the ablation study. We refer to SC and $D for the details. (i) Number of bootstrapped-Q. We attempt different numbers K of bootstrapped-Q in PBRL, and find the performance to be reasonable for K > 6. (ii) Penalization in tiu We conduct experiments with the penalized reward discussed in 33.2 and find the penalized reward does not improve the performance. (iii) Factor f.n in Pin We conduct experiments with different 5mn from (0.1, 0.01, 0.001, 0.0001} to study the sensitiveness. Our experiments show that PBRL performs the best for 3im E [0.0001, 0.01]. (iv) Factor Bood. We use large Bood at the initial training stage to enforce strong regularization over the OOD actions, and gradually decrease Bood in training. We conduct experiments by setting Bood to different constants, and find our decaying strategy generalizes better among tasks. (v) The learning target of OOD actions. We change the learning target of OOD actions to the most pessimistic zero target = 0 and find such a setting leads to overly pessimistic value functions with suboptimal performance. (vi) Regularization types. We conduct experiments with different regularization types, including our proposed OOD sampling, L2 be-regularization, spectral normalization, pessimistic initialization, and no regularization. We find OOD sampling the only reasonable regularization strategy and defer the complete report to $
6 CONCLUSION
In this paper, we propose PBRL, an uncertainty-based model-free offline RL algorithm. We pro- pose bootstrapped uncertainty to guide the provably efficient pessimism, and a novel OOD sampling technique to regularize the OOD actions. PBRL is closely related to the provable efficient offline RL algorithm under the linear MDP assumption. Our experiments show that PBRL. outperforms several strong offline RL baselines in the D4RL environments. PBRL. exploits the optimal trajectories con- tained in the suboptimal dataset and is less affected by the behavior policy. Mcanwhile, we show that PBRL produces reliable uncertainty quantifications incorporated with OOD sampling.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The code of our work available at vironments, datasets, and hyper-paramettes of our experiments given in the appendix. All settings, assumptions, lemmas, and theorems are proved and are discussed detail in our appendix.
The en-
ACKNOWLEDGEMEENT
This work was supported in part by the National Natural Science Foundation of China under Grant 51935005, in part by the Fundamental Research Program under Grant JCK JCKY20200603C010 The authors thank the anonymous reviewers, whose invaluable comments and suggestions have helped us to improve the paper.
REFERENCES
Yasin Abbasi- Abbasi-Yadkori, Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances neural information processing systems, volume 24, pp. 231202320, 2011.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in neural infor- mation processing systems, 2021.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. In International Conference Learning Representations, 2021.
Gaon An, Seungyong Moon, Lang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline rein- forcement learning with diversified g-ensemble. In Advances in neural information processing systems, 2021.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In International Conference Machine Learning, 2017.
Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q g-networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference Machine Learning, pp. 577-587. PMLR, 2021.
Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer offline rl. arXiv preprint arXiv:2106.091 19, 2021.
Jonathan D Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigat- ing covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2/06.03220 03207, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in neural information processing systems, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in neural information processing systems, 2021.
10

---

Published as a conference paper at ICLR 2022
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off- deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062, PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning, pp. 1050-1059.
PMLR, 2016.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691, PMLR, 2021.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: An optimisation perspective. In International Conference on Machine Learning, volume 139, pp. 3734 3744, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 2010.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32:12519- 12530, 2019.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pp. 2137-2143. PMLR, 2020.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Infor- mation theoretic regret bounds for online nonlinear control. arXiv preprint arXiv:2006. /2466, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 11810-21823, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- fems, volume 32, pp. 11794-11794 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191, 2020.
Byungjun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim, and Kee-Eung Kim. Batch reinforcement learning with hyperparameter gradients. International Conference on Machine Learning, pp- 9725-57355 PMLR, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference on Ma- chine Learning, pp. 6131-6141. PMLR, 2021a.

---

Published as a conference paper at ICLR 2022
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to- to -onliine reinforcement learning via balanced replay and pessimistic q-ensemble. In Anmual Conference on Robot Learning. 2021b.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164) 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Annual Conference Robot Learning, 2021.
Horia Mania, Michael Jordan, and Benjamin Recht, Active learning for nonlinear system identifi- cation with guarantees. arXiv preprint arXiv:2006. 10277, 2020.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofrr Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations, 2020.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning. pp. 4424-4434. PMLR, 2019.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. Advances in Neural Information Processing Systems, 31:2419-2430, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-555 2015.
Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration low-rank MDPs. arXiv preprint arX1v:2102.07035, 2021.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information-
directed exploration for deep reinforcement learning. In International Conference on Learning Representations, 2019.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, volume 29, pp. 4026- 4034, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018a.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018b.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32:13991-14002, 2019.
Brendan O' D'Dongguue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp. 3836-3845, 2018.
12

---

Published as a conference paper at ICLR 2022
Shidch Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. arXiv preprint 2.Xx::2006.06431,, 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning. pp. 8583 8592. PMLR, 2020.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. International Confer- on Learning Representations, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Guy Tennenholtz, Nir Baram, and Shie Mannor. GELATO: geometrically enriched latent model for offline reinforcement learning. CoRR, abs/2102.11122 2021.
Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank MDPs. arXiv preprint arXiv:21 2021.
Ruosong Wang, Simon S Du, Lin F Yang, and Ruslan Salakhutdinov. On reward-fres free reinforce- ment learning with linear function approximation. In Advances in neural information processing systems, 2020a.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020b.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: 1911.11311, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning, volume 139, pp. 11319-11328, 2021.
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. pessimism for offline reinforcement learning. In Advances neural information processing sys- tems, 2021a.
Tengyang Xie. Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. arXiv preprint urXiv:2106.04890 2021b.
Lin Yang and Mengdi Wang- Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning, pp. 6995-7004. PMLR, 2019.
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey, arXiv
preprint arXiv: 08796, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129-14144 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Advances in neural information processing systems, 2021.
Andrea Zanette, Martin I Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning. Advances in neural information processing systems, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, 2020.
13

---

Published as a conference paper at ICLR 2022
A THEORETICAL PROOF
A.1 BACKGROUND OF LCB- LCE-PPENALTY IN LINEAR MDPS
In this section, we introduce the provably efficient LCB- CCnapenalty in linear MDPs (Abbasi-Yadkori et al. 2011, Jin et al. 2020 (02T). We consider the setting of y 1 in the following. In linear MDPs, the feature map of the state-action pair takes the form of & S - R and the transition kernel and reward function are assumed to be linear in ø. As result, for any policy т, the state- action value function also linear in Φ (Jin et al. 2020), that is,
Q" (s), ot) :i (s)) a.).
(11)
The parameter wi can be solved in the closed-form by following the Least-Squares Value Iteration (LSVI) algorithm, which minimizes the following loss function,
0 = min
( at) ww r(s), a²) - Ve+ (s) + A- 11w12
(12)
wERd
where V4+1 is the estimated value function in the (t - 1)-th step, and r(s at) + V Ve+ is the (2) takes the form of
target of LSVI. The explicit solution
1 A, Σ
ass) at) (Ve+1 + +r(s) at)), A, E (l) at) + (13)
Here Ar accumulate the state-action features from the training buffer. Based on the solution of wis the action-value function can be estimated by Qi(st, ax) ti 6(s,, at). In addition, in offline RL with linear function assumption, the following CCB-penalty yields an uncertainty quantification,
(lc, a4) B [d(s), TA, [(s 1/2
(14)
which measures the confidence interval of the Q-functions with the given training data (Abbasi-
Yadkori et al. 2011 Jin et al. 2020 2021). In offline RL, the pessimistic value function (s a.) penalizes Qe by the uncertainty quantification pleb (1c,(se, at) as a penalty as follows,
(u(5,, az) Qe(st, ae) plch az)
(15)
= = (0t, az) - I (st, a:),
where w/ is defined in Eq. 12 Under the linear MDP setting, such pessimistic value iteration known to be cetematioo-thhoreticaee optimal (Tin et al.) [02T]. In addition, exploration with (lbb az) as a bonus is also provably efficient in the online RL setting (Abbasi-Yadee Yadkori et al.
2011 Jin et al. 2020).
A.2 CONNECTION BETWEEN THE BOOTSTRAPPED UNCERTAINTY AND Tleb
In the sequel, we consider a Bayesian linear regression perspective of LSVI in Eq. 112 According to the Bellman equation, the objective of LSVI is to approximate the Bellman target 6j r(s], a)) +
V (s++)) with the Q-function Q., where V the estimated value function in the (t+ 1)-th step. In linear MDPs, We parameterize the Q-function by Qe(st,a,) = w| d(s,, at). We further define the noise in this least-square problem as follows,
6 bi w! at),
(16)
where We is the underlying true parameter. In the offline dataset with Dm = {(s), at s/+ he)) denote by wi the Bayesian posterior of given the dataset Din- In addition, we assume that we are given a Gaussian prior of the parameter w ~ (O, I/A) as a non-informative prior. The following Lemma establishes connections between bootstrapped uncertainty and the LCB- LCB-penalty Ilb
Lemma (Formal Version of Claim I]. We assume that e follows the standard Gaussian distribution AO 1) given the state-action pair (s), at). It then holds for the posterior we given D,, that
Vars, (Qx at)) Vara, (6(s),at) w.) a(st, at) N
Vist E S (17)
14

---

Published as a conference paper at ICLR 2022
Proof. The proof follows the standard analysis of Bayesian linear regression. Under the assumption that e ~ (0, 1). we obtain that
(s), o0 N ((s), a(),
(18)
Recall that we have the prior distribution (O, I/A). Our objective is to compute the posterior density We = w Din holds from Bayes rule that
log logp(1 @0 Dm) = log ( + p( [T) + Const.,
(19)
where p(-) denote the probability density function of the respective distributions. Plugging (88) and the probability density function of Gaussian distribution into (19) yields
log p(
Dm)) = 11
W (6 ot) 1/1l3/2 Const.
i=1
-w me)
A, t (w- pt) Const.,
where we define
H4 A Σ
as aiv
A Σ o(s) dt)o(s) +
(20)
1=1
Then obtain that We w Din NNH,, AT It then holds for all (st, 0z) E E x A that
Var(ø(si) at) (ix Var( (ar(Qe((t,))) af)) p(s), at) A, 1 af),
(21)
which concludes the proof.
In Lemma [ we show that the standard deviation of the Q-posterior is equivalent to the LCB- penalty Var (((((Q(s,))) (8r, at) A, (08,, at) introduced in AA.1 Recall that our proposed bootstrapped uncertainty takes the form of
U( az) = (td,
(22)
which is the standard deviation of the bootstrapped Q- Q-functions. Such bootstrapping serves as estimation of the posterior of Q-functions (Osband et al. 2016). Thus, our proposed uncertainty quantification can be seen estimation of the LCB- C-B-eeealyy under the linear MDP assumptions.
In addition, under the tabular setting where the states and actions are finite, the LCB- ACB-penalty takes a simpler form, which show in the following lemma.
Lemma 2. In tabular MDPs, the boolstrapped uncertainty U(s, a) is approximately proportional to the reciprocal-cous of (s, a), that is,
u(s,a) R plch (l. a),, VNa.a +
(23)
Proof. In tabular MDPs, consider the joint state-action space d |S| x AA]. Then j-th state-
action pair can be encoded as a one-hot vector as Φ(8, a) E Rd, where j E 00, d.d-1], thus a special
case of the linear MDP Yang & Wang 2019
Jin et al. 2020). Specifically, we define
(3),,a)) =
E RRd
E R ddd
(24)
where the value of os az) is the j-th entry and elsewhere. Then the matrix A, Em d(st, at ())))) a²)² A is the sum of ((8,, (3)),, over (8,, E Dms which takes the form of
A,
(25)
15

---

Published as a conference paper at ICLR 2022
where the j-th diagonal element of A, is the corresponding counts for state-action (8j, Qy), i.e.
n Ne,,ay
thus holds that
[d(s,, / (83), a()
(26)
VNejay +X
which concludes the proof.
A.3 REGULARIZATION WITH OOD SAMPLING
In this section, discuss how OOD sampling plays the role of regularization in RL, which regu- larizes the extrapolation behavior of the estimated Q-functions on OOD samples.
we consider the setup of LSVI-UCB (Tin et al. 2020) under linear MDPs. Specifi- cally, we assume that the transition dynamics and reward function takes the form of
Similar to A.2
(i(4++1 st, at) = (11(s+1)), 6(s,, r(st, ax) = 0 p(5),0)). V($ (4, si) E ESXAX. S,
(27)
where the feature embedding Φb S A - Rd is known, We further assume that the reward function r: x A + [o, 1] is bounded and the feature is bounded by ||o||2 S 1. Given the dataset Din, LSVI
iteratively minimizes the least-square loss in Eq. (2). Recall that the explicit solution Eq. (22 takes the form of
m
A
Σ
(ist, (Ve+1 ((s. i=1
A = Σ a (3) of) + + (28)
We remark that for the regression problem in Eq. (12) ) the Le-regularizer A- |/|]3 enforces a Gaus- sian prior under the notion of Bayesian regression. Such regularization ensures that the linear func- tion approximation ø wi extrapolates well outside the region covered by the dataset Din-
However, as shown in $D we observe that such Le-regularization is ineffective for offline DRL. To this end, we propose OOD sampling in our proposed PBRL. To demonstrate the effectiveness of OOD sampling as regularizer, we consider the following least-squares loss with OOD sampling and without the Læ-regularizer,
m
min Σ (d(s). a.) rss, Ve+z($(+1))
E
(ø(sond andd) aadd T
y)²
woRd
(suat (snd,pn ())
(29)
The explicit solution of Eq. 29p takes the form of
ii = A-
Σ ( at) (r(s), at) + Ve+1(s(41)) +
Σ
ans aond)y
(30)
(gned and y)
where
A Σ (ss) (f)
Σ
( and ( a amd))
(31)
(ame,aa (0) ,o) Den
Hence, we further set y = 0 all oooo aood, v) Doed, then 29 enforces a Gaussian prior with the covariance matrix A A1 where we define
Aood
Σ
oos ao0d (olo Tod)T
(32)
(atM), .otM D..
Specifically, if we further enforce Dood = {(s), a² H)}e with o(s a²) = e², where ed
Rdd
is the unit vector with the j-th entry equals one, it further holds that Acot A I and Eq. 29
equivalent to Eq. [2). In addition, under the tabular setting. by following the same proof as in
Lemma 2 having such OOD samples in the training is equivalent to setting the count in Eq. 26 to be
Ñ Naj.0 Nas.a, Arood
16

---

Published as a conference paper at ICLR 2022
where Nzy is the occurrence of (sy, az) in the dataset and Noodd is the occurrence of (s), in 8y
the OOD dataset.
However, to enforce such a regularizer without affecting the estimation of value functions, we need to set the target of the OOD samples to zero. In practice, we find such setup to be overly pessimistic. Since the Q-network is smooth, such a strong regularizer enforces the Q-functions to be zero for state-action pairs from both the offline data and OOD data, show in Fig. 12 and Fig. 1 of [C We remark that adopting a nonzero OOD target y does not hinder the effect of regularization as it still imposes the same prior in the covariate matrix Aç. However, adopting nonzero OOD target may introduce additional bias in the value function estimation and the corresponding uncertainty quantification. To maintain a consistent and pessimistic estimation of value functions, one needs to carefully design the nonzero OOD target y.
To this end, we recall the definition of a f-uncertainty quantifier in Definition Das follows.
Definition (S-Uncertainty Quantifier (Jin et al. 2021)). The set of penalization (Trherryy forms E-Uncertaintyy Quantifier if it holds with probability at least 1 & that
(V, a)) S (T(s,a) for all (s.a) E S x A, where T is the Bellman operator and T is the empirical Bellman operator that estimates T based on the data.
(Tv+1(8,a)
We remark that here we slightly abuse the notation T of Bellman operator and write TV Elr(s, (s)) s, a]. Under the linear MDP setup, the empirical estimation TV441 is obtained via fitting the least-squares loss Eq. (9). Thus, the empirical estimation TV takes the following explicit form,
TV (8t; uz) o(st, ar) at) We:
where We is the solution to the least-squares problem defined Eq. (30). We remark that such g- uncertainty quantifier plays an important role in the theoretical analysis of RL algorithms, both for online RL and offline RL (Abbasi- Yadkori et al., 2011; Azar et al. 2017 Wang et al. [2200a] Jin al. 2020 2021 Xie al. [021a (). Our goal is therefore to design a proper OOD target y such that we can obtain f-uncertainty quantifier based on the bootstrapped value functions. Our design is motivated by the following theorem.
Theorem 2. Let Aood Ž X I. For all the OOD datapoint (so0d, aod ,y) E Dood. if we set
TV++1 (oood dood), it then holds for Be = OT Vd. (olooo((T))) that
rich (st, (st, az)= Be [0(s), ar) A, (s5))) 1/2
forms valid E-uncertainty quantifier.
Proof. Recall that we define the empirical Bellman operator 7 follows,
TV441 a²) d(st, at) Tww.
suffices to upper bound the following difference between the empirical Bellman operator and Bellman operator
(s, TVe a) -w.).
Here we define as follows
W = 0 /
(33)
where A and ab are defined in Eq. 27p. It then holds that
TV a) (v41 a) = o(s a) (wr - ww:)
m
= = We ((s, A Σ
(5
+ ((
b((s,)) TÃ
Σ
a ood y-
(34)
(go0s aood Ey
17

---

Published as a conference paper at ICLR 2022
where we plug the solution of in in Eq. (30) Meanwhile, by the definitions of De and w, in Eq. (1) and Eq. (33 respectively, we have
o(s a) = o(s,a) a) "ews
a)) A Σ (s) a ())
Σ o(soo, a00d ( 1 aao))
(anst
(35)
Plugging 35 into 34 yields
aV =
(36)
where we define
m
ii) o(s,0) 0) A, Σ
( aT) at) - r(s), (st1()++))
(ii) o(s a) A,=
o(s00d a a00) (TV+, a y).
(goed aa EDM
Following the standard analysis based on the concentration of self-normalized process (Abbasi-
Yadkori et 2011 Azar al., 2017. Wang et al. 2020a Jin al. 2020 2021) and the fact that
Aood 2 A- it holds that
(1) S Bt [d(s,, at) A (02, (s)),
(37)
with probability at least 1 1 E. where Br = O(T vd log(T/E)). Meanwhile, by setting TV++1 (oood ood), ) it holds that (ii) Thus, obtain from (36) that
(TV++1(8,)) VV+1 a)l Bg (0(s), Ar ( 1/2
(38)
for all as E S x A with probability at least 1
Theorem 2 allows us to further characterize the optimality gap of the pessimistic value iteration. In particular, the following corollary holds.
Corollary (Jin et al. (2021). Under the same conditions Theorem it holds that
2
V*(s1) V*)
Σ E. [Tlc (8t, (x) [1]
Proof. See e.g., Jin et al. (2021) for a detailed proof.
We remark that the optimality gap in Corollary [ is information-thhorrticalll optimal under the linear MDP setup with finite horizon (Jin et al.) (021). Intuitively, for offline dataset with sufficiently good coverage on the optimal trajectories such as the experience from experts, such gap small. For dataset collected from random policy, such gap can be large. Our experiments also support such intuition empirically, where the score obtained by training with the expert dataset is higher than that with the random dataset.
Theorem 2 shows that if we set y TV++ (s addd), then our estimation based on disagreement among ensembles is a valid g-uncertainty quantifier. However, such OOD target is impossible to obtain in practice as it requires knowing the transition at the OOD datapoint (oood ood). In practice, TD error sufficiently minimized, then Q++1(8, should well estimate the target TV++1 Thus, in practice, utilize
Qu+1 a ood) - [st good
as the OOD target, where we introduce an additional penalization oood aod) to enforce the pessimistic value estimation.
In addition, we remark that in theory, we require that the embeddings of the OOD sample are isotropic, in the sense that the eigenvalues of the corresponding covariate matrix Aood are lower
18

---

Published as a conference paper at ICLR 2022
bounded. Such isotropic property can be guaranteed by randomly generating states and actions. In practice, we find that randomly generating states more expensive than randomly generating ac- tions. Mcanwhile, reasonable empirical performance since the generated OOD emheddings are sufficiently isotropic. Thus, in experiments, randomly generate OOD actions according current policy and sample OOD states from the in-distribution dataset.
observe that randomly generating actions alone are sufficient guarantee
B IMPLEMENTATION DETAIL
B. ALGORITIMIIC DESCRIPTION
Algorithm PBRL algorithm
1: Initialize: K bootstrapped Q-networks and target Q-networks with parameter 0 and 0 policy ı with parameter y, and hyper-parameters Bin Good
2: Initialize: total training steps H, current frame =
3: while h < H do
4: Sample mini-batch transitions (s, as,0,7, from the offline dataset Dmn
5: Critic Training for offline data.
6:
Calculate the bootstrapped uncertainty Ug- (s', ('s',a')) a²) through the target-networks.
7:
Calculate the Q-target in Eq. (4) and the resulting TD-loss ( a) T'm
8:
Critic Training for OOD data
9:
Perform OOD sampling and obtains Nood OOD actions nod for each
10:
Calculate the bootstrapped uncertainty L4 annd) for OOD actions through Qnettworks.
Calculate the Q-target in Eq. (5) and the pseudo TD-loss 1Q) (s, aood)- -onii---oon Ffodd"" 12
11:
Minimize 1Q (s. Q (s,a))2+1QH+ aood) -roodd (s, )2od) train by SGD.
13:
# Actor Training
14:
Improve Tep by maximizing ming Q' an) - log u (a(anl)) with entropy regularization.
15:
Update the target Q-network via 6 -)0 78.
16:
end while
B.2 HYPER-PARAMETERS
Most bapereparameters of PBRL follow the SAC implementations in ht ptttpp::/// /ttpttgttbbbbtt com/ rail-berkeley/rlktt We use the hyper-parameter settings in Table for all the Gym domain tasks. We use different settings of iin and 3ood for the experiment for Adroit domain and fix the other hyper-parameters the same as Table 2. See $ for the setup Adroit. addition, we use same settings for discount factor, target network smoothing factor, learning rate, and optimizers as CQL Kumar et al. 2020).
Table 2:
of PBRL.
Hyper-parameters
Value
Description
10
The number of bootstrapped networks. Fully Connected (FC) layers with ReLU activations.
Q-network
FC(256,256.256)
Bin
0.01
The tuning parameter of in-distribution target -
Bood
5.0 + 0.2
The tuning parameter of OOD target Food We perform linearly decay within the first 50K steps, and perform expo- nentially decay in the remaining steps. Target network smoothing coefficient.
0.005
7
0.99
Discount factor.
Ir of actor
le-4
Policy learning rate.
Ir of critic
3e-4
Critic learning rate. Optimizer. Total gradient steps.
Optimizer
Adam
H
IM
Nood
Number of OOD actions for each state.
19

---

Published as a conference paper at ICLR 2022
We conduct experiments D4RL with the latest 'v2' datasets. The dataset is released at http://rail: eecs att.eeeeeeeeeuttttttttttt//////e
Baselines.
x1/gym lawaayamaaaaaaaaaaaaaaaaaaaaaaaa v2 We now introduce the implementations of baselines. (i) The implementation of CQL (Kumar et al. 2020) is adopted from the official implementation at / wigghu..bb......... com/ In experiment, we remove the BC warm-up stage since find CQL performs better without warm-up for 'v2" dataset. (ii) For BEAR (Kumar et al., 2019). UWAC (Wu al. 202I) and MOPO (Yu et al. 2020), we adopt their official implemen- tations at https: http//ggttuubub com
old/
avi 1 kuma 22207/CC/
BB BEAR https: /githb.. com/ hopple/ml-wwd. and hetos: /ttpttttthtbbt/ hwt.:/tttu.....tttiooooeeeeoo com/ wtt.://aannnnnnnnnn111..1 mopo respectively. We in training. (iii) Since the original paper of TD3-BC (Fujimoto Gu 202T) reports the performance of Gym in 'v2' dataset in the appendix section, we directly cite The learning curve reported in Fig. 2 is trained by implementation released at httpt::://// / /i thub com/ s sfu jim/TD3 TD3 BC
adopt their default
the reported scores in Table [T]
Computational cost comparison. In the sequel, we compare the computational cost of PBRL against CQL. We conduct such a comparison based on the Halfcheetab-mediuum-22 task. We mea- sure the number of parameters, GPU memory, and runtime per epoch (IK gradient steps) for both PBRL and CQL in the training. We run experiments on a single A100 GPU. We summarize the re- sult in Table We observe that, similar to the other ensemble-based methods such as Bootstrapped DQN (Osband et al.) 2016)) IDS (Nikolov et al., 2019), and Sunrize (Lee et al. 2021a), our method requires extra computation to handle the ensemble of Q-networks. In addition, we remark that a large proportion of computation for CQL is due to the costs of logsumexp over multiple sampled actions (Fujimoto & Gu 2021), which we do not require.
Table 3: Comparison of computational costs.
Runtime (s/epoch)
GPU memory
Number of parameters
CQL
30.3
1.1G
0.42M
PBRL
52.1
1.7G
1.52M
B.3 REMARKS ON FORMULATIONS OF PESSIMISM IN ACTOR AND CRITIC
We use different formulations to enforce pessimism in actor and critic. the critic training, we the penalized Q-function in Eq. (4) and Eq. (5). While in the actor training, we the minimum of ensemble Q-function as in Eq. (7). According to the analysis in EDAC [6], using the minimum of ensemble Q-function minj=1,.... I K Q2 the target is approximately equivalent using Q 9 Std(Q,) with fixed Bo the target. In contrast, in the critic training of PBRL,, we tune different factors (i.e., fi and Bood) for the in-distribution target and the OOD target, which yields better performance for the critic estimation.
In the actor training, since we already have pessimistic Q-functions learned by the critic, it is not necessary to enforce large penalties in the actor. To see such fact, we refer to the ablation study in Fig. [9] where utilizing the mean as the target achieves reasonable performances. We remark that taking the minimum
the target avoids possible large values at certain state-action pairs, which may arise due to the numerical computation in fitting neural networks. As suggested by our abla- tion study, taking the minimum among the ensemble Q-functions as the target achieves the best performance. Thus, we use the minimum among the ensemble of D-functions as the target in PBRL.
20

---

Published as a conference paper at ICLR 2022
C ABLATION STUDY
In this section, we present the ablation study of PBRL. the training, we perform online interac- tions to evaluate the performance for every 1K gradient steps. Since each method for 1M gradient steps totally, each method is evaluated 1000 times in training. The experiments follow such evaluation criteria, and each curve is drawn by 1000 evaluated scores through online interaction.
Number of bootstrapped-Q. We conduct experiments with different numbers of bootstrapped Q-networks PBRL, and present the performance comparison in Fig- E We observe that the performance of PBRL is improved with the increase of the bootstrapped networks. We remark that since the training of offline RL is more challenging than online RL, better to use sufficient bootstrapped Q Q-functions to obtain reasonable estimation of the non-pane non-parametric -pposterior. We observe from Fig. 4 that using K = 6, 8, and 10 yields similar final scores, and a larger K leads to more stable performance in the training process. We adopt K 10 for our implementation.
K=4
K-6
wallke22d-eeiuu-12
hopper-mediun-eeplayyy
100-
80-
deri score
60
200
400
600
600
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 4: The Ablation on the number of bootstrapped ffunctionss.
Uncertainty of in-iistribution target. We compare different kinds of uncertainty penalization in
Tin for in-distribution data. (i) Penalizing the immediate reward only. (ii) Penalizing the next-Q value only, which is adopted in PBRL. (iii) Penalizing both the immediate reward and next-Q value.
We present the comparison in Fig- 5 We observe that the target-Q penalization performs the best, and adopt such penalization in the proposed PBRL algorithm.
Penalty only
Penalty only next-Q
Penalty
next-Q
walker2d-medium-v2
hopper--edium-replaayy22
100-
80
80-
d4rl score
60
60
40
40
20-
20
05
200
400
600
860
1000
ob
200
400
8bo
1000
Gradient Steps (thousands)
Figure 5: The Ablation penalty strategy for the in-distribution data.
Tuning parameter 3.. We conduct experiments with 8i E {0.1, 0.001, 0.0001] to study the sensitivity of PBRL to the tuning parameter Bn for the in-distribution target. We present the
21

---

Published as a conference paper at ICLR 2022
comparison in Fig. 6 We observe that in the 'medium' dataset generated by single policy, the performance of PBRL is insensitive to Pin- One possible reason is that since the 'medium' dataset is generated by single policy, the offline dataset tends to concentrate around a few trajectories and has low uncertainty. Thus, the magnitude of fn has limited effect on the penalty. Nevertheless, in the 'medium-replay" dataset, since the data is generated by various policies, the uncertainty of offline data is larger than that of the 'medium" dataset (as shown in Fig. 3) Correspondingly, the perfor- mance of PBRL is affected by the magnitude of 3m-- Our experiment shows that PBRL performs the best for Bin E [0.0001, 0.01]. We adopt Sin = 0.01 for our implementation.
B., 000001
8w-0.001
-
8.,0.01
Bu-
walker2d-meedum-42
hopper-medium-reelay--
100
80
80-
60
d4ri score
40-
20
20
04
400
600
800
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 6: The Ablation on the tuning parameter Gin the in-distribution target.
Tuning parameter Bood- We use a large Good initially to enforce strong OOD regularization in the beginning of training, and then decrease Bood linearly while the training evolves. We conduct {0.01, 0.1, 1.0]. We observe that in the 'medium'
experiments with constant settings of Bood dataset, large Bood 1.0 performs the best since the samples are generated by fixed policy with relatively concentrated in-distribution dataset. Thus, the OOD samples tend to have high uncertainty and are less trustworthy.
contrast, in the 'medium-replay dataset, a small Bood {0.1, 0.01} performs reasonably well since the mixed dataset has larger coverage of the state-action space and the uncertainty of OOD data is smaller than that of the 'medium' dataset. Thus, adopting smaller Good for the medium-replay dataset allows the agent to exploit the OOD actions and gain better performance. To match all possible situations, we propose the decaying strategy. Empirically, we find such decaying strategy performs well in both the 'medium" and 'medium-replay datasets.
Constant-0.01
Constant-0.1
Constant-1.0
Decay (ours)
walker2dd-edium--2
100
80
d4r1 score
60-
60-
40-
20-
20
0
200
400
600
860
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 7: The Ablation on the tuning parameter good in the OOD target.
We also record the Q value for (s, a) ~ Din in the training process. As shown in Fig. g since both the in-distribution actions and OOD actions are represented by the same Q-network, large
22

---

Published as a conference paper at ICLR 2022
stant Bood makes the Q-value for in-distribution action overly pessimistic and leads to sub- sub-optimal performance. desirable to use the decaying strategy for Food in practice.
Constant-0.01
Constant-0.1
Constant-1.0
Decay (ours)
walker2d-medium-42
hopper-medium-eelaa--2
350
300
200
Q Offline
150
200
150
100
100
0
200
400
660
860
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 8: With different settings of good, we show the Q-value for state-action pairs sampled from Di in the training process.
Actor training. We evaluate different actor training schemes in Eq. 7 i.e., the actor follows the gradients of 'min' (in PBRL), 'mean' or 'max' value among Q-functions. The result in Fig. 2 shows training the actor by maximizing the 'min' among ensemble-Q performs the best. In the 'mdiuum-rpllay" dataset, since the uncertainty estimation difficult in mixed data, taking 'mean' in the actor can be unstable in training. Taking 'max' in the actor performs worse in both tasks due to overestimation of Q-functions.
Actor: Max
Actor: Mean
Actor: Min
walker2d-meddum-v2
hopper-meduun-reellay-
80-
100-
80-
0411 score
60-
60-
40-
40-
20
20-
04
200
400
600
800
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 9: The Ablation on action selection scheme of the actor.
Number OOD samples. The pessimism in critic training is implemented by sampling OOD actions and then performing uncertainty penalization, shown in Eq. (6). In each training step, we perform OOD sampling and sample Nood actions from the learned policy. We perform an abla-
tion study with Nood E (0, 2. 5, 10}. According to Fig 10 the performance is poor without OOD sampling (i.e., Nond = 0). We find the performance becomes better with small number of OOD actions (i.e., Nood 2). Also, PBRL robust different settings of Nood:
OOD Target. In AA.3 we show that setting the learning target of OOD samples to zero enforces
Gaussian prior to fuuntiion under the linear MDP setting. However, in DRL, we find such a setup leads overly pessimistic value function and performs poorly in practice, Specifically, we observe that such a strong regularizer enforces the Q-functions to close to zero for both in-distribution and OOD state-action pairs, shown in Fig. 12 and Fig. 13 In contrast,
shown in Fig. II
23

---

Published as a conference paper at ICLR 2022
N-OOD:
N-OOD:
N-OOD:
N-OOD: 10
walker2d-medlum--2
hopper-meduun-reepllyyyy
100
80-
d4rl score
60
20
20-
200
400
600
800
1000
200
400
800
1000
Gradient Steps (thousands)
Figure 10: The Ablation on different number of OOD actions. the performance becomes better even with very small amout of OOD actions.
proposed PBRL target performs well and does not yield large extrapolation errors, shown in and Fig.
Fig. 12
Zero target for OOD
PBRL
walker2d-medium-22
hopper-mdduu--eeplaa..
80-
80
d4rl score
60-
60
40-
40-
20
20
400
600
800
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 11: The Ablation on OOD target with good (normalized scores).
Zero target for 0OD
PBRL
walker2d-mediuum-22
hopper copper-medium-replaayy
200
175
250
150
Q Offline
200
125
150-
100
75
100
50
50-
200
400
600
800
1000
200
400
600
800
1000
Gradient Steps (thousands)
Figure 12: The Ablation on OOD target with ood = 0. Q-offline is the Q-value for (s, a) pairs sampled from the offline dataset, where a follows the behavior policy.
24

---

Published as a conference paper at ICLR 2022
Zero target for OOD
PBRL
walker28-medium-v2
hopper-eediumrepllayyu
200
250
Currallcy
150
200
125
75
100
50
25
1000
400
600
1000
Gradient Steps (thousands)
Figure 13: The Ablation on OOD target with yodd = O. Q-CurPolicy is the Q-value for (s, ax)
pairs, where ax ([als) follows the training policy æ.
25

---

Published as a conference paper at ICLR 2022
D REGULARIZATION FOR PBRL
In this section, we compare different regularization methods that are popular in other Deep Learning (DL) literature solve the extrapolation error offline RL. Specifically, we compare the following regularization methods.
None. We remove the O0D-sampling regularizer PBRL and train bootstrapped Q-functions solely based on the offline dataset through T'n.
Le-reuularizer We remove the OOD -aD-sampling regularizer and use Lg normalization instead. As we discussed in |3.3 in linear MDPs, LSVI utilizes L2 regularization to control the trapolation behavior of Q Q-functions in OOD samples. In DRL, we add the L2-norm of weights in the Qnetwork in loss functions to conduct Ig-regularization. We attempt two scale factors {le-24 2, lc 4
4} in our experiments.
Spectral Normalization (SN). We remove the O0D-sampling regularizer and use SN instead. SN constrains the Lipschitz constant of layers, which measures the smoothness of the neural network. Recent research (Gogianu et al. 202I) shows that SN helps RL training when applied specific layers of the Q-network. In experiment,
follow Gogianu et al. (2021) and consider two cases, namely, applying SN in the output layer (denoted by SN/-7J) and applying SN in both the output layer and the before it (denoted by SN(-1,-2)), respectively.
Pessimistic Initialization (PI). Optimistic initialization is simple and efficient for RL explo- ration, which initializes the Q-function for all actions with a high value. In online RL, such initialization encourages the agent to explore all actions in the interaction. Motivated by this method, we attempt a pessimistic initialization to regulate the OOD behavior of the Q-network. In our implementation, the uniform distribution Unif b). We try two settings in our experiment, namely, (i) PI-small that (a, b) to (-0.2, 0), and (ii) PI-large that sets (a, b) (-1.0, 0). In both settings, we remove the OOD sampling and use PI instead.
draw the initial value of weights and bias of the Q-networks from
We illustrate (i) the normalized performance in the training process, (ii) the Q-value along the trajec- tory of the training policy, and (iii) the uncertainty quantification along the trajectory of the training policy. We present the results in Fig. 14 Fig. 15 and Fig. 16 respectively. In the sequel, we discuss the empirical results.
We observe that OOD sampling is the only regularization method with reasonable performance. Though Le-regularization and SN yield reasonable performance in supervised learning, they do not perform well in offline RL.
In the 'medium-replay' dataset, we observe that PI and SN can gain some score in the early stage of training. Nevertheless, the performance drops quickly along with the training process. We conjecture that both PI and SN have the potential to be effective with additional parameter tuning and algorithm design.
In conclusion, previous regularization methods for DL and RL are not sufficient to handle the dis- tribution shift issue and conservatism methods may lead to improved empirical performance, which we leave for future research.
in offline RL. Combining such regularization techniques with policy constraints
26

---

Published as a conference paper at ICLR 2022
Pi-large
Pissmall
5N|-1,2]
(2(0-44
(g
PBRL
walker2d-meeduum22
hogper-
100
score
60
60-
40
20
200
400
600
800
1000
200
400
630
800
1000
Gradient [thousands)
Figure 14: Comparision of different regularizers (normalized score)
Pi-large
Pissmall +
SN|-1,2]
-+ SNI-1]
(((e-4
(0le-2
+
None
PBRL
2ikeer2d-medium-22
hogger-mediuu--eelaaa-222
valun Current Folicy
5000
400
3000
250
2000
100-
Gracient Steps (thousands)
Figure 15: Comparision of different regularizers (Q-value along trajectories of the training policy)
Pi-large
Piismall
5N|-1,2]
(((e-4)
(g(1e-2)
PBRL
woller2d-mmeduu--22
hogger-mediuu--eeplaa-222
25
4000
20-
3000
15
Uncertainty
2000
10
200
400
600
800
1000
200
400
600
800
1000
[thousands)
Figure 16: Comparision of different regularizers (Uncertainty along trajectories of the training policy)
27

---

Published as a conference paper at ICLR 2022
E EXPERIMENTS IN ADROIT DOMAIN
In Adroit, the agent controls a 24-DoF robotic hand to hammer a nail, open door, twirl a pen, and move ball, shown in Fig. 17 The Adroit domain includes three dataset types, namely, demonstration data from a human ( ("human"), expert data from an RL policy ( ('expert)), ) and fifty-fifty mixed data from human demonstrations and imitation policy ('cloned'). The adroit tasks are more challenging than the Gym domain in task complexity. In addition, the use of human demonstration in the dataset also makes the task more challenging. We present the normalized scores Table We set Pm = 0.0001 and Bond = 0.01. The other hyper-paramenn parameters in Adroit follows Table 2
In Table 4 the scores of BC, BEAR, and CQL are adopted from the D4RL benchmark (Fu et al, 2020). We do not include the scores of MOPO (Yu et al. (020) as it is not reported and we cannot find suitable hyper-parametts
make it work in the Adroit domain. We also attempt TD3-BC (Fujimoto al. 2018)) with different BC weights and fail in getting reasonable score for most of the tasks. In addition, since UWAC [Wu al. 2021) has different evaluation process, we re-train UWAC with the official implementation and report the scores in Table 4
In Table 4 we add the average score without 'expert' dataset since the scores of 'expert' dataset dominate that of the rest of the datasets. We find CQL (Kumar et al.) 2020) and BC have the best average performance among all baselines. Meanwhile, we observe that PBRL slightly outperforms CQL and BC. We remark that the human demonstrations stored in 'human' and 'cloned' are inher- ently different from machine-gent -egeerrteeee 'expert' data since (i) the human trajectories do not follow the Markov property, (ii) human decision may be affected by unobserved states such as prior knowl- edge, distractors, and the action history, and (iii) different people demonstrate differently as their solution policies. Such characteristic makes the 'human' and 'cloned' dataset challenging for of- fline RL algorithms. In addition, we remark that the recent study in robot learning from human demonstration also encounter such challenges (Mandlekar et al. 2021).
Figure 17: Illustration of tasks in Adroit domain.
BC
BEAR
UWAC
CQL
MOPO
TD3-B8 BC
PBRL
Pen
34.4
-1.0
10.1 +3.2
37.5
0.0
35.4 13.3
Human
Hammer
1.5
0.3
1007
4.4
0.0
0.4 10.3
Door
0.5
-0,3
0.4 10.2
9.9
0.0
0.1 =0.0
Relocate
-0.3
0.0 +0.0
0.2
0.0
0.0 =0.0
Pen
56.9
26.5
16.9
39,2
0.0
74.9 -9.8
Cloned
Hammer
0.8
0.3
0.00
2.1
0.0
0.8 -055
Door
-0.1
-0.1
0.00
0.0
4.6 -4.8
Relocate
-0.1
-0.3
-0.3 +-0.0
-0.1
-0.1 +0.0
Pen
85.1
105.9
98.2 19.1
107.0
0.3
137.7
-3.4
Expert
Hammer
125.6
127.3
107.7 +-2777
86.7
127.5
-022
Door
34.9
103.4
104.7
10.4
101.5
95.7
112.
Relocate
101.3
98.6
105.5
L3.2
95.0
84.5
-1.2
Average
36.73
38.41
37.57
+:3.8
40.31
0.02
46.79
+3.9
Average w/o expert
11.74
3.21
4.35
1.4
11.70
0.02
14.52
12.3
Table 4: Average normalized score over seeds in Adroit domain. The highest performing scores highlighted. Among all methods, PBRL and CQL outperform the best of the baselines.
28

---

Published as a conference paper at ICLR 2022
F RELIABLE EVALUATION FOR STATISTICAL UNCERTAINTY
Recent research (Agarwal et al. 2021 proposes reliable evaluation principles to address the statisti- cal uncertainty RL. Since the ordinary aggregate measures like mean be easily dominated by a few outlier scores, Agarwal et al. (2021) presents several efficient and robust alternatives that are unduly affected by outliers and have small uncertainty even with a handful runs. In this paper, we adopt these evaluation methods for each method in Gym domain with Mask * Neeed runs.
Stratified Bootstrap Confidence Intervals. The Confidence Intervals (CIs) for a samte score estimates the plausible values for the score. Bootstrap CIs with stratified sampling can be applied to small sample sizes and better justified than sample standard deviations.
Performance Profiles. Performance profiles reveal performance variability through score dis- tributions. A score distribution shows the fraction of runs above certain score and is given by
F(r) = (TM I::M,
मे ΣM t D 1 [ 2 7].
Aggregate Metrics. Based on bootstrap Cls, butions, including median, mean, interquartile mean (IQM), and optimality gap. IQM discards the bottom and top 25% of the runs and calculates the mean score of the remaining 50% runs. Optimality gap calculates the amount of runs that fail to meet a minimum score of d) 50.0.
extract aggregate metrics from score distri-
The result comparisons are give in Fig. and Fig. 19 Specifically, Fig. T8 shows aggregate metrics based on 95% bootstrap Cls, and Fig. 19 shows performance profiles based on score distribution. For both evaluations, our PBRL and PBRL-prior outperforms other methods with small variability.
Median
Optimality Gap
PBRL-prior
PBRL TD3-BC
COL
MOPO
UWAC
BEAR
40
60
80
40
60
80
45
80
75
10
15
20
Figure 18: Aggregate metrics on D4RL with 95% CIs based on 15 tasks and seeds for each task. Higher mean, median and IQM scores, and lower optimality gap are better. The CIs are estimated using the percentile bootstrap with stratified sampling. Our methods perform better than baselines.
BEAR
UWAC
MOPO
COL
TD3-BC
PBRL
PBRL-prior
1.00
1.00
Fraction of runs with score >
T
Fraction of tasks with score >
0.75
0.75
0.50
0.50
0.00
0.00
20
40
60
80
100
120
20
40
60
80
100
120
Normalized Score (T)
Normalized Score (T)
Figure 19: Performance profiles on D4RL based on score distributions (left), and average score distributions (right). Shaded regions show pointwise 95% confidence bands based on percentile bootstrap with stratified sampling. The т value where the profiles intersect y = 0.5 shows median, and the area under the performance profile corresponds to the mean.
29


Reviews:
[{"review_id": "xnh5GdYG8dw", "reviewer": "Reviewer_73UJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-- This specific algorithm is novel, although borrows some elements from existing works (which are all referenced properly, as far as I can tell).\n\n-- Experiments are setup up properly and the algorithm shows nice results compared to baselines.\n\n-- The writing is clear and easy to follow.\n\nWeaknesses:\n\n-- I feel that some of the theoretical results in the main text are misleading, especially when compared to what is actually proved in the appendix. Namely, Lemma 1 in the main text asserts that the bootstrapped uncertainties in the algorithm are equivalent to LCB penalties at the heart of recent theoretical works. In contrast, the appendix establishes a completely different equivalence, showing that when performing Bayesian linear regression, the posterior is related to the same LCB penalties used in recent works. There is no established equivalence between Bayesian linear regression and the bootstrapped uncertainties used in the proposed algorithm, and in fact, I don't think there would be any equivalence, since in the only variation/diversity introduced into the critic ensemble is via initialization and SGD, which are irrelevant in the linear closed-form setting.\n\n-- The OOD term of the critic loss seems strange to me. It is effectively regressing each ensemble member Q_i to Q_i - uncertainty(Q), so the Q_i's will be encouraged to decrease indefinitely, until uncertainty(Q) becomes zero somehow. In fact, in a tabular setting, the gradient update would decrease each Q_i by the same amount, learning_rate * uncertainty(Q), and so the uncertainty will be unchanged over SGD updates, and so the Q_i's will diverge to negative infty.\n\n-- I would be interested to see more ablations (I may have missed some of these); e.g., the same algorithm but without the OOD term in the critic loss, or perhaps replacing the OOD penalty with some other penalty (CQL-style penalty?). Perhaps also measuring the uncertainty in other ways -- previous works (BEAR, BCQ) backup a minimum over an ensemble; is this better than penalizing with stddev?", "summary_of_the_paper": "The paper looks at the offline RL problem and claims that existing techniques penalize out-of-distribution actions too aggressively and so perform poorly at generalization and extrapolation. To remedy this, the paper proposes an offline RL algorithm in which an ensemble of critics is trained with an objective composed of (1) a TD error based on actions seen in the dataset with target value penalized by standard deviation over the ensemble of the next Q-values, and (2) a squared error on the Q-values of OOD actions regressing to those same Q-values penalized by the standard deviation over the ensemble. The proposed algorithm is paired with a theoretical analysis connecting it to recent theoretical offline RL papers. The algorithm is evaluated on the D4RL benchmarks and shows favorable performance compared to baselines.", "summary_of_the_review": "Overall, the paper is a nice contribution, but I think some aspects of the presentation need to be changed before it is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cgkhf2zlB2", "reviewer": "Reviewer_HkNt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The proposed PBRL is well motivated and is also backed by theoretical results in a linear MDP setting. The pessimistic value backup itself may not be new in the offline RL context (e.g. Appendix E in [1]), but it was introduced as a practical trick in the previous work, while PBRL's pessimistic bootstrapping is supported by theory, which I think is a good contribution. Also, to my knowledge, the way of exploiting the OOD samples is novel. Experiments and ablation studies are convincing and thorough. Overall, I think the paper made a solid contribution.\n\n- In contrast to $\\hat T^{in}$ in Eq (4), $\\hat T^{ood}$ in Eq (5) is not a contraction mapping. Therefore, repeatedly applying $\\hat T^{ood}$ would yield a divergence to $-\\infty$ of Q-value, which seems problematic?\n\n\n[1] Lee et al., Batch Reinforcement Learning with Hyperparameter Gradients, ICML 2020\n\n", "summary_of_the_paper": "This paper presents Pessimistic Bootstrapping for offline RL (PBRL), a model-free offline RL algorithm that purely relies on an uncertainty-driven method. Bootstrapped Q-functions are trained, and the standard deviation of their estimates is used for the uncertainty quantification. This uncertainty quantification is then used for pessimistic bootstrapping. Also, in contrast to the existing methods that only considers in-distribution target, PBRL optimizes Q-function even for out-of-distribution actions with the pseudo-target that is penalized by the uncertainty quantifier. A theoretical analysis is provided that PBRL is provably efficient in the linear MDP setting. Experimental results demonstrate that PBRL outperforms the state-of-the-art methods.\n", "summary_of_the_review": "The paper is well written. The proposed method is well-backed by theory and the empirical results are convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Rx3JcjNQ_O2", "reviewer": "Reviewer_hc93", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n- This paper is well-written and easy to read. \n- The paper provides the theoretic interpretation of their uncertainty penalization. \n- The proposed method outperforms various offline RL algorithms including model-based algorithms and uncertainty-aware methods. \n- The paper provides informative ablation studies comparing various regularizations and analyzing the effects of hyperparameters.  \n\n**Cons**\n- The proposed method requires additional computation costs and memory for Q-network ensemble and OOD penalization. It will be informative to compare methods in terms of computational cost and space. \n- Formulation of actor-critic objectives is relying on heuristics. For example, the critic loss term uses uncertainty penalized Q-values, whereas the policy loss term uses the minimum Q-value. Is there any reason to use these different forms?\n\n**Additional comments**\n- Does the PBRL use identical hyperparameter settings provided in Table 2 across all of the Gym and Adroit domains? If then, it is very surprising, because each task has a different dimension and complexity. As I experienced, most of the existing offline RL algorithms including CQL, MOPO, etc. are very sensitive to hyperparameters (e.g, lr, penalty coefficient) and must tune their hyperparameters to each domain. Because it is hard to validate models without online interaction, I believe the robustness to hyperparameters is a very important issue in Offline RL. \n- There is a recent paper [1] that is highly related to this submission. The proposed method seems more stable, but however, it will be informative to compare the proposed method with [1] in revision. \n\n[1] An et. al, Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble, To appear at NeurIPS 2021 (https://arxiv.org/abs/2110.01548). \n\n**Post Rebuttal**  \nI appreciate the author's efforts and clarification. The paper becomes much clearer (e.g., hyperparameter settings) and improved by adding computation cost analysis and other baselines. Thus I maintain my score. \n\n", "summary_of_the_paper": "This paper proposed an uncertainty-aware offline reinforcement learning algorithm based on a Q-network ensemble. The proposed method penalizes Q-values on OOD actions and performs pessimistic offline Q-learning. The paper interprets the method with LCB framework. The proposed method outperforms existing offline RL algorithms on D4RL Gym and Adroit tasks. ", "summary_of_the_review": "This paper is well motivated and clearly written with sufficient theoretical and experimental backups. Overall, I believe this is a good paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0LQKkZxWl4", "reviewer": "Reviewer_wgNz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers theoretical support in the linear MDP setting and has compelling empirical results in the D4RL suite (with both MuJoCo and Adroit tasks). Comments:\n\n1. Can the authors spell out all assumptions for the theoretical section of the paper when describing the problem setting? \n\n2. The paper presents why the proposed bonuses can be viewed as a lower confidence bound type penalty. The paper doesn't present a result comparing the value of the learnt policy against the optimal policy under more assumptions (e.g. Bellman completeness and an appropriate coverage/relative condition number assumption). See Chapter 3 in Agarwal et al. and the work of Zanette et al. as a means to obtain such guarantees.\n\n3. The assumption on \\Lambda_ood having eigenvalues lowerbounded by lambda appears rather strong (and rather extraneous). Comments on why this is needed or how it can be relaxed? \n\n4. The theory and practical sections don't particularly relate to each other in that the theory focuses on linear/tabular MDPs. One way to bridge this divide would be through working out uncertainty measures under a Kernel Nonlinear Regulator (KNR) assumption, see [Mania et al., Kakade et al.].\n\n5. Regarding hyper-parameters:\na. Are the hyper-parameters in table 2 used in all environments considered in the paper?\nb. What were the hyper-parameters that exhibited the most sensitivity? Could you mention what ranges these values these hyper-parameters were grid searched on?\nc. For a given hyper-parameter, did you have to perform online interactions after every few gradient steps to examine the value of the policy that you have, or did you perform online interactions once per hyper-parameter (at the end of learning)?\nCould you detail answers to these questions in the appendix?\n\n4. Could you mention what Q offline and Q current policy mean in figure 10?\n\nAgarwal et al. Reinforcement Learning: Theory and Algorithms\nZanette et al. Provable Benefits of Actor-Critic Methods for offline RL\nKakade et al. Information Theoretic Regret Bounds for Online Nonlinear Control\nMania et al. Certainty equivalent control of LQR is efficient.\n", "summary_of_the_paper": "This paper presents a model-free pessimistic bootstrapping approach for offline RL. Specifically, the paper considers an actor-critic approach with an ensemble of Q-functions and utilizing disagreements in their predictions (measured as standard deviation) for learning the Q-functions. The paper also presents a way to regularize the learning with out of distribution state-action pairs which according to the paper is a crucial part in obtaining improved results. \n\n---\nUpdate post author response: I have increased my score.", "summary_of_the_review": "The theory results in their current form are limited, and this can be addressed by expanding its scope to include classes of non-linear dynamics models; furthermore, the theory results are partial in their characterization since they fall short of characterizing the sub-optimality of the learnt policy as a function of properties of the offline dataset (measured in the form of coverage). That being said, the paper is definitely promising in terms of its empirical results. I'd be up for revisiting my reviews if these issues can be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes significant advances in offline reinforcement learning by proposing a new approach of being pessimistic to deal with uncertainties in the offline data.  The proposed approach uses bootstrapped Q-functions to quantify the uncertainty, which by itself is not new, and introduces additional data based on the pseudo-target that is penalized by the uncertainty quantification.  The use of such additional data is the first of a kind, and the paper provides theoretical support for the case of linear MDP and empirical support with the D4RL benchmark.  The reviewers had originally raised concerns or confusions regarding theoretical analysis and experiments.  The authors have well responded to them, and no major concerns remain.
Decision: Accept (Spotlight)