Confidence-Conditioned Value Functions for Offline Reinforcement Learning
Authors: Joey Hong
Aviral Kumar
Sergey Levine
Venue: ICLR 2023 notable top 5% (2023)
Keywords: reinforcement learning
offline reinforcement learning
ensembles
adaptation

Abstract: Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of OOD actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence. We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains. 

Published as a conference paper at ICLR 2023
CONFIDENGE-CONDTTONED VALUE FUNCTIONS FOR OFFLINE REINFORCEMENT LEARNING
Joey Hong, Aviral Kumar, Sergey Levine
University of California, Berkeley (Joey hong, aviral (eviralkkeen waekkkeeereeeey.... edu, svlevine@eech ber.eeeyy........ edu
ABSTRACT
Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distributio (OOD) actions. However, such methods exhibit one notable drawback: policies optimized such value functions can only behave according to fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated we instead are able to learn policies for varying degrees of conservatism at training time and devise method to dynamically choose one of them during evaluation. To do so, this work, we propose learning value functions that additionally condition the degree of conservatism, which we dub confudence--onddiionee value functions. We derive new form of Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence. We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.
INTRODUCTION
Offline reinforcement learning (RL) aims to learn effective policies entirely from previously collected data, without any online interaction (Levine et al., 2020). This addresses one of the main bottlenecks in the practical adoption of RL in domains such recommender systems (Afsar et al., 2021), healthcare (Shortreed et al., 2011; Wang et al., 2018), and robotics (Kalashnikov et al., 2018), where exploratory behavior primarily caused by distribution shift.. Naive algorithms can grossly overestimate the return of actions that are not taken by the behavior policy that collected the dataset (Kumar et al., 2019a). Without online data gathering and feedback, the learned policy will exploit these likely suboptimal actions. One common approach to handle distribution shift in offline RL to optimize conservative lower-bound estimate of the expected return, or Q-values (Kumar et al., 2020; Kostrikov et al., 2021; Yu et al., 2020). By intentionally underestimating the Q-values of out-of-distribution (OOD) actions, policies are discouraged from taking OOD actions. However, such algorithms rely on manually specifying the desired degree of conservatism, which decides how pessimistic the estimated Q-values are. The performance of these algorithms is often sensitive to this choice of hyperparameter, and an imprecise choice can cause such algorithms to fail.
be costly and dangerous. However, offline RL introduces new challenges,
Our work proposes the following solution: instead of learning one pessimistic estimate of Q-values, we propose an offline RL algorithm that estimates Q-values for all possible degrees of conservatism. We do so by conditioning the learned Q-values on its confidence level, probability that it achieves a lower-bone bound on the true expected returns. This allows us to learn a range of lower-bound Q- values of different confidences. These confidene-condiiton conservative RL algorithms could not-cot not-control the level confidence used to evaluate actions. Specifically, when evaluating the offline-learedd Q-values, policies derived from conservative offline
Q-values enables to do something

---

Published as a conference paper at ICLR 2023
RL algorithms must follow static behavior, even if the online observations suggest that they being overly pessimistic or optimistic. However, our approach enables ponfidecee-daapiie policies that can correct their behavior using online observations, by simply adjusting the confidence- used to estimate Q-values. We posit that this adaptation leads to successful policies more frequently than existing static policies that rely on tuning rather opaque hyperparameter during offline training.
Our primary contribution is a new offline RL algorithm that we call wunfiennc-conddiionee value- learning (CCVL), which learns mapping from confidence levels to corresponding lower-bound estimations of the true Q-values. Our theoretical analysis shows that our method learss appropriate lower- bound value estimates for any confidence level. Our algorithm also has a practical implementa- tion that leverages multiple existing ideas in offline RL. Namely, we use network parameterizations studied distributional RI. to predict Q-values parameterized by confidence (Dabney et al., 20;8), Our objective, similar to conservative Q-learning (CQL) (Kumar et al., 2020), uses regularization to learn Q-values for all levels of pessimism and optimism, instead of anti-exploration bonuses that may be difficult to accurately compute in complex environments (Rezaeifar et al., 2021). In addition, our algorithm can be easily extended to learn both lower- and upper-bound estimates, which can be useful when fine-tuning our offline-leareed value function on additional data obtained via online exploration. Finally, we show that our approach outperforms existing state-of-the art approaches in discrete- acttion environments such as Atari (Mnih al., 2013; Bellemare et al., 2013). Our empirical results also confirm that conditioning on confidence, and controlling the confidence from online observations, can lead to significant improvements in performance.
RELATED WORK
Offline RL (Lange et al., 2012: Levine et al., 2020) has shown promise in numerous domains. The major challenge offline RI. is distribution shift (Kumar et al., 2019a), where the learned policy might select out-of- distiiiiitttton actions with unpredictable consequences. Methods to tackle this challenge can be roughly categorized into policy-constraint or conservative methods. Policy-constraint methods regularize the learned policy to be "close" to the behavior policy either explicitly in the objective via a policy regularizer (Fujimoto et al.. 2018; Kumar et al., 2019a; Liu et al., 2020; Wu et al., 2019; Fujimoto & Gu, 2021), implicitly update (Siegel et al., 2020; Peng et al., 2019; Nair et al., 2020), or via importance sampling (Liu et al., 2019; Swaminathan & Joachims, 2015; Nachum et al., 2019). On the other hand, conservative methods learn a lower-bound, or conservative, estimate of return and optimize the policy against it (Kumar et al., 2020; Kostrikov et al., 2021; Kidambi et al., 2020; Yu al., 2020; 2021). Conservative approaches traditionally rely on estimating the epistemic uncertainty. either explicitly via exploration bonuses (Rezacifar et al., 2021) or implicitly using regularization on the learned Q-values (Kumar et al., 2020). The limitation of existing offline RI. approaches is that the derived policies can only act under a fixed degree of conservatism, which is determined by an opaque hyperparameter that scales the estimated epistemic uncertainty, and has to be chosen during offline training. This means the policies will be unable to correct their behavior online, even if it becomes evident from online observations that the estimated value function is too pessimistic or optimistic.
Our algorithm learns -efffidence-onndtiionnedd Q-values that capture all possible degrees of pessimism by conditioning on the confidence level, modeling epistemic uncertainty as a function of confidence. enable policies that adapt how
By doing so, instead of committing one degree of pessimism, conservative they should behave using the observations they sees during online evaluation. Our approach is related to ensemble (Agarwal et al., 2020; Lee et al., 2021; Chen et al., 2021; An et al., 2021) approaches in that they also predict multiple Q-values to model epistemic uncertainty. However, existing ensemble methods train individual Q-values on the same objective, and rely on different parameter initializations. In contrast, each of our Q-values captures a different confidence-level. In addition, standard ensemble approaches do not consider adaptive policies. Recently, APE-V proposes using ensembles to learn adaptive policies that condition on belief over which value function is most accurate (Ghosh et al., 2022). Our approach considers a similar strategy for adaptation, but explicitly parameterizes the value function by the confidence level, introducing a novel training objective for this purpose. In our experiments, we compare to a method that adapts APE-V to our discrete-action benchmark tasks. Jiang & Huang (2020); Dai et al. (2020) propose confidence intervals for policy evaluation at specified confidence-levels We aim to learn a value function across all confidences, and use it for adaptive policy optimization. Finally, distributional RL (Dabney et al., 2017; Bellemare et al., 2017; Dabney et al., 2018b) learns a distribution over values, but only capture aleatoric uncertainty, whereas our focus is on epistemic uncertainty and offline RI...
2

---

Published as a conference paper at ICLR 2023
3 PRELIMINARIES
The goal reinforcement learning is to learn a policy r(-(s) that maximizes the expected cumulative discounted reward in Markov decision process (MDP), which is defined by a tuple A P, R, 7)- S.A represent state and action spaces, P(s'|s, a) and R(s, a) represent the dynamics and reward distribution, and y (0, 1) represents the discount factor. We assume that the reward r(s, a) is bounded in magnitude, i.e., (r(s, a)] S Rmaz for some finite Rmaz- ( represents the (unknown) behavior policy used to collect the offline dataset D that will be used for training, d (s) is the discounted marginal state distribution of IT (als), and the offline dataset D = {(s, r,,')} is formed from interactions sampled from d d (T) (a|s).
Policy evaluation attempts to learn the Q-function Q" : S A of policy at all state-action pairs (s.a) a) S A, Specifically, for policy и, its Q-value (=(s.a) Ez [Σ r're] is its expected mean return starting from that state and action. The Q-function is the unique fixed point [Q(s', a')],
of the Bellman operator 8" given by 8 BTQ((s, a) = r(s, a) + "'Eg''/(''lssss a''x((/))))))) aa meaning Q = _ Q". Q-learning learns Q" = Q" as the fixed point of the Bellman optimality operator 3" given by 1 Q(s, = r(s, a) P (s,))s,a)) |maxa' Q(s', , a')], and derives the optimal policy (a s) I{a arg max, (s, a))
Offline reinforcement learning. In offline RL, we are limited to interactions that appear in the dataset of N samples (s, a, r')', where a E A is derived from some suboptimal behavior policy. Hence, we do not have access to the optimal actions used in the backup of the Bellman optimality operator. Because of this, offline RL suffers from distributional shift (Kumar al., 2019b; Levine et al., 2020). Prior methods address this issue by learning conservative, or lower-bound, value functions that underestimate expected return outside of the dataset. One method to accomplish this to subtract anti-explorution bonuses that are larger for (Rezacifar et al., 2021):
(OOD) states and actions
(Q(s.a)- B*Q - a n(s,a) A1
27
@++1 = arg min Eg, Ega.d'D
Q 2
where a >> 0 is a hyperparameter. Another relevant method is conservative Q-learning (CQL) (Kumar al., 2020), which proposes regularizer the standard objective to learn pessimistic Q-values:
@++1 = arg min max (Ea~Pant(als) (Q(s. a)] E. [Q(s,a)))
+ Ea
(Q(s.a) BB + R(T).
Here, is some policy that approximately maximizes the current Q-function iterate, and R is some regularizer. This objective includes a penalty that ensures Q-values at OOD actions are underestimated compared to in-distribution (ID) actions. Such methods learn lower-bound value functions for fixed confidence-level. that is implicitly captured in hyperparameter a. In this paper, we propose learning value functions that condition the confidence-level explicitly.
Additional notation. Let n A = max(n, 1}. Denote = polylog((S), (1-))))) We let be a polylogarithmic quantity, changing with context.
4
VALUE FUNCTIONS
In this section, we describe our method for learning confidence-conditioned value functions, such that conditioned on some confidence level 8 E (0, 1). the learned Q-function can lower-bound its true value with probability 1 - 8. Because such Q-functions depend not only on state-action pairs, but also the confidence 8, they enable adaptive policies that change behavior based on and adjust delta to maximize online performance. In contrast, pessimistic offline RL is limited to fixed Markovian strategy. We first propose a novel Q-learning algorithm, which we dub confidence-come conditionne value learning (CCVL), then show how such learned Q-function enables adaptive strategies, dubbed confidence adaptive policies. this work, we focus on discrete-actes action environments, but our insights be straightforwardly extended to develop actor-critic algorithms for continuous environments.
3

---

Published as a conference paper at ICLR 2023
4.1 CONFIDENCE- CONDITIIOND VALUE LEARNING
Recall from Section 3 that standard Q-learning involves learning Q-values that satisfy the Bellman optimality update Q B* Q* We are interested in learning confidence-conditonee Q-values, which we define as: Definition 4.1. A confidence-condttioee value function (s,, a, 6) satisfies, for a given d E (0, 1):
Q(s, a, 6) sup q such that Pr[Q" a) > q 1
Note that we include the suprenum to prevent Q(s, a. 8) = Q(s, a. 0) for all other values of The randomness is due noise in dataset sampling, as the dataset is used compute our learned value function. To achieve a high-probability lower-bound on Q" (s( a), we account for two sources of uncertainty: (1) must approximate the Bellman optimality operator, which assumes known reward and transition model, using samples in D, and (2) we need to additionally lower-bound the target Q* used in the Bellman backup. The uncertainty due (1), also called epistemic uncer- tainty, can be bounded using concentration arguments on the samples from D. Namely, we define b(s, a,
5) as a high- high-probabilityy anti-exploration bonus that upper- bounds epistemic uncertainty, or
P ((B'Q'(s,a (s. a) Q* S b(s, a, 6) z 1 8. Such bonuses are well-studied in the prior literature (Burda et al., 2018; Rezaeifar et al., 2021), and can be derived using concentration in- equalities such as Chernoff-Hoeffdiie or Bernstein. Using the former, the bonuses are given by log(1/5)
b(s, a, 5) m(s,ajAi* where n(s, a) is the number of times the state-action pair appears in D. Next, the uncertainty due to (2) can be straightforwardly bounded using our learned Q-function. This gives rise to the iterative update for training the confidence-condittinned Q-function:
[ [clSs..)]
1 @h+z arg min - E..D.DD
log(1/81) max B' @* (s, Vv n(s a) A 81.825
(4)
Q 2
where a > 0 is again some hyperparameter. Theorem 6.1, we show that for any confidence level
5 E (0, 1), the resulting Q-values ((s, a, 6) lime-+o Q* (s, a, d) lower-bounds the true Q-value (s, a) with probability at least
Note that Equation 4 is similar to traditional Q-learning using anti-exploration bonuses, as in Equation 1, but with important differences. conservative Q-learning, the value is not modeled and implicitly captured in the hyperparameter. Equation can be made more similar to Equation 4 by explicitly conditioning on and setting 6, 62 = 6. We believe our approach offers the following advantages compared to using anti-exploration bonuses without conditioning. First, tuning in our approach is easicr as we do not need to commit to a degree of conservatism beforehand. Also, by introducing outer maximization over 51, 62, see that for any iteration & E N, and any 8 E (0, 1). least as tight of lower-bound one that would set
@ (s, a, 5) as the solution to Equation 61 82 = 5. The latter is what Equation implicitly does.
Implicit bonuses via regularization. The objective in Equation requires explicit computation of anti-exploration bonuses, which requires computation of state-action visitations n((s, a) -1 that we discuss in Section objective that is inspired by how CQL achieves pessimistic value functions (Kumar et al., 2020). The key idea is, instead of explicitly subtracting bonus, we can add a regularizer in the objective. Specifically, we have the following iterative update an alternative equation 4:
difficult with neural network value functions. Here, we propose new
log(1/61) @++1 arg min max max / (n(s) A (Ea-D.anT(al)) (Q(s, a, a, EE 1Q(s, a, 8)))
Q 61,528 &
(Q(5) -0- (s)) + R(T).
+ R BRDND
where like in Kumar et al. (2020). R is some regularizer (typically the entropy of T). Note that Equation 5 still relies the computation of n(s) However, we noticed that estimating state visitations is actually much easier than state-action visitations with neural networks: we observed that state-action density estimators were insufficiently discriminative between seen and unseen
4

---

Published as a conference paper at ICLR 2023
actions at given state, although state-only visitations, which do not require estimating densities of unseen samples were a bit more reliable (see Section 5 for details) In Theorem 6.2, we show that the resulting (s,, a, 6) may not point-wise lower-bound Q* (s, a), but will do so in expectation. Specifically, for (s, 6) maxa (s, 6), we have that V (s, 6) lower-bounds the true value V*(s)= = maxa Q" (s, a) with probability at least ó.
The objective in Equation 5 differs from the CQL update in Equation 2 in two notable aspects: (I) we explicitly condition on and introduce a maximization over 81, 82, and (2) rather than a fixed weight of
on the CQL regularizer, the weight now depends on the state visitations. Like with Equation 4, we can argue that (1) implies that for any E N, we learn at least as tight lower-bounds for any d than the CQL update implicitly would. In addition, (2) means that the lower-bounds due the CQL regularizer additionally depends on state visitations in D, which will improve the quality of the obtained lower-bounds over standard CQL.
4.2 CONFIDENCE-ADAPTIO POLICIES
Given a learned Q-function, standard Q-learning would choose a stationary Markovian policy that selects actions according to (a s) II a = arg max ((s, a) }. We can naively do this with the learned confidence-conddttinee Q-function by fixing and tuning especially in offline RL, it can be preferable for the agent to change its behavior upon receiving new observations during online evaluation, as such observations can show that the agent has been behaving overly pessimistic or optimistic. This adaptive behavior is enabled using our confidence-conditionn Q-function by adjusting ó using online observations.
hyper-parameter. However,
Let h be the history of observations during online evaluation thus far. We propose a confidence- adaptive policy that conditions the confidence under which acts on h; namely, we propose a
non-Markovian policy that selects actions #(a 8, h) I(a arg max, Q(s,, a, a(,, 5) where 5 ~ b(h). Here, b(h) is distribution representing the "belief" over which 5 is best to evaluate actions for history h. Inspired by Ghosh et al. (2022), we compute b(h) using Bellman consistency (Xie et al., 2021) surrogate log- agglikelihood. Here, the probability of sampling å under b(h) is:
(8)
Σ
max (Q(s. - r max Qss a', a()
2
(6)
(S.a.):s')E
Note that this surrogate objective is easy update. This leads tractable confidence-adaptive policy गे that can outperform Markovian policies learned via conservative offline RI.
4.3 LEARNING LOWER- AND
natural extension of our method to learn confidence-conditionedd upper-bounds on the true Q- values. Formally, change of notation, let Qe(s, a, 8) be the lower-bounds as defined in Equation 3. We can learn upper-bounde Qu(s, a, 8) as
Qu a, 5) inf 9 s.t Pr[Q" (s, a) S gl 1-
Following analogous logic in Secton 4.1, derive iterative update
[Q(s.a.0) min 8 (s, tay 62525
log(1/81) nls, a) A
@++1 = arg min Q , ED...'D
(8)
Learning both Q and @u presents the opportunity for improved policy extraction from the learned value functions. Instead of simply optimizing the learned lower-bounds, which may lead to overly conservative behavior, we can optimize the upper-bounds but constrained to safe actions whose corresponding lower-bounds are not too low, Formally, our policy can perform
a(a h, = 1 arg max Qu a 6) aEA,
where 8 ~ b(h), and Ar {aa 1e(s, a, 8) 3 max (r(s,
(9)
5

---

Published as a conference paper at ICLR 2023
for some parameter 3 > O. To simplify notation, for the remainder of the paper, again drop the subscript on when referencing lower-bounds. Learning upper-bounds offline is particularly important when fine-tuning the value functions online interactions, which is a natural next-step after performing offline RI.. Existing offline RI. algorithms achieve strong offline performance, but lack the exploration necessary to improve greatly during online fine-tuning. By learning both lower- and upper-bounds, our method can achieve better online policy improvement (see Section 7.3).
5 PRACTICAL ALGORITHM
In this section, we describe implementation details for our CCVL algorithm, and arrive at practical algorithm. We aim to resolve the following details: (I) how the confidence-conditione Q-function parameterized, and (2) how the objective in Equation 4 or Equation 5 is estimated and optimized.
Our Q-function is parameterized by a neural network with parameters 0. To handle conditioning on we build upon implicit quantile networks (IQN) (Dabney al., 20) and propose parametric model that can produce Q(s, a, 6) for given values of 6. Alternatively, of 6, and model our Q-function as an ensemble where each ensemble member corresponds to one fixed 8. We choose the IQN parameterization because training over many different 2(00, 1) may lead to better generalization over confidences. However, when computing beliefs b online, we maintain a categorical distribution over quantized values of
could fix quantized values
In Equation 4 or Equation 5, we must compute the inverse state-action or state visitations. This can be exactly computed for tabular environments. However, non- non-tabular ones, we need estimate inverse counts nss a)- or n(s)- In prior work, O' Donoghue et al. (2018) proposed obtaining linear-value estimates using the last layer of the neural network, ie., n(ss a) extracts state representations, and & is a matrix of d(s;) for states s; E where action was taken. However, we found that such methods were not discriminative enough to separate different actions the dataset from others under the same state. Instead of state-action visitations, the update in Equation 5 requires only estimating inverse state visitations n(s) Empirically, we find that linear estimates such as n(s) states. Hence, we use the latter update when implementing CCVL in non-tabular environments.
((s) ( &b)) Φ(s), where
((s) (Φ -)((((() could successfully discriminate between
Finally, we summarize our CCVL algorithm in Algorithm Note that aside from sampling multiple 5 (00 1) for training. CCVL is more computationally expensive than standard Q-learning, and is on the same order as distributional or ensemble RL algorithms that train on multiple Q-value estimations per state-action pair. Hence, our algorithm is very practical, while enabling adaptive non-Markovian policies as described in Section 4.2.
Algorithm 1
Value Learning (CCVL)
Require: Offline dataset D, discount factor 7, weight a, number of samples N, M Initialize Q-function Qo
2: for step = 1,2,...,, do
3: for L, ......... do
4: Sample confidence ~ (0. 1)
5: For j 1.... M,, sample 5, 1.1,0 6,,2 ~ (0, Compute (0) as inner-term of right-hand
side of equation 4 or equation 5 with 6, 8,,1, 82 = 63,2
6: Take gradient step Og := 0,-1 V, max; (0)
7: Return Q-function Qo
THEORETICAL ANALYSIS
In this section, show that in tabular MDP, the value functions learned by CCVL properly estimate lower-bounds of the true value, for any confidence 8. We show this for both the update using anti-exploration bonuses in equation 4 as well as the one using regularization in Equation 5.
First, we show a simple lemma that CCVL will learn a value function such that the values decrease as the confidence level increases. Formally, we show the following:

---

values with Published as a conference paper at ICLR 2023
Lemma 6.1. The Q-values Q learned via CCVL satisfy, for any 8, E (0.1) such that ò S 8': a(s, 8) Q(s, 8').
Proof. Let 61, 82 S ò be the solution to the maximization for ((s, a, 6) in Equation 4. Since S 8', have 81, 62 S 8'. This implies ((s, a, 6) I Q(s, a, 8'), as desired.
Lemma 6.1 means that as decreases, which equates to estimating a lower bound of higher confidence, our estimated Q-values will monotonically decrease. Using Lemma 6.1 allows us to show the following theorems, which are the main results of this section. We state the results below, and defer proofs to the Appendix A.
The first shows that using when equation 4, our proper lower-bound the optimal Q-values with probability least 1 1 6.
estimates, for any confidence 8 E (0, 1).
Theorem 6.1. For any E (0,1). the Q-values Q learned via CCVL with Equation satisfies
(s., a. 8) (s
a) for all states 8, and actions A
with probability at least 1 6 for some > O.
The second theorem shows an analogous result to Theorem 6.1, but using the update in equation instead. However, using the alternative update does not guarantee pointwise lower-bount on Q-values for all state-action pairs. However, akin to Kumar et al. (2020), we can show a lower- bound on the values for all states.
Theorem 6.2. For any E 1), the value of the policy (ss 6) maxae Q(s,, a, ō), where Q are learned via CCVL with Equation satisfies
((s,8)<V*i)) for all states E S
where (*(s) = aa (s, a) with probability at least 5 for some >> 0.
7 EMPIRICAL EVALUATION
In our experiments, we aim to evaluate our algorithm, CCVL on discrete-action offline RL tasks. We use the iterative update in Equation 5, it achieves stabler performance when the Q-function is a neural network. We aim to ascertain whether the two distinct properties of our method lead to improved performance: (1) conditioning on confidence ō during offline training, and (2) adapting the confidence value 8 during online rollouts. We compare to prior offline RL methods, REM (Agarwal al., 2020) and CQL (Kumar al., 2020), and ablations of our method where we either replace confidence-conditionnn with simple ensemble, which we dub adaptive ensemble value-learning (AEVL), or behave according to fixed confidence online, which we call Fixed-CCVV
Comparisons. REM and CQL are existing state-of- sane--oftte-aate art offline RL algorithms for discrete-action environments. AEVL allows us to study question (1) by replacing suffieencecondditoon random ensemble, where each model in the ensemble roughly has the same level of conservatism. Each ensemble member of AEVL is trained independently using different initial parameters, and which ensemble member to act under is controlled online using Bellman error in proposed method. Note that AEVL can be viewed as a special case of APE-V in Ghosh et al. (2022) for discrete-action domains. Finally, Fixed-CCVL. tests (2) by treating confidence used by the policy as a fixed erper-parameter instead of automatically adjusting during online rollouts. The confidence is selected as the one that minimized Bellman error during offline training. Because AEVL and CCVL change their behavior during evaluation, we maintain a fair comparison by reporting the average score across the adaptation process, including episodes where adaptation has not yet converged.
7.1 ILLUSTRATIVE EXAMPLE ON GRIDWORLD
We first present didactic example that illustrates the benefit of CCVL over standard conservative offline RL algorithms. We consider a x 8 gridworld environment (Fu et al., 2019), with a start and goal state, walls, lava. The reward is upon reaching the goal, but entering lava state results in receiving a reward of 0 for the rest of the trajectory. We consider an offline RL task where the learned policy must generalize to slightly different gridworld environment than the one was trained on.
7

---

Published as a conference paper at ICLR 2023
Dataser Compostion
COL StatesAction Viaitaiions
CCVL State Vistationn
Figure 2: Example gridworld where CQL takes the longer, suboptimal trajectory that appears more frequently in the dataset, but CCVL ultimately adapts ō and takes the optimal one.
In our case, during offline training, the environment is stochastic. in that there is a 30% chance that the agent travels in unintended direction; however, during evaluation, that probability decreases to 15%. This makes previously risky paths more optimal. This is where we anticipate that adaptive methods such as ours will have severe advantage. While CQL will act too conservatively, method CCVL can evaluate and change its level of conservatism on the fly.
We construct an offiin dataset consist- ing of 2.5k samples from a behavior policy, which takes the optimal action with probability 0.5, and a random ac- tion otherwise. In Figure 1, we show
1.0
0.8
0.8
0.7
Nommaized
0.6
Most (lweyy
0.5
0.4
0.4
0.2
0.2
the returns of CQL and CCVL (nor-
CCVL
0.1
malized by optimal return) for various choices of a. We see that because CCVL does not commit to a degree of conservatism beforchand. it does not suffer from overly conservative behavior as CQL does when a 2 0.2. For a = 0.2, we also visualize the process of CCVL adapting 5 over 10 evaluation trajectories, ultimately becoming less conservative. Finally, in Figure 2, we see that for large settings of a, CQL is unable to recover the optimal trajectory-inng sijeetary-instead instead learning the most likely trajectory the dataset- CCVL can.
08.00
0.2
0.4
10 Number of observed online trajectories
0.0
Figure 1: Left. Effect of normalized returns of CQL and CCVL. Right. Adaptation under CCVL.
7.2 OFFLINE TRAINING ON ATARI
Next, we evaluate our algorithm against prior methods on Atari games (Bellemare et al., 2013) with offline datasets of varying size and quality, previously considered by Agarwal et al. (2020); Kumar et al. (2020). We follow the exact setup of Kumar et al. (2022), including evaluating across the same set of 17 games, using the same three offline datasets, with 1% and 5% of samples uniformly drawn from DQN replay dataset introduced in Agarwal al. (2020), as well as more suboptimal dataset consisting of 10% of the initial samples from the DQN dataset (corresponding to the first 20M observations during online DQN). Including this more suboptimal dataset allows to evaluate the degree to which each method can improve over the average performance in the dataset. Following Agarwal et al. (2020), the Atari games have stochastic dynamics. with 25% chance of "sticky actions," Le., executing the previous action instead of new one.
The REM and CQL baselines use exactly the hyperparamter configurations used by Kumar al. (2022). We refer to Table E.1 of Kumar et al. (2022) for a table of hyperparamters used. Across all methods, we found it useful to perform DR3 regularization on the learned state representations (Kumar et al., 2022). Following Agarwal et al. (2021), we report the interquartile mean (IQM) normalized scores, where the normalization gives score 0 to a random policy and 100 to the nature DQN (Mnih et al., 2015), and each score is computed using the average of 100 episodes. We also report 95% confidence intervals (CIs) computed using stratified bootstrapping. The results across all 17 games for the three datasets are in Table 1. We also show complete per-game results in Tables 4-6.
Note that our method CCVL outperforms all baselines that we evaluate against. Though the average improvement across all games is small, see that CCVL sometimes outperforms REM and CQL by over 30% for games such as Asterix or Breakout. We believe this is because REM and CQL can only act according to a fixed level of conservatism across all games, whereas CCVL is able adapt
8

---

Published as a conference paper at ICLR 2023
Data
REM
CQL
AEVL
Fixed-CCVL
CCVL
1%
16.5
56.9
15.2
56.2
59.1
(14.5, 18.6)
(52.5, 61.2)
(53.0, 60.8)
(52.0, 61.4)
(51.8, 65.6)
5%
60.2
105.7
57.2
105.9
110.1
(101.9, 110.9) 73.8
(55.8, 65.1)
(50.9, 63.6)
(102.3, 109.9)
(101.2, 117.4)
Initial 10%
65.8
75.3
64.7
77.8
(69.3, 78) (63.3, 68.3)
(68, 79.5)
(62.7, 67.9)
(69.1, 87.2)
Table 1: Final performance across 17 Atari games after 6.25M gradient updates on 1% data and 12.5M for 5%, 10% terms normalized IQM across random seeds, with 95% stratified bootstrap Cls in parentheses. REM and CQL results are from Kumar et al. (2022). Our method CCVL outperforms prior baselines and ablations across all three datasets.
its level on a per-game basis. We also notice that CCVL outperforms both ablations, showing that both aonfidence-conditionnnn and adaptation are important to the success of our algorithm. Though AEVL is adaptive, because the ensemble members do not represent diverse hypotheses about how to act optimally, adaptation is not useful. Perhaps unsurprisingly, Fixed-CCVL and CQL perform similarly due to the similarities in the objective in Equation 5 and Equation 2. However, CCVL greatly improves over Fixed-CCVL due to being able to adapt the used by the policy online.
7.3 ONLINE FINE-TUNING ON ATARI
often realistic consider that the value functions obtained by offline RL can be improved additional online interactions, which we call online fine-tuning. Our CCVL method, when extended learn both lower- and upper-bounds bounds as discussed in Section 4.3, is well-suited for this setting. This is because our approach can leverage lower-bounds to act pessimistically offline, while using upper-bounds bounds for online exploration. Note that these experiments include additional training with online RI for all methods. Like in previous experiments, all methods receive the same exact amount of data, but must now perform online exploration themselves.
We select 5 representative Atari games, similarly considered in Kumar et al. (2020). We first run offline training across all algorithms on the 1% dataset for 6.25M gradient steps, then run 625k steps of online RL, and report the final performance. We report the gain in normalized IQM after online fine-tuning in Table 2. Our method, CCVL,, achieves the best score across of 5 games. Though CQL often achieves the second best overall score, it often sees the smallest improvement, as conservatism is detrimental to exploration.
Game
REM
CQL
CCVL
Asterix
4.3 -> 45.2
18.6 52.7
25.9 159.5
Breakout
1.2 + 204.2
2+8
- 193.7
2.7 + 202.7
Pong
36.4 + 113.4
100.0 111.6
105.2 + 117.9
Seaquest
13.9 -.9--522 51.2
24 + 60.7
30.9 + 77.8
Qbert
3.4 120.4
111.2 118.9
111.3 139.7
Table 2: Improvement normalized IQM final performance after 625k additional gradient steps of online fine-tuning.
CONCLUSION
In this work, propose coffddence-conditioonne value learning (CCVL), offline RL algorithm that learns a value function for all degrees of conservatism, called confidence-levels. Contrary to standard offline RL algorithms like CQL that must specify a degree of conservatism during training via hyperparameter tuning, CCVL enables condition-adaptivn policies that adjust this degree using online observations. CCVL can be implemented practically, using slight modifications on top of existing offline RL algorithms. Theoretically, we show that in a tabular enviromment, CCVL, for any confidence-level learns appropriate value that is a lower-bound that confidence. Empirically, we demonstrate that in discrete-action environments, CCVL performs better than prior methods. We view CCVL as first-step in proposing conservative offline RL algorithms that adjust their level of conservatism, rather than having the level tuned beforehand via opaque hyperparameter. Many angles for further investigation exist. Theoretically, it remains to see whether the confidence- conditioned values fower-bounde under function approximation. Algorithmically. direction of future work is to extend CCVL to continuous-actoo action environments, which would involve developing actor-critic algorithm using confidence-conditione policies.
important

---

Published as a conference paper at ICLR 2023
ACKNOW
We thank the members of RAIL. at UC Berkeley for their support and suggestions. We thank anonymous reviewers for feedback on early version of this paper. This research is funded in part by the DARPA Assured Autonomy Program, the Office Naval Research, and in part by compute resources from Google Cloud.
REFERENCES
Mohammad Mehdi Afsar, Trafford Crump, and Behrouz H. Far. Reinforcement learning based recommender systems: A survey, CoRR, 0as/2101.06286 2021.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. In Advances in Neural Information Processing Systems, 2021.
Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. J. Artif. Int. Res., 47(1 (7(1):253-279) May 2013. ISSN 1076-9757.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Hoarning-Volumee 70, pp. 449-458. JMLR. org, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 2018.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double gelaamnnn:: Learning fast without a model. In International Conference on Learning Representations (ICLR), 2021.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. arXiv preprint urXiv:1710. 10044, 2017.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. arXiv preprint 1eXivv1806. 06923, 2018a
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018b.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. In Advances in Neural Information Processing Systems, 2020.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Q-learning algorithms. arXiv preprint 1902 2019.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforeement learning. arXiv preprint 1xXix:2106.06860 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 18/2.02900 2018.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, 2022.
10

---

Published as a conference paper at ICLR 2023
Nan Jiang and Jiawei Huang. Minimax confidence interval for off off-policy evaluation and policy optimization. In Advances in Neural Information Processing Systems, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.0555 05955, 2020.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, pp. 11761-11771, 2019a.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off policy q-learning via bootstrapping error reduction. 2019b. URL
wttp:/ggggg/////eeee/ 1906. 00949.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint WrXiv:2006.047799 2020.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, George Tucker, and Sergey Levine.
DR3: value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations (ICLR), 2022.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, volume 12. Springer, 2012.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint wrXiv:2005.01643 2020.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. CoRR, abs/1904.088. 08473,, 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint arXiv:2007.08202. 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprins 1rXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):5295533 533, 2015.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02077 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint w:Xiv:2006.09339, 2020.
Brendan O' D'Dnogguue Ian Osband, Rémi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning. 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. serantage-weightee. regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 190.00/1900000....... 2019.
11

---

Published as a conference paper at ICLR 2023
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning; tale of pessimism. arXiv preprint arXiv:2103.12021, 2021.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. CoRR, abs/2106.06431, 2021.
Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan Murphy, Informing sequential clinical decision-making through reinforcement learning: empirical study. Machine learning, 14(1-2):109-133 2011.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprini arXiv:2002.0833 20099, 2020.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Mach. Learn. Res, 16:1731-1755, 2015.
Wang, Wei Zhang, Xiaofeng He, and Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offiine reinforcement learning. arXiv preprint arXiv: /91 I.I 1361, 2019.
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consiste pessimism for offline reinforcement learning. arXiv preprint, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.
Combo: Conservative offline model-based policy optimization. arXiv preprint arXi::2102.08338 2021.
12

---

Published as a conference paper at ICLR 2023
A PROOFS
In this section, we provide proofs of theorems stated Section 6. Recall from Section that
== polylog((S), -)= ,) is some constant. Our proofs rely the following lemma, which bounds the estimation error due to using the empirical Bellman operator:
Lemma A.I. For all state-action (s, S A such that n(s, a) 2 L. function Qu and ò E (0, 1), we have:
(og(115) nlss a)
IP
B [(s.a)) Q(s
2 - d.
The above lemma is well-known result in reinforcement learning (Rashidinejad et al., 2021), whose derivation follows from Hoeffding's inequalities.
A.1 PROOF OF THBOREM 6.1
Without loss of generality, assume that 5,, 82 S are the solution to the outer maximization of Equation convergence. Using Lemma A.I, we have that
log(1/61) n(s, a) A
@(s, (3) -
log(1/81)
+ 1og(1151)
(B,Q((s,,,,)) 1
< <B'Q(s, S, n(s,a) a)
n(ss a) A
holds with probability at least 61 for any 2 /// Using Lemma 6.1, have
Q s -I-
= Q a) (s. S
holds with probability at least 1 - 61 2 1 - 8, as desired.
A.2 PROOF OF THEOREM 6.2
Recall from Equation that at convergence, we have,
log(1/51)
((s,, a, 8) = arg min max max a
(n(s) A (Fe-D,awwa(al) (Q(s, a, a. 8)] Rx [Q(s, a. 5)))
Q 61,82
()(s.a ((s, -
+ Eg B.
R(T))
log(1/51)
max max arg mina
(n(s) A 1) (Fea.D, (Q(s, a, 5)] Ex [Q(s, a, 5)))
61,82
Q
BR [Q(s. 8 Q(,, -
- (()) R(TT)
For any 61, S and n, we have that the solution to the inner-minimization over Q yields
log(1/6;)
Q(s, a, 6, arg argmin V (n(s) A 11 (Ex-D.a [Q(s, a, 8)] Eg [Q(s, a, 6)])
2 B..aod (o(s. a B- @ a, 52) 27
10g(1/61)
a(aa
S 8 Q(s, a, 02) - va
n((s)
T (a |s)
13

---

Published as a conference paper at ICLR 2023
This arises from taking the derivative of the minimization objective, and solving for Q that makes the derivative equal to O. Note that we can simplify
log( (/1) n(s)
π s) (a S)
log(1 (1)) n(s)
TT 8)
T 8) To(a |s)
= ov
log(1/61) n(s, a)
T 8)
a S)
(a
Without loss of generality, assume that 51, 62 S 8 and æ are the solution to the outer-maximization Substituting the previous result into the equation for Q(,,, a, 6), and applying Lemma A.1 yields,
10g(1/51) a(s.a)
π (a 8)
T |s)
((s, a) B as Vv
n s) xaa 5) TT S) na s)
log(1/61) S ((s Vv n(ss a)
& llog(1/61)
n(ss a)
Note that the middle term is not positive if x(a a' arg max, (s,, a, 8) then raa s) 2 (a Q-values. Therefore, we have
8) T a s). However, we know that for s) by definition of maximizing the learned
(0g(1/)1) a*)
aTaa 8) TT(a" 18) +
(og(1/)1))
(s,,, 8) = a(,,a",)) 8. a(s.aa,))) -
VTa 8))
nts,
< v(, E
holds with probability at least 61 for satisfying
(a 8) TT((a 8)
a 2 1/2 max 8,n
v T s)
Then, using Lemma A.I, we have
(s, < Br B = 0( As E
holds with probability at least - 61 2 1 - ō. as desired.
14

---

Published as a conference paper at ICLR 2023
B ATARI RESULTS
In this section, we provide per-game results across all Atari games that we evaluated on for the three considered dataset sizes. As mentioned in the main paper, we use the hyperparameter configuration detailed in Kumar et al. (2022) for our Atari experiments. For completion, we also reproduce the table in this section.
Table 3: Hyperparameters used by the offline RL Atari agents in our experiments. We follow the setup of Agarwal et al. (2020); Kumar et al. (2022).
Hyperparameter
Setting (for both variations)
Sticky actions
Yes
Sticky action probability
0.25
Grey-scaling
True
Observation down-sampling
(84, 84)
Frames stacked
Frame skip (Action repetitions)
Reward clipping
[-1, I]
Terminal condition
Game Over
Max frames per episode
108K
Discount factor
0.99
Mini-batch size
32
Target network update period
every 2000 updates
Training environment steps per iteration
250K
Update period every
4 environment steps
Evaluation
0.001
Evaluation steps per iteration
125K
Q Qentwwrkk: channels
32, 64, 64
netwwrrk: filter size
8 x 8, 4 X 4, 3 x
Q network: stride
2,
Q netwoork: hidden units
512
Game
REM
CQL
AEVI
FiXed-CCVI.
CCVL.
Asterix
405.7 + 46.5
821.4 75.1
421.2 67.8
874.0 + 64.3
1032.1 86.7
Breakout
2.8
3.2
7.4
28.7 2.8
31.2
Pong
6.3
1..2
-.84
14.7 3.8
15.8 + 4.4
Seaquest
293.3 191.5
446.6 26.9
320.6 154.1
422.0
551.2 42.2
Qbert
436.3 111.5
9162.71193..0 993.6
294.6-1 100.3
9172.31190 907.6
9170.1 1023.5
SpaceInvaders
206.6
77.1
224.2.18..7
355.71 80.2
355.4 81.1
Zaxxon
1726.4
1757.4
2467.8
1747.6 894.3
2273.6 1803.1
YarsRevenge
4460.
16011.3+1409.
4857.1 1012.6
15890.7 1218.2
20140.5 2022.8
RoadRunner
3872.9 1616.4
74928.774484.5
4
22590.3 + 6860.2
26780.5 10112.3
MsPacman
1275.1 + 345.6
2
1
2542.3 188.4
2673.2 226.1
BeamRider
522.9 42222
25.1
600.1 57.3
645.3 40.1
630.2 37.8
Jameshond
157.644635
+ 102.0
114.3 56.7
462.1 + 98.4
452.1 153.9
Enduro
132.4 16.1
24.22
103.2 10.1
244.8 20.9
274.5 23.8
WizardolWor
1663.7 1 417.8
904.6 343.7 343.7
1640.7 383.4
1488, 450.9
1513,8 1 652.1
IceHockey
5.1
-7.8
10.4
-7.6 1.1
-7.1 1.6
DoubleDunk
-1.66 1.5
- 2.8
- 16.8 + 2.9
14.1 1.8
-3.44
DemonAttack
162.0 + 34.7
386.2 + 75.3
183.2 + 44.7
372.9 + 81.7
570.3 + 110.2
Table 4: Mean and standard deviation of returns per Atari game across 5 random seeds using 1% of replay dataset after 6.25M gradient steps. REM and CQL results are from Kumar et (2022).
15

---

Published as a conference paper at ICLR 2023
Game
REM
CQL
AEVL
Fixed-CCVL.
CCVL
Asterix
2317.0 838.1
3318.5 + 301.7
1958.9 + 1050.2
3256.6 395.1
5517.2 -: 1215.4
Breakout
33.4 4.0
166.0 + 23.1
1..7
150.3 + 17.8
172.5 35.6
Pong
-0.7 9.9
17.9 1.1
4.77
17.6
1744 2.8
Seaquest
2 1119.7
2030.7 822.8
2853.0 1089.2
2112.5 856.4
2746.0 1544.2
Qbert
9605.6 1593.5
4 3256.6
9750.7 1366.8
10108.1 2145.5
SpaceInvaders
143.5
2
1243.1 269.8
11516:30022
Zaxxon
1609.7 1814.1
4250.1 626.2
1678.2 1425.6
1060.3 673.1
6470.2 1512.2
YarsRevenge
16930.1 2625.8
17124.7 2125.6
17233.5 2590.8
18040.5 + 1545.9
19233.0 1719.2
RoadRunner
46601.6 2617.2
38432.6 + 1539.7
4 382330
37945.7 + 1338.9
42780.5
41122. 4112.3
MsPaeman
=
2790.6 + 353.1
2148.8 273.4
2501.5 201.3
2680.4 212.4
BeamRider
674.8
785.8
662.9 50.7
782.3 34.9
780.1
Jameshond
130.5 45.7
06.8+4..2
152.2 58.2
112.3 + 81.3
172.1 153.9
Enduro
1583.9 108.7
938.5 + 63.9 63.9
1602.7 135.5
913.2 50.3
1376.2 203.8
WizardOfWor
2
612.0 343.3
1767.5 462.1
707.4+323.2
2723.1 515.6
IceHockey
3.1
-1..0
-9.1 4.8
-1766
-10.2 2.1
DoubleDunk
2.6
16.2 1 1.7
15.2
3.8
DemonAttack
5602.3 +: 1855.5
8517.4 1065.9
2455.3 + 1765.0
8238.7 1091.2
9730.0 +: 1550.7
Table Mean and standard deviation of returns per Atari game across 5 random seeds using 5% of replay dataset after 12.5M gradient steps. REM and CQL results are from Kumar et al. (2022).
Game
REM
CQL
AEVL
FiXed-CCVL
CCVL
Asterix
5122.9 + 328.9
3906.2 +: 521.3
7494.7 + 380.3
3582.1 +: 327.5
7576.0 +: 360.2
Breakout
96.8 + 21.2
70.8 + 5.5
97.1 35.7
75.8 + 6.1
121.4
Pong
7.6
5.5 6.2
7.1 12.9
6.0
13.4
Seaquest
9 + 605.9
1313.0 220.0
877.2 750.1
1232.6 379.3
121144 437.2
Qhert
4126.2 495.7
+
4713.6
5105.5 986.4
4590.9+21114
SpaceInvaders
28.3
938.1 80.3
692.7+101.9
+
1 103.1
7axxon
0.0
836.8 434.7
902.5 895.2
900.1+560.1
1212.2 902.1
YarsRevenge
11924.8
12413.9 2869.7
12508.5
11587.2 2676.8
12502.6 2349.2
RoadRunner
4 1887.9
45336.9 1 1366.7
50152.9
2208.9
44832.6 1329.8
47972.1 2991.3
MsPacman
2268.8 455.0
2427.5 191.3
2515.5
2115.3 1: 108.9
2015.7 1 352.8
BeamRider
4154.9
3468.0
4564.7
3312.3 + 247.3
3781.0 401.8
Jamesbond
-49.3-3044.
89.7
91.9 20.2
152.8 = 42.8
Enduro
1 81.5
959.2 100.3
1204.6 90.3
1585.0 = 102.1
WizardO/Wor
10.0044970
761.7 250.0
1184.3 588.9
749.3 + 231.8
1429.9 + 751.4
IceHockey
-5.9 5.1
-16.0 +: 1.3
-5.2 4.3
-11.9 = 2.5
1.1 + 5.9
DoubleDunk
19.5 2.5
20.6 + 1.0
19.2 2.2
-1.3 1.7
24.6 6.2
DemonAttack
9674.7 1600.6
7152.9 723.2
10345.3 1612.3
7416.8 + 1598.7
12330.5 = 1590.4
Table 6: Mean and standard deviation of returns per Atari game across 5 random seeds using initial 10% of replay dataset after 12.5M gradient steps. REM and CQL results are from Kumar al. (2022).
16


Reviews:
[{"review_id": "OG08bQeAsGD", "reviewer": "Reviewer_Fh1m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new offline RL algorithm, by learning the value function conditioned on the confidence level. The proposed algorithm learns a Q function parameterized by the confidence level of future backup and executes an adaptive policy during test time. The paper shows the learned Q function as a high-confidence bound of the optimal Q function. Empirical results in Atari games show the benefit of the proposed algorithm over CQL and REM.", "summary_of_the_review": "A good paper with several novel ideas. Some clarity issues and additional studies need to be settled before acceptance.", "strength_and_weaknesses": "Strength:\n1. The idea of confidence-conditioned value function learning is a very novel way of encoding confidence levels into offline reinforcement learning algorithms. This is different from all previous papers using explicit or implicit confidence-based penalty in Q learning, as CCVL realize a more ambitious learning goal of learning Q at all confidence level.\n2. At the test time, CCVL executes a non-Markovian policy which adaptively changes the confidence level \\delta. This gives an additional capacity for test time learning/tuning for the policy. It is also a novel contribution of this paper.\n\nWeakness:\n1. Additional ablation study and more description of methods in existing ablation studies.\n - The implementation use Eq (5) instead of Eq (4), what will be the performance of CCVL with the update rule from Eq (4)?\n - A more detailed description of how AEVL and Fixed-CCVL are implemented should be included in the appendix.\n2. The proposed algorithm requires additional sampling or computation. It would be better to conduct an analysis of how much these parts cost additional computation compared with baselines.\n - Eq (5) need to hold for all \\delta. So the CCVL needs to perform the learning over all \\delta or sample \\delta enough times (as in Alg 1) for convergence. \n - Confidence adaptive policy requires more computation at the test time as well.\n\n\nMinor comments:\n1. In section 5, it is unclear why n(s) is approximated by phi(s)^T phi(s). According to the argument in O’Donoghue et al. (2018), ignoring the difference between actions, it should still be phi(s)^T \\Lambda^{-1} phi(s), where \\Lambda is the empirical feature covariance matrix.\n2. Eq (4) and Eq(5) have ambiguity around the operator max_{\\delta_1, \\delta_2}. There should be a parenthesis to clarify the scope.\n3. \\hat{Q} is not well defined without showing the limit exists or contraction, as the Bellman operator is modified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N28OVtBE71w", "reviewer": "Reviewer_gE4Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To solve the distribution shift problem in offline reinforcement learning of conservative methods, the paper let value functions learn on different degrees of conservatism (optimism/pessimism) instead of a fixed degree of conservatism. The paper proposes an algorithm, called CCVL, to achieve its goal in discrete-action environments.\n\nThe major contribution of the paper shows the shift from fixed conservatism to flexible conservatism improves performance on some discrete control domains.\n", "summary_of_the_review": "I really like the idea of using flexible or dynamic level of conservatism/optimism/pessimism in RL. I feel there is an ongoing trend in this direction, in both online and offline RL. However, because of the flaws in the empirical study of the paper, I recommend the paper to be rejected at this point.", "strength_and_weaknesses": "This paper has the idea to learn confidence-conditioned value fundction for adaptive policy optimization. The idea is the strength of the paper. However, there are flaws in the experimental design and experiment discussion, which make it impossible to thoroughly access the new idea.\n\nProblems (ordered by appearance in the paper)\n1. Two important terms, epistemic and aleatoric uncertainty, are unexplained in Section 2. It would be better for the paper to include the definition especially when the paper tries to emphasize its work on epistemic uncertainty.\n\n    1.1 The last sentence of Section 2 gives a false impression that studying epistemic uncertainty is better than aleatoric uncertainty. Focusing purely on epistemic is no better or worse than focusing purely on aleatoric uncertainty. These are two different areas. The paper should address the difference of epistemic (parametric) vs. aleatoric (intrinsic) uncertainties.\n\n    1.2 Could the paper also include a comparison between uncertainty-based work and confidence-conditioned work (this paper)?\n\n2. Last sentence (\"existing Markovian policies that can only act according to a fixed level of pessimism\") of Section 4.2 is a false claim, which shows the paper does not cover a thorough literature review. One NeurIPS paper (Tactical Optimism and Pessimism for Deep Reinforcement Learning, https://arxiv.org/abs/2102.03765) uses the idea of dynamically apply the level of conservatism (optimism/pessimism).\n\n3. The Gridworld example in Section 7.1 is unconvincing. The idea of the setup is similar to the Cliff Walking example in Rich Sutton's RL book, but the reward dynamics is weird: entering a lava state (a dangerous state) only receives 0 reward for the remaining trajectory and without further penalty.\n\n    3.1 Should the paper use a large negative reward for dangerous states instead of 0?\n\n    3.2 The paper only compares the optimal path between CQL and CCVL. However, a shorter path does not always mean a good path (like the SARSA vs. Q-learning performance in the Cliff Walking experiment). It would be more convincing if the paper includes a comparison plot of CQL and CCVL on cumulative reward.\n\n4. Missing study on weight alpha. The paper changes from tuning conservatism \"delta\" to tuning weight \"alpha\" from my understanding. I believe the alpha is important in CCVL. As in Section 4.1, why tuning hyperparameter \\alpha in CCVL is easier (or better in any aspects) than tuning an \"opaque\" hyperparameter (degree of conservatism)? Could the paper clearly explain or elaborate its statements?\n\n    4.1 Also missing alpha in Table 3: Hyperparameters setup\n\n5. As in Table 4 - 6 in Appendix B., the CCVL shows a level of minor to moderate improvement in the majority of the selected Atari games. However, the variance of CCVL is way larger than CQL in all games in Table 4 settings, and way larger than CQL in some games in Table 5-6 settings. Given the level of improvement of CCVL, I believe the introduction of large variance is unacceptable.\n\n    5.1 The paper should include an explanation of the large variance.\n\nMinors:\n1. repeated article \"a\" in the first paragraph of Section 1\n2. \"delta\" -> \"\\delta\" in the first paragraph of Section 4 \n3. mixed use of symbols of \"Pr\" vs. \"\\mathbb{P}\" in Section 4\n4. References order is confusing, making it hard to follow. It's neither alphabetically ordered nor first-appearance ordered. May I ask what is the reference setting?\n\nQuestions:\n1. For Theorem 6.1, what is the drawback of only showing lower-bound on states than state-action pairs?\n2. In Appendix A.2, could the paper include more detailed steps of solving inner-minimization over Q while proofing Theorem 6.2?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JbTGVzpXNSD", "reviewer": "Reviewer_DkfW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CCVL (confidence conditioned value learning), an offline RL method that learns value functions parameterized by the \"confidence level\". The main algorithm is adapted from CQL and anti-exploration bonus, with explicit modifications (supported by theory) to guarantee that the learned Q-values are lower (or upper) bounds with the probability at least being the selected \"confidence level\". Experiments were conducted in a toy grid-world problem as well as Atari benchmark and shows favorable performance compared to CQL and other baselines. ", "summary_of_the_review": "Overall the paper is well-written with good methodological contribution and theoretical analysis, supported by benchmarking results. Some theoretical details can be clarified. ", "strength_and_weaknesses": "**Strengths:**\n- Problem is well-motivated, mathematical descriptions and derivations are detailed. \n- I really like the toy gridworld problem, which is illustrative of the expected effect of using the proposed algorithm. \n\n**Weaknesses:**\n- Some theoretical details need clarification:\n  - Could you clarify the proof for Lemma 6.1, I don't think I follow why $\\delta_1,\\delta_2 \\le \\delta'$ \"implies $Q(s, a, δ) ≤ Q(s, a, δ' )$, as desired\". \n  - How does setting $\\delta_1 = \\delta$ in Eqn (4) reduce it to Eqn (1), the numerator of the fraction under the sqrt is $\\log(1/\\delta)$ instead of 1\n- Is $\\alpha$ still a hyperparameter, how should it be chosen, and what is its interpretation now that $\\delta$ \"confidence\" is introduced. Since in CQL $\\alpha$ is used to implicitly control the confidence level in the pessimistic Q-values. \n- It's best to make clearer how much of the observed performance gains in experiments is from the proposed learning confidence-conditioned values, vs from some of the empirical/engineering decisions in Sec 5 practical algorithms (IQN, approx inverse visitation). Perhaps if there's a simpler (maybe tabular?) problem that doesn't have these deep RL complexities that would strengthen the results and help the reader understand the contribution better - and help to show what the learned Q-values are actually doing. \n- There are some typos throughout, \"hyperparamters\" and \"suprenum\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0FL7nVHTQaI", "reviewer": "Reviewer_pdN5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel approach to estimating lower bounds of $Q$-functions. Based on the approach, the authors propose CCVL and empirically justify that CCVL outperforms several offline RL SOTAs.", "summary_of_the_review": "The paper seems interesting but there are a few unclear points to me. I will raise my score if the authors can address my questions.", "strength_and_weaknesses": "$\\textbf{Strength:}$\n\nThe topic of offline RL is essential to the field. The quantile estimation and confidence adaptation proposed in this paper are novel to me and perform well empirically.\n\n\n$\\textbf{Weaknesses and Questions:}$\n\nThere are a couple of unclear points to me. \n$\\textbf{Q1:}$\nThe authors may want to specify the randomness source in equation (3), as $Q^*(s, a)$ is itself deterministic, so the probability that $Q^*(s, a)>q$ is either $0$ or $1$. I assume that the probability bound means the bound on some estimated $Q^*$ given the dataset, or is there a typo in (3)?\n\n$\\textbf{Q2:}$\nWhich objective ((4) or (5)) does the authors adopt in the empirical comparisons? Is the policy in (9) adopted in empirical comparisons? Does the uniformly sampled $\\delta$, $\\delta_1$, and $\\delta_2$ affect the lower bound estimation in Algorithm 1, as the right-hand side does not align with (4)?\n\n$\\textbf{Q3:}$\nIn addition, the theorem feels insufficient to me. What is the benefit of utilizing different confidence levels in the target estimation of (4)? Can the authors justify their benefit compared with standard pessimistic methods in (1)? \n\n$\\textbf{Q4:}$\nIn the grid world experiments, how does CCVL learns the optimal trajectory given that there seems to be no coverage of such trajectory in the dataset (illustrated in the leftmost figure of Figure 1)? Is the performance of CCVL desirable, as offline RL is supposed to rely more on confident regions from the dataset?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%