Conservative Offline Distributional Reinforcement Learning
Authors: Yecheng Jason Ma
Dinesh Jayaraman
Osbert Bastani
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
offline
risk-sensitive
distributional
conservative

Abstract: Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with different actions. In the online setting, distributional RL algorithms do so by learning the distribution over returns (i.e., cumulative rewards) instead of the expected return; beyond quantifying risk, they have also been shown to learn better representations for planning. We propose
Conservative Offline Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both risk-neutral and risk-averse domains. CODAC adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. We prove that CODAC learns a conservative return distribution---in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator. In our experiments, on two challenging robot navigation tasks, CODAC successfully learns risk-averse policies using offline data collected purely from risk-neutral agents. Furthermore, CODAC is state-of-the-art on the D4RL MuJoCo benchmark in terms of both expected and risk-sensitive performance. 

Conservative Offline Distributional Reinforcement Learning
Yecheng Jason Ma, Dinesh Jayaraman, Osbert Bastani
University of Pennsylvania
{jasonyma, dineshj, obastami)0seas upenn. edu
Abstract
Many reinforcement learning (RI.) problems in practice offline, learning purely from observational data. A key challenge how to ensure the learned policy is safe, which requires quantifying the risk associated with different actions. In the online setting, distributional RL algorithms do so by learning the distribution over returns (i.e., cumulative rewards) instead of the expected return; beyond quantifying risk, they have also been shown to learn better representations for planning. We propose Conservative Offline Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both rest-neutral and risk-averse domains.
CODAC adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. We prove that CODAC learns conservative return distribution-ing particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator. In our experiments, on two challenging robot navigation tasks, CODAC successfully learns risk-averse policies using offline data collected purely from risk-neutral agents. Furthermore, CODAC is state-of- tftte-of-the-art the D4RL MuJoCo benchmark in terms of both expected and risk-sensitive performance. Code is available at: https///ghhhbbbcoom//////o///000
Introduction
In many applications of reinforcement learning, actively gathering data through interactions with the environment can be risky and unsafe. Offline (or batch) reinforcement learning (RL) avoids this
problem by learning policy solely from historical data (called observational data) 9 22 2x]
shortcoming of most existing approaches offline RI. designed to maximize the expected value of the cumulative reward (which we call the return) of the policy. As a consequence, they are unable to quantify risk and ensure that the learned policy acts in safe way. In the online setting, there has been recent work on distributional RL algorithms 6 27 3 [7], which instead learn the full distribution over future returns. They can use this distribution to plan in way that avoids taking risky, unsafe actions. Furthermore, when coupled with deep neural network function approximation, they can learn better state representations due to the richer distributional learning signal [4 226,, enabling them to outperform traditional RL algorithms
46 20
21
48 18]
is that they are
even on the risk-newt neutrallral, expected return objective 7 6 47 [4].
We propose Conservative Offline Distributional Actor-Critic (CODAC), which adapts distributional RL to the offline setting. key challenge in offline RL is accounting for high uncertainty on oui-of-distribution (OOD) state-action pairs for which observational data is limited 223 20]; the value estimates for these state-action pairs are intrinsically high variance, and may be exploited by the policy without correction due to the lack of online data gathering and feedback. We build on conservative Q-learning [21], which penalizes Q values for OOD state-action pairs to ensure that
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the learned Q-function lower bounds the true Q-function. Analogously, CODAC uses a penalty to ensure that the quantiles of the learned return distribution lower bound those of the true return distribution. Crucially, the lower bound is data-driven and selectively penalizes the quantile estimates of state-actions that are less frequent the offline dataset; see Figure [
We prove that for finite MDPs, CODAC converges to an es- timate of the return distribution whose quantiles uniformly lower bound the quantiles of the true return distribution; in addition, this data-driven lower bound is tight up to the approximation error in estimating the quantiles using finite data. Thus, CODAC obtains uniform lower bound on all integrations of the quantiles, including the stan- dard RI. objective of expected return, the risk-sensitive iaaiiiional-vllu--t---ee
CODAC
[(als)
?tyaas))
F F
(Z) (T)
(CVaR) objective [35). as well as many other risk-sensitive objectives. We additionally prove that CODAC expands the gap in quantile estimates between in-distribution and OOD actions, thus avoiding overconfidence when extrapolating to OOD actions [[]]
Figure 1: CODAC obtains conservative estimates of the true return quantiles (black); it penalizes outoo--distribution actions, uu(a s), more heavily than in- distribution actions, xz(aa s).
Our theoretical guarantees rely on novel techniques for analyzing the distributional Bellman operator, which is challenging since it acts on the mnfnite-dimensional func- tion space of return distributions (whereas the traditional
Bellman operator acts on a finite-dimensional vector space). We provide several novel results that may be of independent interest; for instance, our techniques can be used to bound the error of the fixed-point of the empirical distributional Bellman operator; see Appendix
Finally, to obtain a practical algorithm, CODAC builds on existing distributional RL algorithms by integrating conservative return distribution estimation into a quantile-based actor-critic framework.
In our experiments, we demonstrate the effectiveness of CODAC on both risk-sensitive and risk- neutral RL. First, on two novel risk-sensitive robot navigation tasks, show that CODAC suc- cessfully learns risk-averse policies using offline datasets collected purely from a risk-neutral agent, challenging task that all baselines fail to solve. Next, the D4RL Mujoco suite [TO], popular offline RL benchmark, we show that CODAC achieves state-of-art results on both the original risk-neutral version well a modified risk-sensitive version [3). Finally, we empirically show that CODAC computes quantile lower-bounds and gap-expanded quantiles even on high-dimensional continuous-control problems, validating our key theoretical insights into the effectiveness of CODAC.
The key challenge in offline RL is to avoid overestimating the value of duitt--disttibutton (OOD) actions rarely taken in the observational dataset 440 44 20). The problem is that policy learning optimizes against the value estimates; thus, even if the estimation error is i.i.d., policy optimization biases towards taking actions with high variance value estimates, since some of these values will be large by random chance. In risk-sensitive safety-critic settings, these actions are exactly the ones that should be avoided.
Related work. There has been growing interest in offline (or batch) RL 22 23
One solution is to constrain the learned policy to take actions similar to the ones in the dataset (similar to imitation learning) by performing support matching [46] or distributional matching [20 12]
However, these approaches tend to perform poorly when data is gathered from suboptimal policies. An altermative is to regularize the Q-function estimates to be conservative at OOD actions 21 48 [8]
CODAC builds on these approaches, but obtains conservative estimates of all quantile values of the return distribution rather than just the expected return. Traditionally, the literature on off- off-policy evaluation (OPE) [32 16. (39 25 37) aims to estimate the expected return a policy using pre- collected offline data; CODAC proposes an OPE procedure amenable to all objectives that can be expressed as integrals of the return quantiles. Consequently, our fine- tine-granee approach not only enables risk-sensitive policy learning, but also improves performance on risk-neutral domains.
In particular, CODAC builds on recent works on distributional RL 4 7 6. 47,, which parameterize and estimate the entire return distribution instead of just point estimate of the expected return (i.e., the Q-function) 29 28). Distributional RL algorithms have been shown to achieve state-of-art performance on Atari and continuous control domains 33; intuitively, they provide richer training signals that stabilize value network training 44) Existing distributional RL algorithms parameterize

---

the policy return distribution in many different ways, including canonical return atoms 4. the expectiles [36], the moments [3I. and the quantiles 7 6 47 CODAC builds on the quantile approach due to its suitability for risk-sensitive policy optimization. The quantile representation provides an unified framework for optimizing different objectives of interest 77. such as the risk- neutral expected return, and a family of risk-sensitive objectives representable by the quantiles; this family includes, for example, the Wang measure [45], and the cumulative probability weighting (CPW) metric 42 Recent work has provided theoretical guarantees on learning CVaR policies [[7]; however, their approach cannot provide bounds the quantiles of the estimated return distribution, which is significantly more challenging since there is no closed-fome fored-foommmo expression for the Bellman update on the return quantiles.
(CVaR) return [35 [35 38 [7]. the
Finally, there has been some recent work adapting distributional RL to the offline setting I 43 First, REM [ builds on QR-DQN 77 an online distributional RI. algorithm; however, REM can be applied to regular DQN (29 and does not directly utilize the distributional aspect of QR-DQN. The closest work to ours is ORAAC
45 which uses distributional RL to learn CVaR policy in the offline setting. ORAAC uses imitation learning to avoid OOD actions and stay close to the data distribution. As discussed above, imitation learning strategies can perform poorly unless the dataset comes from an optimal policy; in our experiments, we find that CODAC significantly outperforms ORAAC. Furthermore, unlike ORAAC, we provide theoretical guarantees on our approach.
2 Background
Offiine RL. Consider a Markov Decision Process (MDP) 33 (S,,,,, 7), where is the state space, A is the action space, P(s' s, is the transition distribution, R(r 8. s.a) is the reward distribution, and y E (0. 1) is the discount factor, and consider stochastic policy x(a s) : S (AA). A rollout using from state s using initial action a is the random sequence E = ((so, oo, ro), ($1, a1, (1,,,,, ....... such that sg = 8, ag = a. at ~ x(- 8t) (for t >> 0), Tt ~ P(- St, at); we denote the distribution over rollouts by D= s,a). R(- St, at), and St+1
The Q-function Q : :S x A + R of x is its expected discounted cumulative return Q= = (Elso))
EE= 7're]- Assuming the rewards satisfy E [Rmin, R then we have Q" (s,
(Vmin: | C [Rmin/ (1- - -),, Rmax/(1 7)].
standard goal of reinforcement learning (RL), which we call risk-neutral RL, is to learn the optimal policy x' such that Q" (s,a)= 2 (" a) for all s E S,a A and all
In offline RL, the learning algorithm only has access to a fixed dataset D == {(s,, a, r; s')}, where ~ R(- 8, and s' P( 8, a). The goal is to learn the optimal policy without any interaction with the environment. Though we do not assume that D necessarily comes from a single behavior
policy, we define the empirical behavior policy to be tt(( (8) == En' 1(s'=s,a'=a)
EVeD I 1(==2) With slight
abuse of notation, we write (s, a, s') D to denote uniformly random sample from the dataset. Also, in this paper, we broadly refer to actions not drawn from (- 8) (i.e., low probability density) as outtof-distribution (OOD).
Fitted Q-evaluation (FQE) 9 uses Q-learning for offline RL, which leverages the fact that Q = =T"Q" the unique fixed point of the Bellman operator T R SIIAI + RIS|[AI defined by
T" a) ERiT(s,a) [r] + 7 (rr 106''
where P= a) = P(s' s, a,)) 5'). In the offline setting, we do not have access to T instead, FQE uses an approximation n- obtained by replacing R and P in T with estimates ft and P based on D. Then, we can estimate Q" by starting from an arbitrary Q" and iteratively computing
@++1 arg min C(Q, {+))
where L(Q,Q') ED(s.a) [[Q(s. a) Q' (s(ss)))²]
Q
Assuming we search over the space of all possible Q (i.c., do not use function approximation), then the minimizer is Q++1 = Q² so Q² = (Tr)*Q². Q tt = TT, then lim Q* = =
Distributional RL. In distributional RL, the goal is to learn the distribution of the discounted cumulative rewards (i.e., returns)
4 Given a policy n, we denote its return distribution as the random variable ndd (s, a) Ei r'r, which is function of random rollout & ~ D= s, a);
3

---

note that 2" includes three sources of randomness: (1) the reward R(- s.a). (2) the transition
P( s, and (3) the policy x(- s). Also, note that (s, (c)(a)) [Z (s, (s. Analogous to Q D-ffucction Bellman operator, the distributional Bellman operator for π is
T ((s,a) = 7Z((' where T~ R(- s, 8 P(-1ssa), d' .(. 1s),
(1)
where - indicates equality in distribution. As with Q", 2" is the unique fixed point of T in Eq.
Next, let Fz(s.n) (z) [Vmin Vima] - [0. 1] be the cumulative density function (CDF) for return distribution (ss, a), and FR(s.a) be the CDF of R(- s, a) Then, we have the following equality, which captures how the distributional Bellman operator T operates on the CDF Fz(s.)) [7:
/ Faceraa))
( (---) dF (Fnssa)(r))
(T=z(sa)())
EpT(s',a' |s,a) s'
(2)
Let X and Y be two random variables. Then, the quantile function (i.e., inverse CDF) Fx' of | I Fx()), and the p- g-Wasserstein distance between X and Y is W,(XYY) = Co F - Fx (r)) (r)))))
X is (T) = inf E
I
Then, the distributional Bellman operator T is Y coontraction in the p 4 ie.. letting (z, Z2) = sup. W, ( (s, a), Zz(s, a)) be the largest Wasserstein distance over (s, a), and = (Z: S x - P(R) V(s, a) [BI 8, a) oo) be the space of distributions over R with bounded p-th moment, then
21, 22) S (d,
(VZ 22 E Z).
(3)
As result, 2" may be obtained by iteratively applying T to initial distribution Z.
As before, in the offline setting, we can approximate T by t using D. Then, we can compute z" (represented as F i) see below) by starting from arbitrary 20, and iteratively computing
Z(),a)
2 arg min z
where C'( = as.)) (W,(Zss,o), Z',,,
(4)
We call this procedure fitted distributional evaluation (FDE).
One distributional RL algorithmic framework is quantile-based distributional RL 7 2
47
27 38 43
where the return distribution Z is represented by its quantile function fz(s,a) (T) [0,
R. Given
distribution (T)) over [0, 1]. the distorted expectation of Z is 4 (Z(s, a)) = Fziz..) (T)g(T)d7, and the corresponding policy is (s))) == arg max. b (Z(s a)) 7 If = Uniform((0 [0, 1)), then (*(ssaa) a) & [(s, a)); alternatively, y Uniform((0, E)) corresponds to the CVaR [5. 5 6 objective, where only the bottom E-percentile of the return considered. Additional risk-sensitive
objectives are also compatible. For example, CPW 42 amounts to g(T) = T (T + (1-77) 7))
and Wang 45) has g(T) Fx (T) + B). where Fxr is the standard Gaussian CDF.
A drawback of FDE is that it does not account for estimation error, especially for pairs (s, a) that rarely appear in the given dataset D; thus, 2* (s, a) may be an overestimate of 2* a) [ 20 21, even in distributional RL (since the learned distribution does not include randomness in the dataset) [4] 3 Importantly, since we act by optimizing with respect to (*, the optimization algorithm will exploit these errors, biasing towards actions with higher uncertainty, which is the opposite of what is desired. In Section 3, we propose and analyze a penalty designed to avoid this issue.
3 Conservative offline distributional policy evaluation
We describe our algorithm for computing a conservative estimate of 2 (s a), and provide theoretical guarantees for finite MDPs. In particular, we modify Eq. 4to include a penalty term:
ナ arg min a (U(()) [ou(s, a). FFZ). (T) + (p(,,
(5)
z
for some state-action dependent scale factor oo: here, = Uniform (0, 1)). This objective adapts the conservative penalty in prior work 21] to the distributional RL setting; in particular, the first term in

---

Distorted expectation lower bound (Corollary 3.7)
Quantile bounds (Theorem 3.6)
Characterization of iterative estimates (Lemma 3.4)
Bellman operator contraction (Lemma 3.5)
Quantile gap- expansion (Theorem 3.8)
Distorted expectation gap- 8ap-expansion (Corollary 3.9)
Figure 2: Overview of our theoretical results.
the objective is a penalty that aims to shrink the quantile values for compared those of in-distribution actions: intuitively, co a) should be larger for OOD actions. For now, we let co be arbitrary; we describe our choice in Section 4 or E R is hyperparameter controlling the magnitude of the penalty term with respect to the usual FDE objective. We call this iterative algorithm conservative distribution evaluation (CDE).
(OOD) actions
Next, we analyze theoretical properties of CDE the setting of finite MDPs; Figure 2 overviews these results. First, we prove that CDE iteratively obtains conservative quantile estimates (Lemma 3.4 and defines contraction operator on return distributions (Lemma 3.5 Then, our main result (Theorem 3.6 that CDE converges a fixed point 2T whose quantile function lower bounds that of the true returns Z". We also prove that CDE is gap-expanding (Theorem (..)mi.e.. it is more conservative for actions that are rare in D. These results translate RL objectives computed by integrating the return quantiles, including expected and CVaR returns (Corollaries 3.7 3.9).
We begin by describing D has nonzero coverage of all actions for states in the dataset [23, 21).
assumptions on the MDP and dataset. First, assume that dataset
Assumption 3.1. For all E D and E A, we have a s) 0.
This assumption is only needed by our theoretical analysis to avoid division-by-zero and ensure that all estimates are well-defined; alternatively, we could assign a very low value vt 8) == e for all actions not visited at state s in the offline dataset and renormalize a s) accordingly. Next, we impose regularity conditions on the fixed point 2 of the distributional Bellman operator T
Assumption 3.2. For all E S and a E A, (s,(s is smooth. Furthermore, there exists Ç R50 such that for all
S and a E A, (2)(s1a) is ç-strongly monotone-iie. i.e. have F22)() (x) 2
The smoothness assumption ensures that the yth moments of (5(s, a) are bounded (since Z [Vmin: Vmaxl is also bounded), which in turn ensures that 2T E Z. The monotonicity assumption is needed ensure convergence of (x.)) Next, assume that the search space in (5) includes all possible functions (i.e., no function approximation).
Assumption 3.3. The search space of the minimum over Z in 5 is over all smooth functions F2(s.)) (for all s S and A) with support on [Vnin: Vmax].
This assumption is required for us to analytically characterize the solution ~++1 of the CDE objective. Finally, we also assume p> >>> (i.e., we use the p- Wasserstein distance for some p >> 1).
Now, we describe our key results. Our first result characterizes the CDE iterates ~ importantly, if co(s, a) > 0, then these iterates encode successively more conservative quantile estimates.
Lemma 3.4. For all E D, a E A. ls E M, and t E [0, 1]. we have
F-1 F2 (T) (F)
-D 7x (s.a) (T) c(s, a) where c(s, a) = -ap-- co 1/6-1))))) signnco a)).
We give a proof in Appendix A.I roughly speaking, it follows by setting the gradient of ( equal to zero, relying on results from the calculus of variations to handle the fact that F (z(s,)) is function.
Next, define the CDE operator T = to be the composition of ÿ with the shift operator O. Z + Z defined by Fo, 20. (T) (()((..) (T) c(s, a): thus, Lemma 3.4 says z++1 Zk Now, we show that 7 is contraction in the maximum Wasserstein distance dp-
5

---

Lemma 3.5. T is a n-contraction in dg. so -t converges to unique fixed point Zn.
The first part follows since f" is a y-contraction dp and Og is a non-expansion in dp. so by composition, 7 is a contraction in dp; the second follows by the Banach fixed point theorem.
Now, our first main theorem says that the fixed point 2T of TT is conservative estimate of n at all quantiles 7-i.e., CDE computes quantile estimates that lower bound the quantiles of the true return; furthermore, it says that this lower bound tight.
Theorem 3.6. For any E Rzu Co(s, a) > O, with probability at least - 6,
F (T) F (s,()))) (T) (1 (1 minfc(('s a') (ss',
(z)( (T) S F (=)(sa)) (T) (1-7)-1 feax a') - A(s', a')}
s'
for all E D, a E A. and 7 E [0, 11, where A(s = \VV sufficiently large (i.e., a 2 maxs,a (aaaaa)) }}, we have (z
5|5
log
1S
Furthermore, for
n(s.a]
2 F=1 (a(0)) (T).
co(s,a)
We give a proof in Appendix A.2 The first inequality says that the quantile estimates computed by CDE form lower bound on the true quantiles; this bound is not vacuous as long as satisfies the given condition. Furthermore, the second inequality states that this lower bound is tight.
Many RL objectives (c.g., expected or CVaR return) are distorted expectations (i.e, integrals of the return quantiles). We can extend Theorem 3.6 obtain conservative estimates for all such objectives:
Corollary 3.7. For any 8 E R2 cols, a) > a sufficiently large, and g(T), with probability at least 1 for all D, a E A, we have
(s, 2 $ (2 (s.
Choosing g= Uniform [0, 1I) gives (" a) > i a -i... lower bound on the Q-function. CQL (21 obtains similar lower-bound; thus, CDE generalizes CQL to other objectives-e.g. can be used in conjunction with any distorted expectation objective (e.g., CVaR, Wang, CPW, etc.) for risk-sensitive offline RL.
Note that Theorem 3.6 does not preclude the possibility that the lower bounds are more conservative for good actions (i.e., ones for which a 8) larger). We prove that under the choice
cu(s, a) = plaa 8) ( s)
(6)
(a s)
for some ıu(a s) * ft((a) s). then 7. gap-expanding-ie.. the difference in quantile values between in-distribution and disttr--iiiiitiion actions is larger under 7 than under T Intuitively, co(s, a) is large for actions a with higher probability under μ than under A (i.e., an OOD action).
Theorem 3.8. For p = 2, sufficiently large, and co as 6 for all E and [0, 1].
(=((s,) [ag(a F-1 (= (T) By(a]s) F2 (T) 2 Esg(ala) (n. (T) Ey(ols) FZ (T).
As before, the eup-exxansion property implies sap-expansion of integrals of the quantiles iee:: Corollary 3.9. For p = 2, sufficiently large, as in (6). and any g( all E, (8, ()) Eu(ale) og (Z (s, a)) 2 ERg(ale) t (2 a)) Bucale) $ (2" (s)
ERS(U)E) 22
Together, Corollaries 3.7 & 3.9 say that CDE provides conservative lower bounds on the return
quantiles while being less conservative for in-distribution actions.
Finally, briefly discuss the condition on in Theorems &.6 3.8 In general, can be taken to be small as long as (8, a) is small for all s E S and a E A, which in turn holds as long as n((8, a) is large-iee., the dataset D has wide coverage.
We may have co a) S 0; we use (o.s.a) co(s, + (1 mins, co(s, a)) to avoid this issue.

---

CODAC-C
CODAC-N
ORAAC
CQL
Figure 3: 2D visualization evaluation trajectories on the Risky PointMass environment. The red region is risky, the solid blue circles indicate initial states, and the blue lines trajectories. CODAC-C the only algorithm that successfully avoids the risky region.
4 Conservative offline distributional actor critic
Next, we incorporate the distributional evaluation algorithm in Section Dectionniinto an actor-critic framework. Following [2T]. propose min-max objective where the inner loop chooses the current policy to maximize the CDE objective, and the outer loop minimizes the CDE objective for this policy:
2 = arg min max (a BU())) [ED(s).a(aa)) F Z(s.a) (T) - ED(a ()) Lp( }
where we have used co as in (6). We can interpret μ as an actor policy, the first term as the objective for μ., and the overall objective as an actor-critic algorithm [8] In this framework, natural choice
for μ is maximum entropy policy ula s) x exp(Q(s, a)) 49 Then, our objective becomes
2+++ arg min
(a= BU(G) ED(()) log Σ (x.(F ED( (2)0 Cp, 7 29)
where U Uniform( [0, 1)); provide derivation in Appendix We call this strategy con- servative offline distributional actor critic (CODAC). To optimize over Z, we represent the quan- tile function as a DNN Go(T;s, a) se FZ(s,a)(T) The main challenge is optimizing the term (2, 2*) = WplZ, T 2 We do so using distributional temporal-differences (TD) [6] For sample (s, as,,)r,s' D and d' T(: s') and random quantiles t, T' U, we have
C,( (ō)
where
r GGo (T's 8 ,a') Go(r;8,a).
Here, d is the distributional TD error, 0' are the parameters of the target Q-network [30 and
[| 1 0)).82/(2xx) if 1f
Cx(6:)) = [ 1 < 0))-((6)-nn22 otherwise
the H-Huber quantile regression loss at threshold x [15]: then, Co r) is an unbiased estimator of the Wasserstein distance that be optimized using stochastic gradient descent (SGD) [19]. With this strategy, our overall objective can be optimized using any off-policy actor- critic method [24 13 11;;
distributional soft actor-critic (DSAC) (27). which replaces the
Q-network in SAC 13 with quantile distributional critic network 7 We provide the full CODAC
pseudocode in Algorithm of Appendix B
5 Experiments
We show that CODAC achieves state-of-the-ee results on both risk-sensitive (Sections 5..1 risk-neutral (Section (.3) offline RL tasks, including our risky robot navigation and D4RL 10 We
& & and
also show that our lower bound (Theorem(366) and gap-expansion (Theorem 3.8) results approximately hold in practice, (Section[5.4) 5.4), validating our theory on CODAC's effectiveness. We provide additional details (e.g., environment descriptions, hyperparameters, and additional results) in Appendix C
5.1 Risky robot navigation
2The D4RL dataset license Apache License 2.0.
7

---

Table I: Risky robot navigation quantitative evaluation. CODAC-C achieves the best performance on most metrics and is the only method that learns risk-averse behavior. This table is reproduced with standard deviations in Table 6 in Appendix
Risky PointMass
Risky Ant
Algorithm
Mean
Median
C
Violations
Mean
Median
CVaRo.1
Violations
DSAC (Online)
-7.69
-3.82
-49.9
94
-866.1
-833.3
-1422.7
2247
CODAC-C (Ours)
-6.05
.4.99
-14.73
0.0
-456.0
-43.44
-686.6
347.8
CODAC-N (Ours)
-8.60
-4.05
-51.96
108.3
-43..7
-395.1
-847.1
936.0
ORAAC
-10.67
-4.55
-64.12
138.7
-788.1
-795.3
-1247.2
1196
CQL
-7.51
-4.18
-43.44
93.4
-967.8
-858.5
-1887.3
1854.3
Tasks. We consider Ant robot whose goal is to navigate from random initial state to the green circle as quickly as possible (see Figure For for visualization). Passing through the red circle triggers a high cost with small probability, introducing risk. risk-neutral agent may pass through the red region, but risk-aware agent should not. We also consider a PointMass variant for illustrative purposes.
We construct offline dataset that is the replay buffer of risk- neutral distributional SAC [27] agent. Intuitively, this choice matches the practical goals of offline RL, where data is gathered from a diverse range of sources with no assumptions on their quality
Figure 4: Risky Ant.
and risk tolerance levels. 23 See Appendix C.I for details the environments and datasets.
Approaches. We consider two variants CODAC: (i) CODAC-N, which maximizes the expected return and (ii) CODAC-C, which optimizes the CVaRo.1 objective. We compare to Offline Risk- Averse Actor Critic (ORAAC) [43, a state-of-the-art offline risk-averse RL algorithm that combines a distributional critic with an imitation-learning based policy to optimize risk-sensitive objective, and to Conservative Q-Learning (CQL) [2T] a state-of-an art offline RL algorithm that non-distributional.
Results. We evaluate each approach using 100 test episodes, reporting the mean, median, and CVaRg.1 (i.c., average over bottom 10 episodes) returns, and the total number of violations (i.c., time steps spent inside the risky region), all averaged over 5 random seeds. We also report the performance of the online DSAC agent used collect data. See Appendix for details. Results are in Table ]
Performance of CODAC. CODAC-C consistently outperforms the other approaches on the CVaR return, well the number of violations, demonstrating that CODAC-C is able to avoid risky actions. also competitive in terms of mean return due to its high CVaRg.1 performance, but performs slightly worse on median return, since it is not designed to optimize this objective. Remarkably, on Risky PointMass, CODAC-C learns safe policy that completely avoids the risky region (i.c., zero violations), even though such behavior is absent the dataset. Appendix C.I we also show that CODAC can successfully optimize alternative risk-sensitive objectives such as Wang and CPW.
Comparison to ORAAC. While ORAAC also optimizes the CVaR objective, it imitation learning to regularize the learned policy to stay close to the empirical behavior policy. However, the dataset contains many sub-optimal trajectories generated early in training, and is furthermore risk-neutral. Thus, imitating the behavioral policy encourages poor performance. In practice, key of offline RL is leverage large datasets available for training, and such datasets will rarely consist of data from a single, high-quality behavioral policy. Our results demonstrate that CODAC is significantly better suited to learned these settings compared ORAAC.
Comparison to CQL. On Risky PointMass, CQL learns a risky policy with poor tail mill-performanc, indicated by its high median performance but low CVaRo.1 performance. Interestingly, its mean performance is also poor; intuitively. the mean is highly sensitive to outliers that may not be present in the training dataset. On Risky Ant, possibly due to the added challenge of nigh-dimenssonality. CQL performs poorly on all metrics, failing to reach the goal and to avoid the risky region. As expected, these results show that accounting for risk is necessary in risky environments.
Qualitative analysis. In Figure 3 we show the 100 evaluation rollouts from each policy on Risky PointMass. As can be seen, CODAC-C extrapolates safe policy that distances itself from the risky

---

Table 2: D4RL results. CODAC achieves the best overall performance in both risk-sensitive (Left) and risk-neutral (Right) variants the benchmark. These tables are reproduced with standard
deviations in Tables & 9 in Appendix
Algorithm
Medium
Mixed
Dataset
BCQ
MOPO
CQL
ORAAC
CODAC
Mean
CVaa
Mcan
CVaRu.3
hallccheetah random
2.2
35.4
35.4
13.5
34,6
Cheetah
33.2
12.0
hopocr-random
10.6
11.7
10.8
9.8
11,0
ORAAC
361.4
91.3
307.1
118.9
walker2d-maodom
13.6
7.0
3.2
18.7
CODAC-N
338
347.7
CODAC-C
335
-27
396.4
238.5
hallcheetah medium
40.7
42.3
44.4
41.0
46.3
walker2d-modiun
53.1
17.8
79.2
27.3
82.0
Hopper
CQL
ORAAC
1007.1
877.9
767.6
693.0
876.3
189.2
$24.9
-21.4
hopper medium
54,5
28.0
58.0
1.48
70.8
CODAC-N
993.7
952.5
1483.9
1457.6
halfeheetah-
38.2
53.1
46.2
30.0
44.
CODAC-C
1014.0
976.4
1551.2
1449.6
hopper mixed
33.1
67.5
48.6
16.3
100.2
walker2d mixxd
15.0
39.0
26.7
28
33.2
Walker2d
ORAAC
CQL.
1524.3
1134.1
663.0
222.0
-69.6
halfeheetah-meddeee
64.7
63.3
62.4
24.0
70.4
CODAC-N
358.7
walkor2d-mcd-cxp
57.5
44.6
98.7
28.2
106.0
CODAC-C
1120.8
902.3
450.0
261.4
hopper-med-exp
110.9
23.7
111.0
18.2
112.0
region before proceeding to the goal; in contrast, all other agents traverse the risky region. For Ant, include plots of the trajectories of trained agents in Appendix C.1 and videos in the supplement.
5.2 Risk-sensitive D4RL
Tasks, Next, we consider stochastic D4RL 43 The original D4RL benchmark [10 consists of datasets collected by SAC agents of varying performance (Mixed, Medium, and Expert) on the Hopper, Walker2d, and HalfCheetah MuJoCo environments [41; stochastic D4RL relabels the rewards to represent stochastic robot damage for behaviors such as unnatural gaits or high velocities; see Appendix C.2 The Expert dataset consists of rollouts from fixed SAC agent trained to convergence; the Medium dataset is constructed the same way except the agent is trained to only achieve 50% of the expert agent's return. The Mixed dataset is the replay buffer of the Medium agent.
Results. In Table 2 (Left), report the mean and CVaRg.1 returns on test episodes from each approach, averaged over 5 random seeds. We show results on the Expert dataset in Appendix C22 CODAC still achieves the strongest performance. As can be seen, CODAC-C and CODAC-N outperform both CQL and ORAAC on most datasets. Surprisingly, CODAC-N is quite effective on the CVa metric despite its risk-neutral objective; a likely explanation is that for these datasets, mean and CVaR performance are highly correlated. Furthermore, we observe that directly optimizing CVaR may lead to unstable training, potentially since CVaR estimates have higher variance. This instability occurs for both CODAC-C and ORAAC-on Walker2d-Medium they perform worse than the risk-neutrs neutral algorithms. Overall, CODAC-C outperforms CODAC-N in terms of CVaRo.1 on about half of the datasets, and often improves mean performance as well. Next, while ORA BRAACIO is generally effective on Medium datasets, it performs poorly on Mixed datasets; these results mirror the ones in Section 5. Finally, CQL's performance varies drastically across datasets; we hypothesize that learning the full distribution helps stabilize training in CODAC. In Appendix C.2 we also qualitatively analyze the behavior learned by CODAC compared to the baselines, demonstrating that the better CVaR performance CODAC obtains indeed translates to safer locomotion behaviors.
5.3 Resk-nlutral D4RL.
Task. Next, we show that CODAC is effective even when the goal is optimize the standard expected return. To this end, we evaluate CODAC-N on the popular D4RL. Mujoco benchmark [T]..
Baselines. We compare to state-of-art algorithms benchmarked in [10] and 488. including Batch- Constrained Q-Learning (BCQ), Model-Based Offline Policy Optimization (MOPO) 488, and CQL. We also include ORAAC as offline distributional RL bascline. We have omitted less competitive baselines included in 10 from the main text; a full comparison is included in Appendix C3
Results. Results for non -on-distributional approaches are directly taken from [10]; for ORAAC and CODAC, we evaluate them using 10 test episodes the environment, averaged over 5 random seeds. As shown in Table (Right), CODAC achieves strong performance across all 12 datasets, obtaining state-of-art results on datasets (walker2d-random, hopper-medium, hopper-mixed, halfcheetah- medium-expert, and walker2d-medium--on mediimm-mddieedeem demonstrating that performance improvements from

---

distributional learning also apply in the offline setting. Note that CODAC's advantage is not solely due to distributional RL- ORAAC also uses distributional RL, but most cases underperforms prior state-of-the-art, These results suggest that CODAC's use of a conservative penalty is critical for it to achieve strong performance.
5.4 Analysis of Theoretical Insights
We perform additional ex- periments to validate that our theoretical insights in Section 3 hold in practice, suggesting that they help ex- plain CODAC's empirical performance.
Table 3: Monte-Carlo estimate vs. critic prediction. The CODAC- predicted expected and CVaRo.1 return is lower bound on MC estimate of the true value.
Walker3d-Medium MC Return Q- Q-Estimate
Walker2d-Mixed
Wallerrd--Meiunm-
Regular
MC Return
Q-Estimatic
Return
Q-Estimate
CODAC CQI.
240.2
55.7
127.1
97.6
370.
39.7
247.2
$3.0
124.5
-5.22
369.7
116.4
ORAAC
245.2
302.2
118.2
x.70x 10
68.2
322.2
Lower bound. We show that in practice, CODAC ob- tains conservative estimates of the Q and CVaR objec- tives across different dataset types (i.e., Medium vs. Mixed vs. Medium Medium-Expert), Given an initial state So, we obtain a Monte Carlo (MC) estimate of Q and CVaR for (so, π (#0)) based on sampled rollouts from 80, and compare them to the values predicted by the critic. In Table we show results averaged over 10 random so and with 100 MC samples for each 8g- CODAC obtains conservative estimates for both Q and CVaR; in contrast, ORAAC overestimates these values, especially on Mixed datasets, and CQL only obtains conservative estimates for Q, not CVaR.
Stechastic
Waller2d-MMeiim MC CVaRg... Z-Estimate
MC CV Z.Estimate
Walker2d-Mixed
Walker2d-Mediums Expert
MC CVaRg..
Z-Estimate
CODAC ORAAC
185.7
204.2
85.6
59.9
201.9
367.6
50.9
x.54x10
199.5
343.5
-ap-Exxansioon Next, we verify that CODAC's quan- tile estimates expand their gap between m-disstibution and nut-of-distribution ac- tions. We use the D4RL, -exiutt-Eppett datasets where CODAC uniformly performs well, making them ideal for under- standing the source of CODAC's empirical performance. We train "CODAC w.o. Penalty", non-com conceesservatiee variant of CODAC (i.e., a 0), and use its actor as μ and its critic as Fzz.. Next, for each dataset, we randomly sample 1000 state-action pairs and 32 quantiles т, resulting in 32000 (s, a. r) tuples; for each one, we compute the quantile gaps for CODAC and CODAC w.o. Penalty. In Table[4] we show the percentage of tuples where each CODAC variant has a larger quantile gap, along with their average return. As can be seen, CODAC has larger gap for more than 90% of the tuples on all datasets, as well as significantly higher returns. These results show that gap-expansion holds practice and suggest that it helps CODAC achieve good performance.
Table 4: Gap-expansion: CODAC expands the quantile gap and ob- tains higher returns than an ablation without the conservative penalty.
HalfCheetab Medium Expert Retura
Hlopper-Medlum Expert
Walker2d Medium-Expert Rostive Gap
Posilive Gap
Posilive Gap 9 Renen
Retun
CODAC: CODAC w.o. Peuulty
95.3
93.6
91.3
111.9
91.1
111.3
47
12.1
8.7
25.8
8.9
5.9
Conclusion
We have introduced Conservative Offline Distributional Actor-Critic (CODAC), a general purpose offline distributional reinforcement learning algorithm. We have proven that CODAC obtains conser- vative estimates of the return quantile, which translate into lower bounds on Q and CVaR values. In experiments, CODAC outperforms prior approaches on both stochastic, risk-sensitive offline RL benchmarks, as well as traditional, risk-neutral benchmarks.
One limitation of our work is that CODAC has hyperparameters that must be tuned (in particular, the penalty magnitude ax). As prior work, we choose these hyperparameters by evaluate online rollouts in the environment. Designing better hyperparameter selection strategies for offline RL is an important direction for future work. Finally, we do not foresee any societal impacts or ethical concerns for our work, other than the usual risks around algorithms for improving robotics capabilities.
10

---

Acknowledgments and Disclosure of Funding
This work is funded in part by an Amazon Research Award, gift funding from NEC Laboratories America, NSF Award CCF-1910769, and ARO Award W91 10F-22001-00 The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv.1607.06450, 2016.
[3] Gabriel
Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional determin- istic policy gradients. arXiv preprint arXiv: 1oXiv:8804.000 1804 004.08617, 17, 2018.
[4] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[5] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120 2017.
[6] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pages 1095-1105. PMLR, 2018
[7] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[8] Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. arXiv preprint 1005v: 2012.
[9] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[10] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[14] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.

---

[15] Peter J. Huber. Robust Estimation of Location Parameter. The Annals of Mathematical Statistics, 35(1):73 101, 1964.
[16] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning, 2016.
[17] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 4436 4443, 2020.
[18] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint awXiv:2005.0599 2020.
[19] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 215(4):143-156, 2001.
[20] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: DrXiv:/906.000 19w..00999.9 2019.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1199-1191. Curran Associates, Inc., 2020.
[22] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[23] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[24] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509.00500..22
2015.
[25] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horionn off- off-policy estimation. arXiv preprint arXiv: 1810.. /2429, 2018.
[26] Clare Lyle, Marc G Bellemare, and Pablo Samuel Castro. A comparative analysis of expected and distributional reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4504-4511 2019.
[27] Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distributional soft actor critic for risk sensitive learning. arXiv preprint arXiv:2004.144. 2020.
[28] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference machine learning, pages 1928-1937. PMLR. 2016.
[29] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13225522,, 2013.
[30] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51897540):59-533, 2015.
[31] Thanh Tang Nguyen, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning via moment matching, 2020.
[32] Doina Precup. Eligibility traces for off off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
12

---

[33] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[34] Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, pages 317-328. Springer, 2005.
[35] Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of banking & finance, 26(7):1443 1471, 2002.
[36] Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In International Conference on Machine Learning, pages 5528-5536. PMLR, 2019.
[37] Simon Shen, Yecheng Jason Ma, Omer Gottesman, and Finale Doshi- Velez. State relevance for off-policy evaluation, 2021.
[38] Yichuan Charlie Tang. Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint arXiv: 111.911...6.,,.1 2019.
[39] Philip Thomas and Emma Brunskill. Data-efficient off aff-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[40] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pages 255- 263. Hillsdale, NJ, 1993.
[41] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE, 2012.
[42] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, (4):297733 1992.
[43] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprini arXiv:2 102.00377 2021.
[44] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[45] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
[46] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1331, 2019.
[47] Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tieyan Liu. Fully parameterized quantile function for distributional reinforcement learning. arXiv preprint arXiv: 191 1402 2019.
[48] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 141422 Curran Associates, Inc., 2020.
[49] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438 Chicago, IL, USA, 2008
13


Reviews:
[{"review_id": "X0WOAZluLu5", "reviewer": "Reviewer_9EDx", "summary": "This paper adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. It proposes a CODAC algorithm for this purpose, proves the convergence of the proposed alg in tabular cases, and shows that it can learn risk-averse policies from offline data collected purely from risk-neutral policies. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: \n-\tThe use of distributional RL for offline learning is a promising direction and this paper explores just that. I think such a combination makes an important bit to further improve the performance of offline RL which is becoming a big deal for RL in practical settings. \n-\tThe experimental results on risk-sensitive and risk-neutral policies are promising though expected \n**Weaknesses**:\n-\tThe approach proposed is quite simple and straightforward without much technical innovation. For example, CODAC is a direct combination of CQL and QR-DQN to learn conservative quantiles of the return distribution. \n-\tSome parts of the paper need clearer writing  (more below) \n\n**Comments and questions**: \n-\tI think in a paragraph from lines 22-30 when discussing distributional RL, the paper lacks relevant literature on using moment matching (instead of quantile regression as most DRL methods) for DRL (Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”). I think this should be properly discussed when talking about various approaches to DRL that have been developed so far, even though the present paper still uses quantile regression instead of moment matching. \n-\tMore explanation is needed for Eq (5). For example, what is the meaning of the cost $c_0(s,a)$? (e.g., to quantify out-of-distribution actions)\n-\tThe use of $s’$ and $a’$ when defining $\\hat{\\pi}_{\\beta}$ at line 107 might cause confusion as $\\mathcal{D}$ contains $(s,a,r,s’)$. \n-\tThis paper is about deriving a conservative estimate of the quantiles of the return from offline data where the conservativeness is for penalizing out-of-distribution actions. In the paper, they define OOD actions as those are not drawn from \\hat{\\pi}_{\\beta}(.|s) (line 109) but in Assumption 3.1. they assume that \\hat{\\pi}_{\\beta}(a|s) > 0, i.e., there is no OOD actions. Thus, what is the merit of the theoretical result presented in the paper? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and social impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concern. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Oj8cbrb8vzy", "reviewer": "Reviewer_Haog", "summary": "The paper introduces a general purpose offline distributional reinforcement learning algorithm CODAC. They prove that CODAC obtains conservative estimates of the return quantile, which translate into lower bounds on values. They provide sound theoretical and thorough empirical analysis for the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized and well written in general. The results presented in the paper looks interesting and has potential of bringing good impact in risk-averse RL.\n\nWhat is x in equation (2). Is there an intuitive way to interpret what's happening here?\n\nThe quantile based formulation is varsatile and is a good addition for risk based optimization. One can get VaR, CVaR and so on simply by updating the quantile. CVaR is known to be not time-consistent. Or in other words, a recursive decomposition does not hold and the policy is history dependent for CVaR optimization in MDPs, unless the state space is augmented [1]. But this paper is providing a recursive formulation without augmenting the state space, even for CVaR. An analytical study in this matter would be greatly appreciated.\n\nThe theoretical results presented in the paper is rigorous and looks good from a high level. Though I did not get the chance to verify the correctness of all the proofs presented in the paper.\n\nIn description of figure 1, the inital states are not represented by 'blue' color, as stated. May be green?\n\nThe empirical study presented in the paper is detailed and well thought. It does provide a comprehensive overview on the behavior of CODAC compared to other baseline methods.\n\n\n[1] Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach. Yinlam Chow, Aviv Tamar, Shie Mannor, Marco Pavone.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and impacts are addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "AhmSRW9D-o", "reviewer": "Reviewer_1we9", "summary": "At its core, the paper introduces an algorithm called conservative distributional evaluation (CDE). In the tabular settings, the CDE algorithms theoretically guarantees to converge to a fixed point whose quantile function lower bounds that of the true return distribution, also its fixed point is more conservative for rare actions in the dataset (the property that is called gap-expanding). These are desired properties for offline scenarios with full coverage (see the comments bellow). CDE is then used in an actor-critic framework, which is called Conservative Offline Distributional Actor Critic (CODAC). This gets a min-max objective: the inner loop chooses the current policy to maximize the CDE objective, and the outer loop minimizes the CDE objective for this policy. Some experimental results are followed to support the theory. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-- The **far too strong** assumption that the dataset $\\mathcal{D}$ has full coverage is ignored to discuss (Assumption 3.1). In real-world offline settings (such problem as in healthcare) we almost always deal with situations where only a tiny fraction of the state-action space is covered by $\\mathcal{D}$. When $\\mathcal{D}$ has full coverage, the problem is simply an **off-policy** learning problem (rather than offline) with nearly complete exploration. This is identical to the *online* case where there is an unknown behavioural policy for each experienced trajectory and where all state-action pairs are seen frequently enough. I found this case very misleading to be called **offline**. \n\n-- The penalty term in (5) is sampled-based. In particular, $F^{-1}_{Z}(\\tau)$ with uniform $\\tau$ may simply get a large value by some chance; hence, incurring a large penalty even when $Z$ is not distributed close to uniform. This may cause an unnecessary instability. Why not using other measures of uncertainty which are deterministic function of distribution’s shape, such as moments of $Z$ or entropy (as entropy is a super-exponential lower-bound for all even moments). Such measures can be computed directly from the quantile values; the regularization term in (5) will then only contain an expectation over $\\mathcal{D}$.\n\n-- Proof of Lemma 3.4 (in the Appendix) -> the first term in expressions after line 518 -> expectation over $\\tau$ should introduce an extra constant coming from the uniform distribution when writing the integral over $\\tau$, which is missing. Further, $\\alpha$ also is missing in the second line. (They shouldn’t change the result when the derivative is set to zero.)\n\n-- Theorem 3.6 -> left-hand-side of the inequality must also be $F^{-1}$. \n\n-- Theorem 3.6 -> A return distribution must sum to one, regardless. Intuitively, if the learned distribution lower bounds the original distribution in all the quantiles it cannot sum to one (unless at least in one (discrete) quantile it overestimates with significant amount such that it compensates for the rest). This theorem sounds counter-intuitive. Am I missing something? (This needs clarification in the text as well).  \n\n-- After equation (6) -> It is not clear where $\\mu$ comes from and how sensitive the results are to this policy. A discussion here is needed. \n\n\nOther comments:\n\n-- Line 100: In general, $[V_{min}, V_{max}] \\subseteq [\\frac{R_{min}}{1-\\gamma}, \\frac{R_{max}}{1-\\gamma}]$ instead of being equal. For instance, in a goal-seeking MDP, $[V_{min}, V_{max}] = [R_{min}, R_{max}]$.\n\n-- Line 103 -> what do you mean by “its correspondence to policy”? If it refers to the optimal polic/value, then fix the last expression by adding asterisk. \n\n-- Line 107 -> definition of $\\hat{\\pi}_{\\beta}$: right hand side should be conditioned on (s,a). \n\n-- Equation (5) -> what is $U$? this should be uniform distribution (?), yet in the previous section you used Uniform(.). If yes, it is better to be consistent. \n\n-- Careful proofread is required. I saw at least a couple of sentences with no full-stop at the end.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a mostly theoretical work. Not at a level to directly cause societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "1ka3mAunZW", "reviewer": "Reviewer_8SBh", "summary": "This paper tackles the offline RL setting, in which a policy is learned entirely from historical data, rather than through real-time online interaction with a simulator. However, existing algorithms such as CQL, while able to learn conservative lower bounds on the true Q-values, learn point estimates, making it difficult to also account for risk. Key contributions include:\n1. bridging the gap between distributional RL and conservative value-based RL (CQL) for the first time, in which a lower bound on the quantiles of the return distribution can be learned through a penalization approach (CODAC). \n2. two key theoretical results ensure that the method works as expected. Theorem 3.6 shows that CODAC will learn quantile estimates that lower bound the quantiles of the true return distribution. Intuitively, a lower bound is preserved when integrating over this distribution, such as when computing CVaR objectives. Theorem 3.8 shows gap-expansion, in which the degree of conservativeness is less for in-distributions actions than it is for out-of-distribution actions.\n3. A practical actor-critic algorithm is developed based on the min-max formulation of the original penalized objective in CQL, but in the distributional RL setting. This algorithm is evaluated on a variety of robotics tasks, to see whether CODAC can learn conservative quantiles while being able to take risk into account.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: I believe the paper has sufficient novelty for NeurIPS, being a novel combination of distributional RL and conservative Q-learning. This novel contribution also solves an important problem of learning risk-sensitive policies from offline data, which is a complex but important problem. existing methods are well discussed, and it is clear how this method differs in problem setting from CQL and work on distributional imitation learning. \n\nclarity: overall, the paper is very well written and the motivation for studying this problem is well argued. some minor details below:\n\n1. line 76 \"paratemerizing\" --> \"parameterizing\"\n2. the notation in section 2, from line 92 is very dense, and some notation isn't well defined: is D(s, a) the data set restricted to s, a pairs? \n3. it could really help readers to provide a simple motivating example early on (before section 2, or at least before going into section 3) for incorporating distributions of the return/risk in the offline RL setting, along with visualizations of what the proposed method CODAC is designed to do (e.g. manage risk while learning offline).\n\nquality: the theoretical results are intuitive, appealing and easy to understand, particularly in the context of existing algorithms in offline RL and distributional RL. the experimental evaluation is quite thorough, testing the ability of CODAC to learn both risk-neutral and risk-sensitive policies, on data-sets of various quality, and baselines appear to be well chosen and comprehensive (see limitations for comments). \n\nsignificance: offline reinforcement learning and risk-sensitive distributional RL are both very active and emerging topics in reinforcement learning. based on this observation, I believe this paper will have a very high impact in the research community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "tightness of lower bounds: original work on CQL shows that the learned bounds are reasonably tight lower bounds on the Q-values. does a similar result hold for the quantiles as well? \n\nvisualization and interpretation of results: the results are quite hard to interpreted for the stochastic domains from tables alone. it is also hard to understand what kind of behavior the agent is learning on the stochastic domains. is it possible to visualize the learned behaviors similar to the first two domains? \n\nfailure modes: it is also hard to understand why the algorithm fails (e.g. risk sensitive cheetah task?). can any commentary be provided to discuss why the algorithm fails here and what could be done in the future to address this?\n\nchoice of risk measures: why is CVaR the approach used here, given it is not novel? it would be nice to provide further reasoning. existing work on distributional RL often allows other risk measures to be defined through other transforms of tau (e.g. Wang transform, etc.) Can the current work be applied for other such risk measures? Are most/all well studied risk measures quantile-based or are some choices not amenable to the analysis in this paper? it doesn't seem that exponential utilities or certainty-equivalents are possible for example, but I may be wrong. \n\noverall: I believe this is a strong paper with non-trivial contributions. the overall approach makes sense and the theoretical results are quite intuitive. furthermore, the limitations of the approach (tuning alpha, etc) are anticipated but they are discussed in the conclusion, where it is noted that such limitations are not exclusive to this work, so I do not see this as an issue. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this is a good paper with sufficient novelty and relevance to the NeurIPS community. Reviewers had several questions, which were adequately answered by the authors. Therefore, I would recommend the *acceptance* of this paper.

I encourage the authors to consider the feedback by reviewers in their revision of the paper, including the followings:

- discuss the tightness of the lower bound
- discuss what other risk measures CODAC can be applied to
- discuss that CVaR's not being time-consistent, and how you avoid it.
- cite and compare with the literature on moment matching for DRL
- Fix the typos, including the one in Theorem 3.6
Decision: Accept (Poster)