Latent Offline Distributional Actor-Critic
Authors: Félicien Hêche
BARAKAT Oussama
Thibaut P Desmettre
Tania Marx
Stephan Robert-Nicoud
Venue: Submitted to ICLR 2023 (2023)

Abstract: Offline reinforcement learning (RL) has emerged as a promising paradigm for real world applications, since it aims to train policies directly from datasets of past interactions with the environment. The past few years, algorithms have been introduced to learn policies from high-dimensional observational states in an offline settings. The general idea of these methods is to encode the environment into a smaller latent space and train policies on the top of this smaller representation. In this paper, we extend this general method to stochastic environments (i.e. where the reward function is stochastic) and considering a risk measure instead of the classical expected return. First, we show that under some assumptions it is equivalent to minimize a risk measure in the latent space and in the natural space. Based on this result, we present Latent Offline Distributional Actor-Critic (LODAC), an algorithm which is able to train policies in high-dimensional stochastic and offline settings to minimize a given risk measure. Empirically, we show that using LODAC to minimize Conditional Value-at-Risk (CVaR), outperforms previous methods in term of CVaR and return on stochastic environments.

Under review as a conference paper at ICLR 2023
LATENT OFFLINE DISTRIBUTIONAL ACTOR-CRITIC
Anonymous authors
Paper under double-blind review
ABSTRACT
Offline reinforcement learning (RL) has emerged as a promising paradigm for real world applications, since it aims train policies directly from datasets of past interactions with the environment. The past few years, algorithms have been introduced to learn policies from high-dimensional observational states in of- fline settings, The general idea of these methods is to encode the environment into smaller latent space and train policies on the top of this smaller representa- tion. In this paper, we extend this general method to stochastic environments (i.e. where the reward function is stochastic) and considering a risk measure instead of the classical expected return. First, we show that under some assumptions it is equivalent to minimize a risk measure in the latent space and in the natural space. Based this result, we present Latent Offline Distributional Actor-Criti Critic (LO- DAC), an algorithm which is able to train policies high-dimensional stochastic and offline settings to minimize a given risk measure. Empirically, show that using LODAC to minimize Conditional Value-at Risk (CVaR), outperforms pre- vious methods in term of CVaR and return on stochastic environments.
INTRODUCTION
In a lot of context, human decisions are stored and build interesting datasets. With the successes of modern machine learning tools comes the hope to exploit them to build useful decision helpers. To achieve this, we could use an imitation learning approach (Hussein et al. 2017). But, in this case, we will best be as good humans. Moreover, the performance of this approach depends heavily on the quality of the training dataset. In this work we would like to avoid these behaviours and thus, we consider another framework: reinforcement learning (RI.. In recent years, RL achieved impressive
results in number of challenging areas, including games (Silver et al. 2016 2018). robotic control
(Gu et al. 2017; Haarnoja et al.) 2018) or even healthcare (Shortreed et al. 2011 Wang et al. 2018). In particular, ar, offline RL seems to be really interesting for real world application since its goal is to train agents from a dataset of past interactions with the environment (Deisenroth et al.) 2013).
With the digitization of society, more and more features could be used to represent the environment. Unfortunately, classical RL algorithms are not able to work with high-dimensional states. Obviously, could manually choose feature subset. However, this choice is not straightforward and it could have a huge impact on the performance. Therefore, it may be be more practical to use RL algorithms capable of learning from high-dimensional states.
It is common to evaluate RL algorithms on deterministic (in the sense that the reward function is deterministic) environment such as the DeepMind Control suite (Tassa al. 2018). However, in a lot of real world applications, environments are not deterministic but stochastic. Therefore, it might be important to develop algorithms which are able to train policies in these special cases. The motivation of this paper to provide method for training policies in an offline settings and working high-dimensional stochastic environment.
In this paper, we present Latent Distributional Actor-Critic (LODAC) an algorithm which is able to train policies in digh-oimmeniooall, stochastic environment and in an offline settings. The main idea is to learn smaller representation and to train the agent directly in this latent space. But instead of considering the expected return, we take into account a risk measure. First, assuming some hypothesis, show that minimizing this risk measure in the latent space is equivalent to minimize the risk measure directly in the natural state. This theoretical result provides a natural framework: : train latent variable model to encode the natural space into latent space and then train a policy on

---

Under review as a conference paper at ICLR 2023
the top of this latent space using risk- RL algorithm. In the experimental part, we evaluate our algorithm on high-dimensional stochastic datasets. In the best of our knowledge, we are the first authors to propose algorithm to train policies in high digh-dimensional, stochastic and offline settings.
RELATED WORK
Before going further into work, we present some related work.
Offinne RL. Offine RL (Levine et al., 2020j is a particular approach of RL where the goal is to learn policies directly from past interactions with the environment.Thie is promising framework for real- world applications since it allows the deployment of already trained policies. Thus, it is not really surprising that offline RL has received lot of attention the recent years (Wiering & Van Otterlo
2012 Levine et al. 2020 Yang et al., 2021 Chen et al. 2021
Liu et al. 2021 Yu et al. 202
Wang et al. 2021).
One of the main problem in offline RL is that the Q-function is too optimistic in the case of out-of- distribution (OOD) Nates-actions pairs for which observational data is limited (Kumar al. (2019) Different approaches have been introduced to deal with this problem. For example, there are algo-
rithms that extend the importance sampling method (Nachum et al.. 2019 Liu et al. 2019) or extend the dynamic programming approach (Fujimoto et al. [2099] Kumar et al. (019). Other authors build
a conservative estimation of the Q-function for OOD states-actions pairs (Kumar et al., 2020 Yu
et al. 1b). Finally it is also feasible to extend the model-based approach, (Rafailov et al. 2021
Argenson & Dulac-Arnold 2020).
Learning with high-dimensional states. In the previous years, algorithms were proposed to train policies directly from high-states, like images (Lange & Riedmiller 2010 Levine et al. 2016 Finn
& Levine, 2017; Ha & Schmidhuber, 2018 Chen al.] [021a). Previous work has observed that learning good representation of the observation is a key point for this type of problem (Shelhamer aT. 2016). In some works, authors use data augmentation techniques to learn the best repre- sentation possible (Kostrikov et al. 2020 Laskin et al., 2020; Kipf et al. 2019). But, it has been intensively studied how should encode the high-dimensional states into the latent space (Nair ct al. 2018 Gelada et al. 2019 Watter et al. [0155] Finn et al. (016). Then, it is a common ap- proach train the policies using a classical RL algorithm, like Soft Actor-Critic (SAC) (Harrnoja et al. 2018). on the top of this latent representation (Han et al. 2019 Lee et al., 2020). Further-
more, also possible make some planifications this latent space to improve the performance
Hafner et al.) 2019b a).
Risk-sensitive RL. Risk-sensitive RL is a particular approach to safe RI. (Gareia & Fernández, 2015). In safe RI., policies are trained to maximize performance while respecting some safety con- straints during training and/or in the deployment. In risk-sensitive RL, instead of maximizing the expectation of the cumulative rewards, we are more interesting in minimizing measure of the risk induced by the cumulative rewards. Risk-sensitive RL has raised some attention the last few years (Fei et al. 2021 Zhang et al. 2021). Depending the context, might consider different risk measures, like Exponential Utility (Rabin] 2013) Cumulative Prospect Theory (Tversky & Kah-
neman
1992) or Conditional Value-at-Risk (CVaR) (Rockafellar & Uryasev 2002). Conditional
Malue-at-Risk has strong theoretical properties and is quite intuitive (Sarykalin et al. 2008 Artzner al. 1999). Therefore, CVaR is really popular and has been strongly studied in context of RL
[Chow & Ghavamzadeh 2014; Chow et al., 2015; Singh et al. 2020 Urpi et al. 2021 Ma et al.,
2021 2020 Ying al.) 2021). Previous work suggest that taking into account Conditional Value- at-Risk instead of the classical expectation, could prevent from the gap of performance between
simulation and real world application (Pinto et al.77017)
PRELIMINARIES
In this section, we introduce notations and recall concepts we will use later.
Coherent Risk Measure. Let F,, F.P) probability space and _ = (Q, F,P). A functional
R: _ + (-xx, +00) called coherent risk measure (Rockafellar 2007 if
1. R(C)= C for all constants C.
2

---

Under review as a conference paper at ICLR 2023
2. R((1 X)X XX) 2 (1 (X) + XR(X) for À E (0,
3. R(X) - ( when X - X'.
4. R(X) Σ 0 when X X + 0 with R(X,) I 0.
5. R(AX)= = XR(X) for > 0.
Coherent risk measures have some interesting properties. In particular, risk measure coherent and only if, there is a risk envelope U such that
R(X) sup [
(1)
SEl4
(Arzznrr et al., 1999 Delbaen 2002 Rockafellar et al. 2002). risk envelope is nonempty convex subset of P that is closed and where P (6 L
0, B(g] 1}.
The definition of risk measure R. might depends on probability distribution p. some cases, it may be useful to specify which distribution are working with. Thus, sometimes use the notation Ry. There exists lot different coherent risk measures, for example the Wang risk measure (Wang) 20000), the entropic Value-at-Riss (Ahmadi-Javid 2011) or Conditional Value-at- Risk (Rockafellar et al. 2000; Rockafellar & Uryasev 2002).
Conditional Value-at Risk (CVaRa) with probability level (0,1) is defined
CVaR (X) = min f +
tER 1 - IFmmax (0, X - +}]}
(2)
Moreover the risk envelope associated CVaR,,, can be written as l {ō B 1,
5 S } (Rockafellar et al. 2006 2002).
This rigorous definitions may be not really intuitive, but roughly speaking CVaR,, is the expectation of X in the conditional distribution of its upper a-tail, and thus is corresponds to the (1 - a) worst case. In this paper, follow the classical definition of Conditional Value-at-Risk presented risk measure literature. In particular, X should be interpreted as loss function.
Offline RL. We consider a Markov Decision Process (MDP), (S, A. No: 7) where the en- vironment space, A the action space, r the reward distribution ( (ry ~ r(-[st, az)), is the transition probability distribution (8t+1 ~ p(-1st, at)), Ao is the initial state distribution and y E (0, 1) denotes the discount factor. For the purpose of the notation, we define p(sp) == (o and we write the MDP (S, A,p,r,
Actions are taking following policy which depends on the environment state, (i.e a. r-+s:)). sequence on the MDP (S, A, 77), T 8q, ao; To.... 8r, with 8; E S and a; E A called trajectory. A trajectory of fixed length H E N is called episode. Given a policy x, a rollout from state-action (s, a) E S x A is a random sequence ((s0. ag ro), (s1, (1,,a,,7 ri)..... where ag = a, x--sst). Given policy and fixed length H,
so s, 81-1 ~ p(-1st, az). ~ r(-|s,, (e|st,a,) dz) and we have trajectory distribution given by
H-1
(T = p(so)
II t=
T (aylss))(s41lls (e,,a)))( (r)r(rr(s,,,)))
The goal of classical risk-neutral RI. algorithms to find policy which maximizes the expected discounted return E, Ex / 7'r,] where H might be infinite. Equivalently, we can look for the policy which maximizes the Q-function which is defined Q A R. Qu az) E, D_! (rrr) Instead of this classical objective function, other choices are possible. For B= + *++[(-(*))) where H denotes entropy. This function has an interesting connection with variational inference (Ziebart 2010 Levine 2018) and it has shown impressive results in recent years (Haarnoja al. 2018 Lee al.
example, the maximum entropy RL objective function E, T
2019 Rafailov et al. 202I).
1t41)), In offline RL, we have access to fixed dataset D {s2,, Tr 8c4 (), where s, E S, ar E A, r(s,, ae) and 8141 ~ p(-1s,, a.), and we aim to learn policies without interaction with the environ-
ment. A such dataset comes with the empirical behaviour policy (a(a)s) = Eimapee (6 2
3

---

Under review as a conference paper at ICLR 2023
Latent variable model. There are different methods to learn directly from high dighesinnsio states Kostrikov et al. 2020; Hafner et al. 2019a b). But in this work, we build on the top of the frame- work presented Stochastic Latent Actor-Critie (SLAC) (Lee et al. 2019). The main idea of this work is to train a latent variable model to encode the natural MDP (S, A, P, 7) into a latent space (Z, A, 9, 7) and to train policies directly in this space. To achieve this, the variational distribu- tion (21.H, Ge+1:1l8 1s1e, (1aa) is factorized into product of inference term (2 81410;), latent dynamic term g(z++1 Ats ax) and policy term (a, 81:0, 01:t-1) as follow
H-1 II
H-1 II
q a141:8 $1:1, a1.)
II 2
(2 az) I (0o1s1..011-1
Using this factorization, the evidence lower bound (ELBO) (Odaibo, 2019) and a really interesting theoretical approach (Levine 2018), the following objective function for the latent variable model derived
log D(s(+1lZ(+1) Dww st, s,, ())la(((41151) ar))
(3)
where DEL is the Ballback-Leible divergence and D a decoder.
Distributional RL. The goal of distributional RL is to learn the distribution of the discounted cu- mulative negative rewards Z-
Σt 7/r, Z, is random variable. A classical approach of distributional RL is to learn Zn implicitly using its quantile function Fz(s.a) [0, + R. which is defined as F (Z(sa)))) (v) == inf(x E R y S (z)) and where F2(s.a) is the cu- mulative density function of the random variable Z(s, A model Qo(1, 8, a) is used ap- proximate F=1
ZZ(eaa (n). For (s, D, a' x(-ls), n. Uniform (0, 1), we define 8 as Qo(n, 8, a). Qo is trained using the Hubeer quantile regression loss at threshold it (Huber 1992)
= r + 7Q2r (
|/ L 18 22k) 17 Isco (10) k/2)
if | < k otherwise.
(5, =
(4)
With this function F2 different risk measures can be computed, like Cumulative Probability Weight (CPW) (Tvrrsky & Kahneman, 1992). Wang measure (Wang] (2000) or Conditional Value- at-Risk. For example, the following equation (Acerbi 2002) is used to compute CVaRa
CVaR, (Zm) a [ (2-1
(5)
It is also possible to extend distributional RL in a offline settings. For example, O-RAAC (Urpi al. 202T) and following previous work (Fujimoto al., 2019), decomposes the actor into two different component imitation actor and perturbation model. CODAC (Ma et al. 2021) extends DSAC (Duan et al. 2021) in a offline settings. More precisely, the Qo(1) 8, a) are trained using following loss function
aEnuU EOND log Σ exp(Qo(n.ssaa) - (s.))D [Qo(n s, a)l
+ Lx(
(6)
where U = Uniform(0, 1]. The first term of the equation, and based on previous work (Kumar al, 2020). is introduced to prevent for too optimistic estimation for OOD state-action pairs. The second
term, (5 T), the classical objective function used to train the Q-function in distributional settings.
THEORETICAL CONSIDERATIONS
In this paper, aim to train policies in high sich-immensional stochastic and offline settings. For practical point of view, our idea is quite straightforward. Since training with high dimensional space directly failed and following previous work (Rafailov al. 2021 Lee al. 2019), encode
4

---

Under review as a conference paper at ICLR 2023
high-dimensional states in more compact representation using ø S Z, where Z = d(S). Using 6, we build a MDP on the latent space and train policies directly on the top of this space.
However, for theoretical point of view, this general idea is not really clear. Is this latent MDP always well defined we find a policy which minimizes a risk measure in the latent space, will also minimize the risk measure in the natural state
The first goal of this section is to rigorously construct a MDP in the latent space. Then, we show that for special coherent risk measure and assuming some hypothesis, minimizing the risk measure in the latent space and in the natural space equivalent. In particular, we show that is the case for Conditional Value-at-Risk.
4.1 THEORETICAL RESULTS
First, we make the following assumptions
1. Va E A we have r(:1s, a) = r((lss, if (()) ((8))
2. We note (P() uz) the probability measure with probability density function p(11st, az).
We suppose that if st,5; E satisfy ((*+) = ((((), then PP(-18,, ax) = TP(11,4, 0t).
3. We note Q(11z,, ag) the probability image of [P(-)s, az) by @ and where 8t any element of x (z,). We suppose Q(-IFe, 0z) admits probability density functions (())=).0)).
Under these assumptions, ø induces a MDP (Z.A, where the reward distribution r' is de- fined as r'(| a) == ((-ls,a) for any s E (z). r' is well defined by surjectivity of and by assumption (1), Q is well defined by hypothesis (2).
Obviously, for given policy x' on the MDP (Z.A. q. 7) and fixed length H, we have trajectory distribution
H-1 II n' (a (())))))(2442)))) (r)r (relze, ax)
Given a policy æ and fixed length H, we denote Q the of all trajectories on the MDP (S, A, r), F the o-algebra generated by these trajectories and Fa the probability with prob- ability distribution Px. Following the same idea and given a policy æ', we denote Q' the set of trajectories of length H on the MDP (Z,A, q,r"), F the o-algebra generated by these trajectories and Qx' the probability with probability distribution gx.. F,F,PPT and (I', F" Qxr) are proba- bility spaces.
6: S > induces a map
Ω: 52'
(80. do, To. . 88) (ø(so), ao, ro ... ((0))
With slight abuse of notation, we write it ạb
We denote the set of all policies и on the MDP (S, p, which satisfies (als) (als') d(s) = ((s'). If E II, then induces a policy x' on the MDP (Z,A,g,r') taking N r'(a|z) o (als) where is any element (z).
For any E II we denote x the associated policy on the MDP (Z, A, 4. as defined above. Furthermore, we note II' == {x' π E III. If T' e II', a policy क E II can be defined ([ass) == ('(o((s)) X is random variable on Ω and X' random variable on !'.
With all these notations, we can introduce our first result.
Lemma 4.1.1. Let II. Then, @r the probability image of P- by d.
This result is not really surprising but it has an interesting implication. Indeed, it applies that if X' o O = X, then [X'] = [X]. And thus, we get the following result.
Corollary 4.1.2. Suppose X' o = X. Then, if a policy x¹, satisfies
T = arg min E. [X']
x ETTT'
5

---

Under review as a conference paper at ICLR 2023
its associated policy n, satisfies
T. = arg min E [X] wIll
For a coherent risk measure R, we write U the risk envelope associated to R in Ω and U' the corresponding associated risk envelope in Q. 16 u. 38 U' such that = 8' o and Vo' E U' 36 E N with ô = 8' a & almost everywhere, we write U = Ul a Φ.
Proposition 4.1.3, Let be coherent risk measure. Suppose that X' o Φ X and = Ul og. Then, if a policy n², satisfies
π argmin anx
its associated policy n. verifies
argmin=en]RR(
This last result points out the role of the risk envelope in this equivalence.
However, the result of the last proposition is also verifies if sup E, (6' X'] suPseu Ep, [6X]. Fortunately, this is the case for Conditional Value-at-Risk:
Lemma 4.1.4. Let U, Ut the risk envelopes associated to CVaRa and CVaRa respectively. Then, we have
sup
Bgg [8'X'] = sup Epp [8x]
8'etr'
Sell
Therefore, we can deduce the following result. Corollary 4.1.5. Suppose that X' o X. Then, if a policy x4, satisfies
n. argmin='ent CV
its associated policy n, verifies
7. En
In particular, for X(T) Et=o -y'r 3H((T-(s:)), and X'(r)) = = -
(with 3 E R) we have
H
H
>'r ())(((1s))) (x' o (T) = Σ -'' (-10(())))))))) = Σ
==0
Thus, we obtain the following result.
Corollary 4.1.6. Let X : Ω + R and X' : S2' + R defined as ( = -30
and X'(r)) = D!' -7're- - 37( (- Then, if policy n², satisfies
मं en
its associated policy n, verifies
argmin nnll CVaR (X)
4.2 DISCUSSION
The results presented the last section provide theoretical equivalence between minimizing the risk measure in the latent space and in the natural space. However, to obtain this guarantee we need to make some assumptions.
Assumptions (1), (2) guarantee that the reward distribution r(-[s;, ax) and the probability measure (- at) are insensitive to any change in ò d=1(2)) (zx). Moreover π E 1l ensures that the policy distri- bution is stable to any change in ø (22). Thus, and roughly speaking these assumptions guarantee we do not loss information encoding s; into (($)), in term reward distribution, transition proba- bility measure and the optimal policy.
These theoretical considerations points out that to learn good latent representation of the natural MDP, should consider all components of the MDP and not only focus the environment space.

---

Under review as a conference paper at ICLR 2023
5 LATENT OFFLINE DISTRIBUTIONAL ACTOR-CRITIC
The theoretical results presented above justify our really natural idea: encode the natural environ- ment space S into smaller representation Z and then risk-sensitive offline RL algorithm to learn policy on the top of this space. This is the general idea of LODAC.
5.1 PRACTICAL IMPLEMENTATIONS OF LODAC
In this section, we present the practical implementation of LODAC. First, we need learn latent variational latent model. To train a such model, and similar to Rafailov et al. (202I), the following objective function used
Ego Σ
log D(5441l241)
2e, (2)1100
Then, the dataset D is encoded into the latent space and stored into a replay buffer Bratet More precisely, Blate contain transitions of the form (21:81 T1.H; 01:H) where 21:11 ~ 00(-181:11, and $1:H1 11:H-1, 41:H1 D. After that, we introduce a latent buffer B. Bsyathetic which contains rollouts transitions performed using the policy wæ, the latent model do and a reward estimator rg.
The actor Te and the critic Qo are trained on B := Bsynthetic Blatent- To achieve this, we could use any offline risk-sensitive RL algorithm. But based on our empirical results we follow Ma et al. 2022). Thus, the critic Qn(n, 24, az) is used to approximate the inverse quantile function. Qo is iteratively chosen to minimize
Ez-B log Σ exp(Qo(1; [0 (a
(Q0(1, 2,
(8)
where U = Uniform (0, 1], 8 r; + 7Qo(n' 24+1,0441 - Qa 2c, az), with (#e: ae, re) B, a++1 ~ No (-)a,) and La is the HuHber quantile regression loss as defined (4p. The actor trained to minimize Conditional Value-at-Risk of the negative cumulative reward, which can be computed using the formula
[
CVaR, (Zx)
Fz-
1
Following previous works (Rafailov al. 2021 Yu et al. 2021b), batches of equal data mixed from Bieent and Byyhhhric are used to train Na and Qa. A summary LODAC can found Appendix
D
5.2 EXPERIMENTAL SETUP
In this section, we evaluate the performance of LODAC.
First, our method compared with LOMPO Rafailov et al. 2021). which is an offline high- dimensional free algorithm. Then. we build a version of LODAC where the actor and the critic are trained using O-RAAC (Urpí et al. 2021). We denote LODAC-O. Moreover, it is also possible to use risk-free offline RI. Thus, risk-free policy is also trained in the latent space using COMBO Yu et al. 2021b).
These algorithms are evaluated using the standard walker walk task from the DeepMind Control suite (Tasaa et al.) 2018), but here, we learn directly from the pixels. As a standard practice, each action is repeated two times the ground environment and episode of length 1000 are used. These algorithms are tested on three different datasets expert, medium and expert-replay. Each dataset consists of 100K transitions steps.
For the stochastic environment, we transform the reward using the following formula
Pt (r(st, ax) Al (2
where r is the classical reward function. r. are hyperparameters and B,,, is a Bernoulli distribution of parameter Po- In experiments, choose À and Po 0.1. Different value of r used

---

Under review as a conference paper at ICLR 2023
Table 1: Performance for the stochastic offline high-dimensional dalker_walk task expert, medium and expert datasets. We compare the mean and the CVaRo.7 of the returns. LODAC outperforms other algorithms on the medium and expert replay datasets while achieving comparable results on the expert dataset. We bold the highest score across all methods.
Dataset type
Algorithm
Mean
CVaRa.7
expert expert
LOMPO
8.67
1.92
COMBO
10.96
3.72
expert
LODAC-O
18.54
4.92
expert
LODAC
18.70
4.32
medium
LOMPO
19.03
13.31
medium
COMBO
16.25
11.18
medium
LODAC-O
21.31
14.99
medium
LODAC
27.85
21.37
expert
LOMPO
23.22
16.03
expert replay
CОMBO
20.01
14.12
expert replay expert.replay
LODAC-O
5.73
0.28
LODAC
39.4
30.14
Table 2: Performance for the deterministic offline high-dimensional walker_walk task on expert, medium and expert replay datasets. compare the mean and the CVaRg.7 of the returns. We bold the highest score across all methods.
Dataset type
Algorithm
Mean
CVaRo.7
expert expert expert
LOMPO
14.56
0.99
COMBO
16.74
1.97
LODAC-O
13.25
2.79
expert
LODAC
16.03
0.0
medium medium
LOMPO
52.2
42.56
COMBO
43.71
25.89
medium
LODAC-O
37.33
31.12
medium
LODAC
33.08
5.79
expert_replay
LOMPO
55.86
35.10
expert.replay
COMBO
62.88
27.69
expert.replay
LODAC-O
15.81
2.90
expert.replay
LODAC
48.01
27.25
for each dataset such that about the half of the states verify r(;; 0x) > T. More details regarding the construction of these datasets can be found in Appendix B
Algorithms have been tested using the following procedure. First of all, to avoid computation time and for more accurate comparison, we use the same latent variable model for all algorithms. We evaluate each algorithm using 100 episodes, reporting the mean and CVaR,, of the returns. LODAC and LODAC-O are trained to minimize CVaRo.7. LOMPO and COMBO are trained to maximize the return. We run 4 different random seeds. As introduced in Fu et al. (2020), we use the normalized score to compare our algorithms. More precisely, a score of 0 corresponds a fully random policy and of 100 corresponds to expert policy the deterministic task. However, and as suggested in Agarwal et al. (202T), instead of taking the mean and the results. we consider the interquantile means (IQN).
8

---

Under review as a conference paper at ICLR 2023
5.3 RESULTS DISCUSSION
In this section, we discuss the results of our experiments. Our experimental results can be read in the table the stochastic environment in table for the deterministic environment. We bold the highest score across all methods. Complete results of all different tests can be found in Appendix
Stochastic environment. The first general observation is that LODAC and LODAC-O gener- ally outperform risk-free algorithms. The only exception is with the expertrreplay dataset where LODAC-O provides worse results. This contains an imitation component and obviously the imitation agent obtains really poor performance on this dataset. Then, it can be noticed, that LODAC provides really interesting results. Indeed significantly outperforms risk-free algorithm in term of CVaRo.7 and return on all datasets. More- over, provides better results than LODAC-O on the medium and the expert dataset, while achieving comparable result the expert dataset. A final observation is that risk-free RL policies provide generally bad performances on this stochastic environment.
really surprising since actors trained with O-RAAC
Deterministic environment. First, drop of performance between the deterministic and the stochastic environment can be noticed. This is not really surprising since stochastic environments more challenging than deterministic environments. However this modification seems to affect more risk-free free algorithms than LODAC-O or LODAC. Indeed, we get a difference of more than 27% with LOMPO and COMBO the medium and expert replay dataset in term of return between the deterministic and the stochastic environment. This difference even achieves 42% for COMBO on the expery dataset. For LODAC-O and for the same tasks, we obtain a deterioration of less than 17%. In the opposite to previous risk-free methods and in a lesser degree LODAC-O, adding stochasticity in the dataset does not seem have big impact of the performance of LODAC. a difference of only 5, 23% can even be noticed.
deed, with this algorithm we observe drop of performance of less than 9%. For the medium dataset,
CONCLUSION
While offline RL appears be an interesting paradigm for real-world application, some of these real- world application are high-dimensional and stochastic. However, current high-dimensional offline RL algorithms are trained and tested deterministic environments. Our empirical results suggest that adding stochasticity to the training dataset significantly decreases the performance of high- dimensional risk-free free offline RL algorithms.
Based on this observation, we develop LODAC. LODAC can be used to train policies in high- dimensional stochastic and offline settings. Our theoretical considerations in section show that our algorithm relies strong theoretical foundation. In the opposite previous risk-free methods, adding stochasticity in the dataset does not seem to have a big impact of the performance of LODAC. Finally, the of LODAC minimize Conditional Value-at-Risk empirically outperforms previous algorithms in term of CVaR and return on stochastic high-dimensional environments.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking Finance,
2002.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural informa- fion processing systems, 0122304-293220 2021.
Amir Ahmadi-Javid. An information-theorete approach constructing coherent risk measures. In 2011 IEEE International Symposium on Information Theory Proceedings, pp. 2125-2127. IEEE, 2011.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008. 05556, 2020.
Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures risk. Mathematical finance, 93)):033-23 1999.
Annie Š Chen, HyunJi Nam, Suraj Nair, and Chelsea Finn. Batch exploration with examples for scalable robotic reinforcement learning. IEEE Robotics and Automation Letters, 63)):401-4408; 2021a.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning. Advances in Neural Information Processing Systems, 34, 2021b.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. arXiv preprint arXiv:/466.38. 2014.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. arXiv preprint orXiv:1506.021880 2015.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp. 1096- 1105. PMLR, 2018.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and trends in Robotics, 2(1-2):388-40)) 403, 2013.
Freddy Delbaen. Coherent risk measures on general probability spaces. In Advances in finance and stochastics, pp. 1-37. Springer, 2002.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems, 2021.
Yingjie Fei, Zhuoran Yang. Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances Neural Information Processing Systems, 34, 2021.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp- 2786-2793. IEFE, 2017.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 0052-2062. PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2023
Javier Garcia and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofrr Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pp. 2170-2179. PMLR, 2019.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 883..000.... 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 191.00163300 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565, PMLR, 2019b.
Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv:1912.1077 2019.
Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne, Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50,2):133,,00 2017.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. 2019.
arXiv preprint 19X1../911.222
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprini arXiv:2004. 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: (ww.1990009999 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 international joint conference on neural networks (IJCNN) pp. 1-8. IEEE, 2010.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-19895 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint 1xXiv:1977.000 2019.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances Neural Information Pro- cessing Systems, 33:741-752, 2020. 2020.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. 2018.
arXiv preprint a8xiv
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):3334-1355 2016.

---

Under review as a conference paper at ICLR 2023
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, and Tie- Liu. Curriculum offline imitating learning. Advances in Neural Information Processing Systems, 34, 2021.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-poolicy policy gradient with 2019.
state distribution correction. arXiv preprint
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint 4rX1v 2020.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforce- ment learning. Advances in Neural Information Processing Systems, 34, 2021.
Ofir Nachum, Bo Dai, Πya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: http:///22.000.444 2019.
Ashvin Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807.04744 04742,, 2018.
Stephen Odaibo. Tutorial: Deriving the standard variational autoencoder (vae) loss function. arXiv preprint arXiv:1907.089%: 2019.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Matihew Rabin. Risk aversion and expected-utilits utility theory: A calibration theorem. In Handbook of the fundamentals of financial decision making: Part I, pp. 241-252. World Scientific, 2013.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168. PMLR, 2021.
R Tyrrell Rockafellar. Coherent approaches risk in optimization under uncertainty. In OR Tools and Applications: Glimpses of Future Technologies, pp. 38-61. Informs, 2007.
Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of banking & finance, (677):443-1147,, 2002.
R Tymell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
Tyrrell Rockafellar, Stanislav P Uryasev, and Michael Zabarankin. Deviation measures in risk analysis and optimization. University of Florida, Department of Industrial & &ystems Engineering Working Paper, (2002-7), 2002.
R Tyrrell Rockafellar, Stan Uryasev, and Michael Zabarankin. Generalized deviations in risk analy- sis. Finance and Stochastics, 10(1):51-74, 2006.
Sergey Sarykalin, Gaia Serraino, and Stan Uryasev. Value-at-risk vs. conditional value-at-risk in risk management and optimization. In ofate-of-the-art decision-making tools in the information- intensive age, pp. 270-294. Informs, 2008
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss its own reward: Self- supervision for reinforcement learning. arXiv preprint arXiv: 102.:7761 2016.
Susan M Shortreed, Eric Laber, Daniel Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Mur- phy. Informing sequential clinical decision dccision-making through reinforcement learning: an empirical study. Machine learning, (4(1-2):19-111 2011.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7787):484 489, 2016.
12

---

Under review as a conference paper at ICLR 2023
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play play. Science, 362 1144, 2018.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distri- butional reinforcement learning. In Learning for Dynamics and Control, pp. 958-968. PMLR, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801. 00690, 2018.
Amos Tversky and Daniel Kahneman. Advances prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 544):297-323, 1992.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learn- ing. arXiv preprint arXiv:2102.05371, 2021.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34, 2021.
Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2447- 2456, 2018.
Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pp. 15 16,36, 2000.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally lincar latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning. and optimization, 12(3), 2012.
Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. 2021.
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline
model-based optimization. Advances in Neural Information Processing Systems, 34, 202
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint xrX1v:2102.083633 2021b.
Kaiqing Zhang, Xiangyuan Zhang, Bin Hu, and Tamer Basar. Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity. Advances in Neural Information Processing Systems, 34, 2021.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Camegie Mellon University, 2010.
13

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 LEMMA
Lemma A.1.1. Let π E LL. Then, Qr the probability image of Pa by c.
Proof. Let B' = Zo Ao x Ro x Zu E F'. We have
(B') = ( (T''''
/ g(20)
(
z0) ........9222/ -((=_l=--,,n---)dda dzz dzo
20
l 1160)
p(so)
r'(col((s))) (rol au
dzи dso
[ (20)
( ((ools))r(rola... ... 11 dzz dso
p(50)
/
(Zo)xxxxo
(
dzæ dspdapdro
=fo Aox 12,)
(rnp ao)
VAN ((21b(s0)) ... 922/2/ ... dzz dsndagdrods
/ x(ooxx... 1Zn)
((s))m(aols)))(((oooo. so) pas../pss/1111111 1 dan
Pr (T)
(-1(B))
A.2 PROPOSITION 4.13
Proposition A.2.1. Let R be a coherent risk measure. Suppose that X' o o = and = U1 o og Then, if a policy n², satisfies
n¹. argmmnn."nn R(X')
its associated policy n, verifies
argmin ((X())
Proof. Since U 11' di we have
sup s'ell'
B [6'X'| sup Ep. [8X]
Sell
Thus
Rg, (X') = sup E,g, 16' X'| = sup Ey, [5X] = Rp, (X)
s'Ell'
sett
Now, let n/. argmin wn Rg (X') and m. the associated policy in S. By contradiction, suppose
there exists 1 with R (X) < Ry,. (X). But in this case and by the above observation, we would
have Ry,; (X') RØy (X').
14

---

Under review as a conference paper at ICLR 2023
A.3 LEMMA 4.1.4
Lemma A.3.1. Let U, UI the risk envelopes associated to CVaR, and CVaRa(X) respectively. Then, we have
sup
Bu [8'X'] sup Ep, [6X]
8'ell'
kEl
Proof. Recall that the risk envelope of the CVaRa. takes the form U = [6 a. B|o] 1}.
Let 8 E U. We define = 8' o Φ. By construction we have $ E Ep [6]1 E,,, 1 and Eg,, [6'X'] = Ep. [6X].
Let E u. By definition, óp* is a density function. Let Pb its associated probability measure. We consider Q the probability image of P- by ø. Then, remark that Qr is absolutely continuous with respect Qar. Indeed, if A E F satisfies Qx 0, have
(A) ( /
((F)px(T)dr
4 (A)
1
Po (A))
Qur (A)=1 =0
a
Thus by Radon-Nikodym, there exists 8' 0, such that for all B F
(B) 8 8'
We will show that à 8' Ut We define C= (r' E (8(()) > g. By contradiction, suppose that Q.. > 0. One one hand have
Qa. (C) / ( c
(or( -- Qm.(C))
And the other hand, have
1 Q. (C)
(T)) S Px(0 (C) -P (0-1(C)
And thus, must have (C) =.
Thus, 0 s 8' < 1 a.c. and therefore E U. And since, grr is the density function of the probability image of P... we obtain Exp, [X] Es [X'].
B DATASETS
In this section, we describe more precisely how we build our training datasets.
Expert. For the expert dataset, actions are chosen according to expert policy which has been trained online, risk-free environment using SAC for 500K training steps. For this training, the states are the classical states provided by DeepMind Control suite.
Medium. In this dataset, actions chosen according to a policy which has been trained using the same method as above, but here, the training has been stopped when it achieves about the half performance of the expert policy.
Expert replay. The expert replay dataset consists of episode which are sampled from the expert policy during the training.
15

---

Under review as a conference paper at ICLR 2023
The setup presented above allow build the deterministic datasets.
For the stochastic environment, we transform the reward using the following formula
Tt (r(st, ae) A (
where r is the classical reward function. F, À are hyperparameters and B is a Bernoulli distribution of parameter Po- In our experiments. we choose for each dataset such that about the half of the states verify r(st, at) > F.
and Pu 0.1. Different value of F are used
C IMPLEMENTATIONS DETAILS
In this section, we present some details related to our implementations.
Latent variable model. Following previous works (Rafailov et al. 2021 latent variable model contains the following components
Yu al. 2021b). our
Image encoder:
h4 Eg(st)
Inference model
60 (- hhe 21 a1
Latent transition model:
21 40 (-1 0t-1)
Reward estimator
ro (1())
Image decoder:
8t D2(-12))
The image encoder Eg is classical Convolutional Neural Network. The inference and the latent transition model are implemented as Recurrent State Space Model (RSSM), (Hainer et al. 2019b). More precisely, latent state 2₁ has components 4 (d,, 22._ de is the deterministic part and 2t the stochastic. de is computed using a GRU cell d, = f0(=1-1 a4-1) and the stochastic component Te is computed using, Ty de (- de), for the inference model, and F; qe (-(dd,, Tt-1, 0t-1) for the latent transition model. Both and the reward estimator are implemented as MLP. Finally, the decoder Dg a Deconvolutional Neural Network.
Actor-critic. First, recall that our initial critic loss function is
E
ErwB
log Σ
exp(Q0(n, 2,
Ezz,a) [Qo(n 2, a)] + (x(),n))
(10)
But following Kumar et al. (2000) equation
Ma et al. (2021), we add two parameters G, E Rso to the last
max.20
Enpu
w EawB log Σ exp((Q(() (1)
(( (Q0(1,
thresholds the difference between ((z,)- [Qs(n, 2, a)] and the regulizer Ez-B [log E. exp(Qo()) a))]. The parameter scales this difference.
Remark that if this difference (scaled to w) smaller than the parameter Ç, then a will be set and only the term La (8, v/') will be taken into account.
Moreover, since the action space A continuous, the computation log tractable. To avoid this problem and as introduced in (Kumar et al. (2020)
aa exp(Qo z, a) in- we use
log Σ exp((Q((),,aa) xe log
1
Σ
M
exp((Q(() 2, a))
Σ
M
(xp(Qo(T,,,a))
2M
U(A)
2M
<(al22)
(A)
a
where U(A) = Un) and we choose M 10.
Then, we use implicit quantile network (IQN), (Dabney et al.) 2018) to represent our critic Qo (n) z, a) and the actor To (-1z) consists simple MLP which is trained to minimize
- Qo(n.2,a)d
CVaR, (Zx)
(11)
16

---

Under review as a conference paper at ICLR 2023
D MAIN ALGORITHM
The full implementation of LODAC can be read algorithm
Algorithm 1: LODAC
Input:
dataset DD models train steps, initial latent steps, number iterations, number actor-critic steps, rollout length L
for models train steps do
(1:H-1) Sample batch of sequence ($1.H, (1:H-1,,':HH- from D and train a variational latent
model using equation 7 and reward estimator
end
for Initial latent step do
Sample batch of sequence (81:31 01:H-1,, (1-H- 11:H-,, from
Sample Z1.H1 from the latent model and add the transitions (Z1.H, 01:H-111:H1) to Biaten-
end
for initial synthetic latent steps do
Sample batch of sequences ($1.H: (HH from from D
10
Sample set of latent states (0 (1:H-1) from the trained latent model. for
E S. do for h E {1: L}} do
11
Sample random action 04-1 and 2n go 9-(-22111,1111111100000111111111 ah-1): estimate the reward
T4 ~ Fig
Add (2h-1, 04-44-11 11 Zh, rh) B Bynaheric-
end
15
end
16 end
17
for
number iterations do for number actor-critic steps do
18
19
Sample transitions from Binren U
Byynheeti
20
Train the critic to minimize (8)
21
Train the actor to minimize CVaR. (Zx) which can be computed using the Q(n, z, a)
and equation 1
22
end for rollout steps do
24
from Sample batch of sequences ($1.H, 01. from D
Sample set of latent states S ~ (- 01:H--1111111 11) from the trained latent model. for
So E 5 do
for h E (1: L} do
Sample action 0h-1 ~ No( 1z- and zn ~ qo 1z--1 On 1); estimate the
reward Yz
Add (2h-1, 4h-1, Th) to B Bymthetic.
29
end
30
end
31
end
end
E COMPLETE EXPERIMENTAL RESULTS
In this section, we present our complete empirical results. More precisely, the table all results of the stochastic environment can be found. And on the table 3 all results on the deterministic can be read.
17

---

Under review as a conference paper at ICLR 2023
Table 3: Complete results for the deterministic offline high-dimensional walker walk task on expert, medium and expert datasets. We compare the mean and the CVaRg.7 of the returns.
Expert
Medium
Expeay
Mean
CVaRR.77
Mean
CVaRo.7
Mean
CVaRp.7
LOMPO
174
65
419
298
410
274
177
98
583
501
523
341
71
26
515
437
601
411
193
37
527
428
586
387
COMBO
217
62
345
263
662
395
190
58
458
296
742
501
201
95
554
476
576
197
38
21
428
246
359
160
LODAC-O
149
68
385
334
107
59
178
44
428
357
194
104
149
73
381
309
224
51
186
67
384
321
180
78
LODAC
265
43
628
458
460
271
103
31
416
52
505
313
306
65
275
134
421
193
113
34
194
56
509
313
Table 4: Complete results for the stochastic offline high-dimensional walker_wall walk task on expert, medium and replay datasets. We compare the mean and the CVaRo.7 of the returns.
Expert
Medium
Expertrreplay
Mean
CVaR
Mean
CVaR
Mean
CVaR
LOMPO
150
62
219
161
258
187
129
57
227
167
259
201
114
62
201
141
252
191
108
46
214
171
238
173
COMBO
169
123
196
153
238
182
167
101
202
159
171
101
73
40
179
132
250
192
118
51
186
136
213
161
LODAC-O
210
110
238
181
60
44
173
69
241
178
97
43
214
105
237
190
92
51
228
69
208
153
118
45
LODAC
303
93
299
244
406
326
112
23
287
235
391
279
227
87
299
241
413
336
200
76
296
228
401
311
18


Reviews:
[{"review_id": "wQZP5G0gWX", "reviewer": "Reviewer_c8jd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for Offline RL by explicitly modeling risk through CVaR. The paper targets stochastic environments and makes the case that modeling risk is essential for  performing well in these settings by demonstrating improved performance in standard control tasks with noise injection in the rewards. ", "summary_of_the_review": "The paper has interesting insights and motivations, but the experimental comparisons are incomplete and missing external baselines, evaluation environments and experiments settings. Please refer to the Weaknesses section above. \n\n--- AFTER REBUTTAL ---\n\nAfter reading the authors' responses, I am keeping my score and not recommending accept for the paper because only my first concern is reasonably addressed, and I am not convinced by the justification for significantly minimal experimental evaluations. ", "strength_and_weaknesses": "Strengths\n\n1. The proposed approach for risk-sensitive RL in a latent space is simple to implement, by combining with several prior offline RL algorithms.\n2. The motivation for necessity of modeling risk in stochastic environments is on point, and should be helpful in developing algorithms for these environments broadly. \n\nWeaknesses\n\n1. The practical implementation section is very vague and it is unclear how exactly the components of LODAC differ from prior approaches like Rafailov et al. (2021) More details regarding the algorithm is essential for better understanding the contributions\n\n2. The experimental settings do not conform to the motivations for risk-sensitive RL. The only stochasticity induced is a form of Bernoulli noise added to the rewards, and the dynamics are still deterministic. I think it is important to perform analyses on environments with stochastic dynamics to really understand the benefits of risk-sensitive algorithms.  \n\n3. The experimental comparisons are insufficient and are significantly minimal compared to evaluations that offline RL algorithms usually do. For example, only a single walker agent is considered for evaluation. I would suggest at least evaluating on the entire D4RL suite and comapring with other baselines like CQL, IQL, COMBO etc. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "mYP8r2uTxwD", "reviewer": "Reviewer_5Q9C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces LODAC, a variant offline actor critic method, which first embeds the state space into a latent space and also include risk measure into consideration. On the theory side, the authors show that minimizing risk measure in the latent space and in the natural space. On the empirical side, the authors consider an artificial environemnt (walker) which forces the deterministic reward function to be stochastic and justifies the efficacy of the proposed method. ", "summary_of_the_review": "The paper has an overall good presentation and some motivation. However, given the weaknesses stated above, I recommend reject. ", "strength_and_weaknesses": "Strength: \n - The equivalence between minimizing risk measure in the latent space and in the natural space seems interesting. \n\nWeaknesses:\n - Third paragraph in the intro: I don’t think I buy the logic here. “Classical” RL methods can also be evaluated under “stochastic” environment since their goal is to maximize **expected** total reward. The importance of developing risk-sensitive methods is to consider risk while maximizing total reward, which should not be counted toward training policies under stochastic environment. \n - The paper seems to be a naive combination of risk-sensitive offline RL and latent offline RL. I did not find any novelty beyond these two points. I suggest the authors further classify this. \n - Experiment seems too simple. I believe walker environment itself is already a low-dim env (18, see (Tassa et al., 2018)). It seems there is no need to further embed into low dimension. I suggest the authors consider a more complex environment. \n - The authors should also compare the results with other SOTA offline RL methods: COMBO, CQL, etc. There should be a trade-off between reward and risk. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9WUHedH4lxn", "reviewer": "Reviewer_eHQL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train a distributional actor-critic algorithm in a latent space for high-dimensional data in offline reinforcement learning. Risk measure instead of the classic expected return is optimized under the risk-sensitive RL framework. ", "summary_of_the_review": "Overall, I am concerned that the originality and depth of this contribution may not quite yet reach the high bar set for ICLR. The writing errors make reading stumble. \n\nWriting errors: \n- Section 1, paragraph 2, “it may be be more …”\n- Section 3, paragraph - Latent variable model, “Using this factorization, the evidence lower bound (ELBO) (Odaibo, 2019) and a really interesting theoretical approach (Levine, 2018), the following objective function for the latent variable model is derived … ” by any reference? \n- Section 3, paragraph - Latent variable model, “ … and D a decoder”\n- Section 3, paragraph – Distributional RL, “For example, O-RAAC (Urp´ı et al., 2021) and following a previous work (Fujimoto et al., 2019), decomposes the actor into two different component an imitation actor and a perturbation model.”\n- Section 3, paragraph – Distributional RL, “… , is introduced to prevent for too optimistic estimation”\n- Section 4, starting paragraph, “Since training with high dimensional space directly failed and following previous work (Rafailov et al., 2021; Lee et al., 2019), we encode our …”\n- ……\n\nTheoretical concern: the first two assumptions require the encoder $\\phi$ to be a one-to-one mapping function, while this is hard to be satisfied when using non-linear neural networks state encoders, for example, $\\phi$ occasionally maps both two states $s$ and $s’$ to zero vectors.  This conflicts with the use of trained variational encoder in the algorithm part, so the theoretical analysis doesn’t support the proposed algorithm. \n\nExperiments are not standard and only walker_walk task is considered. The dataset is collected by running SAC based on classic states provided by DeepMind Control suite. If the dataset is collected by policy which is trained not based on pixel input, why you trained your model on this dataset via pixel input. Limited number of baselines are considered here, and the paper didn’t include implementation details of these baseline methods. \n\nIs LODAC-O a O-RAAC trained with your latent encoder? \n", "strength_and_weaknesses": "Strengths: \n1. This paper combines the methods from prior methods. \n\nWeaknesses:\n1. This paper lacks novelty. The latent model for encoding high-dimensional states is borrowed from Rafailov et al. (2021), and the distributional Q-learning is originated from Ma et al. (2021).\n2. The writing is coarse, and many typos appear abruptly.  \n3. Theoretical analysis doesn’t support the proposed algorithm due to unsatisfactory assumptions. \n4. Experiments are only conducted on customized walker datasets. \n\nRafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154–1168. PMLR, 2021.\n\nYecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-utWAAas2m", "reviewer": "Reviewer_1ct4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the risk measure instead of the classically expected return in the offline RL setting. This paper shows that under some assumptions it is equivalent to minimize a risk measure in the latent space and in the natural space. A practical approach is proposed to use CVaR as the objective function.", "summary_of_the_review": "The paper is easy to follow and combines distributional RL with offline RL. My concerns are:\n1. the idea is incremental and the performance is not surprising. The method simply combines CVaR and an RL method. There is no specific design for the offline setting. Since distributional RL methods like C51 can achieve better performance than classical RL methods, the performance shown in this paper is not surprising.\n2. For experiments, only one environment is used, which is not very convincing. More results should be provided.\n\ntypos: the equation above Eq. 3: s_i+1a_i -> s_i+1, a_i; a MDP -> an MDP; equation before Lemma 4.1.1: \\phi(s_0) -> \\phi(s_H)", "strength_and_weaknesses": "Strengths: easy to follow, good performance\nWeaknesses: Incremental idea, insufficient experiments", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject