Risk-Averse Offline Reinforcement Learning
Authors: Núria Armengol Urpí
Sebastian Curi
Andreas Krause
Venue: ICLR 2021 Poster (2021)
Keywords: offline
reinforcement learning
risk-averse
risk sensitive
robust
safety
safe

Abstract: Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criteria, namely the CVaR. In particular, we present the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns policies with higher CVaR than risk-neutral approaches in different robot control tasks. Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts. We demonstrate empirically that in the presence of natural distribution-shifts, O-RAAC learns policies with good average performance. 


Published as a conference paper at ICLR 2021
RISK-AVERSE OFFLINE REINFORCEMENT LEARNING
Núria Armengol Urpí Dept. of Computer Science ETH Zurich na mazeengoluriii@mai.... com
Sebastian Curi
Andreas Krause
Dept. of Computer Science ETH Zurich
Dept. of Computer Science ETH Zurich
scuriting werriiinfeeee www..eeeee....mm ch
krausea@ethz. ch
ABSTRACT
Training Reinforcement Learning (RL) agents online in high-stakes applications is often prohibitive due to the risk associated with exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criterion. In particular, we present the Offline Risk- Averse Actor-Critic (O-RAAC), model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns poli- cies with higher aivr-e performance than risk-neutral approaches in different robot control tasks. Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts. We demonstrate empirically that in the presence of natural distribution-shine O-RAAC learns policies with good average performance.
INTRODUCTION
In high-stakes applications, the deployment of highly-performing Reinforcement Learning (RL. agents is limited by prohibitively large costs early exploration stages (Dulac-Armold al., 2019). To address this issue, the offline (or batch) RL setting considers learning policy from limited soccision-making typically also risk-averse: we batch of pre-collected data. However, high-stakes assign more weight to adverse events than to positive ones (Pratt, 1978). Although several algo- rithms for risk-sensitive RL exist (Howard & Matheson, 1972; Mihatsch & Neuneier, 2002), none of them addresses the offline setting. On the other hand, existing offline RL algorithms consider the average performance criterion and are nestaal (Ernst et al., 2005; Lange et al., 2012).
Main contributions We present the first approach towards learning a risk-averse RI. policy for high-stakes applications using only offline data: the Offline Risk-Averse Octor-Critin (O-RAAC). The algorithm has three components: distributional critic that learns the full value distribution (Section 3.1), a risk-averse actor that optimizes a risk averse criteria (Section 3.2) and an imitation learner implemented with variational auto-encoder (VAE) that reduces the bootstrapping error due to the offline nature of the algorithm (Section 3.3). In Figure 1, we show how these components in- teract with each other. Finally, in Section 4 we demonstrate the empirical performance of O-RAAC. httpg//tb..m///raaaaa////// Our implementation is freely available at Github: https: // h/g/tttuuu wwm/nuria95///0 RAAC.
1.1 RELATED WORK
Risk-Averse RL The most common risk-averse measure in the literature is the Conditional Value- at-Risk (CVaR) (Rockafellar Uryasev, 2002), which corresponds the family of Coherent Risk- Measures (Artzner et al., 1999), and we focus mainly on these missuressures. Nevertheless, other risk criteria such as Cumulative Prospect Theory (Tversky& & Kahneman, 1992) or Exponential Util- ity (Rabin, 2013) can also be used with the algorithm we propose. In the context of RI., Petrik & Subramanian (2012); Chow & Ghavamzadeh (2014); Chow al. (2015) propose dynamic program- ming algorithms for solving the CVaR of the return distribution with known tabular Markov Decision Processes (MDPs). For unknown models, Morimura et al. (2010) propose a SARSA algorithm for (CVaR) optimization but is limited to the on on-policy setting and small action spaces. To scale to larger systems, Tamar et al. (2012; 2015) propose on-policy Actor-Critic algorithms for Coherent Riakuressures. However, they extremely sample inefficient due sample discarding compute

---

Published as a conference paper at ICLR 2021
Critic Zu
Lentie (w)
a)
Fixed butfer (s,0,r,8))
Actor 750
Lactur (0)
VAE,
LVAE(Ø)
DZ
EZ
Figure 1: Visualization of the algorithm components. Solid lines indicate the forward flow of data whereas dashed lines indicate the backward flow of gradients. Data is stored in the fixed buffer. The VAE, in blue, learns a generative model of the behavior policy. The actor, in green, perturbs the VAE and outputs a policy. The critic, learns the Z-value distribution of the policy. The actor optimizes a risk-averse distortion of the Z-value distribution, which we denote by DZ. On the right, we show a typical probability density function of Z learned by the critic in red. In dashed black we indicate the expected value of Z, which a risk-neutral actor intends to maximize. Instead, a risk-averse actor intends to maximize distortion DZ, shown in dashed green. In this particular visualization, we show the ubiquitous Conditional Value at Risk (CVaR).
the risk-criteria and the Sagganariance of the gradient estimate. While Prashanth et al. (2016) address sample efficiency by considering Cumulative Prospect Theory instead of Coherent Zisk-Measures, their algorithm is limited to tabular MDPs and is also on-policy. Instead, Tang et al. (2020) propose an off-policy algorithm that approximates the return distribution with a Gaussian distribution and learns its moments using the Bellman equation for the mean and the variance of the distribution. Instead, we learn the full return distribution without making the Gaussianity assumption (Bellemare et al., 2017). Perhaps most closely related is the work of Singh et al. (2020), who consider also distributional critic but their algorithm limited to the CVaR and they do not address the offline RL. setting. Furthermore, they use sample-based distributional critic, which makes the computation of the CVaR inefficient. Instead, we modify Implicit Quantile Networks (Dabney el al., 2018) in order compute different risk criteria efficiently. Although (Dabney al., 2018) already investigated risk-related criteria, their scope is limited to discrete action spaces (e.g.. the Atari domain) in an off- off-policy setting whereas we consider continuous actions in an offline setting.
Offinee RL The biggest challenge in offline RL is the Bootstrapping Error: a Q-function is evalu- ated at state-action pairs where there little or no data and these get propagated through the Bellman equation (Kumar et al., 2019). In turn, a policy optimized with offline data induces a state-action distribution that is shifted from the original data (Ross al., 201 1). To address this, Fujimoto et al. (2019) propose to express the actor as the sum between an imitation learning component and per- turbation model to control the deviation of the behavior policy. Other approaches to control the difference between the data-collection policy and the optimized policy include regularizing the poli- cics with the behavior policy using the MMD distance (Kumar et al., 2019) or f-divergences (Wu al., 2020; Jaques et al., 2019), or using the behavior policy as a prior (Siegel al., 2020). An alternative strategy in offline RL. is to be pessimistic with respect to the epistemic uncertainty that arises due data scarcity. Yu al. (2020) take model-based approach and penalize the per-step rewards with the epistemic uncertainty of their dynamical model. Using model-free approach Ku- mar al. (2020); Buckman et al. (2020) propose learn lower bound of the Q-function using an estimate of the uncertainty as penalty in the target of the equation. Our work uses ideas from both strategies address the offline avsk-averse problem. First, we use an imitiation learner control the bootstrapping error. However, by considering a risk-averse criterion, we are also optimizing over pessimistic distribution compatible with the empirical distribution in the data set. The connections between risk-aversion and distributional robustness are well studied in supervised learning (Shapiro et al., 2014; Namkoong & Duchi, 2017: Curi et al., 2020; Levy et al., 2020) and in reinforcement learning (Chow al., 2015; Pan et al., 2019).
2 PROBLEM STATEMENT
We consider a Markov Decision Process (MDP) with possibly continuous S and possibly con- tinuous actions a E A. transition kernel P(-|s, a), reward kernel R(-|s, a) and discount factor y. We denote by stationary policy, i.e., a mapping from states to distribution over actions. We have
2

---

Published as a conference paper at ICLR 2021
access to fixed batch data set collected with unknown behaviour policy gg. We call d the joint state, action, reward, next-state distribution induced by the behaviour policy and p the marginal state distribution. We access this distribution by sampling from the fixed data set. Similarly, for any policy R, call d= the joint state, action, reward, next-state distribution induced by on the MDP.
In rist-neutral RL, the goal is to find a policy that maximizes the expected discounted sum of returns Ed- [ΣE= R(- St, Ar)], where the expectation is taken with respect to the stochasticity introduced by the reward kernel, the transition kernel, and the policy. We define as zdd (s, a) =n -ー=r"=1 R(-SS,, Ae) as the return distribution conditioned on (Si = 8, A a) and following π thereafter. Here = denotes equality in distribution. The risk-neutral RI. objective the expectation of the distribution of ZT.
In risk-averse settings, we replace the expectation with a distortion operator D that is a mapping from the distribution over the returns to the reals. Thus, the goal is find a policy ı that maximizes
max D
(1)
t=1
With this framework, we address many different risk-averse distortions. For example, this includes probability weighting functions (Gonzalez & Wu, 1999; Tversky & Kahneman, 1992). the CVaR (Rockafellar& & Uryasev, 2002). the mean- veriinnr criteria (Namkoong & Duchi, 2017) or the Wang criteria (Wang, 1996; Rabin, 2013).
3 OFFLINE RISK-AVERSE ACTOR-CRITES (O-RAAC)
We now present our algorithm O-RAAC for offline-risk averse RL. One of the main technical chal- lenges in going beyond expected rewards is to find an analogue of the Q-function for the particular distortion operator we want to optimize. Unfortunately, the Bellman target of most risk-analogues does not have closed-form expression. Therefore, we instead learn the full distribution of returns as proposed by Bellemare et al. (2017). In Section 3.1, we describe the training procedure for the distributional critic. Next, in Section 3.2 we define the actor loss as the risk-distortion operator on the learned return distribution and optimize it using a gradient-based approach. Up to this point, the actor-critic template is enough optimize a risk-averse criteria. However, as we focus on the offline setting, we need to control the bootstrapping error. To this end, we use a variational auto-encoder VAE to learn generative model of the behavior policy in Section 3.3. Finally, in Section 3.4 we bring all the pieces together and instantiate our algorithm for different risk distortions D.
3.1 DISTRIBUTIONAL CRITIC LEARNING
To learn the distributional critic, we exploit the distributional Bellman equation of returns (" =D R(s.a) + ( A') for policy evaluation. The random variables S' A' are distributed according s' P(-|s, a) and A' T--Is'. In particular, we represent the return distribution implicitly through its quantile function proposed by Dabney et al. (2018) 8). We use this representation because many risk distortion operators can be efficiently computed using the quantile function of the underlying random variable. We parameterize the quantile function through neural network with learnable parameters w. We express such implicit quantile function as ( a; T), where 7 E [0, 1] is the quantile level. Whereas the neural network architecture proposed by Dabney et al. (2018) is for discrete actions only, we extend it to continuous actions by considering all s, a, the
and as the inputs and only the quantile value the output. To learn the parameters w, distributional variant of fitted value-iteration (Bellemare et al., 2017: Munos & Szepesvári, 2008) using quantile Huber-loss (Huber, 1964) by Dabney et al. (2018). To this end, we use a target network with parameters w' and compute the temporal difference (TD) error at a sample (s, a, r, s') as
surrogate of the Wasserstein-distance as proposed
,r,5" = r r 7. (8', d':-') (3( for 7, 7' independently sampled from the uniform distribution, i.e., 7, U(0,1) 1) and a' ~ m(-ss'). The 7-quantile Huber-loss is
(2)
2x 82 8
if 18 K, otherwise.
Lx(ō;)) T 1 (8<0)
(3)
jK
Quantile loss
Huber loss
3

---

Published as a conference paper at ICLR 2021
We prefer the Huber loss over the L2 or L1 loss it is better behaved due to smooth gradient- clipping (Mnih et al., 2015). Fimally, we approximate the quantile loss for all levels 7 by sampling N independent quantiles T and N' independent target quantiles r'. The critic loss is
N N
Leritic (w) E (x N d(mm(
Σ
L (6nnr3))
(4)
3.2 LEARNING A
ACTOR
In risk-averse applications, generally prefer deterministic policies over stochastic ones because introducing extra randomness is against a risk-averse behavior (Pratt, 1978) and there no benefit of exploration often associated with stochastic policies in the offline setting. Hence, consider parameterized deterministic policies (x()) : + A. Given a learned distributional critic, we define the actor loss
Lactor - (Z (s, mo(s);T))]
where D is the operator that models risk aversion. We consider Markovian policies because these contain the optimal policy for many common risk distortions in the discounted infinite horizon setting (Ruszczyrsski, 2010, Theorem 4). To minimize the actor loss (5), we use pathwise derivatives of the objective (Mohamed al., 2020), computed by backpropagating the actor through the learned critic at states sampled from the offline data set. Minimizing the actor loss is equivalent to maximizing the aiskreverse performance.
We leverage the quantile representation of the critic (c.f. Section 3.1) .I) to compute the risk distortion operator inside the actor loss (5), Given a quantile representation of distribution, common risk distortions D can be efficiently approximated using sampling-base based scheme. In partieular, there exists a quantile sampling distribution Py associated to D such that
D (s, (s, = / Z To (p)::
1 K
Σ (s, (o(S))Te), ~ PD. (6) k=1
For cumulative prospect theory (Tversky & Kahneman, 1992) and coherent risk measures (Artzner et al., 1999), the associated quantile sampling distributions are well-known. the particular case of the CVaR, this is known as the Acerbi's 's formula (Acerbi & Tasche, 2002)
1 - | 2 (s, dT.
CVaR, (2 (s, a: F))
When other representations are used for the critic, computing the risk-distortion D becomes computationally expensive. In some particular cases, there variational formulas compute the risk-distortion 'D that require to solve an inner optimization problem. For example, Singh et al. (2020) the common Rockafellar truncated optimization procedure (Rockafellar & Uryasev, 2002) to compute the CVaR, which is sample inefficient and has high variance (Curi et al., 2020).
3.3 OFF-POLICY TO OFFLINE: CONTROLLING THE BOOTSTRAPING ERROR.
Up to this point, the actor-critic procedure we describe in Sections 3.1 and 3.2 is theoretically suffi- cient to learn risk-averse RL agent. However, in the offline setting the bootstrapping error appears: when evaluating the TD-error (2), the Z-value target will be evaluated at actions where there is no data (Kumar et al., 2019) and propagated through the Bellman equation. To address this issue, (Kumar et al., 2019; Wu et al., 2020; Siegel et al., 2020) propose stochastic policies and penalize deviations from the behaviour policy while optimizing the actor.
As discussed in Section 3.2, on one hand we prefer deterministic policies over stochastic ones for risk-averse optimization and on the other hand, we prefer stochastic policies avoid overfitting to the fixed batch of data. To this end, we use similar parameterization to Fujimoto et al. (2019) and decompose the actor two components: an imitation learning component π II, and a perturbation model Eo such that the policy is expressed as:
(s)) + X-o(-1s,
s.t. b (.Iss.
4

---

Published as a conference paper at ICLR 2021
That is. b is action sampled from the imitation learning component, Çe is conditionally deterministic perturbation model that is optimized maximizing the actor loss (5) and X is a hyper- parameter that scales the perturbation magnitude. Thus, all the randomness in policy arises from the behaviour policy and not from the subsequent optimization. Furthermore, if we would have access to the behaviour policy Ts we could directly replace the imitation learning module by xg-
To learn generative model of the T.. from state-action pairs from the behaviour distribution dl we use a conditional variational autoencoder (VAE) (Kingma & Welling, 2014), which is also done in Fujimoto al. (2019). The main advantage of the VAE compared to behavioral cloning (Bain & Sammut, 1995) is that it does not suffer from mode-collapse which hinders the actor optimization, and compared to inverse imitation learning (Ziebart al., 2008; Abbeel & Ng. 2004) does not assume that the policy is optimal. We chose the VAE over Generative Adversarial Networks (Ho & Ermon, 2016) due to easiness of training (Arjovsky Bottou, 2017).
The conditional variational autoencoder is a probabilistic model that samples an action b ~ VAEg(s, a) according the generative model
(9) where Es, is the neural network encoder, Dø, is the neural network decoder, and sample the code z using the re- paraametterrztioon trick (Kingma & Welling. 2014). To learn the & = (Φr, Φ2) parameters place prior (0(0, 7) the code and minimize the variational lower-bound
Σ = Esr (s, ()) z >~ Nu, E); b- Dos
(VAE(O) = (x)))()))
(a - (s, 2))2
2)) + (L(N(H, I).N(O,
(10)
reconstraction loss
regularization
Upon a new state the generative model of the VAB is N(0, (8,
3.4 FINAL ALGORITHM
Algorithm 1:
Offline Risk-Averse Actor Critic (O-RAAC).
input Data set, Critic Zw and critic-target Zw', VAE, { D&g) Perturbation model and target Es, modulation parameter X, Distortion operator or distortion sampling distribution IPD, critic-loss parameters N, N', K, mini-batch size B, learning rate n, soft update parameter J..
for t = 1,... do
Sample B transitions (s, a, r, 8') from data set.
Sample N quantiles 7 and N' target quantiles 7' from (1 1) and compute 8r,- in (2).
Compute policy н + XEo(( (s, b), s.t. b b VA a) in (9).
Compute critic loss Certtie (w) in (4); actor loss Lactor (0) in (5); VAE loss (vve(d) in (10).
Gradient step w + 17 Ceritie (wr): + V (0); Φ + Ø- (LVLv
Perform soft-update on w' + paw + - p)w;: 0' + pM + (1 - p)0'. end for
We now combine the critic in Section 1. the actor in Section 3.2 and the VAE in Section 3.3 and show the pseudo-code of O-RAAC in Algorithm 1. We replace the expectations the critic loss (4), actor loss (5) and VAE loss (10) with empirical averages of samples from the data set. As an ablation, we also propose the RAAC algorithm, in which the actor is parameterized with neural network and there is no imitation learning component.
The critic's goal is learn the reward distribution, the VAE goal to learn baseline action for the actor and the goal of the perturbation model is to be risk-averse. Although Santara et al. (2018) and Lacotte et al. (2019) propose risk-averse imitation learning algorithms, these interact with the environment in an on-policy way. Furthermore, the goal of the imitation learning component in O-RAAC is not to be risk-averse, but to provide a baseline to the risk-averse perturbation.
O-RAAC requires distortion metric D an input. For different D, it generalizes existing distribu- tional RL algorithms and extends them to the offline setting. For example, when D is the expectation operator, then the agent is risk neutral and O-RAAC is an offline version of D4PG (Barth-Mann et al., 2018). Likewise, when D is the Makkafflla--tuunaatii operator (Rockafellar Uryasev, 2002) recover the algorithm by Singh et al. (2020) for optimizing the CVaR.
5

---

Published as a conference paper at ICLR 2021
Results of RAAC, WCPG, and D4PG in the car example. RAAC learns policy that sat- urates the velocity before the risky region. WCPG and D4PG learn to accelerate as fast as possible, reaching the goal first with highest average returns but suffer from events with large penalty. We report mean (standard deviation) of each quantity.
Table 1:
Algorithm
CVaRo.1
Mean
Risky Steps
Total Steps
RAAC
48.0 (8.3)
48.0 (8.3)
(0)
33 (1)
WCPG
15.8 (3.3)
79.8 (1.3)
13 (0)
24
D4PG
15.6 (4.4)
79.8 (2.0)
13 13)
24 (0)
4 EXPERIMENTAL EVALUATION
In this section, we test the performance of O-RAAC using D = aV risk distortion. We use Acerbi's formula (7) to compute the risk distortion. In particular, we ask:
1. How does RAAC perform as risk-averse agent in the off-policy setting? (Section 4.1)
2. How does O-RAAC perform risk-averse agent in the offline setting? (Section 4.2)
3. How does O-RAAC perform risk-neutral agent in the offline setting? (Section 4.3)
For further details such as hyperparameter selection
selectton and extended results please refer to Appendix A.
4.1 OFF-POLICY SETTING: RISK-AVERS PERFORMANCE
In this experiment, we intend to demonstrate the effectiveness of our algorithm as a risk-averse learner without introducing the extra layer of complexity of the offline setting.
4.1.1
EXPERIMENTAL SETUP
As toy example, we chose I-D car with state (z, v), for position and velocity. The agent controls the car with an acceleration a E [-1,
1]. The car dynamics with a time step At = 0.1 is
Tt+1 It + By At + 0.5);
2t+1 Uz + a,tt.
The control objective to move the car to 2g = 2.5 as fast as possible, starting from rest. To model the risk of crashing or of getting speed fine, introduce penalization when the car exceeds speed limit (u > 1). Hence, we use a random reward function given by
Re a) = 10 3701,, - 251,,>1 Bo.2:
where I is an indicator function and B..2 is Bernoulli Random Variable with probability p 0.2. The episode terminates after 400 steps or when the agent reaches the goal.
Benchmarks We compare RAAC with a risk-neutral algorithm (D4PG algorithm by Barth-Maron et al. (2018) with IQN and 1-step returns) and WCPG by Tang et al. (2020), competing risk-averse algorithm. Both RAAC and WCPG optimize the 0. 1CVR of the returns.
4.1.2 RESULT DISCUSSION
In Table 1, we show the results of the experiment. As expected, D4PG ignores the low probability penalties and learns to accelerate the car with maximum power. Thus it has the largest expected return but the lowest CVaR. WCPG also fails to maximize the CVaR as assumes that the return distribution is Gaussian. In turn, it under-estimates the variance of the return distribution of the maximum acceleration policy. Consequently, it over-estimates the CVaR of the returns and prefers the latter policy over the maximum- CVaR policy. In contrast, RAAC learns the full value distribution Z and computes the CVaR reliably. Thus, it learns to saturate the velocity below the e = threshold and finds the highest CVaR policy. See Appendix A.1 for more experiments in this setting.
Qualitative Evaluation In Figure in Appendix A.1 we show the different trajectories as an illustration of the resulting risk-averse behavior.

---

Published as a conference paper at ICLR 2021
4.2 OFFLINE SETTING: RISK-AVERS PERFORMANCE
In this experiment, we intend to demonstrate the effectiveness of our algorithm as risk-averse learner in the offline setting and in high-dimensional environments.
4.2.1 EXPERIMENTAL SETUP
We test the algorithm on variety of continuous control benchmark tasks the data provided in the D4RL dataset (Fu et al., 2020). We use three MuJoCo tasks: HalfCheetah, Walker and Hopper (Todorov al.. 2012). particular, we chose the medium (M) and expert (E) variants of these datasets. Since the tasks are deterministic, we incorporate stochasticity into the original rewards to have a meaningful assessment of risk and showcase a practical example of when the risk-averse optimization makes sense. As risk-averse performance metric, we chose the 0.1-CVaR of the episode returns. We use the following reward functions:
Half-Cheetah: a)) fr(s.u) u) 70I.,>-- Bo.1, where (r(s is the original environment reward, v the forward velocity, and is a threshold velocity (ö = 4 for the (M) variant and ह = 10 for the (E) variant). As with the car example, this high-velocity penalization models penalty the rare but catastrophic event of the robot breaking we want to be avsk-averse to it. We evaluate the Half-Cheetah for 200 time steps.
Rs a) = Fr(ss, pl Bu.1. where Fe a) is the original environment reward, 0 is the pitch angle, is threshold angle (0 0.5 for the Walker2d-M/F and 0 = 0.1 for the Hopper-M/E) and p = 30 for the Walker2d-M/E and p = 50 for the Hopper-M/E. When |0] > 20 the robot falls, the episode terminates, and stop collecting such rewards. To avoid such situation, we shape the rewards with the stochastic event at > 8. The maximum duration of the Walker2D and the Hopper is 500 time steps.
Walker2D/Hopper:
Benchmarks We optimize O-RAAC for the 0.1-CVaR. To demonstrate its effectiveness opti- mizing other risk distortions, we optimize O-RAAC using the 0.25-CVaR and the CPW distortion proposed by Tversky & Kahneman (1992); Gonzalez. & Wu (1999). As a competing risk-averse algorithm, we augment WCPG with a VAE yield the O-WCPG algorithm. As state-of-the-an risk-neutral agents, we use BEAR (Kumar et al., 2019) and O-D4PG, which is equivalent to BCQ (Fujimoto al., 2019) with a distributional critic. As ablations, we compare RAAC (i.e. O-RAAC without the VAE) and the performance of the VAE as pure imitation learning algorithm. Finally, we estimate the returns of the behavior policy by evaluating the stochastic reward functions on the data set and estimate its variance by bootstrapping batches of transitions.
4.2.2 RESULT DISCUSSION
In Table 2, we show the results of the experiments. all environments, O-RAAC has a higher 0.1-CVaR than the benchmarks. In environments that terminate, it also has longer duration than competitors. When optimizing for other risk-averse criteria, O-RAAC still has good 0. 0.1-RVaR per- formance. The behavior policy has poor risk-averse performance and the VAE ablation imitates such policy, hence performs poorly too. On the other side of the spectrum, RAAC performs poorly in all categories. This indicates that the offline version enhancement of RAAC is crucial for offline prob- lems. O-WCPG performs poorly in terms of the CVaR. This might be due to the return distribution not being Gaussian. Finally, BEAR and O-D4PG usually have good risk-neutral performance, par- ticularly in the medium version of these datasets, but poor risk-averse performance. This is expected as these algorithms are designed to optimize the resk-llutral performance.
Qualitative Evaluation We visualize the risk-averse performance by looking at the support of the risk-event in Figure Most of the support of the data set induced by the behavior policy lies in the risky region. O-RAAC learns to shift the distribution towards the risk-free region (green shaded area). On the other hand, O-D4PG (risk- neutral) struggles to shift the distribution as it ignores the rare penalties in the risky region. Only in the Half Cheetah-E experiment O-D4PG manages to shift the distribution towards the safe region. This is because most of the behavior policy is on the risky region and the average performance of the behavior policy under the new stochastic rewards is low.
7

---

Published as a conference paper at ICLR 2021
Velocity (Half Cheetah-M)
Angle (Walker2D-MD
Angle (Hopper-M)
O-RAAC
O-D4PG
Frequency
Behavior Safe Region
10
0.2
0.0
0.2
Velocity (Half Cheetah-F)
Angle (Walker2D-E)
Angle (Hopper-E)
Frequency
5
10
15
1
-0.2
0.0
0.2
Figure 2: Empirical distribution of the risk-event of O-RAAC, O-D4PG, and the behavior policies. O-RAAC shifts the support towards the risk-free region (green area). On the other hand, O-D4PG ignores the risk-related rewards and imitates the behavior distribution.
4.3 OFFLINE SETTING: RISK-NEEL LESKNNETRAL PERFORMANCE
It is well-known that coherent risk-related criteria have a dual distributional robust criterion formu- lation (Shapiro et al., 2014; Iyengar, 2005; Osogami, 2012). In particular, the following holds:
max D 1 (z. a)] = max min Eg (Z* (I,
(11)
dED
where D is dual set of distribution that is induced by the distortion measure D and the distribution d,- When the distortion set the expectation, the dual set collapses to a singleton D= = {d ). For the CVaR, Chow & Ghavamzadeh (2014) expresses the dual for MDPs in Proposition 1. Given this dual result, it is straightforward to show that Exz- (x, a)] 2 D (zr a)] by definition of the minimum. this sense, optimizing DIZa( (x, a)] is equivalent to optimizing a pessimistic estimate of the risk-neutral performance, in similarly way to Buckman et al. (2020) and Kumar et al. (2020).
4.3.1 EXPERIMENTAL SETUP
Despite the goal being to maximize a risk-neutrod objective, we evaluate whether is beneficial to optimize risk-averse criterion in the offline setting. We test this hypothesis using the same setup as in Section 4.2. Namely, we train O-RAAC using risk-averse metric, but we evaluate on the average value, risk-neutral metric.
Benchmarks
We use the same benchmarks as in Section 4.2.
4.3.2 RESULT DISCUSSION
We show the risk-neutral performance in the "Mean" columns Table In all data sets, O-RAAC performs better or similar to the benchmarks. A reason for this is that the episodes of risk-averse agents last longer and thus collect rewards for more time steps. However, this not the only reason. For example, BEAR in the Walker-expert environment has longer episodes and still lower mean returns than O-RAAC In the Hopper-expert, RAAC and VAE have similar duration to O-RAAC. Yet, O-RAAC has higher average return than each of the ablations. This indicates that optimizing a risk-averse performance is beneficial when comparing the rist-alutral performance, specially in data sets where the data is not very diverse (e.g., in the "expert" data sets).
8

---

Published as a conference paper at ICLR 2021
Qualitative Evaluation In Figure 2, we that in most cases there is distribution shift between the behaviour distribution and both O-RAAC and O-D4PG. As the shift increases, we see the benefits of learning using distributionally robust objectives. As particular example we take the Hopper-E distribution. The behavior policy is safe as it does not terminate before the 500 time steps but much of the data is on the risky region. O-RAAC learns center the distribution in the safe region and yet hops forward efficiently. On the other hand, O-D4PG also learns to shift away of the risky region. However, is not risk averse and it overshoots towards the other end of the risky region, where there is not sufficient data to have good critic estimates.
Table 2: Performance metrics on offline MuJoCo data sets with medium (first column block) and expert (second column block) datasets. We compare the 0.1-CVaR and mean of the episode returns, and the average episode duration. We report the mean (standard deviation) of metric. In all environ- ments, O-RAAC has higher CVaR than benchmarks. In environments that terminate, O-RAAC has longer duration too. Finally, O-RAAC has comparable risk-neutral performance to benchmarks.
Medium
Expert
Algorithm
CVaR
Mean
Duration
CVaRo.1
Mean
Duration
O-RAACo.1
214 (36)
331 (30)
200 (0)
595 (191)
1180 (78)
200 (0)
O-RAAC 0.25
252 (14)
317 (5)
200 (0)
695 (34)
1185 (7)
200 (0)
Half- Half-Cheetah Cheetah
O-RAACcPM
253 (9)
318
200 (0)
358 (67)
974 (21)
200 (0)
O-WCPG
(14)
316 (23)
200 (0)
248 (232)
905 (107)
200 (0)
O-D4PG
66 (34)
341 (20)
200 (0)
556 (263)
1010 (153)
200 (0)
BEAR
130)
312
200 (0)
44 (20)
557 (15)
200 (0)
RAAC
(1)
-52
200 (0)
(13)
30 (3)
200 (0)
VAE
(23)
354 (9)
200 (0)
260 (84)
754 (18)
200 (0)
Behavior
(6)
344 (2)
200 (0)
100 (8)
727 (4)
200 (0)
O-RAACo.1
751 (154)
1282 (20)
397 (18)
1172 (71)
2006 (56)
432 (11)
O-RAACo. 0.
497 (71)
1257 (27)
479 (6)
670 (133)
1758 (48)
436 (7)
O-RAACePw
500 (71)
1304 (16)
477 (3)
819 (89)
1874 (34)
454 (8)
Walker-2D
O-WCPG
(41)
283 (37)
185 (12)
362 (33)
1372 (160)
301 (31)
O-D4PG
31 (29)
308 (20)
249 (9)
(55)
1870 (63)
405 (12)
BEAR
517 (66)
1318 (31)
468 (8)
1017 (49)
1783 (32)
463
RAAC
(2)
(9)
200 (7)
(2)
83 (6)
196 (6)
VAE
-84 (21)
425 (37)
246 (9)
345 (302)
1217 (180)
350 (130)
Behavior
-66
727 (16)
500 (0)
1028 (34)
1894 (7)
500 (0)
O-RAACo.1
1416 (28)
1482 (4)
499 (1)
980 (28)
1385 (33)
494 (6)
O-RAACo
1108 (14)
1337 (21)
419 (6)
730 (129)
1304 (21)
434 (6)
O-RAACcPM
969 (9)
1188 (6)
373 (2)
488 (1)
496 (0)
160 (0)
Hopper
O-WCPG
-87 (25)
69 (8)
100 (0)
720 (34)
898 (12)
301 (1)
O-D4PG
1008 (28)
1098 (11)
359 (3)
(31)
783 (18)
268 (3)
BEAR
(47)
1575 (8)
481 (2)
852 (30)
1180 (12)
431 (4)
RAAC
(23)
113 (5)
146 (4)
474 (0)
475 (0)
500 (0)
VAE
727 (39)
1081 (17)
462 (4)
774 (36)
1116 (13)
498 (1)
Behavior
674 (5)
1068 (4)
500 (0)
827 (12)
1211 (3)
500 (0)
5 CONCLUSION
In high-stakes applications, decision-making is usually risk-averse and no interactions with the en- vironment allowed. For this practical setting, introduce O-RAAC, the first fully offline risk- averse algorithm. O-RAAC is compatible with many common risk-averse criteria such as coherent- risk measures or cumulative prospect theory. Due the distributional- sensitive criteria, it also optimizes risk-neutral criteria under natural distribution shift that occur in the offline setting. Empirically, O-RAAC outperforms other algorithms in terms of risk-averse performance and is competitive with risk-neutral algorithms in terms of risk-neutral performance. Particularly, in cases where there is not much data diversity, such as expert data sets, optimizing risk-averse metrics is beneficial due inherent robustness properties.
properties of risk-

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation program grant agreement No 815943.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Carlo Acerbi and Dirk Tasche. On the coherence of expected shortfall. Journal of Banking and Finance, 2002. ISSN 03784266. doi: 10. 10.1016//50788422600220002
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 1XXi:17001.044622 2017.
Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent risk measures.
Mathematical Finance, 93):203-2228 1999.
Michael Bain and Claude Sammut. framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Manfred Otto Heess, and Timothy P Lillicrap. Distributed Distributional Deterministic Policy Gradients. ArXiv, abs/ abs/1804.0, 2018.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement learning. In International Conference on Machine Learning, pp. 419-4449444 2017.
Jacob Buckman, Carles Gelada, and Mare G. Bellemare. The importance of pessimism in fixed- dataset policy optimization, 2020.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization MDPs. Ad- vances in Neural Information Processing Systems, 4(Jamary):3009933177, 2014. ISSN 10495258.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: A CVaR optimization approach. Advances in Neural Information Processing Systems, 2015-Janua:
2015. ISSN 10495258.
Sebastian Curi, Kfir Y Levy, Stefanie Jegelka, and Andreas Krause. Adaptive sampling for stochastic risk-averse learning. Advances in Neural Information Processing Systems, 33, 2020.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In 35th International Conference on Machine Learning, ICML 2018, 2018. ISBN 77867663.
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint 10X.::1994.19000 2019.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, (Apr))553-5566 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for
Deep Data-Driven Reinforcement Learning, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without
exploration. In 36th International Conference on Machine Learning. ICML 2019, 2019. ISBN
Richard Gonzalez and George Wu. On the shape of the probability weighting function. Cognitive psychology, 38( 1811):129-166, 1999.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
10

---

Published as a conference paper at ICLR 2021
Ronald A Howard and James E Matheson. Risk-Sensitive Markov Decision Processes. Management
Science, 18(7):356 369, 1972. ISSN 00251909, 15265501. URL http: /wwww jator .rrg/ Stable/2629352
Peter J. Huber. Robust Estimation of Location Parameter. The Annals of Mathematical Statistics, 1964. ISSN 0003-4851. doi: 10. 10.121//a 177703777
Garud N lyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):
257-280, 2005.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.004:. 2019.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014 Conference Track Proceedings, 2014.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. CoRR, abs/ (bs/0999000 2019. URL http: org/ abs/ abs/1906 00949.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020.
Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2154-2163. PMLR, 2019.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distribution- ally robust optimization. Advances in Neural Information Processing Systems, 33, 2020.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2-3):267-290, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature. 518(7540):529-533 2015.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti- mation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In ICML 2010 Proceedings, 27th International Conference on Machine Learning, 2010. ISBN 9781699589777.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9MMay):855888 2008
Hongseok Namkoong and John C Duchi. Variance-based regularization with convex objectives. In
Advances in neural information processing systems, pp. 297 1-2980, 2017.
Takayuki Osogami. Robustness and risk-sensitivity in markov decision processes. In Advances in Newral Information Processing Systems, pp. 233-241, 2012.
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pp. -522-8528. IEEE, 2019.
11

---

Published as a conference paper at ICLR 2021
Marek Petrik and Dharmashankar Subramanian. An approximate solution method for large risk- averse markov decision processes. In Uncertainty in Artificial Intelligence Proceedings of the 28th Conference, UAI 2012, 2012. ISBN 9780974903989,
LA Prashanth, Cheng Jie, Michael Fu, Steve Mareus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference Machine Learning, pp. 1406-1415, 2016.
John W Pratt. Risk aversion in the small and in the large. In Uncertainty in Economics, pp. 59-79. Elsevier, 1978.
Matihew Rabin. Risk aversion and expected-utility theory: A calibration theorem. Handbook of the fundamentals of financial decision making: Part I. pp. 241-252. World Scientific, 2013.
Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of banking & finance, 26(7): 144)::1433-171,, 2002.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
Andrzej Ruszezyíski. Risk-averse dynamic programming for markov decision processes. Mathe- matical programming, 125(2):235-261, 2010.
Anirban Santara, Abhishek Naik, Balaraman Ravindran, Dipankar Das, Dheevatsa Mudigere, Sasikanth Avancha, and Bharat Kaul. Rail: Risk-averse imitation learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2062-2063, 2018.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyúski. Lectures on stochastic program- ming: modeling and theory. SIAM, 2014.
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki. Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, 2020.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distribu- tional reinforcement learning. arXiv preprint htll:20005005855. 2020.
Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, 2012. ISBN 1781450312851.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. Proceedings of the National Conference on Artificial Intelligence, 4:2993-29999 2999, 2015.
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst Cases Policy Gradients. In Leslie Pack Kaelbling, Danica Kragie, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1078-1093. PMLR, 2020. URL htwp::ttt///. pttp//eeccceeeeeeeeeeeee/oc ml r press v100 ang aang2a..iaa html.
Empel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE International Conference Intelligent Robots and Systems, 2012. ISBN 9781467317375. doi: 10.1109/RRS.2000 201 2.136.666.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992.
Roman Vershynin. diigh-dimessooal probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Shaun Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin: The Journal of the IAA, 26(1):71-92 1996.
12

---

Published as a conference paper at ICLR 2021
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2020. URL http:///pppppeeeeee..e/ee/eeaaaa
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint 2rXiv22005.13239 2020.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pp. Chicago, IL, USA, 2008.
13

---

Published as a conference paper at ICLR 2021
A EXTENDED EXPERIMENTAL RESULTS
A.1
CAR
We ran the Car environment for RAAC, D4PG and WCPG using independent random seeds. We evaluate final policies for 1000 interactions and report the averaged results with corresponding standard deviation in Table In Figure 3, we show the trajectories when following aforementioned policies.
Reward function design
We use the reward function given by
(s a) 10 370 25I,,>>1 Bo.2:
where I is an indicator function and Bo.2 is Bernoulli Random Variable with probability p 0.2. That is, Ty = 370 is a sparse reward that the agent gets at the goal and Ta -10 is a negative reward that penalizes delays reaching the goal. Finally, the agent receives negative reward of sum of
Tv = 25 with probability 0.2 when exceeds the > threshold. As the returns bernoulli R.V. know that it will be Binomial distribution. For this particular case, expect that if the number of steps is large enough, the Gaussianity assumption that WCPG does is good Binomial distributions are asymptotically Gaussian (Vershynin, 2018). However, the episode terminates after at most thirteen risky steps and the approximation is not good.
We show in Figure 3 the trajectories for RAAC, D4PG and WCPG.
Position
RAAC
WCPG
D4PG
Goal
0
2
Velocity
1
Acceleration
10
25
Timesteps
Figure 3: Evolution of states and input control when following learned policies for RAAC, WCPG and D4PG. We use policies from 5 independent seeds for each algorithm. RAAC learns to saturate the velocity below the speed limit.
14

---

Published as a conference paper at ICLR 2021
A.2 MuJoCo ENVIRONMENTS
We ran 5 independent random seeds and evaluate for 20 episodes the policy every 100 gradient steps for HalfCheetah and 500 gradient steps for Hopper and Walker2d. We plot the learning curves of the medium variants in Figure 4 and expert variant in Figure 5. To report the tests in Table 2, we early- stop the policy that outputs the best CVaR and evaluate on 100 episodes with different random seeds.
Behavior policies For sake of reference, we evaluate the stochastic reward function on the state- action pairs in the behavior data set. Unfortunately, the data sets do not distinguish between episodes. Hence, to estimate the returns, we use the state-action distribution in the data set and split into chunks of 200 time steps for the Half-Cheetah and 500 time steps for the Walker2D and the Hopper. We then compute the return of every chunk by sampling a realization from its stochastic reward function. Finally, random with replacement and estimate the mean and CVaRo.1 of the returns in each batch. We report the average of the bootstrap splits together with the standard deviation in Table 2.
bootstrap the resulting chunks into 10 datasets by sampling uniformly
Half Cheetah-M
Walker2D-M
Hopper-M
400
1500
1500
300
Mean
1000
1000
200
O-RAAC
100
O-DIPG
500
O-WCPG
0
1500
750
CVaR
200
500
1000
100
250
500
0
0
200
Risky Steps
100
75
150
50
50
100
25
500
1000
1000
1000
Evaluation Step
Figure 4: Experimental results across several Mujoco tasks for the Medium variant of each dataset.
A.3 ADDITIONAL EXPERIMENTAL DETAILS
A.3.1 ARCHITECTURES
We neural networks function approximators for all the elements in the architecture.
Critic architecture: For the critic architecture, we build on the IQN network Dabney et al. (2018) but extend it to the continuous action setting by adding additional action input the critic network, resulting in the function:
Z(s, = f(mear (m (msoa(ssss (1bs(s), v (a)l), 24 (T)]
(12)
where V. X + Rd Uh A + Rdd Maa : Rddd Ra meat R
R
IR
R
and
f: R
R
15

---

Published as a conference paper at ICLR 2021
Half Cheetah-B
Walker2D-B
Hopper-E
2000
1000
Mean
1000
1000
500
1000
750
CVaR
1000
500
500
250
0
100
100
O-RAAC
Risky Steps
O-D4PG
OO-WCPG
50
50
0
5000
1000
1000
Evaluation Step
Figure 5: Experimental results across several Mujoco tasks for the Expert variant of each dataset.
For the embedding 1,, we use a linear function of n cosine basis functions of the form cos(TiT) i 1.., n, with n 16 as proposed in Dabney et (2018). For v.s, t/a we use a multi-layer perceptron (MLP) with single hidden layer with d = 64 units for the Car experiment and with 256 units for MuJoCo experiments. For the merging function msa, which takes as an input the concatenation of (s)(s) and (a), we use single hidden layer with = 16 units. For the merging function m we force interaction between its two inputs via a multiplicative function Maar (M1, u2) Uj O U2, where 41 moo (Vss (s), (a)) and u2 (r) and @ denotes the element-wise product two vectors. For f we use a MLP with single hidden layer with 32 units We used ReLU non-linearities for all the layers.
Actor architecture:
The architecture of the actor model is
<alss) b XEo(s,b)
(13)
where & A RIAA and b is the output of the imitation learning component. For the RAAC algorithm we remove and set 1.
For the Car experiments, iments, based on Fujimoto et al. (2019), we used a MLP embedding with 3 hidden layers of sizes 400, 300 and 300. We used ReLU non-linearities for all the hidden layers and we saturate the output with a Tanh non-linearity.
used MLP with 2 hidden layers of size 64. For the MuJoCo exper-
VAE architecture: The architecture of the conditional VAE is also based on Fujimoto et al. (2019). It is defined by two networks, an encoder Es, (s, a) and decoder Do, 2), Each network has two hidden layers of size 750 and uses ReLU non-linearities.
A.3.2 HYPERPARAMETERS
All the network parameters updated using Adam (Kingma & Ba, 2015) with learning rates 7) 0.001 for the critic and the VAE, and 7 = 0.0001 for the actor model, as in Fujimoto et al. (2019). The target networks for the critic and the perturbation models updated softly with μ 0.005.
16

---

Published as a conference paper at ICLR 2021
For the critic loss (4) N N' 32 quantile samples, whereas approximate the CVaR to compute the actor loss (5) (7) we use samples from the uniform distribution between [0, 0.1].
In Figure 6, show ablation the effect of the hyper- ryper-parameter lambda. As correct selection of lambda of crucial performance as it trades-off pure imitation learning with pure reinforcement learning. As - the policy imitates the behavior policy has poor risk-averse performance. As À - 1, the policy suffers from the bootstrapping error and the performance is also low. We find values of À E [0.05, 0.5] dependent. This observation coincides with those in Fujimoto et al. (2019, Appendix D.I).
see,
the best, although the specific À is environment
For all MuJoCo experiments, the À parameter which modulates the action perturbation level was experimentally set to 0.25, except for the HalfChectah-meddin experiment for which it was set to 0.5. As we can see from Figure this not the best value of X, but rather value that performs well across most environments.
Half Cheetah-M
Walker2D-M
Hopper-M
1500
250
800
1250
150
1000
CVai
400
750
50
500
0-
250
0.25
0.50
0.75
1.00
0.00
0.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Half Cheetah-E
Walker2D-E
Hopper-F
1200
1250
1000
CVaRt
400
600
400
250
200
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Lambdaa
Figure 6: Effect of the hyperparameter on the CVaR of the returns for each of the MuloCo envi- ronments. As À + 0, the policy imitates the behavior policy has poor risk-averse performance. As 1+1. 1, the policy suffers from the bootstrapping error and the performance is also low. The best environment dependent.
17


Reviews:
[{"review_id": "ypnmcKP69aq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Brief Summary\nThe authors developed an risk-averse RL algorithm with operates in offline contexts. The main idea consists in putting together the benefits from 3 previous works in the distributional, risk-averse and offline RL fields:\n- the DDPG distributional extension (Barth-Maron et al., 2018), called D4PG\n- the Implicit Quantile Network (Dabney et al., 2018), called IQN\n- actor using an imitation learning component with perturbation to control the bootstrappinge error (Fujimoto et al. 2019)\nThe online version of the resulting algorithm, called RAAC, is then tested on a toy problem. \nLater, the full algorithm (O-RAAC) is tested on 3 MuJoCo tasks, using offline data from the the D4RL dataset (Fu et al., 2020).\nThe performance is analysed on both a risk-averse and risk-neutral point of view.\n\n## Strong Points\n- The state-of-the-art is correctly reported.\n- The paper is clear and easy to follow.\n- The experimental analysis is correct, complete and interesting:\n    + baselines seem to be competitive: they are also augmented with the VAE layer\n    + they  the analysis is sound and provides interesting insights.\n- The proposed approach seems to outperform the baselines on the optimized measure: moreover, as noticed in also in previous work, introducing risk-aversion seems to be useful also to increase the expected return. \n\n## Weak Points\n\n- The algorithm is obtained by composing existing approach: it is not clear to me whether there is a novel methodological contribution or not.\n- The authors claim that their approach is general for any risk-measure, however, the experiments are conducted only for the CVaR case and with a specific percentile.\n- The results in the appendix does not seem to match the ones in the main paper.\n\n## Recommendation\nMy reccomendation is to accept the paper since it provides an interesting approach which puts together, in an original way, techniques from different areas of RL, even if it is not clear whether there is a novel methodological contribution. The shown results seems to be relevant, however, it is still not clear whether they are \n\n## Questions for the authors\n- Performances seem to be worse in the supplementary material, can you explay why?\n- Could you provide experiments with different quantiles for CVaR?\n- Could you provide experiments with different risk-measures?\n- Is it possible to evaluate in a qualitative way the behavior obtained by the agents in the learned task when using the risk-aversion or not?\n\n## Additional Feedback\n- It would be useful to state the dependecies of your approach in a clearer way.", "extra_scores": {}, "extra_text": {}}, {"review_id": "wH1nQMtdrQ3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes O-RAAC, an offline RL algorithm that minimizes the Conditional Value-at-Risk (CVaR) of the learned policy's return given  a dataset by a behavior policy. It learns a distributional critic, a VAE for imitation learning, and an actor that perturbs the VAE output to minimize the risk given by the distortion operator D (Here it uses CVaR).\n\n\nSeveral experiments were performed to show the effectiveness of O-RAAC, both with a simple 1-d driving environment and with a modified version of D4RL dataset.\n\n\n \n##########################################################################\nPros: \n\nThe paper is well written, with comparisons with competing methods throughout. This makes the connection with those methods clear and easy to understand. The discussion around the design choices and the tradeoffs are especially insightful. (e.g. Huber loss over l1 or l2; VAE vs vanilla BC; etc.)\n\nThe contribution to risk averse offline reinforcement learning is novel, with the use of CVaR and the idea of a perturbed version of a imitation learning actor. With the assumption that all the risks is captured within the reward. \n\n\n##########################################################################\n\nSuggestions:\n\nThe risk-neutral experiment in 4.3 was a bit confusing. I was not sure whether you measured the O-RAAC’s risk-neutral performance by setting alpha = 0, or you measured its performance with alpha=0.1, but using a risk-neutral metric. I think you did the latter after a few more reading. More clarification either in Table 2 or in 4.3 would be great. \n\nin 3.3, you said \"...is a perturbation model that is optimized maximizing the actor loss (5),\" This part was confusing as well, since one would normally minimize a loss. In this paper, actor loss = risk aversion, so maximizing risk aversion makes sense, but I wouldn't call it a loss. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h1NTfrC4XAB", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission investigates the risk averse objective in offline RL. Usually, the parametric uncertainty is the main source of worries in offline RL, and dealing with the stochastic uncertainty on top of it, in order to account for risk aversion, is very challenging. The authors expose clearly their method and algorithm, even though we sometimes would have liked a bit more argumentation on why this and why not that. Since no theoretical analysis is provided, the only validation is empirical. It is rather complete regarding both the settings and the domains. The results are quite impressive, in particular in the offline setting. For all these reasons, I recommend to accept the submission.\n\nPlease answer/address these comments in the rebuttal/final version:\n* 1- It is unclear whether d^\\beta is defined as the empirical distribution in the batch, or the true distribution. I believe that it would have been helpful in general to formalize more the distributions definitions.\n* 2- U(.) is not introduced.\n* 3- First sentence of 3.2: It is frequent in offline RL to use stochastic policies in order to leverage the risk taken in the face of parametric uncertainty. It is therefore rather odd to state here that only deterministic policies are considered. Even more when we notice in 3.3 that the actual policy is stochastic, since b is sampled from the stochastic behavioral policy.\n* 4- Please indicate the performance of the behavioural policy (average and cvar) in the experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a4hTtmA_K4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper studies the problem of safe reinforcement learning, where we want to learn risk-averse policies in the offline setting. It proposes “Offline Risk Averse Actor Critic” (ORAAC) which performs competitively as risk-neural agent, and outperforms D4PG based baseline as a risk-averse agent on D4RL benchmark. The algorithm involves modifying the losses to learn risk-averse actor, distributional critic and a VAE-based imitative policy.\n\n\nReasons for the score:\nI vote in favour of accepting the paper. The paper studies safety aspects of learning algorithms which are competitive as a risk-averse agent in offlineRL. I would strongly encourage the authors to include numbers for baselines like CQL/BEAR. Though not risk-averse, these algorithms are conservative by design and are strong baselines in offlineRL.\n\n\nStrengths:\n+ The problem is well motivated and the ideas are presented clearly. The authors perform extensive experiments, ablations and provide empirical evidence where ORAAC outperform baselines like OD4PG on D4RL benchmark tasks.\n+ Though risk-averse by design, the learned policy is competitive as a risk-neutral agent.\n\nWeaknesses:\n- Comparison to SOTA baselines in the risk-neutral scenario are missing. Though CQL/BEAR are not risk-averse my design, these algorithms are competitive in the risk-neutral setting. It would be critical to know how ORAAC performs in comparison to such baselines.\n- Discussion on the choice of hyperparameters and experiment design are under-explored. Elaborating on this in section 4 would significantly improve the readability of the paper.\n\n[1] CQL : https://arxiv.org/abs/2006.04779\n[2] BEAR : https://arxiv.org/abs/1906.00949\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FRIPFWbdRH9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an RL algorithm for learning risk-averse policies from offline data. Empirically, it is shown that it can outperform some existing risk-neutral approaches on a number of challenging robotic control tasks under risk-sensitive performance measures. Although the empirical results are encouraging, the theoretical properties of the proposed algorithm are unclear and therefore it is not clear how easy it can be implemented in other tasks.\n\nOverall, the paper is easy to read and the presentation is clear.\n\nThe authors address a very important issue that is faced by RL practitioners. Learning risk-averse policies in a fully offline setting is inherently ill-posed and the burden is in incorporating sufficiently strong prior and bias in regularizing the learning system. Unfortunately the current paper falls short in this regard, where the only idea here is to use imitation learning in the form of autoencoders. So the main contribution in terms of new idea is rather incremental.\n\nFurthermore, the theoretical properties of the proposed algorithm are largely unclear. For example, with respect to a fixed distribution for the start state $S_1$, the optimal CVaR policy for Eq.(1) can be non-stationary. The authors restrict the policy search to the space of stationary deterministic policies. Now suppose that the neural network is over-parameterized, and assuming arbitrarily large training set, which policy does the algorithm converge to? Does the algorithm converge?\n\nConsidering the 3 gradient updates in Algorithm 1, one wonders how to choose all those parameters to make it work, and whether a single step-size parameter is enough. In particular, $\\lambda$ seems hard to choose. How sensitive are the empirical results with respect to the choice of these parameters? In the truly offline case, one presumably can only perform fine-tuning using a separate validation set, how should this be done here?\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes O-RAAC, an offline RL algorithm that minimizes the Conditional Value-at-Risk (CVaR) of the learned policy's return given a dataset by a behavior policy. The reviews are generally positive with most agreeing that the paper presents interesting empirical results. 

The experiments are limited to simpler domains, and could be extended to include harder tasks from other continuous control domains. Some examples could be domains such as in Robosuite (http://robosuite.ai/) or Robogym (https://github.com/openai/robogym). These environments have higher dimensional systems with more clearer safety settings. 
Agreeably, asking for comparisons with unpublished results may be unfair, however, it would be recommended to authors to include additional comparisons with latest methods in Offline/Batch-RL, including the ones which don't guarantee risk, such as CQL, BRAC, CSC.

Further, The theoretical properties of the proposed algorithm are largely unclear. It would help to analyze the effect of both convergence rates, and fixed points, further what is the effect of addition of risk, does the algorithm converge to a suboptimal solution or get there slower. Finally empirical reporting of cumulative number of failures (discrete count) during training as well as during evaluation would be very useful to practitioners. 

Other relevant and concurrent papers to potentially take note of:
Distributional Reinforcement Learning for Risk-Sensitive Policies (https://openreview.net/forum?id=19drPzGV691)
Conservative Safety Critics for Exploration (https://openreview.net/forum?id=iaO86DUuKi)

I would recommend acceptance of the paper. I would strong encourage release of sufficiently documented and easy to use implementation.  Given the fact that the main argument is empirical utility of the method, it would be limit the impact of this work if readers cannot readily build on O-RAAC. 

Decision: Accept (Poster)