Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory
Authors: Karthik Somayaji NS
Yu Wang
Malachi Schram
Jan Drgona
Mahantesh M Halappanavar
Frank Liu
Peng Li
Venue: Accepted by TMLR ()

Abstract: Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.

Published in Transactions on Machine Learning Research (05/2024)
Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory
Karthik Somayaji NS
karthi@ucsb.eed edu
Deportment of Electrical and Computer Engineering University of California Santa Borbara
Yu Wang
gu9500ucsb. edu
Department of Electricol and Computer Engincering University of Colifornia Santa Borbara
Malachi Schram
schram@jlab. ory
Thomas Jefferson National Accelerator LaboratoryS
Jan Drgona
jun. dan.dryona@ppnnl.grr
Pacific Northwest National Laboratory
Mahantesh Halappanavar
hala@pnnl.goo
Pacific Northwest National Laboratory
Frank Liu
fliu Glododd.co edu
School of Data Science Old Dominion University
Peng Li
@p edu
Deportment of Electrical and Computer Engincering University of Colifornia Santa Barbara
Reviewed on OpenReview: https: // openreviee. net/ forum? 098mb 06uhA
Abstract
Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, formulate the extreme values of the statc-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT), We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVa-based parameterization. Importantly, we thcoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms diverse range of benchmark tasks, each encompassing distinct risk scenarios.

---

Published in Transactions on Machine Learning Research (05/2024)
1 Introduction
In the recent years, there has been wide array of research in leveraging reinforcement learning (RL) as a tool for enabling agents to learn desired behaviors with safety guarantees (Gattami et al. 2021 Eysenbach et al.
2017 Pinto et al. 2017 Smirnova el al., 2019 Xu & Mannor 2010). Risk averse RL involves training an RL agent to optimize a risk measure unlike risk neutral RL where agents are trained to maximize the expected value of future discounted rewards. In risk averse RL, the accurate quantification of risk is particularly crucial in preventing catastrophic failures and finds relevance in safety-critical domains such as accelerator control (Rajput et al. 2022), finance (Daluiso et al. 2023), and robotics (Pan al. 2019). where agents must navigate risky or hazardous states.
In risk averse RL, risk measures play vital role in assessing the uncertainty and the potential negative consequences associated with an agent's decisions. Risk-averse RL algorithms employing varions risk measures often consider the distribution of returns to quantify risk. Among these risk measures, Conditional Value at Risk (CVaR) is widely used, relying on the expected value of extreme quantiles of the return distribution to quantify risk (Tamar et al.) 2a Hiraoka et al.] 2019). In contrast, expectiles, which minimize the expected absolute deviation from the target, offer a robust estimate of central tendency (Marzban et al. 2021 Simpler methods for quantifying risk, such as estimating the variance of the return distribution (Xia
2016
PrashanthL. & Fu, 2018) are also utilized.
These conventional approaches often require modelling the entire distribution of returns (state action value distribution) to accurately calculate the risk measure. In risk-averse RL, the focus is on estimating and quantifying low-probability risks, where the distribution of returns may display extreme skewness or heavy tails. Conventional methods model the distribution of returns typically through sampling, a data-intensive process that raises concerns about the accuracy of the modelled distribution, especially in low probability regions. Furthermore, in scenarios where collecting many samples is restrictive, the tail regions of the distribution of returns may not be accurately represented. Percentile risk measures, such CVaR (Tamar ct al. 2014a Hiraoka et al. 2019), demand preciscly modelled tail regions of the distribution of returns for accurale risk measure estimations. When the extreme tail regions of the distribution are inaccurately modeled, the risk measure becomes unreliable and may compromise the risk aversion of the RL agent. Thus, in risk averse RL, accurately modelling the tail region of the state action value distribution is key to good risk averse behavior. Extremely infrequent risk events, labeled as extreme risks (Troop et al., 2019), , naturally exhibit cww-robabilltty skewed tails. Additionally, in situations where sample collection is expensive and adequate samples from the tail are lacking, traditional risk-averse RL algorithms face challenges such high-variance risk measure estimates which are unreliable for oplinization, as noted in (Pan et al. 2020) Beranger et al., 2021). These challenges may lead risk-averse RL algorithms to overlook extreme risk scenarios potentially resulting in catastrophic consequences in safety-critical applications.
We provide a simple demonstration on how the high variance risk measure estimates, can affect risk aversion. We use the Calf-Cheetah environment (Brockman et al., 2016) for our demonstration. Following the setup of (Urpí et al., 2021), we devise scheme where the agent is penalized rarely for moving over certain velocity threshold. We provide all details on implementation and set up in Section g The rareness of the penalty is controlled using Bernoulli random variable which penalizes the reward with a Bernoulli penalization rate p. We train risk averse RL agents based on (Urpí et al., 2021) and our proposed method, EVAC, using the CVaR risk measure [Tamar et al. 2014a) for different rare risk scenarios p = 5%, 15% and ascertain the percentage of times the Half-Cheetah agent. crosses the threshold velocity in an episode. As p decrcases the penaltics become rarer and the risks become more extreme. We illustrate the percentage of crossing the threshold velocity as function of the Bernoulli penalization rate in Figure Ia
In Figure Ic we present the empirical distribution and the modeled distribution according to Urri et al. 2021 for a specific state-action pair, depicted through their cumulative distribution functions (CDF). It is noted that under extreme rare risk conditions (at p p= p55%,, the tail of the modeled distribution (illustrated in red) does not accurately represent the true empirical distribution (illustrated in blue). Furthermore, Figure 16 includes a depiction of the variance associated with the risk mcasure, Conditional Value at Risk (CVaR), for the distribution modeled by (Urpí et al.) (202]) and our proposed method, EVAC. We observe from Figure Ta and Figure 1h that higher variance of the risk measure estimate leads to poor risk aversion
2

---

Published in Transactions on Machine Learning Research (05/2024)
Percentage Failure
Variance
Urpi
40
40
30
10
0
5%
15%
5%
15%
(a) Risk aversion as function of extreme risk
(b) Variance of the C'VaR as function of ex- treme risk in (Urpí al. 2021 and EVAC
in
Urpi al. 2021 and EVAC
5% Bernoull Penalty
15% Bernoulli Penalty
empirical CDF modelled CDF
Empirical Fraction
0.4
0.2
0.2
0.0
0.0
80
85
90
95
100
105
52.5
55.0
57.5
60.0
62.5
65.0
67.5
70.0
72.5
(c) Comparison between the empirical distribution returns and the modelled distribution extreme risk (5% Bernoulli penalty) and moderate risk (15% Bernoulli penalty) in (Urpí al. 2021)
Figure I: Risk aversion and distribution modelling performed by (Urpí et al. 2021 function of extreme
risk: As the risks become extremely rare, traditional risk averse RL methods face challenges in modelling the tail distribution and also exhibit higher variance in the risk measure estimate. Thus, they may end up overlooking risky states and exhibit poor risk aversion.
generally. Under conditions of extreme rare risks, conventional risk-averse methodologies tend to display an increased variance in the estimate of the risk measure in comparison to EVAC. The high variance in the risk measure estimates may equally overestimate or underestimate the risk and obstruct the policy from effectively converging towards a risk-averse policy. As shown Figure [a traditional risk-averse reinforcement learning (RI) methods tend to overlook extremely rare risky events (with a 5% Bernoulli penalization ratc), loading to inadequate risk aversion. However, as the penalty frequency increases (to 15% Bernoulli penalization rate), the variance of the risk measure drops and the risk-averse RL algorithm shows improved risk aversion capabilities. Consequently, the failure to accurately model the tail in scenarios of extreme and rare risk can result in suboptimal risk aversion.
Thus, there an acute need to improve risk aversion by accurately modelling the prediction extreme values of the return (state-action value) distribution. In this work, we propose to develop methods that help extreme risk mitigation by modelling the state-action value distribution accurately. In particular, we develop EVAC, extreme value theory (EVT) based method to model the quantiles of the tail region of the state
3

---

Published in Transactions on Machine Learning Research (05/2024)
action value distribution. Extreme value theory (EVT) posits that the tail distribution follows certain parametric form, which we propose to fit for the particular case of the state action value distribution. We theoretically and empirically demonstrate the reduction in the variance of the risk measure while employing EVAC which leads to better risk aversion even under extreme rare risks. Thus, our contributions are that:
1. We recognize the challenges faced by conventional distribution- based RL methods (a et al. 2020
Dabney et al. 2017 Urpi et al. 2021) in effectively modelling and mitigating extremely rare risks.
2. We propose a novel approach to model the tail region of the state-action value distribution, inspired by extreme value theory (EVT) by utilizing a General Pareto distribution. Importantly, we demonstrate a reduction in variance in the estimation of the quantiles of the tail distribution.
3. We propose a novel actor-critic distributional RL algorithm called Extreme Valued Actor- Critic (EVAC) and conduct comprchensive empirical evaluations of our proposed method across various extreme risk environments. We compare its risk aversion and performance against, widely used baselines.
2
Related Work
There has been an extensive study in incorporating reinforcement learning in risk avoidance. Chow & Ghavanzadeh] 2014 Tamar et al. 2014b Chow et al. 2015b) have studied quantification of risk in terms of percentile criteria, specifically the CVaR (conditional value at risk). Other risk averse measures also include
Xia 2016 PrashanthL. & Fu, 2018) which incorporate variance as a risk measure, while (Chow et al. 2015a
uses notion of range to discuss risk. All the above metrics do require an understanding of the distribution of the quantity of interest. Most works in the RL context deal with risk as it is connected to the distribution of the state action value distribution (or the Q-function). The work of (Bellemare et al. 2017) offers a
distributional perspective on the value functions. (Dabney et al. 2017) approximates the distribution of the value functions in terms of the quantiles of the distribution. Other works like (Ma et al.) 2020 Urpi et al.,
2021 Tang et al.
2019 use the notion of quantiles of the value function distribution to quantify risk and (Sriniaasan et al.
2020 also discusses methods for fine tuning agents in transfer learning scenarios.
Liu et al. 2022 perform post-hoc risk analysis of players in a sports game while accounting for epistemic uncertainty. They use distributional RL framework as policy evaluation tool to evaluate player's performance. Specifically, they employ offiine dataset for performance prodiction of a player and assess risky behavior using quantile based distributional modeling, Furthermore to account for distribution shift between the offline dataset and the test dataset, the work employs epistemic uncertainty estimation via density cstimation for calibration. (Ma et al., 2021) is a distributional RL framework used for risk aversion in the offline learning paradigm. Compared to the offline risk averse setting of (Urpí et al. 2021 namely (O-RAAC), (Ma et al., 2021) penalize the quantiles of OOD state action pairs pointwise. They demonstrate superior risk aversion in the offline setting in comparison to other offline risk averse RL methods. (Greenberg) ct al. 2022) propose method to balance high return policies against risk averse policies. They introduce soft risk mechanism and cross-entropy method to enhance sample elficiency and maintain risk aversion. (Hau et al.) 2022) consider alternate risk measures such as the entropic risk measure and the entropic value at risk in tabular settings for robust. risk averse applications, where the agent needs to mitigate performance loss in uncertain states. Similar in spirit to (Liu et al. 2022). they combine epistemic uncertainty estimates with these risk measures along with a dynamically changing risk level to achieve risk aversion and robustness to model uncertainty. (Killian et al. 2023) operate in the offline RL setting and investigate estimating the risk associated with policy in terms of the expected worst case outcomes. The goal is to develop distributional value function to provide an early warning of risk over possible future outcomes. They use separate dead end and terminal state reward signals to construct separate MDPs to derive two value distributions. A state action pair is classified a dead end if the CVaR of both the value distributions lies below a given threshold. Luo et al.) 2023) study risk aversion as it corresponds to avoidance of highly noisy states. They lay out foundations on why the variance as a risk measure falls short for risk averse policy estimation. They propose Gini deviation, a new risk measure that overcomes the shortcomings of the variance risk mcasure and also demonstrates good risk aversion.
4

---

Published in Transactions on Machine Learning Research (05/2024)
While numerous studies have explored risk aversion using various risk mcasures and techniques, there has been minimal focus on increasing the accuracy of the modeled return distributions that these risk measures rely on, particularly in scenarios involving extreme and rare risks.
When addressing rare risky events, extreme value theory (EVT) (Haan & Ferreira, 2006 provides a framework to characterizing the asymptotic behavior of the distribution of the sample extrema. EVT finds extensive application in modelling rare events across domains from finance, to operations research and meteorology to
multi-armed bandit problems (Roncalli 2020 Santiñaaue et al. 2023 Can et al. 2023 Troop et al., 2019)
In the reinforcement learning context, (Garg et al. 2023 recently discusses the use of EVT in estimating the maximum of the Q-value in the Bellman update. Our work is different from (Garg et al. 2023) on three main fronts. (i) Firstly, (Garg et al., (023) operates in the non-distributional RL setting and aims to learn the optimal value function in the max-entropy RL, inspired by principles in EVT. In max-entropy RL, the optimal Bellman operator requires estimating the optimal value function, which is intractable in continuous action spaces. (Garg ct al.) 2023) uses EVT inspired Gumbell regression to compute the optimal value function in the non-distributional max-entropy RL setting. However, our work operates in the distributional RL setting and aims to perform risk averse decision making in extremely rare risky scenarios by modelling the extreme quantiles of the state action value distribution. Particularly, our work draws inspiration from EVT to model the entire distribution of the state action value distribution unlike (Garg et al. 2023). (ii) Garg et al. estimate the max entropy optimal value function. However, our work considers modelling the entire state action value distribution precisely by using the principle of asymptotic conditional excess distributions estimate the underlying tail behavior of the state action value distribution. (iii) (Garg et al. 2023) uses Gumbell regression a tool to define and train a soft Q-function by gathering inspiration from the Gumbell distributed Bellman errors. This is accomplished by using the Fisher Tippet Theorem (Theorem 4.1 which provides the limiting behavior of sample maxima. In our work, we estimate the realizations of the critic (state action value) distribution over a threshold by employing another key theorem in extreme value theory namely the Pickands- Balkema-de Haan Theorem (Theorem 4.2p. Particularly, our method uses maximum likelihood estimation of the GPD distribution to estimate the state action value distribution.
2023) uses the fact that the sample maxima can be characterized by the Gumbell distribution to
3 Notations
In this rest of the paper, adopt the notation for the standard Markov Decision Process (MDP) characterized by the tuple (S,A, A, Pn, y), where S is the state space, A is the action space, Pn is the stochastic reward kernel such that PR: S x A + P(R), where R is the reward set. Ps S x A P(S) is the probabilistic next state transition kernel, and y is the discount factor. The policy π of the agent is a mapping x S 5 - P(A). We denote S,, A,, RG as the encountered state, action and reward, respectively at time step t. The future sum of discounted returns a random variable denoted by (s'(s,)) = Ei= 2 where R₂ (S A)) and A, ~ =(S)) with So = s; Ao = a. We denote the distribution corresponding to the random variable J z=. In this work, we primarily operate on stationary environments with dense rewards and continuous action space settings.
4 Background
4.1 Distributional Reinforcement Learning
Distributional reinforcement learning (distributional RL) Bellemare et al. (2017); 1: Dabney al. (2017) entails the modelling of the complete distribution of the state-action value function. In contrast to traditional RL which focuses solely on modelling the expected value of the state-action value function's distribution as a point estimate, distributional RL aims to model the entire distribution of the state action value function.
The state action value distribution Z under policy æ is updated using the distributional Bellman operator
T. 7"d 7", a) r(s, a) + F.g T Po(e.)) 7t (a)
(1)
5

---

Published in Transactions on Machine Learning Research (05/2024)
The Bellman operator T" P(R ) operates on the space of probabilities over the reals R, for each state action pair. Focusing attention on Equ 1 reveals that the update of the I,HS distribution (*(s,a) happens via sampling the mixture distribution Es (s..) by r(s,a) a), which is the sampled scalar reward. Thus, the update of (",s, a) can be viewed as a scale and shift. operation of the mixture distribution (es=P)(ee (s',, a'). The distribution function (*,s, (s'','0'
(s' $'',ad a'), scaling it by and shifting it
characterizes the values that the random variable (s, a) can assume. Thus, knowledge of the distribution function (* a) aids in understanding the extreme values that (s, a) can be assigned. In risk averse RL, state action pairs whose distributions (s,s, a) assume low extremal values denote risky states and need to be avoided. Thereby, distributional RL provides tool to quantify the uncertainty and risk for risk averse applications.
4.2 Quantile Regression
One of the popular methods to model the distribution of the state-action value function in distributional RL is through the quantiles of the distribution. The quantiles of the distribution are often estimated through the quantile regression framework (used in risk averse RL applications including Ma et al. (2020); Dabney ct al. (2017): : Urpí et al. (2021)). Quantile regression proposed by Koenker Bassett Jr (1978) estimates the true quantile value of distribution by minimizing the pinball loss. Assume random variable Y with its unknown true distribution function (.) and probability density function fy (.). The goal lies in estimating
the true quantile values of Fy denoted by 0, for E [o, 11, the quantile level. The quantile predicted by the quantile regression framework is a unique minimizer of the pinball loss [(0,) given by
[(0,) = Ey (y 0,)(T - 1y-n-
In deep reinforeement learning, a modified smooth loss function called the empirical quantile Huber loss Huber 19644) is instead used for better gradient back propagation.
4.3 Extreme Value Theory (EVT)
Modeling the extreme values of distributions under low data availability is challenging. Extreme Value Theory (EVT) is statistical framework that focuses modelling and analyzing the tail behavior of probability distributions, particularly the distribution of extreme values. It provides methods to characterize and predict rare and extreme events, making it valuable in assessing and managing risks associated with tail events. We formally introduce the two main theorems in EVT below.
Theorem 4.1 (Fisher Tippet Theorem Basrak (2011)) Let X1,... Xx, be a sequence of IID random variables, with distribution function (CDF) F. Let Mn represent the sample mazimum. If there erist constants a, > 0 and bn and a non-degenerate distribution function G such that:
lim P{ Mn M = 100
an
then the distribution function G(z) is called the Generulized Extreme Value distribution (GEV) and can be decomposed into either the Gumbell, Frechet or the Weibull distribution.
Intuitively the Fisher Tippet Theorem describes that the normalized sample maxima of distribution F, converges in distribution to the GEV distribution.
The other central theorem in EVT is the Pickands- Balkema -de Haan Theorem (PBD Theorem) which inspects the conditional exceedance probability above a threshold u, of a random variable X with a distribution function F..
Theorem 4.2 (Pickands-Balkema--de Haan Theorem Pickands II1 1975) Let X.... Xn be a
quence of IID random variables with distribution function (CDF) given by F whose limiting behavior approuches the GEV distribution. Let Fu(x) P(X u S x > u) be the conditional excess distribution. Then,
lim Fx(x) Dg (x))
2-000

---

Published in Transactions on Machine Learning Research (05/2024)
where Hg is the Generalized Pareto distribution (GPD) with parameters E,
Intuitively Theorem422 4.2 describes that the conditional distribution Fu of the random variable X approaches the GPD distribution for large enough u. The CDF of the GPD distribution Fg.o(() is given by:
- 1/E
1 1
Em
for g *
(3)
1 exp( ) for 0
5 Motivation
Modelling the quantiles of the state-action value distribution, especially the low-probability extreme realizations (tail quantiles) is crucial in risk averse RL. However, this is challenging when the extreme tail regions of the underlying distribution are skewed or heavy tailed. This challenge arises as the traditional sampling based estimations of those quantiles tend to exhibit high varianco, leading to potential compromises in risk aversion strategies. To address this issue, our approach this study involves leveraging extreme value theory- parameterized asymptotic distributions to effectively model the low-probability tail regions of the state-action value distribution.
To underscore the importance of employing extreme value theory-based parameterized distributions for tail distribution characterization, initially examine the challenges associated with utilizing sampling based methods to estimate the quantiles of the tail distribution through quantile regression.
5.1 Challenges when employing the sampling distribution in quantile regression
Assume a random variable Y with its unknown true distribution function Fy(.) and probability density function (.) Assume N samples sampled from Fy, {81, y2. ..NN. The aim to find the quantile value estimate g. using quantile regression, by minimizing the empirical pinball loss for given quantile level т, which follows from Eqn
1 N Σ (ys 8TT N
((0%)
1u- op)
(4)
Importantly, the asymptotic convergence of the quantile regression estimator to the true quantile value is discussed in Koenker & Bassett Jr (1788
VN 0,, N10,
where the variance the estimator 8N is given by
X2 T(1-7) N N (2(0,)
(5)
The variance of the quantile regression estimate 8N is dependent on the number of samples N from the distribution that are used to estimate the quantiles and also on the squared probability density f (0,) at the quantile value 8,.
Under a fixed number of samples, the variance of the quantile regression estimate increases the case of rare extreme valued state-action value distributions when the density function (O,) assumes low probability values in the tail regions (r + 1 ie. heavy tails). Beranger et al. (2021); b Pan et al. (2020) also discuss the estimation inaccuracy of the quantile estimates under lower data availability. Bai et al. (2021) specifically discusses the inherent undercoverage associated with quantile regression estimator for tail quantile regions. Such evidence coupled with the high variance property acts as a deterrent to exclusively choosing the sampling distribution for estimating extreme quantiles in the case of distributions with low probability extreme events.
Thereby, we investigate the modelling of the state action value distribution when the sampling distribution assumes rare extreme values.

---

Published in Transactions on Machine Learning Research (05/2024)
5.2 EVT based modelling of the state-action value distribution
Given the naturally occurring heavy tails in rare extreme events, direct sampling from the return distribution (state-action value distribution), may yield insufficient information regarding extreme values, Consequently, we propose an approach that involves fitting a paramoterized distribution to the tail regions the statc-action value distribution. This approach draws inspiration from the principles of Extreme Value Theory (EVT), which advocates the fitting asymptotic distributions the extreme tail regions of the state action value distribution.
6 Method
Extremely rare reward penalties like the ones discussed in Section cause the state action value distribution to be characterized by low probability tails. The higher probability occurrences of the true random return Jd (s, a) can be learnt reasonably well, unlike the low probability extreme valued realizations. Thereby, we propose to decompose the state action value distribution into two regions namely, the non-tail region and the tail region.
Remark 6.0.1 Although higher rewards are considered better in reinforcement learning, negate the rewards to maintain consistent notation with literature in EVT. This does not affect any analysis. Because of the negated reward, we swap focus from the left tail of the state action value distribution to the right tail.
In order to capture the extreme regions within the state-action value distribution, we depict the tail distribution and the non-tail distribution, in Figure 2 For a given threshold u, we denote by (7, a), the non-tail distribution. The subscript 'L' is used for denoting support values of Z" lower than u. Similarly, we denote by Z (s, a), the tail distribution. The subscript 'H' denates distribution with support values higher than u, which is assumed to be a sufficiently high threshold for the state action pair (s, a). We assume that the area under ZΣ a) and (5, a) to be 1. So, the state action value distribution Z (s, a) is obtained by rescaling Z7, as)a) and Z7/ (s) a).
(s (7 a) + +1 (1-1) n) (8,
(6)
Traditional tail modeling
as,
($.
ait =1
EVT based tail modeling
(s.a)) n (.. 1). () *n + (1 ,)..
17. 7. (s)
(1 (1-7)_27(s.a)
Figure 2: Modelling the tail and non-tail distributions of the state action value function. The area under the non-tail distribution (s, a) and tail distribution 2 (s, a) is assumed to be 1.
The fundamental idea guiding our methodology for modelling extreme values within the state-action value distribution is conceptualize the tail region of the distribution as the conditional excess distribution beyond certain threshold. To formalize this, we invoke the Pickands Balddsa-Bleeaeeee Haan theorem (Theorem 4.2)
to approximate the tail regions of the state-action value distribution z=. According to Theorem 4.2 the
8

---

Published in Transactions on Machine Learning Research (05/2024)
conditional excess can be expressed through the Generalized Pareto Distribution (GPD) with parameters ç, 0. Consequently, for each state-action pair (8,a) E S x A within the state-action value distribution (s, (s, the tail region can be effectively modeled by an associated GPD distribution with parameters E(s, a a). Therefore, the tail regions of each Z= a) can be viewed as paramcterized GPD distributions in the S A
space, denoted as GPD E(s, a), a(s, a)) with distinct thresholds u for each a). The threshold u for each (s, a) pair is the value of the support distribution (= a), where its CDF has a value of 17. In the limit, the tail distribution (8, a) can be represented by GPD ((s, o(s,a) a)) and appropriately rescaled to obtain Ze (s,a) as outlined in Eqn. G
Fitting the asympotic GPD distribution enables effective extrapolation and the generation of new data points during sampling, which makes the EVT based tail modelling data efficient. The merit of this approximation lies in its ability to capture the extreme value behavior of 2" even with limited data availability, owing the fitting of asymptotic Extreme Value Theory (EVT) distributions. We provide a detailed explanation of the GPD fitting procedure in Section 7 and the algorithmic flow of approach in Algorithm
6.1 Variance reduction in the quantile regression estimate using EVT based tail modelling
Having characterized the tail region of the state-action value distribution using the parameterized Generalized Pareto Distribution (GPD), our objective is to explore the implications replacing the sampling distribution with the proposed parameterized state-action value distribution when performing quantile regression.
The proposed methodology involves updating quantiles of the state-action value distribution through quantile regression. As detailed in Section 5.1 the variance of the quantile regression estimator is influenced by the square of the probability density at a specific quantile level. This prompts crucial inquiry: how does the variance of the quantile regression estimate change when utilizing a GPD approximation for the lail region of the underlying distribution? To address this question, we conduct an analysis of the variance of the quantile regression estimator for a distribution with a paramcterized GPD tail and compare it a distribution without. such tail modeling.
Assume random variable Y with its distribution function Fy. Let the 7th quantile of Fy be denoted as 0,. For any sufficiently large quantile level 1) E [0,1), 1), and smaller increment quantile level t E [0, 1 7) such that, + t E [0, 1), we have the corresponding quantiles of the distribution 6, and Byte. We define the excess value of the quantile as x = dgte - Og. We define the CDF of the GPD distribution by Fit... and its density function by f
(0 = P(V < 0, +
= = + (1 11) P( ((rr-pee - -0, S x/Y > 0n)
+ (1 11) Frt (x)
As mentioned earlier for sufficiently large n, P(Y 00 < x 0g) approaches the GPD distribution (e) in the limit. Thus, we have
P(Y < 0g + x) R 1) + (1 P(X s x)
(8)
where X ~ Hg... It follows from Eqn 8 that,
fr (0,, +2) R (1- n)fH....
(9)
If we represent the quantiles of Fig.. by 04, then we have the following relationship between the quantiles of Fy and the quantiles of the GPD distribution:
8 0xte R On + gH
(10)
-
9

---

Published in Transactions on Machine Learning Research (05/2024)
We are interested in representing the quantiles for sufficiently large and higher. Following Eqn. 5 the variance of the quantile regression estimator in estimating the quantile opte of the distribution function Y is
(1 (1-1))111111 11
d2 N. (.(1-n)3fttt
A -
In Eqn. 10 6, corresponds to the thth quantile of the distribution corresponding to the threshold quantile level 7. Assumiing that the distribution has sufficient probability mass at quantile level r, On may be accurately estimated with the provided samples. If 0, is known, may simply estimate the I- GPD distribution and shift it by Og. Thus the quantile regression estimator's variance in estimating 1 + quantile of Y using the GPD is given by:
th quantile of the
A (1/0-n))1--(CC
11/111/1111 1))
N. N.fHt.. ( {
We can verify that Xς > X for large values of e.g., close to 1.0. Therefore, we show that GPD based modelling the tail region of the distribution helps reduce the variance in the estimation of higher quantiles. We also illustrate this in Figure 3 with a few candidate values for vj = 0.75, 0.8, 0.85.
5
n
n= 0.85
Variance values
4
3
2
0.00
0.05
0.10
0.15
0.20
0.25
Increment quantile
Figure 3: Dashod lines indicate X2. and the solid lines indicate X².- X3 is always upper bounded by the X which illustrates variance reduction while using GPD for approximating the tail.
The preceding analysis establishes that when n large enough (and the tail distribution converges to the GPD distribution), the variance of the quantile regression estimator with the parameterized state-action value distribution is smaller in comparison with the variance of an estimator with non-paraeeterized state-action value distribution.
The reduced variability in the quantile regression estimate is beneficial as it also lowers the variability in the estimate of the risk measure. For instance, the Conditional Value at Risk (CVaR) risk measure, employed in optimizing risk-averse policies (as outlined in Equation 12p. shows reduced variability when there is less variability in the tail quantile estimates. This decrease in risk measure variability can, in turn, expedite the process of developing risk-averse policies with limited number of interactions with the environment.
10

---

Published in Transactions on Machine Learning Research (05/2024)
7 Extreme Valued Actor Critic (EVAC)
7.1 Actor Critic Framework
To scale the GPD modelling to large and continuous state action spaces, we introduce an actor-critic deep reinforcement learning method, named Extreme Valued Actor-Critia (EVAC). This approach utilizes parameterized Extreme Value Theory (EVT) based distributions to represent. the state-action value distribution. Aligned with existing frameworks like Lillicrap al. (2016); Fujimoto et al. (2018); [Urpí et al.) (2021), our architecture comprises an actor and critie network. Similar to Urpí et al. (2021). ) the actor network is employed for optimizing the risk measure and suggesting optimal risk-averse actions. Conversely, the critic network is dedicated to modelling the quantiles of the state-action value distribution. Specifically, the critic network is utilized to capture both the quantiles of the non-tail region and those of the EVT-based parameterized tail distribution. We present the framework for training the critic that models the state-action value distribution.
7.2 Training the state-action value distribution
The state-action value distribution models the distribution of the future returns and is updated using the Bellman update:
T (s, = r(s.a) + > 27's', a') (1- ( (s',a'))
(11)
where (s, a, 8,,a), a') represent the state, action, reward, next-state and next-action tuple. The distribution (s', ('sss,'))==)) is docomposed into the non-tail distribution (s''',, a²) and the tail distribution 2² (" d') as Eqn 6 EVAC, we propose learning the stale action value distribution (=,s, a) and the parameters of the GPD distribution GPD(E(s, a), (s,,
(s's,'')= =n 77 ( + 7)
The state action value distribution (s, a) is modelled using quantiles. We denote the 7thh quantile of 2 (8, as z² (8. Firstly, the non-tail distribution function (s,aa) is defined as:
(7, a)l; = as) E [0.
The GPD distribution with parameters E(s, a), (s, a) is denoted as GPD(E a), as, a)). The tail distribution (s,a) (s) is obtained by shifted version of the GPD distribution and may be represented as
(s, 7" (s. a)l, (P
where (s, a)ln is the scalar shift and represents the threshold quantile of the state action value distribution (s,,aa) a) corresponding to quantile level 77.
Our aim is model the quantiles of (= a) accurately by utilizing samples from the non-tail distribution function Zi('s, a') and the tail distribution (s', (5',,')) a'). Our proposed training procedure to train the state action value distribution (= a) in Equ encompasses the following distinct components:
Sampling from the non-tail distribution (Zi(s', d')) and the tail distribution
Quantile regression to estimate the quantiles of the state action value distribution (s(saa).
Updating the parameters (E(s, a), ()) of the GPD distribution which controls the tail distribution as)
7.2.1 Sampling the non-tail distribution function and the tail distribution
Our goal is to estimate the quantiles the state action value distribution 2 a) of the current state action pair (s, a) in Eqn [1 through quantile regression. In order do so, we need obtain samples from the RHS of Eqn 11 by sampling the tail distribution ( a²) with proportion (1 -1-pa 17) and the non-tail distribution (5', a') with proportion 17.
11

---

Published in Transactions on Machine Learning Research (05/2024)
We obtain samples from the non-tail distribution Z7,(s', .') using inverse transform sampling. For sampled quantile levels 7 (O0,,) v)); the queried samples 7d (s', a')], correspond to sampling from the non-tail distribution (7')''a").
We obtain samples from the tail distribution (s' ,') by querying the GPD model. Specifically ( = GPD(E(s
(s's,, a') d')ln + GPD (E(d', d'), ass' )d)). First, we sample from the GPD distribution GPD (((,, and shift by the scalar threshold quantile value 2s'ss'a')ln
The shifted GPD distribution models the tail region of the state action value distribution, while the samples from (I''', model the non-tail region. Having obtained samples from the RHS of Eqn the state action value function Z" a) can be trained using quantile regression. It is to be noted that although the state action value function is modelled by quantile regression, our sampling procedure involves sampling of (s', d') and the shifted GPD distribution which more accurately models the tail behavior, unlike Ma et al.
(2020) Dabney al. (2017); Urpí et al. (2021).
7.2.2 Quantile regression update quantiles of Z" (s, a)
The samples obtained from the tail distribution, 27 (s',a') (with proportion 1 1-7) and the non-tail distribution Z7, (s', a'), (with proportion n), are used to estimate the quantiles of the state action value distribution Zd (s, Consider N such samples from the tail and non-tail distribution, each represented as y,. Particularly, to estimate the 7h quantile 0, of (= a), the pinball loss [(0T) minimized.
N
((07)
Σ (y) 0x)(T= - 10)-0, =11
N
This procedure accurately models the quantiles of the state action value distribution (s) by obtaining samples from both the non-tail and the tail distributions. Since the pinball loss is non-smooth, we employ a smooth approximation of the pinball loss namely, the Huber loss Huber (1944..
7.2.3 Updating the parameters of the GPD Distribution
The previous procedure uses samples from the tail and non-tail distribution to update quantiles of the state action value distribution. However, the tail distribution (7,s, a) which is modelled by the shifted GPD distribution itself needs to be updated. Thereby, it is imperative to appropriately fit the parameters E(s, als,a), a).aaa (s,, a) of the Generalized Pareto Distribution (GPD) to accurately model Equ. II The GPD is used to model the distribution of excess values above a given threshold (threshold quantile value).
CPD(E(s..).oss..)) 2H (s,
2d (s a)l
where (²(s,a)l, is the threshold quantile. To acquire these excess values, we initially generate samples 7 (s, and subtract them from the threshold (s,a))== to obtain the surplus beyond the threshold. Subsequently, the GPD parameters E(s, a), asssa) a) can be determined through maximum likelihood estimation of the GPD distribution applied to the samples of excess over the threshold.
Thus, we provide a novel framework for using extreme value theory for state action value distribution estimation. In Section A we provide a proof of the convergence of the Bellman operator in Eqn 11 In Scction B we provide details of the MLE estimation procedure to update parameters of the GPD distribution.
7.3 Policy Optimization
Once the critic, (encompassing both the tail and non-tail segments) has been trained under a fixed policy æ to produce the state action value distribution (", a), our focus shifts towards the actor's policy optimization, aimed at achieving a risk-averse behavior. In order to train risk averse policies, extreme values of the state-action value distribution are employed to guarantee optimal worst case performance. We propose to
12

---

Published in Transactions on Machine Learning Research (05/2024)
employ the CVaR risk measure Tamar et al. 7 (8,a) for mitigating extreme risk. The CVaR (Conditional value at risk), is risk measure that denotes the average worst case performance by integrating the quantiles of the state action value distribution between quantile levels 21 and 1.0. The optimal policy I² in Equ is obtained through policy gradients over the CVaR. We choose 21 = 0.95 in all experiments. Note the negation of the reward (Remark 6.0.1) which paves the way for higher values of æ1).
(2014b;; Ying et al. (202)) the state-action value distribution
We expect that for a given amount of collected data, lower variance estimates of the CVaR. for a given (s, pair lead to better risk averse policies. We evaluate the both the variance and consequently the quality of the risk averse policies produced by EVAC and other basclines in the next section.
π² arg min CVaR((x)
(12)
[
Z (ss dr
arg min
32 I1 Tmmi
7.4 Algorithm for EVAC
Algorithm 1: Extreme Valued Actor Critic: EVAC
Input:
Initialize GPD parameters (((s,a),((s.d)) a)] the critic 2(s, a), policy rss) and threshold quantile level n
Policy Iteration:
Sampling from the parameterized distribution and updating the critic (Section 7.2.
7.2.2
for tuple (s, (s,,,,,s',d = r(s')) do
~ CPD(((s),a')
Define 2/ = 2(5',d")rr=n
Define 4 = 2(s' ((s),d')l===oo where To ~ Unif(0,,7)
Sample p ~ Bernoulli,
Define Bellman target: =r 1
1
Update the critic Z using samples Zr through quantile regression
Updating the GPD parameters ((s, (), a(s, (Section 7.2.3) for tuple (s do y ~ Z(s, ((s,a), a(s,a) MM.E/GPD( 7(8,)) r=
Policy Improvement: (Section
Update policy according Equ 12
We provide an algorithmic flow of EVAC in Algorithm ! The EVAC algorithm receives the critic and actor network parameter initializations, the GPD parameter initializations and the threshold quantile level n inputs. Firstly, any tuple (s, s,,,,, = (ss)) equivalently batch of such tuples is considered for updating the state action value distribution. To accomplish this, the excess over the threshold quantile level n. is computed by sampling the GPD distribution i.e. x ~ GPD(E(s', a'), (8',''') a')). Next, two kind of samples are obtained. One from the non-tail region (Zz,) and another from the tail region (Zz)) Then these samples are selected in proportions 7) and (1-1) respectively to update the critic through standard quantile regression performed on the entire batch.
Secondly, tuple of state action pairs (s, a) or equivalently a batch of them is considered to update the GPD parameters. The updated state action value distribution is sampled to obtain the the excess over the threshold quantile for all elements in the batch. The parameters of the GPD distribution are then updated using maximum likelihood estimation on the excess samples.
Finally, the policy updated using CVaR optimization on the updated state action value distribution.
8 Experimental Evaluation
As detailed earlier, extreme rare risk scenarios can have catastrophic outcomes. Although it very pertinent to real world complex control environments to study risk aversion, such complex scenarios often lack
13

---

Published in Transactions on Machine Learning Research (05/2024)
open source simulators. As result, resort to ereating such extreme rare risk scenarios in open source environments.
Thus, in order to evaluate rare risk aversion of each agent, we define special environments following the work of Ma et al. (2021): Urpí et al. (2021) where risk scenarios are simulated. However, to simulate extreme risk scenarios we introduce some modifications.
8.1 Generating extreme risk scenarios
In order to replicate scenarios where the state-action value distribution 2d (s, a) encompasses exceedingly rare extreme values with minimal probabilities, we deliberately design rewards r(s, a) that can take very low values (representing catastrophic events) with extremely low probabilities. This mirrors realistic situations where the reward for given state-action pair is drawn from distribution that seldom yields low values, corresponding to high penalties.
To illustrate a strategy simulating low-probability, high-penalty events, we incorporate penalty term into the reward with a low probability. Essentially, this penalty term is influenced by a Bernoulli random variable parameterized by p. offering control over the rarity of penalizing events. The parameter p determines the infrequency of penalties.
We experiment on two benchmark Open-AI Al environments Brockman et al. 2016 namely Mujoco environments and Safety-gym environments Ji al. 2023
8.2 Mujoco Environments
For creating rare risky events in the Mujoco Environments, we modify the reward using a wrapper function which penalizes the reward for certain state action pairs with certain probability (which typically small to simulate rare risky events), Our reward penalization setup is similar to the set up of Urpi et al. (2021) b. We primarily experiment with three Mujoco environments namely, HalfCheetah, Hopper and Walker2d environments. Denoting R(s, a) as the original non-penalized reward, we define the stochastic penalized reward r(s,a) For the Half-Cheetah environment,
r(s, = R(s, B,
where II is the indicator function, v is the velocity of the HallCheetah agent, L = -50 a penalization weight, a = 2.5 the threshold velocity of the HalfCheetah agent above which the agent gets penalized rarely. The sample of the Bernoulli random variable denoted by B,. To simulate extreme rare penaltics, the Bernoulli distribution parameter p is made equal 0.05 ie. (p = 0.05), which indicates the frequency which the state action pair (s, a) is penalized.
For the Hopper environment,
r(s, a) = a(ss.a) Bp
where Il is the indicator function, |0| denotes the angle Hopper, -50 is a penalization weight, for the reward, a 0.03 is the threshold angle over which the agent gets penalized. The Bernoulli distribution parameter is to p 0.03.
For the Walker2d environment,
r(s a) a(ss,a)+
where ll is the indicator function, 10) denotes the angle of the Walker2d agent, L = -30 is a penalization weight for the reward, a = 0.2 is the threshold angle over which the agent gets penalized. The Bernoulli distribution parameter p = 0.03.
For all Mujoco environments, we analyze the CVaR (in Equ 12 and its variance. Next, in order test for risk aversion of the agent, we define metric namely the percentage failure which is defined as
Percentage Failure FA [T2a]] 100
14

---

Published in Transactions on Machine Learning Research (05/2024)
where Ey represents the empirical mean over N episodes, q is the quantity of interest, i.. the velocity v for the Half-Cheetah and the angle |0] for the Hopper and Walker.
The percentage failure is indicative of the fraction of the times in an episode that the agent enters into state-action pairs that are penalized rarely. The percentage of failure naturally quantifies the risk aversion ability of the agent question. Additionally, we also include the cumulative reward collected each episode to assess the performance of the agent in addition to its risk aversion capability.
8.3 Safety-gym gym Environments
We also perform extensive experiments on another suite of environments, namely the safety-gym benchmark Ji al. (2023). Safety-gu gymetyy-ymmm environments consist of configurable robots with programmable reward functions. We use the 'Point' robot and the 'goal' task. Thus, in our setup a particular robot (the 'point' robot) is tasked with reaching goal location the arena. The robot receives a reward with respect to distance from the goal.
We follow the method of reward penalization used by Ma et al. (2021) to introduce rare risky rewards. We define certain circular regions on the arena, termed 'hazards' which rarely lead to penalizations. To introduce extreme rare risks, the nature of reward penalization
r(s, R(s, alsx Bp
where R(s, a) is the original non-penalized reward, rss, a) is the stochastic penalized reward, q denotes the current location of the robot in the arena, H. includes positions of all the hazards in the arena. The sample of the Bernoulli random variable is denoted by B, and is the Bernoulli frequency of penalization. The penalty weight L = -00 Thus, the agent suffers a rare penalty when in the hazard region. We create two scenarios for extreme risks with respect to the placement of the hazard and the size (radius) of the hazard.
Safety-gym Scenario-A: In this experimental setup, there is single hazard of large radius placed along the straight line path (shortest distance) between the start position and the goal position. The agent needs to learn to avoid the large hazard region. We set the Bernoulli penalization rate p 0.05 for Scenario-A.
Safety-gym Scenario-B: In this more challenging experimental setup, there are multiple hazards of smaller radius placed closer to the goal. The agent needs to discover the optimal path to avoid all hazards and reach the goal. We make the setup even more challenging by setting the penalization parameter to p == 0.03 to introduce more extreme risks.
We provide visual reprosentations of Scenario-A and Scenario-B in Figure and Figure B respectively in Section H of the Appendix.
8.4 Baselines
We compare EVAC over different baseline algorithms such as DDPG-RAAC
Urpi al. (2021) TD3-RACC
(TD3 version of DDPG-RAAC) and DSAC (with single critic) Ma et al. (020 RAAC uses quantile regression to construct 2 (s, a) and uses the CVaR to optimize the policy. DSAC additionally maximizes the entropy of the policy to ensure optimal exploration. The difference between our approach (EVAC) and the compared baselines is that we additionally approximately model the tail of the state-action value distribution using the shifted and scaled GPD distribution.
8.5 Result Discussion
As can be seen in Table D the EVAC agents exhibit good risk averse behavior by avoiding risky reward state actions. The percentage failure for the EVAC agents is noticeably smaller when subject to even extreme risk scenarios.
Secondly, the CVaR of the EVAC agents higher than the baseline methods. Additionally, the standard deviation of the CVaR across different runs for the EVAC agent is smaller in comparison to other baselines.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Environment
Algorithm
Percentage Failure
Cumulative Reward
CVaRR
RAAC-DDPG
16.551 4.43
637.811: 319.78
135.18 + 13.02
RAAC-TD3
41.3# 16.6
836.8+ 195.85
129.81 + 38.75
HalfCheetah
D-SAC
30.04 + 22.26
1356.05 + 269.63
149.76 + 27.38
EVAC
2.87 + 1.3
1502.46 94.25
156.71 = 11.07
RAAC-DDPG
66.94 11.79
410.62 + 315.12
27.58 1: 34.41
RAAC-TD3
48.78 = 12.36
718.53 = 355.55
27.87 + 65.35
Hopper
D-SAC
57.34 + 11.42
782.85 = 400.78
51.5 95.98
EVAC
2.09 + 4.18
875.89 = 299.75
76.61 + 28.15
RAAC-DDPG
68.05 11.62
123.17 66.46
1.31 + 6.6
RAAC-TD3
22.18 + 13.02
308.66 + 134.99
-9.52 + 14.83
Walker2d
D-SAC
24.75 11.33
598.5 = 128.38
3.76 + 8.34
EVAC
4.98 + 7.23
668.18 - 463.68
4.37 + 3.22
Table 1: Performance metrics on Mujoco environments under a penalization rate of p 0.05 for the Half Cheetah environment and p = 0.03 for the Hopper and Walker2d environments. We record CVaR at x 0.95 (0.05 reward not negated) and threshold quantile level 1j = 0.96. Inference done on trained agents. Each trained agent completes an episode in inference mode acting on the learnt policy. The results are expressed mean + standard deviation across 5 trained agents.
Environment
Algorithm
Percentage Failure
Cumulative Reward
CVaR
RAAC-DDPG
14.03 11.4
5.41 1 0.24
1.59 1 0.09
RAAC-TD3
12.13 5.94
5.55+ 0.23
1.39 + 0.13
Scenario-A A
D-SAC
22.31 + 13.67
2.13 + 4.07
1.33 + 0.18
EVAC
0.0 0.0
5.71 + 0.0
1.7 = 0.08
RAAC-DDPG
20.2+ 14.66
5.224 0.41
1.58 + 0.1
RAAC-TD3
22.34 9.93
4.87 0.47
0.64 = 0.41
Scenario-B
D-SAC
15.47 4.84
5.1 = 0.24
1.56 = 0.09
EVAC
2.47 3.9
5.61 + 0.12
2.0 = 0.06
Table 2: Performance metrics on the Safety-gym environment under penalization rate of p 0.05 for Scenario-A and 0.03 for Scenario-B. We record CVaR at ø = 0.95 (0.05 if reward is not negated). We set the threshold quantile 17 = 0.96. The results are expressed as mean = standard deviation across trained agents.
This is concurrence to the analysis Section 6.1 consistently and precisely quantify risk in the tail region of the state action value distribution and thereby avoid risky state action pairs (which is demonstrated by the low percentage failures).
The lower variance implies that the EVAC agent is able
The cumulative rewards in T show higher performances than other baselines. In the Mujoco environments, the cunulative reward of DSAC is slightly smaller to EVAC and comes at the cost of entering rare risky regions. We observe that RAAC agents are not as risk averse and do not improve cumulative performance either in extreme rare risk scenarios. EVAG agents while achieving higher cumulative performance, exhibit. very good risk aversion too. This implics that the EVAC agents explore systematically within the 'logal' and non-risky regions to maximize cumulative reward. We observe similar trends in risk aversion and the CVaR in the gem environments in Table 2 The cumulative rewards collected by EVAC agents is the maximum in case of the safety-gym environments which demonstrates the EVAC agents' ability to recognize and avoid extremely rare risk hazards (at even 3% extreme penalty rate) while trying to maximize cunulative reward.
summary, the obtained CVaR values demonstrate that EVAC agents improve optimization of the risk sensitive objective. The higher cumulative rewards indicate that EVAC agents explore well and are not too conservative. The near zero failure percentages demonstrate that the EVAC agents can be risk averse even under extremely penalizations.
16

---

Published in Transactions on Machine Learning Research (05/2024)
We perform several ablation studies in the Appendix. We perform ablation studies varying the penalization rate to assess the ability of EVAC to adjust to varying levels of extreme risks in Section D We also discuss the sensitivity of the EVAC algorithm on the threshold quantile level in Section F The CVaR quantile 21, controls risk averse behavior. We perform ablation studics with stricter I1 values on all baseline algorithms
in Section G lo ascertain simply changing the #1 value leads to conservative behavior. We find that simply increasing 21 does not lead to risk aversion extreme risk scenarios and only additional tail modelling aids in risk aversion. We perform a comparative analysis between the estimated tail distribution of the RL agents against the ground truth distribution for EVAC and other baselines in Section
B We describe the experimental setup and hyperparameters in greater detail in Section C In Section H we provide visual illustrations of the risk averse trajectories of the EVAC agents on the Safety-gym environments. We also provide a proof of the convergence of the Bellman operator used in EVAC in Section
A for a fixed policy. We
provide details of the MLE estimation procedure in Section B of the Appendix.
Acknowledgement
This research was supported by the U.S. Department of Energy, through the Offce of Advanced Scientife Computing Research's "Data-Driven Decision Control for Complex Systems (DnC2S)" project. The research participation of the University of California at Santa Barbara was supported under award number DE- SC0021321. Pacife Northwest National Laboratory is operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract No. DEAC05-76R1.016300 Oak Ridge National Laboratory is operated by UT- TTT-Batelle LLC for the U.S. Department of Energy under contract number 02220255000 The Jefferson Science Associates (JSA) operates the Thomas Jefferson National Accelerator Facility for the U.S. Department of Energy under Contract No. DE-AC05-060R2317.
References
Yu Bai, Song Mei, Huan Wang, and Caiming Xiong, Understanding the under-coverage bias in uncertainty estimation. Advances in Neural Information Processing Systems, 34:18307-18319, 2021.
Bojan Basrak. Fisher- Tippett Theorem, pp. 525 526. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-64--04808-2. doi:
254.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, 2017.
Boris Beranger, Simone A Padoan, and Scott A Sisson. Estimation and uncertainty quantification for extreme quantile regions. Extremes, 24(2):349 375, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.0155 2016.
Sami Can, John Finmahl, and Roger Laeven. Two-sample testing for tail copulas with an application to equity indices. Journal of Business Economic Statistics, pp. 1-29, 01 2023. doi:
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. ArXio, abs/ 1406.3339 2014.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. ArXiv, abs/1552.00162, 2015a.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a ovar optimization approach. ArXie, abs/1506.02188 2015b.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In A AAT Conference on Artificial Intelligence, 2017.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Roberto Daluiso, Marco Pinciroli, Michele Trapletti, and Edoardo Vittori. Cva hedging with reinforcement learning. Proceedings of the Fourth ACM International Conference on AT in Finance, 2023. URL
http:://api. sepainieaantccs / wtt.:ooorr.gg/CouuuSDD.... :654447822647777 265447857
Benjamin Eysenbach, Shixiang Shane Gu, Julian Tbarz, and Sergey Levine Leave no trace: Learning to reset for safe and autonomous reinforcement learning. ArXiv, abs/1711.06782, 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXio, abs/1802.09947,, 2018.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Frmon. Extreme q-learning Maxent rl without entropy. ArXiv, abs/2301.02333 2023. URL http:p/errpgg...aaaaareeeeaaaarr
Ather Gattami, Qinbo Bai, and Vaneet Aggarwal. Reinforcement learning for constrained markov decision processes. In International Conference Artificial Intelligence and Statistics, 2021.
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk- reinforcement learning. ArXiu, (bs/2225.05135 2022. URL http::://ap.sseaanntcccchooorr wwwpcechooaarro/rrrrooroeo /tto/oopsSII22486355000
Laurens Haan and Ana Ferreira. Extreme Value Theory: An Introduction. 01 2006. ISBN 978-0-387-23946--0 doi: 10.1007 1110070--37-3447---1
Jia Lin Hau, Marek Petrik, Mohammad Ghavamzadeh, and Roazul Hasan Russel. Rasr: sote--- mdps with ovar and entropic risk. ArXiv, abs/2209.04067, 2022. URL
org/Corpus : 252185446
Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. In Neural Information Processing Systems, 2019.
URL http://ppp..emmnicsccchcllooeeee wrtg/Corpps1::1111666...
Peter J. Huber. Robust estimation of location parameter. Annals of Mathematical Statistics, 35:492-514 518, 1964.
M. Jakobsson, XiaoFeng Wang, and S. Wetzel. Stealth attacks in vehicular technologies. In JEEE 60th Vehicular T'chnology Conference, 2004. VTC2004-Fo6 Fall, 2004, volume pp. 1218 1222 Vol. 2, 2004. doi: 10.1109/VETECC:2000 20001ECF2000...4021.6
Jiaming Ji, Borong Zhang, Xuchai Pan, Jiayi Zhou, Juntao Dai, and Yaodong Yang. Safety-gymmasium. Gilffub repository, 2023.
Taylor W. Killian, S. Parbhoo, and Marzych Ghassemi. Risk sensitive dead-end identification in safety-critical
offline reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023. htt.aattpss/ppp..mmaattttooooooo
org/Corpus :B: 255825752
Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, pp. 33-50 33-50, 1978.
Edouard Leurent. An environment for autonomous driving decision-making httpp::/giituub.com/eeee/// highway- env 2018.
Timothy P. Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.0299 2016.
Guiliang Liu, Yudong Luo, Oliver Schulte, and Pascal Poupart. Uncertainty-aware reinforcement learning for risk-sensitive player evaluation sports game. In Neural Information Processing Systems, 2022. URL
https://api.... htt.aapp...mmaanncccccllala
wttpechho......ree/rruuusu...aaa 258509511
Yudong Luo, Guiliang Liu, Pascal Poupart, and Yangehen Pan. An alternative variance: Gini deviation for risk-averse policy gradient. ArXiv, abs/2301 23077088777. 2023. URL https://api. /tt.attp://ppp....annccoooo http::///pp..emmanicccccoollr www.sticcccooor..orro..
CD:595ID::299951221
18

---

Published in Transactions on Machine Learning Research (05/2024)
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arxio: Learning, 2020.
Yecheng Jason Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributiional reinforcement. learning. In Neural Information Processing Systems, 2021. URL attp:://ap...emmntiicccloll..
2
Saced Marzban, Erick Delage, and Jonathan Yu-Meng Li. Deep reinforcement learning for equal risk
pricing and hedging under dynamic expectile risk measures. ArXiv, abs/ abs/2109.04001, 2021. URL https
//ap..semantiicscoolle wttpsiiccccoaaaooorrrrorreoeeeee
237452748
Qiyun Pan, Young Myoung Ko, and Funshin Byon. Uncertainty quantification for extreme quantile estimation with stochastic computer models. IEEE Transactions on Reliability, 70(1):134- 145, 2020.
Xinlei Pan, Daniel Seita, Yang Gao, and John F. Canny, Risk averse robust adversarial reinforcement
learning. 2019 International Conference on Robotics and Automation (ICRA), pp. 8522-8528, 2019. URL
https://api. semanticscholaatte. www.:o/rg///r...o...ommm
oetp.eaeetttttto......o.oooooooa 0ttp//oopuuIDD:00222110 90262410
James Pickands III. Statistical inference using extreme order statistics. the Annals of Statistics, pp. 119 131, 1975.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Kumar Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, 2017.
A. PrashanthL. and Michael C. Fu. Risk-sensitive reinforcement learning: constrained optimization viewpoint. ArXiu, abs/1810.09126,, 2018.
Kishansingh Rajput, Malachi Schram, and Karthik Somayaji. Uncertainty aware deep learning for particle accel- erators. Arxie, 0t/p/2309.15550 2022. URL https://api semantccscoolarr http:///pppp.eeemnnccccccoo..... http://aaa.....toooooooooooooooa
Thierry Roncalli. Handbook of Financial Risk Management. 04 2020. ISBN 9781315144597. doi: 10.1201/ 9781315144597.
Florencia Santiñaque, Juan Kalemkerian, and Madeleine Renom. Spatial clustering of extreme annual precipitation in uruguay. Revista Brosileira de Meteorologio, 01 2023. doi: 10.159000100--7777730077
Stefan Schneider, Stefan Werner, Ramin Khalili, Artur Hecker, and Holger Karl. mobile-env: An open platform for reinforcement learning in wireless mobile networks. In Network Operations and Management Symposium (NOMS). IEEP/IFIP 2022.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. ArXiv, abs/1902.08708, 2019.
Krishna Parasuram Srinivasan, Benjamin Eysenbach, Schoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with a safety critic. ArXio, abs/ abs/2010.14603, 2020.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the cvar via sampling. In AAAI Conference on Artificial Intelligence, 2014a. URL https://app: www.ssicscccholarrrt.........a.. org/CorpusID 14002092
Aviy Tamar, Yonatan Glassner, and Shie Mannor. Policy gradients beyond expectations: Conditional value-at-risk. ArXiv, abs/14044 1862,,403,,62 2014b.
Yichuan Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. In Conference on Robot Learning, 2019.
Dylan Troop, Frédéric Godin, and Jia Yuan Yu. Risk-averse action selection using extreme value theory estimates of the evar. arXiv preprint 1rXi::1912.011718 2019.
Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offiine reinforcement learning. In Proc. International Conference Learning Representations (ICLR), May 2021.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Li Xia. Optimization of markov decision processes under the variance criterion. Autom., 73:269-20 278, 2016.
Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Math. Oper. Res., 37:288 300 2010.
Chengyang Ying, Xinning Zhou, Dong Yan, and Jun Zhu. Towards safe reinlorcement learning via constraining conditional value-at-risk. ArXiu, 218/2200004436, 2022.
20

---

Published in Transactions on Machine Learning Research (05/2024)
A Proof of convergence of the Bellman Operator in EVAC
In this section, we set out to prove the convergence of the Bellman update Equ
7 (=) = + (s ,') (1 (1-7))((((,d)
Definition 1: For any two random variables (1 a) and J2(s, a) with distributions a) and (2(s, a) with inverse CDF functions (1(.a) and F (2((,)) respectively, the Wasserstein distance dp is defined as:
(f. 1 (T) ( (u) 1/P
dp (Fr,(sa)) Fostna))
Equivalently, the maximal Wasserslein distance dp is defined as:
FF F11) sups,a ()i Fis(sra))
Property 1: For scalar constant r, the shifted random variables as +r and J2(s, tr have
(hitea)tttFFpsssaattt dp (F)((a)) F12(2.)))
Property 2: For real constant scaling factor 0 1, the scaled random variables (s,
and 7J2(8,a) a) have
dp ( ( FrJe(en)) S Sdp (F)(())) Fas(S.))
Definition 1: For distribution Z, quantile level and its corresponding quantile Zn, define the non-tail distribution Zμ = Pr(Z < Z,) and the non-tail distribution Zμ = Pr(Z 1
Zn).
Theorem 1: Let Z denote the space of all state action value distributions. For the state action value distribution as (s) (1 - 1)) 2 (s,a), where Zj represents the non-tail region of Z and Zu represents the taal region of Z (as described in Definition 1), the Bellman operator T" Z x Z, is contraction under the maximal Wasserstein distance metric dp-
Note: For notational convenience, we express dp (F))(en), Fsa(ews) as dg (1 a), Z2(8,a)).
Proof:
(T TT
((. = (100:-/..60000--...a...-.------- (1- (s, a'))
= <d -1(1,221((,))))-------0111111111 7 (,-
S sdz (41 (
yd Z 2((5)
dT 71, 72) sup, dp
S sup (Zi(s,),, (2(s,))))))))))) 2()
ddp (s) (2(,,a)
21

---

Published in Transactions on Machine Learning Research (05/2024)
From the above equation, prove that the 7 operator is contraction and that the Bellman update with the GPD tail distribution converges.
B MLE Estimation of the parameters of the GPD Distribution
The CDF of the GPD distribution (e. is given by:
1
1 + Ex
for 5 + 0
exp( / for
The log-density function (Iog-PDF) of the same GPD distribution is given by:
log(()) + 1/c - 1) log (1 [1+t2//o for & -og((o)) )- - log fe.o = )
(13)
2/a for 6 =
i log (x)
- -1/5 - 1 ( (1+ )
T + log (+ Ex/o) (1+82/)(11/5)
0E
a log (1)
--1 + (-115- 1 Ex/o =
8o
However when 5 0; we are left with the MLE estimation of the parameter a of the exponential distribution
i log (o,)(1) Oo
x σ²
(14)
We make the GPD parameter
a) be parameterized and represent it by oo (,, a). Given a batch of transition tuples (8,, 0g, T,, 84) ;i = B, where B is the batch size, source samples from the GPD distribution by sampling K samples Fş from Zjr a), the constructed GPD distribution. We then compule the empirical log-likelihood loss L that needs to be maximized.
B
K
2 log J
L.
(15)
B
K
1=1
k=1
1
B
K
2 log Je (T
Co
(16)
B
K
dog
where 2ạ as, K is the number of samples sampled from the GPD distribution (s, a). We set == 100 and B == 128 for all experinentation. We perform gradient ascent the parameters 0 by using the empirical loss:
0:0 + Cir Lo $ : ④ + Oir Lo
(17)
where ag, is the learning rate, Thus, there is a computation overhead of O(B-K) in updating the parameters of the GPD distribution.
C Experimental details and Hyperparameter setting
We use an actor critic framework, where the critic is quantile critic. Both the actor and critic have layers with hidden size being 128.
22

---

Published in Transactions on Machine Learning Research (05/2024)
C.1 Mujoco Environments
During training and inference, the max episode length of the agent is set. to 1000. During training, the agents were trained for 100,000 time steps on the whole. The batch size B = 128 and we set K, the number of samples sampled from the GPD distribution to 50. We set the learning rates for the actor and critic to 0.001 in all cases. The discount factor = 0.99 for all cases too. The soft update parameter T == 0.02 for all our experiments on the Hopper and Walker2d, while r = 0.01 for the HalfCheetah environment.
C.2 Safety Gym Environments
During training and inference, the max episode length of the agent is set 1000. During training, the agents were trained for 70,000 time steps on the whole for the safety-gym suite.
In all of the ablation studies, report results on different trained agents.
D Risk Awareness and Performance with changing Penalization rate p
We recall that the penalization rate p is defined as the Bernoulli distribution parameter that is used to create rare risky rewards of the form r(s, a) = R(s, a)+lvaaLLBB for the Half- Chef-Cheetah environment. As becomes smaller, the penalty becomes rarer, thereby indicating more extreme risk scenarios.
In this section we change this penalization rate p of the reward for the HalfCheetah-v3 environment. We move from an extreme rare reward penalty of p 0.03 a fairly frequent penalty rate of p = 0.1. For our experimentation purposes, we fix the threshold quantile 17 0.96 for both EVAC and RAAC-TD3. From Table 8 however, the penalization rate p becomes smaller (rare risky events), the EVAC agent still exhibits very small percentages of failure while the cumulative reward is still very high. This bolsters the fact that the EVAC agent still explores reasonably while being risk averse.
Algorithm
p
Percentage Failure
Cum. Reward
CVaR
EVAC
0.03
6.43 + 4.99
1412.81 + 292.22
140.66 + 23.57
0.05
2.87 + 1.3
1502.46 = 94.25
156.71 11.07
0.1
1.07 1 0.63
1160.8 1 264.46
111.48 1 33.65
RAAC-TD3
0.03
30.23 = 12.21
847.86 + 145.21
85.66 = 24.12
0.05
41.31 16.6
836.8.1 195.85
129.81 + 38.75
0.1
13.97 + 5.58
366.0 + 453.85
104.92 + 7.08
Table 3: Table showing various performance metrics as the penalization rate is varied for the HalfChoctah environment (with fixed threshold quantile v) = 0.96).
E
Comparison of the modelled distribution with the Ground-truth distribution
Since we believe that the use of EVT theory helps in accurately modelling the tail of the distribution of the future sum of discounted returns, we compare the modelled distribution (7 a) of every algorithı (baselines and EVAC) against the true empirically obtained distribution the sum future discounted returns.
We froeze the initial state to be the same across all algorithms (RAAC-DDPG, RAAC-TD3, D-SAC, EVAC) and collect trajectories according to optimal policy defined for each algorithm. For example, each algorithm has defined optimal policy and trajectory T [R1, ... is collected by following the optimal policy of that algorithm. For such trajectory T we compute the future discounted sum of rewards J -== E. 7R, R. for the very first state in the trajectory. We repeat this process of trajectory collection for the same initial state N times to estimate the distribution of the sum of future discounted returns for each algorithm from [J
On the other hand we sample values from the quantile critic of each algorithm to obtain the modelled distribution zdd (s, a). We collect { M where 2f 7(8, (s)). It. is to be noted that since each algorithm
23

---

Published in Transactions on Machine Learning Research (05/2024)
has different optimal policy, the empirical distribution of the sum of future discounted returns would vary for each algorithm.
RAAC-DOPG
RAAC-TD3
D-SAC
3.0
1.0-
1.0
10
2-diertoucion
0.5
0.6
0.5
50
160
130
1.00
150
(a) HalfCheetah
RAAC DOPG
RAAC TD3
D SAC
EVAC
Z-diarfbuution
0.5
0.4
0.2
0.2
0.2
0.2
0.0-
0.0
0.0
0.0
1.00
110
260
139.4
(b) Hopper
RAAC-DOPG
RAAC-TD3
D-SAC
EVAC
3.0
groundench d-dAAtoooioo
2.5
0.5
0.5
0.0
0.4
1.00
125
130
60
80
120
125
97.5 103.0
202.3
(c) Walker2d
Figure 4: Comparison of the empirical CDF of the modelled distribution (in dashed red) and the empirical CDF of the ground truth (in solid blue) sum of discounted rewards. All plots are plotted for non-negated rewards. The tail behavior (left tail) of the EVAC agents almost precisely matches the true empirical distribution of the sum of the future discounted rewards. (Note that the scale of the X-axis is different. for each algorithm and environment)
We then plot the empirical CDF of both {J (which called the ground truth) and M (modelled Z-distribution). These empirical CDFs are plotted for each algorithm and for each Mujoco agent in Figure
We compute the 1-Wasserstein distance between the ground truth and the modelled distribution in Table The 1-Wasscrstein distance di (X,Y) between random distributions X and Y is defined as:
di xxx,
(f [F (w) -F Fu))dudu (u)ldu)
Firstly for the HalfChoctah, we see that the left tail of the ground truth and the modelled distribution for EVAC, both concentrate in the same region of around 100-120. For the Hopper, the ground truth and the modelled Z distribution both are almost perfectly aligned at value of around 129 to 130. For the Walker2d,
24

---

Published in Transactions on Machine Learning Research (05/2024)
Environment.
Algorithm
1- Wasserstein distance (4)
HalfChectah
RAAC-DDPG
55.05
RAAC-TD3
57.69
D-SAC
37.55
EVAC
14.89
Hopper
RAAC-DDPG
38.22
RAAC-TD3
28.77
D-SAC
86.87
EVAC
0.64
Walker2d
RAAC-DDPG
42.70
RAAC-TD3
21.95
D-SAC
47.40
EVAC
3.32
Table 4: Table showing J-Wasserstein distance between the empirical ground truth distribution of the sum of discounted rewards and the modelled distribution as shown in Figure 4
similar trend in the tails of the groundtruth and modelled distribution for the EVAC algorithm. (Note that the scale of the X-axis is different for each algorithm and environment). From Table 4 and Figure we notice that there is generally smaller discrepancy between the ground-truth distribution of the EVAC agent and its corresponding modelled distribution, in comparison with other baselines that do not use EVT to model the tail behavior. Thus, EVAC agents are able to better model the true underlying distribution, especially the tail regions.
F Risk Awareness and Performance with changing threshold quantile parameter 7)
We next address the sensitivity to risk awareness when the threshold quantile parameter 1j is changed for the HalfCheetah-v3 environment. The value of r) indicates the threshold quantile level after which the GPD approximation for 2 (s, a) is made.
We fix the penalization rate p 0.05 while changing v continuously. We also fix 21 0.95 and 122 1.0 for CVaR calculation.
Environment
?
Percentage Failure
Cum. Reward
CVaR
HalfCheetah
0.90
35.17
23.26
946.65 - 234.75
183.68
35.38
0.96
2.87 1 1.3
1502.46 + 94.25
156.71
11.07
Table 5: Table illustrating various performance metrics as the threshold quantile is varied for the HalfCheetah environment (with fixed penalization rate 0.05).
We observe that the choice of the threshold quantile 7) plays an important role in risk mitigation too. We observe that risk sensitivity increases with increase in the value of 1.. We posit that this is due to better approximation of the GPD distribution as 1) increases (Theorem 4.2). Table as r increases, the EVAC agents become very risk averse while producing good cumulative rewards. This underscores the effectiveness of using EVT theory for risk mitigation in reinforcement. learning.
G Effect of increasing I1 in CVaR optimization (more conservative behavior)
We recall from Equation 12 in the manuscript that 21 is used control the level of risk aversion. We re-write Equation 12 from the manuscript below.
25

---

Published in Transactions on Machine Learning Research (05/2024)
arg min CVaR(x1)
Z (0,,(())]1-
arg min
32
21
T=X)
As 21 - 1, the policy is incentivized to optimize the right tail and is thus made more risk averse. Instead of 21 0.95, we set 21 0.99 ensure the highest level of conservatism.
Environment
Algorithm
Percentage Failure
Cum. Reward
CVaR
RAAC-DDPG
47.07 17.64
677.15 173.03
126.22 + 115.43
HallCheetah
RAAC-TD3
30.47 + 4.97
360.55 = 242.4
104.21 + 12.0
D-SAC
35.7 40.42
1089.13 = 395.36
143.37 = 50.41
EVAC
1.03 = 1.26
872.62 = 171.62
151.45 + 4.33
Table 6: Table showing the the percentage failure and also the cumulative reward as risk sensitivity parameter #1 is made stricter at 0.99 instead of 0.95 for the HalfCheetah environment (with fixed penalization rate p = 0.05 and threshold quantile 70..66).
From Table 6 even when the CVaR objective made very risk averse (at ø1 = 0.99), the bascline agents still do not demonstrate good risk mitigation. The vanilla quantile based methods which depend on sampling exclusively, may not model the tail behavior accurately enough. This implics that even if æ1 is made arbitrarily large (more risk averse) in such agents, proper risk mitigation may still not be observed. EVAC the other hand demonstrates good risk aversion. This further acts motivation to using EVAC-like agents which seem to evidently show better risk mitigation.
H Trajectory Visualization on Safety-gym environments
In this section, we provide visual depictions of the trajectories followed by different agents in the Safety-gym environment. We represent the hazards by red circles. The starting position is indicated in green and the goal position indicated in blue Figures 5 and 6 As discussed the manuscript, the goal of the agent is navigate to the goal while avoiding the red 'hazard circles'. In the scenarios A and B, we observe that the EVAC agent is able to efficiently learn the locations of the hazard and avoid such hazards at all times. However, due to the extremely rare penalty rate of p 0.05 for Scenario-A and p 0.03 for Scenario-B, the other baseline agents, do not seem to recognize the low probability events and often enter the hazardous blue circles.
Histograms showcasing risk avoidance
We illustrate the histogram of the quantities of interest the Mujoco environments Figure 7 We perform inference on the trained agents and record the quantities of interest for each Mujoco environment. We observe that the quantity of interest. (i.e. velocity of the HalfCheetah and the angle of the Hopper) are mostly within the safe region anticipated.
J Experiments on Discrete Action Space
To extend the EVAC algorithm to discrete action spaces we experiment on an autonomous driving environment [Leurent] 2018) where the agent is lasked with driving a car on a highway. The agent is rewarded for moving
fast and avoiding any collisions. However, the car agent may get penalized rarely by highway patrol for
spoeding over the speed limit. The goal of the agent thereby navigate on the highway at the speed limit while avoiding any collisions with other cars on the highway. To make speeding penalties rare, we ensure that the highway patrol present only 5% of the times.
26

---

Published in Transactions on Machine Learning Research (05/2024)
(a) RAAC-DDPG
(b) RAAC-TD3
(c) DSAC
(d) EVAC
Figure 5: Trajectories visualized for the Safety-gyn gym Scenario -A with p 0.05. The EVAC agent successfully
navigates to the goal by recognizing the blue hazards.
(a) RAAC-DDPG
(b) RAAC-TD8
(c) DSAC
(d) EVAC
Figure 6: Trajectories visualized for the Safety-gym Scenario-B with p 0.03. The EVAC agent successfully navigates to the goal by recognizing the blue hazards.
The highway environment has a continuous state space with dimension 25. It has discrete action space with actions. We rate the highway environment below in Figure g
27

---

Published in Transactions on Machine Learning Research (05/2024)
Velocity (HallCheetah))
RAAC DOPG
Angle (Hopper)
DS
RAACTOS
EVAC
region
EVAC Safee region
-0.2
-0.1
0.0
0.1
0.2
(a) Histogram of the velocity of the HalfCheetah en-
vironment
(b) Histogram of the angle of the Hopper environment
(Hopper)
RAAC-DDPG
RAAC-TD3
DSAC
Safe region
-1.0
-0.5
0.0
0.5
1.0
(c)
Histogram the angle of the Walker environment
Figure 7: Histograms of the quantities of interest for the Mujoco environments
50
100
0
100
200
300
400
500
Figure 8: Simple illustration of the highway environment
28

---

Published in Transactions on Machine Learning Research (05/2024)
We compare the discrete version of EVAC with QR-DQN (Dabney al. 2017 percentage failure, cumulative reward and the CVaR value similar to Section g
metrics such the
Environment
Algorithm
Percentage Failure
Cum. Reward
CVaR
QR-DQN
42.58
133.2 + 80.39
31.54 19.87
Highway
EVAC
0.0 - 0.0
185.27 + 6.59
37.95 + 16.84
Table 7: Table showing the percentage failure, cumulative reward and CVaR for the discrete action space Highway Environment
Evolution of the percentage failure
We plot the percentage failure as function of the number of training iterations (same as the number of data samples collected) in Figure g We observe that the EVAC agents exhibit lower percentage failures throughout training and understand high extreme risk regions accurately.
HalfCheetah-v3
Hopper-v3
Walker2d-v3
RAAC-DDPG
60
RAAC-TD3
DSAC
60
50
EVAC
Percentage Fallure
40
40
30
20
20
10
0
25000 50000 75000
25000 50000 75000
0
25000
50000
75000
Training Iterations
Figure 9: Figure indicating the evolution of the percentage failure as a function of number of samples/ training iterations. The EVAC agent reaches smaller percentage failure rates faster comparison to the other baselines.
L Experiments with other complex risk scenarios
L.1 Risk aversion against rare malicious attackers in mobile communication environment
Often times when cell phone users move about in given area, they need to connect to different transmitting base stations near them, so that they communicate over the channel and observe high Quality of experience (QoE). Given multiple base stations and cell phone users the control problem is to assign users to differont base ital so Ihat. the over all QoE is maximized. However, in such a real world set there may exist. security compromised base stations that perform stealth attacks. Stealth attacks [Jakobsson et al. 2004) are characterized by base stations that perform rare attacks on agents a manner that remains undetectable. Such rare security compromises could lead devastating consequences cell phone users and need to be flagged unsafe.
29

---

Published in Transactions on Machine Learning Research (05/2024)
We employ mobile-env (Schneider et al. 2022). an open source environment that simulates the connections and QoE. between several base stations and cell phone users. To simulate stealth attacks, we nominate one of the base stations to be malicious meaning that it very rarely causes data privacy issues for the cell phone users. The design of a risk averse policy in this scenario involves actively disconnecting from the unknown malicious base station for any cell phone user.
We simulate these stealth attacks in two scenarios for the wireless communication.
Scenario A: We design 4 base stations and just 3 cell phone users who randomly move about with certain speed. There is single malicious base station that is unknown to the agent. The observation space of the agent is 27 dimensional and consists information on the connection between the cell phone and the base station, SNR of the channel etc. The reward is rarely penalized at rate p = 5% by a negalive penalty for any connections made to the malicious base station. The action space is continuous valued and its dimensionality corresponds to the number of cell phones.
Scenario T:ehe setup is similar to Scenario-A except we have base stations and 5 cell phone users that randomly move at a certain speed. The observation space is 45 dimensional. This set up is more challenging than the previons oncs as it involves multiple cell phone users trying to connect to a limited number of basc stations.
We evaluate the percentage failure (which indicates the number times the malicious base stations was connected to in th episode), the cumulative reward and the mean CVaR below.
Scenario
Algorithm
Percentage Failure
Cum. Reward
CVaR
Scenario-A
RAAC-TD3
38.89 + 11.34
-4.4 = 2.17
33.46 + 45.92
DSAC
88.89 9.07
-1.21 = 0.95
11.9 14.71
EVAC
3.7 + 5.24
-6.5 + 2.3
34.8 + 4.13
Scenario-B
RAAC-TD3
59.26 41.9
-3.97 + 1.51
-65.75 + 14.93
DSAC
67.38 + 12.94
-2.79 + 0.8
1.71 + 14.7
EVAC
29.63 + 22.83
-4.99 = 0.97
37.52 + 4.97
Table 8: Table showing various performance metrics for risk avoidance when malicious base stations are introduced into the mobile communication environment
A good risk averse policy in this setup will not connect to the malicious base station. However, being not connected to the base station reduces overall reward (QoE). Thus, we see that EVAC which mostly understands not to connect to the malicious base station (demonstrated through low percentage failures), incurs smaller cumulative rewards.
Thus, EVAC which uses EVT based tail modelling risk averse even the presence of rare malicious stealth allacks by a base station.
L.2 High dimensional risk region for the Walker2d environment
The Walker environment as described in (Brockman et al. 2016 is a locomotive robot model that is tasked with walking upright with as high a speed as possible. However certain gaits of the walker may lead to potentially catastrophic consequences under certain environment settings. In this regard, we penalize the walker when it moves with very high velocity ,wen its head angle is bent and when it hops instead of walking.
We monitor the velocity v, head angle 0 and the vertical height h of the walker robot from its state space. We define hyperplane risk surface to introduce rare penalties. We define the risk hyperplane as below:
Wy (u (v-co) + wa (10) -0) + wn (h-ho) ho) > 0
where Wa, we. wA indicate the importance weight for velocity, head angle and height of the walker. We set this to be fixed w. = 0.3, we 1.0, wi 0.5. Any state space variable satisfying the inequality is considered
30

---

Published in Transactions on Machine Learning Research (05/2024)
risky. Unlike, the single paramter threshold, this is more restrictive and requires the agent to keep track of multiple state space variables. We introduce rare risky penalty with Bernoulli rate p 5% and assess the risk aversion of EVAC and other baseline algorithms.
Algorithm
Percentage Failure
Cum. Reward
CVaR
RAAC-DDPG RAAC-TD3
13.76 + 5.08
263.57 + 243.66
158.18 + 15.82
10.8 + 2.13
201.98 + 106.36
67.47 7.21
DSAC
16.76 + 4.78
566.47 127.98
167.97 28.88
EVAC
1.76 + 1.35
267.74 + 51.04
160.68 + 6.34
Table Table showing various performance metrics for the complex risk hypersurface
EVAC shows small amounts of cross over to the risky hypersurface while having lower variance in CVaR values.
M Limitations and Future Work
The use of EVT for risk mitigation in distributional reinforcement is novel. Our work tries to illustrate the benefit of using EVT theory in risk aware reinforcement learning. One of the challenges in the use of EVT RL the overhead involved in training parameters the GPD distribution through maximum likelihood estimation. The added overhead may prove expensive for both training and inference, especially on edge devices. We quantitatively discuss this computational overhead in the concluding part of Section B.We plan to further investigate the usefulness of EVT RL in other domains and study its scaling in other complex environments. Furthermore, would like to investigate the applicability of EVAC in the oflline reinforcement. learning setting.
31


Reviews:
[{"review_id": "J0vwJ3olUZ", "reviewer": "Reviewer_CjPa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "###Strengthes:\n\n- This paper is overall well-written and organized. The\n- The proposed method is well motivated and supported by both theoretical analysis and empirical evaluation.\n- The experiments are conducted from different perspectives, including performance comparison, risk-aversion evaluation, ablation and hyperparameter analysis.\n\n&nbsp;\n\n###Weaknesses:\n\n- Some details of the proposed method are not clear.\n- Sample efficiency is not presented, since the GPD distribution approximation is expected to require smaller number of samples as claimed.\n- The proposed method is only evaluated in continuous control environments with dense rewards.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies risk-aversion RL with a focus on the setting where low-probability but high-penalty risks exist, which poses great challenges to existing risk-aversion RL methods.\n\nTo my knowledge, the contributions of this papers are:\n\n- This paper reveals that existing distribution-based risk-aversion RL methods struggle in modelling heavy-tail value distribution and thus lead to poor aversion of rare risks.\n- This paper presents a new method to model state-action value distribution, inspired by Extreme Value Theory, through leveraging a decomposed approximation with the non-tail distribution and the tail distribution separately. This brings reduced variance and better approximation performance in accuracy and efficiency.\n- The proposed method is integrated into typical AC algorithms, whose effectiveness are demonstrated in MuJoCo and Safety-gym Environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. At the top of page 5, ‘distinct thresholds $u$ for each $(s,a)$’ is mentioned. However, I do not find more information about it. The details of how it is realized/learned should be clarified.\n2. The sample efficiency of the proposed method seems lacking. I find no learning curves for specific performance metric v.s. sample number.\n3. Although I agree that the overhead introduced for the GPD training and sampling is not a big point. I recommend the authors to give quantitive discussion because it can be useful to readers for practical consideration.\n4. I recommend the authors to discuss the potential of extending the proposed method to discrete-action and sparse-reward environments.\n5. The information on how many random seeds/independent trials are used for the results are missing.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3XZoG8GfVT", "reviewer": "Reviewer_wbX5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis paper is well organized and lays out a clear narrative that motivates the proposed approach for risk-sensitive RL. I appreciated the principled analysis used to argue for the use of extreme value theory to model the long tail of the state-action return distribution. The use of robust risk estimators such as CVaR has been well supported in the risk-sensitive RL community, this paper contributes yet another demonstration of the efficacy of using this form of objective when formulating decisions in environments that incorporate outsized negative risks.\n\nThe intended contributions and proposed claims are well framed within the context of related literature. The experiments are appropriately designed to test and analyze these claims and appropriate baselines are chosen to compare with.\n\nDespite an extensive set of weaknesses, to be outlined and summarized next, I do believe this to be an interesting paper and of worth to the TMLR community. I am confident that it is well within the author’s capabilities to improve this work and shore up their intentions and demonstrated contributions + claims that the current paper fails to completely justify or validate.\n\n## Weaknesses\n\nAs I go through the following list of concerns and questions I had while reviewing the paper, I have attempted to put the most serious items at the beginning of this section. Many of these can be attributed to an incomplete framing of the problem and inconsistent or partial explanation of concepts, notation or definitions. Thankfully many of these issues can be addressed through a more careful revision of the text, with particular focus on justifying and clarifying each equation and modeling assumption. Another major consideration is a lack of adequate grounding of the narrative justifying the problem setting. As currently written, this connection is suspicious and lazy. I’ll discuss more about this below. **Finally, the last major (and perhaps the most serious) issue I found with this paper is that it omits perhaps the most appropriate baseline comparison to the proposed EVAC among the literature it cites, specifically CODAC-C from Ma, et al (2021).** In addition to this, the experimental benchmarks it uses are somewhat rudimentary from a performance standpoint (even with consideration of the extremely rare, high-risk adjustments to the domains). \n\n**I found the paper somewhat hard to follow in places as the exposition was incomplete and makes somewhat vague claims and justifications.** Perhaps it was the use of unfamiliar terminology that made it hard for me to follow. This unfortunately starts early in the paper. After introducing challenges of estimating the risk of seldom extreme events (beginning of page 2) the next paragraph refers to a “phenomenon” that hasn’t been adequately characterized or explained. Perhaps what is meant is that there are characteristics of current risk estimation approaches that produce inadequate or incomplete understanding of true risks. I do not want to belabor this point and highlight each instance throughout the paper where similar vague posturing occurs, possibly in an attempt to be more formal or make the insights appear more important than they are. It is however a pretty consistent issue throughout the current version of the paper. I would highly recommend that the authors take a more careful pass through the writing and revise their statements to be more direct, clear and simple while introducing the paper, the concepts used to develop the proposed EVAC as well as when highlighting the limitations of prior work. The paper would greatly benefit from this level of fine-grained revision.\n\nPart of the vague writing problems throughout the paper is the absence of a clear problem statement. From what I could put together, the intention and focus of the work is only as applied to continuous control problems with full observation and dense rewards, using value-based algorithms. It would help improve the clarity of the paper if the type of RL problems the proposed methods and approach are intended for.\n\n**The focus and importance of the problem setting used in this paper is poorly motivated and comes across as a post-hoc, hurried connotation to justify the existence of the work.** I acknowledge that this is a harsh statement from a reviewer. However, there are no real descriptions of where poor decisions in a high-risk environment (perhaps from poor modeling of the tails of the return distribution, incurring high variance?) could be detrimental. There is currently no real connection to the intended problem setting and how the proposed contributions could potentially be used to address current limitations. Without this the proposed approach and the demonstrated results are forgettable. I urge the authors to reconsider the narrative framing used to justify the need for better modeling of rare, high-risk events in safety critical scenarios. There are several clear examples of how to do this well in recent risk-sensitive RL literature (Greenberg, et al 2022; Hau, et al 2022; Liu, et al 2022; Killian, et al. 2023; Kastner, et al 2023; Leo, et al 2023) that was unfortunately omitted from this paper. Beyond being good examples, these papers are relevant to this work and possibly should be included as citations (full references at the bottom of this section).\n\nTo connect with the next paragraph, **it is concerning that the authors intentionally omit the use of CODAC-C from Ma, et al (2021) as a baseline** despite using a very similar experimental setup descried in Section 8.3. This algorithm is already demonstrated to improve beyond RAAC and D-SAC which are the only baselines used in this work. I cannot fully get behind advocating for this work without an honest effort to include CODAC-C as a baseline. Without full and adequate comparison to relevant baselines, the utility and overall performance of the proposed EVAC algorithm is called into question.\n\nAs may have been easy to tell from the previous paragraphs, I was largely underwhelmed by the experimental results. They are indeed promising and do appear to provide adequate conservatism without reducing overall performance. However, the experimental domains are purposefully crafted to reward cut and dry avoidance with pre-defined risk regions and are relatively easy control problems given the dense reward settings. The initial experimental analyses (namely the results in Figure 1a) are unsurprising, particularly given dense reward settings. The training and evaluation setup is not apparent and it is unclear how reliable the presented results in Section 8 are. **How are the training and evaluation settings different?** Are the values presented in Tables 1 and 2 drawn from the latter stages of training or from a held out set of evaluation tasks within the same domain? \n\nI’m somewhat confused by the modeling choices made in the use of EVT to represent the long tail of the estimated return distribution. It is unclear what is meant by using values that are “large enough” for $u$, $\\eta$, etc… With the emphasis on a large value for $\\eta$, isn’t the effect of the GPD modeling of the extreme values somewhat minimized or ignored? It’s not clear how much the intended contributions are actually used? There is a lot left to be desired in how the method and the modeling choices are presented in Section 6. There’s a lot of emphasis on the reduction in high variance estimation of extreme events using EVT and that the use of quantile regression, focused solely on the pinball loss, is insufficient in Section 5 but this very same loss is used for EVAC?! Does this make sense? Are the anticipated weaknesses the pinball loss not fully addressed by EVAC because of the mixture distribution used to model the tail and non-tail portions of the state-action return distribution? __This calls into question the claim made at the end of page 10 in Section 7.2.1__. In total, Section 7 seems to make a lot of postulating without providing sufficient contextualization or justification/proof. This could be amended by more directly and clearly expressing the intended insights in the discussion.\n\nIn a related note, it’s somewhat confusing that the authors choose to evaluate and discount previous distributional risk-sensitive RL approaches via the Pinball loss which is not used in practice. As acknowledged by the authors, these approaches use a different loss function. If there are similar variance issues, it would be beneficial for this paper for the foundational analysis that motivates the use of EVT to be based on what is actually used in “competing” risk-sensitive RL approaches. This leads me to question whether the implementations of the baselines compared to EVAC use the pinball loss instead of their actual loss functions?\n\n**The claim that EVAC performs well, even in the presence of limited data is not justified or demonstrated in any of the experiments.** If this is what is intended by the sparse rates of the extreme values occurring (presented in Figure 1), then this claim needs to be rephrased as it is currently misleading. There is a difference between making determinations of risk from limited data (handled in Killian, et al 2023) and relative sparsity of extreme events. I would encourage the authors to be very clear about what the intended claims are.\n\nThe first two paragraphs of Section 8 are repetitive and could be combined.\n\nWhat determined the choices of the rates of penalization of the Mujoco environments? This is unclear and seems to be inconsistent between environments. Is there a principled reason behind the choices? I have a hard time relying on the relevance of the domains and the communicated results since the domains seem specifically crafted for the intended algorithmic solution… Additionally, a replication of Figure 2 in Urpi, et al (2021) would be particularly notable to demonstrate the efficacy of EVAC in avoiding the risk regions in these domains where they are clearly defined a priori and broken down by the applied continuous actions. \n\n### References\n\n`[Ma, et al (2022)]` Ma, Y, D Jayaraman, and OBastani. \"Conservative offline distributional reinforcement learning.\" NeurIPS 2021.\n\n`[Greenberg, et al (2022)]` Greenberg, I., Chow, Y., Ghavamzadeh, M., & Mannor, S. “Efficient risk-averse reinforcement learning”. NeurIPS 2022\n\n`[Hau, et al (2022)]` Hau, J. L., Petrik, M., Ghavamzadeh, M., & Russel, R. “RASR: Risk-averse soft-robust mdps with EVAR and entropic risk”. arXiv preprint arXiv:2209.04067.\n\n`[Liu, et al (2022)]` Liu, G., Luo, Y., Schulte, O., & Poupart, P. “Uncertainty-Aware Reinforcement Learning for Risk-Sensitive Player Evaluation in Sports Game”. NeurIPS 2022\n\n`[Killian, et al. (2023)]` Killian, T. W., Parbhoo, S., & Ghassemi, M. “Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning.” TMLR 2023\n\n`[Kastner, et al (2023)]` Kastner, T., Erdogdu, M. A., & Farahmand, A. M. “Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning”. NeurIPS 2023\n\n`[Luo, et al (2023)]` Luo, Y., Liu, G., Poupart, P., & Pan, Y. “An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient”. NeurIPS 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presented in this paper features an approach to conservative, risk-sensitive Reinforcement Learning by modeling the tail of a state-action return distribution’s tail using extreme value theory (EVT). The proposed approach is developed to reduce the variance when estimating extremely high risk events that rarely occur, a demonstrated limitation of standard quantile regression. \n\nThis motivates a new actor-critic based approach that alternates between training the return distribution using quantile regression and then maximizing the likelihood of the extreme value distribution modeled to cover the high-risk tail behavior (using a generalized Pareto distribution). A control policy is optimized using a CVaR risk measure, seeking to minimize the selection of actions that lead to possible extreme outcomes. Altogether, this algorithmic approach (named EVAC) is tested against a collection of rudimentary benchmarks from Mujoco and Safety-Gym, adapted to test the risk-sensitivity (and risk aversion) of the proposed approach against a small collection of relevant baselines derived from two prior methods from the literatures.\n\nIn total the paper makes the following claims (to the best of my understanding):\n 1. Better modeling of high-risk extreme events is provided by using EVT in place of traditional quantile regression.\n\n 2. Variance reduction in the estimation of rare events via EVT is maintained even when data is limited.\n\n 3. The proposed EVAC algorithm and policy optimization approach provides improved cumulative performance while significantly reducing the effect of possible extreme events through learned avoidance of a priori defined risky regions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would highly recommend that the authors take a more careful pass through the writing and revise their statements to be more direct, clear and simple while introducing the paper, the concepts used to develop the proposed EVAC as well as when highlighting the limitations of prior work.\n\nThere needs to be proper grounding of the work and the demonstrated empirical justification in actual problems of import. Why is reducing variance of tail behavior in a distribution important? How do the experiments demonstrate this?\n\nThere should be an effort to incorporate CODAC from Ma, et al (2021) into the experimental analysis. Granted the algorithm is designed for offline RL, but so was RAAC (Urpi, et al (2021)) which didn’t restrict the authors from using it in this work as a comparative baseline.\n\nA more thorough literature review of risk-sensitive RL in safety-critical domains. I’ve provided a set of relevant references from recent years that address relevant issues and are highly related to the approaches taken in this work.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QaKfwWoOCp", "reviewer": "Reviewer_czsJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This paper proposes a mathematically rigorous method for avoiding bad but rare events in RL.\n+ The empirical results support the claims and the experiments are well designed.\n+ The paper studies an important topic in RL\n\nWeaknesses:\n- The paper is very dense and hard to follow for a reader unfamiliar with EVT.\n- It is unclear how sensitive the algorithm is to hyperparameter choices.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes a novel approach for risk mitigation in reinforcement learning using extreme value theory. They show that prior approaches do not well model tail risk and learn a parameterized distribution of the state-action value distribution. Using this parameterization the paper proposes an actor-critic framework that can effectively optimize a policy to avoid rare extreme events. The evaluations show good empirical success and perform better than baselines. The authors study their approach in Mujoco tasks where there are rare penalties on velocities and angles and in safety gym where there are hazards that provide a rare penalty. The results show quantitatively and qualitatively that the proposed EVAC algorithm achieves the desired policy while avoiding the rare catastrophic events.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "When citing papers parenthetically \\citep{} should be used rather than \\cite{}.\n\nWhy just split the distribution into tail and non-tail. How do you decide where to split and what are the downsides of modeling the distribution in this way? Would it be better to model the distribution with more than 2 pieces?\n\nThe authors show that prior work on distributional RL is not sufficient for mitigating rare bad events, but then it seems that the proposed approach still relies on distributional RL. I would like clarification on how the EVT approach compares to prior work.\n\nI would also like more details on how the different hyperparameters were chosen for the domains.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission presents a novel and interesting use of extreme value theory (EVT) for risk-averse RL. The authors provide both theoretical and empirical justification for their work, and the presentation is clear and well-motivated. The authors have successfully addressed the reviewers concerns, so we are all in agreement to accept this work.
Decision: Accept (poster)