Extreme Value Policy Optimization for Safe Reinforcement Learning
Authors: Shiqing Gao
Yihang Zhou
Shuai Shao
Haoyu Luo
Yiheng Bing
Jiaxin Ding
Luoyi Fu
Xinbing Wang
Venue: ICML 2025 poster (2025)
Keywords: constrained RL
extreme value theory
constraint satisfaction
Primary area: reinforcement_learning

Abstract: Ensuring safety is a critical challenge in applying Reinforcement Learning (RL) to real-world scenarios. Constrained Reinforcement Learning (CRL) addresses this by maximizing returns under predefined constraints, typically formulated as the expected cumulative cost. However, expectation-based constraints overlook rare but high-impact extreme value events in the tail distribution, such as black swan incidents, which can lead to severe constraint violations. To address this issue, we propose the Extreme Value policy Optimization (EVO) algorithm, leveraging Extreme Value Theory (EVT) to model and exploit extreme reward and cost samples, reducing constraint violations. EVO introduces an extreme quantile optimization objective to explicitly capture extreme samples in the cost tail distribution. Additionally, we propose an extreme prioritization mechanism during replay, amplifying the learning signal from rare but high-impact extreme samples. Theoretically, we establish upper bounds on expected constraint violations during policy updates, guaranteeing strict constraint satisfaction at a zero-violation quantile level. Further, we demonstrate that EVO achieves a lower probability of constraint violations than expectation-based methods and exhibits lower variance than quantile regression methods. Extensive experiments show that EVO significantly reduces constraint violations during training while maintaining competitive policy performance compared to baselines.

Extreme Value Policy Optimization for Safe Reinforcement Learning
Shiqing Gao Yihang Zhou'
Shuai Shao Haoyu Luo Xinbing Wang'
Yiheng Bing'
Jiaxin Ding' Luoyi Fu'
Abstract
et al., 2018; Xu et al., 2020) and games (Vinyals et al., 2019; Yu et al., 2022a). However, the absence of safety guaran- tees poses significant barrier to the deployment of RL in real-world scenarios. To overcome this limitation, Con- strained Reinforcement Learning (CRL) (Ding et al., 2020; Stooke et al., 2020; Xu al., 2021; Yang et al., 2022) aims to optimize policies by maximizing cumulative rewards while satisfying predefined constraints. In safety-critical domains such as autonomous driving and financial trading, safety guarantees are highly susceptible to extreme events, which occur with low probability but can lead to significant consequences (NS et al., 2023). For instance, black swan incidents are especially concerning due to their impact to cause catastrophic outcomes (Masys, 2012).
Ensuring safety is a critical challenge in apply- ing Reinforcement Learning (RL) to real-world scenarios. Constrained Reinforcement Learning (CRL) addresses this by maximizing returns under predefined constraints, typically formulated as the expected cumulative cost. However, expectation- based constraints overlook rare but high-impact extreme value events in the tail distribution, such as black swan incidents, which can lead to severe constraint violations. To address this issue, we propose the Extreme Value policy Optimization (EVO) algorithm, leveraging Extreme Value The- ory (EVT) to model and exploit extreme reward and cost samples, reducing constraint violations. EVO introduces an extreme quantile optimization objective to explicitly capture extreme samples in the cost tail distribution. Additionally, we pro- pose extreme prioritization mechanism dur- ing replay, amplifying the learning signal from rare but
Most CRL methods evaluate constraints based the ex- pected sum of costs (Achiam et al., 2017; Stooke et al., 2020; Ding et al., 2021), optimizing policies to ensure the expected cost remains below a predefined threshold. However, this expectation- based evaluation only guarantees compliance on average, neglecting the inherent variability of the cost distribution, particularly in extreme samples arising in the tail. Consequently, even when the expected cost satisfies the constraint, frequent constraint violations can occur due to the presence of these extreme events, as shown in Figure la.
extreme samples. Theoret- ically, we establish upper bounds on expected constraint violations during policy updates, guar- anteeing strict constraint satisfaction at zero- violation quantile level. Further, we demonstrate that EVO achieves a lower probability of con- straint violations than expectation -ased methods and exhibits lower variance than quantile regres- sion methods. Extensive experiments show that EVO significantly reduces constraint violations during training while maintaining competitive pol- icy performance compared to baselines.
To minimize the probability of constraint violations, prob- abilistic constraint methods (Chow et al., 2018; Hiraoka et al., 2019) optimize policies based on constraint distribu- tions. WCSAC (Yang et al., 2021) employs Distributional RL (Bellemare et al., 2017) with a Gaussian approximation of the cost distribution, computing Conditional Value-at- Risk (CVaR) (Rockafellar & Uryasev, 2002) to guide policy optimization. However, the Gaussian distribution fails to accurately capture the tail behavior, as shown in Figure Ib. QCPO (Jung et al., 2022) introduces quantile-basee con- straint objective with Value-at-Risk (VaR), deriving approx- imated policy gradient. However, these methods overlook the critical impact of extreme samples during training.
1. Introduction
Reinforcement Learning (RL) has achieved remarkable suc- cess across various fields such robot control (Haarnoja
Extreme samples, characterized by their low probability but high impact (NS et al., 2023), play a crucial role in CRL. Specifically, extreme reward samples provide insights into achieving task objectives, while extreme cost samples highlight critical constraints. Unfortunately, these samples
'Shanghai Jiao Tong University, Shanghai, China. Correspon- dence to: Jiaxin Ding <jiaxinding @ wjtt.cddu.cn..
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
are naturally scarce, arising infrequently and often requiring risky or expensive exploration obtain. This searcity poses challenges in modeling tail distribution, leading to high variance (Urpí et al., 2021). Therefore, accurately modeling tail behavior and effectively exploiting these rare but crucial samples is essential for reducing constraint violations and improving policy performance in CRL.
treme prioritization mechanism, enhancing learning from extreme samples.
3. We theoretically and empirically demonstrate that EVO achieves lower expected constraint violations, lower violation probability, and reduced variance compared to baselines. We provide the code for EVO in https: / / /qgitub.... com/ ShigingGao//VV.
expected cumulative cost
condrrint threshold
2. Related Work
Expectation Constraint. The optimization methods in CRL with expectation- constraints can be categorized into primal-dual and primal approaches. Primal-dual meth- ods (Ding et al., 2021; Ying et al., 2024) convert con- strained problems into unconstrained ones using dual vari- ables. NPG-PD (Ding et al., 2020) establishes global conver- gence with sublinear rates. PID Lagrangian (Stooke et al., 2020) introduces proportional and differential control to mitigate cost overshoot and oscillations. In contrast, primal methods directly optimize constrained problems in the pri- mal space (Yang et al., 2020; Yu et al., 2022b; Gao et al., 2024). CPO (Achiam et al., 2017) enforces performance and constraint violation bounds within a trust region. FOCOPS (Zhang et al., 2020) solves the constraint problem in the nonparametric policy space and projects the policy back to the parametric space. CUP (Yang et al., 2022) provides gen- eralized theoretical guarantees for surrogate functions with generalized advantage estimator (Schulman et al., 2015b), However, these expectation- -ased methods fail to capture variability in the constraint distribution, resulting in fre- quent constraint violations. In this paper, we model the tail extremes of constraints using GPD and propose an op- timization objective based on extreme quantile to reduce constraint violation probability.
comninint vislation s0c4
609
602
Coruriatvve Cost
Cumulative Cost
(b)
Figure I: (a) Probability density function of cumulative cost.
Even when the expected constraint satisfies the threshold, there remains a high probability of constraint violations. (b) Fitting cumulative cost distribution with GPD and Gaussian, GPD captures tails more accurately.
In this paper, we propose the Extreme Value policy Opti- mization (EVO) algorithm, which leverages Extreme Value Theory (EVT) (Pickands III, 1975) to model and exploit these samples to enhance policy optimization. To miti- gate violations caused by extreme cost samples, we intro- duce an extreme quantile constraint, explicitly incorporating tail extremes modeled using a Generalized Pareto Distri- bution (GPD) based on EVT. Additionally, we propose extreme prioritization mechanism based on quantile levels derived from the GPDs of reward and cost, prioritizing low- probability but high- nigh-information samples during experience replay to exploit the learning signals in extreme samples. To address the high variance of extreme samples, we apply an off-policy importance resampling approach, augmenting extreme samples and enhancing tail distribution modeling. Theoretically, we establish an upper bound on expected constraint violations during policy updates in EVO, ensur- ing strict constraint satisfaction at a zero-violation quantile level. We further prove that EVO achieves lower constraint violation probability compared to expectation-basee CRL methods and exhibits lower variance compared quantile regression methods. Experimental results across multiple environments validate that EVO significantly reduces con- straint violations while maintaining strong policy perfor- mance. The main contributions of this work are as follows:
Probability Constraint. Probability-based methods (Chow et al., 2018; Hiraoka et al., 2019; Urpí et al., 2021) optimize the probability distribution of constraints to reduce violations. WCSAC (Yang et al., 2021) uses CVaR (Rockafellar & Uryasev, 2002) as a safety measure, approximating the constraint distribution with a Gaussian model. However, the Gaussian approximation fails to capture the tail decay rate, introducing significant bias for small tail probabilities. QCPO (Jung et al., 2022) transforms outage probability constraints into quantile-based ones and proposes an approximate policy gradient. However, these methods neglect the high variance of extreme samples and fail to fully exploit them. EVAC (NS al., 2023) employs EVT to reduce variance in extreme returns, improving risk aversion in RL, but does not address constraint satisfaction in CRL. In this paper, we propose an extreme quantile constraint optimization objective, leveraging EVT to model extreme samples and reduce variance, Additionally, we
1. We propose the EVO algorithm, integrating Extreme Value Theory (EVT) into CRL to model and exploit extreme samples, addressing the limitation in handling rare but high-impact events.
2. We propose an extreme quantile constraint based on Generalized Pareto Distributions (GPDs) and an ex-
2

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
introduce an extreme prioritization mechanism to fully exploit these extreme samples.
where is threshold and (x) denotes the GPD, follow- ing:
1- 1-(1+
1 1+ Ex
5
(3)
3. Preliminaries
3.1. Constrained Reinforcement Learning
Theorem 3.1 describes that the conditional distribution above a threshold t approaches the GPD.
CRI. can be modeled as a Constrained Markov Decision Process (CMDP), denoted by a tuple (S, A, R,, P, p. p)7), where S is the state space, A is the action space, R : S X A + R is the reward function, P S A + [] is the transition probability function, P is the initial state distribution, and
We introduce the necessary notation in this paper. Let X be random variable with cumulative distribution function P(X I x)) The tail distribution, representing the probability of X exceeding threshold x: F(x) P(X > z) = - F(z). Given quantile level μ. the quantile qu is defined as the smallest value such that P(X s k/)) 2 i.e.
F
(0, 1) is the discount factor. The cost function Cy S x A - I maps state- aatte-actionntiii pairs to costs c. denotes the constraint threshold.
P(X > qa) < 11
Starting from an initial state so sampled from the initial state distribution p. the agent perceives the state s, from the environment each time step i, selects action at S + A, receives the reward
4. Methodology
according to the policy r, and cost Ct. and transfers to the next state 81+1 based on P($t+1|st. at). The set of all stationary policies is de- noted as II. The discounted future state visitation distri- bution is defined as d' (s) : (1 - 7) Er= P(ss = s|mr). The value function for policy π (s) == E o ae) sg = s], and action-value function is (s) == Ey E R(s, R())s,)))s
fc
fiting with fir
EVT-hased Constraint
peak set Yn
extreme Ze
Extreme
Prioritization
14
qu
Autu d
00/1-
2
Figure 2: Extreme quantile constraint qu+- and EVT-bbasd constraint qu + - The peak set Yu, containing sam- ples exceeding the safety boundary 9us is used to fit GPDs. The extreme set Zc. containing samples exceeding the risk boundary 4μ + _ is used to compute EVT-based con- straints for reducing violations and calculate extreme priorities for exploiting extreme samples.
s, An a].
The advantage function measures the advantage of action a over the mean value: (7, a) = (7 a) - (s).
The cost value function (s), cost action value function (Z, a) and cost advantage function (7, a) in CMDP can be obtained as MDP by replacing the reward r with the cost C. The expected discounted return JR(TT) == E 1 (st, a,)]. and the expected cumulative discount cost (T( ;= En o CCC(se, at)l, where T= r-Ss, Au, $1, 01,' .) is the trajectory under x.
In this section, we present the modeling of extreme samples using EVT and derive the constrained optimization objective based on extreme quantile. We then introduce an extreme prioritization mechanism to effectively capture information from extreme samples and enhance extreme value model- ing using off-policy samples to mitigate the high variance. Finally, we provide theoretical guarantees for EVO.
The CRI. aims to find an optimal policy by maximizing the expected discount return over the set of feasible policies Ie = { E II (c( S d):
arg (a) #Ell
(1)
stt. Jc( 5d
4.1. Extreme Quantile Constraint
To quantify the probability of constraint violations, play constrained quantile function that ensures the con- straint is satisfied at a specified quantile level:
3.2. Extreme Value Theory
Extreme Value Theory (EVT) is statistical method for analyzing the distributional properties of extreme events.
arg max.JR(T) FET
Theorem 3.1. X1,...,X, denote a sequence of IID random variables with cumulative distribution function F. which approaches the Generalized Pareto distribution (GPD) asymptotically. Denote the conditional excess distri- bution as Fx(x) = P(X - 5 x|xx > t), then:
(4)
s.t.
9μ <d d
where the quantile for the distribution of the cumulative cost C = Er 2'c under policy и is defined as:
qu inf 2resal
(5)
lim (x(x) - (xx) t0000
(2)
3

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
and je is the quantile level. When the quantile satisfies the constraint threshold qu S d, the probability P(C - d) 2 / according to the definition of the quantile.
Then we derive the quantile level in the GPD from Equation 8: P(Z S =) =) T- Denote the corresponding quantile in the GPD as q then the risk boundary quantile Gutz is expressed using the quantile of GPD:
Extreme samples in the cost distribution tails are crucial for reducing the probability of constraint violations, as they carry critical signals strongly correlated with task con- straints. To capture extreme samples, we propose an extreme quantile constraint that explicitly incorporates tail behavior:
H qutv qu + qu + 9
where the excess value+ 2 is asymptotically equal to - under the GPD. Furthermore, the relationship between the probability density function fc of C' and I' of GPD is:
Ou+v S d
(6)
We denote 4u in qu+ as the safety boundary, dividing the distribution into the body and the tail capture extreme samples. Ou-v denotes the risk boundary, integrating ex- treme values into the overall distribution. // denotes the ex- ploitation range in the tail, shown in Figure 2. The proposed quantile-based optimization objective minimizes constraint violations by optimizing the risk boundary, effectively ac- counting for extreme values in the distribution.
fc + +) = ()
(10)
Then we propose EVT-based optimization objective:
arg max xell
(1I)
s.t. 9u + 9 S
d
where the constraint term denotes:
Extreme samples in the tail distribution are characterized by low probabilities and limited sample scale, posing chal- lenges for accurate modeling. To address this, we leverage Extreme Value Theory to effectively capture extreme sam- ples and reduce variance.
inf Pr|C S 2| 2 zdJ
(12)
inf S 2/2 0) 2 1 s
In our work, the safety boundary 9N is determined by the expectation of the cumulative cost, reflecting the average behavior of the cost distribution. In the trust region, we give the surrogate optimization objective:
As stated in Theorem 3.1, samples exceeding a threshold follow a Generalized Pareto Distribution (GPD). To validate this, we empirically analyze cost samples from the envi- ronment, fitting the cumulative cost distribution to both GPD and a Gaussian distribution. The results in Figure lb demonstrate that the tail distribution better aligns with the GPD. By accurately capturing the tail behavior, the GPD enables more effective modeling of extreme samples, which are crucial for reducing constraint violations. Building on this insight, we propose EVT-based approach for tail distribution modeling.
Tk+1 = arg max Egnd" awm (A xElla
JO(TTK) + 1-7 Bend .~m [A (s, + t S d
((T[|TT) 5
(13)
where Hg is the policy set parameterized by 8, D ([Tm))
By [Dxz (T][<>]]))]]]]] DKL is the KL divergence and 8 >> is the trust region size. { E n. D(=[| I d} de- fines the trust region. We provide the detailed optimization methods to solve the objective 13 in Appendix A.
We first establish the relationship between EVT-based tail distribution modeling and the proposed quantile objective 6. Let z = qu+v 4u > 0 denote the excess value beyond the safety boundary 9u- used explicitly model the tail. The cumulative distribution function (CDF) of C is given by Fo(qu-u) P(C S )u+v) = P(C S qu + z). We decompose the distribution of C' into two parts: the body distribution below 91 and the tail distribution beyond qu:
As shown in Figure 2, during training, samples generated by the current policy are used compute the safety boundary qu- Samples where C' >> 4u- called peaks, collected to form the peak set yr.- The peaks are then used to fit the GPD, enabling the update of the risk boundary as qu+v
Fc (g = P(C S 9p +
4n + TH Samples exceeding the updated risk boundary
(7)
qu + g 1-"
=P(C S (/)) + P(C
qn S= qu)
are identified as extreme samples and utilized for further analysis and policy optimization.
Let Z = C' - qu represent the excess random variable over 9p- We use the GPD to model the tail distribution Z, where (z) denotes the CDF of the GPD. According to Theorem
The parameters of the GPD, (shape) and (scale), are estimated using maximum likelihood estimation:
3.1, the conditional probability P(C qn I =|C > qu) in Equation? 7 asymptotically follows the GPD Fn (z). Thus P(C < 9u+ + 2) can be expressed as:
N2
log C(E,o) = -N,, logo - (1+
1
Σ 1=1
log(1 + 5
P(C S qa + 2) = 1 + U u + (1- P(Z 2) (8)
(14)
4

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
where Y denotes the sample in the peak set, N2 is the number of peaks. The risk boundary can be obtained by:
where N is the total number of samples in the buffer. The extreme prioritization mechanism directly correlates sam- ple's replay frequency with its quantile levels in the reward and cost GPDs, emphasizing high- -nghrinformtiion extreme sam- ples during experience replay.
H. T 4u +
= 9r + o &
(1-
N2 1ー- -1))
(15)
where n is the total number of samples.
4.3. Importance Resample for Extreme Samples
4.2. Extreme Prioritization Resampling
The GPD is fitted using a sparse extreme set in EVT. How- ever, limited samples can lead to high variance, as individual extreme values exert a significant influence.
Extreme samples in CRL are essential for guiding policy optimization. Extreme reward samples provide valuable insights for achieving task goals, while extreme cost sam- ples offer critical information for satisfying task constraints. However, their low occurrence probability results in a rarity of these information-rich samples. To learn an optimal safe policy, is essential to effectively exploit the scarce extreme samples. We model extreme reward and cost samples based on EVT and propose an extreme prioritization mechanism to enhance exploitation of learning signals within extreme samples during experience replay.
To address this issue, we introduce an off- aff-policy importance resampling approach augment extreme samples and re- duce the variance of the GPD. By leveraging stored samples generated under previous policies no. apply importance sampling to correct for the distributional shifts between sam- ples generated by earlier policies and current policy æ. Then the resampled values previous samples Aş and C' can be denoted as:
A'2 <(ass) To(a 8)
AR. C' <(|ss) To(ass)
As discussed in Section 4.1, extreme cost samples are de- tected using EVT when their cost value exceeds the risk boundary. These samples form the extreme cost set, denoted as Zc {C > 4u + - }.
c
(18)
By augmenting the scale of extreme samples through off- policy importance resampling, the variance of the GPD is effectively reduced, enhancing its stability, as supported by Theorem 4.3.
Similarly, EVT is applied to detect extreme reward samples in the replay buffer. The quantile of. is computed based on the expectation reward value AR- Samples exceeding qfe termed reward peaks, are collected into the reward peak {AR(s, a) > T.. The reward peak set is then used to fit by GPD, which determines the reward boundary ff.r qf. + q -_=- following the process outlined in Equation 14 and 15. Samples with reward values exceeding the reward boundary have high return to achieve the task goal, termed extreme reward samples. Denote the extreme reward set as ZR: {AR % + q H.r
The algorithm process presented in Algorithm 1.
set: Y
4.4. Theoretical Analysis
We provide an upper bound on the expected constraint vio- lation during policy updates in EVO.
Theorem 4.1 (Constraint violation upper bound), Suppose
Tk+1, Th are related by quantile- based constraint objective 11, the upper bound on constraint of Tk41 is:
To effectively exploit the learning signals in extreme sam- ples, we assign higher priorities to these samples based on their quantile levels under the GPD. Samples with larger quantile levels typically indicate lower probabilities and higher value, and thus are given higher prioritization. The prioritization score p is composed of two components: re- ward prioritization and cost prioritization. Reward priori- tization is determined by the quantile level W, of a sample under the reward GPD, while cost prioritization is based on the quantile level we under the cost GPD:
Jo( S _ I-
(TK+1)
+ 1 1-7
E B
Drv (T [s]]
1 1-7
(19)
where EC Je+1 := max, Ea [AZ (s, a)]l TV-divergence
Drv [T<+2l|<Ta[s] = (1/2) E. ( - a (a|s))
The zero-violation exploitation range vo in GPD satisfies:
Nu (1
E
vp
(7)
p Wy We
(16)
27cc 1 1-7
Ju+1
Eg awitt (aNTEEL
(
[+]]]]]] +1
The probability of a sample s; being replayed with extreme prioritization is:
(20)
which guarantees the expectation of updated policy T strictly satisfies the constraints. where N₂ is the number of peaks and the total number of samples.
p(s+)
P(si)
(17)
Σ k== ((())
5

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Algorithm 1 EVO: Extreme Value policy Optimization
Theorem 4.2 indicates that EVO has a smaller constraint violation probability than the expectation-based objective when μr corresponds the quantile level of the constraint expectation.
Input: Initialize policy network no, value networks Vi:
Vg.
Output: The optimal policy parameter A.
1: for epoch k = 0, 1,2,... do
Theorem 4.3 (Variance of EVO). Let q denote the quan- tile in for random samples of size N from a population with inverse distribution function q = (T If the cumulative distribution function Fh is continuous and has continuous and positive density fH at " then the quantile estimator _ to the true quantile value lr2 con- verges distribution to an Gaussian random vector with mean and variance Qt:
2: Sample trajectories under the current policy Tex Process the trajectories to C-returss, and calculate advantage functions with R and V
4: for K iterations do
5: Update value networks V and Vg.
6: end for
7: Perform importance resampling to adjust off-policy
samples using Eq. 18.
8: Fit the cost GPD with off-policy samples according Eq. 14.
VN - -
(23)
9: Compute the risk boundary qu t using the fitted cost GPD (Eq. 11).
The variance of EVO to estimate the quantile
"
I-a
10:
Fit the reward GPD with off-policy samples accord- ing to Eq. 14.
v リー N(1 - 1
(24)
11:
Compute the extreme prioritization (Eq. 17) from the reward and cost GPDs
where μ denotes the safety boundary quantile, v denotes the exploitation range, and N denotes the number of samples from the distribution that are used to estimate the quantile values. EVO has lower variance than that in quantile regression methods:
12:
Update policy by optimizing the new objective (Eq.
I1) using on-policy samples.
13: end for
14: Return: Policy parameters 0x+1-
a2 +u++D([1 -1
proof is provided in Appendix B.I.
(25)
N N88(4ptv)
Theorem 4.1 demonstrates that EVO has a tighter constraint violation upper bound compared to the expectation optimiza- tion objective in CPO (Achiam al., 2017). Theoretically, with the zero-violation exploitation range vo. EVO guaran- tees that the expectation of the updated policy Th+1 strictly satisfies the constraints.
where fC denotes the probability density function.
A proof is provided in Appendix B.3.
Note that Theorem 4.3 indicates the variance of EVO is negatively correlated with the number of samples used to estimate the quantile, which demonstrates that extending the extreme sample scale through off-policy importance resampling effectively reduces the variance of EVO.
Additionally, we analyze the probability of constraint viola- tions under the quantile-based objective in EVO.
Theorem 4.2 (Constraint violation probability). When the safety boundary qu is determined by the expectation of cu- mulative cost C, the constraint violation probability of EVO with the zero-violation exploitation range vo satisfies:
5. Experiment
We evaluate the performance of EVO through compar- ison with multiple baselines, including the expectation- based constraint method CPO (Achiam et al., 2017), the probability-based constraint method WCSAC (Yang et al., 2021), and state augmentation approaches Saute (Sootla et al., 2022a) and Simmer (Sootla et al., 2022b), which em- phasize zero-constraint violation. Results for additional baselines provided in Appendix C.1.1. The experi- ments address the following questions: 1) Does EVO reduce constraint violations while maintaining policy performance compared to baselines? 2) Can the modeling and exploita- tion of extreme samples within EVO help mitigate variance and enhance policy learning?
P(C > d) (J(E +1))+ 1))
(21)
where:
I 5 JC((T() +
1 - Bxndre ard (A (s, a])
+1
E E a(1-7) anttenl 1-7
27.c. Th12
Drv ( [
(22)
EVO has a lower constraint violation probability than the expectation-basel CRI. by a margin of Vo-
proof provided in Appendix B.2.
6

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
SukttCarrhdet--
SulayCarrGaatt00
SukttPannCrde1-v0
Sa
02
02
10
0.2
0.4
10
Saleyy Circlttvo
CalatyCa
SattryyannnCCrrlle.... 1-00
Saltoyaanttoad 1-40
10
100
139
20
107
147
EVO
CPO
WCSAC
Saute
Simmer
Figure 3: Comparison of EVO to baselines on Safety Gym. The x-axis is the total number of training steps, the y-axis is the average return constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
Environment. The environments in our experiments con- sist of Safety Gymnasium and MuJoCo (Ji et al., 2024). The Safety Gymnasium tasks simulate a simplified version of autonomous driving, where the robot is required to reach a goal while avoiding obstacles. The Safety MuJoCo tasks focus on robot motion control, with agents being rewarded for maintaining straight path while adhering to speed limit to ensure safety and stability. Details are provided in Appendix C.3.
contrast, EVO achieves the best overall performance while strictly adhering to the constraint threshold. In the Safety MuJoCo tasks, as illustrated in Figure 4, when the initial pol- icy lies within the feasible domain, EVO maintains zero con- straint violations throughout training, providing strong em- pirical validation of the zero-violation guarantee in Theorem 4.1. Despite using a probabilistic constraint optimization objective, WCSAC presents significant constraint violations in SaretyCarCirelelvvO and SafetyHallChhetabb Velocity- 1, because fails to account for the substantial impact of extreme samples. We design an evaluation metric to com- prehensively measure policy performance and constraint satisfaction during training, results in Appendix C.2 show that EVO outperforms across multiple environments.
Experiment Setting, All experiments followed uniform conditions ensure fairness and reproducibility, with a total of 10? training time steps and a maximum trajectory length of 1000 steps. To reduce randomness, 6 random seeds were used for each method, and the results are presented as mean and variance. The parameter settings are in Appendix C.4.
Ablation Study. Ablation experiments are designed to validate the effectiveness of each component in EVO. (1) Ablating EVT-based Constraint Objective: We replace the EVT-based constraint objective with a constant quantile con- straint objective. The results in Figure 5a and 5b show that the constant quantile objective achieves constraint satisfac- tion by diminishing policy performance. Conversely, the EVT- based constraint objective in EVO, derived from the ex- treme samples in the tail, captures critical information about the task goal. This enables EVO to achieve a more favorable trade-off between policy performance and constraint satis- faction. (2) Ablating Extreme Prioritization: The results in Figure 5a and 5b show degradation in performance when extreme prioritization during experience replay is removed. This indicates that prioritizing extreme samples enhances the optimization process by fully exploiting the task-related learning signals embedded in these low-probability, high- impact samples. (3) Ablating Off-policy Importance Resam- pling for GPD Fitting: As shown Figure 6, eliminating
Performance and Constraint. In the navigation tasks of Safety Gymnasium, the initial policy is infeasible, pre- senting significant challenges due multiple obstacles and complex decision boundaries. Figure 3 shows the learning curves of EVO and baselines. The results indicate that EVO rapidly converges to the feasible domain and subsequently maintains zero constraint violations throughout training. This is attributed to its optimization objective, which lever- ages extreme constraints to effectively capture violation signals from extreme cost values, thereby significantly re- ducing the probability of constraint violations, as supported by Theorems 4.1 and 4.2. Furthermore, EVO achieves pol- icy performance comparable to CPO while outperforming other methods such as Saute and WCSAC that emphasize zero constraint violations. It is important to note that CPO violates the constraint threshold in SufetyPoim so a higher return than EVO does not indicate a better policy. In

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
SuieeySwimme
2000
1900
5000
1500
4000
1000
3000
3000
500
2000
1000
10
1x7
Survey Swimme Voocity-v
150
117
EVO Figure 4: Comparison of EVO to baselines on Safety MuloCo. The x-axis the total number of training steps, the y-axis the average return constraint. The solid line is the and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
CPO
WCSAC
Saute
Simmer
Limit Saineyhhhnnnne
Lintt SelesfoonnCCaeltr>
Adamnn FV E-Tipuniiee Pun Addingg
Evicceell
tvD
Add Tottiiatnn
0.2
08
1.0
(a) Return Ablation
(b) Constraint of Ablation
(c) Return of Sensitivity
(d) Constraint Sensitivity
Figure 5: (a) Ablation Study, including ablating EVT--based Constraint Objectives and ablating extreme prioritization.
(d) Cost Limit Sensitivity.
Standard Deviation of Cost
Standard Deviation of Quantile
to inaccurate tail modeling.
10.517
27.
Cost Limit Sensitivity. We further evaluate the robustness of EVO across various threshold levels. Figure 5e and 5d indicate that EVO effectively satisfies different cost limits while maintaining policy performance. The ability to learn from extreme values enables EVO to adapt to diverse policy preferences. In eecurity-orientee settings with limit 0, cost extremes in EVO provide learning signals related to task constraints, and in performance-oriene settings with limit 35, reward extremes in EVO offer significant reward signals about the task goals.
Standard Deviation
Standard Deviation
7,370
15.662
13
E50 Abletir poltc samples
Ablating off policy semples
Figure 6: Ablating off off-policy samples for GPD fitting.
The Fitting Accuracy of GPD. To evaluate the fitting accuracy of GPD in our experiments, we collected train- ing data across multiple environments and fitted both GPD and Gaussian distributions. Furthermore, employed the Baanogorov-Smmroon (KS) test to quantify the fit accuracy, where lower values indicate more accurate fits. As shown in Figure 7, the GPD presents robust fitting performance across various data distributions. provides a more accurate fit for extreme samples than the Gaussian and more effectively
off- off-policy samples leads to increase in the variance of the GPD, providing empirical validation of Theorem 4.3. This effect is due to the sparsity of extreme samples, particularly cost extremes, which become increasingly rare as training progresses. Without sample augmentation, GPD fitting be- comes overly sensitive to individual extreme values, leading
8

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
65 Statalic Geusdert 3.20A, STC: 0.032
K-5 Steistc Gaussinn 0:343, GPD: 3.04
0.0:0.04
<S
Statisiic Gruszion 0.125, GPO. 0.086
435
Gaussin GPD
Gaursann C21
Gaussion
GPD
Gauedaa C20
C25
A30
413
(a) SarrtyCarCirceel-v0
(b) 10
(c) Safety-oinCCirclelvOO
(d) Safety -vva
6-5 Stativlre: Gazawimr: 0.025, GPD: 0.038
K-S Statadic: Catzedarc D.319, GPD: 0.05
Statitic: Gaussian 0.037
65
Gasssian
Gausskar
Goussion
GPO
625
Gaursion
GPO
0400
(c) Sefett-A (f) SaaceyyHallhhccaavoocctttt (g) Safety (h) SafetySwinmerreloccttyy Figure 7: Distribution curves fitted by GPD and Gaussian on the training data in EVO. The fitting accuracy quantified using the Somiooorov-Smiroo test, as shown in the titles, where lower values indicate higher fitting accuracy. GPD presents
accurate fitting performance across various data distributions.
SuktpPannCrda--00
SukttPainEidde--00
75.2
30
17.5
913
25
38
04
03 10
52
0.4
2.2
100
02
04
0.6
0.5
02
D4
05
0.0
Figure 8: The training curves of EVO with diffferent sample sizes across multiple environments, indicating that EVO remains effective even with small sample sets.
capturing the tail behavior of the distribution. However, in special cases where the gap between extreme and normal values is subtle, the GPD may not provide a satisfactory fit due to the requirement in EVT that the threshold be sufficiently large. To address this issue, one can employ non- linear transformations to amplify the differences between extreme and normal values prior to GPD fitting, thereby enhancing the accuracy of tail modeling.
mance compared to baselines, indicating that it remains effective even with small sample sets.
6. Conclusion
In this paper, we propose cero-constraint violation method, Extreme Value policy Optimization (EVO) algorithm. EVO leverages Extreme Value Theory (EVT) to model extreme samples and exploits them to enhance safety policy opti- mization. To address the challenge of high variance in extreme samples, we we employ the GPD to explicitly model the tail extremes and augment the extreme samples with off offf-polccy samples. Recognizing the high information of extreme samples, we introduce an extreme prioritization mechanism to exploit them. We provide both theoretical and experimental evidence that EVO achieves zero con- straint violations during training, while maintaining high returns and low variance.
Sample Size of GPD Fitting. We conducted experi- ments by varying the sample size used for GPD fitting and evaluated the corresponding policy performance. As shown in Figure 8, increasing the sample size gener- ally lead to improved constraint satisfaction. Notably, in SefetyyPintCirclel-vee EVO maintains strong constraint sat- isfaction and performance even with as few as 10 samples. In RafittPPinnGoall-vo constraint satisfaction is consis- tently achieved once the sample size exceeds 20. In our experiments, with the same sample size of 20, EVO demon- strates superior constraint satisfaction and policy perfor-

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Acknowledgment
Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., Huang, W., Geng, Y., Liu, M., and Yang, Y. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25 (285):1-6, 2024.
This work was supported by NSF China under Grant No. T2421002, 62202299, 62020106005, 62061 146002.
Impact Statement
Jung, W., Cho. M., Park, J., and Sung, Y. Quantile con- strained reinforcement learning: A reinforcement learn- ing framework constraining outage probability. Ad- vances in Neural Information Processing Systems, 35: 6437-6449, 2022.
This paper aims to advance the field of Reinforcement Lear- ing by introducing the EVO algorithm, which significantly improves safety of policy during RL training. EVO en- ables safer and more efficient deployment of RI. in real- world domains such as large language models, autonomous driving, and robotics, thereby contributing to the broader progress of Machine Learning. There potential socictal consequences of our work, none which we feel must be specifically highlighted here.
Koenker, R. and Bassett Jr, G. Regression quantiles. Econo- metrica: journal of the Econometric Society, pp. 33-50, 1978.
Masys, A. J. Black swans to grey swans: revealing the uncertainty. Disaster Prevention and Management: An International Journal, 21(3):320-3333 -302000-33,100 2012.
References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on ma- chine learning, pp. 22-31, 2017.
NS, K. S., Wang, Y., Schram, M., Drgona, J., Halappanavar, M., Liu, F,, and Li, P. Extreme risk mitigation rein- forcement learning using extreme value theory. arXiv preprint arXiv:2308. /3011, 2023.
Bellemare, M. G., Dabney, W., and Munos, R. distribu- tional perspective on reinforcement learning. In Inter- national conference on machine learning, pp. 449-458.
Pickands III, J. Statistical inference using extreme order statistics. the Annals of Statistics, pp. 119-131, 1975.
PMLR, 2017.
Rockafellar, R. and Uryasev, Conditional value-at- risk for general loss distributions. Journal of banking & finance, 26(7):1443-14) 1443-171,, 2002.
Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 2015a.
Ding, D., Zhang, K., Basar, T., and Jovanovic, M. Natu- ral policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Informa- fion Processing Systems, 0188378-8390 2020.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized ad advantage estimation. arXiv preprint 106iv:150.002435. 2015b.
Ding, D., Wei, X., Yang. Z., Wang, Z., and Jovanovic, M. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics, pp. 3304-3312, 2021.
Sootla, A., Cowen-Rivers, A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., and Ammar, H. Sauté rl: Almost surely safe reinforcement learning using state augmenta- tion. In International Conference on Machine Learning, pp. 20423-20443, PMLR, 2022a.
Gao, S., Ding, J., Fu, L., Wang, X., and Zhou, C. Exterior penalty policy optimization with penalty metric network under constraints. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 4007-404- 2024.
Sootla, A., Nowen-Rivers, A. I., Wang, J., and Ammar, H. B. Effects of safety state augmentation on safe exploration. arXiv preprint arXiv:2206.0888 02675, 2022b.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International conference on machine learning, pp. 1861-1870, 2018.
Stooke. A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133 9143, 2020.
Hiraoka, T., Imagawa, T., Mori, T., Onishi, T., and Tsuruoka, Y. Learning robust options by conditional value at risk optimization. Advances in Neural Information Processing Systems, 32, 2019.
Tessler, C., Mankowitz, D. J., and Mannor, S. Re- ward constrained policy optimization. arXiv preprint arXiv: /074, 2018
10

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Urpí, N. A., Curi, S., and Krause, A. Risk-averse offline reinforcement learning. In International Conference Learning Representations, 2021.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-33) -3444, 2019.
Xu, J. Tian, Y., Ma, Rus, D., Sueda, S., and Matusik, W. defiictionguuider oulti-obeective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607-10616 PMLR, 2020.
Xu, T., Liang, Y., and Lan, G. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference Machine Learning, pp. 11480-11499 PMLR, 2021.
Yang. L., J.. Dai, J.. Zhang, L., Zhou, B., Li, P., Yang. Y., and Pan, G. Constrained update projection approach to safe policy optimization. arXiv preprint MxXiv:209.007089 2022.
Yang, Q. Simão, T. D., Tindemans, S. H., and Spaan, M. T. Wesac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 35, pp. 10639- 10646, 2021.
Yang. T.-Y., Rosca, J., Narasimhan, K., and Ramadge, Projection based constrained policy optimization. arXiv preprint arXiv:2010.03152 2020.
Ying, D., Zhang, Y., Ding. Y., Koppel, A., and Lavaei, Scalable primal-dual actor-critic method for safe multi- agent with general utilities. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Yu, C., Velu, A., Vinitsky, F.,, Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooper- ative multi-agent games. Advances Neural Information Processing Systems. 35:24611-24661 2022a.
Yu, H., Xu, W., and Zhang, H. Towards safe reinforce- ment learning with safety editor policy, arXiv preprint arXiv:2201. 12422222..222..... 2022b.
Zhang, Y., Vuong, Q., and Ross, K. First order constrained optimization in policy space. Advances in Neural Infor- mation Processing Systems, 53:15338-15349 2020.

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Notations
c
cost
cumulative cost
d
constraint threshold
Ar
reward advantage function
Ac
cost advantage function
q
quantile
quantile level
safety boundary
exploitation range in extrem samples
qutv
risk boundary
2
excess value beyond the safety boundary excess random variable
Z
F
cumulative distribution function
probability density function
quantile in GPD
S
shape parameter of GPD
scale parameter of GPD peaks
Zcc
extreme cost set trust region size
8
A. Policy Optimization in EVO
The CRL aims find an optimal policy by maximizing the expected discount return over the set of feasible policies le { 11 (o(T) s d):
arg maxJAa(T) TEll
(26)
s.t. Jc(x) $ d
The following equation briefly gives the performance difference of arbitrary two policies (Schulman et al., 2015a), which represents the expected return of another policy x' in terms of the advantage function over
1
Jn( Jn(TT) =
E Eangy' arw [A (s, a)]
(27)
1-n
This implies that iterative updates to the policy, (s() arg max, A7 a), lead to performance improvement until convergence to the optimal solution.
According to the performance difference equation (27), CRL defined as a constrained optimization problem:
12

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Mh+1 arg max Bxwd" anm [AR (s, a)] TEllo
(28)
s.t. JC( +
1-7 B and [AC (5, [s, S
where policy π E I1 is parameterized with parameters 0, and тą represents the current policy. In this paper, we propose the quantile-based objective,
arg (() Tell
(29)
s.. 4u
+ 9
H
< d
where the safety boundary quantile Qμ is determined by the expectation (T)()) Then the objective can be denoted as:
T = arg max Esnd" aawn [A (s, a)] #Ells
(30)
s... JT(TT) 1-
Bannd and [A (8,
H +
S d
T
We propose the optimization method based on CPO, EVO-CPO, to solve the optimization 29.
A.1. CPO-based EVO
The complex dependency of state visitation distribution d (s) on unknown policy makes Equation 29 difficult to optimize directly. To address this, this paper uses samples generated by the current policy n5 to approximate the original problem locally. We seek to solve the following optimization problem in the trust region:
Tk+1 = arg max Egwd aa [A (s, a)] sElls
s.t. JT(TT)) + 1-7 E (A (s, alx)) + T <d
(31)
D(T[ I 5
where Tl is the policy parameterized by parameter 0, D( Rg-ari (T|[(T| DRr. is the KI. divergence and 8 0 is the step size. The set {nr 1I (( s 5} is the trust region.
In the EVO-CPO method, we approximate the reward objective and cost constraints with first-order expansion and approximate the KI.-divergence constraint with second-order expansion. The local approximation to Equation 31 is:
Ox+1 arg max TT (0-0z)
s.t. c+ (( 0s) 50
(32)
(0- 0²)²² H 0x) 6
where g denotes the gradient of the reward objective in equation 31, gc denotes the gradient of quantile-based constraint Equation 31, e = (Tx) - d, H is the Hessian of the KL-divergence. When the constraint satisfied, the analytical solution can be obtained using the primal-dut method. The solution the primal problem is:
1
0" 00
+ A. " gcv")
(33)
where À and v are the Lagrangian multipliers of the Kl.-divereence term and the constraint term the Lagrangian function, respectively, X", v²* the solutions the dual problem:
1 = max (0, X u
(34)
13

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
q) + -
we, e
ifAc u
X
arg max A20
(35)
1 ( 16) .otherwise,
where q q= H g. 9 H gc. ( gc.
When the constraint is violated, we use the conjugate gradient method to decrease the constraint value:
28
0" 02 - -(Toc)" (1c))-H111
9-
(36)
9c
B. Theoretical Proof
B.1. Constraint Violation Upper Bound
Lemma B.1. (Achiam et al., 2017) For any policies T', and any cost function C 7 R, with c
max, |Eawn
[A ( The following bound holds:
1
2 I-7
Jc(=') Jc(x) 1-7 Ex a (A)(s,a)
Drv (x'(m())
(37)
where Drv(2'||T)(s) = (1/2) (alss) t(a|s)] is the total variational divergence between action distribution at s.
Theorem B.2 (Constraint violation upper bound). Suppose Th+1 Te are related by quantile-based constraint objective 11, the upper bound on constraint of T is:
C7C
5k+1
Jc( s - -h= (T&+1) + Bendtx, antTe+ 1-7 Drv(Fo+=ll/=))s)
(38)
1-7 where &c Th-1 max, Fuwoen (s, a)]l, (1V))l (1/2) (x+1(a)s)) - Tx (als)) The zero-violation exploitation range Va in GPD satisfies:
N₂ 1 vo
- 2xce B Dry (TT+I||TR)(S))
Bo- Bend aowte+
(39)
n
1-7
which guarantees the expectation of updated policy #k-1 strictly satisfies the constraints, where N₂. is the number of peaks and n is the total number of samples.
Proof. According to the Lemma B. in CPO (Achiam et al., 2017), for any policies n' and n, and any cost function, the following bound holds:
1
27
(c(=)))o((r) < Exwdr awa' (AZ(s, + Drv (rv(('(lm)
(40)
I->
Then the bounds of two neighboring policies Ty and Th41 in the policy update follows:
2 mki1
- 1 7 E E th aa [A7 (s, a) +
Dry (Fe+Il|TA\(s]
(41)
1 1-7
2
Je(TR+1) S (T(TTK) +
1- B awaa 147 (8,
Drv (TVtTT+I||
(42)
1-
Suppose T. and mh+1 are related by the optimization objective in EVO, which is:
9u n < d
(43)
14

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
where we the expectation (c( denote the quantile value qus which is approximated in the trust region as:
Jo(Tx) + 1-7 R a [AC (s, a)] - S
(44)
According to Equation 42 and Equation 44, then the upper bound on the constraint expectation of 7+++1
1
1
27 mx+1 DTV (TT [TT)s]]
JC(TT+1) (TT( + Bond aa [A (s, a)] + 1- Bend a0r
1-
1-7
(45)
27Ec th+1
<d
I1 quu
+
Be aank
7
The risk boundary in GPD d 2 O, which indicates that EVO has tighter constraint violation upper bound compared the expectation optimization objective in CPO, which is:
c
Tkil
(c(T))1) <d+
1 1-y E R anwait 1-7
Drv [TV214+1|
(46)
Theoretically, if value of d-= is greater than - Bandd aaanr the expectation of updated policy Th+1 strictly satisties the constraints.
2rec
Jh+1
DrV (TK+1 our method ensures that
a
1-y
The risk boundary in EVO is obtained by:
H We derive the zero-violation exploitation range un by making:
(
un ) -
(47)
&
N₂
27
uh+1
9 p
R a008111
Drv (Te41 [x]]
1 1-7
(48)
(1-
(1
vn N₂
Eand a
2
Drv ( [s]]]
Then we get the zero-violation exploitation range M/:
N₁ - ( 6 coweet 1-9 o01 7))
2 Th11
Dry (rv( -
(49)
B.2. Constraint Violation Probability
Theorem B.3 (Constraint violation probability). When the safety boundary 9u is determined by the expectation of cumulative cost, the constraint violation probability of EVO with the zero-violation exploitation range Vo satisfies:
P(C > d) (
(50)
where:
J 5
5 Jo(Tk) + 1- Exnav anm A (s, [s) +
1
(51)
27 J4-1
Bend a0oit4
D (T Tx)]ss]]]
0(1 -
15

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
EVO has lower constraint violation probability than the expectation-based CRL by a margin of Vo-
Proof. In this paper, the expectation Jc(x) of cumulative cost C to determine the safety boundary qu
qu Jc(T)
(52)
which is used to the GPD with:
Ex - +
5 0
(53)
0
Then we get the safety boundary quantile level ja:
1 (Jo(T)
(54)
where JC(T) is approximated in the local region as:
Jc( = J((TK) + 1-7
Exnd am [A (s, a]]
(55)
Then we get the zero-violation exploration range according to Theorem B.2:
(
5 Ba aawaes 1-7
T++l 27E6 (Tv(TK+||TTE)) +1
-2)
yo (1 - 1d) 1
a(1 7)
(56)
-
=
I -(1+ {JO(T)) (1+5))
(1-(01177) B
27EC
Th+1 DTV (TT++1 (Tk [+]] +
1-7
According to the definition fo the quantile EVO, the constraint violation probability EVO satisfies:
P(C> > d) <1 μ 10
g (1-(o11-)))
27cc (Tv(mA+21lm)))11111 -
=1 (1 (1-1)
E Be aowTe+1 7g)
1-y
-(1 11) (011-) (7)
E
27cc
Thil
Dr Drv(me+illTa)) +1)
B JoNTet
7)
1 1-
(Jo(T))
( Bond
(0/1- -
- Drr(TA+IITWW)s)) +1)
27
1
1-7
- (
c (o( +
& o(1 (1 Bond (A +1)
E
c7
Bondi aanten anttee
(I +1
1 1-7
(57)
To simplify the representation, denote:
J E Jo(Tk) + 1
Exnave and [A (8, [i)] +
1-
(58)
8 Band anwmes
27 :4-1
E = 0(1 -
DTV ( [x]]]]]]]]]]]]]]]
16

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Then the constraint violation probability under EVO is less:
P(C > (J(E 1))
(59)
The constraint optimization objective in expectation-base based CRL is:
9r Jc < d
(60)
Then the constraint violation probability under expectation-based CRL is:
P(C > d) <1
(61)
EVO has lower constraint violation probability than the expectation-based CRL by margin of vo:
1 1 (1 μr vo) M/u
(62)
B.3. Comparison of Variance with Quantile Regression
Lemma B.4. The asymptotic distribution theory (Koenker & Bassett Jr, 1978). Let {qw..... "ppx: } with < μ2 <... μM < 1, denote sequence of unique sample quantiles from random samples of size N from a population with inverse distribution function qu F (p) If FH is continuous and has continuous and positive density fH at qup i 1, M, then the quantile regression estimator qu the true quantile value d7, converges in distribution to Gaussian random vector with mean 0 and variance Qt:
VN(qu- - (1d) + (0,
(63)
The variance "( (PM) with typical element:
ps(1 - 11) Wij
(64)
Theorem B.5 (Variance of EVO). Let qHt I- denote the quantile in GPD for random samples of size N from population with inverse distribution function q T FH ( the cumulative distribution function Fh is continuous and has continuous and positive density fH at - then the quantile estimator "²=_, to the true quantile value qf 9i converges in distribution to an Gaussian random vector with mean O and variance Ω
VN (:mm (0,
(65)
The variance of EVO estimate the quantile
_ is: I-w
u(1 p リ-
Ω
(66)
N(1 - f
where μ denotes the safety boundary quantile, denotes the exploitation range, and N denotes the number of samples from the distribution that are used to estimate the quantile values. EVO has a lower variance than that in quantile regression methods:
S (1 + v)(1 1
(67)
Nff2gutv)
where Ic denotes the probability density function.
17

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Proof. According Lemma B.4, the variance of the estimator under the quantile level is:
7(1 7(1-T) T) (0,)
Ω
(68)
where N denotes the number of samples from the distribution that are used to estimate the quantile values.
The variance Ω2 of EVT to estimate the quantile
T-p is:
Q4 = (1-)(1- I-H
N
(69)
u(1 (1-
N(1 - (H
The variance 02 of quantile regression to estimate the quantile μe is:
2 (u + 1)(1 -
(70)
NSc(quto)
where fo denotes the density the overall cost distribution. According to Equation 10, fc is related to fH:
Sola, +) (1 (1- 1))((()))
(71)
folqutu) = (1 W)HH 1
(72)
Then get the variance Q2 of quantile regression is:
S2 lu+ui (u+ 11-
N (n v)(1 1
(73)
N(1 - W(1-M)3BH
Thus the variance of EVO is lower than that of quantile regression: Ω Z2..
C. Experiment
C.1. Additional Experiments
C. 1.1.1. ADDITIONAL BASELINES
conducted additional experiments across multiple tasks, comparing EVO with PPO-Lagrangian and RCPO (Tessler al., 2018). Lagrangann is primal-dual method that transforms constrained problems into unconstrained ones by introducing dual variables, and then optimizes the objective based the PPO algorithm. RCPO proposes multi-timescale Lagrangian method, which utilizes penalty signals to guide policy updates toward constraint satisfaction.
As shown in Figures 9 and 10, PPO-Lagrangian exhibits significant oscillations during training, and both PPO-Lagrangian and RCPO frequently violate constraints. In contrast, EVO consistently maintains constraint satisfaction across all tasks while achieving superior policy performance. SafetyCarCirclel1v.. although RCPO achieves slightly higher returns, exhibits substantial constraint violations, indicating that its policy is unsafe.
18

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
SuktPoanCrde1-0
Safeby Goal1-00
26
RCPO
RROL.g HCMO
PPO HOPO EVD
RF0.0
EYD
RGP
0.2
04
10
02
06
10
02
0.
10
02
06
10
Falttlyaln ++0
SuletyPoinSeal--e VV
Safety CirclotvvO
CarbaaCa
120
200
ACPO
100
ROR3
PvD
HOM
100
80
167
Figure 9: Comparison of EVO to PPO-Lagrangian and RCPO on Safety Gym. The x-axis is the total number of training steps, the y-axis is the average return or constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
6000
FFCCLag
15,H4
5000
EVO
X00
3000
300
2000
PAO
1000
1942
PvD
PvD
02
0.4
24
08
10
02
0.4
66
0.
0.5
1.0
02
2.4
0.
06
10
SatetyAnttVeectty
SafesySw Veltyv
100
nopo
nomo
nono
Rcnc PvD
APOLag
30
Figure 10:
Comparison of EVO to PPO-Lagrangian and RCPO on Safety MuJoCo. The x-axis is the total number of training
steps, the y-axis is the average return or constraint.
1.2. TRAINING TIME
To evaluate the computational overhead of EVO, we conducted experiments across different environments, measuring both total training time and the time spent on GPD fitting. As shown in Table EVO introduces only a limited increase in training time compared to CPO, and GPD fitting process is highly efficient, taking only few seconds in total. This indicates that EVO introduces minimal computational overhead.
C.2. Additional Results
In CRL algorithms, it is not sufficient compare rewards or constraint violation rates in isolation. To quantitatively compare the overall performance of different algorithms in terms of both policy performance and constraint satisfaction during training, we design the following evaluation metric:
Rutrage
ratio- Constraint iolatiionae +
(74)
where AverogeReture denotes the average return across all episodes during training, and ConstraintV Violation Rate represents the proportion of episodes which constraint violations occur. The constant Ç is introduced to ensure numerical
19

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Environment
CPO
EVO(10 samples)
EVO(20 samples)
EVO(50 samples)
EVO(100 samples)
SafetyPointCirelel
11h 23m 28s
11h 53m 19s (8s)
11h 52m 28s (8s)
11h 53m 31s (8s)
11h 55m 22s (9s)
SafetyPointGoal
11h 29m 42s
11h 58m 51s (8s)
11h 59m 47s (7s)
11h 58m 33s (8s)
11h 59 8s (9s)
Table 1:
EVO training time with different sample sizes compared to CPO. The time data includes the total training time for EVO, with the GPD fitting time shown in parentheses "O". EVO increases limited training time compared to CPO. GPD fitting takes only few seconds in total.
stability. The final ratio is then normalized. The corresponding results of ratio are shown in Table 2. The results show that EVO achieves the best performance across multiple tasks, effectively balancing policy performance and constraint satisfaction.
Environment
EVO (ours)
CPO
Saute
Simmer
WCSAC
SafetyCarGoal1
0.267
0.169
0.190
0.212
0.163
SafetyCarCircle1
0.237
0.147
0.229
0.228
0.156
SafetyPoiniGoal1
0.275
0.175
0.192
0.186
0.172
SafetyPointCirclel
0.236
0.186
0.177
0.191
0.211
SafetyAntVelocity
0.250
0.132
0.245
0.244
0.129
SaferyttallChhetaaWeeccctt
0.233
0.166
0.231
0.224
0.146
SuferyHuumnniidVelocctty
0.219
0.168
0.213
0.214
0.186
SaferySwinmerVelocityy
0.213
0.159
0.214
0.212
0.180
Table 2: The mean performance of ratio across mutiple environments, with the bolded data indicating the maximum ratio.
C.3. Environments
Our experimental environments consist of two types of tasks, navigation in Safety Gymnasium and velocity in Safety MuJoCo.
(a) PointGoal1
(b) CarGoall
(c) PointCirclel
(d) CarCirelel
Figure 1I: Environments in Safety Gym.
C. SAFETY GYM
Figure shows the environments in the Safety Gym. Safety Gym is the standard API for safe reinforcement learning developed by Open AL. The agent perceives the world through the sensors the robots and interacts with the environment via its actuators Safety Gym. In this work, we consider two agents, Point and Car, and two tasks, Goal and Circle.
20

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
(a) Ant
(b) HalfCheetah
(c) Swimmer
(d) Humanoid
Figure 12: Environments in Safety MuloCo.
The Point is simple robot constrained to two-dimensional plane. is equipped with two actuators, one for rotation and another for forward/backward movement. has small square in front of it, making easier to visually determine the orientation of the robot. The action space in Point consists of two dimensions ranging from -1 1, and the observation space consists of twelve dimensions ranging from negative infinity positive infinity.
The Car is more complex robot that moves in three-dimensional space and has two independently driven parallel wheels and freely rotating rear wheel. For this robot, both steering and forward/backwwrd movement require coordination between the two drive wheels. The action space of Car includes two dimensions with a range from -1 to 1, while the observation space consists of dimensions with range from negative infinity to positive infinity.
The agent is required to navigate towards the location of the goal. Upon successfully reaching the goal, the goal location randomly reset to new position while maintaining the remaining layout unchanged. The rewards in the task of Goal are composed of components: reward distance and reward goal. In terms reward distance, when the agent closer to the Goal it gets positive value of reward, and getting farther will cause negative reward. Regarding the reward goal, each time the agent successfully reaches the Goal, it receives positive reward value denoting the completion of the goal. In SafetyGoall, the Agent needs to navigate the Goal's location while circumventing Hazards. The environment consists of 8 Hazards positioned throughout the scene randomly.
Goal:
Agent is required to navigate around the center of the circle area while avoiding going outside the boundaries. The optimal path is along the outermost circumference of the circle, where the agent can maximize its speed. The faster the agent travels, the higher the reward accumulates. The episode automatically ends the duration exceeds 500 time steps. When out of the boundary, the agent gets an activated cost.
Circle:
C.3.2. SAFETY MuJoCo
The agent in Safety MuJoCo is provided by OpenAl Gym, and it is trained move along straight line while constrained with velocity limit. Figure 12 illustrates the different environments.
Velocity tasks are also an important class of tasks that apply RL to reality, requiring an agent to move as quickly as possible while adhering to velocity constraint. These tasks have significant implications various domains, including robotics, autonomous vehicles, and industrial automation.
If velocity of current step exceeds the threshold of velocity, then receive an scalar signal 1, otherwise
C.4. Hyperparameters
The exploitation range is adaptively adjusted based on the safety and performance of the current policy т. The objective for the adaptive is:
((v) (J(((T)) d)
(75)
The gradient with respect is:
VoL(v) (Jo(T) d)
(76)
21

---

Extreme Value Policy Optimization for Safe Reinforcement Learning
Thus is updated follows:
clippy -Viipy-aaa_(()..00
(77)
where a denotes the learning rate, and the clip operation ensures that 2 0. If Jc(T) > d, v increases, raising the quantile in GPD, which in turn reduces Jc and promotes safer policy. Conversely, if (TT) < d, decreases, lowering the quantile in GPD and encouraging exploration.
experiments are implemented in Pytorch 2.0.0 and CUDA 11.3 and performed on Ubuntu 20.04.2 LTS with a single GPU (GeForce RTX 3090). The hyperparameters are summarized Table 3.
Parameter
EVO
Saute
Simmer
CPO
WCSAC
hidden layers
2
2
hidden sizes
64
64
64
64
64
activation
tanh
tanh
tanh
tanh
tanh
actor learning rate
le-3
le
le-3
le-3
le le-3
critic learning rate
le-3
le-3
le-3
le-3
le
batch size
128
128
128
128
128
trust region bound
0.01
N/A
N/A
0.01
N/A
discount factor gamma
0.99
0.99
0.99
0.99
0.99
GAE gamma
0.95
0.95
0.95
0.95
0.95
normalization coefficient clip ratio
le
1e
1e-3
1e-3
le
N/A
0.2
0.2
N/A
0.2
conjugate gradient damping
0.1
N/A
N/A
0.1
N/A
initial lagrangian multiplier
N/A
1e-3
1e-3
N/A
1e-3
lambda learning rate
N/A
0.035
0.035
N/A
0.035
Table 3: Hyperparameters
22


Reviews:
[{"review_id": "69q3KL6W4t", "reviewer": "Reviewer_csFL", "summary": "The authors propose a novel approach, Extreme Value Policy Optimization (EVO), to handle rare but high-impact extreme value events by using the Extreme Value Theory (EVT). The EVO introduces an extreme quantile optimization objective and an extreme prioritization mechanism. Extensive experiments are conducted to demonstrate the effectiveness of EVO in terms of the reduction in constraint violations while maintaining the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer csFL.\n\n>The paper depends on the assumption that the extreme value events follow Generalized Pareto Distributions, which may not be true.\n\n**Response:**\nWe appreciate the reviewer’s insightful comments. \nAccording to extreme value theory, as extreme values increase, these samples asymptotically follow the GPD, which is proven to be effective for modeling extreme events in prior studies [1][2]. \n\nTo evaluate the fitting accuracy of GPD in our experiments, we collected training data across multiple environments and fitted both GPD and Gaussian distributions. Furthermore, we employed the Kolmogorov-Smirnov (KS) test to quantify the accuracy of the GPD and Gaussian fits, where lower values indicate more accurate fits. As shown in Figure 1 (<https://anonymous.4open.science/r/11409-C857/experiments1.pdf>), GPD presents accurate fitting performance across various data distributions. And GPD provides a more accurate fit for extreme samples than the Gaussian distribution and better captures the tail behavior of the distribution.  \n\nWe also acknowledge that in special cases, such as when the distinction between extreme and normal values is small, GPD may not provide a satisfactory fit. To address this issue, we can first apply methods like nonlinear transformations to amplify the difference between extreme and normal values, and then fit the data with GPD for improved accuracy.  \n\nWe thank the reviewer’s constructive feedback and view this as a promising direction for extending EVO to more practical applications in future research.\n\n\n>Could the authors provide more details about the sample size for EVO to fit the GPD?\n\n**Response:**\nWe appreciate the reviewer’s valuable comment regarding the sample size in EVO. \nWe conducted additional experiments varying the sample size used for GPD fitting and evaluated the corresponding policy performance. As shown in Figure 2 (<https://anonymous.4open.science/r/11409-C857/experiments1.pdf>), larger sample sizes generally lead to improved constraint satisfaction. Notably, in SafetyPointCircle1-v0, EVO maintains strong constraint satisfaction and performance even with limited 10 samples. In SafetyPointGoal1-v0, constraint satisfaction is consistently achieved once the sample size exceeds 20.\n\nIn our main experiments, with the same samples size of 20, EVO demonstrates superior constraint satisfaction and policy performance compared to other baselines, indicating that it remains effective even with relatively small sample sets.\n\n\n[1] NS, K. S., Wang, Y., Schram, M., Drgona, J., Halappanavar, M., Liu, F., and Li, P. Extreme risk mitigation in reinforcement learning using extreme value theory. arXiv preprint arXiv:2308.13011, 2023.\n\n[2] Siffer, A., Fouque, P.-A., Termier, A., and Largouet, C. Anomaly detection in streams with extreme value theory. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1067–1075, 2017.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in the paper are aligned with the target problem.", "theoretical_claims": "The proofs for theoretical claims are correct.", "experimental_designs_or_analyses": "The experimental designs are sound and valid.", "relation_to_broader_scientific_literature": "The paper addresses the limitation of expectation-based constrained reinforcement learning (Achiam et al 2017). EVO makes the improvement by explicitly modeling extreme events by EVT.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength: The paper uses EVT to constrained RL to capture rare but high-impact extreme value events that previous methods overlook.\nWeakness: The paper depends on the assumption that the extreme value events follow Generalized Pareto Distributions, which may not be true.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Could the authors provide more details about the sample size for EVO to fit the GPD?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j60Qt3L03b", "reviewer": "Reviewer_bFMW", "summary": "This paper presents the Extreme Value policy Optimization (EVO) algorithm for safe reinforcement learning. It integrates Extreme Value Theory (EVT) to model and utilize extreme samples. EVO introduces an extreme quantile constraint and an extreme prioritization mechanism. Theoretically, it has a lower constraint violation probability and variance than baselines. Experiments in Safety Gymnasium and MuJoCo show that EVO reduces constraint violations while maintaining competitive policy performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer bFMW.\n\n>In lines 210-211, the conditional probability does not follow the GPD. According to theorem 3.1, it only follows GPD as $q_\\mu \\to \\infty$. The following argument should establish in approximately way.\n\n**Response:**\nWe appreciate the reviewer’s insightful comment and agree with the reviewer’s perspective. According to extreme value theory, extreme value samples asymptotically follow GPD. Therefore, we use the asymptotic equality symbol to reflect this relationship, as shown in Equations (8) and (9) in our paper.\n\nEmpirically, our experiments show that GPD provides a good fit for modeling extreme value samples. To validate this, we collected training data from multiple environments and fitted both GPD and Gaussian distributions. Furthermore, we employed the Kolmogorov-Smirnov test to quantify the fitting accuracy, where lower scores indicate better fit. As shown in Figure 1 (<https://anonymous.4open.science/r/11409-C857/experiments2.pdf>), GPD presents accurate fitting performance across various data distributions, and it consistently outperforms the Gaussian distribution in capturing tail behavior.\n\nWe also acknowledge that in special cases, such as when the difference between extreme and normal values is small, GPD may not accurately capture the extremes. To address this issue, nonlinear transformations or similar methods can be applied to amplify the difference between extreme and normal values, and then fit the data with GPD for improved precision.\n\n\n>Why (8) holds? It seems that $Z \\le z \\iff C – q_{\\mu} \\le q_{\\mu + \\nu} – q_{\\mu} \\iff C \\le q_{\\mu + \\nu}$.\n\n**Response:**\nWe appreciate the reviewer’s valuable comment.\nIn this paper, the $z$ denotes the excess value beyond $q_\\mu$, which contains the condition $z = q_{\\mu + \\nu} - q_\\nu > 0$, as shown in Eq (12) in the paper. So $P(Z \\le z) = P(C-q_\\mu \\le z|z>0) = P(C-q_\\mu \\le z| z>0)$, and $P(Z\\le z) \\neq P(C \\le q_{\\mu + \\nu})$ .\n\nIn this paper, we use the GPD $P(Z)$ to separately model the portion $Z$ of the overall distribution $C$ that exceeds $q_\\mu$. According to EVT, $P(C - q_\\mu \\le z|C>q_\\mu)$ asymptotically follows GPD, thus $P(C - q_\\mu \\le z|C>q_\\mu) \\backsimeq  P(Z \\le z)$.\n\nAccording to the Eq. (7) in the paper:\n\\begin{equation}\n\\begin{aligned}\n    F_C(q_{\\mu+\\nu}) = P(C \\le q_\\mu + z) = P(C \\le q_\\mu) + P(C >q_\\mu) P(C - q_\\mu \\le z|C>q_\\mu) \n\\end{aligned}\n\\end{equation}\n\nThen we can get Eq (8):\n\\begin{equation}\n\\begin{aligned}\n    F_C(q_{\\mu+\\nu}) = P(C \\le q_\\mu + z) = \\mu + \\nu \\backsimeq \\mu + (1-\\mu)P(Z\\le z) \n\\end{aligned}\n\\end{equation}\n\nWe appreciate the reviewer’s valuable comment for enhancing the clarity of our paper.\n\n\n>In Theorem 4.1, I cannot understand this sentence:\"$\\pi_{k+1}, \\pi_{k}$ are related by quantile-based constraint objective 11.\"\n\n**Response:**\nWe appreciate the reviewer’s valuable comment. \nThis means that policy $\\pi_{k+1}$ is obtained by optimizing policy $\\pi_k$ according to objective function (11). Here, we follow the notation and description adopted in constrained RL works such as CPO and PCPO.\n\n\n>The metrics investigated in section 4.4 should be defined formally and discussed more. Now It is confused that why these theorems demonstrate the advantage of EVO.\n\n**Response:**\nWe appreciate the reviewer’s valuable comments. \nTheorem 4.1 is about the constraint expectation, demonstrating that the updated policy in EVO has overall lower constraints compared to expectation-based CRL methods, which is empirically validated in multiple environments in Figures 3 and 4 of the paper.\n\nTheorem 4.2 focuses on the constraint violation probability, showing that EVO has a lower constraint violation probability during training compared to expectation-based CRL methods, as validated in Figures 3 and 4 in the paper across multiple environments.\n\nTheorem 4.3 is regarding the variance of extreme value estimation, proving that EVO provides more stable extreme value estimates compared to quantile regression methods, which is verified by the results in Figure 6 of the paper.\n\nWe hope these discussions help clarify the theorems in Section 4.4.\n\n\n\n> In the right column of line 399, there is a typo in \"Figure 5a and 5a\".\n\n**Response:**\nWe greatly appreciate the reviewer’s careful review and will correct this typo in the revised manuscript.", "claims_and_evidence": "Most of the claims in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in the paper are consistent with the research problems.", "theoretical_claims": "I check the theoretical claims in the main text. I have some confusions with the theoretical parts:\n\n- In lines 210-211, the conditional probability does not follow the GPD. According to theorem 3.1, it only follows GPD as $q_\\mu \\to \\infty$. The following argument should establish in approximately way.\n- Why (8) holds? It seems that $Z \\le z \\iff C – q_{\\mu} \\le q_{\\mu + \\nu} – q_{\\mu} \\iff C \\le q_{\\mu + \\nu}$.\n- In Theorem 4.1, I cannot understand this sentence: “$\\pi_{k+1}, \\pi_{k}$ are related by quantile-based constraint objective 11 .”\n- The metrics investigated in section 4.4 should be defined formally and discussed more. Now It is confused that why these theorems demonstrate the advantage of EVO.", "experimental_designs_or_analyses": "The paper is reasonable and effective in experimental design and analysis", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "See other parts.", "other_comments_or_suggestions": "-\tIn the right column of line 399, there is a typo in“Figure 5a and 5a”.", "questions_for_authors": "See other parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTRdPVzUqe", "reviewer": "Reviewer_hnao", "summary": "The paper proposes Extreme Value Policy Optimization (EVO), a novel algorithm for safe reinforcement learning (RL) that addresses rare but high-impact extreme events in constrained RL (CRL). Traditional CRL methods optimize expected cumulative costs (e.g., $J_C(\\pi) \\leq d$), which overlook tail risks (e.g., \"black swan\" events). EVO integrates Extreme Value Theory (EVT) to model tail distributions of costs/rewards using Generalized Pareto Distributions (GPDs).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer hnao.\n\n>Comments regarding the accuracy of GPD fitting and the bias in GPD parameter estimation.\n\n**Response:**\nWe appreciate the reviewer’s insightful comments and agree that parameter estimation bias exists. In our work, we use maximum likelihood estimation for GPD parameters, which inevitably introduces bias. This bias is inherent in statistical distribution fitting and not unique to our method.\n\nTo assess the impact of bias, we saved samples during training and fitted both GPD and Gaussian distributions. Using the Kolmogorov–Smirnov test to measure fitting accuracy (lower values indicate better fit), we found that GPD consistently provides accurate fits across various data distributions despite with estimation bias, as shown in Figure 1 (<https://anonymous.4open.science/r/11409-C857/experiments3.pdf>).\n\n>Comparisons to non-EVT quantile methods are missing in ablation studies.\n\n**Response:**\nWe conducted an ablation study on the EVT-based quantile, as shown by the green curves in Figure 5(a)(b) in the paper. The results show that removing it reduces policy performance, demonstrating the advantage of explicitly modeling extremes in EVO.\n\n>Questions regarding the selection and update of $q_\\mu$.\n\n**Response:**\nIn this paper, the quantile $q_\\mu$ represents the safety boundary reflecting the overall constraint distribution. Rather than assuming $q_\\mu$ equals the expectation, we explicitly set it to the expected constraint value under the current policy. The corresponding quantile level $\\mu$ is then derived from the constraint distribution at each update.\n\n>Figure 6  lacks metrics like convergence speed.\n\n**Response:**\nWe provide the learning curves corresponding to Figure 6, with off-policy importance resampling ablated. As shown in Figure 2 (<https://anonymous.4open.science/r/11409-C857/experiments3.pdf>), constraint satisfaction in EVO converges after $1 \\times 10^6$ steps. \n\n\n>Omitting sensitivity analysis for $\\nu$ adaptation.\n\n**Response:**\nWe constructed experiments to evaluate the sensitivity of different $\\nu$, as shown in Figure 3 (<https://anonymous.4open.science/r/11409-C857/experiments3.pdf>). The results show that  EVO is robust to the initial choice of $\\nu$, as it is adaptively updated during training based on current policy performance, as shown in Appendix C.4.\n\n\n>Connections to distributional RL are under-explored.\n\n**Response:**\nWe will include a more detailed discussion on distributional RL in related work.\n\nWhile distributional RL models the full return distribution instead of the expectation, it does not address constraints. WCSAC expands distributional RL into constrained RL by introducing a distributional safety critic based on CVaR.\n\n>EVT may fail to non-i.i.d. extremes.\n\n**Response:**\nFor non-i.i.d. data, methods such as Block Maxima or clustering modeling can be applied firstly to make the extreme samples approximately i.i.d., thereby enabling the effective application of EVT.\n\n\n> Training time is not quantified.\n\n**Response:**\nWe conducted experiments across different environments, measuring both total training time and the time spent on GPD fitting. As shown in Table 1 (<https://anonymous.4open.science/r/11409-C857/experiments3.pdf>), EVO increases limited training time compared to CPO, indicating minimal computational overhead.\n\nWe also evaluate the impact of sample size and found that the additional time remains minimal as the sample size increases. The GPD fitting is highly efficient, taking only a few seconds in total, with negligible overhead even at larger sample sizes.\n\n\n>How ensuring policies $\\pi_{k+1}$ and $\\pi_k$ adhere to the quantile-based objective with off-policy data in Theorem 4.1?\n\n**Response:**\nFor off-policy data, we apply an importance resampling method, as shown in Equation (18) of the paper, to correct the distributional discrepancy between the current policy and the off-policy samples.\n\n\n>Why omit PPO-Lagrangian/RCPO?\n\n**Response:**\nWe conducted additional experiments across multiple tasks, comparing EVO with PPO-Lagrangian and RCPO. As shown in Figure 4 and Figure 5 (<https://anonymous.4open.science/r/11409-C857/experiments3.pdf>), PPO-Lagrangian exhibits significant oscillation during training, and both PPO-Lagrangian and RCPO frequently violate constraints. In contrast, EVO consistently maintains constraint satisfaction across all tasks while achieving superior policy performance. In SafetyCarCircle1-v0, although RCPO achieves slightly higher returns, it exhibits substantial constraint violations, indicating that its policy is unsafe.\n\n\n>Does EVO require more samples to collect extremes?\n\n**Response:**\nEVO does not require more samples to collect the extremes. In all experiments its using the same size of samples as the other baselines.", "claims_and_evidence": "Supported by experiments (e.g., Figure 3-4) and Theorem 4.1-4.3. However, comparisons to non-EVT quantile methods (e.g., VaR) are missing in ablation studies.\n\nTheorem 4.3 shows $\\Omega < \\Omega_2$ (EVO vs. quantile regression), but bias in GPD parameter estimation (e.g., $\\xi$, $\\sigma$) is not addressed.", "methods_and_evaluation_criteria": "EVT-based tail modeling is novel but sensitive to threshold selection for $q_\\mu$. The prioritization mechanism ($p = \\omega_r + \\omega_c$) is intuitive but assumes accurate GPD fits.\n\nTasks like SafetyCarCircle1 are standard but lack dynamic/adversarial scenarios to test robustness.", "theoretical_claims": "- **Theorem 4.3**: Variance reduction is valid but ignores bias in $\\xi$ estimation, which affects $q^{H}_{\\frac{\\nu}{1-\\mu}}$.", "experimental_designs_or_analyses": "Figure 6 shows variance reduction with resampling but lacks metrics like convergence speed.", "relation_to_broader_scientific_literature": "EVO advances CRL by integrating EVT for tail risks, contrasting expectation-based (CPO) and Gaussian-based (WCSAC) methods. Connections to distributional RL (e.g., Bellemare et al., 2017) are underexplored.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**:\n\n First to integrate EVT with CRL for tail risk mitigation.\n\nTheorems 4.1-4.3 and ablation studies validate contributions.\n\n Zero violations in experiments are critical for real-world safety.\n\n**Weaknesses**:\n\nEVT assumes i.i.d. extremes, which may fail in non-stationary RL.\n\nGPD fitting and prioritization add complexity; training time is not quantified.", "other_comments_or_suggestions": "No", "questions_for_authors": "1.  How does EVO ensure policies $\\pi_{k+1}$ and $\\pi_k$ adhere to the quantile-based objective with off-policy data in Theorem 4.1? \n2.  How is $q_\\mu$ initialized/updated? Could biased thresholds harm GPD fits?\n3.  Why omit PPO-Lagrangian/RCPO?\n4.  Does EVO require more samples to collect extremes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXYc01HpXr", "reviewer": "Reviewer_o4Co", "summary": "The paper introduces Extreme Value policy Optimization (EVO), an algorithm that enhances safety in reinforcement learning by leveraging Extreme Value Theory (EVT) to model and exploit extreme reward and cost samples. EVO features an extreme quantile constraint to capture tail risks and an extreme prioritization mechanism to amplify learning signals from rare but impactful samples. Theoretically, EVO guarantees strict constraint satisfaction at a zero-violation quantile level and exhibits lower constraint violation probability and variance compared to existing methods. Extensive experiments validate EVO's effectiveness in reducing constraint violations while maintaining strong policy performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer o4Co.\n\n\n>The integration of EVT and the proposed mechanisms adds complexity to the algorithm. This might make it challenging to implement and tune for practitioners, especially those without a strong background in EVT.\n\n**Response:**\nWe appreciate the reviewer’s insightful comment. \n\nTo facilitate rapid implementation and support ease of use, we have open-sourced our code along with parameter settings. Moreover, EVO introduces almost no additional hyperparameters, greatly simplifying the tuning process. \n\nWe thank the reviewer's insightful suggestions, and we plan to further refine and package EVO into a user-friendly library, enabling practitioners to apply our method via a simple function call without requiring deep knowledge of EVT.\n\n\n>What are the minimum sample requirements for EVO to effectively fit the GPD and achieve reliable performance? How does the algorithm handle environments where extreme samples are very scarce?\n\n**Response:**\nWe appreciate the reviewer’s valuable comments.  \n\nWe conducted experiments with varying sample sizes (10, 20, 50, and 100) to evaluate the minimum number of samples required for EVO.\nAs shown in Figure 1 (<https://anonymous.4open.science/r/11409-C857/experiments4.pdf>), larger sample sizes generally lead to improved constraint satisfaction. Specially, in SafetyPointCirclel-v0, EVO maintains strong constraint satisfaction and performance even with limited 10 samples. In SafetyPointGoal1-v0, constraint satisfaction is consistently achieved once sample size exceeds 20\nOverall, a minimum of 20 samples is sufficient for achieve reliable performance in EVO.\nAdditionally, in our main experiments, with the same sample size of 20, EVO demonstrates superior constraint satisfaction and policy performance compared to other baselines, indicating its effectiveness even with limited samples.\n\nTo address the challenge of very scarce extreme-value samples, EVO augments extreme samples using off-policy samples and apply importance resampling to correct the distribution shift. Furthermore, in special cases where extreme samples are nearly absent or difficult to collect, offline extreme samples can be provided in advance to ensure GPD fitting remains feasible.\n\n\n\n>Could the authors provide more details on the computational overhead associated with fitting the Generalized Pareto Distribution (GPD) and performing off-policy importance resampling? Specifically, how does the computational cost scale with the size of the dataset and the complexity of the environment?\n\n**Response:**\nWe appreciate the reviewer’s valuable comments. \n\nTo evaluate the computational overhead of EVO, we measured total training time for both EVO and CPO across multiple environments, along with the time spent on GPD fitting. As shown in Table 1 (<https://anonymous.4open.science/r/11409-C857/experiments4.pdf>), EVO only adds a limited training time compared to CPO, and GPD fitting is highly efficient, taking only a few seconds in total. This indicates that EVO introduces minimal computational overhead.\n\nWe also assessed the impact of dataset size by training EVO with varying sample sizes. The results show that increasing the sample size has negligible effect on overall training time.\n\nTo evaluate scalability of EVO with environmental complexity, we compared training times across environments of varying difficulty. While more complex environments naturally require more time, EVO’s additional overhead remains comparable to that of CPO.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the paper's proposed methods and evaluation criteria are well-suited for the problem and application.", "theoretical_claims": "I did not find any obvious issues in the proofs for theoretical claims in the paper.", "experimental_designs_or_analyses": "I did not find any obvious issues in the experimental designs or analyses in the paper.", "relation_to_broader_scientific_literature": "Traditional CRL methods, such as Constrained Policy Optimization (CPO) (Achiam et al., 2017), focus on optimizing policies to ensure that the expected cost remains below a predefined threshold. However, these methods often fail to account for the variability in the cost distribution, especially in the tail, leading to frequent constraint violations. EVO addresses this limitation by explicitly modeling the tail behavior using EVT.\n\nMethods like WCSAC (Yang et al., 2021) use probabilistic constraints and approximate the cost distribution with a Gaussian model to compute Conditional Value-at-Risk (CVaR). However, Gaussian approximations are inadequate for capturing the tail behavior accurately. EVO improves upon this by using the Generalized Pareto Distribution (GPD) to model the tail, providing a more accurate representation of extreme events.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\nThe introduction of the extreme quantile constraint based on the Generalized Pareto Distribution (GPD) is a unique contribution. This mechanism allows the algorithm to explicitly model and exploit extreme samples, which is crucial for reducing constraint violations.\n\nThe paper provides strong theoretical guarantees on constraint satisfaction, violation probability, and variance reduction. These guarantees are essential for building trust in RL systems and ensuring their reliability.\n\n\n\n### Weaknesses\n\nThe integration of EVT and the proposed mechanisms adds complexity to the algorithm. This might make it challenging to implement and tune for practitioners, especially those without a strong background in EVT. In contrast, WCSAC with a Gaussian approximation seems much easier to use.\n\nThe use of EVT relies on certain assumptions about the distribution of extreme events. While the paper demonstrates that these assumptions hold in the tested environments, they might not generalize to all real-world scenarios.\n\nThe paper assumes that sufficient extreme samples can be collected to fit the GPD accurately. In environments where extreme events are extremely rare, this might be a limiting factor.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1 What are the minimum sample requirements for EVO to effectively fit the GPD and achieve reliable performance? How does the algorithm handle environments where extreme samples are very scarce?\n\n2 Could the authors provide more details on the computational overhead associated with fitting the Generalized Pareto Distribution (GPD) and performing off-policy importance resampling? Specifically, how does the computational cost scale with the size of the dataset and the complexity of the environment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the Constrained Reinforcement Learning problem and proposes the Extreme Value policy Optimization (EVO) algorithm, providing both theoretical and empirical evidence to demonstrate its superiority over existing baselines.

All reviewers acknowledge the novelty of introducing Extreme Value Theory (EVT) into constrained RL to effectively capture rare events. Although some assumptions in the paper—such as those related to the distribution of rare events and the additional computational cost—were questioned, these concerns were thoroughly addressed during the rebuttal period.

The authors clarified the assumptions used in the paper, including the rationale for modeling extreme events with the Generalized Pareto Distribution (GPD). Additionally, they provided experimental details regarding the computational overhead involved in fitting the GPD. The authors also addressed issues related to the proof details. In summary, the authors’ responses adequately resolved the concerns raised by the reviewers.

In light of the reviewers’ assessments following the rebuttal, this paper is suitable for acceptance.
Decision: Accept (poster)