Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees
Authors: Dohyeong Kim
Taehyun Cho
Seungyub Han
Hojun Chung
Kyungjae Lee
Songhwai Oh
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe Reinforcement Learning
Risk Constraint
Spectral Risk Measure
Primary area: reinforcement_learning
TL;DR: We propose a safe RL algorithm with spectral risk constraints, which shows convergence to an optimum in tabular settings.

Abstract: The field of risk-constrained reinforcement learning (RCRL) has been developed to effectively reduce the likelihood of worst-case scenarios by explicitly handling risk-measure-based constraints.
However, the nonlinearity of risk measures makes it challenging to achieve convergence and optimality.
To overcome the difficulties posed by the nonlinearity, we propose a spectral risk measure-constrained RL algorithm, spectral-risk-constrained policy optimization (SRCPO), a bilevel optimization approach that utilizes the duality of spectral risk measures.
In the bilevel optimization structure, the outer problem involves optimizing dual variables derived from the risk measures, while the inner problem involves finding an optimal policy given these dual variables.
The proposed method, to the best of our knowledge, is the first to guarantee convergence to an optimum in the tabular setting.
Furthermore, the proposed method has been evaluated on continuous control tasks and showed the best performance among other RCRL algorithms satisfying the constraints.
Our code is available at https://github.com/rllab-snu/Spectral-Risk-Constrained-RL.

Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees
Dohyeong Kim! Taehyun Cho
Seungyub Han'
Hojun Chung! Kyungjae Lee
Songhwai Oh
Dep. of Electrical and Computer Engineering, Scoul National University Dep. of Statisties, Korea University
Abstract
The field of risk-constrained reinforcement learning (RCRL) has been developed effectively reduce the likelihood of worst-case scenarios by explicitly han- dling risk-meauure-bsee constraints. However, the nonlinearity of risk mea- sures makes it challenging to achieve convergence and optimality. To overcome the difficulties posed by the nonlinearity, we propose a spectral risk measure- constrained RL algorithm, seeecraalrisk-coosstriieeee policy optimization (SRCPO), a bilevel optimization approach that utilizes the duality of spectral risk mea- sures. In the bilevel optimization structure, the outer problem involves opti- mizing dual variables derived from the risk measures, while the inner problem involves finding an optimal policy given these dual variables. The proposed method, to the best of our knowledge, is the first to guarantee convergence an optimum the tabular setting. Furthermore, the proposed method has been evaluated on continuous control tasks and showed the best performance among other RCRL algorithms satisfying the constraints. Our code is available https:// // gi thub. com com/rllallla http/-lllbbblbbbbeeerr wwwlllasaaassseeeeeeeeeeeee Snu/Spectral- -ii---iiiis-ssaaaaeaaeee Ses---osstrrinnd-Ree.
Introduction
Safe reinforcement learning (Safe RL) has been extensively researched [2, 15, 25], particularly for its applications safety-critical areas such as robotic manipulation [3, 17] and legged robot locomotion [28]. In safe RL, constraints are often defined by the expectation of the discounted sum of costs [2, 25], where cost functions are defined to capture safety signals. These expectation-base constraints can leverage existing RI techniques since the value functions for costs follow the Bellman equation. However, they inadequately reduce the likelihood of significant costs, which can result in worst-case outcomes, because expectations do not capture information on the tail distribution. In real-world applications, avoiding such worst-case scenarios particularly crucial, as they can lead to irrecoverable damage.
In order avoid worst-case scenarios, various safe RL methods [10, 26) attempt to define constraints using risk measures, such as conditional value at risk (CVaR) [20] and its generalized form, spectral risk measures [1]. Yang et al. [26] and Zhang and Weng [30] have employed distributional critic to estimate the risk constraints and calculate policy gradients. Zhang et al. [32] and Chow et al. [10] have introduced auxiliary variables to estimate CVaR. These approaches have effectively prevented worst-case outcomes by implementing risk constraints. However, the nonlinearity of risk measures makes difficult to develop methods that ensure optimality. To the best of our knowledge, even for well-known risk measures such as CVaR, existing methods only guarantee local convergence [10, 32]. This highlights the need for method that ensures convergence to global optimum.
Corresponning authors: kyungjae_lees lee ao.ea..o...k... songhwai@snu.cem
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we introduce a spectral risk measure- consstriinn RL (RCRL) algorithm called spectral- risk-constrained policy optimization (SRCPO). Spectral risk measures, including CVaR, have dual form expressed as the expectation over a random variable [7]. Leveraging this duality, we propose a bilevel optimization framework designed alleviate the challenges caused by the nonlinearity of risk measures. This framework consists of two levels: the outer problem, which involves optimizing a dual variable that originates from the dual form of spectral risk measures, and the inner problem, which aims to find optimal policy for given dual variable. For the inner problem, we define novel risk value functions that exhibit linearity in performance difference and propose policy gradient method that ensures convergence an optimal policy in tabular settings. The outer problem is potentially non-convex, so gradient-based optimization techniques are not suitable for global convergence. Instead, we propose method to model and update distribution over the dual variable, which also ensures finding an optimal dual variable in tabular settings. As a result, to the best of our knowledge, we present the first algorithm that guarantees convergence an optimum for the RCRL problem. Moreover, the proposed method has been evaluated on continuous control tasks with both single and multiple constraints, demonstrating the highest reward performance among other RCRL algorithms satisfying constraints in all tasks. Our contributions are summarized as follows:
We propose a bilevel optimization framework for RCRL, effectivlly addressing the com- plexities caused by the nonlinearity of risk measures.
The proposed method demonstrates convergence and optimality in tabular settings.
The proposed method has been evaluated on continuous control tasks with both single and multiple constraints, consistently outperforming existing RCRL algorithms.
2
Related Work
RCRL algorithms can be classified according to how constraints are handled and how risks are estimated. There are two main approaches to handling constraints: the Lagrangian [26, 29, 32] and the primal [15, 30] approaches. The Lagrangian approach deals with dual problems by jointly updating dual variables and policies, which can be easily integrated into the existing RL framework. In contrast, the primal approach directly tackles the original problem by constructing subproblem according to the current policy and solving it. Subsequently, risk estimation can be categorized into three types. The first type [29, 32] uses an auxiliary variable to precisely estimate CVaR through its dual form. Another one [26, 30] approximates risks as the expectation of the state-wise wise risks calculated using distributional critic [8]. The third one [15] approximates CVaR by assuming that cost returns follow Gaussians. Despite these diverse methods, they only guarantee convergence to local optima. In the field of risk-sensitive RI. Bäuerle and Glauner [7] have proposed algorithm for spectral risk measures that ensures convergence to an optimum. They introduced a bilevel optimization approach, like method, but did not provide method for solving the outer problem. Furthermore, their approach to the inner problem is based, making it complicated to extend to RCRL, which typically requires a policy gradient approach.
3 Backgrounds
3.1 Constrained Markoy Decision Processes
A constrained Markov decision process (CMDP) is defined as (5, A, P. P, R, (C.). where S is a state space, is an action space, P is a transition model, p is an initial state distribution, y is discount factor, R S x A 5 > Rumax R.uxx] is a reward function, C, S x A S + 00, Cunax] is a cost function, and N is the number of cost functions. A policy т : S + P(A) is defined as mapping from the state space the action distribution. Then, the state distribution given is defined as d7 (s) == (1 - 7) Ei= 7 'P(ss s), where so ~ a T T-1ss)), and 8 ~ P(- az) for Vt.
The state-action return of defined as:
(3 a) = Σ R( ac s141) so = 8, ao = a, a ~ x(1ss)), s441 ~ P(- Vt.
t=0
The state return and the return are defined, respectively, as:
Y (s) a)) ~~ (((s), G = Yū{ss) s~pp
2

---

The returns for cost, Zč; Yo, and G2; are defined by replacing the reward with the ith cost. Then, a safe RL problem can defined as maximizing the reward return while keeping the cost returns below predefined thresholds to ensure safety.
3.2 Spectral Risk Measures
In order reduce the likelihood of encountering the worst-case scenario, it required to incorporate risk measures into constraints. Conditional value at risk (CVaR) is a widely used risk measure in finance [20], and it has been further developed into a spectral risk measure in [1], which provides more comprehensive framework. A spectral risk measure with spectrum or is defined as:
RS (x) = S Fx (u)o
where o an increasing function, o 2 So ( 1, and Fx is the cumulative density function (CDF) of the random variable X. By appropriately defining the function 0, various risk measures can be established, and examples are as follows:
/(1 a); CVaR., (u) 1l2a/(1 -a., a), Pow. : - u/
(1)
where [0, 1) represents risk level, and the measures become risk neutral when 0. Furthermore, it is known that the spectral risk measure has the following dual form [7]:
R. (X) = inf Elg(X)) + [. g" (o(u))
(2)
9
where g is an increasing convex function, and g" (y) sup,, -y- g(x) is the convex conjugate function of g. For example, CVaR is expressed as C'VaR. = inf El (X- ] + B, where (xx) 3)+/(1 - ar), and the integral part of the conjugate function becomes B. We define the following sub-risk measure corresponding to the function follows:
(x) = Blg(X)) + S'' 9" ("(ol)))du.
(3)
Then, (X) inf, (X)... Note that the integral part is independent of X and only serves as a constant value of risk. As result, the sub-risk measure can be expressed using the expectation, provides computational advantages over the original risk measure in many operations. We will utilize this advantage to show optimality of the proposed method.
3.3 Augmented State Spaces
As mentioned by Zhang et [32], risk-constrained RL problem is non-Markovian, so a history- conditioned policy is required to solve the problem. Also, Bastani et al. [6] showed that an optimal history-conditionn policy can be expressed as a Markov policy defined in newly augmented state space, which incorporates the discounted sum of costs as part of the state. To follow these results, define an augmented state as follows:
5 = (St, (e,, M be), where 6,o = 0, 6,,411 (Ci(s), s(+1) + t,,//, = 74. (4)
Then, the CMDP is modified as follows:
1++1 (st+1, (GG, + e,.t)//) ybe), where s4+1 ~ P(-lst, ar), C, Cy 04,
and the policy T( is defined on the augmented state space.
Proposed Method
The risk measure-constrainee RL (RCRL) problem is defined follows:
max E [ s.t. R., (Ge, S d, vi E 21,
(5)
where d, the threshold of the ith constraint. Due to the nonlinearity the risk measure, achieving an optimal policy through policy gradient techniques can be challenging. To address this issue,
3

---

Algorithm 1 Overview of Spectral-Risk-Consttiie
Policy Optimization (SRCPO)
Input: Spectrum function d; for i E [1,..,,]]
Paramcterize 9: o, defined in (3) using S, E B for t = to T do
If Parameterizaation Section 6. I.
for 3 = [3i...., BN) E BN do
Update TT, using the proposed policy gradient method. end for
// Inner problem. Section 5.2.
Calculate the target distribution of using na.r, and update & Outer problem, Section 6.2. end for
Output: Ta.r, where B ęr.
propose solving the RCRL problem by decomposing it into two separate problems. Using feasibility function F,² the RCRL problem can be rewritten follows:
max E Σ =1 F(Ro, (GZ,) - d:) = max E
inf F(RS a. (Ge,) d;)
sup {m
max E GR
F (Rg. (G di), (6)
(b)
where (a) is the inner problem, (b) is the outer problem, and {g: J.V is denoted as a dual variable, is known that if the cost functions are bounded and there exists a policy that satisfies the constraints of (5), there exists an optimal dual variable [7]. which transforms the supremum into the maximum in (6). Then, the RCRL problem
be solved by 1) finding optimal policies for various dual variables and 2) selecting the dual variable corresponding to the policy that achieves the maximum expected reward return while satisfying the constraints.
The inner problem is then a safe RL problem with sub-risk measure constraints. However, although the sub-risk measure is expressed using the expectation as in (3), the function g causes nonlinearity. which makes difficult to develop a policy gradient method. To address this issue, we propose novel risk value functions that show linearity in the performance difference between two policies. Through these risk value functions, we propose a policy update rule with convergence guarantees for the inner problem in Section 5.2.
The search space of the outer problem is function space, rendering it impractical. To address this challenge, we appropriately parameterize the function gi using parameter B3 B C. RM-1 in Section 1. Then, we can solve the outer problem through a brute-force approach by searching the parameter space. However, it is computationally prohibitive, requiring exponential time regarding to the number of constraints and the dimension of the parameter space. Consequently, it is necessary to develop efficient method for solving the outer problem. Given the difficulty of directly identifying an optimal 3 we instead model distribution & to estimate the probability of a given / being optimal. We then propose a method that ensures convergence to an optimal distribution E*, which deterministically samples an optimal 33*, in Section 6.2.
By integrating the proposed methods to solve the inner and outer problems, we finally introduce an RCRL algorithm called spectral-risk ristoii-riii--ctaacinneee policy optimization (SRCPO). We can obtain an optimal 3 by solving the outer problem and set of optimal policies { |3 E B) by solving the inner problem, ultimately achieving the optimal policy zg- of (5). An overview of SRCPO is presented in Algorithm 1, and a detailed pseudo-code of the proposed method is described in Algorithm 2. In the subsequent sections, we will provide details of the proposed methods to solve both the inner and the outer problems, respectively.
Inner Problem: Safe RL with Sub-Risk Measure
In this section, we propose policy gradient method to solve the inner problem, defined as: max B[GA] s.t. Rg. (G S de Vi,
(7)
FF(z) 0 if < else

---

where constraints consist of the sub-risk measures, defined in (3). In the remainder of this section, we first introduce risk value functions calculate the policy gradient of the sub-risk measures, propose policy update rule, and finally demonstrate its convergence to an optimum in tabular settings.
5.1 Risk Value Functions
In order to derive the policy gradient of the sub-risk measure, first define risk value functions. To this end, using the fact that 6i, 2j=o 7////// and = = where {ei,)]==1 and by is defined in the augmented state 8t, we expand the sub-risk measure of the ith cost as follows:
Roy (G // gi (o.((())) - [ g((2)foz, (z)dz -EEoup [ g:(2)) 91(2) (60)
E [- 9i(=)//))) Pe (5.) (z)dz = P.X.P
P.5.P [ (i
(z) (x))))
vt,
where fx is the probability density function (pdf) of a random variable X. To capture the value of the sub-risk measure, we can define risk value functions follows:
[ ((e))( (2))2 (īg(5.a) J (22)f (00)))))
V. (3)
Using the convexity of 9. we present the following lemma the bounds of the risk value functions: Lemma 5.1. Assuming that a function is differentiable, Vig (5) and Q (3, a) are bounded by (((e,b).g(edt (0mmax(1119))) and [QT.g(5 a) Vī,((3) (3)1 S 6C', where C' 1 g (1may ) and == (s, b).
The proof is provided in Appendix A. Lemma 5.1 means that the value of Q (3, Vig (5) is scaled by To eliminate this scale, we define a risk advantage function follows:
A (5, == (Q7.g (5, a) (s))/b.
Now, we introduce theorem on the difference in the sub-risk measure between two policies. Theorem 5.2. Given two policies, π and n', the difference in the sub-risk measure is:
Rg. (G Rgg (GO,) Bay'. . [AT.g, (8, a)] /(1 (1-
(8)
The proof is provided in Appendix A. Since Theorem 5.2 also holds for the optimal policy, beneficial for showing optimality, as done in policy gradient algorithms of traditional RL [4]. Additionally, to calculate the advantage function, we use a distributional critic instead of directly estimating the risk value functions. This process described in detail in Section 7.
5.2 Policy Update Rule
In this section, we derive the policy gradient of the sub-risk measure and introduce policy update rule to solve the inner problem. Before that, first parameterize the policy πe. where e O, and denote the objective function as B[G = JR(TT) = (R(() and the ith constraint function as Rg: (GT,) Jc, (T)) Jc. (0). Then, using (8) and the fact that (1 a)] 0, the policy gradient of the ith constraint function is derived as follows:
VæJc, (0) = Ex," mm [A (3, a) V log(To(a()))((11
Now, we propose policy update rule as follows:
0. 0 (FF (JR(0) - G. Ei de (0))
10-0.
if constraints satisfied,
O
B, + GFF( (GurJR(()- EN Je. (8) 0=0;
otherwise,
where Ge is a learning rate, F(8) is Fisher information matrix, At is Moore-Penrose inverse matrix of A, (Ar,1.... Ne,N, } C [0, Amax) are weights, and Xτ, 0 for E {ilJe, (0) < d, and Σ, A = I when the constraints are not satisfied. There are various possible strategies for determining At, and vn which makes the proposed method generalize many primal approach-basee safe RL algorithms [2, 15, 25]. Several options for determining X and Vt, as well our strategy, are described in Appendix B.
5

---

5.3 Convergence Analysis
We analyze the convergence of the proposed policy update rule in a tabular setting where both the augmented state space and action space are finite. To demonstrate convergence in this setting, we use the softmax policy parameterization [4] follows:
To(a == (xp0 a)/ , exp0(s,a') V(3,a) E x A.
Then, we can show that the policy converges an optimal policy if updated by the proposed method. Theorem 5.3. Assume that there exists feasible policy n such that Jc, (TT) S dN nor Vi, where ? > Then, if policy is updated by the proposed update rule with a learning rate Sa that that follows the Robbins-Mont condition [19], will converge an optimal policy of the inner problem.
The proof is provided in Appendix A.I, and the assumption the existence of a feasible policy is commonly used in convergence analysis in safe RL [5. 12, 15]. Through this result, we can also demonstrate convergence of existing primal approach-basee
safe RL methods, such as CPO [2], PCPO [27], and P3O [31], by identifying proper Aτ, and vi strategies for these methods. We also provide an analysis of the convergence rate of the proposed method in Appendix A.2, employing a similar approach used in [25].
6 Outer Problem: Dual Optimization for Spectral Risk Measure
(
In this section, we propose a method for solving the outer problem that can be performed concurrently with the pol-
15
icy update of the inner problem. To this end, since the
44
860)
domain of the outer problem is function space, first
f5
introduce procedure to parameterize the functions, al-
lowing the problem to be practically solved. Subsequently, we propose a method to find an optimal solution for the parameterized outer problem.
1/2
7/1
0 (1) (22 (4g 04
1=0g
Figure 1: Discretization of spectrum.
6.1 Parameterization
Searching the function space of g is challenging. To address this issue, we discretize the spectrum function illustrated in Figure since the discretization allows g to be expressed using finite number parameters. The discretized spectrum is then formulated as:
(u)) = Ō(u)= == 1n + Σ j=1 (11+1 77,) 1uzay
(10)
where 0 S 175 S 7/+1, 0 S a, S aj+1 S 1, and M is the number of discretizations. The proper values of fl and or can be achieved by minimizing the norm of the function space, expressed as:
[ (a(u) ( ō(u)ddu s.t. S. ō(u)du =
min (nz Lay (a
(11)
Now, we can show that the difference between the original risk measure and its discretized version bounded by the inverse of the number of discretizations.
Lemma 6.1 (Approximation Error). discretized version is:
The difference between the original risk measure and its
RR. - Rs (X)l Cmaxo (1)(11))))))))
The proof is provided in Appendix A. Lemma 6.1 means that as the number of discretizations increases, the risk measure converges to the original one. Note that the discretized version of the risk measure is also a spectral risk measure, indicating that the discretization process projects the original risk measure into specific class of spectral risk measures. Now, we introduce parameterization method as detailed in the following theorem:
Note that these safe RL algorithms are designed to handle expectation based constraints, so they are not suitable for RCRL.

---

Theorem 6.2. Let us parameterize the function 9 using parameter 3 E B C.R. M-M
as:
g2(z) == 11 + Σ M- (1/+1 - 71)(2 Blb))+
where 81]] S B + for {-..., M 2}. Then, the following is satisfied:
(xx i 3 R. (X), where R. Blgs(X)) + L.¹ (z(ō(
According Remark 2.7 in [7], the infimum function in (2) exists and can be expressed as an integral of the spectrum function if the reward and cost functions are bounded. By using this fact, Theorem 6.2 can be proved, and details are provided in Appendix Through this parameterization, the RCRL problem is now expressed as follows:
sup max BIGHT
Σ 1= F 8: (Ge,) d.),
(12)
(9.)
where & B RM-1 is parameter of the ith constraint. In the remainder of the paper, denote the constraint function R. (GZ,) as Jc, (T;; (),, where (B.) and the policy for mg.
6.2 Optimization
To obtain the supremum of in (12), brute-force search can be used, but it demands exponential time relative to the dimension of B. Alternatively, ,3 can be directly optimized via gradient descent; however, due to the difficulty in confirming the convexity of the outer problem, optimal convergence cannot be assured. To resolve this issue, we instead propose to find a distribution on B, called sampler & E PBB) that outputs the likelihood that a given ,3 is optimal.
For detailed descriptions of the sampler ç. the probability of sampling / is expressed as E(B) II - (B:)), where {(i] is the ith element of and 8 is G[i]. The sampling process is denoted by G~ &. where each component is sampled according to B; ~ Eli]. Implementation of this sampling process is similar to the stick-breaking process [23] due to the condition [,Dū] B, + 1] for
j E (1...., M 2}. Initially, [,[1] is sampled within [0. Cmaxx - [)] and subsequent values
B + 1] are sampled within [S:[1], Cmax /(1 7)]. Then, our target is to find the following optimal
distribution:
(B) 22 0 if E (B)JR(m²)) = Jn(
(13)
otherwise,
where 3 is an optimal solution of the outer problem (12). Once we obtain the optimal distribution, ( can be achieved by sampling from g".
In order to obtain the optimal distribution, we propose novel update process by parameterizing the sampler and building loss function. First, we parameterize the sampler as &s. where 4 and define the following function:
J(z: = J=(m) - K X, (Jc. (T: (J:,(T::))) -d)+
is known that for sufficiently large > 0, the optimal policy of the inner problem, Ti. is also the solution of max, [m: B), which means Jn( = max, 33 [31]. Using this fact, we build target distribution for the sampler as: (() exp(J( B)), where T3. is the policy for ( at time-step t. Then, we define a loss function using the cross-cntropy (H) as follows:
(o()) H(E,, E.) Ee-es log (3)) = - EB-c, (Jaes[[[T[)) (T),
VoL((o) Eg V (B)) 3)].
Finally, we present an update rule along with its convergence property in the following theorem.
Theorem 6.3. Let us assume that the space of (3 is finite and update the sampler according to the following equation:
Φ+t1 o
L.(())
(14)
where Ç is a learning rate, and (t)) is the pseudo-invers, of Fisher information matrix Es- Then, under softmax parameterization, the sampler converges an optimal distribution defined in (13).
The proof provided in Appendix As the loss function consists of the current policy n3. the sampler can be updated simultancously with the policy update.
7

---

Laikago-vo
Reward Sum
1.00
Cost Rate
1.00
Cost Rate
0.6
Cost Rate 3
SACAO lours)
200
0.75
0.75
0.6
-400
0.50
0.25
0.23
0.50
0.2
CPPO
0000
0.0
CV&R.CPO
steps
SDAC
Reward
Cost
Rate
Cassle-uo
-200
eydn
WCSAC-D
400-
-600
Threshold
D.0
Enviotmental stess
Endrrommntaal Steps
1ed
Envronmental steps
1m0
Envionmennbl steps
166
Training curves of the legged robot locomotion tasks. The upper graph shows results for the quadrupedal robot, and the lower one for the bipedal robot. The solid line in each graph represents the average of each metric, and the shaded area indicates the standard deviation scaled by 0.5, The results are obtained by training each algorithm with five random seeds.
Figure 2:
7 Practical Implementation
In order to calculate the policy gradient defined in (9), it is required to estimate the risk value functions. Instead of modeling them directly, we use quantile distributional critics [11] 7c,, (5, a; B3 S x A x BM + RA, which approximate the pdf of z (5, a) using L Dirac delta functions and are parameterized by vp E V. Then, the risk value function can be approximated as:
(3,(3, ~ Σ g(be; + bZc, (8, az ()()))
and (3) can be achieved from (a-7))(111)) 1 (3, a)}- The distributional critics are trained to minimize the quantile regression loss [1]] and details are referred to Appendix C. Additionally, to implement& Eo. we use truncated normal distribution N(1, o², a. b) [9], where [a, b] is the sampling range. Then, the sampling process of Eo can be implemented as follows:
abb = -1 k=1
A&[[), where A3, ~ (M,, of oi, 0, Cmax/ (1- Vi and Vj.
Additional details of the practical implementation are provided in Appendix C.
Experiments and Results
Tasks. The experiments are conducted on the Safety Gymnasium tasks [14] with a single constraint and the legged robot locomotion tasks [15] with multiple constraints. In the Safety Gymnasium, two tobots-pointtooo robots point and car-att used to perform two tasks: goal task, which involves controlling the robot to reach a target location, and a button task, which involves controlling the robot to press a designated button. In these tasks, cost is incurred when the robot collides with an obstacle. In the legged robot locomotion tasks, bipedal and quadrupedal robots are controlled to track a target velocity while satisfying three constraints related body balance, body height, and foot contact timing. For more details, please refer to Appendix D.
Baselines. Many RCRL algorithms use CVaR to define risk constraints [15, 29, 32]. Accordingly, the proposed method evaluated and compared with other algorithms under the CVaR constraints with a= 0.75. Experiments on other risk measures are performed in Section 8. The baseline algorithms are categorized into three types based on their approach to estimating risk measures. First, CVaR- CPO [32] and CPPO [29] utilize auxiliary variables to estimate CVaR. Second, WCSAC-Dist [26] and SDPO [30] approximate the risk measure R, (GT) with an expected value E.-D R2 (Y)( Finally, SDAC [15] approximates GT, Gaussian distribution and uses the mean and standard deviation of G., to estimate CVaR. The hyperparameters and network structure of each algorithm detailed in Appendix D. Note that both CVaR- -PO and the proposed method, SRCPO, use the augmented state space. However, for a fair comparison with other algorithms, the policy and critic networks of these methods are modified to operate on the original state space.
Results. Figures 2 and show the training curves for the locomotion tasks and the Safety Gymnasium tasks, respectively. The reward sum in the figures refers to the sum of rewards within episode,
8

---

SafetyPointGeall
SafetyCarGoall-v0
SafetyPointButton--e
SafatyCarButtonI.wo
30
SRCPO (ours)
Reward Sum
20
50
23
2.5
CPPO
0.0
CVaR-CPO
Ensrrommnnt steps 1nd
Enviromenntal steps leb
Enstrooment nts steps
1s7
Endrrommnta steps 1e?
SDAC
0.00
0.100
0.20
Cost Rate 4
0.08
0.11
WCSACDD
0.04
0.50
0.10
SDPO
0.025
0.02
0.22
Threshold
0.000 0
6.000
0,000
00
0.2
088
02
CA Cnvtrrmmnnt siapo 107
Enviroementa stape 105
Enveniimental staps
1e5
Enshermente stops 107
Figure 3: Training curves of the Safety Gymnasium tasks. The results for each task are displayed in columns, titled with the task name. The solid line represents the average of each metric, and the shaded area indicates the standard deviation scaled by 0.2. The results are obtained by training each algorithm with five random seeds.
3.0
2.5
Cast Rate
Distribution
3D
PQ
Distribution
DV
1.5
75
PD
VY
DD
FS
SO
SV
1.0
NG
P5
W
12
14
15
18
20
22
24
Reward Sum
0.00
0.04
0.0b
0.05
0:10
0:00
0.02
0.05
0.08
0.10
Wang 0-0.5
CVaR 0-0.5
Rew 0-05
Cost Rale
Cast Rate
Wang 0-100
CWRR 0.75
Fow 0.0.75
Wlnng
d-0.75
a-0.9
a-0.75
Figure 4: (Left) correlation graph between cost rate and reward for policies trained in the point goal task under various risk measure constraints. The results are achieved by training policies with five random seeds for each risk measure and risk level. The center and radius of each ellipse show the average and standard deviation of the results from the five seeds, respectively. (Middle) Distribution graphs of the cost rate under different risk measure constraints. Locations of several percentiles (from the 50th the 99th) level of each risk measure is selected to have similar cost distributions have heen collected by rolling out the trained policy across 500 episodes. (Right) Distribution graphs of the cost rate with different risk levels, a, under the CVaR constraint.
marked on the plot. The risk rate. After training a policy in the point goal task,
while the cost rate is calculated the sum of costs divided by the episode length. In all tasks, the proposed method achieves the highest rewards among methods whose cost rates are below the specified thresholds. These results are likely because only the proposed method guarantees optimality. Specifically in the locomotion tasks, an initial policy often struggles to stabilize the balance of the robot, resulting in high costs from the start. Given this challenge, is more susceptible falling into local optima compared to other tasks, which enables the proposed method to outperform other baseline methods. Note that WCSAC-Dist shows the highest rewards in the Safety Gymnasium tasks, but the cost rates exceed the specified thresholds. This issue seems to arise from the approach to estimating risk constraints. WCSAC-Dist estimates the constraints based on the expected risk for each state, but it is lower than the original risk measure, leading to constraint violations.
8.1 Study on Various Risk Measures
In this section, we analyze the results when constraints defined using various risk measures. To this end, we train policies in the point goal task under constraints based on the CVaR, and Pow, risk measures defined in (2), as well as the Wang risk measure [24]. Although the Wang risk measure is not a spectral but a distortion risk measure, our parameterization method introduced in Section 6.1 enables it to be approximated as a spectral risk measure, and the visualization of this process is provided in Appendix E. We conduct experiments with three risk levels for each risk measure and set the constraint threshold as 0.025. Evaluation results are presented in Figure 4, and training curves are provided in Appendix E. Figure 4 (Right) shows intuitive results that increasing the risk level effectively reduces the likelihood of incurring high costs. Similarly, Figure (Left) presents the trend across all risk measures, indicating that higher risk levels correspond to lower cost rates and decreased reward performance. Finally, Figure (Middle) exhibits the differences how each risk measure addresses worst-case scenarios. In the spectrum formulation defined in (I), CVaR applics

---

uniform penalty to the tail of the cost distribution above specified percentile, whereas measures like Wang and Pow impose heavier penalty higher percentiles. As result, because CVaR imposes relatively milder penalty the worst-case outcomes compared to other risk measures, it shows the largest intervals between the 50th and 99th percentiles.
Conclusions and Limitations
In this work, we introduced a spectral-risk-conntranne RL algorithm that ensures convergence and optimality in tabular setting. Specifically, in the inner problem, we proposed generalized policy update rule that can facilitate the development of new safe RL algorithm with convergence guarantees. For the outer problem, we introduced notion called sampler, which enhances training efficiency by concurrently training with the inner problem. Through experiments continuous control tasks, we empirically demonstrated the superior performance of the proposed method and its capability to handle various risk measures. However, convergence to an optinum is shown only in a tabular setting, so future research may focus extending these results linear MDPs function approximation settings. Furthermore, since our approach can be applied to risk-sensitive RL, future work can also implement the proposed method in this area.
Acknowledgments and Disclosure of Funding
This work was partly supported by Institute of Information & Communications Technology Planning Evaluation (ITP)) grant funded by the Korea government (MSIT) (No. 2019-0-01 190, [SW Star Lab] Robot Learning: Efficient, Safe, and Docially-Acceptar National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-0021 13357 Smart Assembler: Robot Active Learning for Unseen Parts Assembly, 30%)
Machine Learning, 70%) and the
References
[1] Carlo Acerbi. Spectral measures of risk: coherent representation of subjective risk aversion. Journal of Banking & Finance, 26(7):1505-1515 2002.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of International Conference on Machine Learning, pages 22-31, 2017.
[3] Patrick Adjei, Norman Tasfi, Santiago Gomez-Rosero, and Miriam AM Capretz. Safe reinforce- ment learning for arm manipulation with constrained Markov decision process. Robotics, 13(4):
63, 2024.
[4] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 2021.
[5] Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3682-3689, 2022.
[6] Osbert Bastani, Jason Yecheng Ma, Estelle Shen, and Wanqiao Xu. Regret bounds for risk- sensitive reinforcement learning. In Advances in Neural Information Processing Systems, pages 36259-36269, 2022.
[7] Nicole Bäuerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94(1):35 69, 2021.
[8] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on rein- forcement learning. In Proceedings of International Conference on Machine Learning, pages 449-458, 2017.
[9] John Burkardt. The truncated normal distribution. Department of Scientific Computing Website, Florida State University, 1(35):58, 2014.
10

---

[10] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (1):6070-1120, 2017.
[11] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[12] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gra- dient primal-dual method for constrained Markov decision processes. In Advances in Neural Information Processing Systems, pages 8378-8390, 2020.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of International Conference on Machine Learning. pages 1861 1870, 2018.
[14] Jiaming Ji, Borong Zhang, Xuchai Pan, Jiayi Zhou, Juntao Dai, and Yaodong Yang. Safety- gymnasium. Gitllub repository, 2023.
[15] Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based safe distributional reinforcement learning for multiple constraints. In Advances in Neural Information Processing Systems, 2023.
[16] Dohyeong Kim. Mineui Hong, Jeongho Park, and Songhwai Oh. Scale-invariant gradient aggre- gation for constrained multi-objective reinforcement learning, arXiv preprint arXhv:2403.00282, 2024.
[17] Puze Liu, Haitham Bou-Ammar, Jan Peters, and Davide Tateo. Safe reinforcement learning on the constraint manifold: Theory and applications. arXiv preprint arXiv:2404.099 09080, 2024.
[18] Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: sherior-point policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4940-4947, 2020.
[19] Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400-407, 1951.
[20] R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of Risk, 2:21 4-, 2000.
[21] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of International Conference on Machine Learning, pages 1889-1897. PMLR, 2015.
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 177.071707.06347, 2017.
[23] Jayaram Sethuraman. constructive definition of Dirichlet priors. Statistica Sinica, pages 639 650,,5500111 1994.
[24] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of Risk and Insurance, pages 15-36, 2000.
[25) Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of International Conference on Machine Learning, pages 1 11480-1 1491, 2021.
[26] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Safety- Safety--onstrained reinforcement learning with a distributional safety critic. Machine Learning, 11933)859-8877 2023.
[27] Tsung- Ken Yang. Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. In Proceedings of International Conference on Learning Representations, 2020.
11

---

[28] Tsung- Yen Yang, Tingnan Zhang, Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu. Safe reinforcement learning for legged locomotion. In Proceedings of IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2454-2461, 2022.
[29] Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. In Proceedings of International Joint Conference Artificial Intelligence, 2022.
[30] Jianyi Zhang and Paul Weng. Safe distributional reinforcement learning. In Proceedings of Distributed Artificial Intelligence, pages 107-128, 2022.
[31] Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205. /8814, 2022.
[32] Qiyuan Zhang, Shu Leng, Xiaoteng Ma, Qihan Liu, Xueqian Wang. Bin Liang. Yu Liu, and Jun Yang. conntconstr policy optimization for safe reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
12

---

A Proofs
Lemma 5.1. Assuming that a function is differentiable, Vi (8) and Q (3, a) are bounded by
(o(e:b),g(ett + 6C (////(11111)) -))],, and (5,g( Vi)((3) I bC, where C' Cuux g' (1=a) ) and 3 = (s, (e b).
Proof. Since 0 S Y2, (8), (8, S Cmax/(1 - -?,,
fbe, + bbe (8) fhe, +b2g (f.a)(2) 0,
when be; or be: bc max/ (1-7). Using the fact that g increasing and convex function,
bbe Gums
Guiss
g(be;) S Vigg (8)
(ee (2) S g(be, + bC /D - -)))
beg
Likewise, Qr (5, a) also satisfies the above property. As a result,
Vig (3), (5, [g(bes), g(be; 5C -]]]-
Then, the maximum of QT (5, a) Vir (3) is g(be; + 3C -1->)) - g(be;). Since b = 7 and e Ei 7 C ar 9 * 3 where is the current time step.
be, + ma (1 7) S Cmax/ (1 7).
Consequently,
17 (3, - Vin(() S y(be; + (1
g(bes)
< b Cmax g' be; +
Cmax
Cmax g6 1 1-7
C'axx
S b
Theorem 5.2. Given two policies. π and n', the difference in the sub-risk measure is:
Rs. (G),)
RR. (G Eag .n [A (3, a)] /(1 -7).
(8)
Proof. Since the following equation is satisfied:
| g(z)fbr, 107 t2 ((18.a) [ g(z) olalffeeetcceeeo))))a
= Ex
g(2)fu (3) (2)d2
can say Qr, Eg.-P((11..) V. (3') Using this property.
Bay'or xr (A (5) /(1 7)) = E,"
r' A.. (5e, an)
t=
E,
Q (8r, az) - 4. (8x)
(15)
E,,
(V.7g, (5r41) - Vi (5x)
1=0
lim
E
[V (3:) - EsoNp [Vig (50)
Ro: (GT;) + / gi (o)
lim
Ey
[V (st)
1-500
13

---

Using the definition of V (5t) and the dominated convergence theorem,
lim 1-900
E
[ 1.00
(5t)]
lim E
9:(2) (on
1-900
[ ff()), 2 " (8) (2))
lim Er
(16)
81000
= Ex' [/ ai())ozz (z)dz
RR; (G L. gi (i((o(u))ddu
By combining (15) and (16),
Ro (GC,) + J gi" (o:(o(u)))
Egy' x [A, (8, [)] (11 ()) = lim En V i.g.
(3:)]
t-000
Rg. (G),) R. (Go,).
Lemma 6.1 (Approximation Error). The difference between the original risk measure and its discretized version is:
(X) - Ra (x)x s
Proof. The difference is:
(X) Ro Rx(X) 2 / |FX (u)llo)o {u))ldu S l| 811111Fx 11 =11o-
Cmax
The value of (11) is |10 - 8111 and is smaller than the value of the following problem:
min / [a(u) =(u)ldu s.t. [' ē(u)du= = - no Tg
a()
Vj,
(17)
M
M
since the search space is smaller than the original one. As the value of (17) is smaller than max, (o(u) ō(u)] s (1)/M, ||o 8||1 I (1)/M. Consequently,
(X) - (x) s ||o ō|1 i I (1)
Cmaxx
(1 (1-7)A -)
Theorem 6.2. Let parameterize the function g using parameter B E B C M-1 as: M-1
gg(z) 11 ESn (m)+1-n)(< B15))+
where B[]] Blj + for j fl..., M 2}. Then, the following satisfied:
R& = inf R2x()) where (x) = Elga(X)] + [ (z)
Proof. Since (1r) is an increasing convex function, Rx(X)= < RS (X) R$ In addition, according Remark 2.7 in [1], there exists such that ğ = arg inf (X), and its derivative is expressed as ÿ' = (x)) Using this fact, we can formulate ğ by integrating its derivative as follows:
M-1
((x)
17 x + Σ (7/+1 (1=+1- 7z) Fx (a,)) +C,
where C' is a constant. Since for any constant C. RS (X) has the same value due to the integral part of ÿ" in (3), we set as zero. Also, we can express ((x) as g g2(x), where BBj] Fx (ay) due the definition of gg. As result, the following satisfied:
Rx) R (X) R (X) S R (X).
+ (X)
= (X)
14

---

Theorem 6.3. Let us assume that the space of [3 is finite and update the sampler according to the following equation:
0r+1= - (o
(14)
where Ç is a learning rate, and (T(() is the pseudo-inverse of Fisher information matrix E. Then, under softmax purameterizatin, the sampler converges to an optimal distribution defined in (13).
Proof. Since we assume that the space of / is finite, the sampler can be expressed using the softmax
parameterization as follows:
exp(o((3))
(3) Eg (xp((((3')
we update the parameter of the sampler as described in (14), the following is satisfied as in the natural policy gradient:
Dc+ (5+1 2.(3) +
( (J(TT,;: Σ (J(Ta.u:)) k=1
Furthermore, due to Theorem 5.3, a policy Ta, is updated by the proposed method, the policy converges to an optimal policy TB. Then, if [3 is not optimal,
lim Σ
((J(TB,1))) - J (TT (;)))) -00,
1=1
since 1om JTT (3) (IT J( (3) - : O. As result,
lim (B) = lim explor(B) -r((3))) Σ exp(or(33) o,()))
6000
t000
= lim exp
Σ ( 1811
/
exp())(()) - (x(33)) = 0.
1-000
Thus, Eg, converges an optimal sampler.
A.1 Proof of Theorem 5.3
Our derivation is based the policy gradient works by Agarwal et al. [4] and Xu et al. [25]. Due to softmax policy parameterization, the following satisfied:
tt+1 (al3) = (a(3) exp(Or+1 (5, a) (3,
(18)
Z,(8)
where Z(8) EaEA Te cxp(()+) - 0, a)). By the natural policy gradient,
02 + G(AA - 5: Ex A (/11)/1111)) -)
Jc, (0.) S di Vi, otherwise.
P
(19)
02 + Al E, A ig //11 -7)
Before proving Theorem 5.3, we first present several useful lemmas. Lemma A.1. log (3) is non-negative
Proof. Due to the advantage functions in (19), EeEA (cl33) (x, 0. Using the fact that logarithmic function is concave,
log (5) log Σ (a(3) exp(O.+1 0,(5,))) Σ
0,(5,a)) 0.
aed
As result, log (x) 2 0.
Lemma A.2. DTv(T ne(-1)) max, (3,4) 0c(3,a)) 110011 021loo
15

---

Proof. Using the Pinsker's inequality,
(11) - V (++( [ (+ss))
2 a
T (3 log Zr (8))
2
118
2 = -
max [ (5, a) - (0 (((1))
where (i) follows Lemma A.1, and (ii) follows the fact that > [r(a] a) (((ss)))
As result,
DTv(Te+1(13). m (-13) S max (axx|0e+1185 (0,+1(3, - 02(3,a)] S 10,+1 - -0elooo
Theorem 5.3. Assume that there exists a feasible policy y such that Jc, (T+) for Vi, where n >> Then, if policy is updated by the proposed update rule with learning G. that follows the Robbins-Monro condition [19], will converge to an optimal policy of the inner problem.
Proof. We divide the analysis into two cases: 1) when the constraints are satisfied and 2) when the constraints are violated. First, let consider the case where the constraints are satisfied. In this case, the policy is updated as follows:
0(41
0₁ 1 y
St
A A
E
(20)
Using Theorem 5.2,
-
- Jn((e) + G E An,(Ja - (0+t1))
1
(3)
Σ (a) (3, - Se Σ Se.A (x (3,a))
p
?
1
d
T++
(cl5)) -Ge Σ A5
1-7
1
2
Σ
d
max (h(5,a) Ce Σ (1) (21)
(8) 2|10++1 0ellxc
de
Jrtt1 (3)
max Ak((5,0) -Ge Σ Ar. (5.
1- ?
a
2|10,11 Oelloc || Ge Σ
Ar.:41 (1 1d-2) -1 2111/ Se Σ
2G:
Aes 2
< 1- ?
< 26. (Bmax + NGN 12/(1211))) 7)
16

---

where (i) follows Hölder's inequality, and (ii) follows Lemma A.2. Using (18) and (20),
In(0++1)-Jnihe) + G
Σ (0,:), (0++1)
1
dp d (3)
Σ
T441 - E 20 (5,
BES
1
(
(22)
BES
AEA
(3)
+ log Zi(3))
C1 SE
c+++ (5) log (5)
2
@ -
Σ
((s)
log (z)),
Ct SE3
SEB
where (i) is from the fact that (5) (1 - and (22), the following is satisfied for any pt
E,
P(( =) 2) (1)))(() By combining (21)
())
1-2 Σ P(3 log Zi(3) s (1 (1-7)4 (Rmax + SeN AaNx
2c
(23)
Then, the following satisfied for any policy म:
(R( - JR(TT) tc
- (a)) (d(())
Σ (3) Σ [Acl)) - Ce Σ Ae
SES
aEA
1
(a)))
BEA
(24)
1
D (1 log (5))
4.
1
8) DKL (
SES
2G
+ (1 7)5
|Rmax G.N N.Nammaeeaaa
where (i) follows (23) by substituting p into dz. Now, let us consider the second case where constraints are violated. In this case, the policy is updated as follows:
Oc+1 = 0. +
Ge (GurAR
Σ Ar.
(25)
1-7
17

---

As (21) derived in the first case,
(u((JR((411) -nn(()) E ( (0(+1))
g (8)
1 1-1 Σ (mt41(8)
- (a))) Σ Ne (8,0)
(26)
Σ d g max (unAA((3, Σ X1 N (5, a) (5)
7
2 2110 0x0
dp (5) max (cnA (5.
1-7
2Gr
|1GerAAR Σ Ae s 22Go(GeAmae Rmax + NAmax mmax)² /2/(1 7)
(1 - 7)2
Using (18) and (25).
Guu(Jn(Ptt))= - -Jn((t)) + Σ (c,(0)))))
1
d
Σ
m Σ Ain (3,
8EB
Σ
d T
=
(27)
SES
d ( log (x)
=
SES
2
d (5) log (x)
2
Σ
p(3) log (5).
SE8
BES
By combining (26) and (27), the following is satisfied for any p:
1 1-7 Σ p(g) (3) < (1 (1- (StAmax + N Ndmax m
26:
(28)
As in (24), the following is satisfied for any policy म:
(J)() JR(T()) + Σ (T)() Jc,
Σ
(3)
Σ
[[dB)(Gr Di (5,a))
SE
aEA
1
BEA
(29)
1
D KL (# (+(11)) (.(1)) (| log (5))
G
1
<
Te( (
4.
SES
25,
+ (1 75 mae Rmax + NAmax )
Now, the main inequalities for both cases have been derived. Let us denote by N the set of time steps in which the constraints are not violated. This means that if Jc. (mr) I de Vi. Then, using
18

---

(24) and (29), we have:
Σ
- -(((T)))((Ti)) - Σ (to( Jc
VEN
Σ
((((JA(t) - -R(TT) Ge Σ (e,(Jo,(F)) - (T((
+
MEN
(30)
2G?
DKL( (1) + Σ
(s)
(1 7) (Rmax SeN Ama maxx)
#ES
EEN
267
+
Σ
7)5 GeAmax-maa + N N
ME
If t & N and Ar; 0, Jc. (T+) > and Ae, 1. Thus, E, (,( Ja(())) S -..
By substituting it into of in (30),
Σ IEN
-(((())))(((m()) Jn(TT)) - &t Σ (1)(Ja(TT)) Ja (a(()))
Σ (Gm Gu((n((TT) Jn(()))
KgW
d (5) DKL (KL (T)((1))
8ES
2 max 1)2
+
(1-7)² (max (max Rmax + N Nmmax NAmax
K1
For brevity, we denote DKL( (Ty(TT()T)) =
Eses de (3) (()) Since EiEN St
E.S. St - ELEN Se. (30) can be modified as follows:
- Σ Jn(()) INN
- + Es
(IL(Ty/ITT) + K2 Σ 27 Σ
G E - Jc.
tEN
E (GGU(TT)()T) Jn(())) IEN
s (IL(TT + EG Ki + 2N Amax
Cinax
2>
Rina
m:
It can be simplified follows:
Σ G(JR(T))- - JR(m)) -) + Σ < (RI(FS|(TO) + K₂ Σ G2. BEA
If mg is set with positive values in all actions, DKL. ( (m)) is bounded for Va. Additionally, due to the Robbins- Mobbins-Monro condition, RHS converges to some real number T goes to infinity. Since v) Σ, G in LHS goes to infinity, EIEN (eJR((T)) JR(ML) 77) must go to minus infinity. It means that
19

---

ELEN oo since |Ja(T)] S Rmax/ (1 7). By substituting á into x² in (30),
E ((J((TT) - - -Es, Se
Σ - (T+))
-
EN
IEN
2GF
DKL(T" (|mo) + Σ
(1- (Ruax N. Amansee
IEN
262
7)5 (Se (SeAmaxlBoa + N DmmaxCmax)
IEN
Σ
(x+)) (c,))) (()( Gur(n((+)) -
LEN
IEN
I (TL(T"||TTT) + K2 Et.
Since E; -J. S 0 for t & N,, the above equation is rewritten follows:
-G(Ja(t') -aout)-JJllll s DEL ([(=)) + K₂ Σ G. IEN
Since ExEN oo and the Robbins-Monne condition is satisfied, the following must be held:
Jn(Ø*) - lim JR(TNN1)) = 0,
t-00
where M[t] is the tth element of N. Consequently, the policy converges to optimal policy,
A.2 Convergence Rate
In this section, we analyze the convergence rate of the proposed method. We adopt the finite-time analysis done by Xu et al. [25], and to do this, we make a few minor modifications to the policy update rule described in Section 5.2 as follows:
0, + (O. (n(()) - - SEN )v,
if Jc. 5 + Vi,
O411
A=A. C (veJn(()- Ein de,id,, (0)
otherwise,
0=0,
where 17 is a positive number. Then, we can show the convergence rate of the proposed method.
Theorem A.3. Let us set Ç = (1 (1-7)2.5
and = (2D + 20K3) - 2 where D = D. d (K) (+]))]][TTmm((() and K == max(N (N Amax Cmax: Amax Riaxx Rmax 1). Then,
the followings satisfied:
EEN < (2D + (/(()))(111))))) ((1--)) La (T.)) d, < (21 20k2) 2003)/((11 72) VT) for Vi.
Proof. By subtituting - / (1-7) and Jc., (T² Jc. (Te)l S 20 20max/ (1-
7) into (30),
Σ (S([R((=)) - JR( - NAma - 7))
EEN
Σ (Sn -22a Amax - 7))
+
< D
2Pmaxx + CNN Cmax /2/(1 - 773
IEN
Σ
2ç (CAmax + N NAmax /mxx 7/35.
+
14N
Using the definition of K,
Σ (J(())))J((TT))++ + iEN
KK/ (1-7).
20

---

Given that D + 2 1)3 -1-7 1)) S Cn/T2, one of the following must S. Let us assumme that neither condition is
hold: 1) [A 2 T/2 or
satisfied. Then,
< (T -WI]
Es E - - + > S"
2
LEN
IEN
LEN
11 71-7)5
2TÇ² (1-7)
ST 2
< D
which leads to contradiction. Therefore, one of the two conditions must hold. Furthermore, if we assume IV| = follows that (T/l' S /2, which also a contradiction. Hence, [N] of 0. [ T/2.
Jn(n") [.NN(Jn(TT)] EnEN(JR(TT) - JR(
W
2(D + (T 2
(1-7)5
+
(I-7)
2D 20K2
(1 - 7)
The above inequality also holds for
JR(T()) S O. Consequently,
JR(TT) - EI-N((R(TT)) S (2D + ((1-)))25)
Also, (J., d, S (2D 20K2) /((1 - -)2..
B Weight Selection Strategy
In this section, we first identify 24 and Ae, for existing primal approach- based safe RL algorithms and present our strategy. For the existing methods, analyze several safe RL algorithms including constrained policy optimization (CPO, [2]), projection-busee constrained policy optimization (PCPO, (27]), constraint-rectffeed policy optimization (CRPO, [25]), penalized proximal policy optimization (P30, [31]), interior-point policy optimization (IPO, [18]). and safe distributional actor-critic (SDAC, [15)). Since the policy gradient is obtained differently depending whether the constraints satisfied (CS) or violated (CV), we analyze the existing methods for each of the two cases.
Constraints are satisfied (CS), First, CPO [2] and SDAC [15] find direction of the policy gradient by solving the following linear programming with linear and quadratic constraints (LQCLP):
Case 1:
max (e, s.t.V.e. (0.)²' g + Jc. (0.) s d, Vi, 9 F(0.)9 Sc g
where is a trust region size. The LQCLP can be addressed by solving the following dual problem:
X" vt arg max
VJR(0.)) 9'((0,))''(A,,)) + Ai(V (0.) 9" (2, + (0.( d:)
A20,020
+ +ul 90,, ((0)))((A,)))
where g" - (VJR(0.) V.A, Then, the policy is updated in the follow-
ing direction: 9t = Fi.) G. Σ min(); Amax Jc. (O.)),
(31)
which results in Ar,; min(>;/ /c. ). P3O [31], PCPO [27], and CRPO [25] update the policy only using the objective function in this case as follows:
= F (z,))))(()))) + Ae, = 0.
IPO [18] update policy in the following direction:
gt = ().)
VJR(0.) & Σ (J),(0.)/(4 - Jc, (0:))
21

---

where k. is penalty coefficient. As result, A is computed as min(k/((d - Jc. (mm))
Case 2: Constraints violated (CV). CPO, PCPO and IPO did not handle the constraint violation in multiple constraint settings, so we exclude them in this case. First, SDAC finds a recovery direction by solving the following quadratie programming (QP):
min 20 F(0r)g (.t.)ggst.VV..... V., Jc. (0,) - d: for i {d]Je. (0:) > d::
where we will denote (zlJo. d4} by lov. The QP can be addressed by solving the following dual problem:
1
x = arg max 29 (A)* F(O))
E
((V Jc, -d:),
A20
sElev
where = -Fi(()) Lielor =FF(())Eiiiov V.c.
(0t). Then, the policy is updated in the following direction:
g== = (O()
Σ VELce
EjElow -
X5 X5 VJc.
X
= ur =0 de,i = Ejelor >3
X
if E Icv else 0.
If there is only single constraint, PCPO is compatible with SDAC. Next, CRPO first selects violated constraint, whose index denoted k, and calculates the policy gradient to minimize the selected constraint follows:
F* (0.) (- V.,, = = de, = if i = k else 0.
there is only a single constraint, CPO is compatible with CRPO. Finally, P3O update the policy in the following direction:
9t = F VJn((.) - Σ VJc (0.)
iETcv
which results in v min(1/ [cvl), Amax), and X = 1/l/ if E Icv else 0.
Implementation of trust region method. To this point, have obtained the direction of the policy gradient 9t, allowing the policy to be updated to 0r + Sest. If we want to update the policy using trust region method, introduced in TRPO [21], can adjust the learning rate C as follows:
"
Se Clip(v of F(O.).:, Imin Imax)
(32)
where is trust region size following the Robbins- Babbins-Monro condition, Imin and Jmax are hyper- paramters, and Clip(z, a, b) min (ainmmxx2,, a), b). By adjusting the learning rate, the policy can be updated mostly within the trust region, defined as {0, "وg olg"FF(B) F(0.)g S {f), while the learning rate still satisfies the Robbins-Monro condition. Accordingly, we also need to modify Xτ, for the CS case
and De for the CV case, they are expressed with the learning rate G. We explain this modification
process by using SDAC as an example. The direction of the policy gradient of SDAC for the CS case expressed as follows:
9t (+.))VV)))())) -x Σ (iin(A//ex Amax Vc. (0:)),
where & in (31) is replaced with The learning rate Se is then calculated using 9, and (32). Now, we need to express gi as Ft (v.)((nn(()) Ge E; d Jc. (0t)). To this end, we can use S. et/O, where C' = Clip( / (T ((,, (0,)))) Jmin Imax). Consequently, Ae, is expressed as follows:
A = C min(); / Amax);
and the maximum of Xr,; and v, is adjusted to 9max 9max
Proposed strategy. Now, we introduce the proposed strategy for determining Xτ, and We first consider the CV case. Kim al. [15] empirically showed that processing multiple constraints simultancously at each update step effectively projects the current policy onto a feasible set of policies.
22

---

Therefore, we decide to use the same strategy as SDAC, which deals with constraints simultaneously, and it written follows:
Pr Ar, = {x5/ (Ejete X3) if E Icv. 10
otherwise,
where X arg 220 (2) + (VJ,, (A) + Jc. -d:) and g" (A) = - (0.) Eieley (ft)) Next, let us consider the CS case. We hypothesize that concurrently addressing both constraints and objectives when calculating the policy gradient can lead to more stable constraint satisfaction. Therefore, CPO, SDAC, and IPO can be suitable candidates. However, CPO and SDAC methods require solving the LQCLP whose dual problem is nonlinear, and IPO can introduce computational errors when Jc, (0x) is close to d, To address these challenges, we propose calculate the policy gradient direction by solving QP instead of an LQCLP. By applying a concept of transforming objectives into constraints [16] to the safe RL problem, we can formulate the following QP:
1 min 2 s.tt) VJR(0,) e, (J.)" Jo, (0.) de Vi,
where e == V VJ FU,(
According to [16], represents the maximum value
F(0:)g of (R.)) when g is within the trust region, [glg"F((0)) ) S e Therefore, the transformed constraint, (O,) g 2 e, has the same effect of improving the objective function Jn(O) within the trust region. The QP is then addressed by solving the following dual problem:
X
arg max (A, M) F(0:)9 (A, u(e Jonn())n( ((,00))")(,,))))) ~20 2
Ac..c (0:)² (A, Jc, - dz),
where g² - -
=())(U(RR))))) Eieley A.V (0.)). Then, the policy is updated in the
following direction:
9t F()))
vJn((.) a Σ min
x ixttt
Amax
VJc. (81) vJc:
and the learning rate is calculated using (32) to ensure that the policy is updated within the trust region. Finally, X is expressed as C' minn(A A (v=et), Amax), where C' = Clip( v of F(O.)gt. Imin Imax).
C Implementation Details
In this section, we present practical algorithm that utilizes function approximators. such as neural networks, to tackle more complex tasks. The comprehensive procedure is presented in Algorithm 2.
First, it is necessary train individual policies for various B, so model policy network to be conditioned on B3 expressed as To 8; (3). This network structure enables the implementation of policies with various behaviors by conditioning the policy different & values. In order to train this (-conditioned policy across a broad range of the f space, à e-greedy strategy can be used. At the beginning of each episode, , is sampled uniformly with probability of e; otherwise, it is sampled using the sampler Eg. In our implementation, we did not use the uniform sampling process by setting 0 to reduce fine-tuning efforts, but it can still be used to enhance practical performance.
After collecting rollouts with the sampled B3, the critic, policy, and sampler need to be updated. For the critic, a distributional target is calculated using the TD(A) method [15], and we update the critic networks to minimize the quantile regression loss [11], defined as follows:
p (z ZC.. (31)]
((V)
E -~ S~Bo, (5,0;3)
where pr(u) == u (1 Ia and 2cc (8, a: (3) is the target distribution of 7c. a; B3). The policy gradient for specific & can be calculated according to the update rule described in Section
23

---

Algorithm 2 SopectralRiisk-Conenn
Policy Optimization (SRCPO)
Input: Policy parameter 0, sampler parameter Φ, critic parameter v and replay buffer D. for epochs
P do for episodes = 1 to E do if u c, where Sample [3 from uniform distribution: B:Di] U - 1], / (1-))) Vi and vj,
O0, 1]. then
where [.
else
Sample 3 ~ Eo.
end if
Collect a trajectory = (8t, Cr, Tr, Tr, (e. using na(a|8;83) and store (3,7) end for
Set G = and G =
for updates to U do
Sample (3, ~
Update the critics, (5, a: (3 and Zc. (3, (3), using the quantile regression loss. Calculate the policy gradient of No to(a]:; B) using the update rule, described in Section 5.2, and store 6
Calculate the gradient of the sampler &s using the update rule, described in (14), and store it to G end for
Update the policy parameter by the average of G Update the sampler parameter by the average of Go. end for
Output: To(-1: B), where B Ev.
(a) Point goal task
(b) Car button task
(c) Quadrupedal robot
(d) Bipedal robot
Figure 5: Rendered images of the Safety Gymnasium and the legged robot locomotion tasks.
5.2. Consequently, the policy is updated using the policy gradient calculated with 3 and rollouts, which are extracted from the replay buffer. Finally, the gradient of the sampler for given / can be calculated as in (14), so the sampler is updated with the average of gradients, which are calculated across multiple B.
D Experimental Details
In this section, we provide details on tasks, network structure, and hyperparameter settings.
D.1 Task Description
There are two environments, legged robot locomotion [15] and Safety Gymnasium [14], and whose rendered images are presented Figure 5.
Legged Robot Locomotion. The legged robot locomotion tasks aim to control bipedal and quadrupedal robots maintain specified target velocity, which consists of linear velocity in the X and Y directions and rotational velocity in the Z direction, without falling. Each robot takes actions as PD target joint positions at a frequency of Hz, and PD controller each joint operates at 500 Hz. The dimension of the action space corresponds to the number of motors: ten for the bipedal robot and 12 for the quadrupedal robot. The state space includes the command, gravity vector, current joint positions and velocities, foot contact phase, and history of the joint positions and

---

velocities. The reward function is defined as the negative of the squared Euclidean distance between the current velocity and the target velocity. There are three cost functions to prevent robots from falling. The first cost is to balance the robot, which is formulated 1 g(2)/1 2ol22c00(x/12) where is the gravity vector. This function is activated when the base frame of the robot is tilted more than 15 degrees. The second cost is to maintain the height of the robot above predefined threshold, which is expressed as <b, where & the height of the robot, and b set as 0.7 in the bipedal robot and 0.35 in the quadrupedal robot. The third cost is to match the foot contact state with the desired timing. It defined as the average number of feet that do not touch the floor at the desired time. We set the thresholds for these cost functions as 0.025/ (1- 7), 0.025/ 7), and 0.4/(1 (1 - 7), respectively, where (1 - 7) converts the threshold value from an average time horizon scale to an infinite time horizon scale.
Safety Gymnasium. We use the goal and button tasks the Safety Gymnasium [14], and both tasks employ point and car robots. The goal task is to control the rohot to navigate to target goal position without passing hazard area. The button task is control the robot to push designated button among several buttons without passing hazard area hitting undesignated buttons. The dimension of the action space is two for all tasks. The state space dimensions are 60, 72, 76, and for the point goal, car goal, point button, and car button tasks, respectively. For more details on the action and state, please refer to [14]. The reward function is defined to reduce the distance between the robot and either the goal or the designated button, with bonus awarded at the moment of task completion.
When the robot traverses hazard area or collides with an obstacle, it incurs a cost of one, and the threshold for this cost is set as 0.025/( -)).
D.2 Network Structure
Across all algorithms, we use the same network architecture for the policy and critics. The policy network is structured to output the mean and standard deviation of a normal distribution, which is then squashed using the Tanh function as done SAC [13]. The critic network based on quantile distributional critic [11]. Details of these structures presented in Table 1. For the sampler Ça, we define trainable variable ø E R NXM-11 and represent the mean of the truncated normal distribution as [i,ols] exp(ø[i, j(). To reduce fine-tuning efforts, we fix the standard deviation of the truncated normal distribution to 0.05.
Table 1: Details of network structures.
Parameter
Value
Policy network
Hidden layer
(512, 512)
Activation
LeakyReLU
Last activation
Linear
Reward critic network
Hidden layer
(512, 512)
Activation
LeakyReLU
Last activation
Linear
of quantiles
25
of ensembles
2
Cost critic network
Hidden layer Activation
(512, 512)
LeakyReLU
Last activation
SoftPlus
# of quantiles
25
# of ensembles
2
D.3 Hyperparameter Settings
The hyperparameter settings for all algorithms applied all tasks are summarized in Table
25

---

Table 2: Description on hyperparameter settings.
SRCPO (Ours)
CPPO [29)
CVaR-CPO [32]
SDAC [15]
SDPO [30]
WCSAC-D [26]
Discount factor
0.99
0.99
0.99
0.99
0.99
0.99
Policy learning rate
100-
3 6-10-5
3 1006
6-10-4
Critic learning rate
-10-
10
3-10-4
-10
100
3-10-
of largel quantiles
50
50
50
50
50
Coefficient of TD(A)
0.97
0.97
0.97
0.97
0.97
Soft update ratio
0.995
Batch size
10000
5000
5000
10000
5000
256
Steps per update
1000
5000
5000
1000
5000
100
# of policy
20
20
10
updute iterations
of critic
40
40
40
40
40
10
update iterations
Trust region size
0.001
0.001
0.001
0.001
0.001
Line search decay
0.8
PPO Clip ratio (22]
0.2
0.2
0.2
Length of replay buffer
100000
5000
5000
100000
5000
1000000
Learning rate for
0.05
0.01
auxiliary variables Learning rate for Lagrange multipliers Log penalty cocfficient Entropy threshold
0.05
0.01
0.001
5.0
-14]
Entropy learning rate
0.001
Entrupy coefficient
0.001
0.001
Sampler learning rate
1 1-10-3
for sampler target
E Additional Experimental Results
E.1
Computational Resources
All experiments were conducted on PC whose CPU and GPU are Intel Xeon E5-2680 and NVIDIA TITAN Xp, respectively. The training time for each algorithm on the point goal task is reported in Table 3.
Table 3: Training time for the point goal task averaged over five runs.
SRCPO (Ours)
CPPO
CVaR-CPO
SDAC
WCSAC-D
SDPO
Averaged training time
9h 51m 54s
6h 17m 16s
4h 43m 38s
8h 46m 8s
14h 18m 20s
6h 50m 39s
E.2 Experiments on Various Risk Measures
We have conducted studies on various risk measures in the main text. In this section, we first describe the definition of the Wang risk measure [24], show the visualization of the discretization process, and finally present the training curves of the experiments conducted in Section 8.1.
The Wang risk measure defined as follows [24]:
Wang, (X) = 5/0[( (u)dx((()), where (n) == 4( (0) aa)
+ is the cumulative distribution function (CDF) of the standard normal distribution, and x is called distortion function. It can be expressed in the form of the spectral risk measure as follows:
Wang, (X) = f. Fx (u)a where (u()) = Φ( ((τ-1(u)-a)
df Φ (u))
and oi the pdf of the standard normal distribution. Since aa (u) goes to infinity when u approach- ing one, (u) at 1 is not defined. Thus, it is not spectral risk measure. Nevertheless, our
26

---

parameterization method introduced in Section 6.1 can project Wang,, onto discretized class of spectral risk measures.
The discretization process can be implemented by minimizing (11), and the results are shown in Figure 6. In this process, the number of discretizations M is set five. Additionally, visualizations of other risk measures are provided in Figure 7, and the optimized values of the parameters for the discretized spectrum, which are introduced in (10), reported in the Table 4. Note that CVaR is already an element of the discretized class of spectral risk measures, as observed in Figure 7. Therefore, CVaR can be precisely expressed using parameter with M 1.
Finally, we conducted experiments on the point goal task to check how the results are changed according to the risk level and the type of risk measures. The training curves for these experiments are shown Figure 8.
Table 4: Discretization results.
Wanga
Pow
1M
[0.515, 0.790, 1.091, 1.493, 2.191]
[0,200, 0.600, 1.000, 1-400, 1.800] [m
pM--1 f-
(0.263, 0.541, 0.770, 0.926]
(o.k-1
[0.200, 0.400, 0.600, 0.800)
1.0
10.294,0 0.734, 1.417, 5.517]
0.75
[0.046, 0.574, 1 1.347, 2.308, 3.424]
{a. piu
0.409, 0.701, 0.878, 0.968]
[o:)
[0.417, 0.615, 0.765, 0.890
[0.180, 0.834, 2. 2.253, 5.678, 16.419
(n
[0.003, 0.947, 2.705, 5.216, 8.383
(a,) M
(0.579, 0.834, 0.945, 0.989)
[a,)er Ma
[0.701, 0.821, 0.898, 0.955
27

---

Wang a 0.5
Wang a 1.0
Wang a 1.5
Original Discretized
Original Discretized
Original Discretized
20
Spectrum d
Spectrumt
Spectrum
10
8.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
0.2
0.6
1.0
Figure Visualization of the spectrum function Wang risk measure and the discretized results.
0.5
Pow
0.75
Pow 0.9
10
Original Discretized
Original
Original Discretized
Spectrum
Spectrum
Discretized
Specorum
0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0,6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
CVaR
CVaR
0.75
CVaR
Original
Original
Original Discretized
Spactrum
Discretized
Spectrum
Discretized
Spectrum
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Figure Visualization of the spectrum function of the Pow and CVaR risk measures and the discretized results.
Pow risk measure
CVaR risk measure
Wang risk measure
30
Reward Sum
Reward 5um
Reward
2
Environmental steps
Environmental steps
Environmental steps
0.08
0.08
0.08
0.06
Cost Rate
0.06
Rate
0.06
0.02
0.02
0.02
0.00
0
0.00
0.00
0
Environmental steps 1c6
Environmental steps 1e6
Environmental steps 1c6
0.75
0.9
0.5
0.75
0.9
=0=
Figure 8: Training curves of the point goal task with different risk levels and risk measures. Each column shows the results on the risk measure whose name appears in the plot title. The solid line in each graph represents the average of each metric, and the shaded area indicates the standard deviation scaled by 0.2. The results are obtained by training algorithms with five random seeds.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstact and introduction reflect the contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the last section of the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: We have provided the full set of assumptions and proof, which is presented in Section A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have enclosed the code and provided the pseudo-code for reproducibility in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have enclosed the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//cccccccpcppppp gewic//gguee...ee CoodeSummii agionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting and details are provided in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We have reported the standard deviation of experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
31

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have described the computational resources in Section E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the cstuideliidddnnnnn dines???
NeurIPS Code of Ethics https: / http///eerr.pp.eee
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work has no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This work has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have cited the benchmark papers and included the licence for assets in the code.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This works does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "8KaVb0Pemx", "reviewer": "Reviewer_s1fZ", "summary": "The paper provides the new spectral-risk-costrained policy optimization algorithm, that uses the duality of spectral measures, and bilevel optimization approach to address the risk constrained reinforcement learning problem, solving the inner primal problems and outer dual problem. The paper provides global convergence guarantees in the tabular setting. The paper supports the results with the experimental comparisons to other methods.", "questions": "all questions are above in the weaknesses section", "limitations": "The limitations are discussed. There is no potential negative societal impact.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well written, despite many complex notions used in the flow, they all are clearly introduced. The theoretical result seems to be solid and important to the community. The idea to solve the dual problem of the risk-constrained rl  using a sampler distribution seems to be novel too. The proofs are provided, however I did not check them carefully. The experiments demonstrate a good performance of the proposed method.", "weaknesses": "There are several things which are not very clearly defined and a few minor typos that I noticed. \n\nline 88: F_X is not defined, I believe it is a probability function?\nline 168: where the log in eq. (9) comes from?\n\nConflicting notations: \n- Now, aplha is used both for denoting the learning rate and the risk parameter. Maybe, the authors could use different notations for these notions. \n- F is probably a probability function, and then a feasibility function (line 114). \n- in theorem 6.2 i is used to iterate over the discretization 1:M-1,  but later used to iterate over constraints 1:N, and j is used for discretizations. maybe use j in thm. 6.2 and other places consistently too?  \n\nTypos: \n\nline 102: an -> a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for the positive review and valuable feedback on our paper. We appreciate the effort in reviewing our work. Below, we respond to each of the points mentioned.\n\n**Weakness 1 (line 88 and 168)**:\n\n$F\\_X$ is the cumulative density function (CDF) of the random variable $X$. We will add the definition of $F_X$ to the paper after line 88.\n\nThe logarithmic operator in eq. (9) comes from the following derivation of the policy gradient of the sub-risk measure (by differentiating $\\pi'$ in eq. (8)): \n$\\nabla\\_\\theta \\mathcal{R}\\_\\sigma^g(G^{\\pi_\\theta}) = \\nabla\\_\\theta\\mathbb{E}\\_{d\\_\\rho^{\\pi\\_\\theta},\\pi\\_\\theta}[A\\_g^\\pi(s,a)]/(1-\\gamma) = \\mathbb{E}\\_{d\\_\\rho^{\\pi\\_\\theta}}[\\sum_a \\nabla_\\theta \\pi_\\theta(a|s) A_g^\\pi(s,a)]/(1-\\gamma) = \\mathbb{E}\\_{d\\_\\rho^{\\pi\\_\\theta}, \\pi_\\theta}[\\nabla_\\theta \\log \\pi_\\theta(a|s) A_g^\\pi(s,a)]/(1-\\gamma).$\n\n**Weakness 2 (conflicting notations)**:\n\nThanks for the comments on the conflict notations. We will change the notations as follows:\n\n1. learning rate: from $\\alpha$ to $\\omega$.\n2. Feasibility function: from $\\mathcal{F}$ to $\\chi$.\n3. Iterator of the discretization in Theorem 6.2: from $i$ to $j$.\n\nWe will also correct the typo too.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h8Fu4GcqWF", "reviewer": "Reviewer_WtS2", "summary": "In this paper, the authors have considered the framework of risk-constrained reinforcement learning (RCRL) by tackling risk-measure-based constraints. The non-linearity of risk meaures makes the  convergence of RL schemes challenging. In this paper, a spectral risk measure-constarined RL algorithm is proposed. The proposed algorithm is based on a bilevel optimization. The outer problem deals with optimization of dual variables, while the inner one finds the optimal solution given the a set of dual variables. The authors claims that the proposed scheme is the first method which convergs to optimality for tabular MDP problems. Simulation results are presented where the proposed methods are tested on continuous control tasks.", "questions": "1. The proposed approach approximates the spectrum by a discretized spectrum. What is the gap in performance with respect to the optimal solution of the original system? Although in Theorem 6.2, it is derived that as $M$ becomes large, performance gap reduces to zero,  how does this $M$ scale as the number of states and actions increases?\n\n2. The intuition behind the sampling process and why it works is not clear.\n\n3. What is the rationale behind the loss function proposed by the authors? Why is the loss function novel? Is there any other loss function for which the scheme does work well? If yes, can the authors come up with a more general loss function? If no, then this is a limitation of the work that it works for only a specific type of loss function (constructed by the authors).", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The results and claims presented in the paper appears to be correct.", "weaknesses": "There are some limitations with respect to the constructed loss function and scalability of the adopted linear interpolation mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback on our paper. We appreciate the effort in reviewing our work. We have carefully considered the reviewer's comments. Below, we address the concerns the reviewer has raised:\n\n**Q1 (performance gap):** \n\nIn the general response above, we analyzed the performance gap caused by the spectrum approximation; please refer to the general response.\nBriefly, we first bounded the gap by the difference in performance of the original problem at different thresholds.\nAnalyzing the performance difference at different thresholds in a constrained optimization problem is challenging due to discontinuous changes in the feasible set or optimal points.\nThus, we made some assumptions to analyze the gap and found that the gap is bounded by $KR_\\mathrm{max}/(C_\\mathrm{max}M)$.\nThis suggests that the gap affects the scale of the reward and costs rather than the state and action spaces.\nTherefore, we conjecture that the value of $M $can be set independently of the state and action spaces.\n\n**Q2 (intuition behind the sampling process):**\n\nThanks for pointing out the need for clarification.\nSection 6.2 introduced *1)* why we need to learn the distribution of $\\beta$ (sampler $\\xi$), *2)* the training method, and *3)* the technique for sampling $\\beta$ from $\\xi$.\nThen, in Section 7, we presented the practical implementation method, which seems to be pointed out by the reviewer.\nAs mentioned in lines 230-231, $\\beta[j]$ is sampled within the range $[\\beta[j-1], C_\\mathrm{max}/(1-\\gamma)]$.\nTo implement this, distributions with finite intervals, such as the beta or truncated normal distribution, can be used.\nHowever, defining the interval of the distribution with $\\beta$ complicates the gradient computation graph in PyTorch, increasing the likelihood of computational errors.\nTo address this, we decided to sample the difference of $\\beta$, which is motivated from the fact that $\\beta[i] \\sim [\\beta[i-1], C_\\mathrm{max}/(1-\\gamma)]$ is equivalent to $\\beta[i] = \\beta[i-1] + \\Delta$, where $\\Delta \\sim [0, C_\\mathrm{max}/(1-\\gamma) - \\beta[i-1]]$.\nTo further remove $\\beta$ in the interval, we instead sampled $\\Delta$ within $[0, C_\\mathrm{max}/(1-\\gamma)]$, but it can result in $\\beta[i] > C_\\mathrm{max}/(1-\\gamma)$.\nNevertheless, if $\\beta$ increases, the value of $J_C(\\pi; \\beta)$ also increases due to the conjugate function in (2), resulting in reducing $J_R(\\pi)$.\nAs a result, the distribution of $\\beta$ is trained to output low probabilities for such cases.\nWe will add this explanation to the appendix.\n\n**Q3 (loss function):**\n\nFirst, we want to clarify the meaning of line 235, which seems to be pointed out by the reviewer.\nThe meaning is that the process of solving the outer problem is novel rather than claiming that the proposed loss function itself is novel.\nSpecifically, the proposed method enables the inner and outer problems to be solved concurrently rather than sequentially.\nThus, we will change the phrase to \"novel update process.\"\n\nNext, the rationale for structuring the loss indicated in lines 236 and 240 is as follows:\nThe goal of the sampler is to produce a high probability for the optimal $\\beta$. \nTo achieve this, we need an indicator that can provide feedback on how well the RCRL problem is solved for a given $\\beta$.\nBeing inspired by the paper [R1], which converts constrained RL to unconstrained RL using penalty functions, we defined the indicator as shown in line 236.\n\nAdditionally, as observed in the proof of Theorem 6.3, any function $J$ that satisfies $\\max\\_\\pi J(\\pi; \\beta)= \\max\\_{\\pi \\in \\{ \\pi|J_{C_i}(\\pi;\\beta) \\leq d_i }} J_R(\\pi)$ can be used.\nOne such $J$ could be $J(\\pi;\\beta) = J_R(\\pi)$ if $J_{C_i}(\\pi;\\beta)\\leq d_i$ for $\\forall i$; otherwise, $-R_\\mathrm{max}/(1-\\gamma)$.\nAlternatively, instead of providing feedback, a method that uses the policy's value function to compute the optimal distribution in a closed form could be proposed.\nAs mentioned earlier, since we have proposed a new approach to solving the outer problem, it does not seem necessary to propose a more general loss function.\nDeveloping more efficient and reliable loss functions could be done in future work.\n\n**References**\n- [R2] Zhang, Linrui, et al. \"Penalized proximal policy optimization for safe reinforcement learning.\" arXiv preprint arXiv:2205.11814 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UGrHcQsEVj", "reviewer": "Reviewer_tq31", "summary": "This paper proposes a spectral risk measure-constrained RL algorithm, called spectral-risk-constrained policy optimization (SRCPO). This algorithm leverages the duality of spectral risk measures and treats the risk-constrained RL problem as a bilevel optimization. In this bilevel optimization problem, the outer problem is to optimize dual variables derived from the risk measures, while the inner problem is to find an optimal policy given these dual variables. The proposed algorithm is the first to guarantee convergence to an optimum in the tabular setting. Furthermore, the authors conduct experiments on continuous control tasks, and show that the proposed algorithm achieves the best performance among the compared RCRL algorithms.", "questions": "Please see the weaknesses above.", "limitations": "Please see the weaknesses above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe studied problem, i.e., spectral risk measure-constrained RL, is well-motivated and can be applied to safety-critical scenarios, e.g., healthcare and finance.\n2.\tThis paper is very well-written and easy to follow.\n3.\tThe authors design a general algorithm framework and provide the convergence guarantee for a family of spectral risk-constrained RL problems. \n4.\tEmpirical evaluations are also provided to demonstrate the performance superiority of the proposed algorithm in practice.", "weaknesses": "1.\tThe idea of handling risk-sensitive RL by solving an inner problem (i.e., find an optimal policy under a fixed dual variable) and an outer problem (i.e., optimize dual variables) is not new. This idea appears in prior risk-sensitive RL works, e.g., (although these prior works may not consider constraints)\n\n[1] Wang, Kaiwen, et al. \"Near-minimax-optimal risk-sensitive reinforcement learning with cvar.\" International Conference on Machine Learning, 2023.\n\n[2] Chen, Yu, et al. \"Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback.\" International Conference on Learning Representations, 2023.\n\n2.\tThe current format of Algorithm 1 is not clear enough. The specific algorithm steps should be included, e.g., the policy update, the distribution update (how to find $\\xi$) and the sampling.\n3.\tIt seems that Theorems 5.3 and 6.3 are the theoretical guarantees for the inner and outer problems, respectively. Can the authors give the theoretical guarantee of the overall performance for the spectral risk measure-constrained RL problem?\n4.\tThe provided results only state that the algorithm will converge to the optimal policy. Can the authors comment on the convergence rate or sample complexity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review and insightful comments on our paper. \nWe appreciate the effort to review our work.\nThe following is our response to the reviewer's comments. \n\n**Weakness 1 (inner and outer problems)**\n\nAs the reviewer commented, handling risk by separating the given problem into the inner and outer problems can be considered as not new; instead, the proposed method for solving each problem can be considered novel.\nTherefore, we will modify lines 50-51 to emphasize the proposed process of the inner and outer problems rather than the bilevel optimization framework itself.\n\n**Weakness 2 (algorithm details)**\n\nAlgorithm 1 only provides a brief overview of the proposed method, but Algorithm 2 in Appendix C shows details of the method, including information on the policy and the distribution update.\nTo improve clarity, we will change the title of Algorithm 1 to \"Overview of SRCPO,\" and at the end of Chapter 4, we will add a note indicating that a detailed algorithm is provided in Appendix C.\n\n**Weakness 3 (overall performance)**\n\nThrough Section 5, we can find an optimal policy of the inner problem for various $\\beta$. \nIt means that we can find a set of optimal policies, $\\Pi^*=${ $\\pi_\\beta^*|\\beta \\in B^N$ }, where $\\pi_\\beta^*$ denotes the optimal policy for $\\beta$.\nAdditionally, through Section 6, we can obtain an optimal sampler $\\xi^*$ that samples only $\\beta^* = \\arg\\max_\\beta J_R(\\pi_\\beta^*)$.\nThus, by solving the inner and outer problems, we obtain $\\Pi^*$ and $\\xi^*$. \nSampling $\\beta^*$ from $\\xi^*$ and finding the corresponding policy in $\\Pi^*$ leads to the optimal policy $\\pi_{\\beta^*}^*$ of the RCRL problem.\nIn conclusion, by solving the inner and outer problems respectively, we can obtain an optimal policy for the original RCRL problem defined in equation (5), thus ensuring overall performance.\n\n**Weakness 4 (convergence rate)**\n\nWe have addressed the convergence rate of the proposed method in the general response above. \nPlease check the general response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZaO2jS8TZg", "reviewer": "Reviewer_DPN9", "summary": "This paper considers spectral-risk safe RL algorithm with convergence guarantee. In particular, the paper considers a constrained MDP approach where the objective is to maximize the expected cumulative reward subject to the constraint that the spectral-risk measure is below a certain threshold. The main challenge lies in solving the optimization problem as it requires solve a bi-level optimization framework. The paper proposes first proposes a policy gradient based approach for solving the inner problem, and then proposes a gradient-based approach to sample according to the optimal spectral risk-measure.", "questions": "1. In order to solve the inner-optimization problem, the paper considers a primal-dual type algorithm for solving the feasibility function. What types of feasibility functions $F$ have been considered? Can the authors provide some outline on how the dual-variable updates as shown in Appendix B ensure feasibility?\n\n2. For solving the outer optimization, the paper relies on minimizing the loss function. However, optimal policy that maximizes $J(\\pi,\\beta)$ is difficult because of the non-linearity in the feasibility function. Hence, the reviewer did not get how the algorithm solves the inner-optimization problem which seems to be different compared to the inner-optimization problem described in Section 5. \n\n3. The convergence result (Theorem 6.3) relies on the Robbins-Monroe condition. Is it satisfied (in particular, for the optimization framework considered in Section 6 which is different from Section 5)?", "limitations": "Not as such.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Developing a safe RL policy with spectral risk measure is important research problem.\n\n2. The paper provides convergence guarantees and effective policies.\n\n3. The paper also provides practical approaches for implementation. The empirical performance is good.", "weaknesses": "1. The paper is a little-bit dense to parse everything. \n\n2. While the paper provides a convergence guarantee, the paper did not provide any non-asymptotic rates for convergence. In particular, the sample-complexity guarantee has not been provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review and valuable feedback on our paper.\nWe appreciate the effort to review our work.\nBelow is our response to the reviewer's comments.\n\n**Weakness (convergence rate):**\n\nThanks for pointing out the need for convergence rate analysis.\nWe have addressed the convergence rate of the proposed method in the general response above. \nPlease check the general response.\n\n**Q1 (feasibility):**\n\nThe feasibility function is defined as $\\mathcal{F}(x)=0$ if $x\\leq0$ else $\\infty$, which is commented in the footnote of line 114.\nThis function is used to express the constrained optimization problem as an unconstrained one in equation (6), since two problems \"$\\max_\\pi J_R(\\pi)$ s.t. $J_{C_i}(\\pi)\\leq d_i$\" and \"$\\max_\\pi J_R(\\pi) - \\sum_{i=1}^N\\mathcal{F}(J_{C_i}(\\pi) - d_i)$\" are equivalent.\nWhen solving the risk-constrained RL problem, the feasibility function is not used directly; instead, the constraints are used as they are, as shown in equation (7).\n\nAccording to Theorem 5.3, an optimal policy for the inner problem can be obtained if the policy is updated using the proposed update rule, which is described below in line 169.\nAs mentioned in lines 171-172, the proposed update rule requires satisfying the following conditions on $\\lambda_t$ and $\\nu_t$, which the reviewer refers to as \"dual-variables\": *1)* $\\lambda_{t,i}$ and $\\nu_t$ should exist within $[0, \\lambda_\\mathrm{max}]$, and *2)* $\\lambda_{t,i}(J_{C_i}(\\theta)-d_i)\\geq0$ and $\\sum_i \\lambda_{t,i}=1$ when constraints are not satisfied.\nVarious methods for choosing $\\lambda_t$ and $\\nu_t$ that satisfy the conditions are presented in Appendix B. \nBy using one of these methods to calculate $\\lambda_t$ and $\\nu_t$, an optimal policy can be obtained according to Theorem 5.3, ensuring that the constraints are satisfied.\nA brief insight into the proof of Theorem 5.3 is as follows: the update rule is designed to give more weight to the objective function when the constraints are satisfied and more weight to the constraints when they are not. As a result, through repetitive policy updates, the policy converges within or on the boundary of the constraints.\n\n**Q2 (outer optimization):**\n\nThe reviewer seems to think that calculating $J(\\pi; \\beta)$ requires solving another inner problem, which is different from Section 5.\nIn other words, the reviewer seems to think that in order to train the sampler, it is necessary to find a policy that maximizes $J(\\pi; \\beta)$.\nHowever, this is not correct. \nThe policy is only updated using the method introduced in Section 5, and only the sampler is trained in Section 6. \nThere is no other inner problem.\nAs seen in the equation below line 240, the sampler's loss function is composed of $J(\\pi_{\\beta, t}; \\beta)$, where $\\pi_{\\beta, t}$​ is the policy obtained by solving the inner problem from Section 5 for $\\beta$ over $t$ iterations.\n$J(\\pi_{\\beta, t}; \\beta)$ can be calculated using value functions corresponding to $\\pi_{\\beta, t}$​, and the sampler is trained to output higher probabilities for $\\beta$ that yield a higher $J(\\pi_{\\beta, t}; \\beta)$ value.\n\nTo be more specific, policies are trained for each of several $\\beta$ using the method introduced in Section 5, and these policies are expressed as { $\\pi_{\\beta,t} | \\beta \\in B^N$ }.\nThen, we use these policies to calculate $J(\\pi_{\\beta,t};\\beta)$ and use these values to train the sampler.\nTo implement the set of policies, $\\beta$ along with the state are added as inputs to the actor-network $\\pi_\\theta(a|s ; \\beta)$, as detailed in Appendix C.\n\nAdditionally, the explanation on lines 237-238 does not mean that we need to find $\\max_\\pi J(\\pi; \\beta)$, but rather demonstrates that $J(\\pi; \\beta)$ can be used as a measure of how well RCRL is solved.\nIf there is any misunderstanding on our answer, please provide further clarification.\n\n**Q3 (Robbins-Monro condition):**\n\nSince it is possible to adjust the learning rate freely, the Robbins-Monro condition can be satisfied.\nFurthermore, as mentioned in the above response to Q2, there is no additional inner problem in calculating $J(\\pi;\\beta)$. \nTherefore, the method described in Section 6 does not need to satisfy the Robbins-Monro condition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We appreciate all the reviewers for their insightful comments and suggestions. 
In this response, we will address the common concern raised by the reviewers: convergence rate analysis and performance gap.

**Convergence rate**

We will analyze the convergence rate of the proposed method in Section 5 through finite-time analysis, similar to the approach used in CRPO [R1]. 
To achieve this, we make a few minor modifications to the policy update rule in line 169 as follows:
1. "if constraints are satisfied" is changed to "if $J_{C_i}(\pi_t)\leq d_i + \eta$ $\forall i$," where $\eta$ is a positive number.
2. The time varying step size $\alpha_t$ is changed to a fixed learning rate $\alpha$.

Then, the following is satisfied:
> Let $\alpha=(1-\gamma)^{2.5}/\sqrt{T}$ and $\eta = (2D + 20K^2)/((1-\gamma)^{2.5}\sqrt{T})$, where $D:=\sum_s d_\rho^{\pi^*}(s)D_\mathrm{KL}(\pi^*(\cdot|s)||\pi_0(\cdot|s))$ and $K:=\max(N\lambda_\mathrm{max}C_\mathrm{max}, \lambda_\mathrm{max}R_\mathrm{max}, R_\mathrm{max}, 1)$.  
> Then, $J_R(\pi^*) - \mathbb{E}\_{t\sim \mathcal{N}}[J_R(\pi_t)] \leq (2D + 20K^2)/((1-\gamma)^{2.5}\sqrt{T})$, and $\mathbb{E}\_{t\sim\mathcal{N}}[J_{C_i}(\pi_t)] - d_i \leq (2D + 20K^2)/((1-\gamma)^{2.5}\sqrt{T})$ for $\forall i$.

**Proof:**

By subtituting $|J_R(\pi^*)-J_R(\pi_t)|\leq 2R_\mathrm{max}/(1-\gamma)$ and $|J_{C_i}(\pi^*)-J_{C_i}(\pi_t)|\leq 2C_\mathrm{max}/(1-\gamma)$ into Eq. (30),  
$\sum_{t\in\mathcal{N}}(\alpha(J_R(\pi^*)-J_R(\pi_t))-2\alpha^2N\lambda_\mathrm{max}C_\mathrm{max}/(1-\gamma))+\sum_{t\notin\mathcal{N}}(\alpha\eta-2\alpha^2\lambda_\mathrm{max}R_\mathrm{max}/(1-\gamma))$  
$\leq D + \sum_{t\in\mathcal{N}}2\alpha^2(R_\mathrm{max}+\alpha N\lambda_\mathrm{max}C_\mathrm{max})^2/(1-\gamma)^5+\sum_{t\notin\mathcal{N}}2\alpha^2(\alpha\lambda_\mathrm{max}R_\mathrm{max}+N\lambda_\mathrm{max}C_\mathrm{max})^2/(1-\gamma)^5$.

Using the definition of $K$,
$\sum_{t\in\mathcal{N}}\alpha(J_R(\pi^*)-J_R(\pi_t))+\sum_{t\notin\mathcal{N}}\alpha\eta\leq D+2T\alpha^2K^2(\alpha+1)^2/(1-\gamma)^5+2T\alpha^2K/(1-\gamma)$.  
Also, due to $J_R(\pi^*)-J_R(\pi_t)\geq0$ for $t\in\mathcal{N}$, $(T-|\mathcal{N}|)\alpha\eta\leq D+2T\alpha^2K^2(\alpha+1)^2/(1-\gamma)^5+2T\alpha^2K/(1-\gamma)$.

Since $\alpha\eta T/2\geq D+2T\alpha^2K^2(\alpha+1)^2/(1-\gamma)^5+2T\alpha^2K/(1-\gamma)$, we can get $(T-|\mathcal{N}|)\alpha\eta\leq T\alpha\eta/2\Rightarrow |\mathcal{N}|\geq T/2$.

Then, $J_R(\pi^*)-\mathbb{E}\_{t\sim\mathcal{N}}[J_R(\pi_t)]=\sum_{t\in\mathcal{N}}(J_R(\pi^*)-J_R(\pi_t))/|\mathcal{N}|$  
$\leq2(D+2T\alpha^2K^2(\alpha+1)^2/(1-\gamma)^5+2T\alpha^2K/(1-\gamma))/(\alpha T)\leq(2D+20K^2)/((1-\gamma)^{2.5}\sqrt{T})$.

Also, $\mathbb{E}\_{t\sim\mathcal{N}}[J_{C_i}(\pi_t)]-d_i\leq\eta=(2D + 20K^2)/((1-\gamma)^{2.5}\sqrt{T})$.

**Performance Gap**

Let $A(d)=${ $\pi|\mathcal{R}\_{\sigma}(X)\leq d$ }, $\tilde{A}(d)=${ $\pi|\mathcal{R}\_{\tilde{\sigma}}(X)\leq d$ }, where $X=G_C^\pi$.
Then, the performance gap is $G:=\max_{\pi\in A(d)}J_R(\pi)-\max_{\pi\in\tilde{A}(d)}J_R(\pi)$.
According to Lemma 6.1, $|\mathcal{R}\_{\sigma}(G_C^\pi)-\mathcal{R}\_{\tilde{\sigma}}(G_C^\pi)|\leq K/M$, where $K=C_\mathrm{max}\sigma(1)/(1-\gamma)$, so $A(d-K/M) \subset \tilde{A}(d)$.
Thus, $G\leq\max_{\pi\in A(d)}J_R(\pi)-\max_{\pi\in A(d-K/M)}J_R(\pi)$.

As a result, the gap is bounded by the performance difference between the two thresholds.
However, in constrained optimization, this type of problem is challenging due to discontinuous changes in the feasible set or optimal points.
Thus, we will use some assumptions to analyze the gap practically, which will be introduced below.

Le $d_\rho^\pi(s,a):=(1-\gamma)\sum_t \gamma^t\mathbb{P}(s_t=s, a_t=a)$.
Then, $(1-\gamma)(J_R(\pi)-J_R(\pi'))=\mathbb{E}\_{(s,a)\sim d_\rho^\pi}[A_R^{\pi'}(s,a)]=\langle d_\rho^\pi,A_R^{\pi'}\rangle=\langle d_\rho^\pi-d_\rho^{\pi'},A_R^{\pi'}\rangle\leq||d_\rho^\pi-d_\rho^{\pi'}||\_1 R_\mathrm{max}/(1-\gamma)$.
The performance gap is expressed as $G=\max_{\pi'\in A'}\min_{\pi\in A}(J_R(\pi')-J_R(\pi))\leq\max_{\pi'\in A'}\min_{\pi\in A} ||d_\rho^{\pi'}-d_\rho^\pi||\_1 R_\mathrm{max}/(1-\gamma)^2$, where $A=A(d-K/M)$ and $A'=A(d)$.
As a result, we need to find $\max_{\pi'\in A'}\min_{\pi \in A}||d_\rho^{\pi'}-d_\rho^\pi||\_1$.

Since the one-norm above is the maximum distance between the two nearest policies in set $A$ and $A'$, we only need to evaluate the boundaries $\partial A$ and $\partial A'$.
If $\pi'$ on $\partial A'$ is given, we need to find the closest policy $\pi$ on $\partial A$: $\min_{\pi\in\partial A}||d_\rho^{\pi'}-d_\rho^\pi||\_1=\min_\pi||d_\rho^{\pi'}-d_\rho^\pi||\_1$ s.t. $J_C(\pi')-J_C(\pi)=K/M$.
Let $J_C(\pi)=\inf_g\mathcal{R}\_\sigma^{g}(\pi)=\mathcal{R}\_\sigma^{g^*}(\pi)$.
Then, $J_C(\pi')-J_C(\pi)\leq\mathcal{R}\_\sigma^{g^*}(\pi')-\mathcal{R}\_\sigma^{g^*}(\pi)=\langle d_\rho^{\pi'}-d_\rho^{\pi},A_{g*}^\pi\rangle/(1-\gamma)\leq||d_\rho^{\pi'}-d_\rho^\pi||\_1 C_\mathrm{max}/(1-\gamma)^2$.
Here, we assume that there exists an occupancy measure $d_\rho^{\pi}$ that satisfies the equality in the above equation.
Then, $\min_{\pi\in\partial A}||d_\rho^{\pi'}-d_\rho^\pi||\_1=K(1-\gamma)^2/(C_\mathrm{max}M)\Rightarrow\max_{\pi'\in A'}\min_{\pi\in A}||d_\rho^{\pi'}-d_\rho^\pi||\_1=K(1-\gamma)^2/(C_\mathrm{max}M)$.
Finally, the performance gap is bounded by $KR_\mathrm{max}/(C_\mathrm{max}M)$.

Without the assumption, it is not guaranteed that the performance gap will always be bounded by $KR_\mathrm{max}/(C_\mathrm{max}M)$.
Nevertheless,  an insight can be gained that the gap is influenced by the scale of rewards and costs rather than the state space and action space.
Therefore, it seems possible to conjecture that the proposed approximate approach is scalable.

**References**
- [R1] Xu et. al. "CRPO: A new approach for safe reinforcement learning with convergence guarantee." ICML, 2021.

Meta-review: The paper proposes a spectral risk measure-constrained RL algorithm, as well as a bilevel optimization approach. The proposed algorithm comes with optimality guarantees in the tabular setting, and the good performance is also demonstrated via numerical experiments. The reviewers appreciated the importance of the problem, strength of the results (both theoretical and experimental) and they praised the clarity of the paper. The rebuttal from the authors addressed the reviewers' concerns, and there is a consensus towards accepting the paper. I concur with this evaluation: this will be a nice addition to the NeurIPS program.
Decision: Accept (poster)