Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Authors: Whiyoung Jung
Myungsik Cho
Jongeui Park
Youngchul Sung
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Reinforcement Learning Theory
Constrained Reinforcement Learning
Safe Reinforcement Learning

Abstract: Constrained reinforcement learning (RL) is an area of RL whose objective is to find an optimal policy that maximizes expected cumulative return while satisfying a given constraint. Most of the previous constrained RL works consider expected cumulative sum cost as the constraint. However, optimization with this constraint cannot guarantee a target probability of outage event that the cumulative sum cost exceeds a given threshold. This paper proposes a framework, named Quantile Constrained RL (QCRL), to constrain the quantile of the distribution of the cumulative sum cost that is a necessary and sufficient condition to satisfy the outage constraint. This is the first work that tackles the issue of applying the policy gradient theorem to the quantile and provides theoretical results for approximating the gradient of the quantile. Based on the derived theoretical results and the technique of the Lagrange multiplier, we construct a constrained RL algorithm named Quantile Constrained Policy Optimization (QCPO). We use distributional RL with the Large Deviation Principle (LDP) to estimate quantiles and tail probability of the cumulative sum cost for the implementation of QCPO. The implemented algorithm satisfies the outage probability constraint after the training period.

Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Whiyoung Jung, Myungsik Cho, Jongeui Park, Youngchul Sung'
School of Electrical Engineering, KAIST
Daejeon 34141, Republic of Korea juyg, ms. cho, jongeui. park, yksung) ac. .rr
Abstract
Constrained reinforcement learning (RL) is an area of RL whose objective is find an optimal policy that maximizes expected cumulative return while satisfying given constraint. Most of the previous constrained RL works consider expected cumulative sum cost as the constraint. However, optimization with this constraint cannot guarantee target probability of outage event that the cumulative sum cost exceeds given threshold. This paper proposes a framework, named Quantile Con- strained RL (QCRL), to constrain the quantile of the distribution of the cumulative sum cost that is a necessary and sufficient condition to satisfy the outage constraint. This the first work that tackles the issue of applying the policy gradient theorem to the quantile and provides theoretical results for approximating the gradient of the quantile. Based on the derived thcoretical results and the technique of the Lagrange multiplier, we construct constrained RL algorithm named Quantile Constrained Policy Optimization (QCPO). We use distributional RL with the Large Deviation Principle (LDP) to estimate quantiles and tail probability of the cumulative sum cost for the implementation of QCPO. The implemented algorithm satisfies the outage probability constraint after the training period.
Introduction
Reinforcement learning (RL) has been developed in the direction of finding an optimal policy that maximizes expected cumulative return for a given environment. Thus, most of the works in RL consider only rewards given by the environment to optimize the policy. However, many real-world control problems impose constraints on the behavior of a policy. Constrained RI. is area of RI. whose objective to find an optimal policy that maximizes expected cumulative return while satisfying a certain constraint on the cumulative cost. A conventional constrained RI. problem (ExpCP) can be written as
Maximize subject to
V=(so) := E, I E (Tr(s) Ca (s) E, Et=o 2 2ec(s1, (i)) < dan.
(ExpCP)
where the cost constraint is that the expectation of the sum of costs is less than or equal to a threshold parameter dea. Note that the threshold des is on the average (i.e., expectation) of the cumulative sum cost to avoid undesired high-cost events in this formulation. If we do not want any event causing a positive cost, du should be set as sufficiently small value, i.e., deh ss 0. On the other hand, we can afford events with low costs, we can set de, properly as we desire. Most of the previous constrained RL works solved the problem (ExpCP)
25 28 29
partly because the
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

constraint on the expectation of the cumulative sum cost in (ExpCP) is well fit with the objective given by the expectation of the sum reward, and this makes the problem amenable. However, solving the problem (ExpCP) may have an undesirable outcome for real environments that typically need a constrained behavior on the event that the cost exceeds the threshold deh- For example, in the case of an autonomous driving car, what we want for our sure safety is to control and limit the probability of accident itself. In the case of telecommunication system, what we want to control is the probability of packet Ioss through the communication system. These probabilities are called outage probability" in general. Thus, in many real-world systems, the system requires a constraint on the outage probability, i.e., the probability of critical or unsafe events. In this case, constraint the expectation of critical events, as in (ExpCP), cannot guarantee the desired target probability of critical events. To illustrate this, let us consider the following example.
Consider a simple two-path environment, as shown in Fig. Ta The objective of the environ- ment for an agent to reach the goal by driving car. The environment gives reward when the agent reaches the goal, and gives costs until the agent reaches the goal. There exist two paths to reach the goal, and each path has different cost distribution. Fig. shows the distribution of the cumulative sum cost (curve) and its mean (vertical line) for each path: blue for path and red for path 2. As tion for path has a lower average than that for path 2, but has a longer right tail than that for path 2. If we use the expectation of cost constraint, then following path is better choice, but in this case, the probability of a high cumulative sum cost (e.g. > 10.0 in Fig. (b) is higher than following path 2. If the threshold of 10 represents catastrophic event, we should constrain the probability of events exceeding 10 to a small value. Thus, solving the problem with the expectation constraint (ExpCP) does not necessarily have precise control over the target outage probability. When the event that the cumulative sum cost exceeds deh is a critical unsafe event, this means that such critical event can occur in high probability even if we solve the constrained problem (ExpCP). Therefore, in this paper, we aim to solve the following constrained RL problem with an outage probability constraint:
Path
0.2
Path
Goal
15
(a) Two-path environment
(b) Distribution of cost
Figure 1: A simple example of environment.
in Fig. Tb the distribu-
Maximize
= 7'r(st, az) Subject to Pr -E= Y (e,SS, Ad) > del S ED
V (80) = E, 1>
(ProbCP)
for So = so, A (-IS,),, S.+1 ~ (--(S),AA).
Our approach to this problem is first to convert the outage probability constraint in (ProbCP) into quantile constraint oh-eo (so) == inf {x| Pr P 7'c(Se, AA) S x)) 2 1 - co} S deh which is equivalent to the outage probability constraint (See Fig. D. and then to solve the optimization:
min max (V*T(s)) - (nT-co (so) den)) X
where À is the Lagrange multiplier, based on pol- icy gradient with the parameterized policy. How- ever, we note that the policy gradient theorem, which is the most basic theorem for on-policy RL, cannot be applied directly to compute the gradient of ① with respect (w.r.t.) the policy parameter due to the quantile term qi (so). Therefore, we derive theoretical results for ap- proximating the gradient of the quantile (in Sec- tion B). Then, based on the derived theoretical results and the technique of the Lagrange multi- plier, we construct our algorithm named Quantile Constrained Policy Optimization (QCPO) solve the outage probability constrained RL problem (in Section 4. Here, we use distributional RL with the Large Deviations Principle (LDP) to estimate quantiles and tail probability of the cumulative sum cost for implementation of QCPO. The implemented algorithm satisfies the outage probability constraint after the training period. To the best of our knowledge, this is the first work that tackles the issue of applying the policy gradient theorem to the quantile and obtains (approximate) policy gradient for the quantile, and this one of the main contributions of this paper, together with the QCPO algorithm.
Area 1
Area co
---ea de
Figure 2: Equivalence between the outage proba- bility constraint and the quantile constraint
2

---

2 Background and Related Works
Constrained RL constrained Markov decision process (CMDP) is defined as a tuple (S, (S.A,T, T, C, M, 7), where S is the state space, A is the action space, : S x A + R is the re- ward function, : S x A - Rzo is the cost function, M xx [0, 1] is the state transition probability, and is the discount factor.
Constrained RL or Safe RL is an area of RL whose objective is find an optimal policy for a given CMDP that maximizes the expected return E,, 1- 7'r(s;, at) while satisfying a constraint on the cumulative sum cost Er= r'c(s; az). If one wants to constrain the average cumulative sum cost of the policy и,, then constraint (50) == E, D2 ('c((s, I den can be considered. On the other hand, if one wants to constrain the outage behavior of the policy then constraint Pr (x*())) > der) < En should be considered, where (=())) is a random variable defined as Ei=0 7'((S,, A1) with S. = Su. A (.)) S++1 ~ M(|S,, Ad) for ( = 0, 0,
X" (80)
Most of the previous constrained RL works considered constraint on the expectation of the cumula- tive sum cost: O* (so) S deh 12 24 [25 (8 29]. In order to solve the constrained optimization problem with this expectation-bast constraint, researchers considered the Lagrangian multiplier
method 1 21,, Lyapunov-based methods 6 7 iejjection-baeed methods [28] safety-layer methods
10
Table 1: Comparison of constrained RL with probabilistic constraints
Papers
Algorithm Type
Distribution Modeling
Theory
Deep RL
On-policy
No
Risk-Const. RL
Yes
No
(Trajectory-baset) (need only trajectory samples)
Gaussian
WCSAC 27
Off-policy
No
Yes
(on all range of distribution)
QCPO (This paper)
On-policy
LDP with Weibull
Yes
Yes
(State-basee
(only on tail)
Quantile (i.e., Value at Risk, VaR) and Conditional Value Risk (CVaR) two well-known tech- niques to manage undesirable events in the domain of finance 18]. In the context of RL, the definitions of the quantile and the CVaR for the distribution of the cumulative by q. (so) == inf{= Pr(X* (80) S x) 2 u} and CVaR == E, (30) (²(ss) respectively.
cost for given
given
) 2 o (so)],
The CVaR was previously used in RL to constrain undesirable events and the problem with a CVaR constraint is explicitly formulated as
Maximize Subject to
EE 12>30 70r((e, ar) CVaR (50) S den.
(CVaR-CP)
Note that the (1 CaR denoted CVaR -o (s) is always greater than equal the (1 6a)- quantile denoted as of- (s) for all E S because of the definition of the CVaR. Therefore, satisfying the CVaR constraint CVaR (sp) < der in (CVaR-CP) is sufficient condition for satisfying the probabilistic constraint Pr (Xx (so) > deh)) S ED in (robCPP). and hence (CVaR-CP) is stricter problem than (ProbCP). Therefore, algorithms proposed to solve (CVaR-CP) can be used for solving PPobbPP. and this should satisfy the probabilistic constraint in theory.
Chow et al. proposed trajectory- CVaR method and provided convergence for their method. They used trajectory-bart based policy gradient to their Lagrangian, and it is simple to compute. Like most trajectory-basee RL algorithms, however, it suffers from sample inefficiency since it collects a number of trajectories and updates its parameter once. Recently, Yang et al. [27] proposed an off-policy algorithm solve VCVRR-CP.. They only estimated the mean and variance of the cost distribution using a technique in distributional RI. and computed CVaR v, (s} F as the CVaR of the Gaussian distribution of the estimated mean and variance. However, the distribution of Xx x²(s) is not Gaussian in general, and the Gaussian approximation has limited capability to capture the decay rate of the tail probability because Gaussian probability density function (PDF) has the form of exp(- with fixed rate function x², Therefore, this algorithm can yield a poor estimation
2 They actually consider the CVaR CVaR (s, a) of the cumulative sum cost X (T,ss a) for given (s, a) pair.
3

---

of the CVaR of the tail, especially for small tail probability, and cannot guarantee to satisfy the CVaR constraint (see Section F). Furthermore, note that the CVaR and the quantile are two different measures for undesirable events, and the choice between the two depends on what we desire. For example, an insurance company prefers the CVVRR of undesirable events to determine an insurance premium. On the other hand, company developing an autonomous driving car system needs the quantile of undesirable events to guarantee the accident probability for safety. Thus, in the context of safe learning, our work focuses directly on the constraint on the quantile, which is equivalent (i.e., necessary and sufficient) constraint to the outage probability constraint in (ProbCP). To the best of our knowledge, this is the first work that provides state- baste policy gradient for the quantile and required theoretical results regarding the quantile-come constrained RL problem. Moreover, our implementation approximates the tail distribution of X (s) with a Weibull distribution (a particular case of generalized Gamma distribution), which is general enough to capture various rates of decay of the tail probability. Table D summarizes the previous constrained RL methods and this paper.
Large Deviation Principle (L.DP) Large deviation principle (LDP) 11 is a technique for es- timating the limiting behavior of a sequence of distributions, especially on the tail. A simple example is the empirical mean X, = # > Xx of i.i.d. random variables X,, We say that a sequence { satisfies LDP if the sequence of its log probability distribution 1. log Pr (X,, TT) satisfies the following condition 1 log Pr (Xn, E TT) "= -infzer I (xx) for some function [(x).. The function [xx) satisfying such limiting behavior called the rate function of X,. The
rate function 1 (x) is also related to the cumulative distribution function (CDF) Fx, (r) since
1 Fx (ro) = Pr (Xn E [so, x)) RJ exp (-n inf 20) (xx) for some xo > [[X] and suffi- ciently large n. LDP can be applied to finite-state Markov chains, and there exists a rate function for given Markov chain M. In this paper. we consider the tail probability of the distribution of the cu- mulative sum cost X (50) = Ei 7'c(st, a.). Finding its analytic rate function is hard. Therefore, we instead approximate the rate function directly as I (x) ss (#/3(s))a(a) with learnable param- eters (s)) and (ss), which results in a Weibull distribution: 1- - Fx (z) exp (xp(-(x/(s))eee We use this distribution to approximate the tail probability of (x) (x) of (s).
3 Quantile Constrained RL
In this section, explain equivalent form of (ProbCP) that we use to learn an optimal constrained policy under the outage probability constraint and then explain the difficulty of applying the policy gradient theorem to optimize the Lagrangian of the equivalent problem. Finally, we provide theoretical results that circumvent this difficulty in Section 3.2
3.1 Motivation: Problem of Applying Policy Gradient Theorem to Quantile
Solving (ProbCP) with a direct approach is too hard in making a loss function for л based on the outage probability. Thus, we convert the probability constrained problem to an equivalent form of a quantile constrained problem:
Maximize
Ex, ii r'r(s,, ar) Subject to T--e S den,
(QuantCP)
where (s) inf(x Pr(XX* (s) <x) ut is the u-quantile of the random variable (s) of the 7 A.) with So s, A. 7-11.)). S ~ (- A.). t = 0, 1, 2, .... Note that - (so) I de is equivalent to Pr [x > dan] < eg due to the definition of the quantile. We propose a direct approach to solve the equivalent problem (QuaniCP) instead of (ProbCP). Although we have an equivalent form of (ProbCP). it is still difficult to solve QuantCP). We explain what makes solving the problem (QuantCP) still hard below.
cumulative sum cost: (=) 2
In the case of (ExCCP) the Lagrange-based optimization of (EpCCP is given by minx20 max, Lezpl A) == V (so) - > (C" (su) - den), where (**(s)
E, E 7 (r at)
and C* (so) = E 1E - ' d'c(s, a.)]. Then, due to the form of C* (so), the policy gradient theorem 22 can directly be applied, and the gradient of the Lagrangian w.r.t. a is given by the expectation
form:
V ( -)= E0 (s) Σ Vi(als) (s) XA)(s,a)),
(2)

---

where p=(s) := Ei= y Pr(S: = s|su, T). A5 :=
:= r(s, + + [V* V" (s), and
(s a) c(s. a problem QuantCP)
w [C (s')] (s) However, the gradient of the Lagrangian of the
min max Lequant (T, A) V (50) - (8) dea))
(3)
A20
w.r.t. the policy ı cannot be expressed as an expectation form:
V n Luaatt (T, A) + Ex V log π (a|s) (s, (s,
(4)
where (s, a) == c(s, a) ~M 1 (s')] (s). This is because the u-quantile qu (s) is not the expectation of the cumulative sum cost. However, if the u-quantile (T) can be written as
00
qu (50) Ex
Σ r (c(s) ae) Eu(s)
(5)
for some function (s, a) that is independent of the policy т, we can apply the policy gradient theorem by defining the advantage function for the quantile term:
(s, = c(s,a) a) + (8, a) + 7E la (s')l a. (s).
(6)
This fact motivates us to search for such (s, a). For this, under mild assumptions, we first show the existence of policy-dependent additional cost e (s.a) and then show that the additional cost can be approximated by a cost ž (s, a) for some fixed independent of π except the requirement max, (L(((((-s)) || 7 (-1ss) I 6.
3.2 Theoretical Results
We here provide theoretical results showing the existence of additional cost (8, and showing that this can be approximated as another cost 255 (s, a) for base policy n' independent of æ, only requiring max, KL (+'(-11 (-Is) || T(-1s) I for some 8 > 0. These theoretical results make the quantile constrained policy optimization tractable by enabling application of the policy gradient theorem. For the theoretical results, we assume that the CDF Fx=(s) (z) strictly increasing on [0, 0o), and it is continuously differentiable for all S. The proofs of the theoretical results are in Appendix B
We begin with deriving the temporal-different (TD) relation between the u-quantiles of (*)(s) at 8t and st+1- Theorem states the TD relation for the u-quantile under the following assumptions of boundness of quantile difference and smoothness of CDF of X*
Assumption 1 (Boundness of quantile difference). For a given policy n, the following two quantities are bounded
(o(8, a) + 740 - (T((s) SRR (5(5)=((3,00 -(3,a) c(s,a) )) S R
(x) - -xx (Fxc(e)
(s)
(8)
?
for all (s,a, E S Ax S such that (aas)) (s'ls, a) 0.
Assumption (Smoothness of CDF of X (*)).. For each state s, the average slope of Fx*(s) between qi (8) and E (s) R, (s) + R] is bounded by
I-E (x)(s) (s)( < (x=(a) (x)( Fx)(o) (2)
-- (x= (s)
(9)
y
for small 0 <
Theorem 1. Under Assumptions 7 and 2 the u-quantile of the random variable xT((s) satisfies the following temporal-difference (TD) relation. For some constant R and small > 0,
Bx n (St, ac s1+1) (c(se, + (00(()41) - -[
T-ER. R,
(10)
where pi (8, a.))) Pxx())) ((()))c((0)))
(Px=(() (s)). Here, the expectation for the action 0r (t)) and the next state st+1 M (- a.). (St is given.)
5

---

Note that for the expectation of the cumulative sum cost (*(ss considered in ExpCPp. the expectation of TTD under the policy π follows
Er [c(se, az) + 7 ($c+1))))))))))) C* =0
(11)
by the Bellman equation. The TD relation (11) for expectation has a similar form to that for the u-quantile (TO), but the difference is that (TO) is the weighted expectation of the TD (c(St, at)
79 ($t+1) the (s:)). The numerator (x
1. (x)))) of the weight pe (St, de, St+1) in
(10) involves two quantities: 1) a target quantile ((()))c()) and 2) the PDF of the sum of costs
7
X" (8++1)= Ex= " (++4+1) starting from state 81+1- Here, the value T. (+)))-(()))))) is the target value of the sum of costs Ek=o 7 [((+++++1, ac+x+1) from the next state $141 such that the sum of costs Σx=o 7 (5 as+x) for a given pair (3t, ae) at is the u-quantile q.7 (st). Thus,
of the weight (5(ss ae, St41) in (10) is the probability of the event that the cumulative sum cost starting from (st, becomes the u-quantile qu (sx) at 8t from the perspective of the next state 8(+1. Based on Theorem [ we obtain the following corollary:
the numerator (x+())+1)
2 (1l1)))saa))
Corollary 1. Under Assumptions bounded as
7 andd 2
the u-quantile qū (8e) of the random variable X X*(st) is
(s() - Er Mi (st, as, 8(41) (c(st, a.) (O0()
-
1-c R.
(12)
Proof: Note that the term t, (5t) can go outside the expectation in (0) since the expectation is over
(ae, St41). From eq. (26 in Appendix A.II the expectation of the numerator of fot (st, Ce: S141)
v. (8t)
is the same as the the denominator of the weight, i.e., Ex (x 1) (x.) (st)), and this leads to E. [u] ax, S04 ])] = 1. So, we have the claim.
As seen in Corollary | the u-quantile s, can be approximated as weighted expectation of c(S,, 4z) + 70% (8++1), and the weight is proportional to (x+()) [[))))(())))))))) This means that the more probable is the pair (st, at, $(+1) to achieve q. (st), the higher weight is multiplied to c(st, + + (St+1) for approximating 9. (st). Furthermore, if we assume that the transition dynamics of CMDP are deterministic, i.e., $1+1 = h(st, ae) as in many real-world control problem, we can approximate the u-quantile qHl (so) 80 as the expectation of the sum of costs under distorted policy R... as stated in the following lemma:
Lemma 1. Suppose that the state transition dynamics are deterministic, i.e., 8141 h(st, Then, under Assumptions and 2
the u u-quantile qu (su) of the random variable X=(ss) is expressed as
q (()()) - Ex.
Σ t=
00
2r(c(s), at)
S (1-c))1-7)
eR
(13)
where म. (als) = T(als): phe (s, a, h(s, a)) x T(als) Px=(h(sa))
O (())))((aa)))
Now, plugging (33) into the quantile term in the Lagrangian of the problem (QuantCP. we may apply the policy gradient theorem based on the chain rule since the u-quantile qu (5p) expressed as the expectation of the sum of costs. However, the gradient of 7, w.r.t. æ for chain rule is too
complicated due to the μ" term in Lemma [ Thus, we find another expectation form of 4% (sp) using
an additional cost function ट. (s, a), as stated in the following theorem:
Theorem Under deterministic dynamics s expressed as
h(se, and Assumptions
and
q. (s) can be
(o0) En
Σ
r't (d(s), + 25d (50, arl)
S
cR
(14)
(1 - (1 7)
where (3,8 = (NT (8, a, li(s, a)) 1) [c(s,a) a) + 7 (1(s
Note that the additional cost दي (st, a;) in Theorem 2 is a policy-dependent cost function. Under an additional mild assumption, we can find upper bound of (4) which replaces the policy-dependent cost function ट5 (s.a) with another cost क (s, a) for some fixed x' independent of π, only requiring max, KL( (7'(-1s || T-1ss) < d for some d > 0. The additional assumption is as follows:

---

Assumption& 3 (Lipschitz continuity of (s, a) over T). For any given fixed u E (0.1) and any policies x and n', there exists a coefficient C, such that
(s) (s) - C. max KL (a'(1ss)) ((1s')) S,EE
(15)
Basically, Assumption 3 is that the function e function of π is continuous, which is expected to be satisfied if there is no abrupt change in the associated distributions. With Assumption [3 and Theorem 2 we obtain an expression for the quantile q. (so) as a form of desired expected sum:
Theorem 3. Under deterministic dynamics 81+1 = h((s, ae) and Assumptions | 2 and 3 the
u-quantile qi (8a) is expressed the expectation of the sum of actual cost and π r--indppendent
additional cost ctd (s.a) for X' satisfying max, (L( (-1s) |I x x(-1s) S 6:
(c (n
eR
Cu
Co (80)
8.
(16)
(1 (1 (17))
1 1-7
By Theorem 3 we can approximate the u-quantile q (sa) as the expectation of the sum of costs plus w-independent additional costs द.. (s, a) for a base policy n', and this approximation is tighter when the distance between the current policy т and the base policy x' is smaller. Theorem Caan be interpreted the other way around. As in the case of PPO [20]. if we first simply set n' the policy before the update, denoted as Told. then the updated æ is near from the base policy Told = T', and we can compute the corresponding KL distance between Told (= r') and x. Then, still, the inequality (16) holds for 5 = max; KL (Toaa ([)][[[(((ss) Now, this result enables us to solve the quantile constrained problem (QuaniCP) by applying the policy gradient theorem.
4 Quantile Constrained Policy Optimization
Using the theoretical results in Section 3, we now construct algorithm named quantile con- strained policy optimization (QCPO) to solve QQantCCP based on on-policy RL algorithm: PPO 120]. The QCPO is a direct method to constrain the outage probability and consists of three parts: 1) estimation of the u-quantile qu (s) of (*)(s for a given policy æ, 2) estima- tion of PDF of X (s) to compute the additional cost aيp (s, a), and 3) updating method of the Lagrange multiplier to control the outage prob- ability. We first explain the overall structure of QCPO and the base loss function for the policy, which has similar form to that in [1). Then, we provide a condition for policy improvement for the proposed method. The implementation of the proposed algorithm is based on the imple- mentation of (21). and details of the implemen- tation, including the network structure, the loss functions, the Lagrangian multiplier update method, and the hyper-parameterss are in Appendix github.com/wyjunn hibhubbcoo/wyjung005//000P0
Weibull(a,, 3)
96.1 96.3 96.5 L8.90. 9..9
Figure 3: Illustration of the quantile approximation and tail- probbabbbbbyy approximation. Green-dash curve is the unknown PDF (x+(s) and up-arrow points are the estimated u-quantiles (s). We approximate the PDF (x)(s) on the right tail by a Weibull distribution (blue curve) and this is approximated using the rightmost quantile points (blue arrow).
The implementation code of QCPO is available at
4.1 Overall Structure of QCPO
The agent of QCPO uses function approximators for the policy π, the value function (*(())
E, 12 7'r(st, az)] and the quantile function (s) -ūm (s), Th
(s) of (*)
E. n'e(s,, a;) for the policy æ. These functions are parameterized by deep neural networks with parameters 0. Φi and th respectively. We denote Oatd. Data, and vootd as their old parameters. Note that the quantile function (s) outputs nhy values, and the i-th value represents an estimate of the w;-
quantile q., (s) of X (s) for fixed target CDF values [u1, U2 .., Un,] with U1 U2 < < Uny
7

---

The basic loss function of In addition to these parameterized functions, we need another function that approximates the PDF of X*(s) on the right tail. (Here, the right-tail tail probability most matters since the target outage probability is typically small.)
As aforementioned in Section 2 know that (s) follows LDP with rate function (x*(a) (Ir).
However, finding the rate function (x)(())) in analytic approach is hard. Therefore, in QCPO,
the agent approximates the rate function of the form of (x) = (a/B(s)) and learns the state-dependent parameters a(8) and (s)) by using the quantiles on the right tail approximated by its quantile function for the right tail: Qun.- (8), ....> lüng (s). This approximation of the rate function results in approximation probability on the right tail as a Weibull distribution, whose tail
distribution is 1- (x) (1) (ce/B())))) In order to obtain the state-dependent parameters ass) and (ss) for right-tail distribution approximation, we again parameterize them by neural networks with parameters & and (. respectively. Fig. 3 shows both the quantile approximation and right-tail approximation of QCPO. Note that our approach actually learns the rate function governing the tail- puil-probability decay rate, whereas the previous Gaussian approximation [27] on the PDF of X (s) fixes the rate function as quadratic x², which is not the correct rate function in general.
The overall procedure of QCPO is follows: 1) estimate the value function V*(s) for return
and estimate the quantile function q, (s). u [u1, U2,..., Un, for the cumulative sum cost, 2) approximate tail distribution (x* (z) on the right tail using Weibull distribution with parameters (s)) ((s). 3) compute the additional cost for the base policy n², -7- (s,a) for the quantile advantage AT-ro (s, a) == c(s, a) + (s, + 7-T--o (s') 9T-co (s) 4) take policy gradient using the sum of the value advantage and the quantile advantage (s, a) -AAT-a (s. a). 5) update the Lagrange multiplier A. Since QCPO is based on PPO [20], the loss functions for the policy and
the value function are similar to those of PPO (20). Please see Appendix E.2 and E.3 for detail.
4.2 Policy Loss Function and Policy Improvement Condition
Let consider the policy loss function of QCPO to solve (QuantCP) QCPO for a given Lagrange multiplier X is given by
L (me) Ö₁ max (L(TTaa(-1s) na (-(ss)
(17)
where
Toood (T) (Viuld Aqp (80)) + Barpruud, [Apoid (8), AA (st, a)]
(18)
A Totd (s, a) r(s, a) + 71E [V Fodd (s')] VFold (s)
(19)
A (s, a) c(s, a) + - (s, a) + (( sold (5'] ( (
(20)
and Vold = Thas the policy that collects the most recent batch of samples, (s) =
Ee rd Pr(S. s|s0: Tote) is the stationary state distribution under Tolds C₁ a constant (see Appendix B.5p. Now, consider the relationship between the actually-desired maximization ob-
jective L Liuaan( TT, 2) = (=((s) - A (g (sv) den) in (3) and the practical QCPO objective Lou (0) C1 max, (LTaa(())s) || NH (-1s) in 17 The relationship between the two is given by
the following theorem.
Theorem 4. Let knew = nee be the solution of the problem of maximizing
LF ( - Ö₁ max KL(Tooa(-1s) || mT(+1s))
(21)
for some constans Ōn O. Then, under deterministic dynamics st+1 h(se, as) and Assumptions and 3 the following inequality holds:
Lyuand
(osane(Tnees (Tnew: A) Lquone (Told A)
(22)
L Told (Tnww) Rold (Tota) Ö KLmaz (Told Tnow)
(23)
I-e
approximation loss
for given Lagrange multiplier À > 0, some constant O2 and small 0.
8

---

Note that the term C2 in 23 is due to our approximation of the quantile as an expected sum to apply policy gradient. Therefore, by Theorem 4 when the improvement - (Trew) L (Totd) C1 KLKLmar (Tota) [new) (> 0) by the policy update from the QCPO loss function is large enough to compensate for the approximation loss, the desired quantity will also be improved by our policy update. That is, the Lagrangian for the quantile constrained problem for Tnew will be higher than that for Told-
Experiments
5.1 Environments
(a) SimpleButtonEnv
(b) DynamicEny
(c) GremlinEnv
We examined the performance of the proposed QCPO and com- pared it to that of WCSAC, which uses the CVaR constraint. The environments we considered are SimpleButtonEnny Dy- nnmicEnv (27], and GremlinEnv, which are based on Safety Gym [17]. MuJoCo [23], and OpenAl Gym The environments can be considered as simplified versions of a real environment of an automatic serving robot and are illustrated in Fig. 4 The goal of these environments is for a robot (red sphere) reach a goal (orange sphere wrapped by a grey translucent pillar, or green pillar) while avoiding the non-goal button (orange sphere), hazards (blue circle) or moving gremlins (purple box). Once the robot reaches the current goal, the environments generate the next goal deterninistically (Sttole or randomly (DynamicEnv, GremlinEnv), so the task complexity increases in the order of SimpleButtonFny DynamicEnv, and GremlinEnv. When the robot performs an action at time f, it receives a reward P1 Proal 2 llPt - Paeallly (|ا] 1 goal reached. where is the position (æ, y) of the robot at time and Proal is the current goal position at time t. It also receives a cost +1 if the gremlin) and 0 otherwise. Thus, for the robot, it receives a higher return when the robot touches more goals in maximum timesteps T = 1000, and higher of costs when the robot touches one of the other objects more often. A more detailed explanation of the environments is in Appendix
Figure 4: The considered environments
robot touches non-goal object (the non non-goal button, a hazard,
5.2 Empirical Results
We compared the performance of the proposed algorithm (QCPO) with that of PPO with the La- grangian multiplier method (PPO Lag] for (ExpCP) and that of WCSAC [27 for CVaR-CT CP which is a stricter problem than (ProbCP). We set the threshold des 15 in (EpCP]. CVVR and QQantCCP and the target outage probability 60 = 0.1,
0.1,0.. 0.2 in (CVa - and (QuantCP).
Fig. 5 shows the results of the considered algorithms SimpleButtonEnw, DynamicEnv. and Gremli- nEnv. All experiments were done with 10 different random seeds, and the real line and the shaded area represent the average and average + standard deviation, respectively. PPO with the Lagrangian multiplier method for (ExpCP) (green) keeps the average of the sum cost around the threshold de = 15 well (please see the graph in Appendix DD33), and its outage probability becomes around 0.35 as we can observe in Fig. [dd] [5 [5d As aforementioned, the CVaR approach (WCSAC) should satisfy sufficient condition for satisfying the outage probability constraint in (ProbCPj. It is seen that WCSAC (En = 0.2 (purple), 6o = 0.1 (red)) achieves a lower or similar outage probability to the threshold 6o in Fig. 5d but the algorithm does not satisfy the outage probability constraint exactly Fig. 5e and 51 This means that the Gaussian distribution approximation of the distribution of (x()) has limited capability to capture the decay rate of the tail probability. On the other hand, the proposed QCPO (Eo = 0.2 (blue), 6g = 0.1 (orange)) maintains the outage probability around the desired target outage probability very well, as shown in Fig. 5d 5e
and 5f
Now consider the average return of these algorithms. allowed to have higher sum of costs, then it has a higher return. Thus, as seen in Fig. [d [5], and PPO Lag induces the highest outage probability, so it has the highest average return, as shown The direct comparison between WCSAC and QCPO is less meaningful in
constrained RI, in general, if an algorithm
5f
Fig.
5a 5b
and
5c
We used the implementation code in http:p///gttabo..oocetttttt/t//p
projects/safo (MIT License)
Wee used the github code that the authors of the paper uploaded: https://ggithu..coo/A/.gg//
WCSAC
(MIT License)

---

3
(a) Average Return
(b) Average Return
(c) Average Return
Aitat 0.00N
WSAC XN0R
04
(d) Outage Probability
(c) Outage Probability
(f) Outage Probability
Figure (left) SimpleButtonEnv, (middle) DynamicEnv, and (right) GremlinEnv: (upper row)
average return and (lower row) outage probability of the most current 100 episodes.
DynamicEnv and GremlinEnv, since WCSAC does not satisfy the outage probability constraint, but it is fair in BumpleButtonEny because both algorithms satisfy the outage probability constraint. As seen in Fig. 5a QCPO achieves higher average return than WCSAC for the same target probability constraint eg 0.1, 0.2. This is because QCPO satisfies the target outage probability exactly, i.e., uses the given cost budget fully for a higher return. We provided more results in Appendix
Conclusion
We have proposed the framework of quantile-constrained RL to constrain the outage probability by adopting constraint on the quantile, which is equivalent to the outage probability constraint. We have investigated issues in applying the policy gradient theorem to Lagrangian of the quantile- constrained RL problem and have converted the quantile into an additive form of costs so that the application of the policy gradient theorem feasible. Based on our derivation, we have constructed the QCPO algorithm, which uses distributional RL techniques to learn the u-quantile of the cumulative sum cost (s), and Weibull distribution to approximate the tail distribution of X (s). We also proved the policy improvement condition for QCPO and showed that there exists approximation loss due to our approximation of the quantile. Empirical results show that QCPO constrains the outage probability well the desired target value. The meaning of such exact satisfaction of the outage probability is two-fold: First, the constraint on the outage probability is satisfied to control the probability of unsafe events, and second, the exact satisfaction of the cost constraint enables us to exploit the cost budget fully and obtain a higher return. Empirical results demonstrated the effectiveness of the proposed scheme.
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) NN220022-0- Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones, 50%) and by Institute Information Communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (No. No.2022-0-00124 Development of Artificial Intelligence Technology for Self-Improving Competency-Ane Aware Learning Capabilities, 50%)
10

---

References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pages 22-31. PMLR, 2017.
[2] Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deter- ministic policy gradients. In International Conference on Learning Representations, Apr 2018.
URL https:://penrrrvier wwwtteorum/e/eeeeeeeeeeeeeeeettt Syyimmph
[3] Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Leurning, pages 449-458. PMLR, 2017.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang. and Wojciech Zaremba. Openai gym, 2016.
[5] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrainee reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 1 18(1):6070-611 61220, 2017.
[6] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman and Mohammad Ghavamzadeh. A Lyapunov-based approach safe reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, Dec 2018.
[7] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-base based safe policy optimization for continuous control. arXiv preprint arXiv:1901.1003 2019.
[8] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning, In International Conference on Machine Learning, volume 80, pages 1096-1 105. PMLR, Jul 2018
[9] Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[10] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv: 18X1:/800100757 2018.
[11] Amir Dembo and Ofer Zeitouni. Large Deviations Techniques and Applications. Springer, 1998.
[12] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference Artificial Intelligence and Statistics, pages 3304-3312, PMLR, 2021.
[13] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, volume 119, pages 5556-5566. PMLR, 2020.
[14] Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: Interior interior-point policy optimization under constraints. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 4940-4947, 2020.
[15] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[16] Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning, pages 4424-4434. PMLR, 2019.
[17] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration deep rein- forcement learning. 2019,

---

[18] R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of Banking Finance, 26(7):1443-14) 1471, 2002.
[19] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, volume 37, pages 1889-1897. PMLR, 2015.
[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06347 2017.
[21] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In International Conference on Machine Learning, volume 1 119, pages 9133-9143. PMLR, 2020.
[22] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[23] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 0026-5033. IEEE, 2012.
[24] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. In Advances in Neural Information Processing Systems, volume 33, 2020.
[25] Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 1 14880-11491, PMLR, 2021.
[26] Derek Yang. Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. In Advances in neural information 2019.
processing systems, volume 32, pages
[27] Qisong Yang, Thiago D. Simão, Simon H Tindemans, and Matthijs T. J. Spaan. WCSAC: Worst-case soft actor critic for safety-constraiied reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10639-10646, 2021.
[28] Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection based constrained policy optimization. In Internationall Conference on Learning Representations, 2020. URL https: /tttps://openrevieee wewt//oorrrrid--eeeetttt.tt
[29] Ming Yu, Zhuoran Yang. Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. In Advances in Neural Information Processing Systems, volume 32, pages 3127-3139, 2019.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We mentioned theoretical bound and approximation loss Section 3ectionBandd]3 and
(c) Did you discuss any potential negative socictal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss] have read the guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please see Appendix B
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Appendix
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] The implemen- tation code and instructions are uploaded in github. Please see github address in page
7.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Appendix E
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] Experimental results show mean and standard deviation with 10 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix [
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Please see footnotes in page 9.
(b) Did you mention the license of the assets? [Yes] Please footnotes in page 9.
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Please see github address in page 7.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "fuilKLoJzI3", "reviewer": "Reviewer_doKY", "summary": "This paper considers the problem of solving a constrained MDP under quantile constrained, i.e., that the tail probability of cost is below a certain desired value.  The problem falls under the general framework of risk constrained reinforcement learning and is relevant to systems where a stranger metric than expected cost constraints are desired.  The approach is to derive a proxy for the policy gradient of the quantile probability, and utilize it in a policy gradient approach to ensure the constraint is met.  The derivation of the appropriate policy gradient term is the major contribution.", "questions": "Please see weaknesses.  Addressing the gaps in the logic, namely deterministic transition, nearness of base policy to current policy and hence finding delta and using the bound in in the implementation should be addressed.  I would increase my score if this is satisfactorily addressed.", "limitations": "Yes, it appears to be sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The paper makes a novel observation in obtaining the policy gradient of the Lagrange dual of the risk constrained problem. The policy gradient in this case depends not he tall probability itself, as well as a const that is upper bounded by what appears to be a policy independent term.  \n\n+ The authors use a standard PPO codebase to implement their approach and verify its performance in relevant scenarios.\n\nWeaknesses\n\n\n- The imprecations of Theorem 3 are unclear.  It appears that the result depends on deterministic transitions, and the availability of a base policy $\\pi'$  whose KL divergence from $\\pi$ should be less than $\\delta.$  How doe we obtain $\\pi'$?  Furthermore, the actual problem does not have deterministic transitions.  How does this result apply to the problem?\n\n- Related to the above question, in section 4, step 2 is on estimating $X^\\pi(s)$ to compute the cost $\\tilde{c}^{\\pi'}_u(s,a).$  How is this done?  The last paragraph of the section in step 3 says that we must compute the additional cost, again without showing how to do it.  Appendix E.2 only presents loss functions, not how this step is accomplished.  \n\n- The above seem to be fundamental to the analytical claims of the approach as well as its implementation.  Otherwise, the implementation would become some kind of constraint on the rate function of the occupancy measure generated by the policy.  I am not sure if the implantation is consistent with the theory developed.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CLK6ZoNdeI3", "reviewer": "Reviewer_mHDz", "summary": "The paper \"Quantile Constrained Reinforcement Learning\" studies the problem of reinforcement learning for constraining outage probability. The paper converts this constraint into an equivalent quantile constraint. The policy gradient theorem can not be directly applied, due to the lack of a closed form gradient under this constraint. The paper uses large deviations principle and derives an \"approximate\" gradient for the quantile. Using this, a policy gradient based reinforcement learning algorithm \"Quantile Constrained Policy Optimization\" (QCPO) is proposed. For theoretical result, an improvement theorem is provided. The experimental results show the improvement over existing methods.\n\n", "questions": "1. Please state the assumptions used in the theorems clearly.\n\n2. The improvements shown in the approximations degrade as the complexity of environments increase. For example, the algorithm seems to perform better in simple button environment compared to other algorithms, but in Dynamic Env, its performance benefits are not much.", "limitations": "The limitations are not discussed in the paper. The authors are suggested to include a discussion in the paper about the possible limitations. The problem has a probability constraint, hence, with a small probability, the catastrophic events happen.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "So far the literature is mainly focused on expected cumulative sum cost constraint, very few policy gradient results exist for VAR and CVAR.  It is not a straightforward derivation of policy gradient for risk constrained problems as in expected cost constraint. This paper tries to overcome this issue by computing approximate gradients. The technical contribution is sound, and the paper is readable. These results are significant in the risk constrained RL literature. That said, in the main paper, the theoretical results have many assumptions. The repeated usage of phase \"under the mild assumptions\" is misleading (Theorem 1, Corollary 1, Lemma 1, Theorem 2, etc). Authors should clearly state the assumptions. In line 204, transition dynamics of the CMDP are assumed deterministic for the theorems to work. This has to be stated as an assumption. The experimental evaluation part is too simple and weak.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8IEko9UrAX4", "reviewer": "Reviewer_BZhD", "summary": "This paper proposes a framework, named Quantile Constrained RL (QCRL), to constrain the quantile of the distribution of cumulative\nsum cost that is a necessary and sufficient condition to satisfy the outage constraint. The paper converts the outage probability constraint in (ProbCP) into a quantile constraint which is equivalent to the original probabilistic constraint and solve the equivalent optimization using  Lagrange multiplier. ", "questions": "1. How to know that the underlying distribution of the sum of costs is right-tailed in practice if we do not have much information about the true environment. How does this method perform in practice if the model assumption is mis-specified. For example, what if the real distribution is left-tailed or multi-modal?\n2. Similar to Q1, how to choose proper values of $\\epsilon_0$ and $d_{th}$ if there is no much information about the environment.\n3. The experiments are done with 10 different seeds while the variance seems to be large. Is the proposed method sampling efficient and how the estimations especially the quantiles are affected by the data sample?\n4. There are a lot of studies about Distributional RL which uses the Huber loss to learn the quantiles. Can this approach be applied here?", "limitations": "This paper considers a more general more general outage probability constraint for Constrained Reinforcement Learning. However, this paper fails to provide enough insight about this setup and the experimental results do not provide a strong support. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. This paper considers the more general outage probability constraint in (ProbCP) than the classic expectation constraint (ExpCP) and modifies the optimization problem using quantile constraint.\n2. The paper applies policy gradient theorem to the quantile and provides theoretical results for approximating the gradient of the quantile. \n\nWeakness:\n\n1. This paper does not fully clarify why it is necessary to employ the outage probability constraint in practice and what may be the potential impact of this setting to the real-world examples.\n2. The empirical part is not strong enough to show the advantage of the proposed method, and some more realistic and interesting examples such as the autonomous driving or the telecommunication system that are mentioned by the authors may be helpful. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6uXrsFfRkWm", "reviewer": "Reviewer_uaQ3", "summary": "This paper proposes a framework QCRL to constrain the quantile of the distribution of cumulative sum cost, which is equal to guarantee a target probability of outage event that the cumulative sum cost exceeds a given threshold. This work is a good complement work for the constrained reinforcement learning area by considering a \"new\" quantile constraint. By constraining the quantile (instead of the expectation) of the cumulative sum cost,  we can have \"precise\" control over the target outage probability, which is very critical in practice , especially when high-cost event is unsafe.\n\nThis paper can be seen as a great work following WCSAC, the main contributions of this paper are: (1) better estimation for the quantile of the sum cost (2) focusing directly on the constraint on quantile.", "questions": "\nIn this paper,  three different constrained RL problems and related algorithms are discussed:  (1) ExpCp (2) ProbCP ( equivalent to QuantCP) (3) CVaR-CP.  The manuscript would benefit from adding a formal formulation for CVaR constraints problems with the similar type as the other two problems.  Besides, the WCSAC is not for ProbCP,  but for CVaR-CP.\n\nThe second question is about the experiments. The authors compare three different methods, QCPO, PPO_Lag, and WCSAC. However, these three methods are for three different problems (ExpCP, ProCP, CVaR-CP) respectively. It is not surprising that QCPO is the best in terms of controlling outage probability. Is that fair to compare the performance for different methods when given different constraints? What are these experiments designed for? \n\nThe third issue is about the results on DynamicEnv-v0.  WCSAC does get a bad estimation of the tail probability such that we can not compare the return of WCSAC with QCPO later.  How about adding another experiment of WCSAC by replacing the quantile and tail estimation part with the proposed method in section 4.1.  \n\nAnother question is about the range of outage probability $\\epsilon_0$? If the $\\epsilon$ is smaller (say 0.05, 0.02), could the proposed method still work well? What if the $\\epsilon$ is very large (say 0.5)?\n \nThe last question is about the conclusion \"the implementation algorithm learns an optimal policy while it keeps satisfying the outage probability constraint during its learning process\".   I am confused about whether the constraint is always satisfied when you use a Lagrange-based optimization and also keep updating the Lagrange multiplier $\\lambda$ during the learning procedure. It is also obvious from the experimental results that the constraint is not always satisfied, especially for the early stage. \n\n#################\nupdate:  Thanks for authors' response, which has addressed most of my concerns.  I would like to see the results of WCSAC with a Weibull approximation included in the final version.  Therefore, I would like to keep my score as 7.", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find it is easy to understand the ideas, motivation, and methods of this paper, results are also well-understandable. Overall, I think this paper gives a very critical and interesting setting for constrained RL theoretically and practically. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept