Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints
Authors: Dohyeong Kim
Kyungjae Lee
Songhwai Oh
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement learning
Safety
Multiple Constraints
Distributional Critic
TL;DR: We propose a safe reinforcement learning method based on the trust region method and distributional critics for multiple constraints.

Abstract: In safety-critical robotic tasks, potential failures must be reduced, and multiple constraints must be met, such as avoiding collisions, limiting energy consumption, and maintaining balance.
Thus, applying safe reinforcement learning (RL) in such robotic tasks requires to handle multiple constraints and use risk-averse constraints rather than risk-neutral constraints.
To this end, we propose a trust region-based safe RL algorithm for multiple constraints called a safe distributional actor-critic (SDAC).
Our main contributions are as follows: 1) introducing a gradient integration method to manage infeasibility issues in multi-constrained problems, ensuring theoretical convergence, and 2) developing a TD($\lambda$) target distribution to estimate risk-averse constraints with low biases. 
We evaluate SDAC through extensive experiments involving multi- and single-constrained robotic tasks.
While maintaining high scores, SDAC shows 1.93 times fewer steps to satisfy all constraints in multi-constrained tasks and 1.78 times fewer constraint violations in single-constrained tasks compared to safe RL baselines.
Code is available at: https://github.com/rllab-snu/Safe-Distributional-Actor-Critic.

Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints
Dohyeong Kim', Kyungjae Lee", and Songhwai Oh
Dep. of Electrical and Computer Engineering and ASRI, Scoul National University
2 Artificial Intelligence Graduate School, Chung-Ang University dohyeong. kimērllab. smu kr, kyungjae leelaieee.... cau. ac.kr, songhwai Baggwaa@ssnu.co ar.
Abstract
In safety-critical robotic tasks, potential failures must be reduced, and multiple constraints must be met, such as avoiding collisions, limiting energy consumption, and maintaining balance. Thus, applying safe reinforcement learning (RL) in such robotic tasks requires to handle multiple constraints and use risk-averse constraints rather than risk-neutral constraints. To this end, we propose trust region-based safe RI. algorithm for multiple constraints called safe distributional actor-critic (SDAC). Our main contributions are follows: I) introducing gradient integration method to manage infeasibility issues in multi-constrained problems, ensuring theoretical convergence, and 2) developing TD(A) target distribution to estimate risk-averse constraints with low biases. We evaluate SDAC through extensive experiments involving multi- and single-constrained robotic tasks. While maintaining high scores, SDAC shows 1.93 times fewer steps to satisfy all constraints in multi-constrained tasks and 1.78 times fewer constraint violations in single-constrained tasks compared to safe RI. baselines. Code available at: https: //gi tiu.......h.. com com/rllab aalaaab-sssuuaeeeeeeeeeeee
snw.snuuuaaaaae SSffe www.trrbbiinnal-eeeee
Introduction
Deep reinforcement learning (RL) enables reliable control complex robots [Merel et al. 2020 Peng al.) [ Rudin al. [002]. In order to successfully apply RL to real-world systems, systems,, it is essential to design proper reward function which reflects safety guidelines, such as collision avoidance and limited energy consumption, as well as the goal of the given task. However, finding the reward function that considers all such factors involves a cumbersome and time-consumee conee-conuminngg process since RL algorithms must be repeatedly performed to verify the results of the designed reward function. Instead, safe RL, which handles safety guidelines as constraints, is an appropriate solution. A safe RL problem can be formulated using constrained Markov decision process [Altman 1999]. where not only the reward but also cost functions are defined to provide the safety guideline signals. By defining constraints using expectation risk measures of the sum of costs, safe RL aims to maximize returns while satisfying the constraints. Under the safe RL framework, the training process becomes straightforward since there is need search for reward that reflects the safety guidelines.
While various safe RL algorithms have been proposed to deal with the safety guidelines, their applicability to general robotic applications remains limited due to the insufficiency in 1) handling multiple constraints and 2) minimizing failures, such as robot breakdowns after collisions. First, many safety-critical applications require multiple constraints, such maintaining distance from obstacles, limiting operational space, and preventing falls. Lagrange-based based safe RL methods (Yang et al. 2021 Zhang and Weng. 2022 Bai al. 2022 which convert safe RL problem into dual problem and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

update the policy and Lagrange multipliers, are commonly used to solve these multi- conutiiennttrinee problems. However, the Lagrangian methods are difficult to guarantee satisfying constraints during training theoretically, and the training process can be unstable due to the multipliers [Stooke et al. 2020]- To this end, trust region-basee -aased methods (Yang et al. 2020 Kim and Ohl [202aa] J which can ensure to improve returns while satisfying the constraints under tabular settings [Achiam et al. [017]. have been proposed as an alternative to stabilize the training process. Still, trust region-based methods have a critical issue. Depending on the initial policy settings, there can be infeasible starting case, meaning that no policy within the trust region satisfies constraints. To address this issue, we can sequentially select a violated constraint and update the policy to reduce the selected constraint (Xu eet al. 2021]. However, this can be inefficient as only one constraint is considered per update. It will be better to handle multiple constraints once, but it is a remaining problem to find a policy gradient that reflects several constraints and guarantees to reach a feasible policy set.
Secondly, as RL settings are inherently stochastic, employing risk-neutral measures like expectation to define constraints can lead to frequent failures. Hence, it is crucial to define constraints using risk measures, such conditional value at risk (CVaR), as they can reduce the potential for massive cost returns by emphasizing tail distributions (Yang et al. 2021 Kim and Oh 2022a] In safe RL, critics are used to estimate the constraint values. Especially, y. to estimate constraints based on risk measures, it is required to use distributional critics (Dabney et al. 2018b). which can be trained using the distributional Bellman update (Bellemare et al. 2017]. However, the Bellman update only considers the one-step temporal difference, which can induce a large bias. The estimation bias makes it difficult for critics judge the policy, which can lead to the policy becoming overly conservative or risky, as shown in Section 5.3 In particular, when there are multiple constraints, the likelihood of deriving incorrect policy gradients due estimation errors grows exponentially. Therefore, there is need for a method that can train distributional crities with low biases.
In this paper, we propose a trust region-based safe RL algorithm called safe distributional actor-critic (SDAC), designed to effectively manage multiple constraints and estimate risk-averse constraints with low biases. First, to handle the infeasible starting case by considering all constraints simultaneously, we propose a gradient integration method that projects unsafe policies into a feasible policy set by solving quadratic program (QP) consisting of gradients of all constraints. It guarantees to obtain fcasible policy within a finite time under mild technical assumptions, and we experimentally show that it can restore the policy more stably than the existing method (Xu et al.] [021]. Furthermore, by updating the policy using the trust region method with the integrated gradient, our approach makes the training process more stable than the Lagrangian method, as demonstrated in Section 5.2 Second, to train critics to estimate constraints with low biases, we propose a TD(A) target distribution which can adjust the bias-variance trade-off. The target distribution is obtained by merging the quantile regression losses (Dabney et al., 2018b) of multi-step distributions and extracting a unified distribution from the loss. The unified distribution is then projected onto a quantile distribution set in memory-effictte -fffcccinn manner. We experimentally show that the target distribution can trade off the bias- hias-variance of the constraint estimations (see Section (.3))
We conduct extensive experiments with multi-constrained locomotion tasks and single-constrainedd Safety Gym tasks [Ray et al., 2019] evaluate the proposed method. In the locomotion tasks, SDAC shows 1.93 times fewer steps to satisfy all constraints than the second-best baselines. In the Safety Gym tasks, the proposed method shows 1.78 times fewer constraint violations than the second-best methods while achieving high returns when using risk-averse constraints. As a result, it is shown that the proposed method can efficiently handle multiple constraints using the gradient integration method and effectively lower the constraint violations using the low-biased distributional critics.
2 Background
Constrained Markov Decision Processes. We formulate the safe RL problem using constrained
Markov decision processes (CMDPs) [Altman] 1999]. A CMDP is detined as (S. A, P, R. CI,.,
p. 7). where S is a state space, A is an action space, P S x A x S + [0, 1] is a transition model, R : S x A x S + R is a reward function, CkE(I..... : S x A x S + R20 are cost functions, p S - [o, 1] is an initial state distribution, and 7 E (0, 1) is a discount factor. Given a policy π from a stochastic policy set II, the discounted state distribution is defined as dd (s)( == (1 (1-)) 7/Pr(s: = a|m), and the return is defined as (7, a) == Σ: 7² R(st, Cee 8t-1) where 50 8, ag a, a I( (-1ss), and s1+1 ~ P(-|se, az). Then, the state value and state action
2

---

value functions are defined as: (s) := E (s, (alss a x-1ss), substituting the costs for the reward, the cost value functions Va, (s) and Qo, (s,a) are defined. In the remainder of the paper, the cost parts will be omitted since they can be retrieved by replacing the reward with the costs. Then, the safe RL. problem is defined as follows with a safety measure F:
:= (zz(s.))) By
maximize, (TT s.t. F(ZZ, (s, a)ls ~ P.a (T-ss) S dx vk,
(1)
where (x) := EE (2) ~ p. a ~ T[-[s)] + AE 1 H (=((ss())))) n, PI. 3 is an entropy coefficient, H is the Shannon entropy, and d₃ is threshold of the kth constraint.
Trust-Region Method With a Mean-Std Constraint. Kim and Oh [2022a] have proposed a trust region-based safe RL method with risk-averse constraint, called a mean-std constraint. The definition of the mean-std constraint function as follows:
F F(Z;a) =- E|z] + (o( (6($0((0)) (a))/aa-
((o))aa))Std(]) Std[Z],
where a E (0, 1J adjusts the risk level of constraints, Std[Z] is the standard deviation of Z. and ø and are the probability density function and the cumulative distribution function (CDF) of the standard normal distribution, respectively. In particular, setting = 1 causes the standard deviation part to be zero, so the constraint becomes a risk-neutral constraint. Also, the mean constraint can effectively reduce the potential for massive cost returns, as shown in Yang et al. (2021), Kim
Kiim and Ohl [2022a b). In order to calculate the mean-std constraint, it is essential to estimate the standard deviation of the [022al] define the square value functions:
cost return. To this end, Kim and Oh|
O (8) = E [Z (s,a) () <(1s)]],
St, (s, == E [Z8, (s)
(3)
Since Std[2|² = B[2z] E, the kth constraint can be written as follows:
FA(TT; a) = Jc, + (6 (6(00-11000 (a))/a) VJs. JC,((T)) S dR.
(4)
where Jc, (T) (x) = E [V (s)ls p] (T) E [S (s)ls ~ p]. In order apply the trust region method [Schulman et al. 2013]. it is necessary to derive surrogate functions for the objective and constraints. These surrogates can substitute for the objective and constraints within the trust region. Given a behavioral policy μ and the current policy Kold. we denote the surrogates as Ju (x and FR Fat (T; a.. For the definition and derivation of the surrogates, please refer to Appendix A.77 A.77 and [Kim and Oh 2022a]. Using the surrogates, a policy can be updated by solving the following subproblem:
maximize, JuH (x') s.t. (EL(= S F S de Vk:
where DRL. (x(x)) == Bgnde [DKL (1))))) DKL. is the KI. divergence, and e is trust region size. This subproblem can solved through approximation and line search (see Appendix A.8). However, it is possible that there is policy satisfying the constraints of (5). In order to tackle this issue, the policy must be projected onto a feasible policy set that complies with all constraints, yet there is lack of such methods. In light of this issue, we introduce efficient feasibility handling method for monti-constrained RL problems.
Distributional Quantile Critic. Dabney et al. (2018) have proposed method for approximating the random variable Z follow quantile distribution. Given a parametric model, 0 S A + IR M 7 can be approximated as RR, called a distributional quantile critic. The probability density function of nn is defined as follows:
Pr ==) m=1 (0),ss.
(6)
where M is the number of atoms, Om (s, a) is the rath atom, 8 is the Dirac function, and 6z) 8(2-
a). The percentile value of the mith atom denoted by Tm (To = 0, T. = i/M), In distributional RL, the returns are directly estimated to get value functions, and the target distribution
be calculated from the distributional Bellman operator [Bellemare et al. 2017]: T ZR(ss a) DD R(s, a, a,, + y Rs(s' aa), where s' PP s, a) and a' (--ss'). The above one-step distributional operator can be expanded to the n-step one: T Zz(So, ap)) _ Ei=o 7 R(s, ae, $1+1) + > Zizssn an), where a ~ T(-(s)) for t = 1..... n. Then, the critic can be trained to minimize the following quantile regression loss (Dabney al. 2018b):
((0)- - Σ ELDDyD [B (BzTTzR (2 - 0 (m,(s)))]] where p. (x) = x . (r - lxca). (7)
2.g (9m))
3

---

Fy(ma)) I ds
Feasible region
Naive
Pclip
81
(x: S
H5 S
Figure I: Left: Gradient integration. Each constraint is truncated to be tangent to the trust region indicated by the ellipse, and the dashed straight lines show the truncated constraints. The solution of (8) is indicated in blue, pointing the nearest point in the intersection of the truncated constraints. If the solution crosses the trust region, parameters are updated by the clipped direction, shown red. Right: Optimization paths the proposed and naive method in a toy example. The description is presented in Appendix(A.2] The contour graph represents the objective of the toy example. The optimization paths exhibit distinct characteristics due to the difference that the naive method reflects only one constraint and the proposed method considers all constraints once.
D is a replay buffer, Fi == (Tm-1 Tm)/2, 2, and FoR (fm) denotes the quantile regression loss for a single atom. The distributional quantile critic can be plugged into existing actor-critic algorithms because only the critic modeling part is changed.
3 Proposed Method
The proposed method comprises two key components: 1) feasibility handling method required for multi-constrained safe RL problems and 2) target distribution designed to minimize estimation bias. This section sequentially presents these components, followed by a detailed explanation of the proposed method.
3.1 Feasibility Handling For Multiple Constraints
optimal safe RL policy can be found by iteratively solving the subproblem (5). but the feasible set of (5) can be empty in the infeasible starting cases. To address the feasibility issue in safe RL with multiple constraints, one of violated constraints can be selected, and the policy is updated to minimize the constraint until the feasible region is not empty [Xu et al., 2021]. which is called a naive approach. However, it may not be easy to quickly reach the feasible condition if only one constraint at each update step is used to update the policy. Therefore, we propose feasibility handling method which reflect all the constraints simultaneously. called gradient integration method. The main idea is to get gradient that reduces the value of violated constraints and keeps unviolated constraints. To find such a gradient, the following quadratic program (QP) can be formulated by linearly approximating the constraints:
argmin 59 Hg s.t. ghg + cs SOk,, c == min( ( 2eg2 H - 9k, Fe(Tx;:)) - de + C)2
(8)
where H is the Hessian of KL divergence between the previous policy and the current policy with parameters v. 9k is the gradient of the kth constraint, C3 is a truncated threshold to make the kth constraint tangent to the trust region, is a trust region size, and R50 is slack coefficient. The reason why we truncate constraints is to make the gradient integration method invariant to the gradient scale. Otherwise, constraints with larger gradient scales might produce dominant policy gradient. Finally, we update the policy parameters using the clipped gradient as follows:
/ き + min(1, 2e/(g" Hg Hg")))"
Figure T Iillustrates the proposed gradient integration process. In summary, the policy is updated by solving (): there is no solution to (5), it is updated using the gradient integration method. Then, the policy can reach the feasibility condition within finite time steps.
Theorem 3.1. Assume that the constraints are differentiable and convex, gradients of the constraints are T.-Lipschitz continuous, eigenvalues of the Hessian are equal or greater than positive value R. Ez0. and FglFF < dy, Vk} # 0. Then, there exists E Ezo such that if

---

O < c C E and policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
Note that the first two assumptions of Theorem 3. are commonly used in multi-task learning Liu
al. |021 Yu et al. 2020 Navon al. [022]. and the assumption on eigenvalues is used in most trust region based RL methods [Schulman et al. 2015 Kim and Oh 2022a]. so the assumptions in Theorem 3.1 can be considered reasonable. We provide the proof and show the existence of a
solution 8 in Appendix [. The provided proof shows that the constant E is proportional to Ç. This means that the trust region size should be set smaller as decreases. Also, we further analyze the worst-case time to satisfy all constraints by comparing the gradient integration method and naive approach in Appendix A.3 In conclusion, if the policy update rule (5) is not feasible, a finite number of applications of the gradient integration method will make the policy feasible.
3.2 TD(A) Target Distribution
The mean-std constraints can be estimated using the distributional quantile critics. Since the estimated constraints obtained from the critics are directly used to update policies in (5). estimating the constraints with low biases is crucial. In order to reduce the estimation bias of the critics, we propose a target distribution by capturing that the TD(A) loss, which is obtained by a weighted sum of several losses, and the quantile regression loss with a single distribution are identical. A recursive method is then introduced so that the target distribution can be obtained practically. First, the n-step targets for the current policy ιT are estimated as follows, after collecting trajectories (sy, Oe: S441,..) with a behavioral policy j.:.
2/2r) (s),,a) _ Rs + 7Ree+1 +... +....-"'-1
+7" - Retn-1 +
(8e4n dtnn),
(10)
where R₂ R(st, at, 1t+1), and θctn ~ T(([st4n). Note that the n-step target controls the bias- variance tradcoff using n. If n is equal to 1, the n-step target is equivalent to the temporal difference method that has low variance but high bias. On the contrary, if increases to infinity, it becomes Monte Carlo estimation that has high variance but low bias. However, finding proper n is another cumbersome task. To alleviate this issue, TD(A) [Sutton] 1988 method considers the discounted sum of all n-step targets. Similar to TD(A), we define the TD(A) loss for the distributional quantile critic as the discounted sum of all quantile regression losses with n-step targets. Then, the TD(A) loss for single atom is approximated using importance sampling of the sampled n-step targets in (10) as:
Cim(0m) (1-1) Σ
X X' [EZWTH (n.olocas) [p7m (Z 6m -sn
(+ (ayls)))
=(1 1)
X A'Erraae>
B2nTH 2n.0(a)ar))
G=t+1 p(az(s;) Prm (Z 0m (m(
(I)
1=0
(1 - X)
X ((E
II T(alls;)
Σ m=1
M
1
prm R.m (sr,a)) as) (m(ssc
j=++1 ((a))s;)
M
1=0
where is a trace-decay value, and 2(4) is the mth atom of 2(0) Since 200) (Sc. 0r) 2 R, 7 2/ (1--1) (8:41, " at+1) is satisfied, 11 is the same as the quantile regression loss with the following
single distribution 2,ot, called a TD(X) target distribution:
Pr = =2) := (1 (1 X)
244
II T(as)s))
Σ
1 T
(onnee) (2)
No
p(ay))))
&mm
Σ
M
6 a
(2)
(att1 (8++1)
n (agls;)
Σ 22
M
1
(())))))))))
No
M
c.m
u( 1241)
f=142 m(ag(ss)
M
n.m
(12)
M
c0 (x) + Non x(a441151411 No p(act1/8441
Pr(Re + +2200 2),
N
M
where A₁ a normalization factor. If the target for time step I + 1 is obtained, the target distribution for time step becomes the weighted sum of (a) the current one-step TD target and (b) the shifted
5

---

Projection
2(1)
W1
Ex
2
atoms
Restore CDF
+
f
+ y2
0
0oo
W2
M' atoms
Figure 2: Procedure for constructing the target distribution. First, multiply the target at f + step by and add Rr. Next, weight-combine the shifted previous target and one-step target at step and restore the CDF of the combined target. The CDF can be restored by sorting the positions of the atoms and then accumulating the weights each atom position. Finally, the projected target be obtained by finding the positions of the atoms corresponding to M'' quantiles the CDF, Using projected target, target at 1 step can found recursively.
previous target distribution, so can be obtained recursively, as shown in (12). Since the definition requires infinite sums, the recursive way is more practical for computing the target. Nevertheless, to obtain the target in that recursive way, we need to store all quantile positions and weights for all time Therefore,
steps, which is not memory--fficieee quantile distribution with a specific number of atoms, M' (we set M' > .M to reduce information loss). The overall process get the TD(A) target distribution illustrated in Figure 2 and the pseudocode is given in Appendix A.5 Furthermore, we can show that a distribution trained with the proposed target converges to the distribution of Zí-
propose to project the target distribution into
Theorem 3.2. Let define a distributional operator Tx" whose probability density function is: Pr(Tx"" Z(s, a) =)=
E. X II maglos)) E ((ay)ss)) at ((
Pr (
(13)
-,
Then, a sequence, Zk+1 (s, a) Ze(ssa V(s, a), converges 7%.-
The TD(A) target a quantile distribution version of the distributional operator Th F in Theorem 3.2 Consequently, a distribution updated by minimizing the quantile regression loss with the TD(A) target converges to the distribution proof of Theorem 3.2 is provided in Appendix A.4 After calculating the target distribution for all time steps, the critic can be trained to reduce the quantile regression loss with the target distribution. To provide more insight, experiment with toy example in Appendix A.6] and the results show that the proposed target distribution can trade off bias and variance through the trace-decay X.
2ī, if the number of atoms infinite, according to Theorem 3.2 The
3.3 Safe Distributional Actor-Critic
Finally, we describe the proposed method, safe distributional actor-critic (SDAC). After collecting trajectories, the policy updated by solving (5). which can be solved through line search (for more detail, see Appendix A.8). The cost value and the cost square value functions in 4 can be obtained using the distributional critics as follows:
M
Q2(s,a)= L :Pr(ZZ(s,a) = 2)dz RS
Σ
Om (m,
M
m=1
(14)
(s / Pr ()) 2)d2
0m (m,
M 1121
If a solution of (5) does not exist, the policy is projected into feasible region through the proposed gradient integration method. The critics can also be updated by the regression loss (7) between the target distribution obtained from 12 The proposed method is summarized in Algorithm
4 Related Work
Safe Reinforcement Learning. Gareia and Fernández (2015) and Gu al. [2022] have researched and categorized safe RL methodologies from various perspectives. In this paper, introduce safe RL

---

Algorithm Safe Distributional Actor-Critic
Input: Policy network Ty. reward and cost critic networks Z 28,,0 and replay buffer D. Initialize network parameters v), f, and replay buffer D.
for epochs to F. do
for 1 to T do
Sample at (=(s) and get 8141, R(st, de, (e, 8141), and Cket Ck (s2, at 8141)) Vk. Store (st, ae. Tu (t)(s))) Te, ((1.....K]:.. St+1) in D.
end for
Calculate the TD(A) target distribution (Section 3.2 using D and update the critics to minimize
the quantile loss defined in 77
Calculate the surrogates for the objective and constraints defined in 37 using D.
Update the policy by solving (5). but if (5) has no solution, take recovery step (Section 3.1
end for
methods depending on how to update policies to reflect safety constraints. First, trust region-based safe RL methods [Achiam et al. 2017| Yang et al.) [020 Kim and Oh 2022a find policy update directions by approximating the safe RL problem
lunear-quadratic constrained linear program and update policies through a line search. Yang et al. (2020) also employ projection to meet a constraint; however, their method is limited to single constraint and does not show to satisfy the constraint for the infeasible starting case. Second, Lagrangian- based methods [Stooke et al. 2020 Yang et al. 2021 Liu al.) 2020) convert the safe RL problem to dual problem and update the policy and dual variables simultaneously. Last, expectation- expectation-maxiimzzttiee (EM) based methods [Liu et al. 2022 Zhang al. 2022] find non-parametric policy distributions by solving the safe RL problem in E-steps
and fit parametric policies to the found non-parametric distributions in M-steps. Also, there are other
ways to reflect safety other than policy updates. Qin al. [202T], Lee al. [2022 find optimal state
or state-action distributions that satisfy constraints, and Bharadhwaj et al. 2002]. Thananjeyan et al.
[202]] reflect safety during exploration by executing only safe action candidates. In the experiments. only the safe RL methods of the policy update approach are compared with the proposed method.
Distributional TD(A). TD()) (Precup al., 2000] be extended to the distributional critic to trade off bias varianei, Gruslys et al.] [2188] have proposed a method to obtain target distributions by mixing n-step distributions, but the method is applicable only in discrete action spaces. Nam et al.] [2021] have proposed a method to obtain target distributions using sampling to apply to continuous action spaces, but this is only for on-policy settings. A method proposed by Tang et al. [0022] updates the critics using newly defined distributional TD errors rather than target distributions. This method is applicable for off- aff-policy settings but has the disadvantage that memory usage increases linearly with the number of TD eтror steps. In contrast to these methods, the proposed method is memory-efficient and applicable for continuous action spaces under off-policy settings.
Gradient Integration. The proposed feasibility handling method utilizes a gradient integration method, which is widely used in multi-task learning (MTL). The gradient integration method finds single gradient to improve all tasks by using gradients of all tasks. Yu et al. [2020) have proposed projection-basse gradient integration method, which is guaranteed converge Pareto-stationary sets. A method proposed by Liu et al. [002]] can reflect user preference, and Navon et al. 2022) proposed gradient-scale invariant method prevent the training process from being biased by few tasks. The proposed method can be viewed as a mixture of projection and scale-invariant methods as gradients are clipped and projected onto a trust region.
Experiments
We evaluate the safety performance of the proposed method in single- and multi-constrained robotic tasks, For single constraints, the agent performs four tasks provided by Safety Gym [Ray et al. 2019) and for multi-constraints, it performs bipedal and quadrupedal locomotion tasks.
5.1 Safety Gym
Tasks. We employ two robots, point and car, to perform goal and button tasks in the Safety Gym. The goal task is to control robot toward a randomly spawned goal without passing through hazard
7

---

Reward Sum 1
Cost Rate
Total CVG
30
SCAC_D.2
3000
CITTRC_0.25
0.04
2000
VCSAC_
10
01.02
1000
GPO
WCSAC_11D
RoinbGool
Carboal PointButton Carautton
0.00
Pointcca
CarSnel
WnmBuctooCCrruuma
Pointoo
CarGos
PaintButtnn CarButton
CVPO
(a) Results of the final reward sum, cost rate, and total number of CVs. The number after the algorithm name in the legend indicates a used for the risk-averse constraint.
Rewart
0.15
Cost
Toral
A-O 0 (TDI
20
60000
A-G
n10
AH0.97
A-10 A
10
0.05
20000
1e5
Starrs
100
Staron
100
(b) Training curves of the point goal task according to the trace-decay X. The solid line represents the average value, and the shaded area shows half of the std value.
Figure 3: Safety Gym task results. The cost rates show the cost sums divided by the episode length, and the dashed black lines indicate the threshold of the constraint. All methods are trained with five random seeds.
regions. The button task is to click a randomly designated button using a robot, where not only hazard regions but also dynamic obstacles exist. Agents get a cost when touching undesignated buttons and obstacles or entering hazard regions. There only one constraint for the Safety Gym tasks, and is defined using 44) with the sum of costs. Constraint violations (CVs) are counted when the cost sum exceeds the threshold. For more details, see Appendix B
Baselines. Safe RL methods based on various types of policy updates are used as baselines. For the trust region-based method, we use constrained policy optimization (CPO) [Achiam et al., 2017] and off-policy trust-region CVaR (OfTRC) [Kim and Oh] 2022a]. which extend the CPO to an off aff-policy and mean-std constrained version. For the Lagrangian brggrangian-based based method, distributional worst-case soft actor-critic (WCSAC) (Yang et al. 2022 is used, and constrained variational policy optimization (CVPO) [Liu et al., 2022 based on the EM method is used. Specifically, WCSAC, OffTRC, and the proposed method, SDAC, use the risk-averse constraints, so we experiment with those for 0.25 and 1.0 (when a = 1.0, the constraint is identical to the nest-nl constraint).
Results. The graph of the final reward sum, cost rate, and the total number of CVs are shown in Figure Bal and the training curves are provided in Appendix C.I We can interpret the results as good if the reward sum is high and the cost rate and total CVs are low. SDAC with a = 0.25, risk-averse constraint situations, satisfies the constraints in all tasks and shows an average of 1.78 times fewer total CVs than the second-best algorithm. Nevertheless, since the reward sums are also in the middle or upper ranks, its safety performance of high quality. SDAC with a 1.0, risk-neutral constraint situations, shows that the cost rates are almost the same as the thresholds except for the car button. In the case of the car button, the constraint not satisfied, but by setting a = 0.25, SDAC can achieve the lowest total CVs and the highest reward sum compared to the other methods. As for the reward sum, SDAC is the highest in the point goal and car button, and WCSAC is the highest in the rest. However, WCSAC seems to lose the risk-averse properties seeing that the cost rates do not change significantly according to a. This is because WCSAC does not define constraints as risk measures of cost returns but expectations of risk measures Yangg et al. 2022 OffTRC has lower safety performance than SDAC in most cases because, unlike SDAC, it does not use distributional critics. Finally, CVPO and CPO are on-policy methods, they are less efficient than the other methods.
5.2 Locomotion Tasks
Tasks. The locomotion tasks are to train robots to follow zy-directional linear and z-directional angular velocity commands. Mini-Cheetae from MIT [Katz et al., [219) and Laikago from Unitree [Wang, 2018) are used for quadrupedal robots, and Cassie from Agility Robotics [Xie et al. 2018) is used for bipedal robot. In order to perform the locomotion tasks, robots should keep balancing, standing, and stamping their feet so that they can move in any direction. Therefore, we define three constraints. The first to keep the balance so that the body angle does not deviate from zero, and the second is to keep the height of CoM above threshold. The third is to match the current foot
8

---

Sum
Fate
Cop:
-100
-00
DTTIIC
50AC
0.75
0.10
0.25
0.75
025
9.2
2000
6000
40003
0.00
0.00
40
Ies
(a) Training curves the Cassie task.
Aeaved
Coe Bate
Rate:
Tew CVr.
0.75
-000
SUM,
OfT Ic:
0.50
9.0
NCCACC
0.25
1000
0.0
0.50
155
2.0
0.00
0.0
0.5
1.0
155
9.5
0.5
3.0
15
2.0
(b) Training curves the Laikago task.
Reaved
Coot
Cost fate
Rate
2000
CVa
-200
100
50W
ot tc
MCSAC
0.30
0.25
0.75
0.4
1000
0.0
1.0
0.00
05
3.0
2.0
0.5
1.0
355
1.0
9.0
0.3
3.0
15
2.
(c) Training curves of the Mini-Cheetall task.
Sum
Fane
Con: 2.1
0.75
0.
-100
-200
Prppeeed
2000
0.50
0.00
0.
(d) Training curves the naive and proposed methods for the Cassie task.
Figure 4: Locomotion task results. The hlack dashed lines indicate the thresholds, and the dotted lines in (d) represent the thresholds + 0.025. The shaded area represents half of the standard deviation, and all methods are trained with five random seeds.
contact state with predefined foot contact timing. The reward is defined as the negative 12-norm of the difference between the command and the current velocity. CVs are counted when the sum of at least one cost rate exceeds the threshold. For more details, see Appendix B
Baselines. The baseline methods are identical to the Safety Gym tasks, and CVPO is excluded because it is technically challenging to scale to multiple constraint settings. We set a to for the risk- averse constrained methods (OffTRC, WCSAC, and SDAC) to foeus measuring multi-constraint handling performance.
Results. Figure (a-c) presents the training curves. SDAC shows the highest reward sums and the lowest total CVs in all tasks. In particular, the number of steps required to satisfy all constraints is 1.93 times fewer than the second- best algorithm on average. Trust region methods (OfTTRC, CPO) stably satisfy constraints, but they are not efficient since they handle constraints by the naive approach. WCSAC, Lagrangian method, fails to keep the constraints and shows the lowest reward sums. This because the Lagrange multipliers can hinder the training stability due to the concurrent update with policy (Stooke et al. 2020
5.3 Ablation Study
We conduct ablation studies to show whether the proposed target distribution lowers the estimation bias and whether the proposed gradient integration quickly converges to the feasibility condition. In Figure 3bb the number of CVs is reduced as increases, which means that the bias of constraint estimation decreases. However, the score also decreases due to large variance, showing that À can adjust the bias-variance tradeoff. In Figure 4d the proposed gradient integration method is compared with the naive approach, which minimizes the constraints in order from the first to the third constraint, as described in Section [3.]] The proposed method reaches the feasibility condition faster than the naive approach and shows stable training curves because it reflects all constraints concurrently. Additionally, we analyze the distributional crities in Appendix C.2 and the hyperparameters, such as the trust region size, in Appendix C.3 Furthermore, we analyze the sensitivity of traditional RL. algorithms to reward configuration in Appendix D emphasizing the advantage of safe RL that does not require reward tuning.

---

Limitation
limitation of the proposed method is that the computational complexity of the gradient integration is proportional to the square of the number of constraints, whose qualitative analysis is presented in Appendix E.T. Also, we conducted quantitative analyses in Appendix E.22 by measuring wall clock training time. In the mini-cheete task which has three constraints, the training time of SDAC is the third fastest among the four safe RL algorithms. Gradient integration is not applied when the policy satisfies constraints, so it may not constitute significant proportion of training time. However, its influence can be dominant as the number of constraints increases. In order to resolve this limitation, the calculation can be speeded up by stochastically selecting subset of constraints [Liu et al. 2021]. or by reducing the frequency of policy updates Navon et al. 2022]. The other limitation is that the mean-std defined in (2) is not coherent risk measure. As result, mean-std constraints can be served as reducing uncertainty rather than risk, although we experimentally showed that constraint violations efficiently reduced. To resolve this, using an auxiliary variable, as done by [Chow et al. [017]. However, this solution can destabilize the training process due to the auxiliary variable, as observed in experiments of Kim and Ohl (2022b). Hence, a stabilization technique should be developed to employ the CVaR constraint.
the CVaR constraint, which can be estimated
7 Conclusion
We have presented the trust region- based safe distributional RL method, called SDAC. Through the locomotion tasks, it is verified that the proposed method efficiently satisfies multiple constraints using the gradient integration. Moreover, constraints can be stably satisfied various tasks due to the low-biased distributional crities trained using the proposed target distributions. In addition, the proposed method analyzed from multiple perspectives through various ablation studies. However, to compensate for the computational complexity, future work plans to devise efficient methods when dealing with large numbers of constraints.
Acknowledgments and Disclosure of Funding
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govermment (MSIT) (No. 2019-0-01190, [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceptable Machine Learning, 34%), ), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by General-Purpose Deep Reinforcement Learning Using Metaverse for Real World Applications, 33%), and Institute of Information & Communications Technology Planning & Evaluation (IIP)) grant funded by the Korea govermment (MSIT) (No. 2021-0-01341, AI Graduate School Program, CAU, 33%).
the Ministry of Science and ICT
References
J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In Proceedings of International Conference Machine Learning, pages 22-31, 2017.
A. Agarwal, M. Kakade, J. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22 (1):4431-4066, 2021.
E. Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Q Bai, A. S. Bedi, M. Agarwal, A. Koppel, and V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4), 2022.
M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of International Conference on Machine Learning, pages 449 458, 2017.
M. G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http://www. wttp::/ddssiibbiinnnii..
10

---

H. Bharadhwaj, A. Kumar, N. Rhinchart, S. Levine, F. Shkurti, and A. Garg. Conservative safety critics for exploration. In Proceedings of International Conference on Learning Representations. 2021.
L. Biewald. Experiment tracking with weights and biases, 2020. URL https://www. com/ Software available from wandb.com.
Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18()):670--1100, 2017.
W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of International conference on machine learning, pages 1096-1105, 2018a.
W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI Conference Artificial Intelligence, 32(1), 2018b.
Dennis and R. B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Society for Industrial and Applied Mathematics, 1996.
J. Garcia and F. Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1493)) 2015.
A. Gruslys, W. Dabney, M. G. Azar, B. Piot, M. Bellemare, and R. Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning. In Proceedings of International Conference on Learning Representations, 2018
S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, Y. Yang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Haamoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of International Conference Machine Learning, pages 1861-1870, 2018
B. Katz, J. D. Carlo, and S. Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control, In Proceedings of International Conference on Robotics and Automation, pages 6295 6301, 2019.
D. Kim and S. Oh. Efficient off off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters, 7(3):644-7651, 2022a.
D. Kim and S. Oh. TRC: Trust region conditional value at risk for safe reinforcement learning. IEEE Robotics and Automation Letters, 7(2):2621-2688. 2022b.
A. Kuznetsov, Shvechikov, A. Grishin, and D. Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In Proceedings International Conference on Machine Learning, pages 5556-5566, 2020.
J. Lee, Hwangbo, Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, x(4))cea,, 2020.
J. Lee, C. Paduraru, J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In Proceedings of International Conference on Learning Representations, 2022.
B. Liu, X. Liu, X. Jin, Stone, and Q Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, pages 18878-18890, 2021.
Y. Liu, J. Ding, and X. Liu. IPO: Interior policy optimization under constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):494044947 2020.
Z. Liu, Z. Cen, V. Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Constrained variational policy optimization for safe reinforcement learning. In Proceedings of International Conference on Machine Learning, pages 13644-13668, 2022.

---

W. Meng. Q. Zheng, Y. Shi, and G. Pan. An ooflcy trust region policy optimization method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2223-222 2235, 2022.
J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: Reusable neural controllers for vision-guided whole-body tasks. ACM Transactions Graphics, 39(4), 2020.
T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eabk2822, 2022.
D. W. Nam, Y. Kim, and C. Y. Park. GMAC: A distributional perspective actor-critic framework.
In Proceedings of International Conference on Machine Learning, pages 7927-7936, 2021.
A. Navon, A. Shamsian, Achituve, H. Maron, K. Kawaguchi, G. Chechik, and E. Fetaya. Multi-task learning bargaining game. arXiv preprint arXiv:2202.01017. 2022.
X. B. Peng, Z. Ma, Abbeel, S. Levine, and A. Kanazawa. AMP: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics, 40(4), 2021.
D. Precup, R. S. Sutton, and S. P. Singh. Eligibility traces for off off-policy policy evaluation. In Proceedings of International Conference on Machine Learning, pages 759-766, 2000.
Z. Qin, Y. Chen, and C. Fan. Density constrained reinforcement learning. In Proceedings of International Conference Machine Learning, pages 8682-8692, 2021.
A. Ray, Achiam, and Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
M. Rowland, M. Bellemare, W. Dabney, R. Munos, and Y. W. Teh. An analysis of categorical distributional reinforcement learning. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 29-37, 2018.
N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of Conference on Robot Learning, pages 91-100, 2022.
Schulman, S. Levine, Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of International Conference Machine Learning, pages 1889-1897, 2015.
A. Stooke, J. Achiam, and Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedingg of International Conference on Machine Learning, pages 9133-9143, 2020.
R. Sutton. Learning to predict by the methods temporal differences. Machine learning, 3(1):
9-44, 1988.
Y. Tang, R. Munos, M. Rowland, B. Avila Pires, W. Dabney, and M. Bellemare. The nature of temporal difference errors multi-step distributional reinforcement learning. In Advances in Neural Information Processing Systems, pages 30265-30276. 2022.
B. Thananjeyan, Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang. E. Gonzalez, Ibarz, C. Finn, and K. Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Roboties and Automation Letters, 6(3):4915 4922, 2021.
E. Todorov, T. Erez, and Tassa. MuloCo: A physics engine for model-based control. In Proceedings of International Conference Intelligent Robots and Systems, pages 00326-503,, 2012.
X. Wang. Unitree- Unikreo Pro. hopphttp:// www.uuiiiieeccc.mm http://wwiieeee.e//aaa////////// http:///hhhoonoooooooocoassssass 6kid=355 2018
Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. van de Panne. Feedback control for cassie with deep reinforcement learning. Proceedings of International Conference on Intelligent Robots and Systems, pages 1241-1246, 2018.
12

---

T. Xu, Y. Liang, and G. Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of International Conference on Machine Learning, pages 11480-1144 1149, 2021.
Q Yang. T. D. Simāo, S. H. Tindemans, and M. T. J. Spaan. WCSAC: Worst-case soft actor critic for Intelligence,
reinforcement learning. Proceedings of the AAAI Conference on Artificial (001:):00099-100 2021.
Q. Yang, T. D. Simão, S. H. Tindemans, and M. T. J. Spaan. Safety-constrained reinforcement learning with a distributional safety critic. Machine Learning, 12:8859-887, 2022.
T.-.. Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge. Projection- based constrained policy optimization. In Proceedings of International Conference on Learning Representations, 2020.
Yu, Kumar, A. Gupta, Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pages 5824-5836, 2020.
H. Zhang, Y. Lin, S. Han, S. Wang, and K. Lv. Conservative distributional reinforcement learning with safety constraints. arXiv preprint arXiv:2201.07286, 2022.
J. Zhang and Weng. Safe distributional reinforcement learning. In J. Chen, J. Lang, C. Amato, and D. Zhao, editors, Proceedings of International Conference on Distribuled Artificial Intelligence, pages 107-128, 2022.
13

---

A Algorithm Details
A.1 Proof of Theorem 3.1
We denote the policy parameter space as V C Rd, the parameter at the tth iteration as 1 E L the Hessian matrix as H (+t) = v y DKL( | and the kth constraint as (x(())) Fi (To a). As we focus on the tth iteration, the following notations are used for brevity: H ((h) and 9s (1bx). The proposed gradient integration at tth iteration is defined as the following quadratic program (QP):
1 Hg s.t. of + for for Ak,
9: argmin 29
(15)
where C = min( 2egg H FT((T)) dx + C). In the remainder of this section, we introduce
the assumptions and new definitions, discuss the existence of solution (15). show the convergence to the feasibility condition for varying step size cases, and provide the proof of Theorem 3.1
Assumption. 1) Each F& is differentiable and convex, 2) F is T--Lipschitz continuous, 3) all eigenvalues of the Hessian matrix (1)) are equal or greater than R R. for Vah v, and 4) (1b) + < dy for Vk} j 0.
Definition. Using the Cholesky decomposition, the Hessian matrix can be expressed H B- D" where B is a lower triangular matrix. By introducing new terms, Is == B B=* 99 and b, == B" 9t. the following satisfied: 9 H gk 12 Additionally, we define the in in-boundary and out- uut-boundrry
sets as:
IB₃ =
(IFF(W) - d+ + + V 22TF
OB, VIFF(W) - d +
The minimum of || in OB, is denoted as m,, and the maximum of || in IBg is denoted as MA. Also, ming.mg and max; are denoted as m and M, respectively, and we can say that m is positive.
Lemma A.1. For all k, the minimum value of my is positive.
Proof. Assume that there exist {l..... K} such that my is equal to zero at a policy parameter 2i E OB2, i..., / Fa (b+ O. Since F& is convex, vf* is a minimum point of Fk, min. (i)) Fu(++) < dę - Ç. However, (x > d₃ as vi* E OB, my is positive due to the contradiction. Hence, the minimum of my is also positive.
Lemma A.2. A solution of (15) always exists.
Proof. There exists a policy parameter & E (xb))( + < de for Vk} due to the assumptions.
Let g the Then, the following inequality holds.
of (v v/r) + ca I ( - vhr) (b())) + de S Fx(e) + dR. (: Fb is convex.)
+ of (b whr) + cs < (.) + ds < 0 for Vk.
Since & th satisfies all constraints of 15 the feasible set is non-empty and convex. Also, H is positive definite, so the QP has unique solution.
Lemma A.2 shows the existence of solution of (15). Now, we show the convergence of the proposed gradient integration method in the case of varying step sizes.
Zemii Lemma A.3. If 2eM < Ç and a policy is updated by tht1 = the + Sigr. where 0 < 3t < L 16 and B, < 1, the policy satisfies (xb()) < du for Vk within a finite time.
Proof. We can reformulate the step size as 3 2yZemR 8i, where 8 < 2/2cm and 0 < 3 < 1. L1bb
11
Since the eigenvalues of H is equal to or bigger than R and H is symmetric and positive definite,
14

---

-I-H- is positive semi-definite Hence, H S following inequality holds:
1 R |||001 is satisfied. Using this fact, the
(e + Bege) Fa(ls)) S B.V.R( 91 + 4
- 2113,ar11 (:: V. is L-Lipschitz continuous.)
L.
Big 9: +
19119111
2
L
Biok 91 +
D2 H H-1..
(:: 9: B Tbeb
2
L
S S + 2R
1/11bel1?.
(:: gh 9t + S 0)
Now, we will show that to enters IB₃ in finite time for V/ OB2 and that the kth constraint is
satisfied for Ve E IBx. Thus, we divide into two cases, 1) th E OB₁ and 2) the E IBg. For the first
case, C = /2c 1l so the following inequality holds:
(ab + 9e9i) - F&(()) S B
(-V2c[10nl + Belbbll?
2R
(16)
S 1. 11/1 + mm8²)
S Be. - <.
The value of F& decreases strictly with each update step according to (16). Hence, th can reach IB; by repeatedly updating the policy. We now check whether the constraint is satisfied for the second case. For the second case, the following inequality holds by applying c =x((1))) ds + C:
L
Palve + Beg.) - F& (vhe) s B,d - B.F((())) - 3.5 +
3 1|b.11
2R
Pib( + Begi) de S (1 - 3.) dx) + 3,(-5 + v 2emB))
Since de IBg.
Fa(1h)) d2 S ve -S V2eM 55
Since m S M and 3 1,
-5 2em3, < <- + 22 <
Hence, (a(2/2 + Beg:) S de, which means that the kth constraint is satisfied if t/ E TBg. As the reaches IBg for vk: within a finite time according to (T6), the policy can satisfy all constraints within a finite time.
Lemma A.3 shows the convergence to the feasibility condition in the case of varying step sizes. We introduce a lemma, which shows ||b,|] is bounded by Ve, and finally show the proof of Theorem 3.1 which can be considered special case of varying step sizes.
Lemma A.4. There exists T E lg50 such that |lbel| TV.
Proof. Let define the following sets:
= (k(Pa()))) + (- de 0), U (K)F)()z) + de = 0), C(e) (glok و ca(() S Ak4),,
I = (H)Fa((r)) + Ç de < 0), :=
(17)
I6 = (glgk' g + Fa(W)) + ( C dr S 50.K- E T},
where cu(e) min( minm 229 H --1 gr, F& (TT) du +). Using these sets, the following vectors be 2 defined: g(c) argmingec1)) 7 Hg. b(e) BT g(c). Now, will show that [|b(c)] is bounded above and [lb(s)]l x ve for sufficiently small >
First, the following is satisfied for sufficiently large e:
C(e) {glgī g + Fe(1))) +5 de S 0 Vk).
(18)
15

---

Since & 2) E C(e), where e is defined in Lemma A.2 Therefore, |b({)]] is bounded above.
[ - 2/00- 12) H( vh) for Ve.
Second, let us define the following trust region size:
1 :== 2 minxeo
Fi (we) + + - dk
>> 0.
(19)
|
if, S the following satisfied:
C(e) Ta (olok 9 + 24 Skll S A4 E O, g S vk E UU) *
(20)
Thus, Oa(c) = وele] g + 22d S A4 E O, gf S Ak E U} is not empty. If define g(e) g00 g Hg. the following is satisfied:
5(c) = V00 for 0 for SosE.
(21)
Then, c S é (minker (dk: - Fs FL(( ((() g(() since Ø(c) E Ig. Consequently, by defining trust region size:
min 1, minser de - (CFFF((r) F((U)t) )3) > 0,
(miisee
(22)
g6((2
g(t) = Vefég((t for E Therefore, [[b(c)]] x V e*.
Finally, since [[b(c]] | is bounded above and proportional to ve for sufficiently small c, there exist constant 7 such that [ib(c]] s TVcc.
Theorem 3.1. Assume that the constraints are differentiable and convex, gradients of the constraints T--Lipschitz continuous, eigenvalues of the Hessian are equal greater than positive value R E Rz and {1b] (e(TT;; 0 ES E and a policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
Ç < de, yk} * 0. Then, there exists F. E R20 such that if
Proof. The proposed step size is 3 = min(1, /2e/ [|be)] II), and the sufficient conditions that guarantee the convergence according to Lemma A.3 are followings:
2/2em
V 2e M S Ç and B. < and 3,
The second condition is self-evident, To satisfy the third condition, the proposed step size B, should satisfy the followings:
V2e
2 RemB
2mR
||ball
||bel|
L||be|12
If < (L())) the following inequality holds:
2mR
2mR
Ve <
7 ||bll < T
Lemma A.4
LT
L
Hence, if S E 1. min 2M12 4( m8 )²), the sufficient conditions are satisfied. LT
A.2 Toy Example for Gradient Integration Method
The problem of the toy example in Figure T defined as:
minimiza (V3z, 3z, x2 + 2)² + 4(x1 V32g 4)2 s.t. 21 2 0, 21 2r S 0, #2:002
(23)
where there are two linear constraints. The initial points for the naive and gradient integration methods are -2 2.5 and /2 = -300, which do not satisfied the two constraints. We use the Hessian matrix for the trust region as identity matrix and the trust region size 0.5 in both methods. The naive method minimizes the constraints in order from the first the second constraint.
16

---

A.3 Analysis of Worst-Case Time to Satisfy All Constraints
To analyze the sample complexity, we consider tabular MDP and use softmax policy parameteriza- tion as follows (for more details, see [Xu et al. 2021]):
exp (s)
TTe (als)
En exp V(s,a) E8
(24)
According to Agarwal et al. [021], the natural policy gradient (NPG) update is as follows:
expl e [ (s.
1/+1 = wr + 3 Twe4s (als) = Tv. (als)
(25)
(x)
where /3 is a step size, A E RIII is the vector expression of the advantage function, and (s) = En Th (a 8) exp (B (s,a))/((1 -) Analyzing the sample complexity of trust region- based methods is challenging since their stepsize is not fixed, so we modify the gradient integration method to use the NPG follows:
g = argming 9 "0 Hig s.t. gf + + SOVK E (A(FA(TT > dxf.
(26)
1 = v. + 3g" 0001/"//112
In the remainder, we abbreviate n... , and Fn (TTg a) as nr A', and (T²). respectively. Since g" always exists due to Lemma A.2 can write the policy using Lagrange multipliers X 2 as follows:
1/+11 the - 3 Σ No N W
Σ X Mg 12,
(-
B
(27)
Tt41(a|s) () = (elass) exp
We (-we
Σ Ac. (s, a) /Z((s),
where (s) is a normalization factor, and A = 0 for (A(TT) S dR. The naive approach can also be written above, except that A is one-hot vector, where i-th value X is one only for corresponding to the randomly selected constraint. Then, we can get the followings:
Σ (Fa((e))))) Fa
1 1- Eand
Σ TetI(als)
Σ (s)
B11 -7) Bend Σ Te+ TTT (als) log
1
Te+1 I (a|s) (s))
Te (a(ss)
1
(28)
Eend [DK
log [s(s
[B1 ))
$ B(1 - 7) Eanavet1 log (s)
(: DxL(('1Il 2 20)
1 Bomp [log (s)),
<
(::11d /pllo 2 7)
We can also get the followings by using the Lemma Xu et al. (00T]:
Σ C Fx(T:))/W, 3(1 7) EE Exnnd [TL TT)
(KL [
(29)
2,0
Σ 21 (1 ??W?
where T, is an optimal policy, and Cmax is the maximum value of costs. Xf > 0, Fx (T₂) Fa (T. Thus, Ee (FF((TT.) (T()) -5E Xf. If the policy does satisfy the
constraints until T step, the following inequality holds by summing the above inequalities from / 0
to T:
28Cmmax
3(1-7)(C-
Σ
X/WW, S Esaa [DKI(TT.[TOO)
(30)
(1 7
t=0
17

---

Let denote + Ex= M/Y,W, as E, A/Wi), and we can get Wt - 11 X A En |12|S|| m/xpmma////// (1 7). Then, the maximum 7 can be expressed as:
DKL 280 E E X/WW,]
DKL $ -(1-7) 2280
T
T Tmaxs
(31)
8(1 - 71--))(--
[I-7)²
where we abbreviate Rgndt (TKI.(TT [[TT]] DRL. Finally, the policy can reach the feasible region within Z max steps.
The worst-case time of the naive approach is the same the above equation, except for the A part. In the naive approach, X is a one-hot vector, as mentioned earlier. In other words, only E,
E, E X/Il E, X,AC, [l2] is different. Let assume that the advantage vector follows a normal distribution. Then, the variance of Σ3 MEAAC is smaller for X with distributed values than for one-hot values. Then, the reciprocal the 2-norm becomes larger, resulting in decrease the worst-case time. From this perspective, the gradient integration method has benefit over the naive approach as it reduces the variance of the advantage vector. Even though we cannot officially say that the worst-case time of the proposed method is smaller than the naive method because the advantage vector does not follow the normal distribution, we can deliver our insight on the benefit of gradient integration method.
A.4 Proof of Theorem
In this section, we show that sequence, Zk+1= = Tx" Zu. converges to the Zī-- First, we rewrite the operator Tha for random variables operator for distributions and show that the operator is contractive. Finally, we show that 2% is the unique fixed point.
Before starting the proof, we introduce useful notions and distance metrics. As the return 27 (s, a) a random variable, we define the distribution of (7 a) as (5, a). Let v be the distribution of random variable X. Then, variable, aX + b, using the pushforward operator, which is defined by Rowland et al. [2018] as (fa,t) (n). To measure a distance between two distributions, Bellemare et al. 2023 has defined the distance lp as follows:
express the distribution of affine transformation of random
4(n. 12) = ( (x) Fop (x:) F
1/p
de
(32)
where (x) is the cumulative distribution function. This distance is 1/p-homogeneous, regular, and p-convex (see Section 4 of Bellemare et al.) [2023] for more details). For functions that map state-action pairs to distributions, a distance can be defined as [Bellemare et al. [003]:: Ip (21, 1/2) := (P (PI (s, a), (2 a)). Then, we rewrite the operator 7 for random variables in
(s a
13 as operator for distributions as below.
Tha v(s, a) = 1- A
N
1=0
(v( = (33)
[
x Ey
Ed Ea'nat swn) [fgtt
5, 55,
where 7188 salss)
s(als] and Wi is a normalization factor. Since the random variable 7(s, a) and the distribution v(s, a) is equivalent, the operators in (13 and 33 are also equivalent. Hence, we are
going to show the proof of Theorem 3.2 uuing (33 instead of 13 We first show that the operator Thu has a contraction property.
Lemma A.5. Under the distance in and the assumption that the state, action, and reward spaces are finite, Thas Th is MP //oottrttact
18

---

Proof. First, the operator can be rewritten using summation as follows.
20
i Pr (Su, aus Tos 8842
"k""vss, a) - N
Σ
x
Σ
==
a'EA
x (r(a|ss41))( Ei-o
x
PIy(T)
A
E1 1smmsi1
a'EA
s'ES
II Tr (f the (nd
rois
k==
(34)
E E
N
8'ES
To.
II m(s),a))
II 1 mrx
x En
8
ΣE E Σ ΣEΣΣ ros
aw aa Diso =
N
Since the sum of weights of distributions should be one, we find the normalization factor
= (1 - A) E: E.es Easa Enp. W...o. Then, the following inequality can be derived using the homogeneity, regularity, and convexity of Ip:
15 vi (8, THa v2(
1-1
00 ΣEEΣ
X Waaaaroo Di 7' (V12(s()))))
N
0= aES DEA roi
1- D0
ΣΣΣΣ
N
0= SES a ros
A)A 1p (1
SES Tos
(futt otre (p2(s,
(35)
2)A a. rois
N
SES Tos
A)
N
A. 241 (Tp (11,
N
SEEAA Tos
5> (Tp (2),
Therefore, 1 (T v1, 12) S 2/PTT, (1I, 12).
By the Banach's fixed point theorem, the operator The has a unique fixed distribution. We now show that the fixed distribution vß.
Lemma A.6. The fixed distribution of the operator The is vic
19

---

Proof. From the definition of R. the following equality holds [Rowland et al. 2018 1/ (s. a) E (ff,)) ( (('s'''))))))) a.)))] Then, can be shown that VR is the lixed distribution by applying the operator Tx to uR
T (8 a) = N
1=0
x
E
/
(83 az)
az)
Be Ber [U Ei (1 80 = s, 00 a
(36)
X
A
Xv =v)(ss,)).
Theorem 3.2. Let define a distributional operator Tx"", whose probability density function is:
Pr(TA 2))
E m(alass)
(13)
R' i+1 ((s141,))==:
p(ay(s)) (-le-r))
80 80
Then, a sequence, Z (8, a) =THi ZE(s, a) V(s, a), converges to Z
Proof. The operator TxaT is //P ccottractocteieeeeeeee under the distance 7, according Lemma A.5 Also,
the fixed distribution of the operator is vī which is equivalent to 23. according to Lemma A.6 By the Banach's fixed point theorem, the sequence, Z (s, a) T (s, a) V(s.)) converges to the fixed distribution of the operator, Ză-
A.5 Pseudocode of TD(A) Target Distribution
provide the pseudocode calculating TD(A) target distribution for the reward critic in Algorithm The target distribution for the cost critics can also be obtained by simply replacing the reward part with the cost.
Algorithm TD(A) Target Distribution
Policy network Ty. critic network zg, and trajectory {(8t, at, ((el)s)) rt, di 8t11) Sample action a/r+1 (g(3541) and get 210t rr (1 dr 2z (8T+1, d/41)- Initialize the total weight Whot = X. for I. T to do
Input:
Sample an action a/41 ~ (st+1) and get 2,00) + (1 de Z2 (s411.
Set the current weight w
Combine the two targets, (2) w) and (2 (tot) Wron), and sort the combined target according
the positions of atoms.
Build the CDF of the combined target by accumulating the weights at each atom.
Project the combined target into quantile distribution with M' atoms, which is 2, (proi) using the CDF (find the atom positions corresponding to each quantile). [=(a)(s)) (1 14 A + wrot).
Update (tot)
T:-1 + (1 de-1)72,pro) (pro)) and Wtot > pla. 8x)
end for Return (2 T
11=1
20

---

A.6 Quantitative Analysis on TD()) Target Distribution
We experiment with a toy example to measure the bias and variance the reward estimation according to A. The toy example has two states, and S2; the state distribution is defined as an uniform; the reward function is defined r(s1) ~ WV 0.02) and r(s2) (0.0055 0.03). We train parameterized reward distributions by minimizing the quantile regression loss with the TD(A) target distribution for
0, 0.5, 0.9, and 1.0. The experimental results are presented in the table below.
Table I: Experimental results of the toy example.
5th iteration
10th iteration
15th iteration
20th iteration
25th iteration
0.0
4.813 (0.173)
4.024 (0.253)
3,498 (0.085)
3.131 (0.103)
2.835 (0.070)
X 0.5
4.621 (0.185)
3.688 (0.273)
2.925 (0.183)
2.379 (0.134)
2.057 (0,070)
X 0.9
4.141 (0.461)
2.237 (0.402)
1,389 (0.132)
1.058 (0.031)
0.923 (0.019)
X 1.0
2.886 (0.767)
1.733 (0.365)
1.509 (0.514)
1.142 (0.325)
1.109 (0.476)
The values in the table are the mean and standard deviation of the past five values of the Wasserstein distance between the true reward return and the estimated distribution. Looking the fifth iteration, it is clear that the larger the À value, the smaller the mean and the higher the standard deviation. At the 25th iteration, the run with À = 0.9 has the lowest mean and standard deviation, indicating that training has converged. On the other hand, the with À 1.0 has the biggest standard deviation, and the mean is greater than À = 0.9, indicating that the significant variance hinders training. In conclusion, measured bias and variance quantitatively through the toy example, and the results are well aligned with our claim that À can trade off bias and variance.
A.7 Surrogate Functions
In this section, we introduce the surrogate functions for the objective and constraints. First, Kim and Oh (2022a] define a doubly discounted state distribution: d5 == (1 12) Ee y 2 8 T).
Then, the surrogates for the objective and constraints are defined as follows Kim and Oh 2022a
Ju. (T¹)
E [V
(so)]
BE [M( [+)] + E
(Qh(s,))))
40"0p
du,a!
1
J (x')
E
VE, (so)]
+ - 7 "x'
E
[00 (s. a)]
1p~p
(37)
Jg. (x') = E
Se, (Sv)] + 1- di
E
|Sa (s, .ar
5p
p (a))
(a))
(( (T')),
where μ4, n, n² are behavioral, current, and next policies, respectively. According to Theorem in [Kim and Oh 2022a]. the constraint surrogates bounded by DKL (I,, r²). We also show that the surrogate of the objective is bounded by DKL ( X') in Appendix AA.9 As a result, the gradients of the objective function and constraints become the same the gradients of the surrogates, and the surrogates can substitute the objective and constraints within the trust region.
A.8 Policy Update Rule
To solve the constrained optimization problem 5 we find a policy update direction by linearly approximating the objective and safety constraints and quadratically approximating the trust region constraint, as done by Achiam et al. [2017). After finding the direction, we update the policy using line search method. Given the current policy parameter v V, the approximated problem can be expressed as follows:
1 - K argmax 9" s.t. 7
Hx Se, 65 x + c 50
(38)
ZEW
where g V JM. (TT))
V H V3 (T. TT Mympe by V FH (
oembe and
Ck FA(TTi a) - dr. Since (38 is convex, we can existing convex optimization solver.
21

---

However, the search space. which is the policy parameter space V, is excessively large, so we reduce the space by converting (38) to a dual problem as follows:
ul 1 x Hx c) + X (Txxx +c]
s(A,v) ming (xx, X, v) min, min-(-g'
-1
H 1H-1 g 2g FH B A BH BT)
+ c ME
(39)
2v
=x
=:S
(q
2r X SX) + X c-
2v
where B (b)..., bk). c (ci, ..., CK)² and À E K 2 0 and v E R 0 are Lagrange multipliers. Then, the optimal À and can be obtained by convex optimization solver. After obtaining the optimal values, (A","") ) = (r.)
the policy update direction x" are calculated by 5e (g X*). Then, the policy is updated by d = th + fxx" where 3 is step size, which can be found through a backtracking method (please refer to Section 6.3.2 of Dennis and Schnabel 1996]).
Before using the above policy update rule, we should note that the existing trust-region method with the risk-averse constraint [Kim and Oh, [022a]] and the equations (I 37 | are slightly different. There are two differences: 1) the objective is augmented with an entropy bonus, and 2) the surrogates are expressed with Q-functions instead of value functions. To use the entropy-regum regropy-rgul objective in the trust-region method, it is required show that the objective is bounded by the KI. divergence. We present the existence of bound in Appendix A.. Next, there is no problem using the Q-functions because mathematically equivalent between the original surrogates [Kim and Oh, [022a] and the new ones expressed with Q-functions (37). However, we experimentally show that using the Q-functions in off-policy settings has advantages Appendix AA
A.9 Bound of Entropy-Augmented Objective
In this section, we show that the entropy-regulane objective in (T) has a bound expressed by the KL divergence. Before showing the boundness, we present new function and lemma. A value difference function is defined as follows:
8d (s) == E [R(s, a, 5') + >V" V (*) a ('1)), ~ p(-|s,a]]= E (s, am
where (s, a) = a) - -
Lemma A.7. The maximum of 100h (s) - (s)) is equal or less than cn /nv2DRD 22 (T)( where
CR max |A a)).
5,a
Proof.
The value difference can be expressed in a vector form.
8f (5) (s) (5)) Σ ( (als) T(als) (s, (z'
T(-1s) (T,,,
Using Hölder's inequality, the following inequality holds:
10 (s) 8*)(s) < ||=((-1s) (111s)|1 ||A loo
(=)) (s,
+ 1180 - 8r1 1l max, 18 (s) - 8T (s)l < 2e Dr (T((1)))
Using Pinsker's inequality, ||8" - 8- ERV 20max m')
Theorem A.8. Let us assume that max, H((((Iss) xo for V' E II. The difference between the objective and surrogate functions bounded by a term consisting of KL divergence as:
V2y
[J('') ju. (<x)
(1 >)² DRE (=11*()
(Ben + ER 2D (1||=²)
(40)
where ен max ( DRLN (T||TT) max (1L( and the equality holds
when n' =
22

---

Proof. The surrogate function can be expressed in vector form as follows:
Ju (p, V + 1 (dtt 8"0))" + 3(d", 3(d" 1)
where Ha (s) H (=' (-))). The objective function of x' can also be expressed in vector form using Lemma from Achiam et al. 2017].
1 1 E8 [(s,,,,')) 3H (s) d" a'))( P(11.,))
J(x')
E. (s) + 3 (s) + E [s)(
1 sodtt
smp
(p, V + 1-(adr (dt 8r + =31
By Lemma 3 from Achiam et al. inequality is satisfied:
2017], 11dT d d= S 1-y / 2D ( (1)
Then, the following
KL
[[1 (1-))(Jum) (T') - J(z'))
1(d= d", 8th) B(d d'', HD')l
< 1(d d,, 8t + (d d'', H
1(d'' du 8th 50) + B( d=', HH
8 0)
- 1| d 11 - 10 + =111d= - d 11 H 1l
( Hölder's inequality)
< 1
DERY / DKT. (RT.
Dis.e (π||T") + 1 22DRE Dimie (T||)²)
B7EH
(: Lemma A.7 7
7
DRL. (π||)²)
(223em + 2 DRt. (RIX((ul (u||=')
If T'' n, the KL divergence term becomes zero, so equality holds.
A.10
Comparison of Q-Function and Value Function- -aased Surrogates
The original surrogate is defined as follows:
(als) d p(aas)
JuH (T'') == ((T)
E
A (s, (s.
(41)
where (8 a) == Q (s, a) V (s.a), and the surrogate is the same as that of OffTRPO Meng al. 2022] and OffTRC (Kim and Oh 2022a An entropy-regularized version can be derived as:
1
(a(as)) A (s.
(=²((*²) = J(T) + 1 BE H((
E
(42)
de
puals))
Then, the surrogate expressed by O-functions in (37). called SAC-style version, can be rewritten as:
Je. (HTT( ((T) + 1-y BE H(+ + E (sT)aa))) d',
(43)
In this section, we evaluate the original, entropy-regularized, and SAC-style versions in the continuous control tasks the MuJoCo simulators [Todorov et al. 2012]. We use neural networks with two hidden layers with (512, 512) nodes and ReLU for the activation function. The output of value network is linear, but the input is different; the original and entropy-regularicn and the SAC-style version uses state-action pairs. The input of policy network is the state, the output is mean / and std σ, and actions are squashed into tanh( + co), ~ N/(0 1) as in SAC [Haarnoja et al. 2018). The entropy coefficient 33 in the entropy-regularizer and SAC-style versions are adaptively adjusted to keep the entropy above a threshold (set as -d given A C Rd). The hyperparameters for all versions are summarized in Table 2
versions use states,
The training curves are presented Figure 5 All methods are trained with five different random seeds. Although the entropy entropy-regulatee regularized version 42 and SAC-style version (43 are mathematically
23

---

Table 2: Hyperparameters for all versions.
Parameter
Value
Discount factor 7
0.99
Trust region size é
0.001
Length of replay buffer
10
Critic learning rate
0.0003
Trace-decay
0.97
Initial entropy coefficient
1.0
/ learning rate
0.01
10000
3020
8000
4000
2000
2000
1000
Entropy
Original
2000
SAC-aly's
(a) Ant-v3
(b) HalfCheetah-v3
(c) Hopper-v3
150
6000
300
30
2000
2000
30aps
(d) Humanoid-v3
(e) Swimmer-v3
(f) Walker2d-v3
Figure 5: MuJoCo training curves.
equivalent, it can be observed that the performance of the SAC-style version is superior to the regularized version. It can be inferred that this is due to the variance of importance sampling. In the off off-policy setting, the sampling probabilities of the behavioral and current policies can be significantly different, so the variance of the importance ratio is huge. The increased variance prevents estimating the objective accurately, so significant performance degradation can happen. As a result, using the Q-function-basee based surrogates has an advantage for efficient learning.
24

---

B Experimental Settings
(a) Point goal.
(b) Car button.
(c) Cassie.
(d) Laikago.
(e) Mini-Cheetah.
Figure 6: (a) and (b) are Safety Gym tasks. (c). (d), and (c) are locomotion tasks.
Safety Gym. We use the goal and button tasks with the point and car robots in the Safety Gym environment [Ray al.] 2019), shown in Figure 6a and 6b The environmental setting for the goal task is the same as in Kim and Oh (2022b). Eight hazard regions and one goal are randomly spawned at the beginning of each episode, and robot gets reward and cost follows:
R(s, 8') - Adgoal + 1ou 50.3,
(44)
C(s, 8') Sig (0.2 dhazard)),
where dgoal is the distance to the goal, and duazard is the minimum distance to hazard regions. If dgowd is less than or equal to 0.3, a goal is respawned. The state consists of relative goal position, goal distance, linear and angular velocities, acceleration, and LiDAR values. The action space is two-dimensional, which consists of ay-ddirectinnll forces for the point and wheel velocities for the car robot.
The environmental settings for the button task are the same as Liu et 2022).. There are five hazard regions, four dynamic obstacles, and four buttons, and all components are fixed throughout the training. The initial position of robot and an activated button are randomly placed at the beginning of each episode. The reward function is the same as in (44), but the cost is different since there is no dense signal for contacts. We define the cost function for the button task as an indicator function that
outputs one if the robot makes contact with an obstacle or an inactive button or enters hazardous region. We add LiDAR values of buttons and obstacles to the state of the goal task, and actions are the same as the goal task. The length of the episode is 1000 steps without early termination.
Locomotion Tasks. We use three different legged robots, Mini-Cheetabi Laikago, and Cassie, for the locomotion tasks, as shown in Figure 6e 6d and 6c| The tasks aim to control robots to follow velocity command on flat terrain. A velocity command is given by (o_md, ..y gecuud w ) where e ~ U( (i((-1. 1.0, 1.0) for Cassie and (1.1.00 2.0) otherwise, ogmd = 0, and w. ~ ((0.5,, 0.5). set the y-directional linear velocity zero but scale any
To lower the task complexity, non-zero value. As in other locomotion studies [Lee et al. 2020 Miki et al. 2022). central phases are introduced to produce periodic motion, which are defined as (t.
0..0 + for Vi (1,.... Megs) }. where ff is a frequency coefficient and is set to 10, and Φi. is an initial phase. Actuators of robots are controlled by PD control towards target positions given by actions. The state consists of velocity command, orientation of the robot frame, linear and angular velocities of the robot, positions and speeds of the actuators, central phases, history of positions and speeds of the actuators (past two steps), and history of actions (past two steps). A foot contact timing & can be defined as follows:
(s) = - + (g Vi E Nlegs )
(45)
where a value of means that the ith foot is on the ground; otherwise, the foot is in the air. For
the quadrupedal robots, Mini- Chi-Chhetaah and Laikago, we use the initial phases do = {0, म. n, 0}. which generates trot gaits. For the bipedal robot, Cassie, the initial phases are defined as Φo = {0, r). which generates walk gaits. Then, the reward and cost functions are defined as follows:
R R(s,a,8') = -0.1 (Ilo
base
emd I|2 + ||w Base emd 12
10 Rpower). ):
Rlege
(46)
(s. anglezas Co C2 8') = height Ca(s, a,
Σ
(.)((222222 megs)
where the power consumption Rpower- E. |rrus the sum of the torque times the actuator speed, is base is the xyydiireciionl linear velocity of the base added to the reward regularization term, Uary
25

---

frame of robots, whase is the z-directional angular velocity of the base frame, and 5 E {-1, " is the current feet contact vector. For balancing. the first cost indicates whether the angle between the z-axis vector of the robot base and the world is greater than a threshold (a = 15 for all robots). For standing, the second cost indicates the height of CoM is less than a threshold (b- = 0.3, 0.35, 0.7 for Mini-Cheetah, Laikago, and Cassie, respectively), and the last cost is to check that the current feet contact vector & matches the pre-defined timing The length of the episode is 500 steps. There is no early termination, but if robot falls to the ground, the state is frozen until the end of the episode.
Hyperparameter Settings. The structure of neural networks consists of two hidden layers with (512,5 nodes and ReLU activation for all baselines and the proposed method. The input of value networks is state-action pairs, and the output is the positions of atoms. The input of policy networks the state, the output is mean i and std o, and actions are squashed into tanh (μ + ca), 6 ~ (0, 1). We use a fixed entropy coefficient B. The trust region size is set to 0.001 for all trust region-based methods. The overall hyperparameters for the proposed method can be summarized in Table
Table 3: Hyperparameter settings for the Safety Gym and locomotion tasks.
Parameter
Safety Gym
Locomotion
Discount factor y Trust region size
0.99
0.99
0.001
0.001
Length of replay buffer
10a
10
Critic learning rate
0.0003
0.0003
Trace-decay
0.97
0.97
Entropy coefficient 8
0.0
0.001
The number of critic atoms M
25
25
The number of target atoms M'
50
50
Constraint risk level a threshold d, Slack coefficient <
0.25, 0.5, and 1.0
1.0
0.025/(1 - 7)
[0.025, 0.025,0.4//(/ 014 1 -7)
minadæ = 0.025 (1255/11-)) -7)
Since the range of the cost is [0, 1], the maximum discounted cost sum is 1/(1 7)) Thus, the threshold set by target cost rate times //(1-7) For the locomotion tasks, the third cost in (46) is designed for foot stamping, which is not essential to safety. Hence, we set the threshold to near the maximum (if a robot does not stamp, the cost rate becomes 0.5). In addition, baseline safe RL methods use multiple critic networks for the cost function, such target Yang et al. [222]] or square value networks [Kmm and Oh 2022a]. To match the number of network parameters, we use two critics as ensemble, as in Kuznetsov et al. 22200]
Tips for Hyperparameter Tuning.
Discount factor 7, Critic learning rate: Since these are commonly used hyperparameters, we do not discuss these.
Trace-decay X, Trust region size e: The ablation studies on these hyperparameters are presented in Appendix C.33 From the results, we recommend setting the trace-decay to 0.95 ~ 0.99 as in other baseed methods (Precup et al.) 2000]. Also, the results show that the performance is not sensitive the trust region size. However, if the trust region size is too large, the approximation eттоr increases, so it is better to set it below 0.003.
Entropy coefficient 3: This value is fixed in our experiments, but it can be adjusted automat- ically as done in SAC [Haarnoja et al. 2018).
The number of atoms M, M': Although experiments on the number of atoms did not performed, performance is expected to increase the number of atoms increases, in other distributional RL methods [Dabney et al., 2018a].
Lenghh of replay buffer: The effect of the length of the replay buffer can be confirmed through the experimental results from an off policy-based safe RL method [Kim and Oh 2022a]. : According to that, the length does not impact performance unless too short. We recommend setting it to 10 to 100 times the collected trajectory length.
Constraint risk level a, threshold de: the cost sum follows a Gaussian distribution, the mean-std constraint is identical to the CVaR constraint. Then, the probability of the
26

---

worst case can be controlled by adjusting a. For example, if we set a = 0.125 and d 0.03/(1 - 7). the mean-std constraint enforees the probability that the average cost is less than 0.03 during an episode greater than 95% ( (a))=a))))/a)) Through this meaning, proper a and dg can be found.
Slack coefficient Ç: As mentioned the end of Section 3.1 it is recommended to set this coefficient as large as possible. Since d₃ - Ç should be positive, we recommend setting Ç to ming dg.
In conclusion, most hyperparameters are not sensitive, so few need to be optimized. It seems that and dg need to be based on the meaning described above. Additionally, if the approximation error of critics is significant, the trust region size should be set smaller.
27

---

C Experimental Results
C.1 Safety Gym
In this section, we present the training curves of the Safety Gym tasks separately according to the risk level of constraints for better readability. Figure 7 shows The training results of the risk-neutral constrained algorithms and risk-averse constrained algorithms with = 1.0. Figures and | show the training results of the risk-averse constrained algorithms with a 0.25 and 0.5, respectively.
Point Goal
Car Goal
Point Button
Car Button
10
Reytare Sum
10
20
10
10
366
100
166
0..5
Cost Rate
0.05
0.05
0.50
0.00
0.50
0.00
306
3000
2000
2000
2000
1900
1000
Steps
226
Sceps
100
Steps
106
Steps
Figure 7: Training curves of risk-neutral constrained algorithms for the Safety Gym tasks. The solid line and shaded area represent the average and std values, respectively. The black dashed lines in the second row indicate thresholds. All methods are trained with five random seeds.
Point Goal
Car Goal
Point Button
Car Button
Raware Sum
70
20
10
106
Cast Rabe
0.30
0.10
0.30
0.56
0.85
0.05
0.05
0.50
0.00
0.00
0.00
300
106
166
1500
1502
2000
1000
1000
300
500
5t0ps
24b
500ps
5teps
Steps
Figure 8: Training curves of risk-averse constrained algorithms with a = 0.5 for the Safety Gym.
28

---

Point Goal
Car Goal
Point Button
Car Button
S3AC
OFTRC
VCSAC
20
Reward Sum
10
10
0.15
Cost Rate
0.10
0.10
0.30
0.10
0.55
0.01
0.85
0.05
0.00
0.00
0.00
0.00
346
160
140
16b
2000
1502
1000
1020
500
Stops
Staps
Staps
Stops
Figure Training curves of risk-averse constrained algorithms with = 0.25 for the Safety Gym.
C.2
Ablation Study on Components of SDAC
There three main differences between SDAC and the existing trust region-based safe RL algorithm for mean-std constraints [Kim and Oh 2022a], called OffRC:: 1) feasibility handling methods in
multi-constraint settings, the use of distributional critics, and 3) the use of Q-functions instead of advantage functions, as explained in Appendix [a.pendix[A.111.. A.88 and A. Since the ablation study for feasibility
handling is conducted Section 5.3 we perform ablation studies for the distributional critic and Q-function in this section. We call SDAC with only distributional critics as SDAC-Dist and SDAC with only O-functions as SDAC-Q. If all components are absent, SDAC identical to OffTRC (Kim and Oh 2022a]. The variants are trained with the point goal task of the Safety Gym, and the training results are shown Figure IO SDAC-Q lowers the cost rate quickly but shows the lowest score. SDAC-Dist shows scores similar to SDAC, but the cost rate converges above the threshold 0.025. In conclusion, SDAC can efficiently satisfy the safety constraints through the use of Q-functions and improve score performance through the distributional critics.
Score
Cost Rate
Total CVs
20
0.125
2500
0.75
1000
Spaci
0.755
302
SEAC Oit
Officc
2
Steps
Steps
Steps
Figure 10: Training curves of variants of SDAC for the point goal task.
29

---

C.3 Ablation Study on Hyperparameters
To check the effects of the hyperparameters, we conduct ablation studies on the trust region size and entropy coefficient B. The results on the entropy coefficient are presented in Figure Tla showing that the score significantly decreases when entropy fail to improve score performance since they focus satisfying the constraints. Thus, the entropy coefficient should be adjusted cautiously, or it can better to set the coefficient to zero. The results on the trust region size are shown in Figure [Tb which shows that the results do not change significantly regardless of the trust region size. However, the score convergence rate for 0.01 the slowest because the estimation error of the surrogate increases as the trust region size increases according to Theorem A.8
0.01. This indicates that policies with high
Srorp
Cost Rate
Tntal
0.150
p-0.0
2500
20
3-0.002
9.0.01
2000
0.100
10
0820
502
0.00
Steps
320
Steps
Steps
3:0
(a) Entropy coefficient |..
Hate
0.150
30
2000
0.300
1500
0030
302
6.00
Steps
Steps
Steps
(b) Trust region size
Figure 11: Training curves of SDAC with different hyperparameters for the point goal task.
30

---

D Comparison with RL Algorithms
In this section, we compare the proposed safe RL algorithm with traditional RL algorithms in the locomotion tasks and show that safe RL has the advantage of not requiring reward tuning. We use the truncated quantile critic (TQC) [Kuznetsov et al., 2020]. state-of-the-ar afr algorithm in existing RL benchmarks [Todorov et al., [012], as traditional RL baselines. To apply the same experiment to traditional RL, it is necessary design a reward reflecting safety. We construct the reward through a weighted sum as = (R E i= w,C,)/(1
3 wt), where R and C(1,2.3) are used to train safe RL methods and are defined in Appendix B and R is called the true reward. The weights of the reward function (112333 are searched by a Bayesian optimization too! to maximize the true reward of TQC in the Mini-Cheetah task. Among the 63 weights searched through Bayesian optimization, the top five weights are listed in Table
Table 4: Weights of the reward function for the Mini-Cheetai task.
Reward weights
W1
W2
Wy
#1
1.588
0.299
0.174
#2
1.340
0.284
0.148
#3
1.841
0.545
0.951
#4
6.560
0.187
4.920
#5
1.603
0.448
0.564
Figure 12 shows the training curves of the Mini-Cheetahh task experiments where TQC is trained using the weight pairs listed in Table 4 The graph shows that it is difficult for TQC to lower the second cost below the threshold while all costs of SDAC are below the threshold. In particular, TQC with the fifth weight pairs shows the lowest second cost rate, but the reward sum is the lowest. This shows that it is challenging to obtain good task performance while satisfying the constraints through reward tuning.
Tue Reward Sum
Cost Rate
Cost Rate
Cost Rate
0.2
S0aC
T0C1
0.6
T0G2
-150
-150
0A
0.4
0.2
1OCK
10C5
TOC3
-20
6.2
9/3
145
Steps
146
Steps
166
Steps
16l
Figure 12: Training curves of the Mini-Cheetah task. The black dashed lines show the thresholds used for the safe RL method. The solid line represents the average value, and the shaded area shows one-fifth the std value. The number after TQC the legend indicates which of the reward weights in Table is used. All methods are trained with five different random seeds.
WWe use Sweeps from Weights & Biases Biewald 2020
31

---

E
Computational Cost Analysis
E.1
Complexity of Gradient Integration Method
In this section, we analyze the computational cost of the gradient integration method. The proposed gradient integration method has three subparts. First, required to calculate policy gradients of each cost surrogate, Gs. and H H 9b for Vk E 21,2,-.. .... K}. where H is the Hessian matrix of the KL divergence. H H- 9k can be computed using the conjugate gradient method, which requires only constant number of back-propagation on the cost surrogate, so the computational cost can be expressed as K (BackPro)).
Second, the quadratic problem in Section 3.1 is transformed to a dual problem, where the trans- formation process requires inner products between Ok and H-1 gm for vk, E (1, K}. The computational cost can be expressed as K² (InnerProod).
Finally, the transformed quadratic problem is solved in the dual space E R using quadratic programming solver. Since K is usually much smaller than the number of policy parameters, the computational cost almost negligible compared to the others. Then, the cost of the gradient integration K O (BackProp) + K² O(InerProd)) + C. Since the back- pock-propagation and the inner products is proportional the number of policy parameters | the computational cost can be simplified as 2
O(K² |+]).
E.2
Quantitative Analysis
Table 5: Training time of Safe RL algorithms (in hours). The training time of each algorithm is measured as the average time required for training with five random seeds. The total training steps are 5 10 and 3 10 for the point goal task and the Mini-Cheetah task, respectively.
Task
SDAC (proposed)
OfTTRC
WCSAC
CPO
CVPO
Point goal (Safety Gym) Mini-Cheetah (Locomotion)
7.96
4.86
19.07
2.61
47.43
8.36
6.54
16.41
1.99
We analyze the computational cost of the proposed method quantitatively. To do this, measure the training time of the proposed method, SDAC, and the safe RL. baselines. We use workstation whose CPU is the Intel Xeon e5-2650 v3, and GPU is the NVIDIA GeForee GTX TITAN X. The results are presented in Table 5 While CPO is the fastest algorithm, its performance, such as the sum of rewards, is relatively poor compared other algorithms. The main reason why CPO shows the fastest computation time is that CPO is an on on-policy algorithm, hence, it does not require an insertion (and deletion from) a replay memory, and batch sampling. SDAC shows the third fastest computation time in all algorithms and the second best one among off-policy algorithms. Especially, SDAC is slightly slower than OfFRRC, which the fastest one among off-policy algorithms. This result shows the benefit of SDAC since SDAC outperforms OfTRCC in terms of the returns and CV, but the training time not significantly increased over OffTRC. WCSAC, which is based on SAC, has slower training time because it updates networks more frequently than other algorithms. CVPÓ, an EM-based safe RL algorithm, has the slowest training time. In the E-step of CVPO, non-parametric policy is optimized to solve a local subproblem, and the optimization process requires discretizing action space and solving a non-linear convex optimization for batch states. Because of this, CVPO takes the longest train RL agent.
32


Reviews:
[{"review_id": "U4KsFiBQFw", "reviewer": "Reviewer_A8gA", "summary": "The paper presents a safe reinforcement learning (RL) algorithm called SDAC for handling multiple constraints in safety-critical robotic tasks. SDAC incorporates risk-averse constraints and makes two key contributions: a gradient integration method for handling infeasibility issues and a TD($\\lambda$) target distribution for estimating risk-averse constraints. Experimental results show that SDAC outperforms safe RL baselines, achieving fewer constraint violations and faster constraint satisfaction.\n", "questions": "- What is the purpose of adding a policy entropy term in line 91? Is it necessary, given the use of trust-region methods, and how is it maximized?\n- Is there an implicit assumption that the distribution is Gaussian considering the risk measure in Equation 2?\n- What is the reason for balancing bias and variance? Even though the bias is low, are there still some negative effects? How can we mitigate them, especially in the context of risk?\n- How will the algorithm work if the constraints completely conflict with each other?\n- When the cost signal is binary, multiple constraints can be taken as one constraint. Then, a method like WCSAC or CPO can also be used. In this case, what are the main advantages of SDAC?", "limitations": "Yes, the limitations are clearly stated at the end of the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, the work is solid and contributes quite some novel ideas for safe RL.\n\n- The formulation of safe RL problem with multiple constraints is very important for real applications.\n- The infeasibility issues in constrained optimization are explicitly considered.\n- The risk-averse safety measures are practical to safety-critical problems.\n\n", "weaknesses": "- While the measures have low bias, it is still important to consider the bias when making decisions.\n- The presentation of the technical details, particularly the quantile regression part, is challenging to comprehend. It might be beneficial to refer to Dabney's paper, which provides a clearer explanation of the related content.\n- From the empirical analysis, the proposed method cannot ensure safety during the early stage of training, despite the theoretical guarantees.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer A8gA for the feedback and thorough review on our work. \nWe appreciate that the reviewer commented that the proposed method is novel and solid.\nWe respond to the reviewer's comments and questions below.\n\n# **Weakness: Presentation of the quantile regression can be improved.**\nAs the reviewer commented, we found that the explanation of the quantile regression part can be difficult to follow due to the omitting intermediate steps in equations (11) and (12).\nWe will include the derivation of each equation in detail and explain the intuition of each equation.\n\n# **Weakness: Safety violation during early stage of training.**\nAs we mentioned in the introduction, infeasible starting cases are typical if there is no prior knowledge.\nIn our experiments, we randomly initialize the policy networks resulting in the infeasible starting case.\nWe can use the gradient integration method to recover the policy to the feasible region to handle the infeasible starting case, which is one of the main contributions of our paper.\n\n# Q1\nThe reason for adding the entropy term is to improve the exploration as in many existing RL papers, including Soft Actor-Critic (SAC) [R1], and it is not a required term.\nWe additionally discussed that the trust region method can still be used even if an entropy term is added in Appendix A.7.\nTo maximize the entropy term, we can construct the objective value as $Q(s, a) - \\beta\\log{\\pi(a|s)}$ and update the policy using backpropagation and reparameterization trick (please see SAC paper [R1]).\n\n# Q2\nThere is no assumption of Gaussian distribution in equation (2).\nNevertheless, if we assume the return follows a Gaussian distribution, equation (2) becomes equivalent to conditional value-at-risk (CVaR), a well-known risk measure in finance.\n\n# Q3\nIn safe RL, since the safety of the policy is evaluated by the cost critics, a policy with the desired safety level can be generated only by training the critics with low biases.\nHowever, since the variance increases as the bias decreases in the TD($\\lambda$) approach, the reward performance may decrease at $\\lambda=1$, as in the ablation study (Figure 3.b).\nFor this reason, balancing the variance and bias is essential.\nIn order to mitigate balancing, it is important to set $\\lambda$, and according to our ablation results, it is good to set a value between 0.9 and 1.0.\nIn the context of risk, we can also increase the training batch size since risk measurement is more sensitive to sample noise.\n\n# Q4\nIf the constraints conflict with each other, the solution of equation (9) does not exist. \nThen the policy is not updated, so the conflict will make the policy get stuck at one point.\nHowever, according to Lemma A.2 in the Appendix of our paper, the solution of equation (9) always exists under our assumption that the constraints are convex and the policy set that satisfies the constraints is not empty.\nAlso, in our experiments, no conflict occurred because the gradients of constraints was calculated stochastically. (Since the dimension of the gradient is much larger than the number of constraints, there is little chance of conflicts in the case of stochastic gradient calculation.)\n\n# Q5\nAs the reviewer commented, we can merge multiple constraints into a single constraint by the \"and\" operator in case of each cost function is binary.\nHowever, if the cost function is integrated in this way, it is difficult to establish a unified safety level (or threshold) that corresponds to the original constraint settings.\nWe can give an example from the locomotion tasks, which have three constraints regarding body balance, CoM height, and feet timing. \nSince the body balance and the CoM height significantly affect the robot's stability, it is necessary to increase the safety level of the corresponding constraints. \nHowever, the constraint on the feet timing is to prevent the robot from standing still and has little relation to safety. \nThus, if the level of this constraint is increased, the robot control may become unstable. \nEven though the integrated cost function can be defined as the result of the \"and\" operation of all cost functions, finding a proper safety level (or threshold) is tricky.\nFor this reason, SDAC with multiple constraints has an advantage that can set the safety levels of each constraint independently.\n\n# References\n\n[R1] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018."}, {"review_id": "GEvDczaCg3", "reviewer": "Reviewer_dXag", "summary": "The paper tries to address the problem of safe RL with multiple constraints with a safe distributional actor-critic (SDAC) approach. The approach includes a gradient integration method to manage the infeasibility issues in multi-constrained problems and a TD$(\\lambda)$ target distribution to estimate risk-averse constraints with low biases. Experimental results show that the proposed approach outperforms the baselines in both single- and multi-constrained tasks. \n", "questions": "1. Seems the performance of the algorithm depends a lot on $\\alpha$ and $\\lambda$, then how can we choose them in the experiments? \n\n2. Since the algorithm is designed to deal with multiple constraints, could the proposed algorithm deal with constraints with different priorities, for example, collisions with humans should be avoided prior to collisions with obstacles? \n\n3. In Figure 3(a), WCSAC with both $\\alpha=0.25$ and $\\alpha=1.0$ satisfies the cost threshold in all environments while in some environments it also has higher rewards compared with SDAC. Could the authors provide some intuition about these observations? \n", "limitations": "The authors discuss the limitations well in Section 6. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Safe Rl with multiple constraints is an important problem. The presented approach uses a gradient integration method to address the infeasibility issues of the trust-region-based algorithms with theoretical guarantees and proposed a TD$(\\lambda)$ loss to reduce the estimation bias of the critics. Sufficient experimental results and ablation studies are provided to support the claims of the paper. \n", "weaknesses": "1. The writing of the paper can be improved. There are little background information and intuitions introduced, which makes the paper hard to read. For example, In Section 2 when introducing the trust-region method with a mean-std constraint, it would be better if the authors can introduce the intuitions behind the equations instead of just putting the equations there, because [Kim and Oh, 2022a] is not a very well-known paper. Also in Section 3.1, little information is provided about the intuition of equation (9). \n\n2. See “Questions”. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer dXag for the feedback and thorough review on our work.\nWe respond to reviewer dXag's suggestions and comments below.\n\n# Weakness: The writing of the paper can be improved.\n\nAs the reviewer commented, in constructing the subproblem in equation (6), many parts are enumerated without formal meaning, making it difficult to read.\nWe will supplement the explanation of why the subproblem is established and what is necessary to establish it (the square value and surrogate functions).\nIn addition, we attached Figure 1 to improve the understanding of equation (9), but the intuition about why the constraints should be truncated to the trust region was missing.\nThe reason for truncating is to make the gradient integration method invariant to the gradient scale. \nOtherwise, a dominant policy gradient may form for constraints with large gradient scales.\nWe will also supplement this explanation.\n\n# Q1: How can we choose $\\lambda$ and $\\alpha$ in the experiments?\n\n$\\lambda$ adjusts the balancing between the bias and variance of the critics.\nWe have experimented with various values of $\\lambda$ in Section 5.3 and Appendix A.5 and found that the value between 0.9 and 1.0 is good as in other TD($\\lambda$)-based papers [R1].\nThus, we recommend setting $\\lambda$ between 0.9 and 1.0.\nTo give an intuition for setting $\\alpha$, we can assume that the returns follow a Gaussian distribution.\nThen, the mean-std constraint becomes setting the probability that the return will be lower than the threshold.\nThus, if you want to the cost return to be smaller than the threshold with a probability of $p$, you can find $\\alpha$ as $p = \\Phi(\\phi(\\Phi^{-1}(\\alpha))/\\alpha)$, where $\\Phi$ and $\\phi$ are the cdf and pdf of the standard normal distribution.\nAlso, we wrote the tips for hyperparameter tuning in Appendix B, so please refer to it for the details.\n\n# Q2: Dealing with constraints with different priorities.\n\nPriority can be set indirectly through the thresholds of constraints.\nFor the reviewer's example case, it can be implemented by setting the constraint for collision with people and the constraint for collision with obstacles separately and setting the threshold of the people constraint lower than the obstacle constraint.\nBy giving different thresholds, we can generate a policy in which the probability of collision with a person is lower than the probability of collision with an obstacle.\nHowever, in decision-making situations where you have to choose between a collision with a person and an obstacle, it is challenging to always prioritize the choice of collision with an obstacle over a person.\n\n# Q3: Comparison of SDAC and WCSAC results in Figure 3.\n\nWe presume that since WCSAC based on SAC [R2] takes a more significant number of gradient steps than SDAC based on TRPO [R3] (about 1000 times more), WCSAC performed better than SDAC on easy tasks. (All algorithms have high scores in the car-goal and point-button tasks, indicating that the tasks are easy.)\nHowever, as mentioned in Q1 of the global response, WCSAC does not significantly change the number of constraint violations even when the $\\alpha$ value is adjusted, so it can be seen that SDAC is better in terms of the risk-averse property.\n\n# References\n\n[R1] Schulman, John, et al. \"High-dimensional continuous control using generalized advantage estimation.\" arXiv preprint arXiv:1506.02438 (2015).\n\n[R2] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.\n\n[R3] Schulman, John, et al. \"Trust region policy optimization.\" International conference on machine learning. PMLR, 2015."}, {"review_id": "TNgUCwspn2", "reviewer": "Reviewer_HJro", "summary": "This study focuses on multiple constraints setting in safe reinforcement learning, and an interesting method is proposed by leveraging gradient integration methods. Moreover, the feasibility of multi-constraint problems is addressed, TD distribution method is introduced to decrease the estimation bias.", "questions": "1. Can we handle multiple constraints by averaging the multiple constraints as one constraint? \n2. The essential contribution is to design the distributional critics with low biases. How about the distributional actors for estimating different types of constraints?\n3. What is the difference between PCPO and this method? In PCPO, they also make projections to optimize reward and cost.\n4. Why does the $J(\\pi)$ like the formation in page 3, line 93? \n5. Why do we need a Shannon entropy?\n6. How does the study have Equations (4) and (5)? Could the study provide analysis to prove them?\n7. Since the method claims the algorithm is efficient and computation efficiency is better than other baselines, could the study provide the sample complexity?", "limitations": "1. The cost is assumed as convex. However, in most cases, the constraints may be nonconvex.\n2. The computation complexity and sample complexity should be provided to prove the effectiveness of this study.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Multiple constraints are considered.\n2. The code is provided.\n3. The writing quality is good.", "weaknesses": "1. Some papers are not investigated, e.g., [1] and [2]\t\n\n[1] Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., ... & Knoll, A. (2022). A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330.\n\n[2] Garcıa, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1), 1437-1480.\n\n2. The experimental results do not convince me, as shown in Figure 3, other baselines also present better performance than this study, e.g., the WCSAC method.\n3. Could you compare the method with PPO Lagrangian?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer HJro for the feedback and thorough review on our work. \nWe respond to reviewer HJro's comments and questions below.\n\n# Weakness: Some papers are not investigated.\n\nI will add the two papers mentioned by the reviewer to the safe RL part of the related work section.\n\n# Weakness: Experimental results in Figure 3.\n\nAs mentioned in Q1 of the general response, it can be observed that SDAC is better than WCSAC in terms of risk-averse property and multi-constraint handling.\nPlease see the general response.\n\n# Weakness: Comparison with PPO-Lagrangian.\n\nWe completed the experiments of PPO-Lagrangian (PPO-L) in the Safety gym and Cassie task, and the experimental results can be found in the attached pdf of the global response.\nSince PPO-L is an on-policy algorithm, it can be confirmed that the convergence speeds are slow compared to other algorithms.\nAlso, like other lagrangian methods, PPO-L has the disadvantage of unstable training because multipliers and policies are updated simultaneously.\n\n# Q1: Merging multiple constraints into a single constraint.\n\nIt is difficult to handle multiple constraints as a single constraint by averaging because the feasible regions of the multiple constraints, $F_k(\\pi) \\leq d_k$ $\\forall k$, and the averaged constraint, $\\sum_kF_k(\\pi)/K \\leq \\sum_k d_k/K$, are quite different.\nMatching the feasible region using a single constraint requires designing a new integrated cost function rather than taking the average. Still, as mentioned in the introduction as a drawback of RL, this process is time-consuming.\n\n# Q2: How about the distributional actors for estimating different types of constraints?\n\nSDAC can be extended to use other types of constraints using the distributional critic, but it is essential to find the upper bound of the constraint to apply the trust region method.\nNo study has yet derived the upper bound of risk-based constraints except for the mean-std constraint, so we can deal with this extension as future work.\n\n# Q3: Difference from PCPO.\n\nPCPO only deals with a single constraint but can be extended to multiple constraint settings.\nNevertheless, the most significant difference from our method is that PCPO does not guarantee that the policy will converge to the feasible region for the infeasible starting case, but we can.\nHowever, we did not cite the PCPO paper, so we will cite this paper.\n\n# Q4: Why does the $J(\\pi)$ like the formation in page 3?\n\nIf we replace the $Z_R^\\pi$ with the definition in $J(\\pi)$, it becomes $\\sum_t \\gamma^t(R(s_t, a_t, s_{t+1}) + H(\\pi(\\cdot|s_t)))$, equivalent to the entropy-regularized RL problem [R1].\nWe will explain why the entropy term is added in the answer to the next question.\n\n# Q5: Why do we need a Shannon entropy?\n\nThe reason for adding the entropy term is to improve the exploration as in many existing RL papers, including Soft Actor-Critic (SAC) [R1], and it is not a required term.\nWe additionally discussed that the trust region method can still be used even if an entropy term is added in Appendix A.7.\n\n# Q6: How does the study have Equations (4) and (5)?\n\nSince $\\mathrm{Std}[Z_{C_k}^\\pi]=\\sqrt{\\mathbb{E}[(Z_{C_k}^\\pi)^2] - \\mathbb{E}[Z_{C_k}^\\pi]^2} = \\sqrt{J_{S_k}(\\pi) - J_{C_k}(\\pi)^2}$, we can achieve equation (4) by substituting the Std term in equation (2).\nEquation (5) defines the surrogate functions for the objective and mean-std constraints.\nThe condition for the surrogate function in the trust region method is that the difference between the original function and the surrogate should be bounded by the trust region size.\nFor the objective's surrogate, we derive the bound in Appendix A.7, and the bound for the mean-std constraint's surrogate is derived in [R2].\n\n# Q7: Sample complexity.\n\nWe analyzed the sample complexity of the proposed method in Q2 of the general response. \nPlease refer to the general response.\n\n# References\n\n[R1] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.\n\n[R2] Kim, Dohyeong, and Songhwai Oh. \"Efficient off-policy safe reinforcement learning using trust region conditional value at risk.\" IEEE Robotics and Automation Letters 7.3 (2022): 7644-7651."}, {"review_id": "SWd1OPbJW3", "reviewer": "Reviewer_Cyuw", "summary": "The paper proposed a trust-region method for handling multiple constraints in safe RL for CMDPs. If the TRPO-like gradient calculation with multiple constraints is not feasible, the authors proposed a gradient integration method to calculate feasible gradients. The authors also proposed a TD-$\\lambda$ method to calculate the target distribution of distributional critic. Experimental results have shown that the proposed methods seem to outperform baselines.", "questions": "1. I got confused about the figures of the first column in Figure 4. The curves and shades are not continuous. Please explain why.\n\n*Updates after rebuttal: Clarified in the reply. They are random samples.*\n\n2. For Figure 1, what if the $g_1$ and $g_2$ are parallel to each other (no feasible region), or have an arbitrarily small relative angle? Then you integrate them to near zero and get stuck. Then I doubt the conclusion about finite-time convergence to feasible region. You might need more assumptions like the feasible region does exist. Do you want to comment on these situations?\n\n3. Following problems in 2, it seems like no guarantees like the proposed gradient integration could be faster than naive methods could be made. Then how exactly can safe RL methods benefit from the gradient integration?\n\n3. Practically, Solving TRPO is already computationally heavy. Locally linearization of multiple constraints and objectives will cause more approximation and further harm the performance. If you only would like to handle multiple constraints, many straightforward approaches could be used like first optimizing a weighted sum of multiple constraint critics. How does the proposed algorithm outperform these intuitions then?\n\nPlease carefully explain these questions. I will consider increasing the score if the authors give me a reasonable response.", "limitations": "The authors addressed the limitations well.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "## Originality\nThe gradient integration is novel and looks useful for the infeasible initial stage when solving CMDP. Computing the TD $\\lambda$ for the distributional critic is also novel to me. \n\n## Quality\nThe paper is well-written and includes many technical details, which is good for understanding and reproducing the work.\n\n## Clarity\nThe paper is very clear. Especially, I like the two figures which explain the ideas very well and I get the idea immediately. The notation is a little bit messy but clear enough to read.\n\n## Significance\nThe TD-$\\lambda$ method for distributional critic might have a broad effect on other distributional RL algorithms, but I am not sure if similar approaches have been proposed to solve the problem.", "weaknesses": "1. The paper has novelty but the contributions are not clearly demonstrated. From the results in Figures 3 and 4, I did not see too much performance improvement, solid constraint satisfaction, or faster convergence to feasible policies compared to baselines. Given this paper focuses on practical contributions, the performance results are not convincing. \n\n2. Two contributions of this paper are not related. Gradient integration and TD-$\\lambda$ seem to stand alone on their own. The authors should explain why you put these into one paper and how they contribute together to improve the safe RL.\n\n3. The gradient integration problem is fragile and might have some problems in the finite-convergence theorem. I will put more discussion in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Cyuw for the feedback and thorough review on our work.\nWe respond to reviewer Cyuw's comments and questions below.\n# Weakness: The performance are not convincing\n**Performance improvement**\nThe most comparable method with SDAC in the safety gym is WCSAC.\nHowever, as mentioned in Q1 of the general response, it can be said that the proposed method is better than WCSAC regarding risk-averse property and multi-constraint setting.\nPlease see Q1 of the general response.\nAlso, in the locomotion tasks, SDAC shows the highest reward sum and meets constraints the fastest in all tasks.\nThus, it can be seen that the performance is improved in the risk-averse and multi-constraint settings.\n\n**Constraint satisfaction**\nIn the safety gym tasks, since the optimal policy exists on the boundary of the constraint, the training results exist at the boundary rather than rigidly satisfying the constraints.\nNevertheless, a risk-averse constraint can be used by setting $\\alpha=0.25$ to meet the constraint solidly.\nFigure 3.a shows that SDAC with $\\alpha=0.25$ has the lowest cost (showing solid constraint satisfaction) and is in the middle or upper ranks of the reward sum.\n\n**Convergence to feasible policies**\nThe convergence speed can be measured by the shortest step satisfying all constraints. \nThe proposed method satisfied constraints at 1.5e6 step in Cassie, 0.25e6 step in Laikago, and 0.2e6 step in Cheetah, but the second-best method satisfied constraints at 3.5e6 step, 0.7e6 step, and 0.25e6 step in each task.\nThus, the proposed method converged to the feasible policies the fastest.\n# Weakness: Two contributions are not related\nAs mentioned in the introduction, if the estimation bias of critics becomes large, the policy can be trained to be overly conservative or risky.\nIf the number of constraints increases, the possibility of updating the policy in a wrong direction rises exponentially.\nThus, the importance of reducing the bias in the multiple constraint setting increases.\nTo this end, we contribute to reducing the estimation bias of critics while handling multiple constraints.\n# Q1\nIn the implementation, the policy takes random actions during the initial 10000 steps, like other SAC-based algorithms.\nAs a result, low rewards were received, making the learning graphs discontinuous.\nWe have attached an enlarged graph for the initial steps to the pdf of the general response.\nPlease refer to the pdf.\n# Q2\nLemma A.2 in the Appendix shows that the solution of equation (9) always exists under assumptions that the constraints are convex and the feasible policy set is not empty (see the assumptions introduced in Theorem 3.1).\nTherefore, there is no conflict between constraints due to the existence of the solution.\n# Q3\nIn Q2 of the general response, the worst-case time to meet all constraints of our method is expressed as:\n$$\\frac{D_\\mathrm{KL}}{\\beta(1-\\gamma)(\\zeta-\\frac{2\\beta C_\\mathrm{max}}{(1-\\gamma)^2})\\mathbb{E}\\_t[\\sum_i\\lambda^t_i/W_t]}.$$\nThe worst-case time of the naive method is the same as the above equation, except for the $\\lambda$ part.\nIn the naive method, $\\lambda^t$ is an one-hot vector where $i$-th value, $\\lambda_i^t$, is one only for corresponding to the randomly selected constraint.\nIn other words, only $\\mathbb{E}\\_t[\\sum_i\\lambda_i^t/W_t]=\\mathbb{E}\\_t[\\sum_i\\lambda_i^t/||\\sum_i\\lambda_i^tA\\_{C_k}^t||\\_2]$ is different.\nLet us assume that the advantage vector follows a normal distribution. \nThen, the variance of $\\sum_i\\lambda_i^tA_{C_k}^t$ is smaller for lambdas with distributed values than for one-hot values.\nThen, the reciprocal of the 2-norm becomes larger, resulting in a decrease in the worst-case time.\nFrom this, the gradient integration has a benefit over the naive method as it reduces the variance of the advantage vector.\nOf course, we cannot officially say that the worst-case time of the proposed method is smaller than the naive method because the advantage vector does not follow the normal distribution. Still, we can deliver our insight on the benefit of gradient integration.\nAlso, the proposed method shows better results than the naive method in the multi-constrained experiments of Section 5.2.\nIn Figure 4.d, the proposed method takes 1.5e6 steps to meet all the constraints, but the naive method takes about 2.5e6 steps.\nAlso, the naive method shows an unstable training as some constraints are violated even after the 2.5e6 step.\n# Q4\nAs the reviewer commented, TRPO is computationally heavy, but in safe RL, it can be said that TRPO-based policy updates are more effective than first-order (or gradient descent) updates.\nIf a policy is updated by the first-order method, the policy can change rapidly, resulting in the constraints being violated suddenly, but TRPO-based update allows more stable learning because the values of the constraints do not change significantly.\nFor this reason, many safe RL algorithms are based on TRPO [R1, R2].\n\nSecond, as the reviewer mentioned, the policy can be updated using weighted sum of the cost critics to deal with multiple constraints.\nThis method exactly matches the Lagrangian method of updating the policy to minimize the weighted sum of the Lagrange multipliers and cost critics.\nHowever, since these methods usually update the policy and weights simultaneously, the policy update direction can easily fluctuate.\nIn contrast, due to the strong duality of the QP problem, the solution of equation (9) can be expressed as $\\sum_k \\lambda_k H^{-1}g_k$, so the gradient integration we propose can also be seen as a weighted sum method.\nUnlike the Lagrangian methods, it automatically calculates the appropriate weight by solving the QP problem at each gradient step, so it has the advantage of being able to find the policy update direction more stably.\n# **References**\n[R1] Achiam et al. \"Constrained policy optimization.\" ICML, 2017.\n\n[R2] Xu et al. \"Crpo: A new approach for safe reinforcement learning with convergence guarantee.\" ICML, 2021."}]

Author rebuttal:
# General response
We thank all reviewers for their valuable comments and suggestions.
In the following, we respond to the comments on comparison with WCSAC and the sample complexity.
### **Q1. Comparison with WCSAC.**
We first examine the experiments in Section 5.1.
For $\alpha=1.0$ (risk-neutral constraint setting), SDAC achieves higher reward sums in the point-goal and car-button tasks, whereas WCSAC shows higher in the other two tasks.
The same results are observed at $\alpha=0.25$ (risk-averse setting).
However, risk-averse properties should also be taken into account in interpreting the results.
WCSAC uses risk-averse constraints like SDAC, so it should be able to significantly reduce the number of constraint violations at $\alpha=0.25$, but it failed to do so. (For an analysis of this reason, please refer to Section 5.1 of the paper.)
From this observation, SDAC has better risk-averse properties than WCSAC.
Moreover, in the multiple constraints setting (Section 5.2), whether in terms of reward or constraint satisfaction, SDAC outperforms WCSAC.
Hence, it can be concluded that SDAC is better than WCSAC in both the risk-averse and multi-constraint settings.
### **Q2. Sample complexity.**
To analyze the sample complexity, we consider tabular MDPs and use softmax policy parameterization (see [R1]).
It is challenging to analyze the sample complexity of TRPO-based methods since their stepsize is not fixed, so using natural policy gradient (NPG) methods is common, as done in [R1].
Thus, we introduce the NPG version of our method as follows.  
1. (Recovery) If the constraints are not satisfied at $t$ step, we take a recovery step using the gradient integration as: $g^*=\mathrm{argmin}_g\frac{1}{2}g^THg$ s.t. $g_k^Tg+c_k\leq 0$ $\forall k\in$ {$k|F_k(\pi_t;\alpha)>d_k$}, $\psi\_{t+1}=\psi_t+\beta g^*/||g^*||\_2$ ,where $\beta$ is a stepsize, and other notations are the same as in the paper.
2. (Normal) If the constraints are satisfied at $t$ step, we store $t$ in a set $\mathcal{N}$ and maximize the objective as: $\psi_{t+1} = \psi_{t} + \beta A_R^{\pi_t}/||A_R^{\pi_t}||_2,$ where $A_R^{\pi_t}$ is the vector expression of the advantage function $\in \mathbb{R}^{|S| |A|}$. Derivation of the gradient can be referred to [R1].

According to the Lemma A.2 in our paper, $g^*$ in the recovery step always exists and can be expressed as: $g^* = \sum_k \lambda_k H^{-1}g_k = \sum_k \lambda_k A_{C_k}^\pi$, where $\lambda_k \geq 0$.

#### **Analysis of worst-case time to satisfy all constraints**
During the recovery step, the policy can be expressed as:  
$$\psi\_{t+1}=\psi_t-\beta\sum_k\lambda^t_kA\_{C_k}^t/W_t,\ W_t:=||\sum_k\lambda^t_kA\_{C_k}^t||\_2,\ \pi\_{t+1}(a|s)=\pi_t(a|s)\exp{\left(-\frac{\beta}{W_t}\sum_k\lambda^t_kA\_{C_k}^t(s, a)\right)}/Z_t(s),$$
where $Z_t(s)$ is a normalization factor.
We can get the followings by using the Lemma 6 in [R1]:
$$\sum_k\lambda^t_k(F_k(\pi\_{t+1})-F_k(\pi_t))/W_t=\frac{1}{1 - \gamma}\mathbb{E}\_{s\sim\nu_\rho}\left[\sum_a\pi\_{t+1}(a|s)\sum_k\lambda^t_kA\_{C_k}^t(s, a)/W_t\right]\leq-\frac{1}{\beta}\mathbb{E}\_{s\sim\rho}\left[\log{Z_t(s)}\right],$$ 
where we abbreviate $F_k(\pi_t;\alpha)$ as $F_k(\pi_t)$.
We can also get the followings by using the Lemma 7 in [R1]:
$$\sum_k\lambda^t_k(F_k(\pi_*)-F_k(\pi_t))/W_t\geq-\frac{1}{\beta(1-\gamma)}\mathbb{E}\_{s\sim\nu_*}\left[D_\mathrm{KL}(\pi_*||\pi_t)-D_\mathrm{KL}(\pi_*||\pi\_{t+1})\right]-\sum_k\lambda^t_k\frac{2\beta C_\mathrm{max}}{(1-\gamma)^2W_t},$$
where $\pi_*$ is an optimal policy, $C_\mathrm{max}$ is the maximum value of reward and costs.
If $\lambda^t_k > 0$, $F_k(\pi_{t}) - F_k(\pi_*) > \zeta$.
Thus, $\sum_k \lambda^t_k(F_k(\pi_*) - F_k(\pi_{t})) \leq -\zeta\sum_k \lambda^t_k$.
If the policy does not satisfy the constraints until $T$ step, the following inequality holds by summing the above inequalities from $t=0$ to $T$:
$$\beta(1-\gamma)(\zeta-\frac{2\beta C_\mathrm{max}}{(1-\gamma)^2})\sum_{t=0}^T\sum_i\lambda^t_i/W_t\leq\mathbb{E}\_{s\sim\nu_*}\left[D_\mathrm{KL}(\pi_*||\pi_0)\right].$$
Let denote $\frac{1}{T}\sum_{t=0}^T \sum_i \lambda^t_i / W_t$ as $\mathbb{E}\_t[\sum_i\lambda^t_i/W_t]$, and we can get $W_t=||\sum_k \lambda^t_kA\_{C_k}^t||\_2\leq\sum_k\lambda^t_k2|S||A|C_\mathrm{max}/(1-\gamma)$.
Then, the maximum $T$ can be expressed as:
$$
T\leq\frac{D_\mathrm{KL}}{\beta(1-\gamma)(\zeta-\frac{2\beta C_\mathrm{max}}{(1-\gamma)^2})\mathbb{E}\_t[\sum_i\lambda^t_i/W_t]}\leq\frac{2|S||A|C_\mathrm{max}D_\mathrm{KL}}{\beta (1-\gamma)^2\zeta-2\beta^2C_\mathrm{max}}=:T_\mathrm{max},
$$
where we abbreviate $\mathbb{E}\_{s\sim\nu_*}\left[D_\mathrm{KL}(\pi_*||\pi_0)\right]$ as $D_\mathrm{KL}$.
Finally, the policy can reach the feasible region within $T_\mathrm{max}$ steps.

#### **Convergence rate**
For the normal step, we also can get the following inequality as done in the recovery step:
$$
\frac{(1-\gamma)^2\beta}{2C_\mathrm{max}}(J(\pi_*)-J(\pi_t))-\beta^2\leq|S||A|\mathbb{E}\_{s\sim\nu_*}\left[D_\mathrm{KL}(\pi_*||\pi_t)-D_\mathrm{KL}(\pi_*||\pi\_{t+1})\right].
$$
If we sum up the above inequalities from $t=0$ to $T$, we can get the following inequality:
$$
\frac{(1-\gamma)^2\beta}{2C_\mathrm{max}}(\sum_{t\in\mathcal{N}} (J(\pi_*) - J(\pi_t)) + \zeta (T - |\mathcal{N}|)) \leq |S||A|D_\mathrm{KL} + \beta^2T
$$, and according to Lemma 9 in [R1], the set $\mathcal{N}$ is not empty if $T$ is large enough.
Then, if we schedule $\beta = 2C_\mathrm{max}\sqrt{|S||A|/T}$ and $\zeta=2(D_\mathrm{KL} + 4C_\mathrm{max}^2)\sqrt{|S||A|/T}$, we can get convergence rate as: $J(\pi_*)-\mathbb{E}\_{t\in\mathcal{N}}[J(\pi_t)] = 2\sqrt{|S||A|/T}(D_\mathrm{KL} + 4C_\mathrm{max}^2)/(1 - \gamma)^2 = \mathcal{O}(1/\sqrt{T})$.

### **References**
[R1] Xu, Tengyu, Yingbin Liang, and Guanghui Lan. "Crpo: A new approach for safe reinforcement learning with convergence guarantee." International Conference on Machine Learning. PMLR, 2021.

Meta-review: There is a consensus among reviewers that this paper meets the threshold for acceptance. There is a broad view that the paper is tackling an interesting problem, and that dealing with multiple constraints and infeasibility are helpful in expanding the applicability of the contributions. Some drawbacks were also identified by reviewers in the original reviews and discussion. There is some disagreement among reviewers on the clarity of the paper, and some views that the assumptions required in the core theoretical results may be too strong to be applicable in certain scenarios (e.g positive lower bounds on Hessian eigenvalues of constraint functions). Overall the decision is to accept this paper to NeurIPS; I hope the the reviews and results of the author-reviewer discussion help the authors in preparing their camera-ready version.
Decision: Accept (poster)