SDAC: Efficient Safe Reinforcement Learning with Low-Biased Distributional Actor-Critic
Authors: Dohyeong Kim
Kyungjae Lee
Songhwai Oh
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
Safety
Distributional Critic

Abstract: To apply reinforcement learning (RL) to real-world practical applications, agents are required to adhere to the safety guidelines of their respective domains.
Safe RL can effectively handle the guidelines by maximizing returns while maintaining safety satisfaction.
In this paper, we develop a safe distributional RL method based on the trust region method which has the capability of satisfying safety constraints consistently.
However, importance sampling required for the trust region method can hinder performance due to its significant variance, and policies may not meet the safety guidelines due to the estimation bias of distributional critics.
Hence, we enhance safety performance through the following approaches.
First, we propose novel surrogates for the trust region method expressed with Q-functions using the reparameterization trick.
Second, we utilize distributional critics trained with a target distribution where bias-variance can be traded off.
In addition, if an initial policy violates safety constraints, there can be no policy satisfying safety constraints within the trust region.
Thus, we propose a gradient integration method which is guaranteed to find a policy satisfying multiple constraints from an unsafe initial policy.
From extensive experiments, the proposed method shows minimal constraint violations while achieving high returns compared to existing safe RL methods.
Furthermore, we demonstrate the benefit of safe RL for problems in which the reward function cannot be easily specified.

Under review as a conference paper at ICLR 2023
SDAC: EFFICIENT SAFE REINFORCEMENT LEARNING WITH LOW-BIAS DISTRIBUTIONAL ACTOR-CRITIC
Anonymous authors Paper under double-blind review
ABSTRACT
To apply reinforcement learning (RL) to real-world applications, agents are re- quired to adhere to the safety guidelines of their respective domains. Safe RL can effectively handle the guidelines by converting them into constraints of the RL problem. In this paper, we develop a safe distributional RL method based on the trust region method, which can satisfy constraints consistently. However, policies may not meet the safety guidelines due to the estimation bias of distribu- tional critics, and importance sampling required for the trust region method can hinder performance due to its significant variance. Hence, we enhance safety per- formance through the following approaches. First, we train distributional critics to have low estimation biases using proposed target distributions where bias variance can be traded off. Second, we propose novel surrogates for the trust region method expressed with Q-functions using the reparameterization trick. Additionally, de- pending on initial policy settings, there can be no policy satisfying constraints within a trust region. To handle this infeasible issue, we propose a gradient inte- gration method which guarantees to find a policy satisfying all constraints from an unsafe initial policy. From extensive experiments, the proposed method with risk-averse constraints shows minimal constraint violations while achieving high returns compared to existing safe RL methods. Furthermore, we demonstrate the benefit of safe RL for problems in which the reward cannot be easily specified.
INTRODUCTION
Deep reinforcement learning (RL) enables reliable control of complex robots (Merel et al. 2020 Peng et al. 202T Rudin et al. 2022). Miki al. (2022) have shown that RL can control quadrupedal robots more robustly than existing madel-based optimal control methods, Pnng et al. (2022) have performed complex natural motion tasks using physically simulated characters. order to success- fully apply RL to real-world systems, it is essential to design a proper reward function which reflects safety guidelines, such as collision avoidance and limited energy consumption, as well as the goal of the given task. However, finding the reward function that considers all of such factors involves cumbersome and time-consuming task since RL algorithms must be repeatedly performed to verify the results of the designed reward function. Instead, safe RI,, which handles safety guidelines con- straints, can be an appropriate solution, A safe RL problem can be formulated using a constrained Markov decision process [Altman] 1999), where not only the reward but also cost functions, which output the safety guideline signals, are defined. By defining constraints using risk measures, such as condtional value at risk (CVaR), of the sum of costs, safe RL aims to maximize returns while satis- fying the constraints. Under the safe RL framework, the training process becomes straightforward since there is no need to search for a reward that reflects the safety guidelines.
The most crucial part of safe RL is to satisfy the safety constraints, and it requires two conditions. First, constraints should be estimated with low biases. In general RI., the return is estimated using a function estimator called critic, and, in safe RI, additional critics are used to estimate the constraint values. In our case, constraints are defined using risk measures, so it is essential to use distributional critics (Dabney et al.] 2018b). Then, the critics can be trained using the distributional Bellman update (Bellemare et al. 2017). However, the Bellman update only considers the one-step temporal difference, which can induce large bias. The estimation bias makes it difficult for crities to judge the policy, which can lead to the policy becoming overly conservative or risky, as shown in Section 5.3 Therefore, there is need for method that can train distributional crities with low biases.

---

Under review as a conference paper at ICLR 2023
Second, policy update method considering safety constraints, denoted by a safe policy update rule, is required not only to maximize the reward sum but also to satisfy the constraints after updating the policy. Existing safe policy update rules can be divided into the trust region -based and Lagrangian methods. The trust region-based method calculates the update direction by approximating the safe RL problem within trust region and updates the policy through a line search (Yang et al.) 2020 Kim & Ohl 2022a). The Lagrangian method converts the safe RL problem into a dual problem and updates the policy and Lagrange multipliers (Yang et al. 2021). However, the Lagrangian method is difficult to guarantee satisfying constraints during training theoretically, and the training process can be unstable due to the multipliers (Stooke et al., 2020 In contrast, trust region- based methods can guarantee to improve returns while satisfying constraints under tabular settings (Achiam et al.) 2017]. Still, trust region-base based methods also have critical issues. There can be an infeasible starting case, meaning that no policy satisfies constraints within the trust region due to initial policy settings. Thus, proper handling of this case is required, but there is a lack of such handling methods when there are multiple constraints. Furthermore, the trust region- brasd methods are known not sample- efficient, as observed in several RL benchmarks (Achiam| 2018: Rafin et al. (02T).
In this paper, we propose efficient trust eggion-based safe RL algorithm with multiple constraints, called a safe distributional actor-critic (SDAC). First, to train critics to estimate constraints with low biases, we propose a TD(X) larget distribution combining multiple-step distributions, where bias- variance can be traded off by adjusting the trace-decay X. Then, under off-policy settings, we present a memory-efficienne method to approximate the TD(A) target distribution using quantile distributions (Dabney et al. 8b), which parameterize a distribution as sum of Dirac functions. Second, to handle the infeasible starting case for multiple constraint settings, we propose a gradient integration method, which recovers policies by reflecting all constraints simultaneously. It guarantees to obtain a policy which satisfies the constraints within a finite time under mild technical assumptions. Also, since all constraints are reflected at once, it can restore the policy more stably than existing handling methods Xu et al. (202T), which consider only one constraint at a time, Finally, to improve the efficiency of the trust region method as much Soft Actor-Critic (SAC) (Haarnoja et al.. 2018). propose novel SAC-style surrogates. We show that the surrogates have bounds within a trust region and empirically confirm improved efficiency in Appendix B In summary, the proposed algorithm trains distributional critics with low biases using the TD(A) target distributions and updates a policy using safe policy update rules with the SAC-style surrogates. If the policy cannot satisfy constraints within the trust region, the gradient integration method recovers the policy to a feasible policy set.
To evaluate the proposed method, we conduct extensive experiments with four tasks in the Safety Gym environment (Ray et al. 2019) and show that the proposed method with risk-averse constraints achieves high returns with minimal constraint violations during training compared to other safe RL baselines. Also, we experiment with locomotion tasks using robots with different dynamic and kinematic models to demonstrate the advantage of safe RL over traditional RL,, such as no reward engineering required. The proposed method has successfully trained locomotion policies with the same straightforward reward and constraints for different robots with different configurations.
BACKGROUND
Constrained Markov Decision Processes. We formulate the safe RL problem using constrained Markov decision processes (CMDPs) [llmann| 1999). A CMDP is defined as (5, A, P, R. Ci,.. p. 7) where S is a state space, is an action space, P S x A x S + [0, 1] is a transition model, R : S x A x S + R is a reward function, Cx, S x A x S + Rz0 are cost functions, p: S - [0, 1] is an initial state distribution, and y E 10 is a discount factor. The state action
value, state value, and advantage functions are defined follows:
Qh(s,a) == E. T.P
Σ o Rise,, (d,s,,,,, 8(+1) = =8,00
Vi(()
E
Σ
R'Res ae, sc+1) so AAl(s, -)((
-
By substituting the costs for the reward, the cost value functions V2 (s), Q7 (s, A (s. are defined. In the remainder of the paper, the cost parts will be omitted since they can be retrieved by replacing the reward with the costs. Given policy æ from stochastic policy set IT, the discounted
2

---

Under review as a conference paper at ICLR 2023
state distribution is defined d= (s) == ?'Po = s/T), and the return is defined as 7 (s, a) Ei=o 7 R(s,, dt, 8(+1), where = a, at T T-11s1), and 81+1 P(-|st, az).
Prr(s:
Then, the safe RI. problem is defined as follows with a safety measure
max E [Z a)}s ~ p. (+11s) s.t. F(ZZ, (8,0)) ~ p.a S dx Ak.
(2)
x
where de is a limit value of the kth constraint.
Mean-Std Constraints. In safe RL setting,
mean-std as the safety measure: F(Z: 0)
E[Z] + (6( (a))/o) Std where a E (0, 1] adjusts conservativeness of constraints, Std the standard deviation of Z, Φ is the probability density function, and & is the cumulative dis- tribution function (CDF) of the standard normal distribution. The mean-std is identical to the conditional value risk (CVaR) if Z follows the Gaussian distribution, and the mean-std con- straint can effectively reduce the number of constraint violations, as shown by Yang et al. al.] (202T); Kim & Oh (2022b a). To estimate the med of cost returns, Kim & Oh (2022b) define the square value functions: St, (+) = E 78. (8, a) (+10)] 36 (8,a)
E
1 (s. a)
T.P.
and A3, (s, a) == ST, (s, ST, (s). Additionally, ds)s) = (1 - 7²) Et 7 Pr(ss S|T)
denotes doubly discounted state distribution. Then, the kth constraint can be written as follows: (()
FT(TT Jos (T)
V Js, (T) Jc, (T)) de.
where Jc, (x) = srup VE, (s) and Js. (T)
IE grup
(s)(s)
Distributional Quantile Critic. To parameterize the distribution of the returns, Dabney et al. (2018) have proposed an approximation method to estimate the returns using the following quantile distributions, called distributional quantile critic: Pr(Zg,o(s a) = 2) := EM 10== (s)) where M the number atoms, 0 a parametric model, and Om (8, a) is the mith atom. The per- centile value of the mtth atom is denoted by Tm (To 0, T, i/M). ). In distributional RL, the returns are directly estimated to get value functions, and the target distribution can be calculated from the distributional Bellman operator (Bellemare et al. 2017): T (x($,a) :2 R(s, a, 8') R ,'),,,",, where s' ~ P(-|s, a) and a' ~ T(- The above one-step operator can be expanded to the n-step one: ZT ( ao) _ Ei=o 7 R(s,,, as, $1+1) + + "n"ZZa ( an). Then, the critic can be trained to minimize the following quantile regression loss (Dabney al. 2018b):
MM
(())=
Σ m=1
E Ezaz [ptm (Z 0m)), where p=(x)= x (r 1xc0), Am 7m Tm
(4)
2
= (9m)
and FRR (0) denotes the quantile regression loss for single atom. The distributional quantile critic can be plugged into existing actor-critic algorithms because only the critic modeling is changed.
3 PROPOSED METHOD
We propose the following three approaches to enhance the safety performance of trust region- safe RL methods. First, we introduce TD(A) target distribution combining n-step distributions, which can trade off bias-variance. The target distribution enables training of the distributional critics with low biases. Second, we propose novel surrogate functions for policy updates that empirically improve the performance of the trust region method. Finally, we present a gradient integration method under multiple constraint settings to handle the infeasible starting cases.
3.1 TD(A) TARGET DISTRIBUTION
In this section, we propose target distribution by capturing that the TD(A) loss, which is obtained by a weighted sum several losses, and the quantile regression loss with a single distribution are equal. A recursive method is then introduced so that the target distribution can be obtained practically. First, the stspe targets are estimated as follows, after collecting trajectories (St, dr, St.1y...) with a behavioral policy μt:
(/)) _2 Rd 7 Reti + 2 + +...
... + n- R R + 7"' 7 (8tt d(tn),
(5)
3

---

Under review as a conference paper at ICLR 2023
Projection
2(2)
W1
tot
zoroj
If atoms
Restore
R2 + y
0 0p0
CDF
M' atoms
6)mm)
Figure 1: Constructing procedure for target distribution. First, multiply the target at t step by and add R.. Next, weight-combine the shifted previous target and one-step target at t step and restore the CDF of the combined target. The CDF can be restored by sorting the positions of the atoms and then accumulating the weights at each atom position. Finally, the projected target can be obtained by finding the positions of the atoms corresponding M' quantiles the CDF. Using the projected target, the target at t 1 step can be found recursively.
where R. R(st, Ge. St+1), d'+n [T+[St+n), and π is the current policy. Note that the n-step target controls the bias-variance tradeoff using n. If n is equal to 1, the n-step target is equivalent goes
the temporal difference method that has low variance but high bias. On the contrary, to infinity, it becomes Monte-Carlo estimation that has high variance but low bias. However, finding proper n another cumbersome task. To alleviate this issue, TD(A) (Sutton) 1988) method considers the discounted sum of all n-step targets. Similar to TD(A), we define the TD(X) loss for the distributional quantile critic as the discounted sum of all quantile regression losses with n-step targets. Then, the TD(A) loss for single atom is approximated using importance sampling of the sampled n-step targets in (5) as:
1-1 80 M
n. (actg1884)) 8845)
(Qx(0)=(1-A)) Σ X'EZ (2 [2]]
p+ (2((t))
0).
(6)
5=1
where À is a trace-decay value, and (1)
2m, is the mth atom of 2/1)
Since
2.00
D R.
(1-1) is
satisfied, (6) is the same as the quantile regression loss with the following single distribution 2, called TD(X) target distribution:
((r+))lett) pla++s (8tts) mm1
M
Pr(21 2) N
(2)
M
1
(1
1 M 1==1
Σ
M
6
20 (x) -x (act1118411 ((ac41ls441)
Pr(Re + +7 22g1
N
Presious targer
One step TD target
where N is normalization factor. We show that distribution trained with the proposed target converges to the distribution of 2T in Appendix If the target for time step t + 1 is obtained, the target distribution for time step becomes the weighted sum of the current one-step TD target and the shifted previous target distribution, so it can be obtained recursively, as shown in (7). However, to obtain the target distribution, we need store all quantile positions and weights for all time steps, which is not memorr-eeficientt Therefore, we propose to project the target distribution into quantile distribution with a specific number of atoms, .M' (we set M' -M reduce information loss). The overall process to get the TD(A) target distribution is illustrated in Figure | and the pseudocode given in Appendix A.2 After calculating the target distribution for all time steps, the critic can be trained to reduce the quantile regression loss with the target distribution.
3.2 SAFE DISTRIBUTIONAL ACTOR-CRITH
SAC-Style Surrogates. Here, we derive efficient surrogate functions for the trust region method. While there are two main streams in trust region methods: trust region policy optimization (TRPO)
(Schulman et al. 2015) and proximal policy optimization (PPO) (Schulman et al. (017). however, we only consider TRPO since PPO is an approximation of TRPO by only considering the sum of rewards and, hence, cannot reflect safety constraints. There are several variants of TRPO (Nachum et al. 2018, Wang al. 2017). among which off-policy TRPO (Meng et al., 2022) shows signifi- cantly improved sample efficiency by using off policy data. Still, the performance of SAC outper- extend the surrogate of off off-policy TRPO similar
forms off-polic TRPO (Meng et al. 2022),
4

---

Under review as a conference paper at ICLR 2023
to the policy loss of SAC. To this end, the surrogate should 1) have entropy regularization and 2) be expressed with O-functions. If we define the objective function with entropy regularization as: (xx := E ->=="> 2'(R(ss (t. de, St+1) 3H (T((+)s;))) PI, where H is the Shannon entropy,
can defined the following surrogate function:
Ju (x²') = Eso-p [V- (sa)] + - ( d"
1
E
n (a
A (s. + 3E,T (1(6((1))))
(8)
m((as)
where n, ' are behavioral, current, and next policies, respectively. Then, we can derive a bound on the difference between the objective and surrogate functions.
Theorem 1. Let us assume that max, H (m(())) < xc for Vn II. The difference between the objective and surrogate functions is bounded by a term consisting of KL divergence
[J(x') Juh (or [ (1 7)3
7
DRV. (T|((()
(228EH + n (el)=())))
(9)
where Eн max (TT ER max A (s, a)), KK. (π||(T)) max (KL((((ss)] 6,0
and the equality holds when
=
We provide the proof in Appendix
A.3 Theorem 1 demonstrates that the surrogate function can
approximate the objective function with a small error if the KL divergence is kept small enough. We then introduce SAC-style surrogate by replacing the advantage in (8) with Q-function as follows:
J
E (Q" d' ,
as) + 3E [H(T((+s))] + +
(10)
where C' is a constant term for n². The policy gradient can be calculated using the reparameterization trick, as done in SAC (Haarnoj al., 2018). We present the training results on continuous RL tasks in Appendix B where the entropy- regularized (8) and SAC-style (1O) versions are compared. Although ( and (10) are mathematically equivalent, it can be observed that the performance of the SAC-style version is superior to the regularized version. We can analyze this with two factors. First, if using (8) in the off-policy setting, the importance ratios have significant variances, making training unstable. Second, the advantage function only gives scalar information about whether the sampled action is proper, whereas the Q-function directly gives the direction in which the action should be updated, so more information can be obtained from (10).
Safe Policy Update. Now, we can apply the same reformulation to the surrogate functions for the safety constraints, which are defined by Kim & Oh (022a)). The cost surrogate functions Fta can be written in SAC-style form as follows:
1
Jer (nr) E.-p [V2 (5)]
+ 1- Yd"
E Q6 a]]
B VV (5) 1-
J (I') ) = Eamp [SS, (s) + 1 -1
1
B
SS (8, 1-
SS. (+))
(11)
o( (a))
Pf (m':a) a) = 10 (T'')
( (J
Remark that Kim & Oh (2022a) have shown that the cost surrogates are bounded in terms of KL divergence between the current and next policy. Thus, we can construct the following practical, safe policy update rule by adding trust region constraint:
new argmax jui (œ') s.t. FK (T';a) S de Vk = L...,,,, K, DKL(T||T') Se,
(12)
and e is a trust region size. As 12) is non- linear, the objective and constraints approximated linearly. while the KL divergence approx- imated quadratically in order to determine the update direction. After the direction is obtained, a backtracking line search is performed. For more details, Appendix A.5
where (T = E Egnd,
Approximations. In the distributional RL setting, the cost value and the cost square value functions can be approximated using the quantile distribution critics follows:
M
@o(s.a) =Pr(Zo 2)dz
Σ
0m (s,
M
m=1
(13)
So(s, |
2 Pr(ze = 2)dz Σ (s,
Finally, the proposed method is summarized in Algorithm
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Safe Distributional Actor-Critic
Data: Policy network Tw, reward and cost critic networks ZR Z6,,0- and replay buffer D. Initialize network parameters 1b, and replay buffer D.
for epochs=1, E do for 1=1, do
Sample de (o(s)s)) [ss) and get Se+1, R(st, at, 81+1), Ck. Ca(st, ae, 8t+1). Store (s, Fe. (as/81 (8)); Ck,e, #b+1) 1 in D.
end
Calculate the TD(A) target distribution (Section 3.1 with D and update the crities to minimize
Calculate the surrogate (T0) and the cost surrogates 11 with D.
Update the policy by solving 112 but if 12 has solution, take recovery step (Section
end
3.3 FEASIBILITY HANDLING FOR MULTIPLE CONSTRAINTS
The proposed method updates policy using 12). but the feasi- ble set of (12) can be empty in the infeasible starting cases. To address the feasibility issue in safe RL with multiple constraints, one of the violated constraints can be selected, and the policy is updated to minimize the constraint until the feasible region is not empty (Xu et al. 202T). which is called naive approach. How- ever, may not be easy quickly reach the feasible condition if only one constraint at each update step is used to update the policy. Therefore, we propose gradient integration method to reflect all the constraints simultancously. The main idea is to get a gradient that reduces the value violated constraints and keeps unviolated constraints. To find such gradient, the following quadratic program (QP) can be formulated by linearly approximating the constraints:
81
Feltp
92,1
28 Se
Figure 2: Gradient Integration.
1 = argmin 79" 5 Hg s.t. gf + c S 0, Ak E (1. {1...., K}.
(14)
where is the Hessian of KI. divergence at the current policy parameters 1h, Ih the gradient the keth cost surrogate,
H- gk, (a(TTi a)) d2 + (), R>o is a slack coefficient. Finally, we update the policy by e² = & + min(1, 22e/(g" """ Figure Figure@lillustatee illlustrates the proposed gradient integration process. Each constraint truncated by c2 to be tangent to the trust region, and the slanted lines show the feasible region of truncated constraints. The solution of (44 is indicated in red, pointing to the nearest point in the intersection of the constraints. If the solution crosses the trust region. parameters are updated by the clipped direction, shown in blue. Then, the policy can reach the feasibility condition within finite time steps.
trust region size, and e
Theorem 2. Assume that the cost surrogates are differentiable and corvex, gradients of the surro- gates are L-Lipschitz continuous, eigenvalues of the Hessian are equal or greater than positive value R E R2 and {1b) F& (TT:a) + <, de. vk) * Q. Then, there exists E. E R20 such that if < 6 S and a policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
We provide the proof and show the existence a solution (1) in Appendix A.4 The provided proof shows that the constant E is proportional ( and inversely proportional to the number of constraints K. This means that the trust region size should be set smaller as K increases and Ç decreases. In conclusion, if the policy update rule (12) is not feasible, finite number of applications of the proposed gradient integration method will make the policy feasible.
4 RELATED WORK
Safe Reinforcement Learning. There are various safe RL methods depending on how to update policies to reflect safety constraints. First, trust region- based methods (Achiam al. 2017 Yang ctt al. 2020: Kim & Oh (022a) find policy update directions by approximating the safe RL problem and update policies through line search. Second, Lagrangian-basee methods (Stooke 2020

---

Under review as a conference paper at ICLR 2023
Yang et al.) 2021 Liu et al. 2020) convert the safe RL problem to dual problem and update the policy and dual variables simultaneously. Last, expectation expectation-maxinizs maximization (EM) based methods (Liu et al. 2022) find non-parametric policy distributions by solving the safe RL problem in E-steps and fit parametric policies to the found non-parametric distributions in M-steps. Also, there are other ways to reflect safety other than policy updates. Qin et al. (202T); Lee et al. (2022) find optimal state or state-action distributions that satisfy constraints, and Bharadhwaj et al. (2021); Thananjeyan et al. (202I) reflect safety during exploration by executing only safe action candidates. In the experiments, only the safe RL methods of the policy update approach are compared with the proposed method.
Distributional TD(A). TD(A) (Precup al., 2000) can be extended to the distributional critic to trade off bias-variance, Gruslys et al. (2018) have proposed method to obtain target distributions by mixing step distributions, but the method is applicable only in discrete action spaces. [Nam al. (22TT) have proposed a method to obtain target distributions using sampling to apply to continuous action spaces, but this is only for on-policy settings. A method proposed by [Tang et al.] (2022) updates the critics using newly defined distributional TD errors rather than target distributions. This method is applicable for off-policy settings but has the disadvantage that memory usage increases linearly with the number of TD error steps. In contrast to these methods, the proposed method is memory-efficient and applicable for continuous action spaces under off- off-policy settings.
Gradient Integration. The proposed feasibility handling method utilizes a gradient integration method, which is widely used in multi-task learning (MTL). The gradient integration method finds single gradient to improve all tasks by using gradients of all tasks. Yu al., (2020) have proposed a inrrection-bbsedd gradient integration method, which is guaranteed to converge Parcto-stationary sets. A method proposed by Liu et al. (202I) can reflect user preference and Navon al. (2022) proposed a gradient-scale invariant method to prevent the training process from being biased by a few tasks. The proposed method can be viewed as a mixture of projection and iiveeiinvariant methods as gradients are clipped and projected onto a trust region.
5 EXPERIMENTS
We evaluate the safety performance of the proposed method and answer whether safe RL actually has the benefit of reducing the effort of reward engineering. For evaluation, agents are trained in the Safety Gym (Ray et al.) 2019) with several tasks and robots. To check the advantage of safe RL, construct locomotion tasks using legged robots with different models and different numbers of legs.
5.1 SAFETY GYM
Tasks. We employ two robots, point and car, to perform goal and button tasks in the Safety Gym. The goal task is to control robot toward a randomly spawned goal without passing through hazard regions. The button task is to click a randomly designated button using a robot, where not only hazard regions but also dynamic obstacles exist. Agents get cost when touching undesignated buttons and obstacles or entering hazard regions. There is only one constraint for all tasks, and it is defined using 3 with the sum of costs. Constraint violations (CVs) are counted when a robot contacts obstacles, unassigned buttons, or passes through hazard regions.
Baselines. Safe RL methods based on various types of policy updates are used baselines. For the trust based method, we use constrained policy optimization (CPO) (Achiam et al. 2017) and off-policy trust-region CVaR (OfTRRC) (Kim & Ohl 2022a), which extend the CPO to an off-policy and mean-std constrained version. For the Lagrangian-
method, worst-case soft actor-critic (WCSAC) (Yang et al.) 202T used, and constrained variational policy optimization (CVPO) (Liu) et al. 2022) based on the EM method is used. Specifically, WCSAC, OffTRC, and the proposed method, SDAC, use the mean-std constraints, so we experiment with those for a = 0.25, 0.5, and 1.0 (when 1.0, the constraint is identical to the mean constraint).
The graph of the final score and the total number of CVs are shown in Figure 3 and the training curves are provided in Appendix D. If points are located in the upper left corner of the graph, the result can be interpreted as excellent since the score is high and the number of CVs is low. The frontiers of SDAC, indicated by the blue dashed lines in Figure are located in the upper left corners for methods. In particular, SDAC with a 0.25 shows comparable scores despite recording the lowest
Results.
all tasks. Hence, SDAC shows outstanding safety performance compared to other
7

---

Under review as a conference paper at ICLR 2023
Point
SDAC_0.25
20
SDMC_0.5
Hinll Score
15
Finall Score
SDAC_1.0
10
ORTRCC_0.25
OITRCC_0.5
OffTRC_1.0
WCSAC_0.25
WCSAC_0.5
20000
Total number CVs
40000
60000
80000
20000
40000 60000 80000 100000120000
Total number of CVs
WCSAC_1.0
Point
CPO
Fiall score
Final score
20
50000
100000
150000 Total number CVs
200000
250000
50000
100000 150000 20000 250000 300000 350000 Total number of CVs
Figure 3: Graphs of final scores and the total number of CVs for the Safety Gym tasks. The number after the algorithm name in the legend indicates used for the constraint. The center and boundary of ellipses are drawn using the mean and covariance of the five runs for each method. Dashed lines connect results of the same method with different
number of CVs in all tasks. Although the frontier overlaps with WCSAC in the car goal and point button tasks, WCSAC shows high fluctuation of scores depending the value of a. In addition, can be seen that the proposed method enhances the efficiency and the training stability of the trust region method since SDAC shows high performance and small covariance compared to the other trust region- based methods, OffTRC and CPO.
5.2 LOCOMOTION TASKS
Tasks. The locomotion tasks are to train robots follow ayy-directional linear and z-directional angular velocity commands. Mini-Cheetah from MIT (Katz et al. 2019) and Laikago from Unitree [Wang] 2018) are used for quadrupedal robots, and Cassie from Agility Robotics (Xie al. 2018) is used for bipedal robot. In order to successfully perform the locomotion tasks, robots should keep balancing, standing, and stamping their feet so that they can move in any direction. Therefore, we define three constraints. The first constraint for balancing is to keep the body angle from deviating from zero, and the second for standing is to keep the height of the CoM above a threshold. The final constraint to match the current foot contact state with predefined foot contact timing. Especially, the contact timing is defined as stepping off the left and right feet symmetrically. The reward is defined as the negative (2-norm of the difference between the command and the current velocity. For more details, see Appendix C
Baselines. Through these tasks, we check the advantage of safe RL over traditional RL. Proximal policy optimization (PPO) (Schulman et al.] (017). based on the trust region method, and truncated quantile critic (TQC) (Kuznetsov et al. 2020), based on the SAC, are used as traditional RL. base- lines. To apply the same experiment to traditional RI. it is necessary to design a reward reflecting safety, We construct the reward through a weighted sum R = (R -; (;C.)/(1+))) t= w.). where R and C(1,2,3) are used to train safe RL methods and are defined in Appendix C and R is called the true reward. The optimal weights are searched by a Bayesian optimization too which optimizes the true reward of PPO for the Mini-Cheetah task. The same weights are used for all robots and baselines to verify if reward engineering is required individually for each robot.
Results. Figure 4 shows the true reward sum graphs according the r-directional velocity com- mand. The overall training curves are presented in Appendix D.2) and the demonstration videos are attached to the supplementary. The figure shows that SDAC performs the locomotion tasks success- fully, observing that the reward sums of all tasks are almost zero. PPO shows comparable results in the Mini-Cheetah and Laikago since the reward the traditional RI. baselines is optimized for the
task of PPO. However, the reward sum is significantly reduced in the Cassic task, where the kinematic model largely differs from the other robots. TQC shows the lowest reward sums
We use Sweeps from Weights & Biases Biewald 2020
8

---

Under review as a conference paper at ICLR 2023
MnilChhetthh
Laikago
Cassle
SUN
HD
-20
190
-200
-00
-030
-200
-0.5
50
0.3 35
2.0
-1.0
-0.5
0.7
2.5
15
2.0
-2.0
-0.3
0.0
0.
Figure 4: True reward sum graphs according to the x-directional command. The true reward is defined in Appendix C The solid line and shaded area represent average and one fifth of std value, respectively. The graphs obtained by running ten episodes per seed for each command.
Score
Cost Rate
Total CVs
25
And (10)
20-
10000
3-11E
0.023
0.033
Steps
Slepo
Ice
Steps
(a) Training curves the point goal task according the trace-decay X.
True Reward Sum
Cost Rate 1
Cost Rate
Cost Rate 3
0.75
1000
-200
Proposed
0.25
0.2
Nolve
-300
0.00
0.00
0.0
Stepss
166
Sceps
1e6
Steps
leG
Steps
1c6
(b) Training curves the naive and proposed methods for Cassie task.
Figure 5: Ablation results. The cost rates show the cost sums divided by the episode length. The shaded area represents the standard deviation. The black lines indicate the limit values, and the dotted lines in (b) represent the limit values + 0.025.
despite the oftte-o-thhe-e art algorithm in other RL benchmarks (Kuznetsov et al. 2020 From these results, can be observed that reward engineering is required according to algorithms and robots.
5.3 ABLATION STUDY
We conduct ablation studies to show whether the proposed target distribution lowers the estimation bias and whether the proposed gradient integration quickly converges to the feasibility condition. In Figure 5a the number of CVs is reduced as À increases, which means that the bias of constraint estimation decreases. However, the score also decreases due large variance, showing that À can adjust the bias-variance tradeoff. In Figure the proposed gradient integration method is compared with naive approach, which minimizes the constraints in order from the first to the third constraint, described in Section 3.3 The proposed method reaches the feasibility condition faster than the naive approach and shows stable training curves because it reflects all constraints concurrently.
CONCLUSION
We have presented the trust region- region-based safe distributional RL method, called SDAC. To maximize the merit of the trust region method that can consistently satisfy constraints, we increase the per- formance by using the Q-function instead of the advantage function in the policy update. We have also proposed the memory-efficient, practical method for finding low-biased target distributions in off-policy settings to estimate constraints. Finally, we proposed the handling method for multiple constraint settings to solve the feasibility issue caused when using the trust region method. From extensive the experiments, proved performance with minimal constraint violations and successfully performed the locomotion tasks without reward engineering.
have demonstrated that SDAC with med constraints achieved im-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of International Conference on Machine Learning, pp. 22-31, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of International Conference Machine Learning, pp. 449-458, 2017.
Mare G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http:// /www di strsstiiioot rittrriuuiiio bonaioonllllo - lrll
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety critics for exploration. In Proceedings of International Confer- ence on Learning Representations, 2021.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: / /www.
wandb. com/ Software available from wandb.com.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of International conference on machine learning, pp. 1096-1105, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018b.
Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Mare Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learn- ing. In Proceedings of International Conference on Learning Representations, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of Inter- national Conference Machine Learning, pp. 1861-1870, 2018
Benjamin Katz, Jared Di Carlo, and Sangbae Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. In Proceedings of International Conference on Robotics and Automation, pp. 6295-6301, 2019.
Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letlers, 7(3):6444777 7651, 2022a.
Dohyeong Kim and Songhwai Oh. TRC: Trust region conditional value at risk for safe reinforcement learning. IEEE Robotics and Automation Letters, 7(2):2621-2222 2022b.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of International Conference on Machine Learning, pp. 5556-5566, 2020.
Jongmin Lee, Cosmin Paduraru, Daniel Mankowitz, Nicolas Heess, Doina Precup, Kee-Bung Kim, and Arthur Guez. COptDDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In Proceedings of International Conference on Learning Rep- resentations, 2022.
Joonho Lee, Jemin Hwangbo. Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47):eabc5986, 2020.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, pp. 18878-18890, 2021.
10

---

Under review as a conference paper at ICLR 2023
Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: Interior interior-point policy optimization under constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):49400 4947, 2020.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Con- strained variational policy optimization for safe reinforcement learning. In Proceedings of Inter- national Conference Machine Learning, pp. 13644-13668, 2022.
Wenjia Meng, Qian Zheng, Yue Shi, and Gang Pan. An off-policy trust region policy optimization method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transac- fions on Neural Networks and Learning Systems, (3(5))2223-2235, 2022.
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: Reusable neural controllers for vision-guide guided whole-body tasks. ACM Transactions on Graphics, 39(4), 2020.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hut- ter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eabk2822, 2022.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-PCL: An off off-policy trust region method for continuous control. In Proceedings of International Conference on Learn- ing Representations, 2018
Daniel W Nam, Younghoon Kim, and Chan Y Park. GMAC: distributional perspective on actor- critic framework. In Proceedings of International Conference on Machine Learning, pp. 7927- 7936, 2021.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint 2rXiv:2202.01017, 2022.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: Adversarial motion priors for stylized physics-basee based character control. ACM Transactions Graphics, 40 (4), 2021.
Xue Bin Peng. Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics, 41)4), 2022.
Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibilitt traces for off-policy policy evaluation. In Proceedings of International Conference on Machine Learning, pp. 759-766, 2000.
Zengyi Qin, Yuxiao Chen, and Chuchu Fan. Density constrained reinforcement learning. In Pro- ceedings of International Conference on Machine Learning, pp. 8682-8692, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah
Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021. URL httt:://jmm=::: http::rorgpppppppp.. 20 1364. html
htttp:///mll..rrr/pppprrs//////
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Mark Rowland, Mare Bellemare, Will Dabney, Remi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. In Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 29-37, 2018.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of Conference on Robot Learning, pp. 91-100, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of International Conference on Machine Learning, pp. 1889- 1897, 2015.
11

---

Under review as a conference paper at ICLR 2023
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of International Conference on Machine Learning, pp- 9133-9143, 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9 14,9944,11 1988.
Yunhao Tang, Mark Rowland, Rémi Munos, Bernardo Ávila Pires, Will Dabney, and Marc Belle- mare. The nature of temporal difference errors in multi-step distributional reinforcement learning. arXiv preprint arXiv:2207.0755555 2022.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 49:4491--4222, 2021.
Emanuel Todorov, Tom Frez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In Proceedings of International Conference Intelligent Robots and Systems, pp. 5026-5033. 2012.
Xingxing Wang. UnirreeLLaikago ht 2hp?claassic<6$1d--333 2018
Ziyu Wang, Victor Bapst. Nicolas Heess, Volodymyr Mnih, Remi Munos. Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In Proceedings of International Conference on Learning Representations, 2017.
Zhaoming Xie, Glen Berseth, Patrick Clary, Jonathan Hurst, and Michiel van de Panne. Feedback control for cassie with deep reinforcement learning. In Proceedings of International Conference on Intelligent Robots and Systems, pp. 1241-1246, 2018
Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of International Conference on Machine Learning. pp. 1 1180011491, 2021.
Qisong Yang, Thiago D. Simão, Simon H Tindemans, and Matthijs T. J. Spaan. WCSAC: Worst- case soft actor critic for safety-constrained reinforcement learning. Proceedings of the AAAI Conference on Arlificial Intelligence, 35(12):10639-1006 2021.
Tsung- T'sungYYen Yang. Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection- based constrained policy optimization. In Proceedings of International Conference on Learning Repre- sentations, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pp. 2020.
12

---

Under review as a conference paper at ICLR 2023
A ALGORITHM DETAILS
A.1 CONVERGENCE ANALYSIS
In this section, show that the proposed TD(A) target distribution converges the 2" First, express the target distribution using a distributional operator and show that the operator is contrac- tive. Finally, we show that 2 is the unique fixed point.
Before starting the proof, we introduce useful notions, distance metrics, and operators. As the return zd (s, a) is a random variable, define the distribution of Z (s,,,a) a) v² v=(ss, a). Let y/ be the distribution of random variable X. Then, we can express the distribution of affine transformation of random variable, a. + b, using the pushforward operator. which defined by RRolaad et al. 20188) as (Ja.)) (17). To measure distance between two distributions, Bellemare et al. (2023) has defined the distance lp as follows:
4phn.72) 72) = ( (F) Frz (2((2))) (x) dr
1/p
(15)
where (x) the cumulative distribution function. This distance is 1/p-homogeneous, regular, and p-convex (see Section 4 of Bellemare al. (2023) for more details). For functions that map state- action pairs to distributions, a distance can be defined as (Bellemare et al. 2023): 1,(11,2) := s))(e.))ESxall (s, a), (2 a)). Then, the proposed TD(A) target distribution be expressed operator below.
T p(s, a) = N
(16)
x E,
E Ber (1ttt) El (tr)) w (u(ssan, ( 88 8, 8, = a
where 11 a) p(als) Then, the operator Tx"" has a contraction property.
s(aas)
Theorem 3. Under the distance I, and the assumption that the state, action, and reward spaces are
finite, Thtt is -
Proof. First, the operator can be rewritten using summation as follows.
The v(s, a)
Σ x Σ Σ
Pr, (so, ao, ro. To. 8
II m((s,,a))
N
1=0
d'EA
x aaaa|sitt
() Ei xed (u((((()())))))
8
A
PT,(T)
(())) (aalsit4))
Σ 1y=s4
N
o'EA
#EES
x
km
(17)
Σ
Eiso (v(ss
N
d'EA''ES
To.
I I
x Ep
(s) 0z) j=1
a(d' (1t))ss
R=0
Xwsrra (Jytt Eimo (
1=0 a'EA Po.
13

---

Under review as a conference paper at ICLR 2023
Since the sum of weights of distributions should be one, we can find the normalization factor .V (1 X) Ei= Eees Euea Erp. A'Wa,a.ro.- Then, the following inequality can derived using the homogeneity, regularity, and convexity of Ip:
1T 11 (s, a), u22,"
00 E Piss
ΣΣΣΣ
X' Eimo 7' (
ΣEΣΣ> 10 WES aEA Yo:
N
s) (UHHEii
5 ΣΣΣΣ
i=U SES aEA Yo:
(ne 2'me ) (12(5,
(18)
2)X Ws a 1o (Urr)))))((((ss a)), (F
N
SES aEA Pa.
X)X'ss 16 (11 (s(s,
N
aEA Po.
20
s)X 1t2 (Tp (v1, u2))
5 Σ
N
SES aEd Ros
S (h (21,
Therefore, 1p (Tx vı: Tht v2) S y //I (V1, 12).
By the Banach's fixed point theorem, the operator has unique fixed distribution. From the definition of 2", the following equality holds (Rowland et al.) 2018): (= a) Ex [(f [(f + (v= a'))) Then, can be shown that nd the fixed distribution by applying the operator Tx to v²:
1- N
Tht o (8,a)
Σ
X
x E
(())
Earna((ens)) Ein wer) (v (s141,
80 =, 00
(19)
A (frttt ( (111) 80 = =s 0,00 =
N
X' a) = (s,aa)
W
1=0
14

---

Under review as a conference paper at ICLR 2023
A.2 PSEUDOCODE TD(A) TARGET DISTRIBUTION
We provide the pseudocode for calculating TD(A) target distribution for the reward critic in Algo- rithm 2
The target distribution for the cost crities can also be obtained by simply replacing the reward part with the cost.
Algorithm 2: TD(A) Target Distribution
Data: Policy network Til critic network Zğ. and trajectory {(*r, "r, (u(re(o)) sc), rt, de, 84+1))
Sample action d T=($T+1) and get Ziot Initialize the total weight Whot = A. for t=7, / do
(1 - d2 (1
Sample an action d/41 ~ (T(()-1) and get 2, = rt + d.) (5 (5+11, Set the current weight = -.
Combine the two targets, (2 w) and (2(tot) Wenn), and sort the combined target according to the positions of atoms.
Build the CDF the combined target by accumulating the weights at each atom.
Project the combined target into quantile distribution with M' atoms, which is 2, ((mro) using the CDF (find the atom positions corresponding to each quantile).
Update 2(tot) T2-1 + (1 d (pro)) and
Whot = A (o(a))))) (1 11 + Wrot).
u(arler)
end
12(00o0) IT=
Return
A.3 PROOF OF THEOREM
Before showing the proof, we present a new function and a lemma. value difference function defined as follows:
8t = E Ei(,,,,,'') + +VV*(s') (*((s x'(ls), a( E (sT(s,
amx!'
Lemma 4. The maximum of 15 (s) - (s)) is equal or less than E V / D2 (T|[T').
Proof. The value difference can be expressed in a vector form,
8F ( (ss))))sss)))(((())====)))))=))) (r((1s) (
Using Hölder's inequality, the following inequality holds:
18 (s) (*)()) < || (116)11 ||AT((, looo
+ 1|0 10 max. 10 (s) - (s) 22 Drv
Using Pinsker's inequality, ||8T 101 SRRV 2 (m 1')).
Theorem 1. Let us assume that max, H (T(-) oo for VT II. The difference between the objective and surrogate functions is bounded by term consisting of KL divergence as:
(())( -"
-
(1 -))3 7)3 2 v DREx((1(17) (22BeH + s (el(T))
(9)
where es = max (T' (-18)),, = max ( a)), DiNAe (~((=')) = max (xs(((((s)] and the equality holds when n' π.
15

---

Under review as a conference paper at ICLR 2023
Proof. The surrogate function can be expressed in vector form as follows:
1 (dd, 8")) B(d", Hr)))
(m] (p, VT)
where HJ (8) (+((s1)))) The objective function of N' can also be expressed in vector form using Lemma from Achiam et al. (017))
J(x')
+ 3H (s) s ~ d" aa ~ P()s,a) 1 E a(s,a,s')
18 (8) B.. (s) + E [s)(
E.
7 aadd=
smp
= (0, V
(dtt 5rh + +/3H"
1-
By Lemma 3 from Achiam et al. inequality is satisfied:
2017), 1|d - -d''ll 111 $ 1- v
2D ( [T|>²)
Then, the following
KL
(11-)) (T') J(x())
= 1(0 - d", 80' B(d= dF', H
2 1(d d", 8 + B/Kdd 8 1111)1
1(d" d", 8" - - + Kddd" d" .H'Il
(: 8 = 0)
< 1|dT d' 111118 8t 100 -11dF= ddh 11a H 1ls0
(: Hölder's inequality)
< RE
87EH V
DREE (RE (T||TT) + 1-
22D
(: Lemma 4
7
7
(REX(T V2,3H + s PRex(ulli (u||z²)
T' n, the KL divergence term becomes zero, so equality holds.
A.4 PROOF OF THEOREM 2
We denote the policy parameter space as V C Rdd, the parameter the tth iteration the V the Hessian matrix as H (1/)) = V DxI. (TT, ( [ and the kth cost surrogate as Fx(()))
FU (T a). As we focus on the fth iteration, the following notations are used for brevity: H (tbe) and Ok = VF& The proposed gradient integration tth iteration defined as the following quadratic program (QP):
1 T 9: argmin 29 Hg s.t. gf + ck S 0 for Vk,
(20)
where c = min( V /2 - Fe de +)). In the remainder of this section, we introduce
the assumptions and new definitions, discuss the existence of solution (20). show the convergence to the fcasibility condition for varying step size cases, and provide the proof of Theorem 2
Assumption. 1) Each F& is differentiable and convex, 2) V is L-Lipschitz continuous, 3) all eigenvalues of the Hessian matrix H (1)) are equal or greater than R E R>0 for Vib E v, and 4) (v) (bbb) Ç d+ for Vk} + 0.
Definition. Using the Cholesky decomposition, the Hessian matrix can be expressed as H B- where B is lower triangular matrix. By introducing new terms, 9k := B '9 and b, := B" gt. the following is satisfied: of H H-1ge g) = 1l 12.. Additionally, we define the in-boundary and out- boundary sets as:
IBx = V)PE(() - du + 22T
OB (IFF(W) - d + ( (2TTF(())--()))
16

---

Under review as a conference paper at ICLR 2023
The minimum of loell in OB, is denoted my, and the maximum of |l in IB is denoted as Mr. Also, mingmn and Ma is denoted as m and M, respectively, and we can say that m is positive.
Lemma 5. For all k, the minimum value of my is positive.
Proof. Assume that there exist & E fl...., } such that my is equal to zero at a policy parameter th E OBg, i.e. [[ = 0. Since F3 convex, tf* is a minimum point of FE. ming (*(()) = Fa (1b"*) dx S. However, (xb"))) 2 da ( as v E OBg, so my is positive due to the contradiction. Hence, the minimum my is also positive.
Lemma 6. A solution of (00 always exists.
Proof. There exists policy parameter d E (b))F((t) + < de for vkj due to the assumptions.
Let g = v th.. Then, the following inequality holds.
9 (v vxr + cı S of (v - v/) + ((b))) + + de S Fa(1) + dk- (: Ph is convex.)
of (v: v/x) cs (v) C dx 0 for Vk.
Since &b 1/. satisfies all constraints of (0)). the feasible set is non-empty and convex. Also, H is positive definite, so the QP has a unique solution.
Lemma 6) shows the existence of solution of (20). Now, we show the convergence of the proposed gradient integration method in the case of varying step sizes.
Lemma 7. If 2 < Ç and a policy is updated by the1 = the + Begt. where 0 < Be < 2emem and Bg < the policy satisfies (i(()) de for vk within a finite time.
L bb
Proof. We reformulate the step size 3 2//emm
2116.11 Sy where 3 1|bb-11
2//2mm and 0 B < 1.
Since the eigenvalues of is equal to or bigger than R and H is symmetric and positive definite, I H-1 is positive semi-definite. Hence, x! -x S |a//1/ 12 is satisfied. Using this fact, the following inequality holds:
Fs (v + Bege) (v.)))) < V.V.(( gt- +
4 111 (:: V., is L-Lipschitz continuous.)
T
Bigg 9:
3.
2
L
Buol 91 +
320
H H-1b,
(:: 9: B B-Th))
2
L
< -B.ck +
1b.
(:: 9t 9t + S 0)
2R
Now, we will show that vi enters IBg in finite time for Vo OBg and that the kth constraint is
satisfied for Vis E BBg. Thus, we divide into two cases, 1) t/ E OBg and 2) the E IBg. For the first
case, C = /2||/@=ll so the following inequality holds:
(ebe + 3.90) F((U)) S 8,
12 0 + 8.100.11)
2R
(21)
$ B, (- + m.
S 3..2eem
<
The value of Fk decreases strictly with each update step according 21.. Hence, vi can reach IB; by repeatedly updating the policy. We now check whether the constraint is satisfied for the second case. For the second case, the following inequality holds by applying e (u(r)) de + Ç:
L
( + Beg.) (vre)) < B.d+ B.FF(())) B.c+ 2R
311..112
+ Pil( + Brgr)) - dy < (1 (1 3)) - dx) + +-3,-5(+ + v 22m3,))
17

---

Under review as a conference paper at ICLR 2023
Since de IB
FR(11))) ds S v2cllgel1 S V26M - 50
Since m & M and 3 1,
- 2em3 <-c+ V2E 0.
Hence, Fk Bege) I de. which means that the kth constraint is satisfied if th E IB₁. As the reaches IB, for Ak within a finite time according to (21). the policy can satisfy all constraints within a finite time.
Lemma 7 shows the convergence to the feasibility condition in the case of varying step sizes. We introduce lemma, which shows ||öel| bounded by vc, and finally show the proof of Theorem which can be considered special case of varying step sizes.
Lemma 8. There exists T E Ryo such that ||ba]| & TVc
Proof. By solving the dual problem of (20). 9t can be expressed as:
K
Ck Σ jæt Xjg H 9k
= Σ A Th s.t. A max
for Ak.
9 H 9k:
The following inequality holds for vk:
2e [i
2ec
A max [1ge|1:0) 0) < max
|1 1| .0)
Using triangular inequality.
||bel| =|B" 9=1l=1111 Σ BT T H H 9ull S Σ He 9k1
k
&
B" 1 9 K Kv2c.
S 2
Hence, for every constant T /2K, the statement holds.
Theorem 2. Assume that the cost surrogates are differentiable and convex, gradients of the surro- gates are L-Lipcchitz continuous, eigenvalues of the Hessian are equal or greater than positive value R E R2 and {1bl Fu Fa(ma:a)) Ç < d2. vk) + 0. Then, there exists F. E R such that if 0 < E S. and a policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
Proof. The proposed step size is B, = min (1, v /2 (,,l|,,,11)) I), and the sufficient conditions that guaran- tee the convergence according to Lemma 7 are followings:
2emR
2eM $ ( and 0 Be s and 3,
11
The second condition is self-evident. To satisfy the third condition, the proposed step size B, should satisfy the followings:
22c
2.2e
2mR
||bell
||bel|
L||b,112
L
If < ((()))((L))) the following inequality holds:
2mR
2mR
/2e- <
||bell < Kv
(:: Lemma g
LK
Hence, if 5 F. } min(
2 2(
2), the sufficient conditions satisfied.
IK
18

---

Under review as a conference paper at ICLR 2023
A.5 POLICY UPDATE RULE
To solve the constrained optimization problem (12), we find a policy update direction by linearly approximating the objective and safety constraints and quadratically approximating the trust region constraint, as done by Achiam et al. (2017). After finding the direction, we update the policy using line search method. Given the current policy parameter t/e v, the approximated problem can be expressed as follows:
1 x Hr S, by x + ek 50
argmax g²zx s.t.
(22)
2
where - VWJt ( H V DKL (Tw, [+ by V F (Tei or) and
Ck = (T(TTaa) de. Since (22) is convex, we can use an existing convex optimization solver.
However, the search space, which the policy parameter space V is excessively large, so we reduce the space by converting (22) to dual problem as follows:
g(A,u) = min, v) min, (g x + ul 5 x² Hx - c) X- (Bx +c]
-1
+2 g" H- A BH- A
+ + DE
(23)
2v
=ip
=
-1
(q + 2r x S7) + VE, 2v
where B (0,,,.,bk) c (ci, Cx)² and À R 2 and E E 0 are Lagrange multipliers. Then, the optimal À and v be obtained by convex optimization solver. After obtaining the optimal values, (A*,"*) ,/-/ argmax calculated by / H=² x* + g). Then, the policy is updated by th+t th + Bx", where 3 is step size, which can calculated by line search method.
the policy update direction x are
19

---

Under review as a conference paper at ICLR 2023
B ABLATION STUDY ON SURROGATE FUNCTIONS
We have extended the off-policy TRPO (Meng et 2022) to the entropy-regularizz version and re- formulated it as the SAC-style version. In this section, we evaluate the original, entropy-regularized, and SAC-style versions in the continuous control tasks of the MuJoCo simulators (Todorov et al,, 2012). We use neural networks with two hidden layers with (512, 512) nodes and ReLU for the activation function. The output of value network is linear, but the input is different; the original and entropy-rege reguuarreeu versions use states, and the SAC-style version uses state- bute-action pairs. The input of policy network is the state, the output is mean and std and actions are squashed into tanh(ju + cσ), (0, 1) as SAC (Haarnoja et al., 2018). The entropy coefficient / in the entropy-regularized and StC-style versions adaptively adjusted to keep the entropy above threshold (set as -d given IR.). The hyperparameters for all versions are summarized in Table
Parameter
Discount factor Y
0.99
Trust region size
0.001
Length of replay buffer
105
Critic learning rate Trace-decay À
0.0003
0.97
Initial entropy coefficient 3
1.0
learning rate
0.01
Table Hyperparameters for all versions.
The training curves are presented in Figure 6 All methods are trained with five different random seeds. Since there is importance ratio and the Q-functions directly provide policy update direc- tion, the SAC-style version outperforms the others.
6000
10000
8000
2000
200r
6000
4000
1000
2000
Entropy
SAi-lyyle
1w5
(a) Ant-v3
(b) HalfChcctah-v3
(c) Hopper-v3
150
6000
4000
4000
Soore
Steps
160
Steps
165
Steps
100
(d) Humanoid-v3
(e) Swimmer-v3
( Walker2d-v3
Figure 6: MuJoCo training curves.
20

---

Under review as a conference paper at ICLR 2023
(a) Point goal.
(b) Car button.
(c) Mini-Chhettah.
(d) Laikago.
(e) Cassie
Figure 7: (a) and (b) are Safety Gym tasks. (c), (d), and (c) are locomotion tasks.
C EXPERIMENTAL SETTINGS
Safety Gym. We use the goal and button tasks with the point and car robots in the Safety Gym environment (Ray et al.] 2019), as shown in Figure 77 and [bb The environmental setting for the goal task is the same in Kim Ohl (2022b). Eight hazard regions and one goal are randomly spawned at the beginning of each episode, and a robot gets reward and cost as follows:
R(s, a, s') = Adgoall + 50.31
(24)
C(s, a, s') = Sigmoid(10 (0.2 dhazard)).
where dgoat is the distance to the goal, and dhazard is the minimum distance to hazard regions. If dgoal is less than or equal to 0.3, a goal is respawned, and the number of constraint violations (CVs) is counted when duazard is less than 0.2. The state consists of relative goal position, goal distance, linear and angular velocities, acceleration, and LiDAR values. The action space is two-dimensional, which consists of zzy-directonaal forces for the point and wheel velocities for the robot.
The environmental settings for the button task are the same as in Liu et al. (2022). There are five hazard regions, four dynamic obstacles, and four buttons, and all components are fixed throughout the training. The initial position of a robot and an activated button are randomly placed at the beginning of each episode. The reward function is the same as in (24). but the cost is different since there is no dense signal for contacts, We define the cost function for the button task as an indicator function that outputs one if the robot makes contact with an obstacle or an inactive button or enters a hazardous region. We add LiDAR values of buttons and obstacles to the state of the goal task, and actions are the same the goal task. The length of the episode is 1000 steps without early termination.
Locomotion Tasks. We use three different legged robots, Mini-Cheetah, Laikago, and Cassie, for the locomotion tasks, as shown in Figure 7c 7 and [7 The tasks aim to control robots to follow a velocity command on flat terrain. A velocity command is given by (oamd, ..ggdd ( ). where gamd ~ (1-(((110) 1.0) for Cassie and 2 2.0) otherwise, egrm = 0, and w ~ (1-0.5, 0.5). To lower the task complexity. we set the y-directional linear velocity to zero but can scale to any non- zero value. As in other locomotion studies (Lee et al. 2020 Miki et al. 2022), central phases are in- troduced to produce periodic motion, which are defined as (i(() = Pi.0 + f-t for Vi E (1...., Beggs) where f is a frequency coefficient and is set to 10, and Φ is an initial phase. Actuators of robots are controlled by PD control towards target positions given by actions. The state consists of velocity command, orientation of the robot frame, linear and angular velocities of the robot, positions and speeds of the actuators, central phases, history of positions and speeds of the actuators (past two steps), and history of actions (past two steps). A foot contact timing & can be defined as follows:
(x) = 1 + 2 sin( Vi E (1...,,legss,
(25)
where a value of means that the ith foot is on the ground; otherwise, the foot is in the air. For the quadrupedal robots, Mini-Cheetah and Laikago, we use the initial phases as dn {0, म, ग, 0}, which generates trot gaits. For the bipedal robot, Cassie, the initial phases are defined as do (0,T), J). which generates walk gaits. Then, the reward and cost functions are defined as follows:
Als, a s = -0.1 u2.. base cmd 112 + ||w buse
w
.md
112 + 10 Hpower):
Meeea
(26)
8')),,,8') Langleaa, C2(s, s')= Innighega. Ca(s,
D (1 & -.)((2 megg)),
21

---

Under review as a conference paper at ICLR 2023
where the power consumption Rpower = =, |,uul, the sum of the torque times the actuator speed, added to the reward regularizaaion term, "xyy is the zy-directional linear velocity of the base frame of robots, wo is the z-directional angular velocity of the base frame, and & E {-1, J is the current feet contact vector. For balancing, the first cost indicates whether the angle between the z-axis vector of the robot base and the world is greater than a threshold (a = 15 for all robots). For standing, the second cost indicates the height of CoM is less than a threshold Mini- Mini-Cheetah, Laikago, and Cassie, respectively), and the last cost is to check that the current feet contact vector & matches the pre-defined timing Ç. The length of the episode is 500 steps. There is no early termination, but if a robot falls to the ground, the state is frozen until the end of the episode.
0.3, 0.35, 0.7 for
Hyperparameter Settings. The structure of neural networks consists of two hidden layers with (512, 512) nodes and ReLU activation for all baselines and the proposed method. The input value networks is state-action pairs, and the output is the positions of atoms. The input of policy networks the state, the output is mean and std a, and actions are squashed into (anh(u+eco We use fixed entropy coefficient B. The trust region size is set 0.001 for all trust region-based methods. The overall hyperparameters for the proposed method can be summarized in Table 2 Since the range of the cost is [0, 1], the maximum discounted cost sum is 1/(1 7). Thus, the limit
~ (00 1).
Parameter
Safety Gym
Locomotion
Discount factor y Trust region size
0.99
0.99
0.001
0.001
Length of replay buffer
10a
10a
Critic learning rate Trace-decay À
0.0003
0.0003
0.97
0.97
Entropy coefficient, ,9
0.0
0.001
The number of critic atoms M
25
25
The number of target atoms M'
50
50
Constraint conservativeness
0.25, 0.5, and 1.0
1.0
Limit value dN
0.025/(1 -1))
[0.025, 0.025, 0.4]/(1 (1 -n)
Slack coefficient&
mingd4 0.025 (1 - -)
Reward weights un;
5.19, 1.19,3.71.0.33 0.344
Table 2: Hyperparameter settings for the Safety Gym and locomotion tasks.
value is set by target cost rate times 1/(1 7). For the locomotion tasks, the third cost in 26 is designed for foot stamping, which is not essential to safety. Hence, we set the limit value to near becomes 0.5). The reward weights are also
the maximum (if robot does not stamp, the cost presented in Table which are optimized using the existing Bayesian optimization tool. In addition, baseline methods use multiple critic networks for the cost function, such as target (Yang al. 2021) or square value networks (Kim & Oh (0022a). To match the number of network parameters, we use two critics as an ensemble, as Kuznetsov et al. (2020).
Tips for Hyperparameter Tuning.
Discount factor 7, Critic learning rate: Since these are commonly used hyperparameters, we do not discuss these.
Trace-decay X, Trust region size e: The ablation studies on these hyperparameters are presented in Appendix D.4 From the results, we recommend setting the trace-decay to 0.95 ~ 0.99 as in other TD(A)-based methods (Precup et al., 2000). Also, the results show that the performance is not sensitive to the trust region size. However, if the trust region size is too large, the approximation eттог increases, so it is better to set it below 0.003.
Entropy coefficient B: This value is fixed in our experiments, but it can be adjusted auto- matically as done in SAC (Haarnoja al.) (2018).
The number of atoms M, M': Although experiments on the number of atoms did not per- formed, performance is expected increase the number of atoms increases, as other distributional RI. methods Dabney et al. (2018a).
Length of replay buffer: The effect of the length of the replay buffer can be confirmed through the experimental results from off policy-based safe RL. method (Kim & Oh
22

---

Under review as a conference paper at ICLR 2023
2022a).
According to that, the length does not impact performance unless it is too short. We recommend setting it to 10 to 100 times the collected trajectory length.
Constraint conservativeness a, limit value dg: If the cost sum follows a Gaussian distri- bution, the mean-std constraint becomes the CV&R constraint. Then, the probability of the worst case can be controlled by adjusting a. For example, we set a 0.125 and d 0.03/(1 - 7), the mean-std constraint enforces the probability that the average cost is less than 0.03 during an episode greater than 95% = ((0(((((a)))/))) (a))/a). Through this meaning, proper and dg be found.
Slack coefficient Ç: As mentioned at the end of Section 3.3 it is recommended to set this coefficient as large as possible. Since dy - should be positive, we recommend setting to ming dę.
Reward weights: These are used when defining the reward function for traditional RL. methods, so these are not hyperparameters of our method.
In conclusion, most hyperparameters and de need to be set based on the meaning described above. Additionally, if the approximation error of critics is significant, the trust region size should be set smaller.
not sensitive, so few need be optimized. It seems that
23

---

Under review as a conference paper at ICLR 2023
D EXPERIMENTAL RESULTS
D.1
SAFETY GYM
Point Goal
Car Goal
Point Button
Car Button
Scale
20
20
36
346
366
Cost
6.30
146
3300000
60000
200002
200000
#9000
10000C
20000
0
Staps
Stops
Staps
5togs
Figure 8: Training curves of mean constrained algorithms for the Safety Gym tasks. The solid line and shaded area represent the average and std values, respectively. The solid black lines in the second row indicate limit values. All methods are trained with five random seeds.
In this section, present the training curves of the Safety Gym tasks separately according to the conservativeness of constraints for better readability. Figure SeuureSshorr shows the training results of the mean constrained and mean-std constrained algorithms with results of the mean-std constrained algorithms with a = 0.25 and 0.5, respectively. can be observed that the score of SDAC has the fastest convergence speed and that the also converge to the limit values quickly. Observing that all the other methods show the highest total CVs in the car button task, this task challenging to meet the constraint. Thus, SDAC also has higher cost rate for the car button than other tasks. In addition, since decreasing makes the constraints more conservative. the cost rates and the number of total CVs of SDAC are reduced in Figures and T0 For 0.25 and 0.5, SDAC shows the highest scores and the lowest number of CVs in all tasks.
T.O. Figures 9 and 10 show the training
Figure g
24

---

Under review as a conference paper at ICLR 2023
Point Goal
Car Goal
Point Button
Car Button
10
30
Scone
20
20
10
25
306
300
Coss Rate
260
20000
20000
120000
00000
10000
50000
50000
00
Staps
Stops
Staps
340
Stops
Figure 9: Training curves of mean-std constrained algorithms with a = 0.5 for the Safety Gym.
Point Goal
Car Goal
Point Button
Car Button
S3AC
OMTRC
WCSAC
Scale
10
0.15
0.15
0.19
0.10
0.10
0.10
Cost
0:0
0.32
000
0.09
0.00
0.00
0.00
340
150000
20009
15000
100600
10000
50000
50000
5000
Staps
Steps
Staps
366
Stops
Figure 10: Training curves of mean-std constrained algorithms with 0.25 for the Safety Gym.
D.2 LOCOMOTION TASKS
We present the training curve for the locomotion tasks in Figure 11 where the true reward is defined
in 26h. The traditional RL baselines are trained with the reward (R - Σ w;C.)/(1 +
E W:), and the weights are described in Section C The training curve contains graphs of the three cost rates and the true reward sum.
25

---

Under review as a conference paper at ICLR 2023
True Resward Sum
Cosl Rate
Casi Rate
Cosl Raie 3
-32
-130
0.2
0.1
-270
02
M/
00
05
1.0
2.0
1,0
00
03
1.0
1.3 Steps
24
3.0
0.5
2.0
15 Steps
255
20
05
1.0
1.0
50
Steps
Steps
(a) Mini-Cheetah.
True Rewerd Sum
Cost Rate
Cost Rate
Cost Rate 3
0.5
-52
014
0.-
035
-220
0.5
044
04
-330
1.0
1.5 Steps
200
3.0
00
1.0
24
00
05
25 Steps
255
0.0
3.0
1.0
1.0
Steps
106
Steps
(b) Laikago.
0.5
-52
014
0.6
0.5
-150
04-
002
03
0.1
-330
106
Steps
(c) Cassie.
Figure 11: Training curves of the locomotion tasks. The black dashed lines show the limit values used for the safe RL methods. The solid line represents the average value, and the shaded area shows one-fifth of the std value. All methods are trained with five different random seeds.
D.3 ABLATION STUDY ON COMPONENTS OF SDAC
We experiment with variations of SDAC to examine the effectiveness of each component. SDAC has two main components, SAC-style surrogate functions and distributional critics. We call SDAC with only distribution critics, SDAC-Dist, and SDAC with only SAC-style surrogates, SDAC-Q. If all components are absent, SDAC is identical to OffTRC (Kim & Oh 2022a). The variants are trained with the point goal task of the Safety Gym, and the training results are shown Figure[12) SDAC-Q lowers the cost rate quickly but shows the lowest score. SDAC-Dist shows scores similar to SDAC, the cost rate converges above the limit value 0.025. In conclusion, SDAC can efficiently satisfy the safety constraints through the SAC-style surrogates and improve performance through the distributional critics.
Score
Cost Rate
Total CVs
0.130
40000
20
0.125
10000
30
0.250
SONC
10000
SPAC-9
0.225
0.300
Steps
165
Steps
1nc
Stops
Figure 12: Training curves of variants of SDAC for the point goal task.
26

---

Under review as a conference paper at ICLR 2023
D.4 ABLATION STUDY ON HYPERPARAMETERS
To check the effects of the hyperparameters, we conduct ablation studies on the trust region size and entropy coefficient. The results are shown in Figure 13 From the entropy coefficient results, it can be seen that excessive exploration causes the constraint to be violated. Thus, the entropy coefficient should be adjusted cautiously, shows that the estimation error of the surrogates is proportional to the trust region size, it can be observed from Figure 13b that the number of CVs increases with the size of the trust region due to the estimation error.
it can be better to set the coefficient to zero. Since Theorem
Cost
0.130
p-0.0
20000
0.125
p-0.001
20
25000
0.100
0.075
15000
0.250
10000
3000
Steps
165
Steps
166
Steps
365
(a) Entropy coefficient 8.
Score
Cost Rate
Total CVs
25
5-000033
40000
0.125
t-0.001
20
0.100
*=n n03
30000
15
30
0.015
20000
0.290
10000
0.325
0-
0.200
Steps
165
Steps
166
Steps
(b) Trust region size
Figure 13: Training curves of SDAC with different hyperparameters for the point goal task.
E COMPUTATIONAL COST ANALYSIS
In this section, we analyze the computational cost of the gradient integration method. The proposed gradient integration method has three subparts. First, it is required to calculate policy gradients of each cost surrogate, Gk. and H 11-1g -9 Ib for Ak E 21. K). where H is the Hessian matrix of the KL divergence. -- 9k can be computed using the conjugate gradient method, which requires only a constant number of back-propagation on the cost surrogate, so the computational cost can be expressed as K O(BackProp)
Second, the quadratic problem in Section 3.3 is transformed to a dual problem, where the trans- formation process requires inner products between Zb and H- gm for vk, m 21,2, K}. The computational cost can be expressed as (TTnerrrrP)))))
Finally, the transformed quadratic problem is solved in the dual space I using quadratic programming solver. Since K is usually much smaller than the number of policy parameters, the computational cost almost negligible compared the others. Then, the cost of the gradient in- tegration is K O(BackProp + K2 O(InnerProd) + C. Since the back-propagation and the inner products is proportional the number of policy parameters |0b]. the computational cost can be simplified as O(K². [V])).
27


Reviews:
[{"review_id": "n0p4b46kHD", "reviewer": "Reviewer_FqLh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for safe RL or more general constrained RL. Constraints are modelled by the mean-std method. As RL algorithm the authors use TRPO and use a multi-step distributional critic.\nApart from combining these methods, the paper seems to have three independent contributions: \n1. The authors extend the distributional RL approach based on quantile regression and Wasserstein distance to the multistep setting.\n2. Then replace the advantage function with the q-function in estimating the trust region.\n3. Finally, they propose a gradient-based method to find starting policies in the trust region.\nExperiments are conducted on the safety Gym environment and compare to other safe RL approaches w.r.t. constraint violations and performance.", "summary_of_the_review": "The paper proposes a well-performing safe RL-method which combines several recent approaches such as trust-region-based safe RL and distributional critics. Experimental results are also quite convincing. \nOn the other hand, some of the contributions are not well-motivated. For example, the authors claim that it is a good idea to use q-values instead of the advantage function. However,  as the advantage function is the q-function minus the value function and it was introduced to lower the variance, it is kind of unclear why this is done. Later, they add entropy regularization which makes more sense, but it is not very exciting as it is a well-known technique. \nIn general, many improvements seem to be a relatively straightforward combination of existing techniques and a better explanation on why the proposed extension yield a major technical contribution would be desireable.", "strength_and_weaknesses": "strong points:\n* The paper combines and extends several SOTA techniques.\n* Results indicate a superior trade-off between constraint violations and performance\n\nweak points:\n* The paper's contributions are not easily distinguishable from the combined basis methods.\n* it is unclear how the contributions relate to each other. Some of them seem disconnected and not really specific to safe RL. In particular, using a distributional critic seems valid for any AC method. \n* Some of the methods are not well-motivated. For example, I did not get why replacing the advantage function with the q-function is beneficial. \n\nAfter reading the author's comments and the revised paper, explaining why these methods address open problems of constrained RL with trust regions is more precise. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g6mVeZcPAS", "reviewer": "Reviewer_QThN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose SDAC, a safe RL method based on trust regions that uses TD($\\lambda$) for the distributional critic and SAC-style updates.\nThey provide an elegant and memory-efficient way to approximate TD($\\lambda$).\nThey propose to perform gradient integration in order to avoid getting stuck due to unmet constraints.\nThe proposed method was tested on Safety Gym with very nice results.\n", "summary_of_the_review": "This is a nice paper, featuring good writing and a genuine effort at making technical points digest to the reader.\n\nThe experiments are varied and convey a lot of information. They do a good job at validating the claims of the paper.\n\nThe proposed algorithm seems super heavy (k-distributional critics + backtracking + quadratic programming for gradient integration) but the computational cost is not quantified in this work, which I think is regrettable. \n\nIn the current state, it is somehow hard to see how the community will use this work if no open-source implementation and/or not more comments on computational cost and sensitivity of the hyperparameters.\n\nFor all these reasons, though the proposed method is inventive and sound, I only borderline recommend for acceptance. There are low hanging fruits that the authors could choose to address to increase the potential impact of the work.", "strength_and_weaknesses": "### Strengths\n\nThe paper involves technical elements that the authors explain in a clear way.\nFor instance, the introduction is super clear and provides a nice overview of the existing methods. In the background, the notations are clear and precise. The figures do a great job at facilitating the comprehension. The related work is exhaustive and well-organized.\n\nThe results on Safety Gym are quite nice, showing increasing or equal scores while reducing the number of violated constraints, which illustrates the empirical pertinence of the proposed method.\n\n### Weaknesses\n\nThe paper is quite dense and I think this aspect could be improved. Some proofs might appear in the appendix instead.\n\nOn the title: use the term low-bias instead of low-biased?\n\nOn the RL method:\n* It should be made clearer that the recursive formula (eq.7) is particularly useful to implement the method\n* Fig.1 is nice, but a clarification as to why a projection is needed, i.e. different supports for distinct time-steps, would be welcome in the text \n* why the off-policy version of TRPO? This choice should be motivated.\n* Why TRPO over PPO? PPO enforces a trust region by modifying the policy gradient, while TRPO requires backtracking. If I understand correctly this is because backtracking includes the linearized constraints as well? This choice could be better motivated in the text.\n* What is the cost of gradient integration in practice?\n\nOn experiments:\n* There are many hyperparameters for the resulting method, how hard is it to hyperoptimize? I think a discussion on that aspect would benefit the work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SjOQOvAxO_", "reviewer": "Reviewer_SPhH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper concerns safe (deep) reinforcement learning. In particular, the authors present an approach that take the constrained MDP setting, where on top of a reward function, a cost function is given that conveniently reflects safety concerns. The learning goal is then to maximize the reward, while the cost needs to stay below a certain thresholed. In this work, the authors employ actually a set of cost functions. The actual approach presented here entails a distributional RL method that uses trust region optimization, like in other prior works. Here, however, the multipe constraints cause problems, and the authors use (1) a target distribution to estimate constraint biases, and (2) something called the ‘parameterization trick’ to develop a dedicated actor-critic techniqe. The approaches are evaluated by means of several environments from the Safety Gym, and further robotics experiments.\n", "summary_of_the_review": "Good approach, clarity can be improved. ", "strength_and_weaknesses": "++ Important problem fitting for ICLR\n++ Novel approach with multiple cost functions\n++ Interesting actor-critic technique\n\n— The paper is at parts very hard to follow\n— Unclear utility of multiple cost functions in the Safety Gym environments\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CWHNyShBdO", "reviewer": "Reviewer_97Q2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author develop an efficient trust region-based safe RL algorithm with multiple constraints.\n1. introduce a TD(λ) target distribution combining multiple-step distributions to estimate constraints with low biases\n2. present a memory-efficient method to approximate the TD(λ) target distribution using quantile distributions\n2. to increase the efficiency of the trust region method, we propose novel surrogate functions which use the reparameterization trick with Q-functions", "summary_of_the_review": "I have many doubts about the experimental results and opinions in this article, and I hope the authors can explain them in detail.", "strength_and_weaknesses": "Strength\n1. introduce a TD(λ) target distribution combining multiple-step distributions to estimate constraints with low biases\n2. present a memory-efficient method to approximate the TD(λ) target distribution using quantile distributions\n3. propose novel surrogate functions which use the reparameterization trick with Q-functions to increase the efficiency of the trust region method,\n\nWeaknesses\n\n1. I don't understand what figure 1 is trying to say, could the author be more clear?\n2. The authors believe that Figure 3 can be clearly represented, \"the result can be interpreted as excellent\", but from this figure, I have no way to see the cost limit, as well as fine metrics, intelligent roughly, the method of this paper is roughly put in the upper left, but Point Goal and Car Goal seem to overlap in the upper left.\n3. I carefully read through the experimental results of the article, and in Appendix D.1, Figure 8, in Point Goal and Point Button, WCSAC is significantly better than the SDAC proposed in this paper in terms of score, cost rate, and Total CVs, and this seems to be consistent with the Safe RL perspective. From the Safe RL perspective, this seems to be in great conflict with what is discussed in the abstract, \"From extensive experiments, the proposed method shows minimal constraint violations while achieving high returns compared to existing safe RL methods.\" \n    I suggest that the authors should rework Figure 3 or put a more refined result, Figure 8, in the text, because the ambiguous and unrefined Figure 3 leads to a misunderstanding of the performance of the algorithm.\n4. In Figure 8, why is there if a big difference between cvpo in Button's task and Goal's task? cvpo's reward in Goal's task is even negative, while it does have a considerable reward in Button's task.\n5. As far as I know, \\alpha is the discount factor corresponding to cost, so why the comparison in Figure 9 and Figure 10 lack CPO and CVPO.\nBased on the above doubts, I have doubts about the contradictory claims in this article and the results of the experiments, and the authors don't seem to disclose their code in the appendix material. Also based on the list of hyperparameters provided by the authors (and no other baseline details of the experimental parameters are provided), I think the reproducibility of the experimental results of this work is very low.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject