Off-Policy Primal-Dual Safe Reinforcement Learning
Authors: Zifan Wu
Bo Tang
Qian Lin
Chao Yu
Shangqin Mao
Qianlong Xie
Xingxing Wang
Dong Wang
Venue: ICLR 2024 poster (2024)
Keywords: Safe Reinforcement Learning
Primary area: reinforcement learning

Abstract: Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

Published as a conference paper at ICLR 2024
OFF-POLICY PRIMAL-DUAL SAFE REINFORCEMENT LEARNING
Zifan Wu', Bo T Lin'; Chao Yu'! Shangqin Mao', Qianlong Xie', Xingxing Wang', Dong Wang? 'uun Yat-sen University, Guangzhou, China
(wuzf5, 1i in @mail2..... sysu.. edu. en, yuchao3@maill wys.. Pnnstitute for Advanced Algorithms Research, Shanghai, China tangbê aarr ac. cn 'Meituan, Beijing. China (maoshanggin, xieqianlong,
(awp://g000oaanggggggg00000 uanegdoonno/mmmmituu...ooooo com
ABSTRACT
Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such train- ing paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update pro- cesses. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfyingg area by considering the uncertainty in cost esti- mation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help climinate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results benchmark tasks show that our method not only achieves an asymptotic performance comparable state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://gtthub.com www.:/uaacooo/aaaaaaaaaaa/aaaaaa Wu/CAL.
INTRODUCTION
Reinforcement learning (RL) has achieved tremendous successes in various domains (Silver et al., 2016: Andrychowicz et al., 2020; Jumper al., 2021; Afsar et al., 2022). However, in many real- world tasks, interacting with the environment expensive or even dangerous (Amodei et al., 2016). By encoding safety concerns into RL via the form of constraints, safe RL (Gu et al., 2022) studies the problem of optimizing the traditional reward objective while satisfying certain constraints. In recent years, safe RI. has received increasing attention due to the growing demand for the real-world deployment of RL methods (Xu et al., 2022b).
As one of the most important safe RL methods, primal-dual methods convert the original constrained optimization problem to its dual form by introducing set of Lagrange multipliers (Chow et al., 2017: Tessler al., 2018; Ray et al., 2019: Ding et al., 2020; Yang al., 2022). Generally, primal- dual methods alternate between the primal update for policy improvement and the dual update for Lagrange multiplier adjustment. While enjoying theoretical guarantee in converging to constraint- satisfying policies (Tessler et al., 2018; Paternain et al., 2019), primal-dual methods often suffer from severe performance instability during the practical training process (Stooke al., 2020), which can be largely attributed to the high reliance on accurate estimation of the cumulative cost (Xu al., 2021; Liu et al., 2022). An inaccurate cost estimation, even with a small error, can give rise to wrong Lagrange multiplier that hinders the subsequent policy optimization. For example, when the
"Equal contribution. Work done when Bo Tang was at Meituan.
1 "orresponning author

---

Published as a conference paper at ICLR 2024
true cost value is under the constraint threshold but its estimate is over the threshold (and vice- versa), the Lagrange multiplier will be updated towards the complete opposite of the true direction and in turn becomes misleading penalty weight in the primal policy objective. Therefore, compared to the reward estimation where an inaccurate value estimate may still be useful for policy improvement long the relative reward priority among actions is preserved, cost estimation is more subtle since a minor eттоr can significantly affect the overall primal-dual iteration (Lee et al., 2022b; Liu al., 2023). This issue further aggravated when using off-policy methods due the inherent estimation error caused by the distribution shift (Levine et al., 2020). As a result, the majority of existing primal-dual methods are on on-policy by design and thus require large amount of samples before convergence, which is problematic in safety-critical applications where interacting with the environment could be potentially risky.
In this paper, we propose an off-policy primal- dual-based safe RL. method which consists of two main ingredients. The first ingredient is conservative policy optimization, which utilizes an upper confidence bound of the true cost value in the Lagrangian, in order to address the cost underestimation issue existing in naive off- policy primal-dual methods. The underestima- tion of the cost value causes the policy to vi- olate the constraint, leading to an aggressive feasibility boundary (C*) that exceeds the fea- sible policy space ()) illustrated in Fig- ure I(a). In contrast, by considering estimation uncertainty, the conservative policy optimiza- tion encourages cost overestimation, which ul- timately results in conservative boundary (*²) that lies inside the constraint sattiittt--tssyiing policy space. While such conservatism improves con- straint satisfaction, may also hinder reward maximization since it shrinks the policy search space. To tackle this problem, introduce second ingredient, namely local policy convexification, which modifies the original objective using the augmented Lagrangian method (Luenberger et al., 1984), in order convexify the neighborhood area (C) of locally optimal policy ()). Through rectifying the policy gradient (-),, the local policy convexification can stabilize policy learning in the local convexified area and thus gradually reduce the cost estimation uncertainty in this area. Consequently, the conservative search space will expand gradually towards the optimal policy (o), as depicted by the process from Figure (a) to Figure 1(b). The contributions of this work are as follows:
(a)
(b)
Conservative Policy
Aggressive Boundury
Optimal Policy
Conservative Boundary
Gradient Rectification
True Boundary
Local Convexified Area
Feasible Policy Space
Visualization of a part of the policy space, where deeper color indicates higher reward in the feasible (constraint-satiifying area.
Figure 1:
We analyze the problem existing in naïve off-policy primal-dual methods and address it by introducing two critical algorithmic ingredients. Detailed theoretical interpretations and empirical verifications are provided to demonstrate the respective effect of each ingredient and more importantly, their mutual impact on the overall learning process.
We specify a practical algorithm by incorporating the two ingredients into an off-policy backbone. Empirical results various continuous control tasks show that method achieves superior sample efficiency compared to the basclines, i.c., achieving the same reward with fewer samples and constraint violations during training.
We evaluate our method on a real-world auto-bidding task under the semi-batch training paradigm (Matsushima et al., 2021), where the behavior policy is not allowed to update within each long-term data collecting process. Results verify the effectiveness of our method in such scenarios by conservatively approaching the optimal policy.
2 BACKGROUND
This paper studies safe RL problems under the formulation of the Constrained Markov Decision Process (CMDPs) (Altman, 1999), which defined by a tuple (S, P. r, Y, d.d,, where S is the state space, A the action space, P A x [0, 1] the transition kemel, S A R
2

---

Published as a conference paper at ICLR 2024
the reward function, S x A - R+ the cost function, y the discount factor, and d the constraint threshold. Without loss of generality, this paper considers CMDPs with only a single scalar cost to simplify the notation. At each time step f, the agent selects an action Or A based on the current state 8t E S and the policy rT-1sa). The environment then returns reward rt = r[s,, at), ), a cost ez = c(st, 04), and the next state St+1 8 sampled from the transition probability P(- (t). We denote the reward value function and the cost value function respectively as Q7 (s, a) = EE [r 7're] and Q (s, a) = E 1_==0 7'ce), where = {su = 8, ag = a, $1..... is a trajectory starting from and The goal of safe RL methods is to solve the following constrained optimization problem:
max E Exppp. [Q
s.t. (a--p. [Q (s) - d,
(I)
where Ps denotes the stationary state distribution induced by x. The basic primal-dual approach converts Eq. (I) to its dual form, i.e..
min max (emp),anatll)) (0T X (O) d)], X
and solves by alternating between optimizing the policy and updating the Lagrange multiplier. For a large state space, the value functions can be estimated with function approximators, and we denote @F and Q: the estimation of the reward and the cost value, respectively.
METHODOLOGY
This section introduces two key algorithmic ingredients for improving off-policy primal-dual safe RI methods. In Section 3.1, we reveal the cost underestimation issue existing in naive off-policy primal-dual methods and address it by applying the first ingredient, i.e., conservative policy opti- mization. To mitigate the possible reward suboptimality induced by such conservatism, we then introduce the second ingredient, i.e., local policy convexification in Section 3.2. The interpretation and empirical verification for the joint effect of these two ingredients are provided in Section 3.3.
3.1 CONSERVATIVE POLICY OPTIMIZATION
In the primal-dual objective (2), when the Lagrange multiplier is greater than 0, the policy will be updated to minimize the estimation of the cost value. Assuming that the approximation Q= a) = (s, has zero-mean noise e, then the zero-mean property cannot be preserved in general under the minimization operation, i.e., E, (minn,Q(ss T())] < min, Qo(s, (s)), Vs (Thrun & Schwartz, 1993), leading to the underestimation of the cost value. Such underestimation bias may be further accumulated to a large bias in temporal difference learning where the underestimated cost value becomes learning target for estimates of other state-action pairs. This issue becomes even more severe when using poflcy methods that inherently suffer from value approximation error (i.e., larger induced by the distribution shift issue (Levine al., 2020).
We empirically validate the above analysis by showing the results of a naive off-policy baseline: SAC-Lag, which is the Lagrangian version of SAC (Haarnoja et al., 2018). Figure 2 provides the reward (left column) and the cost evaluation (middle column), as well as the difference between the estimations of cost values and their oracle values during training (right column). The oracle value is approximated by testing the current policy in the environment for 100 episodes and averaging the induced discounted cost returns of these episodes. Clearly, we can see that SAC-Lag suffers from severe underestimation of the cost value and also poor constraint satisfaction.
To deal with such cost underestimation, we propose to replace the value estimate Q in the primal- dual objective by an upper confidence bound (UCB) of the true cost value. Specifically, we maintain E bootstrapped Qe networks, i.e., { fm1 which share the same architecture and only differ by the initial weights. All the networks are trained on the same dataset. During cost estimation, each Q estimates the cumulative cost in parallel and independently. The UCB value is obtained by computing "mean plus weighted standard deviation" of this bootstrapped value ensemble:
1 F
F
1 F
QUCB (s, a) = E
Σ
S1(s, a) +k.
Σ
@2(s,a)
- -
(3)
i=1
3

---

Published as a conference paper at ICLR 2024
test reward
test cost
estimation pracie
SAC
110
Samples
Figure 2: Results of SAC-Lag and its conservative version (denoted as "UCB" the legend) on two example tasks. The horizontal dashed line represents the constraint threshold in the cost plot.
where we omit the superscript π in QUCB and Q% for notational brevity. From the Bayesian per- spective, the ensemble forms an estimation of the posterior distribution of Qe (Osband al., 2016), with the standard deviation quantifying the epistemic uncertainty induced by lack of sufficient data (Chua et al., 2018). Despite its simplicity, this technique is widely adopted and is proved to be extremely effective (Ghasemipour et al., 2022; Bai et al., 2022). By taking into account this approximate upper bound of the true value with
uncertainty in Qe estimation, QUCB serves high confidence, as empirically verified by Figure I. Intuitively, the use of UCB induces safety conservativeness in the policy by encouraging actions that not only result in high reward but also yield low uncertainty in cost estimation.
3.2 LOCAL POLICY CONVEXIFICATION
While conservative policy optimization can improve constraint satisfaction by encouraging cost overestimation, it could also impede reward maximization due to the shrink of policy search space (i.e., E[Q5] < B[QUCB] implies (z [ < d) S [T|B[QQ S d)). As illustrated in Figure 1, by narrowing the gap between the conservative boundary (2)) and the true boundary (2), the algo- rithm can expand the search space and thus enhance the probability to obtain a high-reward policy. However, this conservatism gap is co-controlled by the conservatism parameter and the bootstrap disagreement. Since the bootstrap disagreement quantifies the uncertainty and is susceptible to insta- bility, it impractical to precisely control the gap only through tuning the conservatism parameter. In this section, we introduce local policy convexification to stabilize the learning of the policy and the Lagrange multiplier. When working in conjunction with conservative policy optimization, such stabilization helps reduce the bootstrap disagreement and thus gradually narrows down the conser- vatism gap, as interpreted in Section 3.3.
In the estimation of the reward value, even if the estimated values are far from their true values, they can still be helpful for policy improvement as long as their relative order is preserved. Nevertheless, the cost value estimation is less tolerant to estimation error, because the dual update process needs to compare its exact value against an absolute threshold. Thus, even a slight estimation error of the cost value can result in completely wrong À that misleads the subsequent primal optimization. For instance, in objective (2) the optimal value for À is oo if the estimated cost value exceeds the threshold, and is 0 otherwise. Therefore, compared on-policy methods, off-policy methods are less suitable in primal-dus safe RL due to the inherent value estimation eттor (Levine et al., 2020), which can lead to instability in policy learning and oscillations in the overall performance. To stabilize the training of off- off-policy methods, we modify the original objective (2) by applying the augmented I. Lagrangian method (ALM) (see Pages 445-453 of (Luenberger et al., 1984)) as follows?: 2:
For presentational convenience, we only show the performance of k = 2 in Figure 2, and the ablation study of this hyperparameter is provided in Section 3.3. Forr the convenience of the following interpretations, here we use the UCB modified objective by replacing Q in the original objective (2) with QUeR.
4

---

Published as a conference paper at ICLR 2024
min max E' X20
0-
-A X(E
QucB -
@ (B 2
c
(B
- d)
if > wedd-EQQ
E[Qucu]; (4a)
max E
otherwise,
(4b)
where is positive hyperparameter and s Px, Px (-[ss) Such modification does not alter the positions of local optima under a common assumption for inequality constraints (see Appendix A).
Note that (4b) does not involve the cost value, which is reasonable since À S d - [[ implies that the UCB value is below the threshold. As for objective (4a), intuitively, with sufficiently large it will be dominated by the quadratic penalty term and thus will be convex in the neighborhood of d). Theoretically, assuming continuous- time setting, we can define the "energy" of our learning system as E || + _ where the dot denotes the gradient w.r.t. time. Adapting the proof in (Platt & Barr, 1987) to the inequality constraint case leads to the following proposition (see Appendix B for the complete proof):
boundary solution (i.e., a policy satisfying E[QuCB]
Proposition 1. If is sufficiently large, E will decrease monotonically during training in the neigh- borhood of a local optimal policy.
Proof (Sketch). Let (TT) Eg.a Q (s, a)] and g(T) E,,a QuCB (s, a)] d. By transforming the objective into an equivalent form and using the chain rule, the dynamics of the energy E can then be derived as: E 7² Arr, where According to Lemma 1, there exists c* > 0 such that if e > c, A is positive definite at the constrained minima. Using continuity, A is positive definite in some neighborhoods of local optima, hence, the energy function E monotonically decreases in such neighborhood areas.
0 and A - f (A + c(g(T) u + ((Vg))
By definition, relatively smaller E implies smaller update steps of both and A, meaning that the policy will oscillate less when it is close to local minimum. In other words, there will be an "absorbing area" surrounding each suboptimal policy, in which the updating policy can be stabilized and thus be attracted by this local optimum.
3.3 JOINT EFFECT OF CONSERVATIVE POLICY OPTIMIZATION AND LOCAL POLICY CONVEXIFICATION
Differentiating the ALM modified objective (4a) w.r.t. the estimated cost value yields the gradient
B|QUCB SRiV&QUCB (s, o) (s) V (T)),,) where À = max(0, 7- c(d - (s, a)])}. Therefore, the
ALM modification can be seen as applying gradient rectification by replacing À in the original gradient with Å. As illustrated in Figure 1, the effect of such gradient rectification (-) is manifested
in two cases:
1) when the policy crosses the conservative boundary (i.e., E|QU as) > d), the ALM modifica-
tion rectifies the policy gradient by assigning larger penalty weight (i.e., A-c(d-E((eC)) (eedddddeeBB///// (s, a))) > A), which is an acceleration for dragging the policy back to the conservative area; 2) in contrast, when the policy satisfics the conservative constraint (i.c., E| (s, a)] S d) while is still large and not able to be immediately reduced to O, we have A c(d - E EJQU a))) < X, which to some extent prevents unnecessary over-conservativy policy updates from reducing the cost value.
Therefore, applying local policy convexification can force the policy to stay close to the local op- timum, which consistent with the theoretical analysis from the energy perspective. With such a stabilization effect, the collected samples can be focused in line with the distribution of the local optimal policy, which in turn dissipates the epistemic uncertainty in the local convexified area (O in Figure 1). As the uncertainty decreases, the value estimations in the ensemble ((@:) in Eq. (3) will become more accurate, i.e.,
F
Σ ()s,,a) + QT( a),,,),, Std;= (D)(s.a)) +
(5)
E
and the bootstrap disagreement, which quantifies the epistemic uncertainty, will also decrease, i.e.,
1 F ()(s 0.
1 (s),
Σ
E
5

---

Published as a conference paper at ICLR 2024
In this way, the conservativeness induced by the bootstrap disagreement is reduced, and thus the conservative boundary can be pushed gradually towards the true boundary.
reward
test
training
estimation
Greamble Rdd
Crade
0C0-3.5
176
US833
UCB crado
LC3-2-2
Figure 3: Results of applying local policy convexification on two example tasks, averaged over 6 random seeds. The first two columns show the reward/cost learning curves, where the solid lines are the mean and the shadowed regions are the standard deviation. The third column shows the training cost of different timestep intervals by using sox--nddwhissee plots where boxes showing median, 75% (qs) and 25% (41) quarterlies of the distributions respectively, whiskers depicting the error bounds computed as 1.5(gs 21), as well as outliers lying outside the whisker intervals. The last column shows the trend of the standard deviation of the cost value ensemble during training, as well as the difference between the UCB cost value and its oracle value during training (where greater than implies overestimation, otherwise implies underestimation)
To validate the joint effect of the two introduced ingredients, we incorporate them into the off-policy RL method of SAC and denote the final algorithm as Conservative Augmented Lagrangian (CAL) 3 The results are given in Figure 3, where "UCB-k" denotes the method only using UCB with k being the value of the conservatism parameter, and & in CAL is set to 0.5 in the two tested tasks. From the first two columns, conservative policy optimization, applying local policy convexification yields sigmificantly smaller oscillations, indicating that this algorithmic ingredient can stabilize the overall performance both within a single run (the solid line) and across different runs (the shadowed region). Moreover, the third column shows that, 1) as the timestep grows, the episodic training cost becomes more and more stable (embodied in the decreasing box length) and closer to the constraint thresholds; and 2) even compared to sole UCB with a much larger using local policy convexification makes the overall constraint violations and cost oscillations much fewer during training. Additionally, we can see in the last column that the overestimated UCB value gradually approaches its oracle value while the standard deviation of the bootstrapped ensemble decays toward O, which is consistent with our interpretation about the coupling effect of the two ingredients 5.
see that compared to the results of merely depending
the
EXPERIMENTS
In this section, we first compare CAL to several strong baselines on benchmarks in Section 4.1, and then study two hyperparameters that respectively control the effect of the two proposed ingredients in Section 4.2. The experiment on real-world advertising bidding scenario is presented in Section 4.3.
'The pseudo code of CAL can be found Appendix C.
"Note that the learning curves the testing results using extra evaluation episodes after each training iteration, while the hox plots presents the training cost.
'Note that different from the results on Hopper, the UCB value in the PointButton2 task does not exhibit overestimation the early stage of training. This is because the sparsity of the cost in the PointButton2 task increases the difficulty in leaming the value estimates.

---

Published as a conference paper at ICLR 2024
WCSAC
SAGPIDL ag
(Al(ours)
PointButton)
CarButton]
PointButton2
PointPush1
CarButton2
training Cast
TEIAAA THA
2000
HalCChwetah
Ant
Humanoid
sace
(641 CONT
120
training CON
1226
Figure 4: Comparisons with off-policy baselines on Safety-Gym (top half) and velocity-constrained MuJoCo (bottom half).
4.1 COMPARATIVE EVALUATION
We conduct our comparative evaluation the Safety-Gym benchmark (Ray et al., 2019) and the velocity-constraiee MuJoCo benchmark (Zhang et al., 2020). A detailed description of the tested tasks can be found in Appendix D. Baselines categorized into off- and on-policy methods. The off-policy baselines include 1) SAC-PIDLag (Stooke et al., 2020), which builds a stronger version of SAC-Lag by using PID-control update the Lagrange multiplier. 2) WCSAC (Yang al., 2021), which replaces the expected cost value in SAC-Lag with the Conditional Value-at -t-ue- (Rockafellar al., 2000) to achieve controllable conservatism, and 3) CVPO (Liu et al., 2022), which performs constrained policy learning in Expectation Maximization fashion. Due to the space limit, show the comparisons with on-policy baselines Appendix D.2. Implementation details of CAL and the baselines are given in Appendix D.3. Our code can be found in supplementary material.
Figures shows the results on the benchmarks averaged over random seeds. The top/middle rows present the tested reward/cost using extra episodes after each training iteration 6, The bottom row presents the cost induced during training, where the training episodes are quartered in time order and respectively represented by the four boxes for each algorithm. The horizontal dashed lines in the
'Note that the smoothness of the learning curves among the baselines varies due to the different testing intervals, which result from the difference in the training paradigm.

---

Published as a conference paper at ICLR 2024
kew0.0
k.0..5
Hopper
PointButton2
CarButton1
testt rewerrd
0.000
+200
100
16
51.3004
30-75c
1051199
153/2314
training
1-374
113-2506
K-0.5
kw2.0
kw115
c=0
c=1
c=10
c=100
c=1000
Hopper
PointButton2
1est reward
-300
-200
100k
1.5
1144
1/6
3.37c
1.50%
30-126
5L3024
103-1504
26-12
C-0
04.
Co
1o0000
c-1000
co1000
Co 1000
Co0
coi 1.000
Co co1000
Figure 5: Ablation Studies of ls (top half) and e (bottom half).
cost plots represent the constraint thresholds. On all tested tasks, CAL significantly outperforms all baselines in low data regime, and also achieves the same level of reward and constraint satisfaction as that of state-of-the-an on-policy methods (see Appendix D.2), with much fewer samples and constraint violations during training.
4.2 ABLATION STUDIES
An appealing advantage of CAL is that there are only a small number of hyperparameters that need to be tuned. The ablation studies for two key hyperparameters are presented in Figure 5, i.e., in Eq. (3) and e in Eq. (4a), which respectively control the conservativeness in conservative policy optimization and the degree of convexity local policy convexification. As shown the top half of Figure 5, larger & generally leads to lower test/training cost, but overly conservative k can also hinder reward maximization. As a special case, setting A = 0 leads to significantly more constraint violations, and it fails to obtain an asymptotic manssaain-satisffiiii CarButton I. which demonstrates the importance of incorporating conservatism into the algorithm. As for the ablation results of c, increasing results in fewer performance oscillations but slower
policy PointButton22 and
8

---

Published as a conference paper at ICLR 2024
reward improvement. Specially, setting 0 (i.e., without ALM) results in severe performance cillations which hinder both the reward maximization and constraint satisfaction, and an overly large imposes a large penalty on policies exceeding the conservative boundary (where E [QQUCB > d), which significantly slows down the process of expanding the conservative boundary (as illustrated in Figure 1) and thus impedes reward improvement.
In addition, an ablation study for the ensemble size E is shown in Figure 8 in Appendix D.4, where we can see that a larger E generally leads to lower cost, but the overall performance is not very sensitive to this hyperparameter. In theory, increasing the number of Q. networks will improve the reliability of uncertainty quantification. However, Figure 8 shows that in simpler tasks (i.e., Hopper and PointButton2) the benefit of doing this is not as significant as in the harder CarButton2. Since training more networks yields more computational burden, users of the CAL algorithm will need to make some trade-offs according to the difficulty of the target task and the computational expenses.
4.3 SEMI-BATCH SETTING
Semi-batch setting refers to the training paradigm where the agent alternates between 1) collecting a large batch of data with a fixed behavior policy, and 2) updating the target policy on the collected data. Since the policy is not allowed to be adjusted during long-term environment interaction, the online deployment of a policy must ensure a certain level of safety. As a compromise between the online and the offline training (Levine et al., 2020), such training paradigm is crucial for real- world applications of RL (Lange et al., 2012; Matsushima et al., 2021). In this regard, CAL may be particularly useful since by conservatively approaching the optimal policy from the constraint- satisfying area it can ensure both safe online deployment and high offline sample efficiency.
To verify CAL's capability in the semi-batch setting, we conduct experiments in large-scale ad- vertising bidding scenario w.r.t. oCPM (the optimized cost-per-mille), which is a typical semi-bateh application in real world. In the advertising bidding scenario, advertisers bid for advertisement slots that would be exhibited to users browsing the site. After being assigned the target ROI (return on investment) of each advertiser, the auto-bidding policy provided by the platform helps the advertis- ers bid for the slots, in order to maximize the total payments of all advertisers while satisfying each ROI constraint. A more detailed description of this scenario provided in Appendix D.
Our experiment is carried out real-world advertising platform with total amount of 204.8 mil- lion bidding records collected from 47244 advertisers within 4 weeks. During the online deployment of each week, the platform generates 51.2 million records with fixed bidding policy. The policy is only allowed to update once a week on data sampled from the historical dataset. We evaluate the algorithms by summarizing the total advertiser payments and the ROI violation rate within each de- ployment. The experiment results shown in Table demonstrate that CAL significantly outperforms the baselines in terms of payment maximization while maintaining relatively low ROI violation rate, verifying CAL's potential to the large-scale semi-batch setting on real-world applications.
WCSAC
CVPO
CUP
SAC-PIDLag
CAL(our))
Pay/k
Vio/%
Pay/k
Vio%%
Pay/k
Vio/%
Pay/k
Vio/%
Pay/k
Vio/%
Week 1
270.23
30.43
374.38
31.66
433.33
31.86
367.82
32.30
656.31
30.55
Week 2
96.38
25.51
92.23
23.44
116.89
22.63
105.12
25.77
372.51
23.04
Week 3
0
22.57
25.20
21.38
59.47
20.88
29.71
23.12
208.65
18.15
Week 4
25.61
22.70
51.06
20.52
95.80
19.50
58.22
23.07
226.37
17.26
Table 1: Comparison on the oCPM task, where "Pay" refers to the total advertiser payments (value normalized for data security) and "Vio" refers to the ROI violation rate.
CONCLUSION
This paper proposes an off-policy primal-dual method containing two main ingredients, i.e., conser- vative policy optimization and local policy convexification. We provide both theoretical interpreta- tion and empirical verification of how these two ingredients work in conjunction with each other. In addition to the benchmark results which demonstrate a significantly higher sample efficiency com- pared to the baselines, the proposed method also exhibits promising performance large-scale real-world task using a challenging semi-bate batch training paradigm. A possible direction for future work is to extend our approach to fully offline settings.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We gratefully acknowledge support from the National Natural Science Foundation of China (No. 62076259), the Fundamental and Applicational Research Funds of Guangdong province (No. 5213115100122 2946), and the Fundamental Research Funds for the Central Universities-Sum Yat-sen University. This research is also supported by Meituan.
REFERENCES
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7):1-38, 2022.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC press, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
OpenAl: Marcin Andrychowiez, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for incertainty-driver offline reinforcement learning. arXiv preprint arXiv:2202.11560 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q- learning: Learning fast without a model. International Conference on Learning Represen- tations, 2021.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-cone cass-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 33:8378-823 -3390, 2020.
A.M. Duguid. Studies in linear and non-lincar programming, by k. arrow, L. hurwicz and uzawa. stanford university press. 1958. 229 pages. 7.50. Canadian Mathematical Bulletin, 3(3):196-198, 1960. doi: 10.0017750084455025522
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35: 12688877-1 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
Jaaming Ji. Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun. Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint 200i::2300009004 2023.
10

---

Published as a conference paper at ICLR 2024
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurmov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, Reinforcement
learning: State-of-the-ari, pp. 45-73, 2012.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup. Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957 2022a.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup. Kee- gre-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022b.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning, pp. 21 21127-21152 PMLR, 2023.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-poo point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4940-4947, 2020.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Con- strained variational policy optimization for safe reinforcement learning. In International Confer- ence on Machine Learning. pp. 13644-13666 PMLR, 2022.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Con- ference on Machine Learning, pp. 21611-21630. PMLR, 2023.
David Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofrr Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations, 2021.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems, 29, 2016.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained re- inforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32:7555-7565, 2019.
John Platt and Alan Barr. Constrained differential optimization. In Neural Information Processing Systems, 1987.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 19Xiv:1910.01700 7(1):2, 2019.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7787):444 489, 2016.
11

---

Published as a conference paper at ICLR 2024
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR. 2020.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2018.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, volume 255, pp- 263. Hillsdale, NJ, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
2022a.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trust- worthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generaliz- ability. arXiv preprint arXiv:2209.0000 2022b.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference Machine Learning, pp- 11480-1149) 14491. PMLR, 2021.
Long Yang. Jiaming Ji. Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li. Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. In Advances in Neural Information Processing Systems, 2022.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft Iearminn In Proceedings of the AAAI Conference
actor critic for safety- conttrieedd Artificial Intelligence, volume 35, pp. 10639-10646, 2021.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 01115338-134495 2020.
12

---

Published as a conference paper at ICLR 2024
A ASSUMPTIONS AND PROOF FOR "THE ALM MODIFICATION FOR ANEQUALITY-CONSTRAIYY PROBLEMS DOES NOT ALTER THE POSITIONS OF LOCAL OPTIMA"
The ALM modification in Section 3.2 does not alter the positions of local optima under the following strict complementarity assumption (Luenberger al., 1984):
Assumption (Strict Complementarity) E.. (s,a)) = implies that X* well as the second order sufficiency conditions hold for the original constrained optimization problem, where π' denotes the optimal solution the original constrained problem, and X* denotes the optimal solution to the dual problem.
Assumption (Second Order Sufficiency). The second order sufficiency conditions for the original constrained optimization problem include:
Ex F.t (s, a)] =
there exists À 2
V 10 (8, + AVE (Q: (s, a)] =
(7)
the Hessian matrix
VeE,- (s, + AV [Q_ (s, is positive definite on the subspace [ Ex----- (s, = 0}.
(8)
Here we prove that under the strict complementarity assumption, the ALM modification does not alter the position of local optima.
Eg- (-19) original problem can be written as
Proof. Let f(T)
@7 (7 (s. and g(x)) = Egmpa (UC a]] d, the
max f(TT) s.t. g(T) <
(9)
If the strict complementarity assumption is satisfied, then the original problem can written as an equivalent problem with equality constraints (Luenberger et al., 1984):
max f S7
(10)
s.t. g + u == 0, 2 0.
Since e > 0, the optimization problem (10) is equivalent to:
f(T) cl + u 2
(11)
s.t. g(T) 0, 2
which does not alter the position of local optima since the conditions of stationary points of the Lagrangian of Eq. (11) are not changed compared to the original Lagrangian. Then, we formulate the dual function of problem (11) as:
1 <lo(T) + ul?
= max f(T) - x (g( + u] 2 x20,a
(12)
where À is the Lagrangian multiplier. Now show that the optimization of the dual problem, i.e.. minx20 ((A), is equivalent to:
[ 2e (mxx cld - [
min max E
(13)
A20
Given À and the analytical optimal solution ug for problem (12) can be obtain by solving the quadratic problem:
P3 = max ",
c(g;(T)) (T)) + uz1² - (T)) + uz]
(14)
13

---

Published as a conference paper at ICLR 2024
It can be easily computed that the derivative of Eq. (14) is 0 at -9,(T) Az/c. Thus, the optimal
solution to problem (12) is uj = 0 -9, (TT) - >//c cc 0, otherwise uj -9)( ) Ay/c. other
words, the solution can be written
"G max (0. ( - X//c}.
(15)
Substituting Eq. (15) into Eq. (14), we obtain an explicit expression for Pj: 1) for uj = 0, we have:
1
-/2 (1, + (,]] 15);
1
P3 2 gy(5)² (g((()
- 2c
(16)
and 2) for uj 99 9) (T) Xg/c, we have:
Pj = P3=X3/20.
(17)
Combining the two cases together we get:
Pj = 2c
(Imax {0, >3 + c. [,(m]]] 23).
(18)
Therefore, the dual function be written the following form that only depends on À and m:
((A) max (TT) X" [g([)) 020,
clo(m)) + u|?
max [(TT) + Σ P3
(19)
=max f(T))
(1 (mmax {0, A + c. ((T)( 2 2- 1111)
2c
[
1
[max{0, A c(d-E E [QUCh) })² X2
-max IE
2c
According the cquality in Eq. (19), the optimization of the dual problem (12) MinA>0 d(A) A) is equiv- alent to Eq. (13). Based on this result, the objective (4a)(4b) in the main paper be derived, i.e.,
min max A20
E
QUCB
d)
E [QCB]
-d)
a)²
if 2 > d E QUCB
min ((1) X
x
X
+ min max E A20
otherwise.
x
2c
we have reached conclusion that the ALM modification does not alter the positions of local optima.
Since min max
+ x
is equivalent max E
[
A20
B PROOF FOR THE MONOTONIC ENERGY DECREASE AROUND A LOCAL OPTIMUM
In Section 3.2, we define the "energy" of the learning system of Eq. (13) as: E = || 1 where the dot denotes the gradient w.r.. time (assuming a continuous-ting setting). Now we extend the proof in (Platt & Barr, 1987) to show that:
If is sufficiently large, E will decrease monotonically during training the neighborhood of local optimal policy.
Proof. Recall that in the proof in Appendix A, the equivalent problem for the equality constrained optimization (10) can be written
min f(T) + cl u/²
(21)
((T) + 0, a 2 0. The Lagrangian of this constrained objective is:
= - (T) ) + ((g(T) + u) + gd + u/2
(22)
14

---

Published as a conference paper at ICLR 2024
The continuous learning dynamics of Eq. (22) can be written
म 0L
0f
0g
0g
c(g(T) + u) Ôπ
(23)
A (g(x))
+ u)-
During each time interval At, the policy π and the Lagrangian multiplier À are updated to π + # and X + ÄAt, respectively. The dynamics of the energy E = || - 1/2 X2 can be derived as:
d
E
(HT)) ) + =2((()) + =2
dt
2
(24)
d
Σ dt
2 ( (g(+t) + (Vg)
Furthermore, since
d
d
O
X 0g
0g
2 (+:))
#. dt On; 0t;
dg((T) + u)
dt
0m:
0 8 0g 0g
0g
8g ŌT:
c(g(()) u) O,,
Ön,On, Ön,ūn, On, On,
Tj
(25)
derive the following expression for E by substituting Eq. (25) into Eq. (24):
of
0²g 0g 0g θ.. Ōπ,
0
ΣΣ n
c(g(T) + u) Ō,, øj
0g
+ (0(T)) u
mm,
+ T v f - (A + ((g(T)+++))))
# (g(x)) + ()
+ +)) T
= Art,
(26)
where A= V (A c(g(T) + u)) V (V Before discussing this damping matrix A, we first introduce Lemma mentioned in (Duguid, 1960) [Page 172]:
Lemma 1. P is symmetric matrix and if x' Px < for all x + 0 satisfying Q = 0, then for all sufficiently large, P - cQ² Q negalive definite.
Denoting şI p) (mr) + p(g(T) + u) as the Lagrangian function of the original constrained problem (10) and x as an optimal solution for the original constrained problem, we can easily derive that x¹ satisfies: 1) g(Tt) + u O, and 2) there exists critical point ( (T"."²) such that
V²yp(T", = - V (m²") + μ " V"g((")) is positive definite according to the second order suffi- ciency conditions. the point (x*, p²),, ), the damping matrix can be represented by:
A = -==-v2((m + +1 " (V("((")) + <V (V
(27)
=-(P (P--QQTQQ QQTQ),
where P
and Q = Vg(T"). Since P is negative definite, we can apply Lemma and obtain that there exists c* 0 such that if c c², the damping matrix A" is positive definite at the constrained minima (T* "²") Using continuity, the damping matrix A" positive definite in some neighborhoods of the local optima, hence, the energy function E monotonically decreases in such neighborhood areas.
C PSEUDO CODE
The pseudo code of CAL is presented in Algorithm 1.
15

---

Published as a conference paper at ICLR 2024
Algorithm 1 CAL
1: Set hyperparameters: k, and learning rates On. QQQ
2: Initialize policy Tg. double reward Q-functions Qh Q,2 cost Q-function ensemble {0ah 1
fi=1
target networks Q - 1Qa E Lagrange multiplier X, and replay buffer D.
3: for each iteration do
4: for each environment step do
5:
at (-1s)) 81+1 P())st.
6:
7:
D DU (8t, (st, az, r(s,, c(oe; ar),
8:
end for
9:
for each gradient step do
10:
sample experience batch B = {S(A) a(A), 5/2) hi from D
c- -- Ca,EEs,o E.
V log (aals) + Va To(ass) Va min T (s, Voto(als)
sE(1,2)
SAC objective
+λV A.VQuCr QUCB (8, 0) V) [oTo(a where ā= max {0, c (d. B [QUC (8,
27 ,i= Vi 1/ - aQBBsans Vos (c(s,a) + te Q: (8',')) - -x,
13:
1
E
0g x 0, -Q -QQ E Vo. (r(s,a) + min; Ba' (18) 2 (s',a') (s, a)) 27
14:
1,2
- max (0, - axEe,a~8 d QucB (s, (s, as) a))
15:
16:
vi Twts + (1 -T), , == 1,..., E
17:
0, - r0; + -),,, , 1,2
18:
end for
19:
end for
D EXPERIMENT DETAILS
D.1 ENVIRONMENT DESCRIPTIONS
Safety-Gym Safety-Gym (Ray et al., 2019), the agent perceives the world through its sensors and interacts with the world through its actuators. In Button tasks, the agent needs to reach the goal while avoiding both static and dynamic obstacles. In Push tasks, the agent needs to move box series of goal positions. Dense positive rewards will be given to the agent for moving towards the goal, and a sparse reward will be given for successfully reaching the goal. The agent will be penalized through the form of cost for violating safety constraints, i.c., colliding with the static obstacles or pressing the wrong button. To accelerate training, we adopt the modifications on Lafety-Gym tasks made by (Liu et al., 2022). The comparison is still valid since all the algorithms are evaluated on the same set of tasks.
Constrty-Constraineed MuJoCo In MuJoCo tasks (Todorov et al., 2012), we follow the cost def- inition and constraint threshold used in (Zhang al., 2020; Yang et al., 2022) for reproducible comparison. Specifically, the cost calculated as the velocity the agent, and the threshold of each task is set 50% of the velocity attained by an unconstrained PPO (Schulman al., 2017) agent after training for a million samples.
oCPM At the beginning of each day, advertiser s assigns the target ROI Roi, and the maximum total pay P.. At time step t, the advertising system bids pay/,, for advertiser If this bid succeeds, then the advertiser expends pay.,, and obtains revenue gmn,,, (the gross merchandise value) from the exhibited advertisements. In this process, the system has ensure that the average conversion
16

---

Published as a conference paper at ICLR 2024
is larger than the target Roig, i.e.,
Σ,
Rois
Σ
gm
-Rois Pa.
(28)
Ps
Thus, the optimization problem in this scenario can be formulated as:
Σ payse
max
(29)
s.t.
gmoss
RRoi, P.. Vs
We further formulate this problem constrained Markov Decision Process:
The state consists of the user context features, ads context features, cumulative revenue, cu- mulative pay, maximum total pay, target ROI, pCT (predicted click-through rate), pCV (predicted conversion rate) and pGMV V (predicted gross merchandise volume) of one ad- vertiser.
The action to output weight E [0.5, 1.5], which controls the bidding by formula bid k. PCTRACVRRGMV
Rois
The reward is pays,t for successful bid and O otherwise.
The cost is - for successful bid and 0 otherwise.
The constraint budget is RRoi, Note that by definition the ROI value is computed by Expages thus in this scenario replace B[Q2] d in the algorithm by E 1 + Roi gme,,"" Furthermore, the frequency of the change the target ROI is relatively low and does not affect the training.
In this task, the cost and the constraint budget are negative, which means that the constraint is violated at the starting stage of bidding and can only be satisfied over time. During evaluation, each advertiser switches among policies trained by different algorithms, in order to eliminate the differences between advertisers and make the online experiment fairer.
D.2 COMPARISONS WITH ON-POLICY BASELINES
The on-policy baselines include 1) PDO (Chow al., 2017), which applies Lagrangian relaxation to devise cisktconstrained RL algorithm, 2) RCPO (Tessler et al., 2018), which incorporates the constraint penalty into the reward function guide the policy towards constraint-san -satsfiin one, 3) IPO (Liu et al., 2020), which augments the objective with a logarithmic barrier function as a penalty, 4) CRPO (Xu et al., 2021), which updates the policy alternatingly between reward max- imization and constraint satisfaction, and 5) CUP (Yang et al., 2022), which applies a conservative policy projection after policy improvement.
Figure 6 shows the comparative evaluation on two benchmark task sets. As shown by the x-axes, the total number of samples used by on-policy methods are set million in order to present the asymptotic performance of these methods. Note that the curves of CAL look squeezed in these plots. This is because the x-axes are altered to a much larger scale compared to those used Figure 4. We that in each task, CAL achieves reward performance comparable to the on-policy baseline that yields the highest asymptotic reward in this task. Although IPO attains higher reward in several Safety-Gym tasks, its performance on MuloCo tasks is apparently inferior to CAL. Moreover, is worth noting that no on-policy baseline able to achieve constraint satisfaction on all these tested tasks. Most importantly, CAL's advantage sample efficiency is significant, and this particularly important in safety-critical critical applications where interacting with the environment could be potentially risky.
D.3 IMPLEMENTATION DETAILS & HYPERPARAMETER SETTINGS
Baselines
We the implementations in the official codebases, i.e.,
17

---

Published as a conference paper at ICLR 2024
CAlouus
CUP
RCPD
PDO
IPO
CAPO
PointButton1
CarButton1
PointButton2
PointPush1
CarButton2
1051 cost 3
Hopper
HalfCheetah
Ant
Humanoid
2000
2000
2030
2000
3003
1000
1890
Figure 6: Comparisons with on-policy baselines on Safety-Gym (top half) and velocity-constrained MuloCo (bottom half).
http://ghthbbboom//AgggDDDtttttf for WCSAC (Yang et al., 2021)
http://ggiiubbbcco///uuuuivwvvvs for CVPO (Liu al., 2022)
http://giihabbcom///mmn22//////p for CUP (Yang et al., 2022)
for PDO (Chow et al., 2017). RCPO (Tessler et al., 2018), IPO (Liu al., 2020), CRPO (Xu al., 2021). Note that this is infrastrue- tural framework for safe RI. algorithms implemented by Ji et al. (2023).
For SAC-PIDLag, we use the implementation in the CVPO codebase. We adopt the default hyper- parameter settings in the original implementations. Note that the MuJoCo results not presented in the CVPO paper, so the hyperparameters of CVPO are finetuned on MuloCo tasks based on the https///gghuub.co/lluuuuii in WCSAC is set to the risk-averse 0.1.
authors' modified codebase
In addition, the risk hyperparameter
CAL The network structures of the actor and the reward/cost critics of CAL are all the same with the off- off-policy baselines. i.e., 256 neurons for two hidden layers with ReLU activation. The discount factor is set to 0.99 both in reward and cost value estimations. The optimization of the networks conducted by Adam with the learning rate 5e 4 for the cost critics, and 3e for the actor and the reward critics. The ensemble size E is set to 4 for MuJoCo tasks and 6 for Safety-Gyn tasks. The conservatism parameter i parameter e is set to 10 for all tasks except for Ant (100), HalfCheetah (1000) and Humanoid (1000).
0.5 for all tasks except for PointPushl 1 (0.8). The convexity
About the Up-To-Data Ratio To make a more sufficient utilization of each data point and improve the sample efficiency, larger up-to-data (UTD) ratio (i.e., the number of updates taken by an agent environment interaction) (Chen et al., 2021) than normal off- off-policy algorithms is adopted in the implementation of CAL. The UTD ratio of CAL set 20 for all tasks except for Humanoid
'Specifically, we set 1) the number of episodes per data collection to 2) the number of updates per environment sample 0.5; and 3) the constraint of the policy's standard deviation in the M-step to 0.005.
18

---

Published as a conference paper at ICLR 2024
SAC-PIDLag
100
20
740
18
1000
Taxt cONT
SAC-MDLag
WCSAC
CVPO
CAL
roward
100
8
1504
D00
10C
200
Figure 7: Results of different UTD settings (1, 10 and 20) on Hopper (top half) and PointButton2 (bottom half).
(10) and HalfCheetah (40). In the comparative evaluation (Section 4.1), we adopt the default UTD settings for the baselines which are lower than the ratio in CAL. Since the off-policy baselines also have the potential to be improved by using larger UTD ratio, here we provide results of different UTD settings for these baselines (and also CAL) for a more comprehensive comparison. Clearly, can see from Figure 7 that different from CAL, increasing the UTD ratios of the baselines only improves their reward performances cannot improve their constraint satisfactions, indicating that the capability of using large UTD ratio is also one of the non-trivial advantages of CAL..
D.4 ABLATION STUDY FOR THE ENSEMBLE SIZE
See Figure 8.
E RELATED WORK
In recent years, deep safe RL methods targeting high- high-dimensional controlling tasks have been well developed. One of the most commonly adopted approach for safe RL the primal-dual frame- work due to its simplicity and effectiveness. In this regard, Chow et al. (2017) propose primal-dual method and prove the convergence to munsraint-satisffingg policies. Tessler et al. (2018) propose approach and local convergence guarantees. Paternain et al. (2019) prove that primal-dual methods have zero duality gap under certain assumptions. Ding al. (2020) show the convergence rate of natural policy gradient based on the primal-dual framework. Ray et al. (2019) make practical contributions by benchmarking safe RL baselines on the Safety-Gye environment. Nevertheless, the majority of existing primal-dual methods are on-policy by their design. Despite the convergence guarantees, "except for SAC-PIDI.gg PointButton2 (20) and CVPO MuloCo tasks (0.5).
alternative penalty signal to leam constrained policies with
19

---

Published as a conference paper at ICLR 2024
Ew4
E-6
Hopper
PointButton2
CarButton2
2000
350
100c
11.60
10c
1004
1564
2004
book
2036
Y00
23
5.
LOCk
2006
160
53:1000
80
62-
100 25.1
62-
226-7004
trairing
28-758
Figure 8: Ablation study for the ensemble size E.
these methods usually encounter the sample efficiency issues, demanding a large amount of envi- ronment interaction and also constraint violations during the training process. This may impede the deployment of such methods in safety-critich applications. An approach to avoid risky environment interaction is training constraint-satisfyy
policies using only offline data (Le al., 2019; Lee al., 2022a; Xu et al., 2022a; Liu et al., 2023; Lin et al., 2023). Instead, this paper focuses on the online setting and attempts to enhance the safety of RL methods by improving the sample efficiency.
The methods most related to ours are as follows: Stooke et al. (2020) apply PID control on the dual variable and show promising performance when combined with the on-policy PPO method (Schulman et al., 2017). This method is adopted as of our baselines since reported in (Stooke et al., 2020), the PID control can stabilize the update of the dual variable and thus has the potential to be adapted to the off-policy setting. WCSAC (Yang et al., 2021) augments the naïve SAC-Lag method with a variance estimator of the cost value to achieve risk control. Yet, in practice found that the temporal-din wiwooralddiffeeeee-ttyeee learning of this variance estimator is less stable than directly using the variance of an ensemble, which may be one of the causes explaining WCSAC's inferior performance compared CAL. CVPO (Liu et al., 2022) allows off-policy updates by break- ing the brittle bond between the primal and the dual update and introducing variational policy to bridge the two updates. This method shows significant improvement in sample efficiency but suffers from relatively low computational efficiency, due to the frequent sampling operations in the action space when computing the variational policy.
20


Reviews:
[{"review_id": "seSyiisavk", "reviewer": "Reviewer_cCE3", "summary": "This paper considers the problem of safe reinforcement learning in constrained Markov decision processes, where the agent tries to maximize reward while keeping a separate cost signal under a given threshold. The authors propose an off-policy primal-dual deep RL method that combines two techniques. The first, called conservative policy optimization, replaces cost-value estimates with an empirical upper confidence bound based on the variance of the estimates across an ensemble of Q networks. The second, called local policy convexification, applies the augmented Lagrangian method to the primal-dual formulation of constrained RL. While the first is intended to make the policy search more conservative reducing the number of constraint violations due to cost estimation, the second is intended to stabilize the learning process. The algorithm that combines these two techniques, called CAL (Conservative Augmented Lagrangian), is tested on benchmark tasks and on a real advertising system (in a semi-batch fashion) and compared with several baselines, showing a significant improvement over the state of the art both in terms of reward maximization and constraint violation.", "questions": "How many Q networks did you use in the ensemble and how did you choose this hyperparameter?\n\nJust a minor remark: the objective should be stated in terms of the true value functions and not their estimates (equation 1), even if they must be estimated from data in practice.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper considers an important problem, that of safe RL, from a practical perspective.\nIts main strength is in the extensive experimental campaign. The experiments are well designed and documented, and convincingly show the advantages of the proposed method over the state of the art. It is particularly impressive to see experiments on a real system, where the proposed method is also compared against several baselines.\nThe main point of originality is in using the augmented Lagrangian method in this setting.", "weaknesses": "The theoretical motivations are a bit brittle. The authors should be more clear on where the theoretical inspiration ends and where heuristics begin. For instance, in section 3.1, it is suggested that Equation (3) would be an actual upper confidence bound in a linear MDP, which is inaccurate, as the upper confidence bound for linear MDPs has a specific form that does not correspond to equation (3). \nAlso, more words should be spent in explaining how the bootstrapped value ensemble is constructed, and why you chose this form of uncertainty estimation.\nRegarding ALM, I think that the theory that is provided in the appendix is interesting and deserves more space in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u59LIFtzJ8", "reviewer": "Reviewer_Dfxp", "summary": "This paper considers the problem of constrained off-policy reinforcement learning and remedies the issue of inaccurate cost estimation in prior Lagrangian duality-based approaches in two steps: i) the cost value estimate is replaced by an upper-confidence bound, which is derived through the uncertainty, or lack of consensus, in an ensemble of cost value estimators, and ii) the regular Lagrangian is replaced by the augmented version of the Lagrangian in order to make the optimization landscape in the feasible region surrounding the local optimal policy. Essentially, the first component encourages the policy to be conservative and ensures better constraint satisfaction, while the second component encourages the policy to become as aggressive as possible in the space of feasible policies in terms of the primary reward function. Numerous experiments demonstrate the superiority of the proposed method, termed CAL, over multiple baseline methods in a variety of settings and environments.", "questions": "- Could you please provide more details on how the cost value ensemble in (3) is created? Do you have multiple (i.e., $E$) parallel parameterized models that each estimate the cost in parallel? Should the expectation in (3) be replaced by an empirical mean (i.e., $\\frac{1}{E} \\sum_{i=1}^E$)?\n- The last statement in Section 3.1 mentions that the UCB \"encourages the policy to choose actions with low uncertainty in cost estimation.\" Could you comment on why that is the case? Based on (3), it seems that for state-action pairs with low uncertainty in cost estimation, the UCB is close to each of the ensemble estimates, but it does necessarily translate to the action leading to the highest estimated reward in that state.\n- I suggest changing the notation for the hyperparameter $c$ in the augmented Lagrangian, as it has also been used to denote the cost function in the CMDP definition.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Training safe RL policies is a very important research direction, and alongside the aforementioned components of controlled conservativism and aggressiveness, the paper presents a sample-efficient constrained RL training procedure that also has safer behavior during **training**, something that has potentially been overlooked in prior work in this area.\n\n- I very much enjoyed reading the manuscript. The content is very well presented, and the experimental results are, in my opinion, excellent and convincing as to why and how the proposed method works well.", "weaknesses": "One limitation I can think of is the inclusion of only one constraint in the problem formulation, solution, and experiments. The authors have mentioned that having a single constraint does not prevent generality, but I wonder how the method performs in settings with multiple competing constraints, especially as compared to baseline methods. A discussion on how the two introduced components interplay in the presence of several constraints would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bJS1uCHJhc", "reviewer": "Reviewer_M5JF", "summary": "This paper presents primal-dual method to solve constrained RL problem combining the conservative policy optimization and local convex optimization. By conservative policy, the estimated Q-value is calculated via averaging over several copies of Q-value and adds UCB-style bonus term. The local convexification comes from well-posed augmented Lagrangian problem.", "questions": "- Why do we have to reformulate the augmented Lagrangian? What is the advantage of the reformulation according to Luenberger compared to the original augmented Lagrangian (equation (13))?\n\n- For Figure (5), as for Hopper environment, when $c=0$, the test-cost seems to be not that bad in the sense that average is below the threshold. Does this imply the threshold constraint is too loose?\n\n- Please provide page number or section for Luenberger 1984, about the derivation of augmented Lagrangian, which will be helpful to the readers.\n\n\n- In section 3.1, about the underestimation, we have $\\mathbb{E} [ \\hat{Q}_c(s,\\pi(s))]=\\mathbb{E}[Q_c(s,\\pi(s))+\\epsilon]=\\mathbb{E} [Q_c(s,\\pi(s))]$. Why does taking minimum over $\\pi$ implies underestimation bias? Please give some clarifications.\n\n- It would be helpful if the authors formally introduce and rephrase the theorem in A.M. Duguid. \n\n\nA.M. Duguid. Studies in linear and non-linear programming, by k. j. arrow, l. hurwicz and h. uzawa. stanford university press, 1958. 229 pages. 7.50. Canadian Mathematical Bulletin, 3(3):196–198, 1960. doi: 10.1017/S0008439500025522.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The primal-dual approach is known to be difficult to train but combining several technique, the authors show improved performance compared to recent results in safe RL.", "weaknesses": "- I believe there needs to be some clarifications about the proof that ALM does not modify or alter the position of local optima. Regarding equation (20), in general expectation of max or square is not same as the max or square of expectation, i.e., $\\mathbb{E}[X]^2\\leq \\mathbb{E}[X^2]$ and the same for the max operator. Hence, we cannot guarantee the following equation to hold:\n$$ |\\max\\\\{ 0,\\lambda+c \\mathbb{E} [ \\hat{Q}^{UCB}_c(s,a) ] \\\\}  |^2 = \\mathbb{E} [| \\max\\\\{ 0,\\lambda+c\\hat{Q}^{UCB}_c(s,a) \\\\} |^2].$$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fpBQSaSs6V", "reviewer": "Reviewer_oKob", "summary": "The study conducted herein delves into the issue of underestimation in safe off-policy RL. In addressing this problem, the study conducts a series of experiments aimed at assessing the efficacy of the proposed methodology. The outcomes of these experiments hold substantial promise for enhancing RL applications within real-world environments.", "questions": "1. Is it possible to extend Equation (5) into a third-order equation, and if so, what impact might this have on performance?\n2. What is the connection between the variance in Q-cost value estimation and the process of policy optimization?\n3. How does the value of \"c\" affect performance in terms of both cost minimization and reward maximization?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The presentation is clear and straightforward, making it easy to understand.\n2. The research conducted in this study is interesting.", "weaknesses": "1. The study could benefit from a more comprehensive exploration of convergence and stability, particularly concerning policy optimization. The theoretical analysis in this regard appears to be somewhat limited.\n\n2. It is suggested that the on-policy experiments be completed, and the results of training for on-policy RL methods be included for a more comprehensive evaluation. The on-policy experiments for this method are not completed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)