Balancing Constraints and Rewards with Meta-Gradient D4PG
Authors: Dan A. Calian
Daniel J Mankowitz
Tom Zahavy
Zhongwen Xu
Junhyuk Oh
Nir Levine
Timothy Mann
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
meta-gradients
constraints

Abstract: Deploying Reinforcement Learning (RL) agents to solve real-world applications often requires satisfying complex system constraints. Often the constraint thresholds are incorrectly set due to the complex nature of a system or the inability to verify the thresholds offline (e.g, no simulator or reasonable offline evaluation procedure exists). This results in solutions where a task cannot be solved without violating the constraints. However, in many real-world cases, constraint violations are undesirable yet they are not catastrophic, motivating the need for soft-constrained RL approaches. We present two soft-constrained RL approaches that utilize meta-gradients to find a good trade-off between expected return and minimizing constraint violations. We demonstrate the effectiveness of these approaches by showing that they consistently outperform the baselines across four different Mujoco domains.

Published as a conference paper at ICLR 2021
BALANCING CONSTRAINTS AND REWARDS WITH META-GRADIENT D4PG
Dan A. Calian*, Daniel J. Mankowitz², Tom Zahavy, Zhongwen Xu, Junhyuk Oh, Nir Levine & Timothy Mann
Deep.iidd
London, United Kingdom
(danca 1 ian, amankowitz @igankowitz//ooog11111 @ww.gggo........oo..o com
ABSTRACT
Deploying Reinforcement Learning (RL) agents to solve real-world applications often requires satisfying complex system constraints. Often the constraint thresh- olds are incorrectly set due to the complex nature of a system or the inability verify the thresholds offline (e.g, no simulator or reasonable offline evalua- tion procedure exists). This results in solutions where a task cannot be solved without violating the constraints. However, in many real-world cases, constraint violations are undesirable yet they are not catastrophic. motivating the need for coft-constr RL approaches. We present a coft-constrained RL approach that utilizes meta-gradients find a good trade-off between expected return and min- imizing constraint violations. We demonstrate the effectiveness of this approach by showing that it consistently outperforms the baselines across four different Mu- joco domains.
INTRODUCTION
Reinforeement Learning (RL) algorithms typically try to maximize an expected return objective (Sutton & Barto 2018). This approach has led to numerous successes in a variety of domains which include board games (Silver et al., 2017). computer games (Mnih et al. 2015; Tessler et al. 2017) and robotics (Abdolmaleki et al., 2018). However, formulating world problems with only an expected return objective is often sub-optimal when tackling many applied problems ranging from recommendation systems to physical control systems which may include robots, self-driving cars and even aerospace technologies. In many of these domains there are a variety of challenges preventing RL from being utilized as the algorithmic solution framework. Recently, Dulac-Arnodd jtt al. (2019) presented nine challenges that need to be solved to enable RL algorithms to be utilized in real-world products and systems. One of those challenges is handling constraints. All of the above domains may include one or more constraints related to cost, wear-and and-tear, or safety, to name few.
There are two types of constraints that are encountered constrained optimization problems; namely hard-con hard-constraints and soft-constraints (Boyd & Vandenberghe, 2004).. Hard constraints are pairs of pre-specified functions and thresholds that require the functions, when evaluated on the solution, to respect the thresholds. As such, these constraints may limit the feasible solution set. Soft constraints are similar to hard constraints in the sense that they are defined by pairs of pre-specified functions and thresholds, however, a soft constraint does not require the solution to hold the constraint; instead, it penalizes the objective function (according to specified rule) if the solution violates the constraint (Boyd & Vandenberghe, 2004; Thomas et al. 2017).
Hard and Soft Constraints:
Motivating Soft-Constraints. In real-world products and systems, there are many examples of soft-constraints; that is, constraints that can be violated, where the violated behaviour is undesirable but not catastrophic (Thomas et al., 2017; Dullc-Arnod et al., 2020b). One concrete example is that of energy minimization in physical control systems. Here, the system may wish to reduce the amount of energy used by setting soft-constraint. Violating the constraint is inefficient, but not catastrophic to the system completing the task. In fact, there may be desirable characteristics that can only be attained if there are some constraint violations (e.g.. a smoother/faster control policy). Another common setting is where it is unclear how to set a threshold. In many instances, a product
indicates equal contribution.

---

Published as a conference paper at ICLR 2021
manager may desire to increase the level of performance on particular product metric A, while ensuring that another metric B on the same product does not drop by "approximately X%'. The value 'X' is often inaccurate and may not be feasible in many cases. In both of these settings, violating the threshold is undesirable, yet does not have catastrophic consequences.
Lagrange Optimization: In the RL paradigm. a number of approaches have been developed to incorporate hard constraints into the overall problem formulation (Altman 1999; Tessler et al., 2018: 201,, Efroni et al.) 2020: [Achiam et al., 2017: [Bohez et al.) 2019; Chow et al., 2018: Patemaii et al. 2019; Zhang et al. 2020; Eironi et al, 2020). One popular approach is to model the problem as Constrained Markov Decision Process (CMDP) (Altman, 1999). In this case, one method is to solve the following problem formulation: max, JR s.t. Je S B. where π is a policy. Jă is the expected return, Jg is the expected cost and B is constraint violation threshold. This is often solved by performing alternating optimization on the unconstrained Lagrangian relaxation of the original problem (e.g. [Tessler et al. (2018)) defined as: minx>0 m JR X(B- The updates alternate between learning the policy and the Lagrange multiplier X.
In many previous constrained RL works (Achiam et al., 2017: Tessler et al., 2018: Ray et al., 2019; Satija et al,, 2020), because the problem is formulated with hard constraints, there are some domains in each case where feasible solution is not found. This could be due to approximation errors, noise, or the constraints themselves being infeasible. The real-world applications, along with em- pirical constrained RL research results, further motivates the need develop coft-constrained RL optimization approach. Ideally, in this setup, we would like an algorithm that satisfies the constraints while solving the task by maximizing the objective. If the constraints cannot be satisfied, then this algorithm finds good trade-off (that is, minimizing constraint violations while solving the task by maximizing the objective).
In this paper, we extend the constrained RL Lagrange formulation perform soft-constrained opti- mization by formulating the constrained objective as a nested optimization problem (Sinha et all 2017) using meta-gen vetta-gradinns. We propose MetaL that utilizes meta- tetta-graients (Xu et al., 2018; Za- havy al. (020)) to improve upon the trade-off between reducing constraint violations and improv- ing expected return. We focus on Distributed Distributional Deterministic Policy Gradients (D4PG) Barth-Maron al., 2018) the underlying algorithmic framework, state-of-the-arr continuous control RL algorithm. We show that Metal. can capture an improved trade-off between expected return and constraint violations compared to the baseline approaches. We also introduce a second approach called MeSh that utilizes meta- veta-gradieets by adding additional representation power the reward shaping function. Our main contributions are as follows: (1) We extend D4PG to handle constraints by adapting it to Reward Constrained Policy Optimization (RCPO) TTssler et al.) 2018) yielding Reward Constrained D4PG (RC-D4PG); (2) We present a soft constrained meta-gradien technique: Meta- deea-Grrddeent for the Lagrange multiplier learning rate (MetaL (3) We derive the meta- grtt--gradien update for MetaL (Theorem I); (4) We perform extensive experiments and investiga- tive studies to showcase the properties of this algorithm. Metal outperforms the baseline algorithms across domains, safety coefficients and thresholds from the Real World RL suite (Dulac-Aroll let al., 2020b).
2 BACKGROUND
A Constrained Markoy Decision Process (CMDP) is an extension to an MDP (Sutton & Barto, 2018) and consists of the tuple (8, A, P. R, C, where S is the state space; A is the action space; P. S x A + S is a function mapping states and actions to a distribution over next states; R : S x A + R is a bounded reward function and C S A + RK is a K dimensional function representing immediate penalties (or costs) relating to K constraints. The solution to a CMDP is policy π S + AA which is mapping from states to probability distribution over actions. This policy aims to maximize the expected return Jî = E 7 rrre] and satisfy the constraints Jg. E 7'c.,l I B,, = 1... K. For the purpose of the paper, we consider a single constraint; that is, K 1, but this can easily be extended to multiple constraints.
Meta-Gradients is an approach to optimizing hyperparameters such as the discount factor, learning rates, etc. by performing online cross validation while simultaneously optimizing for the overall RL optimization objective such the expected return (Xu al., 2018; Zahavy al., 2020). The goal is to optimize both an inner loss and an outer loss. The update of the 0 parameters on the inner
TTiis is also the first time meta-gradients have heen applicd an algorithm with an experience replay.
2

---

Published as a conference paper at ICLR 2021
loss is defined 6' = = f(,, where 0 E Rd corresponds to the parameters of the policy To(a]s) and the value function (s) (if applicable). The function f IR* - Rd is the gradient of the policy and/or value function with respect to the parameters 0 and is function of an n-step trajectory
-81, 01, r2, 82. 8n), meta-parameters v and is weighted by a learning rate a and is defined as f(),0,n) = d (0.7.1)
where Jby (0, T, 7)) is the objective being optimized with respect to 0. The idea is to then evaluate the performance of this new parameter value 6' on outer loss the meta-gradient objective. We define this objective as J'(T', 0", (0',)) 17) where 7' new trajectory,
dø
o' are the updated parameters and fj is fixed meta- paraaeee (which needs to be selected/tuned in practice). We then need to take the gradient of the objective J' with respect to the meta-panann meea-parsmeters
(J'(r'.0' m))
to yield the outer loss update Ty' = + Og (J'('',8'.)) aJ'(r' 8' m) 00' 87
This gradient is computed as follows:
8y
By The outer loss is essentially the objective we are trying to optimize. This could be a policy gradient loss, temporal difference loss, combination of the two etc (Xu et 2018: Zahavy et al., 2020). Meta-gradients have been previously used to learn intrinsic rewards for policy gradient (Zheng et al., 2018) and auxiliary tasks (Veeriah et al., 2019). Meta-gradients have also been used adapt optimizer parameters (Young al.. 2018; Franceschi et al., 2017). In our setup, we consider the continuous control setting, provide the first implementation of meta- gradients for algorithm that uses experience replay, and focus on adapting meta-parameters that encourage soft constraint satisfaction while maximizing expected return.
D4PG is state-of -f------thhh---- continuous control RL algorithm with a deterministic policy (Barth-
Maron et al. 2018). It is an incremental improvement to DDPG (Lillicrap al., 2015p. The overall objective of DDPG is to maximize J(a,, 02) = E[Qg, (s. a)ls
St, a = To, (st)] where To, (5t) is deterministic policy with parameters 02 and Qo (s, a) is an action value function with param- eters 8.. The actor loss is defined as: Licttr = (VG(VV (s,, la) + +a...) ag.,t ll2 where SG is a stop gradient.
The corresponding gradient update is defined as Jo. = Eo Ve, Te_ (st)]. The critic is updated using the standard temporal dif- ference error loss: Lerstic (r(s, a) + "Q (T') (Q((s',Tyss))) - Qo, (s, a))² where Qr, TT are the target critic and actor networks respectively. In D4PG, the critic is a distributional critic based on the C51 algorithm (Bellemare al. 2017) and the agent is run in a distributed setup with multiple actors executed in parallel, n-step returns and with prioritized experience replay. We will use the non-distributional critic update in our notation for ease of visualization and clarity for the reade
REWARD CONSTRAINED D4PG (RC-D4PG)
This section describes our modifications required to transform D4PG into Reward Constrained D4PG (RC-D4PG) such that it maximizes the expected return and satisfies constraints.
subject to J < 3, where J = E (s)(ssa)|s = St, a = [s))]] and J = E[C(s, a)ls = st, a = Te [T{))]]; the param- eter 0 = ( Oz) from here on in; C(s, a) is a long-term penalty value function (e.g., sum of dis- counted immediate penalties) corresponding to constraint violations. The Lagrangian relaxation objective is defined as J + A(3 JC ). As in RCPO, proxy objective J XJ is used that converges the same set of locally optimal solutions the relaxed objective (Tessler al.] 2018). Note that the constant does not affect the policy improvement step and is only used for the Lagrange multiplier loss update. To optimize the proxy objective with D4PG, reward shap- ing of the form r(s, a) Xc(s, a) required to yield the reward shaped critic loss defined as: Lerätic( A) (r(s, a) Ac(s. a) + 7Qr(s, AT Qo, (s, The actor loss is defined as before. The Lagrange loss is defined as: Liagrange (A) = X(3 -G. where À Z O. Since RC- D4PG is off-policy, it requires storing the per time-step penalties, inside the transitions stored in the experience replay buffer (ER). For training the Lagrange multiplier an additional penalty buffer is used to store the per-episode penalties J The learner then reads from this penalty buffer for updating the Lagrange multiplier. RC-D4PG updates the actor/eritic parameters and the Lagrange multipliers using alternating optimization. The full algorithm for this setup can be found in the Appendix, Algorithm g
The constrained optimisation objective is defined as: max J
TThis can easily he extended include the distribational critic.
3

---

Published as a conference paper at ICLR 2021
4 DETA-GRADDENTE FOR THE LAGRANGE LEARNING RATE (METAL)
In this section, we introduce the MetaL algorithm which extends RC-D4PG to use meta-gradients for adapting the learning rate the Lagrangian multiplier The idea to update the learning rate such that the outer loss (as defined in the next subsection) is minimized. Our intuition is that learning rate gradient that takes into account the overall task objective and constraint thresholds will lead to improved overall performance.
Algorithm MetaL
1: Input: penalty c(-), constraint C(-), threshold 3, learning rates an 000,, 002, Osy, max. number of episodes M
2: Initialize actor, critic parameters 0₂ and f.. Lagrange multiplier À 0, meta parameter ax
3: for 1... .M do
4: Inner loss:
5: Sample episode penalty Jž from the penalty replay buffer
6: X < A - a exp(axx (3 (J)]+
Lagrange multiplier update
7:
Sample a batch with tuples (st, de, dr, ot) from ER and split into training/validation /alidationn sets
8:
Accumulate and apply actor and critic updates over training batch T Traain by:
9:
V0. 0, V0,-
10:
for & = 1 ....Trroin do
11:
R = rı - -ce + 7Q(1, s1+1, a++1 (s(41)));;
12:
v0. += as. RR: (A, st. de 0o))) 0,)
Critic update
13:
v0, += ao, E V (sc at) Vo, To, (84) (o,)
Actor update
14:
0-
t O₂ Tergin Σ Vo.
15:
O₂ 02 + Terain VVo,
16:
Outer loss: Compute outer loss and meta-gradient update using validation T palidate > Meta-parameter update- Theorem
a'3 < an -0g aJ' (aa
17:
deg
18:
x N. + 02a Oe - ax t
return 0a, Oa,
Meta- petaaparametees, inner and outer losses: The meta parameter is defined as = ax- The inner loss is composed of three losses, the actor, critic and Lagrange loss respectively. The actor and critic losses are the same as in RC-D4PG. The Lagrange multiplier loss is de- J where ax is the meta-parameter as defined above.
fined as: Llagrange =x)==)x())){(3 The meta-parameter is wrapped inside an exponential function to magnify effect of an while also ensuring non-negativity of the effective learning rate. The inner loss updates are
060a dLactor
TO' 00
T0a1
J(T,0a. S(T,
Te.
alcoritic Meerriie00... (B
0c
- A
=
0a A
do, dlttgrannge (A) dA
where ag,, ag., a1 are the fixed ac-
f(T,A,m)
a1
tor critic and Lagrange multiplier learning rates respectively. The outer loss is defined as We tried different variants of outer losses and found that this loss empirically yielded the best performance; we discuss this in more detail in the experiments section. This analogous formulating Metal. as the following nested optimiza- tion problem: min., J'(0(ax), A(ax)) s.t. 0, E arg ming - (B J }}. We treat the lower level optimization problem the Lagrange relaxation objective (inner loss). We then treat the upper level optimization as the meta-gradicat objective ('(0(ax), X(ax)) (outer loss). This transforms the optimization problem into soft-constrained optimization since the meta--aramettrr ax guides the learning of the Lagrange multiplier minimize the outer loss while attempting find a good trade-off between minimizing constraint violations and maximizing return (inner loss).
(ax,), ((ax)) Louter Loritio (a.
As shown in Algorithm [I the inner loss gradients computed for À (line 6), 9_ (line 12) and 0, (line 13) corresponding to the Lagrange multiplier, critic and actor parameters respectively. The Lagrange multiplier is updated by sampling episode penalties which is an empirical estimate of J from a separate penalty replay buffer (line 5) to compute the gradient update. The updated multiplier
We plan releasing the code for Metal. in the near future.
4

---

Published as a conference paper at ICLR 2021
is then utilized in the critic inner update (lines 11 and 12) to ensure that the critic parameters a function of this new updated Lagrange multiplier. The actor and critic parameters are updated using the training batch. and these updated parameters along with a validation batch are used to compute the outer loss (line 17). The meta-parameter ax is then updated along the gradient of this outer loss with respect to lj = ax. We next derive the meta- vreta-grddeett update for as, and present it in the following theorem (see the Appendix, Section A for the full derivation). Intuition for this meta-gradient update provided in the experiments section.
Theorem 1. MetaL gradient update: Let B 2 O be a pre-defined constraint violation threshold,
mera-parameter lt = og and Je = C(s, a){s s = S1, a = no( the discounted constraint
violation function, then, the meta- gradient update is:
(- -225-((s.a)-01 exp(ax)
(FE- ( (Vo:Qo a)) Vo (..
d aax
(V00,
where ō is the TD error; on, is the critic learning rate and Ong is the meta-parameter learning rate.
EXPERIMENTS
The experiments were performed using domains from the Real-Worke Reinforcement Learning (RWRL) suite") namely cartpole:swingup. walker:walk, quadruped:walk and humanoid: We will refer these domains as cartpole, walker, quadruped and humanoid from here on in.
We focus on two types of tasks with constraints: (1) solvable constraint tasks where the task is solved and the constraints can be satisfied; (2) unsolvable constraint tasks where the task can be solved but the constraints cannot be satisfied. Unsolvable constraint tasks correspond tasks where the constraint thresholds are incorrectly set and cannot be satisfied, situations which occur in many real-world problems motivated in the introduction. The specific constraints each domain can be found in the Appendix (Section C). The goal is to showcase the soft-constrained performance of Metal., with respect to reducing constraint violations and maximizing the return in both of these scenarios (solvable and unsolvable constraint tasks) with respect to the baselines.
focused on for
The baseline algorithms we focused on for each experiment are D4PG without any constraints, RC- D4PG (i.c., hard constraint satisfaction) and Reward Shaping D4PG (RS-D4PG) (i.c., soft constraint satisfaction). RS-D4PG uses fixed for the duration of training. We compare these baselines to Metal. Note that D4PG, RC-D4PG and Metal have no prior information regarding the Lagrange multiplier. RC-D4PG and MetaL attempt to learn a suitable multiplier value from scratch, i.c. the initial Lagrange multiplier value is set to 0.0. In contrast, RS-D4PG has prior information (i.e. it uses pre-selected fixed Lagrange multiplier).
Experimental Setup: For each domain, the action and observation dimensions are shown in the Appendix, Table + The episode length is 1000 steps, the base reward function is computed within the dm_cont rol I suite (Tassa al., 2018). The upper bound reward for each task is 1000. Each task was trained for 20000 episodes. Each variant D4PG uses the same network architecture (see the Appendix, T for more details).
We use different performance metrics to compare overall performance. We track the average episode return (R), but we also define the penalized return: Reenalized = R - x tha.c, which captures the trade-off between achieving optimal performance and satisfying the constraints. Here, R is the average return for the algorithm upon convergence (computed as an average over the previous 100 episodes); w. is fixed constant that determines how much to weight the constraint violation penalty. For the purposes of evaluation, we want to penalize algorithms that consistently violate the constraints and therefore set λ = 1000. Since the upper bound of rewards for each domain is 1000, we are essentially weighing equally attaining high performance and satisfying constraints. Finally, t0,. = max(0, - - 8) is defined as the overshoot. Here is the constraint violation threshold and defines the allowable average constraint violations per episode; J is the average constraint violation value per episode upon convergence for policy म. The overshoot, & tracks the average constraint violations that are above the allowed constraint violation threshold B.
We investigate each algorithm's performance along variety of dimensions which include differ- ent constraint violation thresholds (see the Appendix, Table 3 for list of thresholds used), safety
https://gi http//ggiiuubboooo www.hooo.ooooooooooooo/o/ooooooo resea rch reaa/eaaaaa wor 1dd_uuitee
5

---

Published as a conference paper at ICLR 2021
coefficients and domains. The safery coefficient is flag in the RWRL suite Dulac-Arnole al. 2020a). This flag contains values between 0.0 and 1.0. Reducing the value of the flag ensures that more constraint violations occur per domain per episode. As such, we searched over the values {0.05, 0.1, 0.2, 0.3). These values vary from solvable constraint tasks (e.g.. 0.3) to unsolvable con- straint tasks (e.g., 0.05). We wanted to see how the algorithms behaved in these extreme scenarios. In addition, analysed the performance across variety of different constraint violation thresholds (see Appendix, Table 6). All experiments are averaged across seeds.
5.1
MAIN RESULTS
We begin by analyzing the performance of our best variant, MetaL, with different outer losses. Then we analyse the overall performance of all methods, followed by dissecting performance along the dimensions of safety coefficient and domain respectively. Finally, we investigate the derived gradient update for MetaL from Theorem and provide intuition for the algorithm's behaviour.
1000
1200
500-
800
Perfermace
600
500
400
400
300
Metal.
Metal (frvser hatue)
Matal.
RC-D4PG RC-D4PG Renabaa
D4PG R
Metal (lettrr Lyphep Lerne)
Revabos
modduped Domain
vo
-200
0.35
01
0.3
Salery Codfiiiont
Figure 1: Metal performance using different outer losses (left) and comparison with D4PG and RC-D4PG (right).
MetaL outer loss: We wanted to determine whether different outer losses would result in improved overall performance. We used the actor loss (Lactor) and the combination of the actor and critic losses as the outer loss (Lactor + Leritse) and compared them with the original MetaL outer loss (Leritic) as well the other baselines. Figure ! shows that using just the actor loss results in the worst performance; while using the critic loss always results better performance. The best performance is achieved by the original critic-only MetaL outer loss.
There is some intuition for choosing critic-only outer loss. In Metal., the critic loss is function of lambda. As a result, the value of lambda affects the agents ability to minimize this loss and therefore learn an accurate value function. In D4PG, an accurate value function (i.e., the critic) is crucial for learning good policy (i.e., the actor). This because the policy relies on an accurate estimate of the value function to learn good actions that maximize the return (see D4PG actor loss). This would explain why adding the actor loss to the outer loss does not have much effect on the final quality of the solution. However, removing the critic loss has a significant effect on the overall solution.
Overall performance: We averaged the performance of Metal. across all safety coefficients, thresh- olds and domains and compared this with the relevant baselines. As seen in Table D MetaL out- performs all of the baseline approaches by achieving the best trade-off of minimizing constraint violations and maximizing retumf] This includes all of the soft constrained optimization baselines (i.e., RS-D4PG variants), D4PG as well as the hard-constrained optimization algorithm RC-D4PG. It interesting analyze this table see that the best reward shaping variants are (1) RS 0.1
which achieves comparable return, but higher overshoot and therefore lower penalized return; (2) RS 1.0 which attains significantly lower return but lower overshoot resulting in lower penalized return. D4PG has the highest return, but this results in significantly higher overshoot. While RC- D4PG attains lower overshoot, it also yields significantly lower overall return. We now investigate this performance in more detail by looking at the performance per safety coefficient and per domain.
Performance as function of safety coefficient: We analyzed the average performance per safety coefficient, while averaging across all domains and thresholds. As seen in Figure [] (right), Metal. achieves comparable average return to that of D4PG, In addition, significantly outperforms both
"MetaL'ss penalized reward (Rpenalized) performance is significantly better than the baselines with all p- values smaller than 10 99 using Welch's t-test.

---

Published as a conference paper at ICLR 2021
Algorithm
Rpenalized
R
max(0, JØ
(8)
D4PG
432.70 + 11.99
927.66
0,49
MetaL
677.93 + 25.78
921.16
0.24
RC-D4PG
478.60 + 89.26
648.42
0.17
RS-0.1
641.41 #
906.76
0.27
RS-1.0
511.70 + 15.50
684.30
0.17
RS-10.0
208.57 + 61.46
385.42
0.18
RS-100.0
118.50 + 62.54
314.93
0.20
Table Overall performance across domains, safety coefficients and thresholds.
D4PG and RC-D4PG terms of penalized return. Figure 2 includes the reward shaping baselines. As can be seen in this figure, choosing different reward shaping value can lead to drastically different performance. This one of the drawbacks of the RS-D4PG variants. Is possible however, to find comparable RS variants (e.g.. RS 0.1 for the lowest safety coefficient of 0.05). However, be in Figure B] for the highest safety coefficient and largest threshold, this RS variant fails completely at the humanoid task, further highlighting the instability of the RS approach. Figure 3 which presents the performance of Metal and the baselines on the highest safety coefficient and largest threshold (to ensure that the constraint task is solvable), shows that MetaL has comparable performance to RC-D4PG (a hard constrained optimization algorithm). This further highlights the power of Metal. whereby it can achieve comparable performance when the constraint task is solvable compared to hard constrained optimization algorithms and fate-offthe-ae performance when the constraint task is not solvable.
Performance per domain: When analyzing the performance per domain, averaging across safety coefficients and constraint thresholds, we found that Metal. has significantly hetter penalized return compared D4PG and RC-D4PG across the domains. A table of the results be seen in the Appendix, Figure 7 Note that, as mentioned previously, the RS-D4PG variants fluctuate drastically in performance across domains.
Peformacc
400
Mecal. R
DC-DAPG It
DMPG Racstsas
RS-01 Booettat
RS-10 Reeallse
RS-10
Reestto
0.66
Safey Conficiint
Figure 2: Performance as function of safety coefficient.
Safety confficion: 0.3. largest only
MD
200
MecaL. ft Meral Spostir
RC DAPG At DCPD4PG Spssttst
D4PG D4PG Racetaii
RS.011
RS
R$ 30
RS:01 Paodtoa
RS-10 Rposttoa
RS-10.0 Rossttis
400
catoos
hanuroid
qushhool
vuleer
Figure 3: Performance per domain. MetaL compared to baselines in terms of average reward and penalized reward across the highest safety coefficient and largest thresholds for each domain.
Algorithm behaviour analysis: Since MetaL is soft-constrained adaptation of RC-D4PG, we next analyze Metal.'s 's gradient update in Theorem understand why the performance of Metal. differs

---

Published as a conference paper at ICLR 2021
from that of RC-D4PG in two types of scenarios: (1) solvable and (2) unsolvable constraint tasks. For both scenarios, we investigate the performance on cartpole for a constraint threshold of 0.115
For (1), we set the safety coefficient to value of 0.3. The learning curve for this converged setting can be seen in Figure H (left). We track 4 different parameters here: the Lagrangian multiplier (red curve), the mean penalty value J (orange curve), the meta-parameter a (black curve) and the scaled Lagrangian learning rate a1 exp((x) (green curve). The threshold 3 is shown as the blue dotted line. Initially there are many constraint violations. This corresponds to a large difference for Jē / (orange curve minus blue dotted line) which appears in the gradient in Theorem 1. As result, the meta-parameter a'A increases in value as seen in the figure, and therefore increases the scaled learning rate to modify the value of À such that an improved solution can be found. Once Jõ satisfying the constraint in expectation (Jă 3 0), the scaled learning rate drops in value due to J - 3 being small. This is an attempt by the algorithm to slow down the change in À since a reasonable solution has been found (see the return for MetaL (green curve) in Figure 4 (right)).
For (2), we set the safety coefficient to value of 0.05 making the constraint task unsolvable in this domain. The learning curves can be seen in Figure E (middle). Even though the constraint task is unsolvable, MetaL still manages to yield reasonable expected return seen in Figure (right). This is compared to RC-D4PG that overfits to satisfying the constraint and, in doing so, results in poor average reward performance. This can be seen in Figure R (middle) where RC-D4PG has lower overshoot than Metal. for low safety coefficients. However, this is at the expense of poor expected return and penalized return performance as seen in Figure E (left). We will now provide some intuition for Metal. performance and relate it to the a gradient update.
In this setting, there are consistent constraint violations leading to a large value for JZ - 3. At this point an interesting effect occurs. The value of oty decreases, as seen in the figure, while it tries to adapt the value of À to satisfy the constraint. However, as seen in the gradient update. there is an exponential term exp(ax) which scales the Lagrange multiplier learning rate. This quickly drives the gradient down to 0, and consequently the scaled Lagrange multiplier learning rate too, seen in Figure | (middle). This causes À to settle on a value as seen in the figure. At this point the algorithm optimizes for stable fixed À and as result finds the best trade-off for expected return at this A value. In summary, MetaL will maximize the expected return for an 'almost' fixed A, whereas RC-D4PG will attempt to overfit to satisfying the constraint resulting in a poor overall solution.
5itt Orehoer
d1 eex(aau
01 exp(a))
harty 200 NO DAROO cooed
1281 300s
2012
2010
1928 2013 LeaTee 2013
1313 2000 1986 20le 20113 2012
Selte siles 10l2e
3490
4530
4590
4090
10000
Figure 4: The learning progress of Metal. for solvable (left) and unsolvable (middle) constraint tasks. In both cases, MetaL attempts to try and maximize the return (right).
DISCUSSION
In this paper, presented a soft-constrained RL technique called MetaL that combines meta- gradients and constrained RL to find good trade-off between minimizing constraint violations and maximizing returns. This approach (1) matches the return and constraint performance of a hard- constrained optimization algorithm (RC-D4PG) on "solvable constraint tasks"; and (2) obtains an improved trade-off between maximizing return and minimizing constraint overshoot on "unsolvable constraint tasks" compared to the baselines. (This includes a hard-constrainel RL algorithm where the return simply collapses in such case). MetaL achieves this by adapting the learning rate for the Lagrange multiplier update. This acts as proxy for adapting the lagrangian multiplier. By amplify- ing/dampening the gradient updates to the lagrangian during training, the agent is able influence the tradeoff between maximizing return and satisfying the constraints to yield the behavior of (1) and (2). We also implemented a meta-gradient approach called MeSh that scales and offsets the
'This threshold chosen varying the safety coefficient this threshold yields both solvable and solvahle constraint tasks which is important for our analysis.
8

---

Published as a conference paper at ICLR 2021
shaped rewards. This approach did not outperform MetaL but is direction of future work. The algorithm, derived meta- eeta--gradeet update and a comparison to Metal can be found in the Appendix, Section I. We show that across safety coefficients, domains and constraint thresholds, MetaL out- performs all of the baseline algorithms. We also derive the meta-gradient updates for Metal, and perform an investigative study where we provide empirical intuition for the derived gradient update that helps explain this mneaa-gradie variant's performance. We believe the proposed techniques will generalize to other policy gradient algorithms but leave this for future work.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin A. Riedmiller. Relative entropy regularized policy iteration. CoRR, Abs/1812.02256 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 22-31. JMLR. org, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.0866 00111800408617, 2018.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volune 70, pp.
JMLR. org. 2017.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Had- sell. Value constrained model-free continuous control. arXiv preprint arXiv:1902.04623, 2019.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman Guzman, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning, 2018.
Gabriel Dulac-Arold, Daniel J. Mankowitz, and Todd Hester. Challenges of real-world reinforce- ment learning. CORR, abs/1904, 904. 12901, 2019.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing. arXiv preprint arXiv:2003. /1000000///888// 1881, 2020a.
Gabriel Dulac-Aroold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing, 2020b.
Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-explootaaion in constrained mdps, 2020.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization, 2017.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.00 550.:02799. 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra. Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(754)):529 533, 2015.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained rein- forcement learning has zero duality gap. In Advances in Neural Information Processing Systems, pp. 7555-7565, 2019.

---

Published as a conference paper at ICLR 2021
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv: 2019.
Harsh Satija, Philip Amortila, and Joelle Pineau. Constrained markov decision processes via back- ward value functions. arXiv preprint 0rXiv:2008.118/1 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550, 2017.
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22 (2):276-295, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillierap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1801 801.00690, 2018.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor, A deep hierar- chical approach to lifelong learning in minecraft. In AAAI, volume 3, pp. 6, 2017.
Chen Tessler, Daniel J Mankowitz. and Shie Mannor. Reward constrained policy optimization. arXiv preprint 1r.iv:1805. 11074, 2018.
Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, and Emma Brunskill. On ensuring that intelligent machines are well-behaved. arXiv preprint 10XX::7008.054440 2017.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of usefull questions auxiliary tasks. NeurlPS, 2019.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-Gradient Reinforcement Learning. NeurlPS, 2018.
Kenny Young, Baoxiang Wang, and Matthew E. Taylor. Metatrace actor-critic: Online step-size tuning by meta-gradient descent for reinforcement learning control, 2018.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado Hasselt, David Silver, and Satinder Singh. Self-Tuning Deep Reinforcement Learning. 2020.
Ruiyi Zhang, Tong Yu, Yilin Shen, Hongxia Jin, Changyou Chen, and Lawrence Carin. Reward constrained interactive recommendation with natural language feedback, 2020.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. NeurIPS, 2018
10


Reviews:
[{"review_id": "eN-Ak1uomP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents two soft-constrained rl approaches built on top of D4PG. Specifically, they use meta gradients for the lagrange multiplier learning rate (MetaL), and use meta gradients for reward shaping (MeSH).\n\nI found the paper to very clearly written, the main algorithm MetaL is clearly presented, and the results are fairly conclusive: the meta gradient approach proposed in the paper works better than the tested baselines. The introduction motivates the different real-world problems very nicely, e.g., hard vs. soft constraints. As someone with a lot of deep RL experience, but not a lot of constrained RL experience, I found the authors did a very good job at explaining all the relevant background. I also really appreciated the detailed experimental analysis of the approach at the end of 6.1 – it highlights exactly why the method works well. \n\nOne critique I do have, is that it would be great to have the intuition for the MeSH update in the main paper. Otherwise, since the results indicate that it performs worse than MetaL across the board – perhaps it would be best to relegate the method to the appendix and change the presentation of the paper, e.g., change “we propose two meta gradient methods”  etc… I feel like the paper would be stronger without MeSH as it takes away from the overall message. \n\nAnother issue I have is that 3 seeds might not be enough to make any meaningful conclusions. More seeds would be needed to make any statistical comparison between metal and Rs-0.1 in table 1. Nevertheless, as highlighted RS-0.1 fails at humanoid – which makes sense – you would need to tune the penalty parameter for each domain (although 0.1 actually works quite well on ¾ domains). Which is why meta gradient approaches make sense. \n\nOverall, assuming the authors add extra seeds and perform the statistical significance testing – I think this paper would have a large impact at ICLR.\n\nSmall notes:\n\nOn the presentation side of the results, I find figures 2 and 3 to be hard to interpret at a glance. It would be better to compare against the best baseline – instead of ~6 of them. \n\nOne final note, \\bar \\lambda = 1000 corresponds to the upper bound of the reward. Unless the algos are consistently achieving this upper bound, \\bar \\lambda seems very high. Ideally, a bunch of lambdas should be tested. This feels arbitrary.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bYSLbn36EUi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThe paper focuses on soft-constrained RL techniques and proposes a meta-gradient approach for the same. It first extends the RCPO (Tessler et al)  algorithm using the methodology of DDPG (Lillicarp et al) to propose an off-policy version of RCPO (called RC-D4PG). The main contribution of the work is the proposal of two new meta-gradients based algorithms for the soft-constrained RL problem that are able to find a good trade-off between constraint violation and maximizing returns. The first proposed algorithm - Meta-L - is based on a meta-learning based adaptive update rule for the Lagrange multiplier's learning rate. The second algorithm is based on similar principles but instead focuses on adapting the reward-shaping update in a meta manner. The author's show the empirical evidence of their method's strengths on a bunch of continuous control based simulator tasks. \n\n#### Strengths:\n\n- **Setting**: The paper shifts focus on the soft-constrained RL (a relaxed CMDP) setting and provide good motivation to work on these setting rather than sticking to the much harder CMDP setting. This makes sense especially for the choice of problems the authors are interested in such as saving energy costs, find some trade-off between two metrics, etc.\n\n- **Method**: One of the biggest problems with the Lagrange Optimization-based CMDP algorithms is that the optimization of the Lagrange multiplier is tricky (and most times very brittle). I like the approach of the authors that aims to adaptively modify the rate of change of the Lagrange multiplier, and I believe that it is a step forward in the right direction.\n\n- **Empirical results**: The authors have strong empirical results for their methods, and it seems that the meta-gradient based approach is able to find the trade-off for the more successfully when compared to the baselines. \n\n- **Analysis**: The empirical analysis of the behavior of the proposed algorithms has been insightful as a reader. I appreciate the extensive experiments and particularly enjoyed the section Algorithm behavior analysis.\n \n\n#### Weakness:\n \n- **Reproducibitlity**: There is no mention of code release. Though, they provide the architectural details in the appendix it is difficult to take the author's claim at face-value when the implementations of the Deep-RL-based methods aren't released. \n\n- **Outer-losses**: The authors propose and test a variety of outer-loss and pick the best one here. I can see why the authors chose this approach, however, from an outsider's perspective, this means for any new application where this algorithm needs to be deployed, a lot of effort and computing needs to go first in finding the appropriate outer loss (on top of hyper-parameter optimization!). It'll be great to have some insights into the choice of outer-loss, other than the hyper-parameter based search that is currently being used.  \n\n- **Solution quality**: No comments have ever been mentioned about the quality of the solution reached. It'll be nice to have some guarantees associated with the solution quality in this case as the setting is motivated by safety and constraint violation.  \n\n\n#### Questions for rebuttal:\n\n- Why is the Meta-parameter update for MetaL (in Algorithm 1, Line 17) different from the derived gradient update in Theorem 1 ?\n- Does updating the Lagrange multiplier adaptively makes the initial learning rate selection process more cumbersome or fragile? \n- Why did the authors choose the off-policy extension of RCPO to build upon, when the original RCPO was purely online? I'm curious to know what breaks (if at all anything breaks) with the original online formulation.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KReeDizK5A", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple approach to soft-constrained deep RL optimization using the unconstrained Lagrangian, by meta-learning the learning rate of the Lagrange multiplier. The authors include extensive evaluation, comparisons and ablations on 4 mujoco domains that establish the efficacy of the approach and provide insight into why it works better. \n\nPros:\n1. Proposes a solution to the important problem of soft constraint optimization in deep RL, which is a challenge to be addressed for real world deployment. This work casts the unconstrained Lagrangian problem in the meta-learning framework, where the inner update adapts the learning rate for the Lagrange multiplier, and the outer loop optimizes the overall loss which takes into account both reward and constraint penalty. This adaptable learning rate prevents the optimizer from fitting too soon to the constraints and enables it to find a solution that can give higher overall performance (sum of reward and penalty). \n\n2. The extensive evaluation includes comparison to prior approaches for constraint optimization in deep RL (soft-constraint with fixed Lagrangian multipliers and hard-constraint with learnable multipliers), tests with varying degree of constraint violations in the environment, ablations with variants of the outer loss across 4 mujoco domains in the Real-world RL suite. The authors also include an analysis of how the and Lagrange multiplier and its learning rate evolve in a setting with high number of safety violations, to show the benefit of using the adaptable learning rate.\n\n\nCons : \n1. The idea of adapting the learning rate in the inner loop of a meta-learning algorithm is not novel, and is often used to enhance performance. With that said, this work convincingly shows that using this simple idea is effective for constrained optimization in the meta-learning framework. \n\n2. The paper introduces another variant of their main approach, but doesn't discuss it adequately in the main paper, and it's unclear why it does worse than the proposed approach.\n\n3. The paper could be further strengthened by experiments on real world domains, where soft-constrained optimization is a critical challenge.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GUAqlaLS4PU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the soft constraints problem in RL. The problem is formulated as a Lagrangian optimization following Tesslaer et al. (2018) where the constraint is treated as a penalty in the reward. The base solution to the Lagrangian optimization is D4PG. \nTo adapt the learning rate of the Lagrangian multiplier and find a good trade-off between reward and penalty, the paper customizes the usage of meta-gradient method (Xu et al., 2018) to the problem in this paper. Two variants, MetaL and MeSH are designed.\n\nThe problem being addressed is an important topic in RL, the method proposed seems to be novel (but the intuition behind it is not clear), while the empirical evaluation is not convincing to me.\n\nStrengths:\n1) Addressing an important problem in RL, potentially could draw attention to the RL community\n2) The paper provides a good motivation of addressing soft constraints in RL\n3) Proposed method is somewhat novel\n\nWeaknesses:\n1) In the paper the motivation of using meta-gradient to solve the formulated Lagrangian optimization is only explained once at the beginning of Page 4 \"Our intuition is that a learning rate gradient that takes into account the overall task objective and constraint thresholds will lead to improved overall performance.\" However it is clear to me what explicitly do you want to achieve? Are you trying to find the \"ground truth\" $\\bar{\\lambda}$ (i.e., 1000 in the experiments)? Does not seem to be the case; Do you want to somehow learn a \"robust\" policy that works well for all $\\bar{\\lambda}$ values? If so it is expected that the authors would show the empirical results on different $\\bar{\\lambda}$ values, and the proposed method works well in all of them.\n2) Following 1), the paper evaluates different forms of outer loss and show that using a critic-only outer loss yields the best performance. What is the intuition behind this? It would be good that the authors can have a more in-depth discussion on this.\n3) In the empirical evaluations, the paper defines a \"penalized return\". This is very un-intuitive -- in reality there is not always such quantized trade-off between reward and penalty of constraint violation. If there is, then one could directly add that to the objective. It would be more interesting to see, for example, given the same reward value, the proposed method always outperforms the baselines in penalty; or the other way round. The \"penalized return\" metric is therefore un-convincing to me.\n4) Assuming the penalized return metric makes sense. From Figure 3 it appears that the performances of the baselines sometimes are very close to the proposed method. What puzzles me is that the performances of the baselines vary dramatically across domains. Can the authors elaborate more on why this happens?\n\nQuestions: See weaknesses points 1) 2) and 4)\n\nLess important points:\n1) In table 1 it seems that overall performance of RS-D4PG monotonically increases w.r.t. $\\lambda$ values. I am curious to see what happens when $\\lambda$ is even smaller.\n2) Page 3, Line 2, $J_{obj}^{\\pi}(\\theta)$ -> $\\tau$ and $\\eta$ are missing in the bracket\n3) Line 4 at paragraph D4PG: $Q_T(s, ...)$ -> s'\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks at the soft-constrained RL techniques and proposes a meta-gradient approach.
One of the biggest problems with the Lagrange Optimization-based CMDP algorithms is that the optimization of the Lagrange multiplier is tricky
The proposed solution and empirical results have promise. The reviewers broadly agree on their evaluation and the major concerns on comprehension, additional experiments and as well as comparison with baselines have been addressed in the rebuttal. 

- Convergence rate and quality of fixed point reached.
The authors mention convergence to local optima but omit the quality of this solution from perspective of safety. It would be useful to include a discussion on the topic, with potential references to concurrent work. 
Other relevant and concurrent papers to potentially take note of:
- Risk-Averse Offline Reinforcement Learning (https://openreview.net/forum?id=TBIzh9b5eaz)
- Distributional Reinforcement Learning for Risk-Sensitive Policies (https://openreview.net/forum?id=19drPzGV691) 
- Conservative Safety Critics for Exploration (https://openreview.net/forum?id=iaO86DUuKi)

I would recommend acceptance of the paper based on empirical results, conditional on release of sufficiently documented and easy to use implementation. 
Given the fact that the main argument is empirical utility of the method, it would be limit the impact of this work if readers cannot readily build on this method. 
Decision: Accept (Poster)