Explicit Pareto Front Optimization for Constrained Reinforcement Learning
Authors: Sandy Huang
Abbas Abdolmaleki
Philemon Brakel
Steven Bohez
Nicolas Heess
Martin Riedmiller
raia hadsell
Venue:  (2021)
Keywords: constrained reinforcement learning
multi-objective reinforcement learning
continuous control
deep reinforcement learning

Abstract: Many real-world problems require that reinforcement learning (RL) agents learn policies that not only maximize a scalar reward, but do so while meeting constraints, such as remaining below an energy consumption threshold. Typical approaches for solving constrained RL problems rely on Lagrangian relaxation, but these suffer from several limitations. We draw a connection between multi-objective RL and constrained RL, based on the key insight that the constraint-satisfying optimal policy must be Pareto optimal. This leads to a novel, multi-objective perspective for constrained RL. We propose a framework that uses a multi-objective RL algorithm to find a Pareto front of policies that trades off between the reward and constraint(s), and simultaneously searches along this front for constraint-satisfying policies. We show that in practice, an instantiation of our framework outperforms existing approaches on several challenging continuous control domains, both in terms of solution quality and sample efficiency, and enables flexibility in recovering a portion of the Pareto front rather than a single constraint-satisfying policy.

Under review as a conference paper at ICLR 2021
EXPLICIT PARETO FRONT OPTIMIZATION FOR CONSTRAINED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many real-world problems require that reinforcement learning (RL) agents learn policies that not only maximize a scalar reward, but do so while meeting constraints, such as remaining below an energy consumption threshold. Typical approaches for solving constrained RL problems rely on Lagrangian relaxation, but these suffer from several limitations. We draw a connection between multi-objective RL and constrained RL. based on the key insight that the constraint-satisfyiin optimal policy must be Pareto optimal. This leads to novel, multi-objective perspective for constrained RL. We propose framework that uses a multi-ohjective RI. algorithm to find a Pareto front of policies that trades off between the reward and constraint(s), and simultancously searches along this front for constraint-satislying policies. We show that in practice, an instantiation of our framework outperforms existing approaches several challenging continuous control domains, both in terms of solution quality and sample efficiency, and enables flexibility in recovering a portion of the Pareto front rather than single constraint-satisfiing policy.
INTRODUCTION
Deep reinforcement learning (RL) has shown great potential for training policies that optimize single scalar reward. Recent approaches have exceeded human-level performance on Atari (Mnih et al. 2015) and Go (Silver et al., 2016), and have also achieved impressive results in continuous control tasks. including robot locomotion (Lillicrap et al. 2016 Schhulmnn et al., (017)), acrobatics (Peng
et al. 2018), and real-world robot manipulation (Levine et al. 2016 Zeng et al. 2019).
However, many problems, especially in the real world, require that policies meet certain constraints. For instance, we might want factory robot to optimize task throughput while keeping actuator forces below a threshold, to limit wear-and-tear. Or, we might want to minimize energy usage for cooling a data center while ensuring that temperatures remain below some level (Lazic et al. 2018).
Such problems are often encoded as constrained Markov Decision Processes (CMDPs) (Altman 1999). where the goal is to maximize task return while meeting the constraint(s). Typical approaches for solving CMDPs use Lagrangian relaxation (Bertsekas) 1999) to transform the constrained opti- mization problem into unconstrained one. However, existing Lagrangian- based approaches suffer from several limitations. First, because the relaxed objective is a weighted-sun of the task return and constraint violation, this assumes convex Pareto front (Das Dennis 1997). In addition, when the constraint is difficult to satisfy, in practice policies can struggle to obtain task reward. Finally, such approaches typically produce a single policy that satisfies specific constraint threshold. However, the exact constraint threshold may not be known in advance, or one may prefer to choose from set of policies across range of acceptable thresholds.
We aim to achieve the goal of CMDPs (i.c., finding a constraint-satise satissyiiiiiiitga policy that maximizes task return) while avoiding these limitations, by introducing a novel, general framework based on multi-objective MDPs (MO-MDP). A MO-MDP can be seen as a CMDP where the constrained objectives are instead unconstrained. Our key insight is that if we have access to the Pareto front, then we can find the optimal constraint-satisfyig policy by simply searching along this front. However, finding the entire Pareto front is unnecessary if only relatively small portion of the policies along the Pareto front meet the constraints. Therefore, propose to also simultaneously prioritize learning for the preferences (i.c., trade-offs between reward and cost) that are most likely to produce policies that satisfy the constraints, and thus cover the relevant part of the Pareto front.

---

Under review as a conference paper at ICLR 2021
To our knowledge, there is existing framework for applying multi-objective RL algorithms to constrained RL problems. Our main contribution is a general framework that enables this, by learning which preferences produce constraint- satisiiinn policies. This framework can be combined with any multi-objective RI. algorithm that learns an approximate Pareto front of policies. Our second contribution is to extend a state-of-the-ar multi-objective RI. algorithm, multi-objective maximum posteriori policy optimization (MO-MPO) (Abdolmalcki et al. 2020). to learn the Pareto front in a single training run. We use this extension of MO-MPO within our framework, and call the resulting algorithm constrained MO-MPO. We show in practice, constrained MO-MPO outperforms existing approaches on challenging continuous control tasks with constraints.
2 RELATED WORK
Constrained RL algorithms seek policies that meet the desired constraints at deployment time. Most approaches use Lagrangian relaxation (Bertsekas) 1999). Recent Lagrangian- based approaches claim convergence to constraint-satisfyin (Tessler et al.,
Constrained reinforcement learning.
2019) or optimal (Paternain et al. (019) solutions, but this is debatable (Szepesvari) (020). Recent works seek to stabilize this optimization by approximating the reward and cost functions with convex relaxations (Yu et al. 2019) or by utilizing derivatives of the constraint function (Stooke et al. 2020).
Other works have applied Lagrangian relaxation to mean-value constraints (Tessler et al. 2019).
convex set constraints (Miryoosefi et al. 2019), and local constraint satisfaction (Bohez et al.
2019
Existing Lagrangian approaches involve linear scalarization, however, and thus cannot find solutions that lie on concave portions of the true Pareto front (Das & Dennis 1997). In contrast, we build on multi-objective RL algorithm that does not rely on scalarization, and in practice, our approach can indeed find constraint-satisfiinn solutions
concave Pareto front (see Sec. 5.1 humanoid walk).
Safe reinforcement learning. In safe RL, the aim is to achieve constraint satisfaction not only during deployment, but also during learning (Garefa & Fernández 2015). Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training (Achiam al. 2017 Berkenkamp et al.) 2017 Chow et al. 2018 2019 Yang al. 2020 Zhang et 2020). These approaches require, however, that the initial policy meets (or almost meets) the constraints. Otherwise, performance degrades substantially and constrained policy optimization approaches (Ray
(CPO) (Achiam et al., 2017), for example, performs worse than Lagrangian-base cct al. 2019). The aim of our work is to find better solutions for constrained RL, rather than safe RL. We discuss in Sec. 6 how our method can be extended to reduce constraint violation during training.
Multi-objective reinforcement learning. Our approach is built on ideas from multi-objective RL (MORL), which consists of single-policy and multi multi-policy approaches. Single policy methods learn a policy that is optimal for a given setting of reward preferences. Most rely on linear scalariza- tion (Roijers et al. (013). which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales. Non-linear scalarizations have been proposed (Tesauro et al., 2008 Van Moffaert et al. 2013 Golovin & Zhang 2020), but these are harder to combine with based RL and have seen limited use in deep RL. Recently, Abdolmaleki et al. (020)) introduced MO-MPO, where the preferences represent per-objective constraints on the policy improvement step. MO-MPO does not rely on scalarization and is thus invariant to reward scales.
Multi-policy MORL aims to find set of policies that covers the whole Pareto front. Recent approaches learn a manifold in parameter space that optimizes the hypervolume of the Pareto front (Parisi et al. 2016 2017). While such approaches could be combined with our framework, their scalability to deep RL remains to be shown. Other works combine single policy approaches with a general objective optimize hypervolume (Xu et al. (2020). The instantiation of our framework is similar in spirit to such two-level methods, but applied to different problem setting.
BACKGROUND AND NOTATION
3.1 CONSTRAINED MARKOV DECISION PROCESSES
constrained Markov Decision Process (CMDP) consists of states s E S, actions a E A, an initial
state distribution (s))), transition function p(s'|s [s, a), reward functions {ru(s, a)){ k constraint
2

---

Under review as a conference paper at ICLR 2021
thresholds {cx)E=) and a discount factor [0, 1]. The (K + 1) reward functions consist of task reward To and constrained rewards T1:K: we will refer to these together as objectives.
A policy I (aass) maps from given state to a distribution over actions. The optimal solution to constrained MDP is a policy that maximizes the expected return for the task reward ro. while ensuring that the expected return for all constrained rewards r1:K satisfy their respective thresholds:
max IT
Ex Extro(et, at)
s.t.
Ex Er'ri(se ai] 2 cs = K,
(1)
where E,, is shorthand for the expectation over trajectories when following the policy æ. given fixed initial state distribution. Here the constraint thresholds are lower bounds, without loss of generality.
The value function V (s) of a policy π is its expected return for objective rи when starting from state The action-value value function Q (s, a) denotes the expected return for objective r& after taking action in state 8 and thereafter acting according to the policy: 0% (8, a) ra a) ypx'' IV? (s')]
Most approaches for solving CMDPs are based on Lagrangian relaxation, where the constrained problem is turned into the following unconstrained optimization problem:
K
min max ES X
Σ I'ro(s).
+
Σ
AB
B
Σ ('ru((t) -c))
These approaches alternate between optimizing for the Lagrange multipliers λ and the policy
While our multi-objectivv perspective on constrained RL aims to achieve the goal of WMMDPs-inding a constraint-satisfyig policy that maximizes task returniit does not directly solve ① Nonetheless, we will show our approach outperforms those that directly solve ( via Lagrangian relaxation.
3.2 MULTI-OBJECTIVE REINEORCEMENT LEARNING
In order to achieve the goal of CMDPs, we propose leverage advances in multi-objective RL. A multi-objective MDP (MO-MDP) is defined in the same way as a CMDP, except without constraint thresholds. Because there are multiple unconstrained objectives, there is not single optimal policy. Instead, there is a set of optimal policies, called the Pareto front. front. A policy is Pareto optimal if there is no other policy that improves return for one objective without decreasing return for another.
Preferences. Each policy on the Pareto front is the optimal policy for a particular setting of preferences (i.e., desired trade-off over objectives) This typically encoded via a preference vector in which each element c represents the relative importance of the corresponding objective rk-
Connection CMDPs. Given CMDP (S, A. Po. Po:KK Ci:K, 7 consider the corresponding MO-MDP (S, A. Po.P, To:k; 7.. Regardless of the thresholds CI:K; the optimal policy for the CMDP must lie on the Pareto front for this corresponding MO-MDPP Thus, we knew the Pareto front, we could simply search along it to find the constraint-satiisying policy that maximizes task return.
Multi-objective Maximum a Posteriori Policy Optimization (MO-MPO). Whereas most MORL algorithms are based on linear scalarization, MO-MPO takes distributional approach to training policies. For each policy improvement step, MO-MPO first computes a non non- non-parametric policy for each objective rık that improves the (parametric) policy with respect to that objective, subject to non-negative constraint é, on the KL-divergence between the improved and old policies. Then, the policy updated via supervised learning the sum of these non-parametric policies. Intuitively, E defines the influence of objective F&k on the final policy; an objective with larger 6 has more influence. However, MO-MPO can only train policies for single preference setting e. Sec. I.I explains how we extend MO-MPO to train sefference-conditionee policies π TTaass, E), in order to use in our framework. We choose extend MO-MPO because it does not suffer from the limitations of linear scalarization, and has been shown to outperform such approaches (Abdolmaleki et al.) 2020).
'This assumes that the choice of preference encoding does not restrict optimal solutions to the convex portions of the Pareto front. Although linear scalarization does not satisfy this requirement, other kinds of preference encodings do, for instance Chehyshev scalarization (Van Moffaert al. 2013) or the per-ohjective Kl.-divergence constraints in MO-MPO, described later in this section.
2This is because if the optimal policy for the CMDP is not Pareto optimal, then there must exist another policy that obtains higher task return while meeting the constraints, which leads contradiction.
3

---

Under review as a conference paper at ICLR 2021
4 APPROACH
Problem Statement. For a given CMDP, we seek to find the preference vectors e in the correspond- ing MO-MDP that produce constraint-satiisying action policies ((lss, e). To do this, we want to find the distribution x (e) that maximizes the probability that the action policies satisfy the constraints 3
Overview of Framework. Our framework enables applying multi- RL algorithms to solve constrained RL problems. is inspired by the observation that for any CMDP, there exist preference settings for the objectives that lead the desired constraint-satiifying solutions. We propose find these preference settings by optimizing for a hierarchical policy o (16))))) ( [a|s, e) that first selects a preference setting e and then takes an action conditioned e. We also maintain a policy evaluation function Q (a, s, e) for each objective.
To train this policy, we decouple learning the preference policy (E) from learning the action policy Ta(a|s, e). Formally, we alternate between optimizing two sub-problems:
1. Learn a new action policy Fnow E) given the current distribution of preferences (E).
2. Learn a new preference policy Tnow (E) given the current action policy [ota(a)s, e).
We aim to converge to an optimal policy π (als)) E) that not only satisfies the constraints but is also Pareto optimal. Note that we can use different time scales for alternating between Steps and 2. One extreme to first fully optimize for a Pareto front curve given an initial preference distribution, and then fully optimize for solutions that satisfy the constraints. In our experiments we take the other extreme: we alternate between taking one learning step on (e( and mh (a(a E) independently.
For Step 1, any multi-ohjective RI algorithm capable of training wreference-condition be used. Sec. descrribs how we use MO-MPO for Step 1, which requires non-trivially extending it to train peeference-oonditoone policies. For Step 2, the notion of learning preference policy is the core novel component of our framework: this is what enables us to apply multi-objective RL algorithms to constrained RL problems. Sec. describes how to do this.
policies can
4.1 LEARNING
ACTION POLICIES (STEP 1)
In this section, we explain how we extend MO-MPO Abdolmaleki et al. 2020) to learn preference- conditioned policies for the current preference distribution (c))
MO-MPO can only train policies π (als) for a single preference c, so it cannot be directly used. This section describes how extend MO-MPO to train single policy Ta (als. conditioned on preference parameters 6 ~ (E) that represents the entire Pareto front. This is a non-trivial extension, that requires making both the action policy and Q-functions preference-onndttiond, and replacing the scalar temperature with a ceeference-conditions modifying the underlying MO-MPO optimization principles. We also introduce hindsight relabeling of preferences to stabilize off- off-policy learning and improve sample efficiency.
temperature network, as well as
Our extended version is policy iteration algorithm with two steps:
Policy evaluation: learn preference-conditionee O-functions for all objectives.
Policy improvement: improve the defference-conditionee policy according to Q-functions.
Preference-condiitoone policy evaluation. In this step, we train a separate Q-function per objective (coined Q-decomposition by Russell & Zimdars 2003), evaluate the current policy <) under the state distribution (u(s). To learn the Q-function, for each learning step we sample L. transitions from the replay buffer {s;, a, (rd K s Since our Q-functions Q 8, e) are also conditioned on preferences, we use hindsight relabeling to augment the states with preference parameters sampled from the current preference policy 6; ~ Tala (E), resulting in transitions {[s,, e,), a, {r. [s/, e.],. Any policy evaluation algorithm can be used to learn these Q-functions. We use distributional policy evaluation with 5-step return (Barth- (Barth-Maron et al., 2018).
peeference-conditione policy improvement. The policy improvement step assumes state distribution u(s), per-objective Q functions (s, a, <), the current action policy Tala(a)s (a)s, c), and the current preference policy (E). It consists of two sub-steps: 1) finding per-objective improved
Without loss of generality, we consider state-independent preference policy ((E) when deriving the update nules. In Sec. 5.3 we show our approach also be used to learn state-dependent preference policies (Elss.
4

---

Under review as a conference paper at ICLR 2021
action distributions, and 2) distilling these distributions into preference-condtiionee action policy via supervised learning.
Finding per-objective distributions: To find per-objective improved action distributions 9u (a e). we optimize the following RL optimization problem for each objective:
max Eman(e)a(s) I (alas, E) Q% (s, a, e) da
(3)
S.L. Be()) [[L maa e))
< ck Ven mou(e.
We solve this problem in closed form to obtain
2h a,E) ,
gu [als, e) x Tota E) exp
(4)
Tho Tww (Ex)
where Bus (Ex) is a preference-dependen temperature function for objective TR, and parameterized by wk. This temperature fuunction is obtained by minimizing the following dual function:
Q2 (s, a,,)
g(we) = E ( Mwn (ex) (ex + log Tata(als, E) exp
da) dd)].
(5)
Mu, (Ex)
In practice, we maintain a single function nh (E) with shared parameters for all objectives
To approximate the expectations over the preference distribution and state distribution, we draw states from the replay buffer and L preferences from the preference policy Toia To approximate the integrals over a, for each (s, <) pair we sample M actions from the current policy (ala, e).
Learning new parameterized action policy: After obtaining per-objective improved policies, use supervised learning to distill these distributions into new parameterized policy:
max
Emaa(s)p(s)) (aa(la Ta] s.t. (T)a()))(s) (To (a s, e) TT (a|s, (al,s,)))))))))) e))] < 3,
subject to trust region with bound / > 0 for more stable learning. To solve this optimization, use Lagrangian relaxation as described Abdolmaleki et al. (2018 2020).
4.2 LEARNING PREFERENCE POLICIES (STEP 2)
We will now explain how to optimize the preference distribution Ty. (T) for selecting action policies (als, E) with better constraint satisfaction. To achieve this, we define a fitness function fa that evaluates satisfaction of constraint threshold Ck2 given the current action policy (a)s e) and Q-function Q a, e). The fitness function should be chosen based on the problem and form of the constraint. In our empirical evaluation, for equality constraints we use fitness function that penalizes the difference between the expected Q-values and constraint threshold, and for inequality constraints it penalizes the amount the expected Q-values fall under the threshold:
[Qgld (a c)]
lk (c) (())))
Ck
(6)
Simer (E) min Ep)()) Qglld (s, (T.((ols,.) e)] -c
Given a chosen fitness function, we now can use any off-the-shelf RI. algorithm to optimize the preference policy (c), initialized by Mc)a(C), to maximize the fitness function. See Appendix for more details on the general procedure underlying our algorithm (Appendix, Algorithm I].
EXPERIMENTS
We find that our approach of Pareto front optimization applied MO-MPO (i.e., constrained MO- MPO), finds solutions that are on par with those found by MORL algorithms (Appendix C.T.) In
'The optimization problem in 3 generalizes the one in MO-MPO, which assumes fixed KL-constraints to a distribution (E over KI. cln-oontrass Thus 4 and can be derived by following steps analogous to those given in Abdolmaleki al. 2020
5

---

Under review as a conference paper at ICLR 2021
our experiments, we compare constrained MO-MPO against existing RL algorithms for constrained MDPs: for a number of challenging continuous control tasks, we analyze the quality of policies found for range of constraint thresholds. Unless otherwise mentioned, use ( as the fitness function. For action policies we use Gaussian distributions parameterized by neural networks, and for the preference policy in constrained MO-MPO, we use discrete distribution. Architecture details and learning hyperparameters are described in Appendix A
Domains. We evaluate our approach in two continuous control domains. First, we use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al. 2018). This domain has a challenging 21 -i-dimensional action space. The agent receives shaped reward for running at 10 m/s or walking at 1 m/s. The constraint is imposed on the expected negative control norm (i.e., - -I|all2), which roughly captures the "energy" expended by the agent. This is relevant for real-world settings (e.g., in robotics or control tasks), where energy consumption often needs to be taken into account.
We also evaluate on the level-two point mass tasks from the Safety Gym suite (Ray et al. [2019). which we refer to as point goal, button, and push. The agent receives sparse reward for either reaching a goal location or pushing a box to a goal. The constraint is imposed on the expected per-episode cost, which is incurred by running into or over objects. There are several types of objects, each with its own cost. Unless otherwise mentioned, the constraint is with respect to the total cost.
Pareto plots. When we plot learned Pareto fronts for the tasks (e.g., in Fig. [ top), the x-coordinate is average per-episode negative cumulative cost and the y-coordinate is average per-episode task reward For both axes, higher values are better. All trained policies for the same task are evaluated on the same set of randomly-initinn initialized environments. For constrained MO-MPO policies, we sample new preference at the beginning of each episode, rather than at every timestep.
Baselines. As a baseline we consider Lagrangian relaxation; because of its simplicity and effective- ness it is common choice (Ray et al. 2019 Stooke et al., (220)). To ensure a fair comparison with our approach, we use MPO to train policies to optimize the Lagrangian dual objective (2). We obtain state-of-the-ar results on Safety Gym tasks with this MPO-Lagrangian bascline (Appendix C.2
5.1 QUALITY OF SOLUTIONS
We evaluated our approach and the baseline for constraint thresholds linearly spaced in the range 1-4-11 for humanoid run, -0.5] for humanoid walk, and [-15, -2] for point tasks. For constrained MO-MPO, the preference policy π (TE) is initialized to uniform and is a discrete distribution over 100 enearly-spaced values starting at 10-3 and up 0.15 for humanoid run, 0.3 for humanoid walk, and 0.2 for the point tasks. Results are shown in Fig. [
We observe the biggest difference in performance for humanoid walk: the Lagrangian baseline only finds policies at the extremes, whereas our approach finds constraint-satiffyan
solutions that achieve non-zero task reward. This may be because the ground-truth truth Pareto front for this task is concave (Sec.
2). On humanoid run, point goal, and point button, the baseline and our approach find similar quality solutions for easier constraint thresholds, but as the constraint threshold becomes more difficult to
satisfy (i.e., smaller in magnitude), our approach dominates (Fig. [ top row; Fig. 2ap. The only exception is point push, for which both approaches find similar-quality solutions for all constraint thresholds. We hypothesize that this is because in this particular task, the cost does not conflict much with the task objective, so is easier to find high-task-
solutions that incur small cost.
Our approach always finds policies that satisfy the constraints for humanoid, whereas the Lagrangian baseline occasionally violates the constraint For Safety Gym tasks, the policies found by both our approach and the baseline slightly violate the constraints (Fig. []] bottom row); this is due to Q-function underestimates for the cost, rather than a limitation of the policy optimization (Appendix C.4).
"The action norm constraint for the humanoid tasks is pir-timestep with 1000 timesteps per episode, so a policy that exactly meets constraint threshold of -2, has an average cumulative episodic cost of -2000. "With longer training, these violations might disappear, but we do not expect the task reward to improve. These violations are due to the relatively unstable behavior of the Lagrangian baseline: when the constraint starts to be violated, this leads to gradual increase in the Lagrange multiplier, and it takes some time for this to propagate into changes in the policy. This can he seen empirically in Fig. 2b (top row) where the average cost dips helow (i.e., violates) the constraint threshold, indicated by the dotted line.

---

Under review as a conference paper at ICLR 2021
hnmaazoid walk
hamanoid ron
point goai
point butioon
point: pasi
1000
15-
Lask reward
took rewaxd
10
500
01-
1000
-10010
1000
15-
task remand
Luck reward
10-
0-
1000
-2000
20
-2
-00
worm
total
total
total cost
MPO Lagrangian
constzained MO-MPO
MPO Lagrangian
constrained MO-MPO
Iramaood walk
hurasoodd run
point goa)
point button
point pash
0-
uctund cood
actand
-2
-10-
-20
-2 cost threshoold
2
-20
-20 cost threshold
-20
cOST: chreshold
cost threshold
cost threshoki
Figure 1: Top row: For harder-to-satise statisfy (i.e., lower magnitude) constraint thresholds, constrained MO-MPO finds policies with higher task performance than those found by the Lagrangian baseline, in all tasks except point push. Middle row: Across all constraint thresholds, constrained MO-MPO performs least well the baseline. The plots in the top and middle row are Pareto plots: for both axes, higher values are better. Each dot corresponds to a separate policy trained for particular constraint threshold. Bottom row: Constrained MO-MPO and MPO-Lagrangian perform comparably in learning policies that satisfy the constraints. Policies that lie in the red region (below the dotted line) violate the constraint.
Our approach is also significantly more sample-efficient than the baseline. For instance humanoid run, constrained MO-MPO learns constraint-satiffyne reward mid-way through training, whereas the majority of MPO-Lagrangian policies obtain zero task reward at the same point (Fig. Bal Bb). This could be because with two conflicting objectives, MPO-Lagrangian cannot optimize for both objectives once, whereas our approach can (Fig. 2b).
policies that reach reasonably high task
Multiple random seeds. We additionally ran experiments with five random policy initializations per constraint threshold, for a subset of four thresholds. The results support the conclusions drawn above. In particular, our approach significantly outperforms the baseline for the harder-to-satise constraint thresholds, in four out of the five tasks (see Appendix C.3 for details).
Scaling to multiple constraints. We also investigated how well constrained MO-MPO scales to multiple constraints, for a more difficult version of point goal (see Appendix [.. for details). There are two constraints: one on the cost of crossing over hazards, and one on bumping into vases. For each type of cost, we fix its threshold at 22 and vary the other between -10 and O. The baseline fails to obtain task reward when the constraint for bazards is fixed at the challenging threshold of -2, whereas constrained MO-MPO finds a range of solutions (Fig. 3c right).
5.2 FLEXIBILITY
Since our approach decouples Pareto front optimization from Pareto exploration, by choosing the appropriate fitness function, one can in theory recover any portion of the Pareto front. This is useful because in practice, RL practitioner may not know the true constraint, and will be better able to
7

---

Under review as a conference paper at ICLR 2021
rel threshold 3.4 cos threshold 2.02 cel threshodd 1.0
1000
taek resoard inprovenent
200
890 1uat return
-200
Dong action TOCI
1000
-2500
constrainl thrmiiod
0.0
0,50.0
0,50.0
0.5
(a)
x10P
stops, x10P
x10"
(b)
Figure 2: Plots are for humanoid run. (a) The difference in average reward obtained by constrained MO-MPO versus MPO- Lagrangian (positive values mean constrained MO-MPO performs better), across sub-ranges of constraint thresholds and with standard error bars. Our approach significantly outperforms the baseline for more difficult (lower magnitude) constraint thresholds. (b) Constrained MO-MPO (bottom) is able to optimize for both objectives at once, whereas the baseline (top) alternates between the two, leading large drops in action norm cost. The horizontal dashed lines indicate the cost threshold.
Intaamood run, mid-teaining
homanoid ran, mid-training
Exxed threshold fixed hezards threshokd
1000
task reward
task zoward
3000
uetual
0-
0-
2000 action aorm etst.
-20
cott threshole
total cost
total
(b)
Figure 3: (a) Pareto fronts and (b) constraint violations for humanoid from the middle of training. (c) Pareto fronts for point goal with two constraints. Refer to Fig. for interpretation of colors.
choose which policy deploy after observing couple of options along the Pareto front. Typical approaches to constrained RL find a single policy that meets the constraint exactly
On humanoid run, evaluate constrained MO-MPO using (7) the fitness function for several inequality constraint thresholds. We find that our approach indeed learns a portion of the Pareto front that satisfies the constraint (Fig. 4ab. These plots are obtained by taking the preferences regular percentiles between 5th and 95th percentiles the learned preference distribution, and evaluating the policy conditioned on each of these preferences, on the same set of handomly-initiaizzee environments.
5.3 STATE-DEPENDENT PREFERENCES
By using a state-dependent preference policy x (c. s). we can extend our approach to satisfy constraints per-initial-state rather than in expectation To do this, we train neural network that maps the state to discrete distribution over preference settings. We implement this for the MPO-Lagrangian baseline by making the Lagrange multiplier state-dependent as well, as introduced by Bohezz et al. (2019); this trains policies that satisfy the constraint per-state-ther is equivalent for the Lagrangian approach that satisfies constraints per-initial-stane
We use both approaches to train policies for humanoid run, for the same settings of constraint thresholds as Sec. 5.1 Our approach finds constraint-satiifyinng policies that achieve higher task reward than the baseline (Fig. 4b). The constraint violation is computed by summing the per-episode
A Although it is possible extend the Lagrangian baseline hy conditioning the policy the threshold, given that constrained MO-MPO finds hetter solutions, we choose to focus evaluation here on constrained MO-MPO. IIn our implementation, sample preference from the preference policy only at the start of every episode, rather than per-timestep. If we did the latter, then we could satisfy per-state constraints.
8

---

Under review as a conference paper at ICLR 2021
humanoid rm
humaraid ruc
1000
1000
Daok rewazd
Luck sowoard
600
300
-2000
2000 action
-000
-2000
25
action
action
action norm cost
action Dorm cost violation
seed sood
sord 2
MPO Lograngian
coustrained MO-MPO
(b)
Figure 4: (a) Pareto plots for constrained MO-MPO on humanoid run for three inequality constraints, with three random initializations for each. The vertical dashed lines indicate the cost threshold. (b) Policies trained by constrained MO-MPO with per-initial-state constraint satisfaction obtain higher task reward, while satisfying the constraints.
constraint violation, obtained by comparing the average per-timestey action norm cost for that episode against the constraint threshold.
6 CONCLUSION AND FUTURE WORK
In this paper, we introduced a novel framework for constrained RL, by leveraging the ability of multi- objective RL algorithms to find Pareto-optimal solutions. This framework can be combined with any algorithm that finds a Pareto front of solutions. Our empirical results show that an instantiation of this framework, constrained MO-MPO, outperforms the commonly Lagrangian relaxation approach in terms of solution quality, stability, and sample-efficiency. In particular, the Lagrangian approach struggles to train policies that obtain task reward when the constraints are difficult to meet, or when the ground-trute truth Pareto front is concave. Our framework also enables flexibilityy via the choice of fitness function and can be extended to meet constraints per- per-initial-state rather than in expectation.
One limitation of this work is that the support of the preference policy's discrete distribution must include preference setting(s) that satisfy the constraint threshold. In practice, this is straightforward to do for MO-MPO, because the encode preferences in a way that is independent of the objectives' reward scales. In future work, worth exploring other distributions for the preference policy that would overcome this limitation, for instance mixture of Gaussians.
Another limitation this work cannot be applied directly to safe RL. We plan to extend our approach to reduce the cost incurred during training, perhaps by initializing with conservative distribution over preferences, that prioritizes minimizing cost. Finally, we also plan to exploit the flexibility of our framework to solve constrained RL problems with more than one unconstrained objective.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. In Proceedings of the Sixth International Conference on Learning Representations (ICLR), 2018.
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Reidmiller. A distributional view on multi-objective policy optimization. In Proceedings of the 37th International Conference Machine Learning (ICML), 2020.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Eitan Altman. Constrained Markov Decision Processes. CRC Press, 1999. ISBN 9780849303821.

---

Under review as a conference paper at ICLR 2021
Gabriel Barth-Marowe Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference Learning Representations (ICLR), 2018.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems 30, 2017.
Dimitri P. Bertsekas. Nonlinear Programming. Athena Scientific, edition, 1999.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv: 2019.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Lyapunov- based approach safe reinforcement learning. In Advances in Neural Information Processing Systems 31, pp. 8092-8101, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint /r01 2019.
I. Das and E. Dennis. A closer look drawbacks minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural Optimization, 14:63-69, 1997.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Daniel Golovin and Qiuyi Zhang. Random hypervolume scalarizations for provable multi-objective black box optimization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the Third International Conference on Learning Representations (ICLR), 2015.
Nevena Lazic, Craig Boutilier, Tyler Lu, Echern Wong, Binz Roy, MK Ryu, and Greg Imwalle. Data center cooling using model preei-tredietive control. In Advances in Neural Information Processing Systems 31, 2018.
S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
T. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the Fourth International Conference Learning Representations (ICIR), 2016.
Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert Schapire. Reinforce- ment learning with convex constraints. In Advances in Neural Information Processing Systems 32, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller. Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Simone Parisi, Matteo Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through continuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227, 2016.
10

---

Under review as a conference paper at ICLR 2021
Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold-basee multi-objective policy search with sample reuse. Neurocomputiing 263:3-14 14, 2017.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems 32, 2019.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, 37 (4), 2018.
Alex Ray, Joshua Achiam, and Dario Amodei, Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making, Journal of Artificial Intelligence Research, 48(1 48(1))67-111,, 2013.
Stuart Russell and Andrew L. Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning (ICML), pp. 656-663, 2003.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Sutskever, T. Lillicrap, M. Leach, K. Kavukeuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484-503, 2016.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Csaba Szepesvari.
Constrained MDPs and the reward hypothesis. hypotthhssyppot.tss.....o htm
https://readi ngsm
blogspot com/2020/03/ccm/ constt/ccnnsaaaaa mnd.:/anntta.aammmm----aaaaa March 2020. Accessed: 1-October-2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.00640 2018.
Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey Kephart, David Levine, Freeman Rawson, and Charles Lefurgy. Managing power consumption and performance of computing systems using reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1497-1504, 2008.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 191-199 IEEE, 2013.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. erediction-guides multi-objective reinforcement learning for continuous robot control. In Proceed- ings of the 37th International Conference on Machine Learning (ICML), 2020.
Tsung-Ye Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection- based constrained policy optimization. In Proceedings of the Eighth International Conference Learning Representations (ICLR), 2020.
Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. In Advances in Neural Information Processing Systems 32, 2019.

---

Under review as a conference paper at ICLR 2021
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbirrary objects with residual physics. In Proceedings of Robotics: Science and Systems (RSS), 2019.
Yiming Zhang, Quan Vuong, and Keith W. Ross. First order optimization in policy space for constrained deep reinforcement learning, arXiv preprint arXiv:2002.06506, 2020.
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
We use an asynchronous off- offlp actor-learner setup. In this setup, actors fetch policy parameters from the learner and act in the environment, writing transitions to the replay buffer. The learner uses the transitions in the replay buffer to update the policies and Q-functions (and optionally the temperature network and Lagrange multiplier network, depending on the approach). We implement this setup in Acme (Hoffman et al., 2020), a framework for distributed RL. To make learning more stable, we maintain a target network for each trained network; we use these target networks for computing gradients, and update them after every 200 gradient steps. We use 32 actors for training. We use Adam (Kingma Ba, 2015) for optimization, with a learning rate of 10 unless otherwise specified.
All policies and Q-functions are implemented as feed-forward neural networks. Our action policies Te output Gaussian distribution with a diagonal covariance matrix, and our preference policies T output a categorical distribution. We found that adding layer normalization followed by a hyperbolic langent (tanh) to the first layer of the policy and Q-function networks improves stability of learning. For the Safety Gym point mass tasks, we train a separate Q-function per objective, with a shared first hidden layer. For humanoid tasks, the Q-values for action norm cost are instead computed exactly from the actions; thus these Q-values have a discount factor of zero
The default hyperparameters we used for our experiments are reported in Table 1 and task-specific hyperparameters are reported in T In our experiments, we trained policies with constrained MO-MPO and the MPO Lagrangian baseline for range of different constraint thresholds, these thresholds are reported in Table 3 We consider the "harder" constraint thresholds (as plotted in Fig. [ top row) to be the 15 lowest-magnitude thresholds for the humanoid tasks, and the megentt- thresholds for the point mass tasks.
Gathering data. We gather data for the replay buffer via the actors. When the action policy ma is at the start of each episode, each actor first samples preference e' from the preference policy Ty, and then acts according to nalals e') until the end of the episode. For the next episode, the actor repeats this, sampling new preference.
Evaluation. For humanoid tasks, we evaluate each trained policy on the same 100 randomly- initialized environments. For Safety Gym tasks, we do the same on the same 500 randomly-initializz environments. Table reports how many actor steps policies are trained for, before evaluation. We evaluate with deterministic action policy Te by using the mean of the Gaussian distribution over actions. When the action policy is conditioned on preferences, at the beginning of each episode during evaluation, first sample preference from the preference policy, and then execute the mean of To (als,, e), sticking with the same sampled preference for the entire episode. However, when the preference policy is state-depende dependeeend,n,, from ( (E(s), because we find that this leads to better satisfaction of per-episode constraints.
instead condition the action policy on the median preference
B EXPERIMENTAL DOMAINS
B.1 HUMANOID
We use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al. 2018)) The observation space is 67-dimensional and the action space is 21-dimensional. Observations consist of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions. Actions correspond to joint accelerations; the minimum and maximum action limits are 1 and 1, respectively. Each episode is 1000 timesteps.
The unconstrained task reward is given by the environment: there is a shaped reward for maintaining a horizontal speed (in any direction) of 10 m/s for humanoid run and 1 m/s for humanoid walk. The shaped reward is equal min(h/h", 1), where h is the speed of the agent and h is the target speed, both in m/s. The constrained reward is the negative /2-norm of the action vector: r Toos (8, a) = - |a]12- This can be thought of limiting the energy usage of the agent.
Available adgithub..ommdceqmmnddd control
13

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Default
Q function network(s): (s, (x(
layer sizes
(512, 512, 256) |500 150)
support
number of atoms
51
n-step returns discount factor y
0.99
action policy network: (o(ass) or (a(as),
layer sizes minimum variance
(256, 256, 256) 10 12 unbounded
maximum variance
preference policy network: (E) or (e(s))
layer sizes for (E(s)
(256, 256, 256) (256, 256, 256) [0.1. 0.1] [10 - 0.15) 100
layer sizes for temperature network (E)
support for Cusk support for Goost number of atoms
both policy networks and Q-fucttion networks
layer norm first layer?
yes yes
tanh output of layer norm?
activation (after each hidden layer)
ELU
Lagrange multipliers: À or g(Als)
layer sizes for g{{ss)
(256, 256, 256)
softmax on output?
initial Lagrange multiplier À (before softmax)
Adam learning rate
10
MPO MO-MPO for action policy network me actions sampled per state default Fk
0.1
KL-constraint on policy mean, B, KL-sonntraint on policy covariance, 8s
10 10-3
10 10-7
initial temperature
Adam learning rate (for dual variables)
10-2
MPO MO-MPO for preference policy network me actions sampled per state default 6z
0.1
Kl-constraint on policy,
10-
initial temperature
Adam learning rate (for dual variables)
10
training
batch size
512
replay buffer size
10
target network update period
Table Default hyperparameters for all approaches, with decoupled update on mean and covariance of the action policy.
B.2 SAFETY GYM
We the level-two point mass tasks in Safety Gym: goal, button, and push (Ray al. 2019 The action space is 2-dimensional, with minimum and maximum action limits of -1 and 1, respectively. The observation space is 60-dimensional for point goal, and 76- 76-dimensional for point button and push. There are 1000 timesteps per episode. At the start of each episode, the entities (i.e., goal, agent, obstacles) are randomly initialized.
Available adggthub.com/opeen///ssetyyyy
14

---

Under review as a conference paper at ICLR 2021
Humanoid run
MPO/ MO-MPO for preference policy network w KL-constraint on policy, |
10
Humanoid walk
preference policy network: (e) (T(e support for Eoost
[10 0.30
MPO MO-MPO for preference policy network w KL-constraint on policy, 3
10-6
Safety Gym point goal, button, push
preference policy network: Ty (c) or (T((())) is)
support for Coust
[10-5, ,0220]
Table 2: Hyperparameters for humanoid and Safety Gym experiments that differ from the defaults in Table
Task
Constraint thresholds
Humanoid run
Goos, E linspace( (-.00, -.00,31
Humanoid walk
Cooo, E linspace( -2.0, -0.5,
Safety Gym point goal, button. push
Goost E linspace(- -1, -2 14)
Safety Gym point goal with two constraints
Chuzard -2, linspacee- (0oppacee-10,,,111
Oyzee 2, Chazard linspace( 10,0, 01,,0, 01)
Table 3: The constraint thresholds that policies are trained on for each task.
We use the default task configurations used in Ray al. (2019), for instance how many obstacles are spawned, the regions in which entities can be spawned, the size of entities, and the type of observations. The only difference is that we use sparse task reward instead of shaped task reward, since the former is more realistic.
A sparse task reward of is given in goal when the agent enters the goal area, in button when the agent touches the target button, and in push when the agent pushes a box into the goal area. The constrained reward is the negative cumulative cost. In goal the agent incurs cost by being in hazardous regions or bumping into vases; in button the agent incurs cost by being in hazardous regions, bumping into gremlins, or touching non-target buttons; push the agent incurs cost being in hazardous regions or bumping into pillars. All obstacles are static, except for gremlins, which move fixed circular pattern; the agent moves vases when bumps into them.
In our experiments with two constraints (Sec. 5.1 we use more difficult variant of the point goal task, with two changes. First, the length of the square area in which entities are spawned is three-quarters of the original length, which makes the density of obstacles higher and thus more difficult for the agent to navigate around. In addition, the radius of the goal region is two-thirds the original radius, which allows for it to be placed in more difficult-to-reach locations: because the random initialization of the scene enforces that there are no collisions between entities, the smaller the goal is, the more potential open areas it could be placed in.
C ADDITIONAL EXPERIMENTS AND ANALYSIS
C.I COMPARISON TO MUL.TI-OBIECTIV RL ALGORITHMS
We first evaluate whether constrained MO-MPO is able to find solutions that lie on the Pareto front. Since the ground-truth Pareto front is not available in general, we approximate this for humanoid run
15

---

Under review as a conference paper at ICLR 2021
Task
Number of actor steps
Humanoid run and walk
500 million
Humanoid run, part-way through training
200 million
Humanoid run, per-episode constraint satisfaction
billion
Safety Gym point goal, button, bush Safety Gym point goal with two constraints
400 million
200 million
Table 4: The number of actor steps that policies are trained for, before evaluation.
MΩ-MPO
contmollable MPO
rontrvillble MO-MPO
comstrained MOOMPO
1000
Fromm 0.35
Woome
Greer
0.15
lask rowarrd
500
-2000
50000 acCion norm cost
20000 action ROOT cost
22000 action norm cost
action corm cost
Figure 5: A comparison of the Pareto fronts for humanoid run found by MO-MPO, controllable MPO, and approaches. For MO-MPO and constrained MO-MPO, each dot corresponds to a separately-trained policy. For controllable MPO and controllable MO-MPO, five policies are evaluated by conditioning range of linearly-spaced preferences (which correspond to weights or Kl--divergence bounds, respectively). The gray triangles denote the MO-MPO Pareto front.
by training policies with MO-MPO for range of preference settings: 6rack 0.1 and fnorm at linearly spaced intervals between 10-5 and 0.15.
We trained preference-conditione policies with five random initializations, for the same Cask and fnorm sampled uniformly from the same range; we call this approach controllable MO-MPO. This produces a Pareto front that is comparable to MO-MPO's s, both in terms of coverage and solution quality (Fig. middle right).
It is also possible to train weefeeenee-onnditoone policies where the preferences correspond to weights for linear scalarization, rather than KL-divergence bounds as MO-MPO. So as baseline, we train weight-conditionel policies with MPO (Abdolmaleki et al. 2018); call this bascline controllable MPO. T'esse policies were conditioned on a Wask of 1.0 and Whorm sampled uniformly between O and 0.15. Compared to controllable MO-MPO, we had to train controllable MPO for longer (900M actor steps instead of 500M) in order to reach reasonable task performance, and controllable MPO was still unable to find solutions with low action norm (Fig. 5 middle left). This expected: Abdolmaleki et al. (2020) show that MO-MPO outperforms MPO with linear scalarization on humanoid run, so one would expect controllable MO-MPO to also outperform controllable MPO. We tried running controllable MPO with higher weights, but these did not show any learning progress within 500M actor steps.
Policies trained with constrained MO-MPO for equality constraint thresholds from -4 to -1 also lie on the MO-MPO Pareto front (Fig. [5] right). We train separate preference-conditin and preference policy for each unique constraint threshold. This supports that learning the preference distribution simultaneously while training the policy and Q-functions does not negatively impact the final solutions that are found.
action policy
C.2 MPO LAGRANGIAN BASELINE
The baseline we use in our empirical evaluation MPO with Lagrangian relaxation. We ran this baseline on all Safety Gym point mass tasks, using the default settings (including dense reward) that
16

---

Under review as a conference paper at ICLR 2021
poiitl gool, level
point goel, level
point buttor, level
point button, level 2
point pash, level
point push, level 2
301
Taol reward
20-
10
100
75
5058 cost
50
8 steps x10"
8
6 8
0
0
steps 100
actor steps x10'
actor steps x167
actor steps 100"
actor steps 10'
Figure 6: Top row: The task reward (averaged over episodes) achieved by policies trained with our MPO-Lagrangian baseline, over the course of training. The horizontal solid and dashed grey lines denote the final task reward obtained by the PPO-Lagrangian and TRPO-Lagrangian approaches, respectively, in Ray et al. (2019). Bottom row: The MPO- Lagranggnnn baseline meets the constraint of incurring less than 25 expected cumulative cost per episode, indicated by the dotted red line.
areraage sadk/ rewunl
1000
humanokd
humanool walk
noceegge stsk /oost romed
point goal
point button
point pash
-1000
-1.8 -1.6 -1.4 -1.2 threshold
-12 -1.1 40.8 cost threshold
-8.0
-6.0 -1.0
-200
-8.0
.6.0 4.. rost threshold
220
-8.0 -6.0 -4.0 -2. cnet threshold
osst. threshald
Figure 7: Constrained MO-MPO (in orange) achieves significantly higher task reward than the baseline (in gray) for the harder-to-satisfy constraint thresholds in humanoid run (-1.4 and 1.22), humanoid walk (-1.2, -1.0, and -0.8), point goal (-4 and -2), and point button (-4 and 2). For the other constraint thresholds, both approaches achieve similar task reward. In terms of satisfying constraint thresholds, our approach performs on-par with or better than the baseline. Each bar shows either the average task reward (solid bars) or cost (clear bars) per episode, averaged over five seeds. For both, higher is better. The error bars refer to standard deviation. The red lines denote the constraint thresholds: clear bars above the constraint threshold indicate that the constraint is met.
we can directly compare against the constraint-satisfiinn policies obtained in Ray et al. 2019)) |T Our MPO-Lagrangian baseline compares favorably with those in Ray et al. (2019), that instead combine Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) with Lagrangian relaxation (Fig. 6 This is with the caveat that policies were trained with TRPO- Lagg-naggann and PPO-Lagrangian for 10 million actor steps, whereas our training curves for Mag-Laggangian go up to 100 million actor steps.
C.3 MULTIPLE RANDOM SEEDS
To verify that the empirical results we observed in Sec. 5.1 are significant, we used our approach and the baseline to train policies for the same constraint threshold, starting from five random initializations. We evaluated this for four constraint thresholds per task, selected from the ranges in Table 3 Results are shown in Fig. 7
Constrained MO-MPO achieves significantly higher task reward than the baseline for the harder-to- satisfy constraint thresholds in four out of the five tasks. For the other constraint thresholds, both
Since noticed that our Qufuciiso underestimate cost (Appendix C.4 we use cost threshold of 15 for training, rather than the actual cost threshold of 25.
17

---

Under review as a conference paper at ICLR 2021
point goal
point button
poant push
actual cost
-10-
-20
-20
20
20 Q-va to estimate
Q--alnn estimate
Q-valoe estimaae
MPO Lagrangian
constmined
MO MPO
(a) The learned Q- O-function for cost underestimates the cost episode. The actual cost is obtained by evaluating the deterministic policy after 200M actor steps, the same 100 randomly-initialize environments.
poial gead
poiut button
push
cost threshold
-10-
-20
-20 Q-value estimate
-20
-20 Q-value estimate
Q-valoe estimate
(b) For both approaches, almost all trained policies satisfy the constraint (i.e., above the dotted line), according to the learned Q-values.
Figure 8: In the plots above, the Q-value estimate is obtained by averaging the Q-values for 100 batches of (s, a) pairs from the replay buffer, after 200M actor steps of training. Each point corresponds to eeparately-traine policy, for different constraint threshold
approaches achieve similar task reward. In terms of satisfying constraint thresholds, our approach performs on-par with or better than the baseline.
As mentioned, the slight violation of constraint thresholds for the Safety Gym point tasks due to underestimation of Q-valuss, rather than limitation of the policy improvement method; this is described in the following subsection.
C.4 Q-FUNCTION ESTIMATES
We noticed that for the Safety Gym point mass tasks, both our approach and the Lagrangian baseline
train policies that slightly exceed the constraints (Fig. [ bottom row). After digging deeper, we realized that this results from suboptimal Q-function learning, rather than suboptimal policy optimization. The learned Q-values for the cost consistently underestimate the actual cost incurred (Fig. 8a) The policy optimization finds a policy that meets the constraints, assuming that the learned Q-vluues for the cost are accurate (Fig.
Thus, better constraint satisfaction can be obtained by improving policy evaluation, so that estimated Q-values are more accurate this orthogonal to our proposed approach, which is focused on policy improvement.
D ALGORITHMIC DETAILS
In this section, give details on how to learn the preference policy and provide algorithm box for the policy improvement step. We also describe the overall procedure we use to collect data and learn from the data.
18

---

Under review as a conference paper at ICLR 2021
D.1 LEARNING PREFERENCE PREFERENCE-CONDITIOND CONDITIONED POLICIES
As discussed in the main paper, we also learn a new preference policy (e( given the current preference policy (ed and fitness functions f& (e) (that evaluate in terms of constraint satisfac- tion). One could add the fitness functions together to obtain Σ. [E), and use any off-the-shelf RL algorithm to optimize for new preference policy. On the other hand. this problem can be been as multi-objective problem where the fitness for each constraint k is an objective. To this end, we use MO-MPO for learning Ti (c). Note that this paper we have at most two constraints. the case of problem with one constraint (i.e., one "objective"), MO-MPO (Abdolmaleki et al. 2020) reduces to MPO Abdolmaleki et al. 2018
More formally, following the MO-MPO algorithm, optimize the following constrained optimiza- tion problem, that can be solved via Lagrangian relaxation (Abdolmaleki et al. 2020):
K
max
(L(p(())l=pp()) s.t. (L(Ta(())Imm(()) <,,
where & defines trust region for more stable learning and p& (x(c) is non non-maracettric improved policy for each objective, i.e,
(fa(E)
(E( x (E) exp
PP
6k is temperature variable that we maintain for each objective (or fitness function) and obtained by optimizing the convex dual function
g(px) as + log / TaIo(E) exp
f((() de
PP
where on for each objective & defines desired KL-divergence bound between the new improved policy (e) and current policy (V)). We use the same value of ag 0.1 for all fitness functions k throughout the paper. For more details on MO-MPO, please refer to (Abdolmaleki et al. 2020).
Note that this section assumes that the preference policy is state-independent, i.c., x(e). The optimization is analogous for state-dependent preference policy x (<[ss) which we consider in Sec.
D.2 GENERAL ALGORITHM
We maintain online network and one target network for each Q-function, action policy and preference policy. We also maintain one online network for the temperature function. Target networks updated every fixed number of steps by copying parameters from the online network. Online networks are updated using gradient descent in each learning iteration. We refer to the target networks by using the subscript/supprrsrritt "old" throughout the paper.
We use an asynchronous actor-learner setup. In this setup actors fetch policy parameters from the learner and act in the environment and write transitions the replay buffer. Note that the beginning of each episode, we first sample one preference parameter e from the preference policy, which remains fixed until the end of the episode. The learner uses the transitions in the replay buffer to update the Q- f-functions, policies and temperature functions. | descrries one step of policy improvement for constrained MO-MPO.
19

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Constrained MO-MPO: One policy improvement step
I: given batch size (L), number of actions and to sample (M), current policy
e), current
(K + Q-functions {Q* e)] lemperature network (e(e). (K) constraint fitness functions
{ff(e)) (K) temperature variables {sp)]==1 (K) bounds {ar )= replay buffer D, first-order
gradient-based optimizer O
2:
3: initialize na(als, e) from the parameters of Tai (a|ss
initialize (e(()) from the parameters of mau(e)
5: repeat
6: // Collect dataset (s'.e',a') 22) where
7: stattsss s¹ D
I. preferences e' ~ (T)
9: #1 M actions a Taa(aas', e') and O (s",
11:
Compute (non-parametric) action distribution for each objective 8w + V. En Σ. [M)(')(M)] c + log E, Mr exp (c))]
of
12:
where [k] is the index of the vector
13:
Update w based on 8w. using optimizer O
14: for k o.... K do
15:
9x exp( Tw
16:
end for
18:
Update action policy
19: du Vo E. E3 EK de log (a
20:
(subject to additional KL regularizattion))
21:
Update based on 8,, using optimizer O
22:
23:
24:
Collect dataset (e', fi.k M. M.K where
25:
M preferences e' TTa(()) and få = fc(E')
26:
27:
Compute epsilon distribution for each constrained objective for k l. K
28:
29:
6,2 + Pe + yps log [, + exp
30:
Update yıu based on 8,2. using optimizer O p oc exp( 12 V2
31:
32:
end for
33:
34:
Update preference policy
35:
5x -v Z. Σx Ph log T (e')
36:
(subject to additional KL regularization)
37:
Update Fy. based on 6g. using optimizer
38:
39: until fixed number of steps
40: return (al s, E) ma(als, E) and Tata(E) (W)
20


Reviews:
[{"review_id": "togqm-C6Mrq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interested in reinforcement learning where one needs to satisfy constraints (for instance energy spent) in addition to maximizing rewards. The proposed approach proposes to extend any method able to approximate the Pareto front of optimal policies by also learning portions of the front that satisfy user constraints (given that only small portion of the front may be valid). In particular, a previous approach MO-MPO is used to approximate the Pareto front in a first step and a second step learns another policy find preference vectors satisfying given constrains. Experiments are carried in two environments and show that the proposed method outperforms Lagrangian baselines (e.g. optimizing alternatively for reward and constrain violation) in particular in settings where constraints are harder to satisfy.\n\nThe main strength of this paper are its sound experimental results and overall improvements compared to the baselines studied. In addition, the paper is well-written and tackle a highly relevant problem. The weakness of the paper is that the method is a bit incremental compared to MO-MPO and that it may be hard to reproduce without code.\n\nFor these reasons, I recommend accepting the paper given its improvement over Lagrangian methods and its clarity. It would clearly help if the code allowing to rerun comparison could be provided so that future work can compare with the proposed approach.\n\n# Additional feedback and suggestions\n\n- The problem of concave Pareto front is indeed problematic for linear scalarization but recent work in scalarization may allow to bypass the issue [1]\n- Figure 1: typo \"contrianed\"\n- Figure 1: Is there a reason or an intuition on why outliers appear in MPO Lagrangian (bottom-left)? Is it because of the unstable behavior explained in Fig 2? \n- \"but as the constraint threshold becomes more difficult to satisfy, our approach dominates\", except for \"point push\" this could be precised. Also is it because the front is closer to a convex?\n- Figure 4: I suggest drawing a vertical line with the constraint at -3000, -2000 and -1500. Also it may be clearer to use a normalized action norm cost (e.g. -2.0 instead of -2000) to make it easier to compare with the cost constrain.\n\n[1] Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization. Golovin et al. ICML2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fTf_BpgKJsE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n \nThis paper introduced a general framework that incorporates multi-object reinforcement learning(MORL) perspective for constrained reinforcement learning to a policy set, called Pareto front, that meets the constrained. The author has instantiated a method based on the previous method MO-MPO. Compared to previous Lagrangian-based approaches, the proposed method has advantages in solution quality, stability, and sample-efficiency in few empirical environments.\n\n##########################################################################\nReasons for score: \n \nOverall, I'd vote for rejection for this paper. My major concerns lie in two aspects: 1) the innovation is minor compared to the previous method(MO-MPO). 2) the empirical evidence for the proposed method is marginal to me. Hopefully, the authors can address my concern in the rebuttal period.  \n\nPros:\n\t1. The paper is straightforward and clear to read.\n\nCons:\n\t1. The motivation of the proposed framework is not strong. Besides that, the improvement of constrained MO-MPO is minor.\n\t2. The domains ( runs & walk) didn't illustrate the idea well. It is not clear to me what the constrained is. The advantage of constrained MO-MPO over lagrangian-based approaches is marginal. The advantages could vary from sed choice. \n##########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "c-HyIMk-I_Z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In Reinforcement Learning (RL) it is common that one searches for policies that optimize the reward and that meet certain constraints. A common approach for handling the constraints is Lagrangian relaxation, i.e., to incorporate the constraints in some way into the objective. In this submission, a different approach is suggested, in which the constraints are treated as additional objectives and then the Pareto front of policies is computed. This is reasonable as the optimal policy for a single objective that satisfies certain constraints must be Pareto optimal when the constraints are interpreted as additional objectives. It is then explained how the new approach can be implemented and several experiments are conducted on well-known data sets. It turns out that it is indeed beneficial for some data set to compute the set of Pareto-optimal policies instead of using Lagrangian relaxation because in some cases Lagrangian relaxation finds only the extreme policies while the Pareto front also contains intermediate policies that balance the criteria better.\n\nI find the suggested approach very natural. The connection between constrained problems and the Pareto front is not very surprising and it is well-known and used extensively in the field of multi-objective optimization (for different problems). The writeup is rather condensed. I found it hard to follow Section 4 in detail (but I am not an expert in the field of RL). The experimental results look convincing. It is hard for me as as a non-expert in RL to give a clear recommendation for this submission. It is not clear to me if there is a novel contribution except for the idea to look at the Pareto front instead of a Lagrangian relaxation. It does not become clear to me if there is any non-trivial contribution in Section 4 or if the implementation follows more or less along the lines of previous RL implementations.\n\nI got the impression (on page 3 this is said more or less explicitly) that the submission makes the assumption that for any Pareto-optimal solution there is a preference vector for which this particular solution is optimal. I don't think that this is true. This way one can characterize the convex hull of solutions but in general there can be Pareto-optimal solution that do not lie on the convex hull and for which there does not exist such a preference vector.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Considering reviewers' comments and comparing with similar papers recently published or submitted, this is a good paper but hasn't reached the bar of ICLR.  We believe that the paper is not ready for publication yet, and strongly encourage the authors to use the reviewers' feedback to improve the work and resubmit to one of the upcoming conferences.

Decision: Reject