A Constrained Multi-Objective Reinforcement Learning Framework
Authors: Sandy Huang
Abbas Abdolmaleki
Giulia Vezzani
Philemon Brakel
Daniel J Mankowitz
Michael Neunert
Steven Bohez
Yuval Tassa
Nicolas Heess
Martin Riedmiller
raia hadsell
Venue: CoRL2021 Poster (2021)
Keywords: constrained RL
multi-objective RL
deep RL

Abstract: Many real-world problems, especially in robotics, require that reinforcement learning (RL) agents learn policies that not only maximize an environment reward, but also satisfy constraints. We propose a high-level framework for solving such problems, that treats the environment reward and costs as separate objectives, and learns what preference over objectives the policy should optimize for in order to meet the constraints. We call this Learning Preferences and Policies in Parallel (LP3). By making different choices for how to learn the preference and how to optimize for the policy given the preference, we can obtain existing approaches (e.g., Lagrangian relaxation) and derive novel approaches that lead to better performance. One of these is an algorithm that learns a set of constraint-satisfying policies, useful for when we do not know the exact constraint a priori.

A Constrained Multi-Objective Reinforcement Learning Framework
Sandy H. Huang,, Abbas Ahdolmaleki", Giulia Vezzani, Philemon Brakel, Daniel Mankowitz, Michael Neunert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, Raia Hadsell Deep London, UK
shhuang, aaboolmmalee10@deepmnnddd com
Abstract: Many real-world problems, especially robotics, require that rein- forcement learning (RL) agents learn policies that not only maximize an environ- ment reward, but also satisfy constraints. We propose a high-level framework for solving such problems, that treats the environment reward and costs as separate objectives, and learns what preference over objectives the policy should optimize for in order to meet the constraints. We call this Learning Preferences and Poli- cies in Parallel (LP3). By making different choices for how to learn the preference and how optimize for the policy given the preference, we obtain existing approaches (c.g., Lagrangian relaxation) and derive novel approaches that lead to better performance. One of these is algorithm that learns of constraint- satisfying policies, useful for when we do not know the exact constraint a priori.
Keywords: constrained RL, multi-objective RL, deep RL
1
Introduction
Many problems, especially in the real world and in robot applications, require that policies meet constraints [1]. For instance, we might want a factory robot to optimize task throughput while keeping actuator forces below a threshold, to limit wear-and-tear. Or, we might want to minimize energy usage for cooling a data center while ensuring temperatures remain below some level [2].
Existing approaches typically formulate such constrained RL problems as constrained Markov Decision Process (CMDP) [3]. To solve it, many rely on Lagrangian relaxation [4], which transforms the constrained optimization be unconstrained, where the objective is a sum of the expected task return and weighted constraint functions. The weights are updated to ensure constraint satisfaction.
However, some constrained RL problems do not fit the CMDP formulation. For instance, might not have an exact constraint threshold in mind, but rather a viable range. Or, we may have soft constraints [5], where violating the constraint is undesirable but not catastrophic. These situations arise because of the inherent trade-off between maximizing task reward and satisfying constraints.
Our key insight is to view constrained RL. from multi-objective perspective, where the task reward and each constraint are separate objectives. To train a policy, we must tell it what preference over objectives to optimize for-i.e., how much to prioritize one objective over another. At the extremes, we could tell the policy to either focus entirely on maximizing task reward while disregarding con- straint satisfaction, focus entirely satisfying the constraints. Some preference settings will lead to policies that satisfy the constraints, but we do not know in advance which ones these are.
Thus we propose to learn which preferences lead to monstraintssaisffiiii policies, while simulta- neously training policies for those preferences. We call this framework Learning Preferences and Policies in Parallel (LP3), shown in Fig. 1. In LP3 there are two main design choices: 1) which multi-objective RL algorithm is used to train the policy, and 2) whether to learn a single preference or distribution over preferences. Lagrangian relaxation is one possible instantiation uses linear scalarization as the multi-objective RL algorithm and learns a single preference (Sec. 4.2).
equal contribution
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Module
learn policy, for the current preferences egg. with lineur acstorizution or distributional
Algorithm
Module 1
Module 2
Lagrangian relaxation
lineur scalarization distributional (MO-MPO)
single
I.P3 [MO-MPO]
single
Module
learn preferences satisfy constraints single c
LP3 [MO-MPO-D]
distributional (MO-MPO) distribution
ts. distribation p(e)
Figure I: We use our proposed framework (left), Learning Preferences and Policies in Parallel (I.P3), to derive two novel algorithms for constrained RL by making different design choices (in italics).
We explore two other instantiations of our framework. In the first, to train the policy we use a state- of-the-art mblti-obeective RL algorithm, Multi-Objective Maximum Posteriori Optimization (MO- MPO) [6]. In the second, we learn a probability distribution over preferences, based on how likely they are to lead to constraint-satisfying policies. This enables finding set of constrait--aatisyiing policies in single training run, that differ in terms of task return and constraint performance.
Our main contributions, among others, are:
We propose multi-objective framework for constrained RL, called LP3 (Sec. 4). This is a general framework, that includes Lagrangian relaxation as one possible instantiation. Although the connection between constrained and multi-objective problems intuitive [7,
8), our work is the first to leverage this to create novel algorithms for constrained RL.
Based on this framework, we derive two novel algorithms for constrained RL. One of these is the first algorithm, to the best of our knowledge, for finding a set of constraint-satin sattsffing policies this is important when the constraint is soft or not exactly known a priori.
We run extensive experiments that compare these two new approaches with Lagrangian- based approaches on continuous control tasks across three task suites, including high- dimensional locomotion tasks. Videos are at http: //stt://sites.... wtrssitttssggggllee.... com/view/eee cmorl.
2
Related Work
Constrained reinforcement learning. Constrained RL algorithms learn policies that satisfy con- straints while maximizing task return. Many approaches formulate this problem a CMDP and use Lagrangian relaxation [4]. Recent works claim this converges to constraint-satisfyigg [9] or op- timal [10] solutions. Other works seek to stabilize this optimization by incorporating convex relax- ations [11] I] or derivatives of the constraint function [12]. Others have applied Lagrangian relaxation to mean-value constraints [9], convex set constraints [13], and local constraint satisfaction [14].
For soft constraints, Metal. [5] uses meta-gradients to adjust the Lagrangian learning rate, to find single policy that achieves high task reward without overly violating the constraints. However, this particular trade-off between the reward and cost may not be desirable. Instead, we propose an approach that returns a set of policies choose from, that make different trade-offs.
Safe reinforcement learning. In safe RL, the aim to achieve constraint satisfaction not only at de- ployment, but also during learning [15]. Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training [16, 17, 18, 19, 20, 21]. These approaches require, however, that the initial policy (nearly) satisfies the constraints; otherwise, performance may decrease significantly [22]. We focus on constrained RL, rather than safe RL, but diseuss how extend our approach to reduce constraint violation during training (Sec. 6).
Multi-objective reinforcement learning. In our framework, the policy is trained with multi- objective RL (MORL). MORL consists of single-policy and multi-policy approaches. Single policy methods learn a policy that is optimal for a given trade-off. Most rely on linear scalarization [23], which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales. Non-linear scalarizations exist [24, 25, 26), but these are harder to combine with value-based RL and have limited use in deep RL. Recently, Abdolmaleki et al. [6] introduced MO-MPO, single-policy approach that does not rely on scalarization and is invariant to reward scales.
Multi-policy MORL aims to find sel of policies that covers the whole Pareto front. Recent works learn a manifold parameter space [27, 28] or combine single policy approaches with a general objective [29], to directly optimize the hypervolume of the Pareto front.

---

3 Background and Notation
3.1 Markov Decision Processes
Markov Decision Process (MDP) is defined by (S, ro: 7): the state space S, action space A. transition function p(s'|s, a). bounded task reward function ro(s, a). and discount factor [0. 1].
Constrained MDP. Constrained RL problems are typically formulated as constrained MDPs (CMDP). A CMDP is defined by (S,A, P, To:K, :1:K: ?)) Compared to regular MDP, CMDP in- cludes additional constraint-specife reward functions (rx (s, a) a)) with corresponding thresholds { All reward functions ry return scalar rewards; we will refer to these as separate objectives.
policy {alss) maps from state to distribution over actions. Let J = E, IE, >" at)] denote the expected return for objective i when following policy τ, given fixed initial state distri- bution. The action-value function Q (8, a) is the expected return for objective k after taking action in state s, and then acting based policy л.
The optimal solution to a CMDP a policy that maximizes expected task return, while ensuring all other expected returns satisfy their thresholds:
max JØ s.t. J 2 = 1,..,..., K.
Many approaches for solving CMDPs are based on Lagrangian relaxation, which turns the con- strained problem into the following unconstrained optimization problem:
min max JOT X20
+
Au
J
k=1
These approaches alternate between optimizing the Lagrange multipliers 2 and the policy
Multi-Objective MDP. objti-obeecctiv MDP (MO-MDP) is defined by (S.A. p. ro:k; 7) In contrast to an (C)MDP, there are multiple rewards to optimize for, and no constraints. The optimal policy depends on the desired preference across objectives. In linear scalarization, the preference e defines a weighting across objectives; the optimal policy is the one that maximizes Σ G.J
When no preferences are given, the solution to a MO-MDP is the set of all non-dominated policies, also known as the Pareto front. A policy is non-dominated if there no other policy that improves its return for one objective without decreasing return for another.
3.2 Multi-Objective MPO (MO-MPO)
Whereas most multi-objective RL algorithms use linear scalarization, MO-MPO [6] takes a distribu- tional approach to combining objectives. MO-MPO extends MPO [30] to the multi-objective setting. MO-MPO is a policy iteration algorithm with two steps: policy evaluation and policy improvement.
Policy evaluation. In this step, MO-MPO evaluates the current policy Tou(a)s) by training a Q-function per objective, as according to Q-decomposition [31].
Policy improvement. Given the previous policy (a)a))))) and Q-functions {Q2 a) k= in
this step MO-MPO improves the policy for a given state visitation distribution j(().. MO-MPO decomposes policy improvement into steps, similar to the view proposed in Ghosh et al. [32]: first find per-objective improved action distributions, and then distill these into a new policy.
Finding per-objective distributions: For each objective k. MO-MPO computes nonparametric policy qu (a) s)) that improves the old policy with respect to that objective, subject to a non-negative constraint eg the KL-divergence between the improved and old policies:
[.[
max Eu(e)) (c(ss)) Q% (s, a) da) qu
s.t. (Pp()) (T-((<(-ls) m=a((1))) 6k
Learning new parameterized policy:: MO-MPO then updates the policy via supervised learning on these nonparametric policies, subject to a trust region with bound 3 for more stable learning:
N
min Σ
Bu(a) KL no (-()s))
s.t. En())
KI(mau(-1s)) Te (-1s)) B.
(4)
The KL-constraints K encode preferences: the larger ex. the more influence objective k has.
3

---

Learning Preferences and Policies in Parallel (LP3)
In this section we introduce LP3, our constrained multi-objective RL framework. We first formulate the constrained RL problem as a Constrained Multi-Objective MDP (CMO-MDP). Following this, we describe how LP3 solves this MDP, and detail how Lagrangian relaxation algorithms are an instantiation of this framework. We then derive two novel constrained RL algorithms based on LP3.
4.1 Constrained Multi-Objective MDP (CMO-MDP)
The CMDP formulation implies a single optimal solution. But in real-world settings, one may be considering a range of viable constraint thresholds, which correspond to different trade-offs between the task reward and constraints. This connects to the multi-objective setting.
We define a constrained multi-objective MDP (CMO-MDP) {S, A, ra.k, BNNK, 7) where N 2 1. In a CMO-MDP, the optimization is over reward functions, where N of these are task rewards, and the remaining K N rewards have constraints them. Thus a CMO-MDP dif- fers from a CMDP in two key ways: 1) the optimization objective is over all rewards, including the constraint-speeific ones; and 2) there may multiple task rewards To.N.
CMO-MDP compares constraint-san santrrant--atisffiing policies in terms of their returns for the task reward and constraint functions- -ttsssitsssssse solution is the set of all constraint-satiisfingg Pareto optimal policies. In contrast, a CMDP ranks policies according to only task return its solution the single constraint- satisfying policy with the highest task return.
4.2
The LP3 Framework for Solving CMO-MDPs
We now describe LP3, a unifying framework (Fig. 1) for deriving algorithms that solve CMDPs and CMO-MDPs. This framework has two modules: learning the policy, and learning which prefer- ence(s) (over the task and constraints) the policy should optimize for to meet the constraints.
For the first module, the key choice is which multi-objective RI. algorithm (such linear scalariza- tion or MO-MPO) is used to train the policy xy to optimize for a given preference.
For the second module, the key choice is whether to learn single preference distribution p(e). The former corresponds to learning a policy Te (a]s). Learning a distribution over preferences, however, requires training preference-condiiiooee policy To(als, E), that learns to op- policies, timize for different preference settings. This enables finding a set of constraint-satisfyii produced by sampling preferences from the final p(e) and rolling out the fully-trained Ta(aas, e), conditioned on that preference. In practice, using a preference distribution can also improve learn- ing, because training
a preference
multiple preferences enables more diverse data collection and exploration.
Note that we can use different time scales for training the two modules in this framework. In our experiments, we alternate between taking one learning step on each.
Connection to Lagrangian Relaxation. As mentioned, Lagrangian relaxation methods are equiv- alent to LP3 with linear scalarization and single preference optimized using gradient descent. To show this, define the preference
[1, A1,
where λ1:K are the Lagrange multipliers.
In the first module, we train the policy to maximize a linear combination of Q-values, weighted by the current preferences old, Σ enld ( (a, s). We can use any standard RL algorithm for this.
gradient descent to optimize for single preference setting E with respect the loss L(e 0) = =K Eafk. Here, fx is the fitness score corresponding to whether the policy respects threshold Bx::
In the second module, given the current policy (a)s))))
En(s)
QQd Er.u(al)) [a)] Bk.
(5)
4.3 Novel Instantiations of LP3
Making different design choices in the LP3 framework leads to two novel algorithms for constrained RL. The first replaces linear scalarization with state-of-the-art multi-objective RI. algorithm, MO- MPO. The motivation is that MO-MPO may be better at training policies for a given preference,

---

since it has been shown to outperform linear scalarization on variety of control problems [6]. We call this LP3 [MO-MPOJ. uses the same second module (for learning the preference) La- grangian relaxation. The second algorithm, in addition to using MO-MPO, learns distribution over preferences. We call this LPP [MO-MPO-D] and describe it below; full details are in Appendix
4.3.1 Module 1: Preference-Conditionel MO-MPO
Learning distribution over preferences requires training preference-conditioned policy To To(als, e), but MO-MPO can only train a policy To (a(as) for single preference c. We extend MO-MPO to train a single policy (a|s, c) conditioned on preferences c ~ Pota(c), by making both the policy evaluation and policy improvement steps preference-conditioneed as follows.
Preference-conditionee policy evaluation. We evaluate the current policy (osa(e)TTTa(CC (a|s, ss E) via off off-policy learning of per-objective Q-functions QE (a, s, E). Note that these Q-functions must be preference-cone conertice-cnnditionee because the policy is. For each learning step we sample L transitions from the replay buffer, {s,, as {r]] s Motivated by prior work on learning from off-policy data [33], we use relabeling to augment the states with preferences sampled from the current pref- erence distribution 6; (e) resulting in transitions {[st, 61), as (r ) K [s/. o. Any policy evaluation algorithm can be used. We use distributional policy evaluation with 5-step return [34].
Tk
Preference-conditiore policy improvement. This step improves the current policy (ata(als, (s E).
Finding per-objective distributions: To find preference-conditione per-objective improved action distributions yi (als, e), we optimize the following policy optimization problem for each objective:
max E ppa(())u(s) I/ qu (als,, e) Q%d (s, a, e) da
s.t. Ep(s) KL [ata e)) < cx YE- (o-Poll(E).
This generalizes (3) from MO-MPO, which assumes fixed KL-constraints e, to a distribution Pola(E)
over KL-constraints. We use closed-form known nonparametric solution for 9k [35, 30).
Learning a new parameterized policy: After obtaining per-objective improved policies, we use su- pervised learning to distill these into a new parameterized wrwpeeeeceeeeeefeeeeeeeeeeooe preference-conditionee policy:
min
[K
8.1. (es (TTTTaaa < B.
(7)
k==0
4.3.2 Module 2: Learning Preferences
The second module optimizes the preference e or preference distribution p(E) to lead to better con- straint satisfaction. This assumes fitness functions f& that evaluate satisfaction of the constraint threshold C.. given the current policy Vold and Q-function Q We can use any gradient- algorithm to optimize the preferences, initialized by Cold Pola(e), maximize the fitness.
In our empirical evaluation, we use fitness functions that give a score zero to preferences that lead to constraint satisfaction, and negative scores to all others. For equality constraints we use fitness function that penalizes the difference between the expected Q-values and constraint threshold, and for inequality constraints it penalizes the amount the expected Q-values violate the threshold:
(E) Bu((e) [Q2 (s. Eral (a], <)] Bx
(8)
f (E) min (0, Ey(e) [2x (s, Bna.Enlld [a), E)] Bz8).
(9)
Experiments
To recap, in the LP3 framework, Lagrangian relaxation corresponds to using linear scalarization (LS) and learning a single preference (Fig. 1). Lagrangian relaxation is a widely-used approach for
'The flexibility in choosing the fitness function enables specifyynng a variety of constraints: e.g., expected return equal to threshold, within interval of thresholds, or within the union of disjoint threshold intervals.
5

---

In toy domain, trained policies for four different constraints on r2 per task. When the Pareto front is concave (right), the LP3 [I.]] baseline cannot recover the optimal solutions, whereas LP3 [MO-MPO] can. This because LP3 [LS] relies linear scalarization (LS), which fundamentally can- not find solutions on concave Pareto fronts [36].
Figure 2:
Sohaileer
Faneeso Flosning
LP3
[MO-MPO]
0.5
LP3 [LS]
-.0
71
homaanod walk
humanold ran
rum. mid--rriinng
point goal
point button
point push
lask rewand
tack reward
10-
1000
1500
0
-1500
10
-10
-10
D
action norm cost
total cost
LP3 [MO-MPO]
LP3 [MO-MPODD]
LP3 [LS]
Figure 3: Using MO-MPO (cireles) for Module results in better policies than using LS (triangles), for hard- lo-satisfy constraints. These plots show the performance of policies trained different constraint thresholds. Each point denotes task reward and cost of a fully-trained policy; points above and to right are better. The LP3 [LS] baseline struggles to find policies that perform well on both the task and cost. LP3 [L.S] also cannot find solutions on a concave Pareto front (humanoid walk) and takes longer to learn the task (run, mid- miidtraiinngg.
constrained RL because of its simplicity and effectiveness [22, 12). We proposed two novel changes: 1) using MO-MPO instead of LS in Module 1, and 2) learning a distribution over preferences rather than a single preference in Module 2. In our empirical evaluation, we first run ablation studies that study the impact of cach of these changes. Then, run further experiments with LP3 [MO-MPO- DJ. that combines both improvements. The main baseline we compare against is LP3 [LS], which uses MPO [30] to optimize the Lagrangian dual objective (2). This is strong Lagrangian-basee baseline that obtains state-of-the-ar results on benchmark constrained RL tasks (Appendix C.2).
Implementation. We train Gaussian policies parameterized by neural networks. Our implemen- tations are based on the open-sourced MPO and MO-MPO in Acme [37]. For LP3 [MO-MPO-D), we learn separate distribution per preference és. initialized to uniform. We choose to use dis- crete distribution because is flexible and interpretable. Unless otherwise mentioned, we use (9) as the fitness function for LP3 [MO-MPO] and use (8) for LP3 [MO-MPO-D]-- that it converges to a single policy, rather than a set of policies, for easier comparison with the single-preference approaches. We sweep over hyperparameters for each approach; details are in Appendix A.
5.1 Module 1: Linear Scalarization MO-MPO
To first assess the impact of using MO-MPO instead of linear scalarization, we compare LP3 [MO- MPO] against LP3 [LS] across several domains. LP3 [LS] is an ablation of LP3 [MO-MPO] with respect to Module I; both learn a single preference in the same way for Module 2.
Toy Domain. First, consider toy domain with continuous action (a E R) and two objectives (r(a) E R²). The constraint is on r2(a). For the reward, we use either the Schaffer or Fonseca- Fleming function, which standard test functions in multi-objective optimization [38]. For each function, we train policies for four constraint thresholds. For Schaffer, both approaches find the optimal solutions. However, for Fonseca-Fleming which has concave Pareto front, only LP3 [MO-MPO] can find all optimal solutions (Fig. 2, right). LP3 [LS] only finds solutions at the two extremes, even after tuning the learning rate. This is likely because linear scalarization cannot find solutions on the concave portions of a Pareto front [36]. Even in this simple setting, we can see the potential advantages of using MO-MPO over linear scalarization, Module of LP3.
Humanoid. We next consider two nigh-dimensional locomotion tasks, humanoid run and walk from the DeepMind Control Suite [39]. The constraint is imposed the expected negative control norm (i.e.. - [|ll22) which roughly captures the "energy" expended by the agent.
We train policies using LP3 [LS] and LP3 [MO-MPO] for variety of constraint thresholds. We plot the average task reward and cost for the fully-trained policies in Fig. 3. L.3 [MO-MPO] finds higher-quality solutions for both tasks, especially for walk, which may be because its ground-truth

---

Task
Algorithm
Task
Cost
Task
Algorithm
Task
Cost
cartpole
Metal
800 = 1
-342 - 1
quadruped
MetaL
+
-615 +: 15*
swingup
D4PG-Lag
376 11
-199 = 29
walk
D4PG Lag
999 1 0*
-632 1: 14*
1.P3 [1SS]
166 20
2993 108t
LPA [1.S]
1000 + 0
-33 +
LP3 [MO-MPO-D]
871 4
-80 +
LP3 [MO-MPO-D]
1000 = 0
0
humanoid
Metal.
659 -: 54
368.3 26
walker
Metal.
966 + 0
1664. 9
walk
D4PG-Lag
571 + 53*
- -301.3 + 19*
walk
D4PG-Lag
905 + 11
-76 -76.8 + 3
LP3 [LSS
668 + 149
100.3 1 41
LP3 [LS]
983
-58 5
LP3 [MO-MPO-D]
828 - 28
-32 +
LP3 [MO-MPO-D]
983 = 0
-61 + 28
Table I: RWRL tasks, LP3 [MO-MPO-D] achieves better task reward and cost than other algorithms. The bounds are on standard error. Results for MetaL and D4PG-Lag are from [5]: both rely on linear scalarization. (p I 0.05) and (p I 0.1) indicate significance based on Welch's 's t-test, compared to LP3 [MO-MPO-D],
Pareto front is concave. LP3 [MO-MPO] is also more sample efficient: for humanoid run, mid way through training LP3 [MO-MPO] policies obtain moderate task reward, while LP3 [LSS] policies have not learned the task at all (Fig. 3, "run, mid-training"). This could be because LP3 [MO-MPO] can optimize for both objectives at once, whereas LP3 [LSS] cannot (see analysis in Appendix C.4).
5.2 Module 2: Single Preference vs. Distribution over Preferences
We combine MO-MPO, because it outperforms linear scalarization, with learning a distribution over preferences to obtain LP3 (MO-MPO-D)) LP3 [MO-MPO] is an ablation of LP3 [MO-MPO-D] with respect to Module 2. For humanoid, both approaches perform similarly (Fig. 3).
A key advantage of using LP3 [MO-MPO-D]] as opposed to single-preference approaches, is the ability to solve CMO-MDP and discover a set of constraint-satiifinng non non-dominated solutions, rather than finding a single solution. This is useful because in practice, we may not know the true constraint ahead of time, and would thus prefer to see several options along the Pareto front. We evaluate this the following section on a task with three objectives, two of which are constrained.
5.3 Evaluation of Full Algorithm
Finally, we focus on comparing LP3 [MO-MPO-D], which combines both improvements, against existing approaches in the literature, as well as our LP3 [LS], a strong Lagrangian bbase approach.
RWRL, We use the same four Real World RL (RWRL) [40] tasks and constraints used by Calian et al. [5], and we compare against the algorithms evaluated in that work (described in Appendix B.5). The cost is binary and depends on the task, e.g., for humanoid walk, the cost is -1 if the joint angles outside of a specified range, and 0 otherwise. For each task, we used LP3 [MO-MPO-D] and LP3 [LS] to train five policies with different random initializations. LP3|MM-MMO-DD performs best on three out of four tasks, incurring less cost while achieving on-par or better task reward (Table 1).
Safety Gym From OpenAI Safety Gym [22], we use the Level 2 point mass tasks: goal, button, and push. There is a sparse reward for either reaching goal location pushing box to goal. There is a cost for running into or over objects. On two out of the three tasks, LP3 [MO-MPO-D] finds better solutions than LP3 [LS], on harder-to-satisff constraint thresholds (Fig. 3). For point push and for sonierr-o-satisfyy constraint thresholds, LP3 [LS] performs on par with LP3 [MO-MPO- DJ (see Appendix C). We hypothesize this is because when meeting the constraint does not conflict much with the task objective, it is easier to find high-task-reward solutions that incur small cost.
Robot. We created simulated robot manipulation task with the Sawyer arm, with sparse reward for reaching a green block and pushing it to a target position. There is a cost for either the gripper or green block colliding with the yellow block (Fig. 4, left). On this task. LP3 [MO-MPO-D] and LP3 [LS] converge to similar task performance and constraint satisfaction, but LP3 [LS] takes about twice as many actor steps to do so (Fig. 4). This is likely due to the use of MO-MPO in Module r-rall that in the first ablation study, this improved sample efficiency in humanoid run (Sec. 5.1).
Multiple Constraints. We also evaluate on tasks with three and four objectives: walker run and humanoid walk from the Control Suite. In walker run, there are three objectives: a torso height reward, moving forward reward, and cost for energy usage (captured by negative control norm). In humanoid walk, there are four objectives: the task reward (from the standard Control Suite task), moving forward reward, moving left reward, and cost for energy usage.
7

---

faok
task yorver
-50-
-30
LP3 [MO-D] LP3 [LS]
1.00
0
10
10
actor stepe (x107)
Figure 4: On robot push task, LP3 [LS] much less sample-efficien: it takes about twice as many actor steps to converge to similar task and cost performance as LP3 [MO-MPO-D]. denotes the constraint threshold.
500
250
moveleli 400, cosl 2: -1.5
move or action borm
coes 2 -25
cast 2 -1.
height reward
1000
1200
move left reword
move left revcard
5000 -800
Dore reward
500
500-
500
-2000
2000
700
2000
1000
-1500
action norm
action HOTEL DOSS
move forward reward
Figure 5: For walker run (left) and humanoid walk (right), use LP3 [MO-MPO-D] to train policies that learn portion of the Pareto front. Walker has three objectives and humanoid has four objectives; both tasks have incquality constraints on two objectives. These plots shows the Pareto front for one random seed; Appendix C.6 contains plots for two others. The dashed lines indicate the constraint threshold. We evaluate each fully-trained policy by conditioning evenly-spaced percentiles from its learned distribution and the median preference for the other objectives. Since we can only plot two objectives at time, we use color indicate the value another. For humanoid walk, the not-plotted fourth objective (task reward) has near maximum reward regardless the preference.
learned preferences: for each objective with constraint, we select preferences at
For both tasks, we set inequality constraints on moving and energy usage. We use LP3 [MO-MPO- D] to train seeference-conditione portion of the Pareto front that satisfies the constraints (Fig. 5), which useful when the constraint is soft or not known exactly a piori. For humanoid walk, the agent learns to smoothly interpolate between walking left and walking forward (Fig. 5, right-most plot). In this task, walking left is a more challenging gait and would not likely emerge naturally without constraint on this objective.
policies using the fitness function (9). These policies learn
Appendix contains further experiments and analysis, including on constraint satisfaction (Ap- pendix C.5) and evaluation with multiple random seeds per constraint threshold (Appendix C.7).
6 Conclusion and Future Work
In this paper, we proposed to formulate constrained RL as constrained multi-objective MDP. Based on this multi-objective perspective, we described general framework, LP3, for solving constrained RL problems. We used this perspective to introduce two new algorithms: 1) using better multi- objective RL. algorithm, MO-MPO, instead of linear scalarization; and 2) learning a distribution over preferences instead of a single preference, to find a set of constraint -tatraiint-tttifyinng non-dominated policies. For safe RL, one could initialize the preference distribution to prioritize minimizing cost.
This work has several limitations. One is that LP3 [MO-MPO(-D]] has not been validated yet on real robot platform. However, MPO with Lagrangian relaxation has been successfully run on robots (e.g. 14], and our ablation study suggests LP3 [MO-MPO] should perform at least as well. Another limitation that LP3 depends on accurate policy evaluation ensure constraint satisfac- tion, because it relies on Q-value estimates to learn preferences. The preference learning rate cannot be too high, otherwise the distribution will converge based on Q-values estimates for a partially- trained
which may not accurately reflect which preferences will lead to constraint satisfac- tion for the fully-trained policy. In addition, using a discrete distribution for the preferences requires that its support includes preferences that satisfy the constraint threshold. Finally, a limitation is that we focus on problems with up to four objectives. In principle, LP3 [MO-MPO-D] can be scaled up to more objectives with minimal additional computational cost, and this can be easily parallelized. However, the more objectives there are that are in conflict with each other, the more difficult the problem is. It would be valuable to investigate performance on these problems in future work.
8

---

Acknowledgments
The authors would like to thank Dan Calian, Andrea Huber, Leslie Fritz, Christopher Schuster, the Control Team, the REAL Team, and many others at DeepMind for their helpful feedback and support for this paper. We would also like to thank our anonymous reviewers for their valuable feedback.
References
[1] G. Dulac-Amold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learn- ing. arXiv preprint arXiv: /2901, 2019.
[2] N. Lazic, C. Bouuiller, T. Lu, E. Wong, B. Roy, M. Ryu, and G. Imwalle. Data center cooling using model-predictive control. In Advances in Neural Information Processing Systems 31, 2018.
[3] E. Altman. Constrained Markov Decision Processes. CRC Press, 1999. ISBN 9780849303821.
[4] D. Bertsekas. Nonlinear Programming. Athena Scientific, edition, 1999.
[5] D. A. Calian, D. J. Mankowitz, T. Zahavy, 7. Xu, J. Oh, N. Levine, and T. Mann. Balanc- ing constraints and rewards with meta-gradient d4pg. Proceedings of the Ninth International Conference on Learning Representations (ICLR), 2021.
[6] A. Abdolmaleki, S. H. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. F. Mar- tins, N. Heess, R. Hadsell, and M. Reidmiller. A distributional view on multi-objective pol- icy optimization. In Proceedings of the 37th International Conference Machine Learning (ICML), 2020.
[7] P. S. Thomas, B. C. da Silva, A. G. Barto, S. Giguere, Y. Brun, and E. Brunskill. Preventing undesirable behavior intelligent machines. Science, 366(64668))999110000) 2019.
[8] H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning (ICML.,, 2019.
[9] C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
[10] S. Paternain, L. Chamon, M. Calvo-Fullana, and A. Ribeiro. Constrained reinforcement learn- ing has zero duality gap. In Advances in Neural Information Processing Systems 32, 2019.
[11] M. Yu, Z. Yang, M. Kolar, and Z. Wang. Convergent policy optimization for safe reinforcement learning. In Advances in Neural Information Processing Systems 32, 2019.
[12] A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. Proceedings of the 37th International Conference Machine Learning (ICML), 2020.
[13] S. Miryoosefi, K. Brantley, H. Daume III, M. Dudik, and R. E. Schapire. Reinforcement learning with convex constraints. In Advances in Neural Information Processing Systems 32, 2019.
[14] S. Bohez, A. Abdolmaleki, M. Neunert, J. Buchli, N. Heess, and R. Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv: 1902.04900 2019
[15] García and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research,
2015.
[16] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning (ICMI), 2017.
[17] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In Advances Neural Information Processing Systems 30, 2017.

---

[18] Y. Chow. O. Nachum, E. Duenez- Guzman, and M. Ghavamzadeh. A Lyapunov-base based ap- proach to safe reinforcement learning. In Advances in Neural Information Processing Systems
31, pages 8092-8101, 2018.
[19] Y. Chow, O. Nachum, A. Faust, E. Quenez-Guzman and M. Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprini urXiv: 2019.
[20] T.-Y. Yang. J. Rosca, K. Narasimhan, and P. J. Ramadge. Projection-badd -bsed constrained policy optimization. In Proceedings of the Eighth International Conference on Learning Representa- tions (ICLR), 2020.
[21] Y. Zhang, Q. Vuong, and K. W. Ross. First order optimization in policy space for constrained deep reinforcement learning. arXiv preprint arXi::2002.06500 2020.
[22] A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
[23] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley. A survey of multi-objective sequen- tial decision decision-making making. Journal of Artificial Intelligence Research, 48(1):67-113, 2013.
[24] G. Tesauro, R. Das, H. Chan, J. Kephart, D. Levine, Rawson, and C. Lefurgy. Managing power consumption and performance of computing systems using reinforcement learning. In Advances in Neural Information Processing Systems, pages 1497-1504, 2008.
[25] K. Van Moffaert, M. M. Drugan, and A. Nowé. Scalarized multi-objective reinforcement learn- ing: Novel design techniques. In 2013 IEEE Symposium Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 191-199, IEEE, 2013.
[26] D. Golovin and Q Zhang. Random hypervolume scalarizations for provable multi-objective black box optimization. In Proceedings of the 37th International Conference Machine Learning (ICML), 2020.
[27] S. Parisi, M. Pirotta, and M. Restelli. Multi-objective reinforcement learning through contin- uous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227, 2016.
[28] S. Parisi, M. Pirotta, and J. Peters. Manifold based multi-objective policy search with sample reuse. Neurocomputiing 263:3-14, 2017.
[29] Xu, Y. Tian, P. Ma, Rus, S. Sueda, and W. Matusik. Irddiction-guiden multi multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
[30] A. Abdolmaleki, J. T. Springenberg, mum posteriori policy optimisation. In Proceedings of the Sixth International Conference Learning Representations (ICLR), 2018
Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi-
[31] S. Russell and A. L. Zimdars. Q-dccomposition for reinforcement learning agents. In Pro- ceedings of the Twentieth International Conference on International Conference Machine Learning (ICML), pages 656-663, 2003.
[32] D. Ghosh, M. C. Machado, and N. L. Roux. An operator view of policy gradient methods. CoRR, abs/2006.1 1266, 2020. URL httts:t/ttssriii/
http::////ardl...rra////00000/0 wwwp/rgg//aa/oooooeeo 1 11266.
[33] M. Andrychowicz. F. Wolski, A. Ray, Schneider, R. Fong. P. Welinder, B. Mc- Grew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. V. Luxburg.
Bengio, H. Wallach. R. Fergus, S. Vishwanathan, and R. Gar- nett. editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: https:// /pwoceedin/ceee/ssseeoeee wwa.ggs..urrs.... weeuurpss.cppapp////000/0 cc/paper/2017/fsle) Paper. pdf.
[34] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. TB, A. Muldal, N. Heess, and T. Lillierap. Distributional policy gradients. In International Conference on Learning Representations (ICLR), 2018.
10

---

[35] J. Peters, K. Mülling, and Y. Altün. Relative entropy policy search. In Proceedings of the Twenty-Fore Fourth AAAI Conference on Artificial Intelligence. AAAI' page 1607-1612. AAAI Press, 2010.
[36] I. Das and J. E. Dennis. A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural Optimization, 14: 63-69, 1997.
[37] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Ab- dolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.09979, 2020.
[38] Okabe, Y. Jin, M. Olhofer, and B. Sendhoff. On test functions for evolutionary multi- objective optimization. In Parallel Problem Solving from Nature PPSN VIII, 2004.
[39] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, Merel, A. Lefrancq. T. Lillicrap, and M. Riedmiller. Deepmind control suite. arXiv preprint (Tf1::800000000 2018.
[40] G. Dulac-Amold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.//888 2020.
[41] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the Third International Conference on Learning Representations (ICLR), 2015.


Reviews:
[{"review_id": "mMUJniGt2t_", "reviewer": "Reviewer_rQa3", "summary": "This work proposes a general framework to leverage multi-objective learning for constraint RL, for which Lagrangian relaxation is one possible option. Further, the authors propose two advances to this setting. First, instead of linear scalarization they use Multi-Objective Maximum-a-Posteriori Optimization (MO-MPO) to allow for a distributional approach that combines objectives. Second, they introduce the use of a preference distribution in place of a single preference. This additionally enables to find a set of constraint-satisfying policies that offer different levels of trade-off between constraints and task performance. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Using multi-objective optimization for constraint RL is an intuitive direction. While the idea in itself is not novel, this work is still introducing the idea to a new area and is at the same time approaching an important field in deep RL and Robotics.\nOverall, this work is well motivated, structured excellently, and provides sufficient background information to understand all aspects presented throughout every chapter. I very much enjoyed reading this paper and the general approach itself was apparent. \n \nAn important weakness with the current approach is the applicability to physical robotics systems, which the authors state in their conclusion as interest for future work. However, given the currently required sample complexity, it might be challenging to directly apply the algorithm in its current form. Other than that, the work does not put much emphasis on demonstrating the limitations of the framework. Only the appendix contains additional discussion, e.g. for the `point` task the constraints are consistently violated due to the Q-function approximation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "First and foremost, the technical quality as well as the clarity of this work is well done and there is little to complain about. \nThe novelty of the proposed framework is mainly given by transferring the existing idea of multi-objective learning for constraint optimization to the (deep) RL space. While this is not a groundbreaking idea, I think it has its place in the given setting.\n\nDespite the good writing, I am still left with some open points: \n- For the preference-conditioned MO-MPO, the replay buffer transitions are relabeled based on the current preference distribution $p(\\epsilon)$. Do you have any results that demonstrate that this indeed leads to improved stability and sample efficiency?\n- While the approach can learn a set of constraint-satisfying policies, how would one choose an appropriate trade-off. In simulation one option would compare multiple policies, albeit, this is assuming a human expert that can search the preference distribution. Considering a physical system, this might not even be an option, especially when trying to avoid potentially system-harming/-damaging trade-offs. \n- For easier comparison LP3 [MO-MPO-D] uses equality constraints to find a single solution. However, wouldn't that lead to a different optimization problem in the first place? \n- Figure 4 shows how the learned set performs, when using inequality constraints, albeit to me it seems the coverage of the Pareto front is quite seed dependent. Exploring this more would also be in the interest of determining the limitations of the framework.\n- For the toy domain I did not find any details about the initialization or seed choice. It might as well be that LP3 [LS] is performing poor simply because of the chosen seed and not necessarily due to the concave Pareto front. \n- For the reward shaping baseline Calian, et al. consider quite large steps for the chosen weights. I expect when tuning those weights for the task the performance for this method will increase and therefore present a better baseline.\n- The appendix also touches slight upon using multiple constraints. How would you expect your method to perform when introducing a larger number of constraints, which for some tasks can be easily possible.    ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "### Minor Issues \n- One of the main references is cited incorrectly: D. A. Calian, D. J. Mankowitz, T. Zahavy, Z. Xu, J. Oh, N. Levine, and T. Mann. Balancing\nconstraints and rewards with meta-gradient d4pg. arXiv preprint arXiv:2010.06324, 2020. --> ICLR 2021\n- Table 1: LP3 [LS] Humanoid walk is neither bold nor marked with a $p\\leq0.05$ for Welch's t-test\n- Comparing the numbers reported in this work and the numbers reported in table 7 by Calian, et al. I cannot find a direct match, most of the results are close but not identical."}}, {"review_id": "XFQtb8mdYUt", "reviewer": "Reviewer_Z5YJ", "summary": "This paper proposes a high-level framework called Learning Preferences and Policies in Parallel (LP3), which can derive new approaches basing on existing methods. Based on the idea of learning two modules separately, this paper solves the problem of constrained multi-objective reinforcement learning with better performance. The article has some innovative significance, and the experimental scheme is basically reasonable. What’s more, the experimental results and data in this paper are relatively good and reliable. In the open simulation environment, the RL system is verified. From the perspective of the algorithm, the verification is obtained, which makes a certain contribution to the development of Reinforcement Learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes the LP3 framework, considering the RL algorithm from the top-level perspective, which improves the efficiency of algorithm promotion, and at the same time has a positive significance for the development of RL. However，for the mathematical concepts and derivation of the traditional RL algorithm, the description in this paper is slightly cumbersome. What's more, the definition of the framework in the title of this paper is not reasonable, because the framework is not universally validated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This article proposes a new RL insight (introducing learning preference), and tries to explain the idea with a general framework. However, the transferability of this framework to other Constrained Multi-Objective Reinforcement Learning problems remains to be verified. Therefore, it is recommended to reconsider the definition of the framework.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1.Lines 105-106, for the traditional definition of non-dominated policy, the meaning of these two sentences in this position and the author's intention are not clear enough. \n\n2.Lines 214-215, the reason for using uniform-discrete distribution is not clear enough. \n\n3.The length of the basic knowledge in RL should be reduced, and the algorithm implementation and experimental results should be analyzed more. For those who are new to this professional field, this paper can also cite classic representative documents to convey basic theories.   \n4. From Figure 1 left, the framework composed of module 1 and module 2 gets a loop iteration, but in fact, only a limited number of combinations are made in the paper. Strictly speaking, this cannot be called a framework, in other words, the description of the framework in the paper is a little exaggerated.\n\n"}}, {"review_id": "LnELgzPixET", "reviewer": "Reviewer_YksG", "summary": "The authors present a more generalized framework for handling multi-objective RL problems which merges concepts of both constrained MDPs and multi-objective MDPs with in a circular two-phase optimization where the RL policy is optimized for a set of preferences followed by the preferences being optimized for a set of constraints and repeating. The authors also claim that the resulting LP3 is more sample efficient, able to find sets of viable policies and able to handle soft constraints.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nS1) The paper has generally decent flow and communicates most of its ideas clearly with sufficient background\n\nS2) I always appreciate the process of starting demonstrations with toy problems and working up to more complex domains as it does better to establish the viability of an approach in more intuitive terms.\n\nS3) I like the idea of addressing soft constraints or constraint bounds as opposed to strict constraints as this can relate to more realistic scenarios.\n\nS4) The website has intuitive visualizations that help drive the key points of the paper home.\n\n### Weaknesses:\n\nW1) It appears to me that, despite talking about a 'multi-objective' setting, many of the experiments relate mainly to just a single reward and a single policy regularization constraint - and even that seems mostly related to actuation costs. How would this method extend to other types of objectives, and how well would the algorithm handle a more than two soft constraints - what are the associated performance and computational tradeoffs? I feel like these are important questions to address that are not addressed in the paper.\n\nW2) I am not convinced that treating MO-MPO as a special module really constitutes a novel instantiation of LP3 in a significantly meaningful sense (at least not in a way that I'd regard a novel module) - unless I am mistaken, the extension proposed in this paper is mostly just input-state augmentation, am I wrong?\n\nW3) It would help if the figure captions were more self-contained and descriptive, with important takeaways summarized. The inconsistency of the formatting for the caption for Figure 2 also interrupted the flow of reading.\n\n### General notes:\n\nOverall, I think the methods proposed here have promise, but need more thorough evaluation to demonstrate broader value. One possible exploration could be in looking at how the different reward components for the various tasks could be satisfied (for example, the gym mujoco control tasks have rewards associated with joint limits, energy consumption, task progress, etc. - so considering them all separately as constraints could make for interesting test cases)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I think the work shows promise, I am as yet unconvinced of broader utility by the experimental results.\n\n(Updated after rebuttal):\nI am more convinced, given the additional results presented and additional contextualization provided. I am still not convinced on the issue of the novelty of MO-MPO, but I am willing to let it slide given that no other major concerns are raised", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Experiments with more than 2 objectives would be the major thing I'd hope to see before I can accept this as a 'multi' objective solution", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "Gvtisozm9lD", "reviewer": "Reviewer_uAjH", "summary": "This work explores two problems in RL, learning a policy with constraints and multi-objective learning. Currently, known methods exist to solve each of these problems.  The authors merge solutions to these two problems so that constraints can be placed on a multi-objective MDP.  To do this, the authors define an MDP with multiple objectives, and each objective can have a constraint on it, which they title a \"constrained multi-objective MDP\".  The method is then derived which combines known work on multi-objective policy optimization (MO-MPO) and learning with constraints to solve this class of MDPs.  The main contribution of this work is that the approach is able to approximate the Pareto front of optimal policies by learning a segment of the front that satisfies the given constraints.  This presents the benefit of finding Pareto optimal policies that balance between satisfying constraints and task reward.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The approach of this work is quite intuitive, and the writing is clear and convincing. This work derives intuitive connections between the constrained optimization and the Pareto front.  The problem of constrained multi-objective learning is highly relevant to robot learning and could make a good contribution to CoRL.\n\nA key strength of this paper is that the results achieved appear to be significant over baselines and the experimental evaluation is quite extensive.  I feel that this work will provide a useful contribution to the set of RL tools and will likely be used.  The clarity of this paper is also good considering the wide range and complexity of methods discussed.  The writing is well polished.\n\nMy main issue is that the method in this paper feels incremental compared to MO-MPO. The authors apply well known methods from constrained policy optimization to the multi objective framework.   The connections found between constrained optimization and the Pareto front are well known in other multi-objective problems but have not been explored in RL, to my knowledge.  Overall, this does help improve the novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am learning towards recommending acceptance of this work and think that it will be useful to the audience of CoRL.  The problem is highly relevant, and the connections made are intuitive.  The evaluation of the method is convincing and the writing is clear.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Mainly concerning the novelty over MO-MPO.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes one general framework which can deal with the constrained multi-objective RL problems. Reviewers own the consistent comments that this framework is promising and useful in some sense. Experimental results are also convincing. However, there are still some concerns raised by reviewers, which need to be well answered to make this work much more qualified. For example, multi-objective setting in the experiments sounds like not so complex and general. Some empirical tuning and algorithmic design should be presented more clearly. 

Although there are still some minor issues in this paper, most of reviewers think authors have addressed the majority of the concerns during the rebuttal period and it has met the quality of CoRL paper. 
Decision: Accept (Poster)