Year: 2020

Abstract:
Many real-world problems require that reinforcement learning (RL) agents learn policies that not only maximize a scalar reward, but do so while meeting constraints, such as remaining below an energy consumption threshold. Typical approaches for solving constrained RL problems rely on Lagrangian relaxation, but these suffer from several limitations. We draw a connection between multi-objective RL and constrained RL, based on the key insight that the constraint-satisfying optimal policy must be Pareto optimal. This leads to a novel, multi-objective perspective for constrained RL. We propose a framework that uses a multi-objective RL algorithm to find a Pareto front of policies that trades off between the reward and constraint(s), and simultaneously searches along this front for constraint-satisfying policies. We show that in practice, an instantiation of our framework outperforms existing approaches on several challenging continuous control domains, both in terms of solution quality and sample efficiency, and enables flexibility in recovering a portion of the Pareto front rather than a single constraint-satisfying policy.

heading: 1 INTRODUCTION
text: Deep reinforcement learning (RL) has shown great potential for training policies that optimize a single scalar reward. Recent approaches have exceeded human-level performance on Atari (Mnih et al., 2015) and Go (Silver et al., 2016), and have also achieved impressive results in continuous control tasks, including robot locomotion (Lillicrap et al., 2016; Schulman et al., 2017), acrobatics (Peng et al., 2018), and real-world robot manipulation (Levine et al., 2016; Zeng et al., 2019).
However, many problems, especially in the real world, require that policies meet certain constraints. For instance, we might want a factory robot to optimize task throughput while keeping actuator forces below a threshold, to limit wear-and-tear. Or, we might want to minimize energy usage for cooling a data center while ensuring that temperatures remain below some level (Lazic et al., 2018).
Such problems are often encoded as constrained Markov Decision Processes (CMDPs) (Altman, 1999), where the goal is to maximize task return while meeting the constraint(s). Typical approaches for solving CMDPs use Lagrangian relaxation (Bertsekas, 1999) to transform the constrained optimization problem into an unconstrained one. However, existing Lagrangian-based approaches suffer from several limitations. First, because the relaxed objective is a weighted-sum of the task return and constraint violation, this assumes a convex Pareto front (Das & Dennis, 1997). In addition, when the constraint is difficult to satisfy, in practice policies can struggle to obtain task reward. Finally, such approaches typically produce a single policy that satisfies a specific constraint threshold. However, the exact constraint threshold may not be known in advance, or one may prefer to choose from a set of policies across a range of acceptable thresholds.
We aim to achieve the goal of CMDPs (i.e., finding a constraint-satisfying policy that maximizes task return) while avoiding these limitations, by introducing a novel, general framework based on multi-objective MDPs (MO-MDP). A MO-MDP can be seen as a CMDP where the constrained objectives are instead unconstrained. Our key insight is that if we have access to the Pareto front, then we can find the optimal constraint-satisfying policy by simply searching along this front. However, finding the entire Pareto front is unnecessary if only a relatively small portion of the policies along the Pareto front meet the constraints. Therefore, we propose to also simultaneously prioritize learning for the preferences (i.e., trade-offs between reward and cost) that are most likely to produce policies that satisfy the constraints, and thus cover the relevant part of the Pareto front.
To our knowledge, there is no existing framework for applying multi-objective RL algorithms to constrained RL problems. Our main contribution is a general framework that enables this, by learning which preferences produce constraint-satisfying policies. This framework can be combined with any multi-objective RL algorithm that learns an approximate Pareto front of policies. Our second contribution is to extend a state-of-the-art multi-objective RL algorithm, multi-objective maximum a posteriori policy optimization (MO-MPO) (Abdolmaleki et al., 2020), to learn the Pareto front in a single training run. We use this extension of MO-MPO within our framework, and call the resulting algorithm constrained MO-MPO. We show in practice, constrained MO-MPO outperforms existing approaches on challenging continuous control tasks with constraints.

heading: 2 RELATED WORK
text: Constrained reinforcement learning. Constrained RL algorithms seek policies that meet the desired constraints at deployment time. Most approaches use Lagrangian relaxation (Bertsekas, 1999). Recent Lagrangian-based approaches claim convergence to constraint-satisfying (Tessler et al., 2019) or optimal (Paternain et al., 2019) solutions, but this is debatable (Szepesvari, 2020). Recent works seek to stabilize this optimization by approximating the reward and cost functions with convex relaxations (Yu et al., 2019) or by utilizing derivatives of the constraint function (Stooke et al., 2020). Other works have applied Lagrangian relaxation to mean-value constraints (Tessler et al., 2019), convex set constraints (Miryoosefi et al., 2019), and local constraint satisfaction (Bohez et al., 2019).
Existing Lagrangian approaches involve linear scalarization, however, and thus cannot find solutions that lie on concave portions of the true Pareto front (Das & Dennis, 1997). In contrast, we build on a multi-objective RL algorithm that does not rely on scalarization, and in practice, our approach can indeed find constraint-satisfying solutions on a concave Pareto front (see Sec. 5.1, humanoid walk).
Safe reinforcement learning. In safe RL, the aim is to achieve constraint satisfaction not only during deployment, but also during learning (Garcı́a & Fernández, 2015). Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training (Achiam et al., 2017; Berkenkamp et al., 2017; Chow et al., 2018; 2019; Yang et al., 2020; Zhang et al., 2020). These approaches require, however, that the initial policy meets (or almost meets) the constraints. Otherwise, performance degrades substantially and constrained policy optimization (CPO) (Achiam et al., 2017), for example, performs worse than Lagrangian-based approaches (Ray et al., 2019). The aim of our work is to find better solutions for constrained RL, rather than safe RL. We discuss in Sec. 6 how our method can be extended to reduce constraint violation during training.
Multi-objective reinforcement learning. Our approach is built on ideas from multi-objective RL (MORL), which consists of single-policy and multi-policy approaches. Single policy methods learn a policy that is optimal for a given setting of reward preferences. Most rely on linear scalarization (Roijers et al., 2013), which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales. Non-linear scalarizations have been proposed (Tesauro et al., 2008; Van Moffaert et al., 2013; Golovin & Zhang, 2020), but these are harder to combine with value-based RL and have seen limited use in deep RL. Recently, Abdolmaleki et al. (2020) introduced MO-MPO, where the preferences represent per-objective constraints on the policy improvement step. MO-MPO does not rely on scalarization and is thus invariant to reward scales.
Multi-policy MORL aims to find a set of policies that covers the whole Pareto front. Recent approaches learn a manifold in parameter space that optimizes the hypervolume of the Pareto front (Parisi et al., 2016; 2017). While such approaches could be combined with our framework, their scalability to deep RL remains to be shown. Other works combine single policy approaches with a general objective to optimize hypervolume (Xu et al., 2020). The instantiation of our framework is similar in spirit to such two-level methods, but applied to a different problem setting.

heading: 3 BACKGROUND AND NOTATION

heading: 3.1 CONSTRAINED MARKOV DECISION PROCESSES
text: A constrained Markov Decision Process (CMDP) consists of states s ∈ S, actions a ∈ A, an initial state distribution p0(s), a transition function p(s′|s, a), reward functions {rk(s, a)}Kk=0, constraint
thresholds {ck}Kk=1, and a discount factor γ ∈ [0, 1]. The (K + 1) reward functions consist of a task reward r0 and constrained rewards r1:K ; we will refer to these together as objectives.
A policy π(a|s) maps from a given state to a distribution over actions. The optimal solution to a constrained MDP is a policy that maximizes the expected return for the task reward r0, while ensuring that the expected return for all constrained rewards r1:K satisfy their respective thresholds:
max π
Eπ [∑
t
γtr0(st, at) ] s.t. Eπ [∑
t
γtri(st, at) ] ≥ ci ∀i = 1, ...,K , (1)
where Eπ is shorthand for the expectation over trajectories when following the policy π, given a fixed initial state distribution. Here the constraint thresholds are lower bounds, without loss of generality.
The value function V πk (s) of a policy π is its expected return for objective rk when starting from state s. The action-value functionQπk (s, a) denotes the expected return for objective rk after taking action a in state s and thereafter acting according to the policy: Qπk (s, a) = rk(s, a)+ γEs′∼p(s′|s,a)[V πk (s′)].
Most approaches for solving CMDPs are based on Lagrangian relaxation, where the constrained problem is turned into the following unconstrained optimization problem:
min λ≥0 max π
Eπ [∑
t
γtr0(st, at) ] + K∑ k=1 λk ( Eπ [∑ t γtrk(st, at) ] − ci ) . (2)
These approaches alternate between optimizing for the Lagrange multipliers λk and the policy π.
While our multi-objective perspective on constrained RL aims to achieve the goal of CMDPs—finding a constraint-satisfying policy that maximizes task return—it does not directly solve (1). Nonetheless, we will show our approach outperforms those that directly solve (1) via Lagrangian relaxation.

heading: 3.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING
text: In order to achieve the goal of CMDPs, we propose to leverage advances in multi-objective RL. A multi-objective MDP (MO-MDP) is defined in the same way as a CMDP, except without constraint thresholds. Because there are multiple unconstrained objectives, there is not a single optimal policy. Instead, there is a set of optimal policies, called the Pareto front. A policy is Pareto optimal if there is no other policy that improves its return for one objective without decreasing return for another.
Preferences. Each policy on the Pareto front is the optimal policy for a particular setting of preferences (i.e., desired trade-off over objectives).1 This is typically encoded via a preference vector , in which each element k represents the relative importance of the corresponding objective rk.
Connection to CMDPs. Given a CMDP {S,A, p0, p, r0:K , c1:K , γ}, consider the corresponding MO-MDP {S,A, p0, p, r0:K , γ}. Regardless of the thresholds c1:K , the optimal policy for the CMDP must lie on the Pareto front for this corresponding MO-MDP.2 Thus, if we knew the Pareto front, we could simply search along it to find the constraint-satisfying policy that maximizes task return.
Multi-objective Maximum a Posteriori Policy Optimization (MO-MPO). Whereas most MORL algorithms are based on linear scalarization, MO-MPO takes a distributional approach to training policies. For each policy improvement step, MO-MPO first computes a non-parametric policy for each objective rk that improves the (parametric) policy with respect to that objective, subject to a non-negative constraint k on the KL-divergence between the improved and old policies. Then, the policy is updated via supervised learning on the sum of these non-parametric policies. Intuitively, k defines the influence of objective rk on the final policy; an objective with larger k has more influence. However, MO-MPO can only train policies for a single preference setting . Sec. 4.1 explains how we extend MO-MPO to train preference-conditioned policies π(a|s, ), in order to use it in our framework. We choose to extend MO-MPO because it does not suffer from the limitations of linear scalarization, and has been shown to outperform such approaches (Abdolmaleki et al., 2020).
1This assumes that the choice of preference encoding does not restrict optimal solutions to the convex portions of the Pareto front. Although linear scalarization does not satisfy this requirement, other kinds of preference encodings do, for instance Chebyshev scalarization (Van Moffaert et al., 2013) or the per-objective KL-divergence constraints in MO-MPO, described later in this section.
2This is because if the optimal policy for the CMDP is not Pareto optimal, then there must exist another policy that obtains higher task return while meeting the constraints, which leads to a contradiction.

heading: 4 APPROACH
text: Problem Statement. For a given CMDP, we seek to find the preference vectors in the corresponding MO-MDP that produce constraint-satisfying action policies π(a|s, ). To do this, we want to find the distribution π( ) that maximizes the probability that the action policies satisfy the constraints.3
Overview of Framework. Our framework enables applying multi-objective RL algorithms to solve constrained RL problems. It is inspired by the observation that for any CMDP, there exist preference settings for the objectives that lead to the desired constraint-satisfying solutions. We propose to find these preference settings by optimizing for a hierarchical policy πψ( )πθ(a|s, ) that first selects a preference setting and then takes an action conditioned on . We also maintain a policy evaluation function Qπk (a, s, ) for each objective.
To train this policy, we decouple learning the preference policy πψ( ) from learning the action policy πθ(a|s, ). Formally, we alternate between optimizing two sub-problems:
1. Learn a new action policy πnew(a|s, ) given the current distribution of preferences πold( ). 2. Learn a new preference policy πnew( ) given the current action policy πold(a|s, ) .
We aim to converge to an optimal policy π∗( )π∗(a|s, ) that not only satisfies the constraints but is also Pareto optimal. Note that we can use different time scales for alternating between Steps 1 and 2. One extreme is to first fully optimize for a Pareto front curve given an initial preference distribution, and then fully optimize for solutions that satisfy the constraints. In our experiments we take the other extreme: we alternate between taking one learning step on πψ( ) and πθ(a|s, ) independently. For Step 1, any multi-objective RL algorithm capable of training preference-conditioned policies can be used. Sec. 4.1 describes how we use MO-MPO for Step 1, which requires non-trivially extending it to train preference-conditioned policies. For Step 2, the notion of learning a preference policy is the core novel component of our framework: this is what enables us to apply multi-objective RL algorithms to constrained RL problems. Sec. 4.2 describes how to do this.

heading: 4.1 LEARNING PREFERENCE-CONDITIONED ACTION POLICIES (STEP 1)
text: In this section, we explain how we extend MO-MPO (Abdolmaleki et al., 2020) to learn preferenceconditioned policies for the current preference distribution πold( ).
MO-MPO can only train policies π(a|s) for a single preference , so it cannot be directly used. This section describes how we extend MO-MPO to train a single policy πθ(a|s, ) conditioned on preference parameters ∼ πold( ), that represents the entire Pareto front. This is a non-trivial extension, that requires making both the action policy and Q-functions preference-conditioned, and replacing the scalar temperature with a preference-conditioned temperature network, as well as modifying the underlying MO-MPO optimization principles. We also introduce hindsight relabeling of preferences to stabilize off-policy learning and improve sample efficiency.
Our extended version is a policy iteration algorithm with two steps:
• Policy evaluation: learn preference-conditioned Q-functions for all objectives. • Policy improvement: improve the preference-conditioned policy according to Q-functions.
Preference-conditioned policy evaluation. In this step, we train a separate Q-function per objective (coined Q-decomposition by Russell & Zimdars 2003), to evaluate the current policy πold( )πold(a|s, ) under the state distribution µ(s). To learn the Q-function, for each learning step we sample L transitions from the replay buffer {si, ai, {rki }Kk , s′i}Li . Since our Q-functions Qπk (a, s, ) are also conditioned on preferences, we use hindsight relabeling to augment the states with preference parameters sampled from the current preference policy i ∼ πold( ), resulting in transitions {[si, i], ai, {rki }Kk , [s′i, i]}Li . Any policy evaluation algorithm can be used to learn these Q-functions. We use distributional policy evaluation with 5-step return (Barth-Maron et al., 2018).
Preference-conditioned policy improvement. The policy improvement step assumes a state distribution µ(s), per-objective Q functions Qoldk (s, a, ), the current action policy πold(a|s, ), and the current preference policy πold( ). It consists of two sub-steps: 1) finding per-objective improved
3Without loss of generality, we consider a state-independent preference policy π( ) when deriving the update rules. In Sec. 5.3, we show our approach can also be used to learn state-dependent preference policies π( |s).
action distributions, and 2) distilling these distributions into a new preference-conditioned action policy via supervised learning.
Finding per-objective distributions: To find per-objective improved action distributions qk(a|s, ), we optimize the following RL optimization problem for each objective:
max qk
Eπold( )µ(s) [ ∫
a
qk(a|s, )Qoldk (s, a, ) da ]
(3) s.t. Eµ(s) [ KL(qk(a|s, )‖πold(a|s, )) ] < k ∀ ∼ πold( ) .
We can solve this problem in closed form to obtain qk(a|s, ) ∝ πold(a|s, ) exp (Qoldk (s, a, )
ηωk( k)
) , (4)
where ηωk( k) is a preference-dependent temperature function for objective rk, and is parameterized by ωk. This temperature function is obtained by minimizing the following dual function:
g(ωk) = Eπold( )µ(s) [ ηωk( k) ( k + log ∫ a πold(a|s, ) exp (Qoldk (s, a, ) ηωk( k) ) da )] . (5)
In practice, we maintain a single function ηω( ) with shared parameters ω for all objectives.4
To approximate the expectations over the preference distribution and state distribution, we draw L states from the replay buffer and L preferences from the preference policy πold( ). To approximate the integrals over a, for each (s, ) pair we sample M actions from the current policy πold(a|s, ). Learning a new parameterized action policy: After obtaining per-objective improved policies, we use supervised learning to distill these distributions into a new parameterized policy:
max θ K∑ k=0 Eπold( )µ(s) [ KL(qk(a|s, )‖πθ(a|s, )) ] s.t. Eπold( )µ(s) [ KL(πold(a|s, )‖πθ(a|s, )) ] < β,
subject to a trust region with bound β > 0 for more stable learning. To solve this optimization, we use Lagrangian relaxation as described in Abdolmaleki et al. (2018; 2020).

heading: 4.2 LEARNING PREFERENCE POLICIES (STEP 2)
text: We will now explain how to optimize the preference distribution πψ( ) for selecting action policies πold(a|s, ) with better constraint satisfaction. To achieve this, we define a fitness function fk that evaluates satisfaction of constraint threshold ck, given the current action policy πold(a|s, ) and Q-function Qoldk (s, a, ). The fitness function should be chosen based on the problem and form of the constraint. In our empirical evaluation, for equality constraints we use a fitness function that penalizes the difference between the expected Q-values and constraint threshold, and for inequality constraints it penalizes the amount the expected Q-values fall under the threshold:
f eqk ( ) = − ∣∣∣Eµ(s)[Qoldk (s,Eπold(a|s, )[a], )]− ck∣∣∣ (6)
f ineqk ( ) = min ( 0, Eµ(s) [ Qoldk (s,Eπold(a|s, )[a], ) ] − ck ) . (7)
Given a chosen fitness function, we now can use any off-the-shelf RL algorithm to optimize the preference policy πψ( ), initialized by πold( ), to maximize the fitness function. See Appendix D for more details on the general procedure underlying our algorithm (Appendix, Algorithm 1).

heading: 5 EXPERIMENTS
text: We find that our approach of Pareto front optimization applied to MO-MPO (i.e., constrained MOMPO), finds solutions that are on par with those found by MORL algorithms (Appendix C.1). In
4The optimization problem in (3) generalizes the one in MO-MPO, which assumes fixed KL-constraints , to a distribution πold( ) over KL-constraints. Thus (4) and (5) can be derived by following steps analogous to those given in Abdolmaleki et al. (2020).
our experiments, we compare constrained MO-MPO against existing RL algorithms for constrained MDPs: for a number of challenging continuous control tasks, we analyze the quality of policies found for a range of constraint thresholds. Unless otherwise mentioned, we use (6) as the fitness function. For action policies we use Gaussian distributions parameterized by neural networks, and for the preference policy in constrained MO-MPO, we use a discrete distribution. Architecture details and learning hyperparameters are described in Appendix A.
Domains. We evaluate our approach in two continuous control domains. First, we use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al., 2018). This domain has a challenging 21-dimensional action space. The agent receives a shaped reward for running at 10 m/s or walking at 1 m/s. The constraint is imposed on the expected negative control norm (i.e., −‖a‖2), which roughly captures the “energy” expended by the agent. This is relevant for real-world settings (e.g., in robotics or control tasks), where energy consumption often needs to be taken into account.
We also evaluate on the level-two point mass tasks from the Safety Gym suite (Ray et al., 2019), which we refer to as point goal, button, and push. The agent receives a sparse reward for either reaching a goal location or pushing a box to a goal. The constraint is imposed on the expected per-episode cost, which is incurred by running into or over objects. There are several types of objects, each with its own cost. Unless otherwise mentioned, the constraint is with respect to the total cost.
Pareto plots. When we plot learned Pareto fronts for the tasks (e.g., in Fig. 1, top), the x-coordinate is average per-episode negative cumulative cost and the y-coordinate is average per-episode task reward.5 For both axes, higher values are better. All trained policies for the same task are evaluated on the same set of randomly-initialized environments. For constrained MO-MPO policies, we sample a new preference at the beginning of each episode, rather than at every timestep.
Baselines. As a baseline we consider Lagrangian relaxation; because of its simplicity and effectiveness it is a common choice (Ray et al., 2019; Stooke et al., 2020). To ensure a fair comparison with our approach, we use MPO to train policies to optimize the Lagrangian dual objective (2). We obtain state-of-the-art results on Safety Gym tasks with this MPO-Lagrangian baseline (Appendix C.2).

heading: 5.1 QUALITY OF SOLUTIONS
text: We evaluated our approach and the baseline for constraint thresholds linearly spaced in the range [−4,−1] for humanoid run, [−2,−0.5] for humanoid walk, and [−15,−2] for point tasks. For constrained MO-MPO, the preference policy π( ) is initialized to uniform and is a discrete distribution over 100 linearly-spaced values starting at 10−5 and up 0.15 for humanoid run, 0.3 for humanoid walk, and 0.2 for the point tasks. Results are shown in Fig. 1.
We observe the biggest difference in performance for humanoid walk: the Lagrangian baseline only finds policies at the extremes, whereas our approach finds constraint-satisfying solutions that achieve non-zero task reward. This may be because the ground-truth Pareto front for this task is concave (Sec. 2). On humanoid run, point goal, and point button, the baseline and our approach find similar-quality solutions for easier constraint thresholds, but as the constraint threshold becomes more difficult to satisfy (i.e., smaller in magnitude), our approach dominates (Fig. 1, top row; Fig. 2a). The only exception is point push, for which both approaches find similar-quality solutions for all constraint thresholds. We hypothesize that this is because in this particular task, the cost does not conflict much with the task objective, so it is easier to find high-task-reward solutions that incur small cost.
Our approach always finds policies that satisfy the constraints for humanoid, whereas the Lagrangian baseline occasionally violates the constraint.6 For Safety Gym tasks, the policies found by both our approach and the baseline slightly violate the constraints (Fig. 1, bottom row); this is due to Q-function underestimates for the cost, rather than a limitation of the policy optimization (Appendix C.4).
5The action norm constraint for the humanoid tasks is per-timestep with 1000 timesteps per episode, so a policy that exactly meets a constraint threshold of −2, has an average cumulative episodic cost of −2000.
6With longer training, these violations might disappear, but we do not expect the task reward to improve. These violations are due to the relatively unstable behavior of the Lagrangian baseline: when the constraint starts to be violated, this leads to a gradual increase in the Lagrange multiplier, and it takes some time for this to propagate into changes in the policy. This can be seen empirically in Fig. 2b (top row) where the average cost dips below (i.e., violates) the constraint threshold, indicated by the dotted line.
Our approach is also significantly more sample-efficient than the baseline. For instance on humanoid run, constrained MO-MPO learns constraint-satisfying policies that reach reasonably high task reward mid-way through training, whereas the majority of MPO-Lagrangian policies obtain zero task reward at the same point (Fig. 3a, 3b). This could be because with two conflicting objectives, MPO-Lagrangian cannot optimize for both objectives at once, whereas our approach can (Fig. 2b).
Multiple random seeds. We additionally ran experiments with five random policy initializations per constraint threshold, for a subset of four thresholds. The results support the conclusions drawn above. In particular, our approach significantly outperforms the baseline for the harder-to-satisfy constraint thresholds, in four out of the five tasks (see Appendix C.3 for details).
Scaling to multiple constraints. We also investigated how well constrained MO-MPO scales to multiple constraints, for a more difficult version of point goal (see Appendix B.2 for details). There are two constraints: one on the cost of crossing over hazards, and one on bumping into vases. For each type of cost, we fix its threshold at −2 and vary the other between −10 and 0. The baseline fails to obtain task reward when the constraint for hazards is fixed at the challenging threshold of −2, whereas constrained MO-MPO finds a range of solutions (Fig. 3c, right).

heading: 5.2 FLEXIBILITY
text: Since our approach decouples Pareto front optimization from Pareto exploration, by choosing the appropriate fitness function, one can in theory recover any portion of the Pareto front. This is useful because in practice, an RL practitioner may not know the true constraint, and will be better able to
choose which policy to deploy after observing a couple of options along the Pareto front. Typical approaches to constrained RL find a single policy that meets the constraint exactly.7
On humanoid run, we evaluate constrained MO-MPO using (7) as the fitness function for several inequality constraint thresholds. We find that our approach indeed learns a portion of the Pareto front that satisfies the constraint (Fig. 4a). These plots are obtained by taking the preferences at regular percentiles between 5th and 95th percentiles of the learned preference distribution, and evaluating the policy conditioned on each of these preferences, on the same set of randomly-initialized environments.

heading: 5.3 STATE-DEPENDENT PREFERENCES
text: By using a state-dependent preference policy π( |s), we can extend our approach to satisfy constraints per-initial-state, rather than in expectation.8 To do this, we train a neural network that maps the state to a discrete distribution over preference settings. We implement this for the MPO-Lagrangian baseline by making the Lagrange multiplier state-dependent as well, as introduced by Bohez et al. (2019); this trains policies that satisfy the constraint per-state—there is no equivalent for the Lagrangian approach that satisfies constraints per-initial-state.
We use both approaches to train policies for humanoid run, for the same settings of constraint thresholds as in Sec. 5.1. Our approach finds constraint-satisfying policies that achieve higher task reward than the baseline (Fig. 4b). The constraint violation is computed by summing the per-episode
7Although it is possible to extend the Lagrangian baseline by conditioning the policy on the threshold, given that constrained MO-MPO finds better solutions, we choose to focus evaluation here on constrained MO-MPO.
8In our implementation, we sample a preference from the preference policy only at the start of every episode, rather than per-timestep. If we did the latter, then we could satisfy per-state constraints.
constraint violation, obtained by comparing the average per-timestep action norm cost for that episode against the constraint threshold.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this paper, we introduced a novel framework for constrained RL, by leveraging the ability of multiobjective RL algorithms to find Pareto-optimal solutions. This framework can be combined with any algorithm that finds a Pareto front of solutions. Our empirical results show that an instantiation of this framework, constrained MO-MPO, outperforms the commonly-used Lagrangian relaxation approach in terms of solution quality, stability, and sample-efficiency. In particular, the Lagrangian approach struggles to train policies that obtain task reward when the constraints are difficult to meet, or when the ground-truth Pareto front is concave. Our framework also enables flexibility via the choice of fitness function and can be extended to meet constraints per-initial-state, rather than in expectation.
One limitation of this work is that the support of the preference policy’s discrete distribution must include preference setting(s) that satisfy the constraint threshold. In practice, this is straightforward to do for MO-MPO, because the k encode preferences in a way that is independent of the objectives’ reward scales. In future work, it is worth exploring other distributions for the preference policy that would overcome this limitation, for instance a mixture of Gaussians.
Another limitation is this work cannot be applied directly to safe RL. We plan to extend our approach to reduce the cost incurred during training, perhaps by initializing with a conservative distribution over preferences, that prioritizes minimizing cost. Finally, we also plan to exploit the flexibility of our framework to solve constrained RL problems with more than one unconstrained objective.

heading: B EXPERIMENTAL DOMAINS

heading: B.1 HUMANOID
text: We use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al., 2018).9 The observation space is 67-dimensional and the action space is 21-dimensional. Observations consist of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions. Actions correspond to joint accelerations; the minimum and maximum action limits are −1 and 1, respectively. Each episode is 1000 timesteps. The unconstrained task reward is given by the environment: there is a shaped reward for maintaining a horizontal speed (in any direction) of 10 m/s for humanoid run and 1 m/s for humanoid walk. The shaped reward is equal to min(h/h∗, 1), where h is the speed of the agent and h∗ is the target speed, both in m/s. The constrained reward is the negative l2-norm of the action vector: rcost(s, a) = −‖a‖2. This can be thought of as limiting the energy usage of the agent.
9Available at github.com/deepmind/dm control.

heading: B.2 SAFETY GYM
text: We use the level-two point mass tasks in Safety Gym: goal, button, and push (Ray et al., 2019).10 The action space is 2-dimensional, with minimum and maximum action limits of −1 and 1, respectively. The observation space is 60-dimensional for point goal, and 76-dimensional for point button and push. There are 1000 timesteps per episode. At the start of each episode, the entities (i.e., goal, agent, obstacles) are randomly initialized.
10Available at github.com/openai/safety-gym.
We use the default task configurations as used in Ray et al. (2019), for instance how many obstacles are spawned, the regions in which entities can be spawned, the size of entities, and the type of observations. The only difference is that we use sparse task reward instead of shaped task reward, since the former is more realistic.
A sparse task reward of 1 is given in goal when the agent enters the goal area, in button when the agent touches the target button, and in push when the agent pushes a box into the goal area. The constrained reward is the negative cumulative cost. In goal the agent incurs cost by being in hazardous regions or bumping into vases; in button the agent incurs cost by being in hazardous regions, bumping into gremlins, or touching non-target buttons; in push the agent incurs cost being in hazardous regions or bumping into pillars. All obstacles are static, except for gremlins, which move in a fixed circular pattern; the agent moves vases when it bumps into them.
In our experiments with two constraints (Sec. 5.1), we use a more difficult variant of the point goal task, with two changes. First, the length of the square area in which entities are spawned is three-quarters of the original length, which makes the density of obstacles higher and thus more difficult for the agent to navigate around. In addition, the radius of the goal region is two-thirds the original radius, which allows for it to be placed in more difficult-to-reach locations: because the random initialization of the scene enforces that there are no collisions between the entities, the smaller the goal is, the more potential open areas it could be placed in.

heading: C ADDITIONAL EXPERIMENTS AND ANALYSIS

heading: C.1 COMPARISON TO MULTI-OBJECTIVE RL ALGORITHMS
text: We first evaluate whether constrained MO-MPO is able to find solutions that lie on the Pareto front. Since the ground-truth Pareto front is not available in general, we approximate this for humanoid run
by training policies with MO-MPO for a range of preference settings: task = 0.1 and norm at linearly spaced intervals between 10−5 and 0.15.
We trained preference-conditioned policies with five random initializations, for the same task and norm sampled uniformly from the same range; we call this approach controllable MO-MPO. This produces a Pareto front that is comparable to MO-MPO’s, both in terms of coverage and solution quality (Fig. 5, middle right).
It is also possible to train preference-conditioned policies where the preferences correspond to weights for linear scalarization, rather than KL-divergence bounds as in MO-MPO. So as a baseline, we train weight-conditioned policies with MPO (Abdolmaleki et al., 2018); we call this baseline controllable MPO. These policies were conditioned on a wtask of 1.0 and wnorm sampled uniformly between 0 and 0.15. Compared to controllable MO-MPO, we had to train controllable MPO for longer (900M actor steps instead of 500M) in order to reach reasonable task performance, and controllable MPO was still unable to find solutions with low action norm (Fig. 5, middle left). This is as expected: Abdolmaleki et al. (2020) show that MO-MPO outperforms MPO with linear scalarization on humanoid run, so one would expect controllable MO-MPO to also outperform controllable MPO. We tried running controllable MPO with higher weights, but these did not show any learning progress within 500M actor steps.
Policies trained with constrained MO-MPO for equality constraint thresholds from −4 to −1 also lie on the MO-MPO Pareto front (Fig. 5, right). We train a separate preference-conditioned action policy and preference policy for each unique constraint threshold. This supports that learning the preference distribution simultaneously while training the policy and Q-functions does not negatively impact the final solutions that are found.

heading: C.2 MPO LAGRANGIAN BASELINE
text: The baseline we use in our empirical evaluation is MPO with Lagrangian relaxation. We ran this baseline on all Safety Gym point mass tasks, using the default settings (including dense reward) so that
we can directly compare against the constraint-satisfying policies obtained in Ray et al. (2019).11 Our MPO-Lagrangian baseline compares favorably with those in Ray et al. (2019), that instead combine Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) with Lagrangian relaxation (Fig. 6). This is with the caveat that policies were trained with TRPO-Lagrangian and PPO-Lagrangian for 10 million actor steps, whereas our training curves for MPO-Lagrangian go up to 100 million actor steps.

heading: C.3 MULTIPLE RANDOM SEEDS
text: To verify that the empirical results we observed in Sec. 5.1 are significant, we used our approach and the baseline to train policies for the same constraint threshold, starting from five random initializations. We evaluated this for four constraint thresholds per task, selected from the ranges in Table 3. Results are shown in Fig. 7.
Constrained MO-MPO achieves significantly higher task reward than the baseline for the harder-tosatisfy constraint thresholds in four out of the five tasks. For the other constraint thresholds, both
11Since we noticed that our Q-functions underestimate cost (Appendix C.4), we use a cost threshold of 15 for training, rather than the actual cost threshold of 25.
−20 0 Q-value estimate
−20
−10
0
ac tu
al co
st
point goal
−20 0 Q-value estimate
point button
−20 0 Q-value estimate
point push
MPO Lagrangian constrained MO-MPO
(a) The learned Q-function for cost underestimates the cost per episode. The actual cost is obtained by evaluating the deterministic policy after 200M actor steps, on the same 100 randomly-initialized environments.
−20 0 Q-value estimate
−20
−10
0
co st
th re
sh ol
d
point goal
−20 0 Q-value estimate
point button
−20 0 Q-value estimate
point push
(b) For both approaches, almost all trained policies satisfy the constraint (i.e., are on or above the dotted line), according to the learned Q-values.
Figure 8: In the plots above, the Q-value estimate is obtained by averaging the Q-values for 100 batches of (s, a) pairs from the replay buffer, after 200M actor steps of training. Each point corresponds to a separately-trained policy, for a different constraint threshold
approaches achieve similar task reward. In terms of satisfying constraint thresholds, our approach performs on-par with or better than the baseline.
As mentioned, the slight violation of constraint thresholds for the Safety Gym point tasks is due to underestimation of Q-values, rather than a limitation of the policy improvement method; this is described in the following subsection.

heading: C.4 Q-FUNCTION ESTIMATES
text: We noticed that for the Safety Gym point mass tasks, both our approach and the Lagrangian baseline train policies that slightly exceed the constraints (Fig. 1, bottom row). After digging deeper, we realized that this results from suboptimal Q-function learning, rather than suboptimal policy optimization. The learned Q-values for the cost consistently underestimate the actual cost incurred (Fig. 8a). The policy optimization finds a policy that meets the constraints, assuming that the learned Q-values for the cost are accurate (Fig. 8b).
Thus, better constraint satisfaction can be obtained by improving policy evaluation, so that estimated Q-values are more accurate—this is orthogonal to our proposed approach, which is focused on policy improvement.

heading: D ALGORITHMIC DETAILS
text: In this section, we give details on how to learn the preference policy and provide an algorithm box for the policy improvement step. We also describe the overall procedure we use to collect data and learn from the data.

heading: D.1 LEARNING PREFERENCE-CONDITIONED POLICIES
text: As discussed in the main paper, we also learn a new preference policy πψ( ) given the current preference policy πold( ) and fitness functions fk( ) (that evaluate in terms of constraint satisfaction). One could add the fitness functions together to obtain ∑ k fk( ), and use any off-the-shelf RL algorithm to optimize for a new preference policy. On the other hand, this problem can be been as a multi-objective problem where the fitness for each constraint k is an objective. To this end, we use MO-MPO for learning πψ( ). Note that in this paper we have at most two constraints. In the case of a problem with one constraint (i.e., one “objective”), MO-MPO (Abdolmaleki et al., 2020) reduces to MPO (Abdolmaleki et al., 2018).
More formally, following the MO-MPO algorithm, we optimize the following constrained optimization problem, that can be solved via Lagrangian relaxation (Abdolmaleki et al., 2020):
max ψ K∑ k=1 KL(pk( )‖πψ( )) s.t. KL(πold( )‖πψ( )) < δ ,
where δ defines a trust region for more stable learning and pk( ) is a non-parametric improved policy for each objective, i.e,
pk( ) ∝ πold( ) exp (fk( ) ϕk ) .
ϕk is a temperature variable that we maintain for each objective (or fitness function) and is obtained by optimizing the convex dual function
g(ϕk) = αk + log
∫ πold( ) exp (fk( ) ϕk ) d , (8)
where αk for each objective k defines a desired KL-divergence bound between the new improved policy pk( ) and current policy πold(ψ). We use the same value of αk = 0.1 for all fitness functions k throughout the paper. For more details on MO-MPO, please refer to (Abdolmaleki et al., 2020).
Note that this section assumes that the preference policy is state-independent, i.e., π( ). The optimization is analogous for a state-dependent preference policy π( |s), which we consider in Sec. 5.3.

heading: D.2 GENERAL ALGORITHM
text: We maintain one online network and one target network for each Q-function, action policy and preference policy. We also maintain one online network for the temperature function. Target networks are updated every fixed number of steps by copying parameters from the online network. Online networks are updated using gradient descent in each learning iteration. We refer to the target networks by using the subscript/superscript “old” throughout the paper.
We use an asynchronous actor-learner setup. In this setup actors fetch policy parameters from the learner and act in the environment and write transitions to the replay buffer. Note that at the beginning of each episode, we first sample one preference parameter from the preference policy, which remains fixed until the end of the episode. The learner uses the transitions in the replay buffer to update the Q-functions, policies and temperature functions. Algorithm 1 describes one step of policy improvement for constrained MO-MPO.
Algorithm 1: Constrained MO-MPO: One policy improvement step 1: given batch size (L), number of actions and to sample (M ), current policy πold( )πold(a|s, ), current
(K + 1) Q-functions {Qoldk (s, a, )}Kk=0, temperature network ηω( ), (K) constraint fitness functions {fk( )}Kk=1, (K) temperature variables {ϕk}Kk=1, (K) KL bounds {αk}Kk=1, replay buffer D, first-order gradient-based optimizer O
2: 3: initialize πθ(a|s, ) from the parameters of πold(a|s, ) 4: initialize πψ( ) from the parameters of πold( ) 5: repeat 6: // Collect dataset {si, i, aij , Qijk } L,M,K i,j,k , where 7: // L states si ∼ D 8: // L preferences i ∼ πold( ) 9: // M actions aij ∼ πold(a|si, i) and Qijk = Q old k (s
i, i, aij) 10: 11: // Compute (non-parametric) action distribution for each objective 12: δω ← ∇ω ∑ k 1 L ∑L i ηω( i)[k] [ ik + log ∑M j 1 M exp ( Q ij k
ηω( i)[k]
)] , where [k] is the index of the vector
13: Update ω based on δω , using optimizer O 14: for k = 0, . . . , K do 15: qijk ∝ exp( Q ij k
ηω( i)[k] )
16: end for 17: 18: // Update action policy 19: δθ ← −∇θ ∑L i ∑M j ∑K k=0 q ij k log πθ(a
ij |si, i) 20: (subject to additional KL regularization) 21: Update πθ based on δπ , using optimizer O 22: 23: 24: // Collect dataset { i, f ik}M,Ki,k , where 25: // M preferences i ∼ πold( ) and f ik = fk( i) 26: 27: // Compute epsilon distribution for each constrained objective 28: for k = 1, . . . , K do 29: δϕk ← ∇ϕkϕkαk + ϕk log (∑ i 1 M exp ( fik ϕk
)) 30: Update ϕk based on δϕk , using optimizer O 31: pik ∝ exp( fik ϕk ) 32: end for 33: 34: // Update preference policy 35: δψ ← −∇ψ ∑M i ∑K k=1 p i k log πψ(
i) 36: (subject to additional KL regularization) 37: Update πψ based on δψ , using optimizer O 38: 39: until fixed number of steps 40: return πold(a|s, ) = πθ(a|s, ) and πold( ) = πψ( )

Review summaries:
The paper is interested in reinforcement learning where one needs to satisfy constraints (for instance energy spent) in addition to maximizing rewards. The proposed approach proposes to extend any method able to approximate the Pareto front of optimal policies by also learning portions of the front that satisfy user constraints (given that only small portion of the front may be valid). In particular, a previous approach MO-MPO is used to approximate the Pareto front in a first step and a second step learns another policy find preference vectors satisfying given constrains. Experiments are carried in two environments and show that the proposed method outperforms Lagrangian baselines (e.g. optimizing alternatively for reward and constrain violation) in particular in settings where constraints are harder to satisfy.

In Reinforcement Learning (RL) it is common that one searches for policies that optimize the reward and that meet certain constraints. A common approach for handling the constraints is Lagrangian relaxation, i.e., to incorporate the constraints in some way into the objective. In this submission, a different approach is suggested, in which the constraints are treated as additional objectives and then the Pareto front of policies is computed. This is reasonable as the optimal policy for a single objective that satisfies certain constraints must be Pareto optimal when the constraints are interpreted as additional objectives. It is then explained how the new approach can be implemented and several experiments are conducted on well-known data sets. It turns out that it is indeed beneficial for some data set to compute the set of Pareto-optimal policies instead of using Lagrangian relaxation because in some cases Lagrangian relaxation finds only the extreme policies while the Pareto front also contains intermediate policies that balance the criteria better.

This paper introduced a general framework that incorporates multi-object reinforcement learning(MORL) perspective for constrained reinforcement learning to a policy set, called Pareto front, that meets the constrained. The author has instantiated a method based on the previous method MO-MPO. Compared to previous Lagrangian-based approaches, the proposed method has advantages in solution quality, stability, and sample-efficiency in few empirical environments.