Explicit Pareto Front Optimization for Constrained Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) has shown great potential for training policies that optimize a single scalar reward . Recent approaches have exceeded human-level performance on Atari ( Mnih et al. , 2015 ) and Go ( Silver et al. , 2016 ) , and have also achieved impressive results in continuous control tasks , including robot locomotion ( Lillicrap et al. , 2016 ; Schulman et al. , 2017 ) , acrobatics ( Peng et al. , 2018 ) , and real-world robot manipulation ( Levine et al. , 2016 ; Zeng et al. , 2019 ) . However , many problems , especially in the real world , require that policies meet certain constraints . For instance , we might want a factory robot to optimize task throughput while keeping actuator forces below a threshold , to limit wear-and-tear . Or , we might want to minimize energy usage for cooling a data center while ensuring that temperatures remain below some level ( Lazic et al. , 2018 ) . Such problems are often encoded as constrained Markov Decision Processes ( CMDPs ) ( Altman , 1999 ) , where the goal is to maximize task return while meeting the constraint ( s ) . Typical approaches for solving CMDPs use Lagrangian relaxation ( Bertsekas , 1999 ) to transform the constrained optimization problem into an unconstrained one . However , existing Lagrangian-based approaches suffer from several limitations . First , because the relaxed objective is a weighted-sum of the task return and constraint violation , this assumes a convex Pareto front ( Das & Dennis , 1997 ) . In addition , when the constraint is difficult to satisfy , in practice policies can struggle to obtain task reward . Finally , such approaches typically produce a single policy that satisfies a specific constraint threshold . However , the exact constraint threshold may not be known in advance , or one may prefer to choose from a set of policies across a range of acceptable thresholds . We aim to achieve the goal of CMDPs ( i.e. , finding a constraint-satisfying policy that maximizes task return ) while avoiding these limitations , by introducing a novel , general framework based on multi-objective MDPs ( MO-MDP ) . A MO-MDP can be seen as a CMDP where the constrained objectives are instead unconstrained . Our key insight is that if we have access to the Pareto front , then we can find the optimal constraint-satisfying policy by simply searching along this front . However , finding the entire Pareto front is unnecessary if only a relatively small portion of the policies along the Pareto front meet the constraints . Therefore , we propose to also simultaneously prioritize learning for the preferences ( i.e. , trade-offs between reward and cost ) that are most likely to produce policies that satisfy the constraints , and thus cover the relevant part of the Pareto front . To our knowledge , there is no existing framework for applying multi-objective RL algorithms to constrained RL problems . Our main contribution is a general framework that enables this , by learning which preferences produce constraint-satisfying policies . This framework can be combined with any multi-objective RL algorithm that learns an approximate Pareto front of policies . Our second contribution is to extend a state-of-the-art multi-objective RL algorithm , multi-objective maximum a posteriori policy optimization ( MO-MPO ) ( Abdolmaleki et al. , 2020 ) , to learn the Pareto front in a single training run . We use this extension of MO-MPO within our framework , and call the resulting algorithm constrained MO-MPO . We show in practice , constrained MO-MPO outperforms existing approaches on challenging continuous control tasks with constraints . 2 RELATED WORK . Constrained reinforcement learning . Constrained RL algorithms seek policies that meet the desired constraints at deployment time . Most approaches use Lagrangian relaxation ( Bertsekas , 1999 ) . Recent Lagrangian-based approaches claim convergence to constraint-satisfying ( Tessler et al. , 2019 ) or optimal ( Paternain et al. , 2019 ) solutions , but this is debatable ( Szepesvari , 2020 ) . Recent works seek to stabilize this optimization by approximating the reward and cost functions with convex relaxations ( Yu et al. , 2019 ) or by utilizing derivatives of the constraint function ( Stooke et al. , 2020 ) . Other works have applied Lagrangian relaxation to mean-value constraints ( Tessler et al. , 2019 ) , convex set constraints ( Miryoosefi et al. , 2019 ) , and local constraint satisfaction ( Bohez et al. , 2019 ) . Existing Lagrangian approaches involve linear scalarization , however , and thus can not find solutions that lie on concave portions of the true Pareto front ( Das & Dennis , 1997 ) . In contrast , we build on a multi-objective RL algorithm that does not rely on scalarization , and in practice , our approach can indeed find constraint-satisfying solutions on a concave Pareto front ( see Sec . 5.1 , humanoid walk ) . Safe reinforcement learning . In safe RL , the aim is to achieve constraint satisfaction not only during deployment , but also during learning ( Garcı́a & Fernández , 2015 ) . Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training ( Achiam et al. , 2017 ; Berkenkamp et al. , 2017 ; Chow et al. , 2018 ; 2019 ; Yang et al. , 2020 ; Zhang et al. , 2020 ) . These approaches require , however , that the initial policy meets ( or almost meets ) the constraints . Otherwise , performance degrades substantially and constrained policy optimization ( CPO ) ( Achiam et al. , 2017 ) , for example , performs worse than Lagrangian-based approaches ( Ray et al. , 2019 ) . The aim of our work is to find better solutions for constrained RL , rather than safe RL . We discuss in Sec . 6 how our method can be extended to reduce constraint violation during training . Multi-objective reinforcement learning . Our approach is built on ideas from multi-objective RL ( MORL ) , which consists of single-policy and multi-policy approaches . Single policy methods learn a policy that is optimal for a given setting of reward preferences . Most rely on linear scalarization ( Roijers et al. , 2013 ) , which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales . Non-linear scalarizations have been proposed ( Tesauro et al. , 2008 ; Van Moffaert et al. , 2013 ; Golovin & Zhang , 2020 ) , but these are harder to combine with value-based RL and have seen limited use in deep RL . Recently , Abdolmaleki et al . ( 2020 ) introduced MO-MPO , where the preferences represent per-objective constraints on the policy improvement step . MO-MPO does not rely on scalarization and is thus invariant to reward scales . Multi-policy MORL aims to find a set of policies that covers the whole Pareto front . Recent approaches learn a manifold in parameter space that optimizes the hypervolume of the Pareto front ( Parisi et al. , 2016 ; 2017 ) . While such approaches could be combined with our framework , their scalability to deep RL remains to be shown . Other works combine single policy approaches with a general objective to optimize hypervolume ( Xu et al. , 2020 ) . The instantiation of our framework is similar in spirit to such two-level methods , but applied to a different problem setting . 3 BACKGROUND AND NOTATION . 3.1 CONSTRAINED MARKOV DECISION PROCESSES . A constrained Markov Decision Process ( CMDP ) consists of states s ∈ S , actions a ∈ A , an initial state distribution p0 ( s ) , a transition function p ( s′|s , a ) , reward functions { rk ( s , a ) } Kk=0 , constraint thresholds { ck } Kk=1 , and a discount factor γ ∈ [ 0 , 1 ] . The ( K + 1 ) reward functions consist of a task reward r0 and constrained rewards r1 : K ; we will refer to these together as objectives . A policy π ( a|s ) maps from a given state to a distribution over actions . The optimal solution to a constrained MDP is a policy that maximizes the expected return for the task reward r0 , while ensuring that the expected return for all constrained rewards r1 : K satisfy their respective thresholds : max π Eπ [ ∑ t γtr0 ( st , at ) ] s.t . Eπ [ ∑ t γtri ( st , at ) ] ≥ ci ∀i = 1 , ... , K , ( 1 ) where Eπ is shorthand for the expectation over trajectories when following the policy π , given a fixed initial state distribution . Here the constraint thresholds are lower bounds , without loss of generality . The value function V πk ( s ) of a policy π is its expected return for objective rk when starting from state s. The action-value functionQπk ( s , a ) denotes the expected return for objective rk after taking action a in state s and thereafter acting according to the policy : Qπk ( s , a ) = rk ( s , a ) + γEs′∼p ( s′|s , a ) [ V πk ( s′ ) ] . Most approaches for solving CMDPs are based on Lagrangian relaxation , where the constrained problem is turned into the following unconstrained optimization problem : min λ≥0 max π Eπ [ ∑ t γtr0 ( st , at ) ] + K∑ k=1 λk ( Eπ [ ∑ t γtrk ( st , at ) ] − ci ) . ( 2 ) These approaches alternate between optimizing for the Lagrange multipliers λk and the policy π . While our multi-objective perspective on constrained RL aims to achieve the goal of CMDPs—finding a constraint-satisfying policy that maximizes task return—it does not directly solve ( 1 ) . Nonetheless , we will show our approach outperforms those that directly solve ( 1 ) via Lagrangian relaxation . 3.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING . In order to achieve the goal of CMDPs , we propose to leverage advances in multi-objective RL . A multi-objective MDP ( MO-MDP ) is defined in the same way as a CMDP , except without constraint thresholds . Because there are multiple unconstrained objectives , there is not a single optimal policy . Instead , there is a set of optimal policies , called the Pareto front . A policy is Pareto optimal if there is no other policy that improves its return for one objective without decreasing return for another . Preferences . Each policy on the Pareto front is the optimal policy for a particular setting of preferences ( i.e. , desired trade-off over objectives ) .1 This is typically encoded via a preference vector , in which each element k represents the relative importance of the corresponding objective rk . Connection to CMDPs . Given a CMDP { S , A , p0 , p , r0 : K , c1 : K , γ } , consider the corresponding MO-MDP { S , A , p0 , p , r0 : K , γ } . Regardless of the thresholds c1 : K , the optimal policy for the CMDP must lie on the Pareto front for this corresponding MO-MDP.2 Thus , if we knew the Pareto front , we could simply search along it to find the constraint-satisfying policy that maximizes task return . Multi-objective Maximum a Posteriori Policy Optimization ( MO-MPO ) . Whereas most MORL algorithms are based on linear scalarization , MO-MPO takes a distributional approach to training policies . For each policy improvement step , MO-MPO first computes a non-parametric policy for each objective rk that improves the ( parametric ) policy with respect to that objective , subject to a non-negative constraint k on the KL-divergence between the improved and old policies . Then , the policy is updated via supervised learning on the sum of these non-parametric policies . Intuitively , k defines the influence of objective rk on the final policy ; an objective with larger k has more influence . However , MO-MPO can only train policies for a single preference setting . Sec . 4.1 explains how we extend MO-MPO to train preference-conditioned policies π ( a|s , ) , in order to use it in our framework . We choose to extend MO-MPO because it does not suffer from the limitations of linear scalarization , and has been shown to outperform such approaches ( Abdolmaleki et al. , 2020 ) . 1This assumes that the choice of preference encoding does not restrict optimal solutions to the convex portions of the Pareto front . Although linear scalarization does not satisfy this requirement , other kinds of preference encodings do , for instance Chebyshev scalarization ( Van Moffaert et al. , 2013 ) or the per-objective KL-divergence constraints in MO-MPO , described later in this section . 2This is because if the optimal policy for the CMDP is not Pareto optimal , then there must exist another policy that obtains higher task return while meeting the constraints , which leads to a contradiction .

Summary: The paper is interested in reinforcement learning where one needs to satisfy constraints (for instance energy spent) in addition to maximizing rewards. The proposed approach proposes to extend any method able to approximate the Pareto front of optimal policies by also learning portions of the front that satisfy user constraints (given that only small portion of the front may be valid). In particular, a previous approach MO-MPO is used to approximate the Pareto front in a first step and a second step learns another policy find preference vectors satisfying given constrains. Experiments are carried in two environments and show that the proposed method outperforms Lagrangian baselines (e.g. optimizing alternatively for reward and constrain violation) in particular in settings where constraints are harder to satisfy.