Explicit Pareto Front Optimization for Constrained Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) has shown great potential for training policies that optimize a single scalar reward . Recent approaches have exceeded human-level performance on Atari ( Mnih et al. , 2015 ) and Go ( Silver et al. , 2016 ) , and have also achieved impressive results in continuous control tasks , including robot locomotion ( Lillicrap et al. , 2016 ; Schulman et al. , 2017 ) , acrobatics ( Peng et al. , 2018 ) , and real-world robot manipulation ( Levine et al. , 2016 ; Zeng et al. , 2019 ) . However , many problems , especially in the real world , require that policies meet certain constraints . For instance , we might want a factory robot to optimize task throughput while keeping actuator forces below a threshold , to limit wear-and-tear . Or , we might want to minimize energy usage for cooling a data center while ensuring that temperatures remain below some level ( Lazic et al. , 2018 ) . Such problems are often encoded as constrained Markov Decision Processes ( CMDPs ) ( Altman , 1999 ) , where the goal is to maximize task return while meeting the constraint ( s ) . Typical approaches for solving CMDPs use Lagrangian relaxation ( Bertsekas , 1999 ) to transform the constrained optimization problem into an unconstrained one . However , existing Lagrangian-based approaches suffer from several limitations . First , because the relaxed objective is a weighted-sum of the task return and constraint violation , this assumes a convex Pareto front ( Das & Dennis , 1997 ) . In addition , when the constraint is difficult to satisfy , in practice policies can struggle to obtain task reward . Finally , such approaches typically produce a single policy that satisfies a specific constraint threshold . However , the exact constraint threshold may not be known in advance , or one may prefer to choose from a set of policies across a range of acceptable thresholds . We aim to achieve the goal of CMDPs ( i.e. , finding a constraint-satisfying policy that maximizes task return ) while avoiding these limitations , by introducing a novel , general framework based on multi-objective MDPs ( MO-MDP ) . A MO-MDP can be seen as a CMDP where the constrained objectives are instead unconstrained . Our key insight is that if we have access to the Pareto front , then we can find the optimal constraint-satisfying policy by simply searching along this front . However , finding the entire Pareto front is unnecessary if only a relatively small portion of the policies along the Pareto front meet the constraints . Therefore , we propose to also simultaneously prioritize learning for the preferences ( i.e. , trade-offs between reward and cost ) that are most likely to produce policies that satisfy the constraints , and thus cover the relevant part of the Pareto front . To our knowledge , there is no existing framework for applying multi-objective RL algorithms to constrained RL problems . Our main contribution is a general framework that enables this , by learning which preferences produce constraint-satisfying policies . This framework can be combined with any multi-objective RL algorithm that learns an approximate Pareto front of policies . Our second contribution is to extend a state-of-the-art multi-objective RL algorithm , multi-objective maximum a posteriori policy optimization ( MO-MPO ) ( Abdolmaleki et al. , 2020 ) , to learn the Pareto front in a single training run . We use this extension of MO-MPO within our framework , and call the resulting algorithm constrained MO-MPO . We show in practice , constrained MO-MPO outperforms existing approaches on challenging continuous control tasks with constraints . 2 RELATED WORK . Constrained reinforcement learning . Constrained RL algorithms seek policies that meet the desired constraints at deployment time . Most approaches use Lagrangian relaxation ( Bertsekas , 1999 ) . Recent Lagrangian-based approaches claim convergence to constraint-satisfying ( Tessler et al. , 2019 ) or optimal ( Paternain et al. , 2019 ) solutions , but this is debatable ( Szepesvari , 2020 ) . Recent works seek to stabilize this optimization by approximating the reward and cost functions with convex relaxations ( Yu et al. , 2019 ) or by utilizing derivatives of the constraint function ( Stooke et al. , 2020 ) . Other works have applied Lagrangian relaxation to mean-value constraints ( Tessler et al. , 2019 ) , convex set constraints ( Miryoosefi et al. , 2019 ) , and local constraint satisfaction ( Bohez et al. , 2019 ) . Existing Lagrangian approaches involve linear scalarization , however , and thus can not find solutions that lie on concave portions of the true Pareto front ( Das & Dennis , 1997 ) . In contrast , we build on a multi-objective RL algorithm that does not rely on scalarization , and in practice , our approach can indeed find constraint-satisfying solutions on a concave Pareto front ( see Sec . 5.1 , humanoid walk ) . Safe reinforcement learning . In safe RL , the aim is to achieve constraint satisfaction not only during deployment , but also during learning ( Garcı́a & Fernández , 2015 ) . Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training ( Achiam et al. , 2017 ; Berkenkamp et al. , 2017 ; Chow et al. , 2018 ; 2019 ; Yang et al. , 2020 ; Zhang et al. , 2020 ) . These approaches require , however , that the initial policy meets ( or almost meets ) the constraints . Otherwise , performance degrades substantially and constrained policy optimization ( CPO ) ( Achiam et al. , 2017 ) , for example , performs worse than Lagrangian-based approaches ( Ray et al. , 2019 ) . The aim of our work is to find better solutions for constrained RL , rather than safe RL . We discuss in Sec . 6 how our method can be extended to reduce constraint violation during training . Multi-objective reinforcement learning . Our approach is built on ideas from multi-objective RL ( MORL ) , which consists of single-policy and multi-policy approaches . Single policy methods learn a policy that is optimal for a given setting of reward preferences . Most rely on linear scalarization ( Roijers et al. , 2013 ) , which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales . Non-linear scalarizations have been proposed ( Tesauro et al. , 2008 ; Van Moffaert et al. , 2013 ; Golovin & Zhang , 2020 ) , but these are harder to combine with value-based RL and have seen limited use in deep RL . Recently , Abdolmaleki et al . ( 2020 ) introduced MO-MPO , where the preferences represent per-objective constraints on the policy improvement step . MO-MPO does not rely on scalarization and is thus invariant to reward scales . Multi-policy MORL aims to find a set of policies that covers the whole Pareto front . Recent approaches learn a manifold in parameter space that optimizes the hypervolume of the Pareto front ( Parisi et al. , 2016 ; 2017 ) . While such approaches could be combined with our framework , their scalability to deep RL remains to be shown . Other works combine single policy approaches with a general objective to optimize hypervolume ( Xu et al. , 2020 ) . The instantiation of our framework is similar in spirit to such two-level methods , but applied to a different problem setting . 3 BACKGROUND AND NOTATION . 3.1 CONSTRAINED MARKOV DECISION PROCESSES . A constrained Markov Decision Process ( CMDP ) consists of states s ∈ S , actions a ∈ A , an initial state distribution p0 ( s ) , a transition function p ( s′|s , a ) , reward functions { rk ( s , a ) } Kk=0 , constraint thresholds { ck } Kk=1 , and a discount factor γ ∈ [ 0 , 1 ] . The ( K + 1 ) reward functions consist of a task reward r0 and constrained rewards r1 : K ; we will refer to these together as objectives . A policy π ( a|s ) maps from a given state to a distribution over actions . The optimal solution to a constrained MDP is a policy that maximizes the expected return for the task reward r0 , while ensuring that the expected return for all constrained rewards r1 : K satisfy their respective thresholds : max π Eπ [ ∑ t γtr0 ( st , at ) ] s.t . Eπ [ ∑ t γtri ( st , at ) ] ≥ ci ∀i = 1 , ... , K , ( 1 ) where Eπ is shorthand for the expectation over trajectories when following the policy π , given a fixed initial state distribution . Here the constraint thresholds are lower bounds , without loss of generality . The value function V πk ( s ) of a policy π is its expected return for objective rk when starting from state s. The action-value functionQπk ( s , a ) denotes the expected return for objective rk after taking action a in state s and thereafter acting according to the policy : Qπk ( s , a ) = rk ( s , a ) + γEs′∼p ( s′|s , a ) [ V πk ( s′ ) ] . Most approaches for solving CMDPs are based on Lagrangian relaxation , where the constrained problem is turned into the following unconstrained optimization problem : min λ≥0 max π Eπ [ ∑ t γtr0 ( st , at ) ] + K∑ k=1 λk ( Eπ [ ∑ t γtrk ( st , at ) ] − ci ) . ( 2 ) These approaches alternate between optimizing for the Lagrange multipliers λk and the policy π . While our multi-objective perspective on constrained RL aims to achieve the goal of CMDPs—finding a constraint-satisfying policy that maximizes task return—it does not directly solve ( 1 ) . Nonetheless , we will show our approach outperforms those that directly solve ( 1 ) via Lagrangian relaxation . 3.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING . In order to achieve the goal of CMDPs , we propose to leverage advances in multi-objective RL . A multi-objective MDP ( MO-MDP ) is defined in the same way as a CMDP , except without constraint thresholds . Because there are multiple unconstrained objectives , there is not a single optimal policy . Instead , there is a set of optimal policies , called the Pareto front . A policy is Pareto optimal if there is no other policy that improves its return for one objective without decreasing return for another . Preferences . Each policy on the Pareto front is the optimal policy for a particular setting of preferences ( i.e. , desired trade-off over objectives ) .1 This is typically encoded via a preference vector , in which each element k represents the relative importance of the corresponding objective rk . Connection to CMDPs . Given a CMDP { S , A , p0 , p , r0 : K , c1 : K , γ } , consider the corresponding MO-MDP { S , A , p0 , p , r0 : K , γ } . Regardless of the thresholds c1 : K , the optimal policy for the CMDP must lie on the Pareto front for this corresponding MO-MDP.2 Thus , if we knew the Pareto front , we could simply search along it to find the constraint-satisfying policy that maximizes task return . Multi-objective Maximum a Posteriori Policy Optimization ( MO-MPO ) . Whereas most MORL algorithms are based on linear scalarization , MO-MPO takes a distributional approach to training policies . For each policy improvement step , MO-MPO first computes a non-parametric policy for each objective rk that improves the ( parametric ) policy with respect to that objective , subject to a non-negative constraint k on the KL-divergence between the improved and old policies . Then , the policy is updated via supervised learning on the sum of these non-parametric policies . Intuitively , k defines the influence of objective rk on the final policy ; an objective with larger k has more influence . However , MO-MPO can only train policies for a single preference setting . Sec . 4.1 explains how we extend MO-MPO to train preference-conditioned policies π ( a|s , ) , in order to use it in our framework . We choose to extend MO-MPO because it does not suffer from the limitations of linear scalarization , and has been shown to outperform such approaches ( Abdolmaleki et al. , 2020 ) . 1This assumes that the choice of preference encoding does not restrict optimal solutions to the convex portions of the Pareto front . Although linear scalarization does not satisfy this requirement , other kinds of preference encodings do , for instance Chebyshev scalarization ( Van Moffaert et al. , 2013 ) or the per-objective KL-divergence constraints in MO-MPO , described later in this section . 2This is because if the optimal policy for the CMDP is not Pareto optimal , then there must exist another policy that obtains higher task return while meeting the constraints , which leads to a contradiction .

Summary: This paper introduced a general framework that incorporates multi-object reinforcement learning(MORL) perspective for constrained reinforcement learning to a policy set, called Pareto front, that meets the constrained. The author has instantiated a method based on the previous method MO-MPO. Compared to previous Lagrangian-based approaches, the proposed method has advantages in solution quality, stability, and sample-efficiency in few empirical environments.