Pareto Policy Adaptation

1 INTRODUCTION . Deep reinforcement learning has a pivotal role in solving several control problems of practical interest that would otherwise be intractable , such as robotic locomotion ( 1 ) , the Atari ( 2 ) and Go games ( 3 ) to name a few . While such approaches have focused on the scalar reward setting , there exist many real-world problems that have multiple , conflicting objectives and , therefore , can not be addressed by the current reinforcement learning tools . In such scenario , it is challenging to find an optimal control policy as the trade-offs ( preferences ) among the objectives may not be precisely known at training time or may differ from user to user . The paradigm of Multi-Objective Reinforcement Learning ( MORL ) provides a generalized framework for dealing with multi-dimensional rewards signals and has been identified as one of the main challenges of real-world reinforcement learning ( 4 ) . However , learning optimal policies for MORL has been proven to be quite challenging , because most strategies require either to have access to or be able to infer a quantification of the relative importance of the objectives , or they perform sophisticated searches in the value or policy space aiming to find an ensemble of policies that are non-inferior to each other . The former methods lack adaptability and generalizability , as their performance is tied to the given or inferred preferences , while the later ones suffer from scalability issues , because they learn multiple policies , and are usually complicated to implement . In this work , we aim to strike a balance between those two families of methods . Related Work : Multi-Objective Optimization ( MOO ) provides the fundamental tools for MORL . In MOO , there are two common solution concepts : scalarization and Pareto optimality . The former one derives a scalar objective and uses standard single-objective optimization techniques ( 5 ; 6 ; 7 ; 8 ; 9 ) . The later method is based on the concept of Pareto dominance and considers the set of all noninferior solutions ( 10 ; 11 ) . Multiple gradient methods that leverage first-order , necessary conditions for Pareto optimality have also been developed ( 12 ; 13 ; 14 ) . Similarly to MOO , existing work in MORL can be roughly divided into two main categories : Singlepolicy methods aim to maximize a single , scalarized reward . These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences . Scalarization is usually performed using a weighted sum of the reward vector ( 15 ; 16 ; 17 ) or , less commonly , using linear mappings ( 18 ) . Different single-policy methods are based on thresholds or lexicographical ordering ( 19 ) or different classes of preferences ( 20 ; 21 ) . More recently , a scalarized Q-learning algorithm has been developed ( 22 ) which uses the concept of corner weight to infer and optimize over preferences . It was extended to handle dynamic preferences in ( 23 ) and to utilize the convex envelop of the Pareto front in ( 24 ) . Finally , a scale-invariant supervised learning approach for encoding the preferences was developed in ( 25 ) . On the other hand , multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front . The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple-policy methods mainly differ on the approximation scheme for the Pareto front . One common approach is to perform multiple runs ( 16 ; 15 ) of a single scalarized reward function over a set of preferences . Unfortunately , such methods lack scalability to high-dimensional rewards . Other approaches leverage convex approximations of the Pareto front ( 26 ) or linear approximations of the value function ( 27 ; 28 ) to learn optimal deterministic policies . Multi-objective fitted Q-iteration ( 29 ; 30 ) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator . A policy gradient method based on discrete approximations of the Pareto front was proposed in ( 31 ) and enhanced to utilize continuous approximations in ( 32 ) . A gradient-based method that learns a manifold on the policy parameter space leveraging episodic exploration and importance sampling was developed in ( 33 ) . Finally , a prediction-guided evolutionary algorithm which is able to find dense approximation to the Pareto front was proposed in ( 34 ) . Single-policy methods have the advantage of learning a single policy and being simple to implement . However , they suffer from instability issues , as a small change on the preferences may lead to performance collapse ( 35 ) , and also rely on heuristics to infer the preferences when they are not known . On the other hand , multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution ( being an approximation to the Pareto front ) encapsulates all trade-offs between the objectives . However , this benefit comes at a higher computational cost as we typically have to learn and store multiple policies . In this paper , we aim to bridge the gab between single and multiple policy MORL methods : Our method learns a single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution . Our preference vectors are inferred , not by using heuristics , but by approximating the Pareto front via a first-order condition ( see Fig . 1 for an example ) . Our method is inspired by the multiple gradient descent algorithm for MOO introduced in ( 13 ) as well as its application in the field of multi-task supervised learning presented in ( 36 ) . Contributions : We propose a policy gradient method for multi-objective reinforcement learning under unknown , linear preferences . Initially , we present Pareto stationarity as a necessary , firstorder condition for Pareto optimality and develop a multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objectives . Following that , we tackle the problem of adapting the policy gradient to any given preference distribution . We utilize our method from the first part and introduce the Pareto Policy Adaptation ( PPA ) , a loss function that penalizes deviations between the given preference distribution and the recovered preferences . Using implicit differentiation , we are able to back-propagate the gradient of PPA bypassing the operations of the projected gradient descent solver , which makes our method applicable to real-world problems . We evaluate our method in a series reinforcement learning tasks . The most closely related work to ours is presented in ( 31 ) . The authors discuss the existence of common ascent directions , introduce a quadratic optimization problem for their computation and give two gradient-based learning algorithms . Even though our method identifies the common ascent directions using a similar optimization problem , we leverage these directions to define the PPA loss function , which is a simple and efficient way 1 ) to incorporate the ascent directions into any learning algorithm and 2 ) to account for a given preference distribution . Leveraging implicit differentiation , PPA enables us to incorporate multiple rewards and preference distributions into any modern policy gradient algorithm by a mere modification of the loss function and by implementing a projected gradient descent optimizer . 2 PRELIMINARIES . We consider the framework of Multi-Objective Markov Decision Process ( MOMDP ) M : = ( S , A , R , P , γ , Ω , fΩ ) , where S and A denote the ( discrete ) state and action spaces , P : S × A × S → [ 0 , 1 ] is the transition kernel representing the environment dynamics , r ( s , a ) ≡ R : S × A → RM is a function mapping state-action tuples to reward vectors , Ω is the space of preferences and fω : RM → R a preference function which produces a scalar utility using a preference ω ∈ Ω . To declutter notation , we assume that there exists a given initial state . We consider linear preferences fω ( r ( s , a ) ) = ωTr ( s , a ) . A fixed preference vector ω ∈ Ω allows to directly compare the vectorized reward and value function by comparing their scalarized utilities , and , therefore , the MOMDP collapses to a standard MDP . A stationary control policy π : S × A → P ( A ) , where P ( A ) is the Borel set of A , is a function that assigns a probability distribution over actions for all states . The value vπ : S → RM×|S| of a policy π is defined as vπ ( s ) : = Eπ [ ∑∞ t=0 γ tr ( st , at ) | s0 = s ] , where γ ∈ ( 0 , 1 ) is a discount factor and the expectation is taken under the distribution over trajectories τ = ( s0 , a0 , s1 , a , s2 , . . . ) obtained by starting from s and following policy π thereafter . The discounted state-action occupation measure under policy π is defined as dπ ( s , a ) : = ∑∞ t=0 γ tP ( st = s , at = a | π ) for all states s ∈ S and allows us to write the value function compactly as vπ ( s ) = Edπ [ r ( s , a ) | s0 = s ] . Definition 1 . Considering all possible returns r̃ : = ∑∞ t=0 γ tr ( st , at ) , we define the Reward Space Pareto Coverage Set ( RS-PCS ) or Pareto frontier as F∗r : = { r̃ : r̃ ∈ RM , r̃′ ≻P r̃ } , where the symbol ≻P denotes Pareto dominance : greater or equal in all objectives and strictly greater in at least one objective . For all possible preferences in Ω and under the linear preference assumption , we define the Reward Space Convex Coverage Set ( RS-CCS ) as C∗R , Ω : = { r̃ : r̃ ∈ R , ∃ω ∈ Ω s.t . ωT r̃ ≥ ωT r̃′ , ∀r̃′ ∈ RM } . ( 1 ) The set C∗R , Ω contains all discounted reward vectors that could be optimal for some preference vector in Ω . Anything in F∗r but not in C∗R , Ω can not be useful under linear preferences . In that case , it suffices to restrict our analysis to C∗R , Ω . Also , we assume , without loss of generality , that∑ m ωm = 1 , ωm ≥ 0 , m = 1 , 2 . . .M for all ω ∈ Ω , i.e. , the preference vector is a convex combination of the objectives . We extend the definition of Pareto optimality from the reward space to the policy space . Definition 2 . A policy π dominates a policy π′ if and only if vπ ≻P vπ ′ . A policy π∗ is called Pareto optimal if there exists no other policy π that dominates π∗ . The set Π∗P of all Pareto optimal policies is called Pareto policy set and its image F∗π : = { vπ } π∈Π∗P is called Pareto policy frontier . The set C∗Π , Ω : = { π : π ∈ Π , ∃ω ∈ Ω s.t . ωTvπ ≥ ωTvπ ′ , ∀π′ ∈ Π , ∑ m ωm = 1 , ωm ≥ 0 } ( 2 ) is called the Policy Space Convex Coverage Set ( PS-CCS ) . Lemma 1 . Let π ∈ C∗Π , Ω be a policy . Then , we have Eπ [ r̃ ] ∈ C∗R , Ω , i.e. , the expected return vector under policy π belongs in the RS-CCS . Lemma 1 ( proof given in the Appendix ) connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later . This is very convenient because searching for Pareto optimal vectors in the reward space is not practically efficient ( e.g. , due to the sparsity of reward signal ) or useful ( e.g. , because they can not be easily mapped to values or actions ) . By finding a policy π ∈ C∗Π , Ω , Lemma 1 guarantees that the expected return vector under π belongs in C∗R , Ω and , therefore , maps the problem of reward space exploration to policy space exploration . Additionally , Lemma 1 imposes no restrictions on the class of policies and allows us to adopt the common practice of parameterizing the policy by high-dimensional vector θ ∈ Θ ( e.g. , a neural network ) . The definition of PS-CCS given Eq . equation 3 carries over to this parametric class as is , i.e. , we can define the PS-CCS in the policy parameter space as follows C∗Θ , Ω : = { θ : θ ∈ Θ , ∃ω ∈ Ω s.t . ωTvπθ ≥ ωTvπθ′ , ∀θ′ ∈ Θ } . ( 3 )

Summary: This paper introduces a policy gradient approach to multi objective RL. The authors learn show that an optimal solution for a specific reward vector can be found by minimizing a constrained quadratic optimization problem. They propose to solve this problem using projected gradient descent. Then, the authors offer an efficient solution for optimizing the Pareto front through "Pareto Policy Adaptation" loss function, which trades off learning the Pareto front and the policy. The authors conduct experiments and ablations studies on four benchmarks, validating their method with improvement over current sota approaches for MORL.