Pareto Policy Adaptation

1 INTRODUCTION . Deep reinforcement learning has a pivotal role in solving several control problems of practical interest that would otherwise be intractable , such as robotic locomotion ( 1 ) , the Atari ( 2 ) and Go games ( 3 ) to name a few . While such approaches have focused on the scalar reward setting , there exist many real-world problems that have multiple , conflicting objectives and , therefore , can not be addressed by the current reinforcement learning tools . In such scenario , it is challenging to find an optimal control policy as the trade-offs ( preferences ) among the objectives may not be precisely known at training time or may differ from user to user . The paradigm of Multi-Objective Reinforcement Learning ( MORL ) provides a generalized framework for dealing with multi-dimensional rewards signals and has been identified as one of the main challenges of real-world reinforcement learning ( 4 ) . However , learning optimal policies for MORL has been proven to be quite challenging , because most strategies require either to have access to or be able to infer a quantification of the relative importance of the objectives , or they perform sophisticated searches in the value or policy space aiming to find an ensemble of policies that are non-inferior to each other . The former methods lack adaptability and generalizability , as their performance is tied to the given or inferred preferences , while the later ones suffer from scalability issues , because they learn multiple policies , and are usually complicated to implement . In this work , we aim to strike a balance between those two families of methods . Related Work : Multi-Objective Optimization ( MOO ) provides the fundamental tools for MORL . In MOO , there are two common solution concepts : scalarization and Pareto optimality . The former one derives a scalar objective and uses standard single-objective optimization techniques ( 5 ; 6 ; 7 ; 8 ; 9 ) . The later method is based on the concept of Pareto dominance and considers the set of all noninferior solutions ( 10 ; 11 ) . Multiple gradient methods that leverage first-order , necessary conditions for Pareto optimality have also been developed ( 12 ; 13 ; 14 ) . Similarly to MOO , existing work in MORL can be roughly divided into two main categories : Singlepolicy methods aim to maximize a single , scalarized reward . These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences . Scalarization is usually performed using a weighted sum of the reward vector ( 15 ; 16 ; 17 ) or , less commonly , using linear mappings ( 18 ) . Different single-policy methods are based on thresholds or lexicographical ordering ( 19 ) or different classes of preferences ( 20 ; 21 ) . More recently , a scalarized Q-learning algorithm has been developed ( 22 ) which uses the concept of corner weight to infer and optimize over preferences . It was extended to handle dynamic preferences in ( 23 ) and to utilize the convex envelop of the Pareto front in ( 24 ) . Finally , a scale-invariant supervised learning approach for encoding the preferences was developed in ( 25 ) . On the other hand , multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front . The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple-policy methods mainly differ on the approximation scheme for the Pareto front . One common approach is to perform multiple runs ( 16 ; 15 ) of a single scalarized reward function over a set of preferences . Unfortunately , such methods lack scalability to high-dimensional rewards . Other approaches leverage convex approximations of the Pareto front ( 26 ) or linear approximations of the value function ( 27 ; 28 ) to learn optimal deterministic policies . Multi-objective fitted Q-iteration ( 29 ; 30 ) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator . A policy gradient method based on discrete approximations of the Pareto front was proposed in ( 31 ) and enhanced to utilize continuous approximations in ( 32 ) . A gradient-based method that learns a manifold on the policy parameter space leveraging episodic exploration and importance sampling was developed in ( 33 ) . Finally , a prediction-guided evolutionary algorithm which is able to find dense approximation to the Pareto front was proposed in ( 34 ) . Single-policy methods have the advantage of learning a single policy and being simple to implement . However , they suffer from instability issues , as a small change on the preferences may lead to performance collapse ( 35 ) , and also rely on heuristics to infer the preferences when they are not known . On the other hand , multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution ( being an approximation to the Pareto front ) encapsulates all trade-offs between the objectives . However , this benefit comes at a higher computational cost as we typically have to learn and store multiple policies . In this paper , we aim to bridge the gab between single and multiple policy MORL methods : Our method learns a single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution . Our preference vectors are inferred , not by using heuristics , but by approximating the Pareto front via a first-order condition ( see Fig . 1 for an example ) . Our method is inspired by the multiple gradient descent algorithm for MOO introduced in ( 13 ) as well as its application in the field of multi-task supervised learning presented in ( 36 ) . Contributions : We propose a policy gradient method for multi-objective reinforcement learning under unknown , linear preferences . Initially , we present Pareto stationarity as a necessary , firstorder condition for Pareto optimality and develop a multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objectives . Following that , we tackle the problem of adapting the policy gradient to any given preference distribution . We utilize our method from the first part and introduce the Pareto Policy Adaptation ( PPA ) , a loss function that penalizes deviations between the given preference distribution and the recovered preferences . Using implicit differentiation , we are able to back-propagate the gradient of PPA bypassing the operations of the projected gradient descent solver , which makes our method applicable to real-world problems . We evaluate our method in a series reinforcement learning tasks . The most closely related work to ours is presented in ( 31 ) . The authors discuss the existence of common ascent directions , introduce a quadratic optimization problem for their computation and give two gradient-based learning algorithms . Even though our method identifies the common ascent directions using a similar optimization problem , we leverage these directions to define the PPA loss function , which is a simple and efficient way 1 ) to incorporate the ascent directions into any learning algorithm and 2 ) to account for a given preference distribution . Leveraging implicit differentiation , PPA enables us to incorporate multiple rewards and preference distributions into any modern policy gradient algorithm by a mere modification of the loss function and by implementing a projected gradient descent optimizer . 2 PRELIMINARIES . We consider the framework of Multi-Objective Markov Decision Process ( MOMDP ) M : = ( S , A , R , P , γ , Ω , fΩ ) , where S and A denote the ( discrete ) state and action spaces , P : S × A × S → [ 0 , 1 ] is the transition kernel representing the environment dynamics , r ( s , a ) ≡ R : S × A → RM is a function mapping state-action tuples to reward vectors , Ω is the space of preferences and fω : RM → R a preference function which produces a scalar utility using a preference ω ∈ Ω . To declutter notation , we assume that there exists a given initial state . We consider linear preferences fω ( r ( s , a ) ) = ωTr ( s , a ) . A fixed preference vector ω ∈ Ω allows to directly compare the vectorized reward and value function by comparing their scalarized utilities , and , therefore , the MOMDP collapses to a standard MDP . A stationary control policy π : S × A → P ( A ) , where P ( A ) is the Borel set of A , is a function that assigns a probability distribution over actions for all states . The value vπ : S → RM×|S| of a policy π is defined as vπ ( s ) : = Eπ [ ∑∞ t=0 γ tr ( st , at ) | s0 = s ] , where γ ∈ ( 0 , 1 ) is a discount factor and the expectation is taken under the distribution over trajectories τ = ( s0 , a0 , s1 , a , s2 , . . . ) obtained by starting from s and following policy π thereafter . The discounted state-action occupation measure under policy π is defined as dπ ( s , a ) : = ∑∞ t=0 γ tP ( st = s , at = a | π ) for all states s ∈ S and allows us to write the value function compactly as vπ ( s ) = Edπ [ r ( s , a ) | s0 = s ] . Definition 1 . Considering all possible returns r̃ : = ∑∞ t=0 γ tr ( st , at ) , we define the Reward Space Pareto Coverage Set ( RS-PCS ) or Pareto frontier as F∗r : = { r̃ : r̃ ∈ RM , r̃′ ≻P r̃ } , where the symbol ≻P denotes Pareto dominance : greater or equal in all objectives and strictly greater in at least one objective . For all possible preferences in Ω and under the linear preference assumption , we define the Reward Space Convex Coverage Set ( RS-CCS ) as C∗R , Ω : = { r̃ : r̃ ∈ R , ∃ω ∈ Ω s.t . ωT r̃ ≥ ωT r̃′ , ∀r̃′ ∈ RM } . ( 1 ) The set C∗R , Ω contains all discounted reward vectors that could be optimal for some preference vector in Ω . Anything in F∗r but not in C∗R , Ω can not be useful under linear preferences . In that case , it suffices to restrict our analysis to C∗R , Ω . Also , we assume , without loss of generality , that∑ m ωm = 1 , ωm ≥ 0 , m = 1 , 2 . . .M for all ω ∈ Ω , i.e. , the preference vector is a convex combination of the objectives . We extend the definition of Pareto optimality from the reward space to the policy space . Definition 2 . A policy π dominates a policy π′ if and only if vπ ≻P vπ ′ . A policy π∗ is called Pareto optimal if there exists no other policy π that dominates π∗ . The set Π∗P of all Pareto optimal policies is called Pareto policy set and its image F∗π : = { vπ } π∈Π∗P is called Pareto policy frontier . The set C∗Π , Ω : = { π : π ∈ Π , ∃ω ∈ Ω s.t . ωTvπ ≥ ωTvπ ′ , ∀π′ ∈ Π , ∑ m ωm = 1 , ωm ≥ 0 } ( 2 ) is called the Policy Space Convex Coverage Set ( PS-CCS ) . Lemma 1 . Let π ∈ C∗Π , Ω be a policy . Then , we have Eπ [ r̃ ] ∈ C∗R , Ω , i.e. , the expected return vector under policy π belongs in the RS-CCS . Lemma 1 ( proof given in the Appendix ) connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later . This is very convenient because searching for Pareto optimal vectors in the reward space is not practically efficient ( e.g. , due to the sparsity of reward signal ) or useful ( e.g. , because they can not be easily mapped to values or actions ) . By finding a policy π ∈ C∗Π , Ω , Lemma 1 guarantees that the expected return vector under π belongs in C∗R , Ω and , therefore , maps the problem of reward space exploration to policy space exploration . Additionally , Lemma 1 imposes no restrictions on the class of policies and allows us to adopt the common practice of parameterizing the policy by high-dimensional vector θ ∈ Θ ( e.g. , a neural network ) . The definition of PS-CCS given Eq . equation 3 carries over to this parametric class as is , i.e. , we can define the PS-CCS in the policy parameter space as follows C∗Θ , Ω : = { θ : θ ∈ Θ , ∃ω ∈ Ω s.t . ωTvπθ ≥ ωTvπθ′ , ∀θ′ ∈ Θ } . ( 3 )

Summary: The paper presents a multi-objective method for linear preferences. The key concept used for the design of the approach is that there always exists a common ascent direction for all the objectives. The authors leverage this property for introducing a novel loss function that can be integrated into different DeepRL methods. The proposed approach is simple and proved to be effective in the proposed experiments.