Year: 2021

Abstract:
We present a policy gradient method for Multi-Objective Reinforcement Learning under unknown, linear preferences. By enforcing Pareto stationarity, a first-order condition for Pareto optimality, we are able to design a simple policy gradient algorithm that approximates the Pareto front and infers the unknown preferences. Our method relies on a projected gradient descent solver that identifies common ascent directions for all objectives. Leveraging the solution of that solver, we introduce Pareto Policy Adaptation (PPA), a loss function that adapts the policy to be optimal with respect to any distribution over preferences. PPA uses implicit differentiation to back-propagate the loss gradient bypassing the operations of the projected gradient descent solver. Our approach is straightforward, easy to implement and can be used with all existing policy gradient and actor-critic methods. We evaluate our method in a series of reinforcement learning tasks.

heading: 1 INTRODUCTION
text: Deep reinforcement learning has a pivotal role in solving several control problems of practical interest that would otherwise be intractable, such as robotic locomotion (1), the Atari (2) and Go games (3) to name a few. While such approaches have focused on the scalar reward setting, there exist many real-world problems that have multiple, conflicting objectives and, therefore, cannot be addressed by the current reinforcement learning tools. In such scenario, it is challenging to find an optimal control policy as the trade-offs (preferences) among the objectives may not be precisely known at training time or may differ from user to user. The paradigm of Multi-Objective Reinforcement Learning (MORL) provides a generalized framework for dealing with multi-dimensional rewards signals and has been identified as one of the main challenges of real-world reinforcement learning (4). However, learning optimal policies for MORL has been proven to be quite challenging, because most strategies require either to have access to or be able to infer a quantification of the relative importance of the objectives, or they perform sophisticated searches in the value or policy space aiming to find an ensemble of policies that are non-inferior to each other. The former methods lack adaptability and generalizability, as their performance is tied to the given or inferred preferences, while the later ones suffer from scalability issues, because they learn multiple policies, and are usually complicated to implement. In this work, we aim to strike a balance between those two families of methods.
Related Work: Multi-Objective Optimization (MOO) provides the fundamental tools for MORL. In MOO, there are two common solution concepts: scalarization and Pareto optimality. The former one derives a scalar objective and uses standard single-objective optimization techniques (5; 6; 7; 8; 9). The later method is based on the concept of Pareto dominance and considers the set of all noninferior solutions (10; 11). Multiple gradient methods that leverage first-order, necessary conditions for Pareto optimality have also been developed (12; 13; 14).
Similarly to MOO, existing work in MORL can be roughly divided into two main categories: Singlepolicy methods aim to maximize a single, scalarized reward. These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences. Scalarization is usually performed using a weighted sum of the reward vector (15; 16; 17) or, less commonly, using linear mappings (18). Different single-policy methods are based on thresholds or lexicographical ordering (19) or different classes of preferences (20; 21). More recently, a scalarized Q-learning algorithm has been developed (22) which uses the concept of corner weight to infer and optimize over preferences. It was extended to handle dynamic preferences in (23) and to utilize the convex envelop of the Pareto front in (24). Finally, a scale-invariant supervised learning approach for encoding the preferences was developed in (25).
On the other hand, multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front. The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple-policy methods mainly differ on the approximation scheme for the Pareto front. One common approach is to perform multiple runs (16; 15) of a single scalarized reward function over a set of preferences. Unfortunately, such methods lack scalability to high-dimensional rewards. Other approaches leverage convex approximations of the Pareto front (26) or linear approximations of the value function (27; 28) to learn optimal deterministic policies. Multi-objective fitted Q-iteration (29; 30) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator. A policy gradient method based on discrete approximations of the Pareto front was proposed in (31) and enhanced to utilize continuous approximations in (32). A gradient-based method that learns a manifold on the policy parameter space leveraging episodic exploration and importance sampling was developed in (33). Finally, a prediction-guided evolutionary algorithm which is able to find dense approximation to the Pareto front was proposed in (34).
Single-policy methods have the advantage of learning a single policy and being simple to implement. However, they suffer from instability issues, as a small change on the preferences may lead to performance collapse (35), and also rely on heuristics to infer the preferences when they are not known. On the other hand, multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution (being an approximation to the Pareto front) encapsulates all trade-offs between the objectives. However, this benefit comes at a higher computational cost as we typically have to learn and store multiple policies. In this paper, we aim to bridge the gab between single and multiple policy MORL methods: Our method learns a single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution. Our preference vectors are inferred, not by using heuristics, but by approximating the Pareto front via a first-order condition (see Fig. 1 for an example). Our method is inspired by the multiple gradient descent algorithm for MOO introduced in (13) as well as its application in the field of multi-task supervised learning presented in (36).
Contributions: We propose a policy gradient method for multi-objective reinforcement learning under unknown, linear preferences. Initially, we present Pareto stationarity as a necessary, firstorder condition for Pareto optimality and develop a multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objectives. Following that, we tackle the problem of adapting the policy gradient to any given preference distribution. We utilize our method from the first part and introduce the Pareto Policy Adaptation (PPA), a loss function that penalizes deviations between the given preference distribution and the recovered preferences.
Using implicit differentiation, we are able to back-propagate the gradient of PPA bypassing the operations of the projected gradient descent solver, which makes our method applicable to real-world problems. We evaluate our method in a series reinforcement learning tasks.
The most closely related work to ours is presented in (31). The authors discuss the existence of common ascent directions, introduce a quadratic optimization problem for their computation and give two gradient-based learning algorithms. Even though our method identifies the common ascent directions using a similar optimization problem, we leverage these directions to define the PPA loss function, which is a simple and efficient way 1) to incorporate the ascent directions into any learning algorithm and 2) to account for a given preference distribution. Leveraging implicit differentiation, PPA enables us to incorporate multiple rewards and preference distributions into any modern policy gradient algorithm by a mere modification of the loss function and by implementing a projected gradient descent optimizer.

heading: 2 PRELIMINARIES
text: We consider the framework of Multi-Objective Markov Decision Process (MOMDP) M := (S,A,R,P, γ,Ω, fΩ), where S and A denote the (discrete) state and action spaces, P : S × A × S → [0, 1] is the transition kernel representing the environment dynamics, r(s, a) ≡ R : S × A → RM is a function mapping state-action tuples to reward vectors, Ω is the space of preferences and fω : RM → R a preference function which produces a scalar utility using a preference ω ∈ Ω. To declutter notation, we assume that there exists a given initial state. We consider linear preferences fω(r(s, a)) = ωTr(s, a). A fixed preference vector ω ∈ Ω allows to directly compare the vectorized reward and value function by comparing their scalarized utilities, and, therefore, the MOMDP collapses to a standard MDP. A stationary control policy π : S × A → P(A), where P(A) is the Borel set of A, is a function that assigns a probability distribution over actions for all states. The value vπ : S → RM×|S| of a policy π is defined as vπ(s) := Eπ[ ∑∞ t=0 γ
tr(st, at) | s0 = s], where γ ∈ (0, 1) is a discount factor and the expectation is taken under the distribution over trajectories τ = (s0, a0, s1, a,s2, . . . ) obtained by starting from s and following policy π thereafter. The discounted state-action occupation measure under policy π is defined as dπ(s, a) := ∑∞ t=0 γ
tP (st = s, at = a | π) for all states s ∈ S and allows us to write the value function compactly as vπ(s) = Edπ [r(s, a) | s0 = s]. Definition 1. Considering all possible returns r̃ := ∑∞ t=0 γ
tr(st, at), we define the Reward Space Pareto Coverage Set (RS-PCS) or Pareto frontier as F∗r := {r̃ : r̃ ∈ RM , r̃′ ≻P r̃}, where the symbol ≻P denotes Pareto dominance: greater or equal in all objectives and strictly greater in at least one objective. For all possible preferences in Ω and under the linear preference assumption, we define the Reward Space Convex Coverage Set (RS-CCS) as
C∗R,Ω := {r̃ : r̃ ∈ R, ∃ω ∈ Ω s.t. ωT r̃ ≥ ωT r̃′, ∀r̃′ ∈ RM}. (1)
The set C∗R,Ω contains all discounted reward vectors that could be optimal for some preference vector in Ω. Anything in F∗r but not in C∗R,Ω cannot be useful under linear preferences. In that case, it suffices to restrict our analysis to C∗R,Ω. Also, we assume, without loss of generality, that∑
m ωm = 1, ωm ≥ 0,m = 1, 2 . . .M for all ω ∈ Ω, i.e., the preference vector is a convex combination of the objectives. We extend the definition of Pareto optimality from the reward space to the policy space.
Definition 2. A policy π dominates a policy π′ if and only if vπ ≻P vπ ′ . A policy π∗ is called Pareto optimal if there exists no other policy π that dominates π∗. The set Π∗P of all Pareto optimal policies is called Pareto policy set and its image F∗π := {vπ}π∈Π∗P is called Pareto policy frontier. The set
C∗Π,Ω := {π : π ∈ Π, ∃ω ∈ Ω s.t. ωTvπ ≥ ωTvπ ′ , ∀π′ ∈ Π, ∑ m ωm = 1, ωm ≥ 0} (2)
is called the Policy Space Convex Coverage Set (PS-CCS).
Lemma 1. Let π ∈ C∗Π,Ω be a policy. Then, we have Eπ[r̃] ∈ C∗R,Ω, i.e., the expected return vector under policy π belongs in the RS-CCS.
Lemma 1 (proof given in the Appendix) connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later. This is very convenient because searching for Pareto optimal vectors in the reward space is not practically efficient (e.g., due to the sparsity of reward signal) or useful (e.g., because they cannot be easily mapped to values or actions). By finding a policy π ∈ C∗Π,Ω, Lemma 1 guarantees that the expected return vector under π belongs in C∗R,Ω and, therefore, maps the problem of reward space exploration to policy space exploration. Additionally, Lemma 1 imposes no restrictions on the class of policies and allows us to adopt the common practice of parameterizing the policy by high-dimensional vector θ ∈ Θ (e.g., a neural network). The definition of PS-CCS given Eq. equation 3 carries over to this parametric class as is, i.e., we can define the PS-CCS in the policy parameter space as follows
C∗Θ,Ω := {θ : θ ∈ Θ, ∃ω ∈ Ω s.t. ωTvπθ ≥ ωTvπθ′ , ∀θ′ ∈ Θ}. (3)

heading: 3 LEARNING WITH UNKNOWN PREFERENCES
text: In this section, we propose a new policy gradient algorithm for MORL under unknown preferences. The key idea of our approach is to infer the preference vector ω, use that ω as scalarization coefficient for the multi-objective criterion and ascent on the gradient of the scalarized value. To search for a Pareto optimal policy, we leverage a fist-order condition known as Pareto stationarity. Definition 3 (Pareto stationarity). A policy πθ is called Pareto stationary if and only if∑M
m=1 ωm∇θ Edπθ [rm(s, a)] = 0 for some ωm ∈ [0, 1], m = 1, 2 . . .M .
Pareto stationarity implies that there exists a vanishing convex combination of the objectives’ gradients and is a necessary condition for Pareto optimality (13). The following theorem connect the PS-CCS with Pareto stationarity. Theorem 1. Let πθ ∈ C∗Θ,Ω be a policy in the PS-CCS. Then, πθ is Pareto stationary.
Proof. Let πθ ∈ C∗Θ,Ω be a policy and ω the corresponding preferences, i.e., we have ωTvπθ ≥ ωTvπθ′ , ∀θ′ ∈ Θ. The last relation can be written as ωT Eπθ [r̃] ≥ ωT Eπθ′ [r̃], ∀θ
′ ∈ Θ. This implies that θ is a global maximizer of ωT Eπθ [r̃] or, equivalently, that its gradient at θ vanishes. Therefore, we have
∇θωT Edπθ [r̃] = M∑
m=1
ωm∇θ Edπθ [rm(s, a)] = 0, (4)
which corresponds to the necessary condition for Pareto stationarity as per Def. 3.
Theorem 1 essentially states that Pareto stationarity is a necessary condition for a policy to belong to PS-CCS. Additionally, as stated in the proof, the convex combination of the objectives that leads to vanishing gradients coincides with the preferences that make the underlying policy part of the PS-CCS. This presents us with a strong necessary condition for recovering the unknown preferences and for identifying an optimal policy. This is because the problem of finding a Pareto stationary is tractable via gradient methods. In fact, if a policy is not Pareto stationary then there exists a common ascent direction for all objectives. To find this descent direction, we use the following theorem. Theorem 2 ((13), Theorem 1). Consider the following quadratic optimization problem
min ω1,...ωM {∥∥∥∥∥ M∑ m=1 ωm∇θ Edπθ [rm(s, a)] ∥∥∥∥∥ 2
2
s.t M∑
m=1
ωm = 1, ωm ≥ 0, ∀m } . (PA)
Let ω∗ be the optimal solution of Problem PA and µ∗ the underlying minimal-norm objective. Then:
1. either µ∗ = 0 and πθ∗ is a Pareto stationary policy
2. or ∑M
m=1 ω ∗ m∇θ Edπθ [rm(s, a)] is a common ascent direction for all objectives.
Problem PA is essentially equivalent to the formulation of the famous Minimum-Norm Point problem, which arises many fields such as optimal control (37), submodular minimization (38) and portfolio optimization (39). The MNP problem has been well studied and several combinatorial and recursive algorithms have been proposed (40; 41; 42). Such methods search for the exact solution and typically have exponential run-time (43), which prevents their application in high-dimensional control policies. Therefore, we adopt convex optimization approach for Problem 2. We start by noting that the constrain set E = {ω ∈ RM : ∑M m=1 ωm = 1, ωm ≥ 0, ∀m = 1, 2 . . .M} is essentially the unit simplex and the projection problem, i.e., ΠE(ω) = argminδ∈E ∥ω − δ∥, can be efficiently solved. In fact, there exists a unique dual variable τ ∈ R such that
ΠE(ω) = (ω − τ)+, (5)
where (x)+ = max(x, 0). To find the dual variable, we enforce the equality constraint, i.e.,∑ m(ωm − τ)+ = 1. This non-linear equation of a single variable can be solved very efficiently using the Newton method. This suggests a projected gradient descent solver for solving Problem PA and recovering the unknown preferences.
Algorithm 1: Multi-Objective Policy Gradient Inputs: Initial parameters θ0, learning rate α
1 foreach k = 0, 1, . . . do 2 Collect trajectories D = {τi}, τi ∼ πθk 3 Compute rewards-to-go rt 4 gm ← 1|D| ∑ τ,t rt∇θ log πθk(at|st) 5 (ω1, . . . ωM )←− PGDSolver(g1, . . . gM , α) 6 θk+1 ← θk + ∑M m=1 ωmgm 7 end 8
9 procedure PGDSolver(g1, . . . , gM , α): 10 Initialize feasible ω = (ω1, . . . ωM ) 11 while not converged do 12 Find τ solving ∑ m(ωm − τ)+ = 1
13 ωm ← ωm + αgTm ∑
n engn, m = 1 . . .M 14 ω ← (ω − τ)+ 15 end 16 return (ω1, . . . ωM ) Our method (Algorithm 1) extends the classical REINFORCE algorithm (44) to take gradient steps in the direction given by the weighted combination of the multi-objective gradient, with the weights being the preferences recovered by the projected gradient descent solver. Lemma 1, Theorem 1 and 2 essentially state that Algorithm 1 learns arbitrary policies in the Convex Coverage Set along with their underlying preference vectors and provide theoretical justification for our method. Algorithm 1 is simple to implement, compatible with all policy gradient and actor-critic methods and provides a practical algorithm for learning Pareto optimal policies under unknown preferences. However, it suffers from a subtle drawback: we have no control over the identified preferences and they cannot be optimized to account for any given preference distribution. This is the problem of policy adaptation which we address in the next section.
It is worth comparing our projected gradient descent solver with (36), where the authors used the Frank-Wolfe method for Problem PA, albeit in the context of supervised learning. Both algorithms have the same convergence rate of O(1/N) and since the projection to the constraint set can be easily calculated, we opted for the projected gradient descent optimizer. The overhead added by solving for the dual variable τ is minimal because the Newton’s method is known to enjoy quadratic convergence rate and is very efficient in practice. The trade-off of using Frank-Wolfe is that requires to obtain the feasible minimizer of the linear approximation of the objective around the current iterant, which is typically solved approximately and worsens the convergence rate by a multiplicative constant (45). Nonetheless, we expect that both methods can be used interchangeably in practice.

heading: 4 PARETO POLICY ADAPTATION
text: The policy gradient algorithm presented in the previous section finds policies that are optimal (i.e., Pareto stationary) for the inferred preferences. In this section, we show how to address the problem of adapting the policy to be optimal for a given distribution Dω over preferences. We start by expressing the optimal solution of Problem PA as a function of the policy parameters ω∗ := ω∗(θ) = (ω1(θ), ω2(θ), . . . ωM (θ)). We introduce the Pareto Policy Adaptation loss function as follows:
L(θ) = Edπθ ,ω [ λ M∑ m=1 ω∗m(θ)rm(s, a)− (1− λ) ∥ω∗(θ)− ω∥ 2 2 ] , (6)
where λ ∈ [0, 1] is a weight that trades off between the two losses. The first term of PPA is the scalarized reward, with the scalarization coefficient being the inferred preferences. The second term penalizes the expected deviation between the given and inferred preferences. To further motivate the PPA loss, consider the special case of a single, given preference vector ω0. In this case, the MOMDP collapses to a standard MDP, with ω0 being the scalarization coefficient for the reward and value functions. Additionally, assume that the policy πθ has been adapted to that preference vector, i.e., it achieves ω∗(θ) = ω0 for some parameter vector θ. Then, Eq. 6 reduces to L(θ) = Edπθ [∑M m=1 ω 0 mrm(s, a) ] . Maximization of this loss corresponds to taking ascent steps
along ∑
m ω 0 m∇θ Edπθ [rm(s, a)], which, per Theorem 2, is an ascent direction guaranteed to in-
crease all objectives. This simplified scenario essentially states the if we exactly match the preferences learned from Problem PA to the given preference vector ω0, then not only the PPA loss function reduces to the loss function for the scalarized reward, but also the ascent steps along its
gradient lead to concurrent improvements of all objectives. We show a geometric illustration of the PPA loss function in Fig. 2.
Despite its simplicity, one important obstacle in using the PPA loss to train agents in practice is that the optimal solution to the Problem PA is analytically known only for M = 2 (solution is presented in the Appendix). For M > 2, it can rather only be approximated by the iterative projected gradient descent solver introduced in Algorithm 1. This makes the backward pass challenging because the gradients need to be back-propagated through the (perhaps thousands) iterations of the projected gradient descent solver. Since back-propagation is known to be a very expensive operation compared to the forward pass, this could make our approach inefficient for practical problems. The problem of back-propagating through solutions of numerical solvers arises in several deep learning architectures such as deep equilibrium models (46) and optimization-based neural network layers (47). Luckily, the implicit value theorem allows us to back-propagate the gradient bypassing the operations of the projected gradient descent solver. Theorem 3. Let l := l(ω∗(θ)) be a generic loss function that depends explicitly on the the optimal solution ω∗(θ) of Problem PA. Then, the gradient of l with respect to the policy parameters is
∇θl = [ HT (θ) 0 0 ]  G(θ) −IM e−D(µ∗(θ)) −D(ω∗(θ)) 0 eT 0 0 −T  dldω∗0 0  , (7) where [G(θ)]m,n = 2gTm(θ)gn(θ), [H(θ)]m,· = − ∑ n ω ∗ n(θ)∇Tθ (gTn (θ)gm(θ)), gm(θ) = ∇θ Edπθ [rm(s, a)], µ∗(θ) is the optimal dual variable, e and 0 are the identity and zero vectors of appropriate dimensions, IM is the identity matrix of dimension M and D(·) create a diagonal matrix from a vector. When applied on a vector quantity, the operator ∇θ denotes the Jacobian matrix and the operator∇Tθ denotes the transpose of the underlying gradient or Jacobian.
The proof of Theorem 3 (given in the Appendix) leverages the implicit value theorem which enables us to differentiate the KKT optimality conditions of Problem PA with respect to the policy parameters. Observe that matrix H(θ) is essentially of dimension M × dθ, dθ being the dimension of θ. This dependence on dθ makes the solution of eq:nmatrix inefficient, as we need to compute and store a matrix of size (2M + 1)dθ. However, in practice, we are not interested in knowing the full gradient of ω∗(θ), but rather in back-propagating the gradient of the loss function; Eq. 7 has an appealing form which allows us to use vector-Jacobian products (VJPs) to evaluate the gradient ∇θl one row at a time. This frees us from having to store matrix H(θ) and compute its product with another matrix. We observe that the solution in Eq. 7 requires knowledge of the optimal dual variable µ∗(θ), which is not directly available. However, the optimal primal solution ω∗(θ) given by the projected gradient descent solver (Algorithm 1) can be utilized to obtain µ∗(θ) by substituting it into the KKT optimality conditions of Problem PA (details in the Appendix) and solving the resulting linear system. That system has M + 1 variables, therefore its solution adds minimal computational overhead.
It is worth discussing similarities and differences between PPA and EQL (24). PPA retrieves and adapts to preferences during the training phase, whereas EQL first learns a preference-dependant policy and then retrieves preferences using a combination of stochastic search and policy gradient. Also, the loss functions are partially similar: the penalty term in PPA is the MSE loss between the given and inferred preferences, whereas the EQL loss function is the MSE loss of vector Q-values. Additionally, given that PPA is a policy method and EQL is a value method, they can be combined in an actor-critic algorithm. This is a very interesting direction for future work.
One limitation of our work is that it relies on Pareto stationarity, which is a necessary but not sufficient condition for Pareto optimality. i.e., there is no guarantee for Pareto optimality only stationarity. Nonetheless, Pareto optimality is hard to be achieved and we often need to resort to first-order approximations. Finally, the optimal policy obtained under a given preference distribution is not, in general, optimal under a different distribution and the model has to be fine-tuned. We expect that we will not need to re-train the network entirely and that we can leverage the previous policy to find the new one. The technical details of how this would be achieved are a direction for future work.

heading: 5 EXPERIMENTAL EVALUATION
text: In this section, we evaluate the performance of our proposed method. In our implementation, we extended the popular Proximal Policy Optimization (PPO) (48) algorithm so that it can handle our PPA loss function. We implemented all experiments on PyTorch using modified versions of OpenAI Gym environments. We give the details of our experimental setup in the Appendix.
Domains: We evaluate on 4 environments (details given in the Appendix): (a) Multi-Objective GridWorld (MOWG): a variant of the classical gridworld, (b) Deep Sea Treasure (DST): a slightly modified version of classical multi-objective reinforcement learning enviroment (49), (c) MulitObjective SuperMario (MOSM): modified, multi-objective variant of the popular video game that has a 5-dimensional reward signal and (d) Multi-Objective MuJoCo (MOMU): a modified version of the MuJoCo physics simulator, focusing on locomotion tasks.
Ablation Study and Baselines: To highlight the extent to which our results are driven by the PPA loss function, we perform an ablation study. In more details, we consider the following three agents: (a) Pareto Policy Adaptation (PPA): This is our original loss function as introduced in Sec. 4. (b) non-adaptive PPA (na-PPA): Same as PPA but the preferences are static, i.e., the policy is not adapted to the preference distribution. This agent is essentially Algorithm 1 without the PPA loss function. (c) Fixed Preferences: Same as na-PPA but the preferences are given. This corresponds to optimizing the scalarized objective. Additionally, we compare the performance of our method against a range of state-of-the-art baselines. In particular, we compare against: the Pareto Following Algorithm (PFA) (31), the Conditional Neural network with Optimistic Linear Support (CNOLS)(22; 23), the Conditioned Network with Diverse Experience Replay (CN-DER) (23), the Envelope Q-Learning (EQL) (24) and the Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL) (34).
Evaluation Metrics: To evaluate the performance of our method and compare against the baselines, we use the following metrics:
1. HyperVolume (HV): Let F be a reward space Pareto front approximation and r0 be a reference vector. Then, we define the HyperVolume as HV := ∫ 1H(P )(z)dz, where
H(P ) = {z ∈ RM : ∃r ∈ F : r0 ≺P z ≺P r} and 1(·) is the indicator function. 2. Pareto Dominance (PD): Let r1t , r 2 t be the reward vectors of trajectories sampled
from agent 1 and 2, respectively. We define the Pareto Dominance metric as PD1,2 := 1 T ∑T t=0 1(r 1 t ≻P r2t ), where 1(·) is the indicator function. This is essentially the fraction of time when the reward vector under agent 1 Pareto dominates the reward vector under agent 2. Note that PD1,2 + PD2,1 ≤ 1.
3. Utility (UT): For a reward vector rt sampled from any agent, we define the Utility metric as UT := Eω[ 1T ∑T t=0 ω
Trt], where the expected value is evaluated by sampling uniformly from the preference space Ω or from the preference distribution Dω .
Main Results: Figure 3 shows the results of the ablation study conducted on the MOGW and DST domains. We compare against the non-adaptive and the fixedpreferences agents. For the later case, we pick three different preference vectors: (ω1, ω2) = (0.5, 0.5), (ω1, ω2) = (0.75, 0.25) and (ω1, ω2) = (0.25, 0.75). These plots facilitate the visual identification of Pareto dominant agents. Focusing on the MOGW (upper row), we observe that our PPA agent converges to a policy that yields a higher reward and lower timepenalty than any other agent, i..e, PPA Pareto dominates all other agents. While a similar statement cannot be made for the DST domain, we note that PPA is not Pareto dominated by any other agent and the only agent that it does not dominate
is the fixed-preference one for (ω1, ω2) = (0.5, 0.5). On the other hand, the non-adaptive PPA (na-PPA) seems to perform relatively poor. However, there exists some merit in using it because, in contrast to the fixed-preference agents, na-PPA does not rely on being given a preference vector. It is also worth noting that in the DST domain the preference vector (ω1, ω2) = (0.25, 0.75) gives very poor results, which implies that handpicking the preferences might lead to performance collapse. Overall, these results suggest that PPA steadily achieves better performance and is superior than the other agents. It is interesting to observe that the reward and time penalty curves are quite similar in the MOWG domain. This agrees with our intuition because in the MOGW domain the reward and time penalty have an ”inverse” relationship (i.e., the sooner we reach the bottom right, the higher the reward).
In the next line of experiments, we evaluate the performance of PPA in continuous control tasks. We modified the MuJoCo Gym interface to output 2-dimensional rewards on each step. The first one is the commonly used reward in such environments, i.e., the difference between two sequential positions of the robot. The second one is the energy consumed by the robot. We show the results on Fig. 4. We compare against the ablated agents and show the Pareto Dominance (PD) and Average
Utility (UT) metrics. The only case where our method does not attain the best performance is the Hopper-v3 environment under the UT metric, where the performance of the fixed-preferences agent (ω1, ω2) = (0.75, 0.25) slightly surpasses the performance of PPA. However, observe that the same agent attains very poor performance in the HalfCheetah-v3 enviroment. This suggests that, given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector is far from optimal and a preference vector that yields good performance in one environment may perform poorly in a different environment or judged under a different metric.
In the last line of experiments, we compare our method against state-of-the-art MORL algorithms (Table 1). Since PD is a pair-wise metric, we first compute the pair-wise PDi,j values for all agents and then average-out the j component to obtain a single value for each agent. Note that this metric is specific to the ensemble of agents that we consider and needs to be re-calculated if we compare a different set of agents. The UT metric is calculated by sampling preferences from the preference space of each domain. We assume assume a uniform preference distribution and sample 1000 points for the 2-dimensional preference case (MOGW, DST, MUMO) and 100 across each dimension for the MOSM case, where the preference space has 4 dimensions. Additionally, we scale the rewards in the range [−1, 1] across each dimension. This scaling is important when calculating the average utility but has no particular influence on the Pareto dominance or hypervolume. Observe that some agents have negative utility, which is indeed possible given the aforementioned reward scaling. Our method systematically outperforms all other baselines across all domains, which is another strong empirical finding for its performance.

heading: 6 CONCLUSION
text: We introduced a policy gradient algorithm for Multi-Objective Reinforcement Learning (MORL) under linear preferences. By approximating the Pareto front via a first-order necessary condition, our method learns a single policy that can be adapted to any preference distribution. The two pillars of our method is a projected gradient descent solver that searches for common ascent direction for all objectives and a novel Pareto Policy Adaptation (PPA) loss function that leverages the solutions of that solver to adapt the policy to any preference distribution. Our method enjoys the advantages of both single and multiple policy MORL algorithms while being simple to implement and compatible with all policy gradient and actor-critic methods. We demonstrated the effectiveness of our method in several reinforcement learning tasks.

heading: A APPENDIX

heading: A THEORETICAL FRAMEWORK
text: In the first part of the Appendix, we give the proofs omitted in Sections 2 and 4 as well as the analytical solution of Problem PA for the special case of 2-dimensional reward signals. For the ease of reference, we re-state the underlying theorem.
A.1 PROOFS
Lemma 1. Let π ∈ C∗Π,Ω be a policy. Then, we have Eπ[r̃] ∈ C∗R,Ω, i.e., the expected return vector under policy π belongs in the RS-CCS.
Proof. Consider a policy π ∈ C∗Π,Ω and a policy π′ ∈ Π such that dπ ′ (s, a) = 0 wherever dπ(s, a) = 0. Then, we can write
vπ ′ = ∑ s,a dπ ′ (s, a)r(s, a) = ∑ s,a dπ(s, a) dπ ′ (s, a) dπ(s, a) r(s, a) = Edπ [r′(s, a)], (8)
where we used the importance sampling identity and defined r′(s, a) := dπ ′ (s, a)/dπ(s, a)r(s, a). Per Def. 2, there exists ω ∈ Ω such that ωTvπ ≥ ωTvπ′ . Leveraging the fact that the value function can be expressed as vπ = Eπ[r̃] as well as Eq. equation 8, we have ωT Eπ[r̃] ≥ ωT Eπ[r̃′]. This implies that Eπ[r̃] ∈ C∗R,Ω and concludes the proof.
Remark 1. Under the additional assumption of normalized rewards, we can prove the converse of Lemma 1. However, since we are are concerned with approximating C∗R,Ω by C∗Π,Ω (and not the other way around), we refrain from making that additional assumption to prove the converse of Lemma 1.
Theorem 3. Let l := l(ω∗(θ)) be a generic loss function that depends explicitly on the the optimal solution ω∗(θ) of Problem PA. Then, the gradient of l with respect to the policy parameters is
∇θl = [ HT (θ) 0 0 ]  G(θ) −IM e−D(µ∗(θ)) −D(ω∗(θ)) 0 eT 0 0 −T  dldω∗0 0  , (9) where [G(θ)]m,n = 2gTm(θ)gn(θ), [H(θ)]m,· = − ∑ n ω ∗ n(θ)∇Tθ (gTn (θ)gm(θ)), e and 0 are the identity and zero vectors of appropriate dimensions, IM is the identity matrix of dimension M and D(·) create a diagonal matrix from a vector.
Proof. We start by writing the Lagrangian of Problem PA
L(ω, λ,µ) = ∥∥∥∥∥ M∑
m=1
ωmgm(θ) ∥∥∥∥∥ 2 + λ ( M∑ m=1 ωm − 1 ) + M∑ m=1 µmωm, (10)
where gm(θ) = ∇θ Edπθ [rm(s, a)]. Let (ω∗(θ), λ∗(θ),µ∗(θ)) be the policy-dependant optimal primal and dual variables. The KKT conditions for stationarity, primal feasibility and complementary slackness are
∂L
∂ω∗m(θ) = 2 M∑ n=1 ω∗n(θ)g T n (θ)gm(θ) + λ ∗(θ) + µ∗m(θ) = 0, m = 1, 2 . . .M (11a)
∂L
∂λ∗(θ) = λ∗(θ) ( M∑ m=1 ω∗m(θ)− 1 ) = 0 (11b)
∂L
∂µ∗m(θ) = µ∗m(θ)ω ∗(θ) = 0, m = 1, 2 . . .M, (11c)
By taking the gradient of the above equations with respect to the policy parameters, we obtain
2 M∑ n=1 ∇θω∗n(θ)gTn (θ)gm(θ) + 2 M∑ n=1 ω∗n(θ)∇θ ( gTn (θ)gm(θ) ) +
∇θλ∗(θ) +∇θµ∗m(θ) = 0, m = 1, 2 . . .M, (12a) M∑
m=1
∇θω∗m(θ) = 0, (12b)
∇θµ∗m(θ)ω∗(θ) + µ∗m(θ)∇θω∗(θ) = 0, m = 1, 2 . . .M. (12c) Using matrix notation, we can rewrite the above system of equations as follows: G(θ) −IM e−D(µ∗(θ)) −D(ω∗(θ)) 0
eT 0 0 ∇Tθ ω∗(θ)∇Tθ µ∗(θ) ∇Tθ λ∗(θ)  = [H(θ)0 0 ] . (13)
By inverting Eq. equation 13 and using the chain rule ∇θl(ω(θ)) = ∂l∂ω∇θl we obtain the claimed result.
We observe that the solution of Eq. equation 13 requires knowledge of the optimal dual variable µ∗(θ), which is not directly available when we use the projected gradient descent solver of Algorithm 1; that solver only gives us the optimal primal variable ω∗(θ). However, we can substitute that optimal ω∗(θ) into the KKT conditions given by Eq. (11a–11c) and solve the resulting linear system of M +1 variables to find the optimal dual variables. The number of variables in that system depends only on the dimension of the reward signal M and not on the dimension of θ, therefore it adds minimal computational overhead. This step would not be necessary if we chose a primal-dual method for solving Problem PA. However, primal-dual methods may add to the complexity of the implementation, which may not be worthy given the simplicity of Problem PA. Exploring such a trade-off is an interesting direction for future work.
A.2 PARETO STATIONARITY: 2-DIMENSIONAL CASE
In general, the problem of finding common ascent directions (Problem PA) can be solved either exactly using the methods from the minimum norm points literature (as discussed in Sec. 3) or approximately using convex optimization methods. Due to the dimensionality of the gradient vector, the former class of methods are not applicable in our case. Therefore, we resorted to the projected gradient descent solver presented in Algorithm 1. This solver is essentially an internal optimizer which we run before every parameter update. Luckily, Problem PA is convex and the gradient descent optimizer converges rapidly in practice with little tuning. Additionally, for the special case of two objectives, we can leverage the analytical solution to achieve further performance improvements. We re-state Problem PA for convenience bellow:
min ϵ1,...ϵM {∥∥∥∥∥ M∑ m=1 ϵm∇θ Edπθ [rm(s, a)] ∥∥∥∥∥ 2
2
s.t M∑
m=1
ϵm = 1, ϵm ≥ 0, ∀m } . (PA)
For M = 2, the problem reduces to minϵ∈[0,1] ∥ϵ∇θ Edπθ [r1(s, a)] + (1− ϵ)∇θ Edπθ [r2(s, a)]∥ 2 2, which is a one-dimensional quadratic optimization problem of a single variable and has the following analytical solution:
ϵ∗ =
[ (∇θ Edπθ [r2(s, a)]−∇θ Edπθ [r1(s, a)])T∇θ Edπθ [r2(s, a)]
∥∇θ Edπθ [r2(s, a)]−∇θ Edπθ [r1(s, a)]∥22
] + , (14)
where [·]+ represents the clipping operator, i.e., [x]+ = max (min (x, 1), 0). Geometrically, Eq. equation 14 implies that minimal-norm point lies either on the boundary on the convex hull (ϵ = 0 or ϵ = 1) and the desired descent direction is parallel to corresponding gradient or is equal to the vector perpendicular to the difference of the gradients. The former one is a corner case and occurs when the ascent direction for one of the objectives is a non-decreasing direction for the other one. While this is only applicable to M = 2, it enables us to replace the inner projected gradient descent optimizer in Algorithm 1 with the analytical solution in Eq. equation 14. Additionally, this special case is applicable to several real-world cases since there are several problems that have the reward-cost structure.

heading: B EXPERIMENTAL DETAILS
text: B.1 ENVIRONMENTS
Our implementation uses modified versions of 4 OpenAI Gym enviroments (Fig. 5). The first two are synthetic domains and act as a proof-of-concept for our method, while the other two are complex, real domains. The details are listed below:
Multi-Objective GridWorld (MOGW): A multi-objective variant of the classic gridworld environment. The agent starts from the top-left corner, moves in a 16 × 16-grid and receives a reward equal to 1 when it reaches the bottom-right corner. Each action results in a small time penalty of −0.1. Any policy that results in a trajectory that minimizes the taxicab distance between the agent’s starting position and the goal is Pareto optimal. Our implementation uses the gym-minigrid1 enviroment, which we modify to output the aforementioned 2-dimensional reward.
Deep Sea Treasure (DST): A classic multi-objective reinforcement learning environment. The agent is located in a 10 × 11-grid and controls a submarine searching for treasures. There are multiple treasures the value of which increases as their distance increases. The agent receives a time penalty of −1 for each action. The Pareto front of non-dominated policies can be easily extracted (49) and is also shown on Fig. 1. As in the MOGW case, we use the gym-minigrid package and extend it to the reward map shown in Fig. 5.
Multi-Objective SuperMario (MOSM): We use a modified, multi-objective variant of the popular video game. There are 5 scalar rewards: x-pos: the difference in Mario’s horizontal position between current and last time point, time: a small time penalty of −0.1, deaths: a large negative penalty of −100 each time Mario dies, coin: rewards for collecting coins, and enemy: rewards for eliminating an enemy. We use the gym-super-mario-bros2 package and the multi-objective extension developed in (24).
Multi-Objective MuJoCo (MOMU): We focus on locomotion tasks and modify the environments provided by the MuJoCo physics simulator to output vector rewards. To do so, we create a custom wrapper for the built-in OpenAI Gym interface. There are 2 scalar rewards: x-pos: the difference in the robot’s current and last horizontal position, r-pos: a positive reward for moving in circular trajectories.
B.2 IMPLEMENTATION DETAILS
We implement our algorithm in PyTorch building on top of the torch-ac package3. Our implementation extends the Proximal Policy Optimization (PPO) algorithm to handle our PPA loss function (Eq. equation 6). We use the clipped version of PPO with the clip ratio threshold set to 0.2. We set the GAE parameter to λ = 0.95 and the discount factor to γ = 0.99.
Network Architecture: We use a consistent network architecture across all our simulation for both the actor and the critic: Two hidden layers of 64 and 64 neurons each and a ReLU non-linearity. In the discreet action case (MOGW, DST, MOSM), we append the actor network with an output layer of dimension equal to the number of actions. In the MuJoCo enviroments, where the actions are
1https://github.com/maximecb/gym-minigrid 2https://github.com/Kautenja/gym-super-mario-bros 3https://github.com/lcswillems/torch-ac
continous, we use that output layer to parametrize a multi-dimensional Gaussian distribution, via its mean and variance, assuming uncorrelated actions. Each non-terminal layer has a ReLU nonlinearity. Even though the hyperbolic tangent is known to be preferred for reinforcement learning applications, we experimentally observed that it leads to vanishing gradients and slows down learning. This may be due to the fact that the PPA loss function requires second-differentiation. In the SuperMario enviroment, we worked on the pixel domain and two convolutional layer of 32 and 64 layers as feature extractors, which were shared between the actor and the critic.
Hyperparameters and Training: To train the neural network, we use the Adam optimizer (β1 = 0.9, β2 = 0.999) with a learning rate of 0.0001. In the MOGW, DST and MOMU domains, we optimize the parameters of the actor and critic separately. In the MOSM domain, given that the actor and critic share the convolutional layers, we sum their losses (using weights 1 and 0.5, respectively) and use a common optimizer. We run all of our simulations in the Google Cloud Platorm using 48 vCores and one NVIDIA Tesla T4 GPU. For the MOGW and DST environments, we train the agent for 0.5M frames and for the MOSM and MOMU for 1M frames. For each 512 frames we perform one update of the network parameters iterating over 10 epochs of the collected data and using a mini-batch size of 64.

Review summaries:
The paper presents a new method for solving MORL problems. The main contributions are: a generic gradient descent MORL algorithm for finding multiple solutions, and another algorithm to retrieves preferences from the given solutions. The authors also present theoretical proofs of their method.

The paper presents a multi-objective method for linear preferences. The key concept used for the design of the approach is that there always exists a common ascent direction for all the objectives. The authors leverage this property for introducing a novel loss function that can be integrated into different DeepRL methods. The proposed approach is simple and proved to be effective in the proposed experiments.

This paper introduces a policy gradient approach to multi objective RL. The authors learn show that an optimal solution for a specific reward vector can be found by minimizing a constrained quadratic optimization problem. They propose to solve this problem using projected gradient descent. Then, the authors offer an efficient solution for optimizing the Pareto front through "Pareto Policy Adaptation" loss function, which trades off learning the Pareto front and the policy. The authors conduct experiments and ablations studies on four benchmarks, validating their method with improvement over current sota approaches for MORL.