Pareto Policy Adaptation
Authors: Panagiotis Kyriakis
Jyotirmoy Deshmukh
Paul Bogdan
Venue: ICLR 2022 Poster (2022)
Keywords: multi-objective reinforcement learning
policy gradient
pareto optimality
policy adaptation

Abstract: We present a policy gradient method for Multi-Objective Reinforcement Learning under unknown, linear preferences. By enforcing Pareto stationarity, a first-order condition for Pareto optimality, we are able to design a simple policy gradient algorithm that approximates the Pareto front and infers the unknown preferences. Our method relies on a projected gradient descent solver that identifies common ascent directions for all objectives. Leveraging the solution of that solver, we introduce Pareto Policy Adaptation (PPA), a loss function that adapts the policy to be optimal with respect to any distribution over preferences. PPA uses implicit differentiation to back-propagate the loss gradient bypassing the operations of the projected gradient descent solver. Our approach is straightforward, easy to implement and can be used with all existing policy gradient and actor-critic methods. We evaluate our method in a series of reinforcement learning tasks

Published as a conference paper at ICLR 2022
PARETO POLICY ADAPTATION
Panagiotis Kyriakis University Southern Califorin Los Angeles, USA
Jyotirmoy V. Deshmukh
Paul Bogdan
University Southem Califomia Los Angeles, USA jde jdeshmuk@use edu
University Southem California Los Angeles, USA
pkyriaki&usc. edu
pbogdan edu
ABSTRACT
We present a policy gradient method for Multi-Objectivv Reinforcement Learning under unknown, linear preferences. By enforcing Pareto stationarity, a first-order condition for Pareto optimality, we are able to design a simple policy gradient al- gorithm that approximates the Pareto front and infers the unknown preferences. Our method relies on projected gradient descent solver that identifies common ascent directions for all objectives. Leveraging the solution of that solver, we in- troduce Pureto Policy Adaptation (PPA), a loss function that adapts the policy to be optimal with respect to any distribution over preferences. PPA uses implicit differentiation to back- pook-pro the loss gradient bypassing the operations of the projected gradient descent solver. Our approach is straightforward, easy imple- ment and can be used with all existing policy gradient and actor-critic methods. We evaluate our method in a series of reinforcement learning tasks.
INTRODUCTION
Deep reinforcement learning has a pivotal role in solving several control problems of practical in- terest in multi-agent autonomy (1), robotics, and cyber-physical systems (2) that would otherwise be intractable, such as robotic locomotion (3), the Atari (4) and Go games (5) to name a few. While such approaches have focused on the scalar reward setting, there exist many real-world problems that have multiple, conflicting objectives and, therefore, cannot be addressed by the current rein- forcement learning tools. In such scenario, it is challenging to find an optimal control policy as the trade-offs (preferences) among the objectives may not be precisely known at training time or may differ from user to user. The paradigm of Multi-Ohjective Reinforcement Learning (MORL) provides a generalized framework for dealing with mimti-ddimenionall rewards signals and has been identified as one of the main challenges real-world reinforcement learning (6). However, learn- ing optimal policies for MORL has been proven to be quite challenging, because most strategies require either to have access to or be able to infer a quantification of the relative importance of the objectives, or they perform sophisticated searches in the value policy space aiming to find an ensemble of policies that are non-inferior to each other. The former methods lack adaptability and generalizability, as their performance is tied the given inferred preferences, while the later suffer from scalability issues, because they learn multiple policies, and are usually complicated to implement. In this work, we aim strike balance between those two families of methods.
Related Work: Multi-Objective Optimization (MOO) provides the fundamental tools for MORL. In MOO, there are two common solution concepts: scalarization and Pareto optimality. The former one derives a scalar objective and uses standard single-objective optimization techniques (7; 8; 9; 10: 11). The later method based on the concept of Pareto dominance and considers the set of all non-inferior solutions (12; 13). Multiple gradient methods that leverage first-order, necessary conditions for Pareto optimality have also been developed (14; 15; 16).
Similarly to MOO, existing work in MORL can be roughly divided into two main categories: Single- policy methods aim to maximize a single, scalarized reward. These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences. Scalarization is usually performed using a weighted sum of the reward vector (17; 18; 19) or, less commonly, using linear mappings (20). Different single-policy methods are based on thresholds or lexicographical ordering (21) or different classes of preferences (22; 23). More recently, scalarized Q-learning algorithm has been developed in (24) which uses the concept of

---

Published as a conference paper at ICLR 2022
corner weight to infer and optimize over preferences. It was extended to handle dynamic preferences in (25) and to utilize the convex envelope of the Pareto front in (26). Finally, a inveriinva supervised learning approach for encoding the preferences was developed in (27).
On the other hand, multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front. The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple policy methods mainly differ on the approximation scheme for the Pareto front. One common approach is to perform multiple runs (18; 17) of a single scalarized reward function over a set of preferences. Unfortunately, such methods lack scalability to high- nigh-dimensional rewards. Other approaches lever- age convex approximations of the Pareto front (28) or linear approximations of the value function (29; 30) to learn optimal deterministic policies. Multi-ohjective fitted Q- Q-iteration (31; 32) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator. A policy gradient method based on discrete approximations of the Pareto front was proposed in (33) and enhanced to utilize continuous approximations in (34). A gradient-based method that learns a manifold on the policy parameter space leveraging episodic ex- ploration and importance sampling algorithm which is able to find dense approximation to the Pareto front was proposed in (36).
developed in (35). Finally, a prediction guided evolutionary
Single-policy methods have the advantage of learning a single policy and being simple to implement. However, they suffer from instability issues, as a small change on the preferences may lead to performance collapse (37), and also rely heuristics to infer the preferences when they are not known. On the other hand, multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution (being an approximation the Pareto front) encapsulates all trade-offs between the objectives. However, this benefit comes at a higher computational cost as we typically have to learn and store multiple policies. In this paper, we aim to bridge the gab between single and multiple policy MORL methods: Our method learns single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution. Our preference vectors are inferred, not by using heuristics, but by approximating the Pareto front via a first-order condition (see Fig. for an example). Our method is inspired by the multiple gradient descent algorithm for MOO introduced in (15) well as its application in the field of multi-task supervised learning presented in (38). Contributions: We propose a policy gradient method for multi-objective reinforcement learning under unknown, linear preferences. Initially, we present Pareto stationarity as a necessary, first- order condition for Pareto optimality and develop multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objec- tives. Following that, we tackle the problem of adapt- ing the policy gradient to any given preference distri- bution. We utilize our method from the first part and introduce the Pareto Policy Adaptation (PPA), loss function that penalizes deviations between the given preference distribution and the recovered preferences. Using implicit differentiation, we are able to back-propagate the gradient of PPA bypassing the op- erations of the projected gradient descent solver, which makes our method applicable to real-world problems. We evaluate our method in a series reinforcement learning tasks.
15
Incuure
10
Pareto Front (ours)
time penalty
Figure 1: True and identified Pareto front for Deep Sea Treasure enviroment.
The most closely related work to ours is presented in (33). The authors discuss the existence of common ascent directions, introduce a quadratic optimization problem for their computation and give two gradient-based learning algorithms. Even though our method identifies the common ascent directions using similar optimization problem, we leverage these directions to define the PPA loss function, which is a simple and efficient way 1) to incorporate the ascent directions into any learning algorithm and 2) to account for given preference distribution. Leveraging implicit differentiation, PPA enables us to incorporate multiple rewards and preference distributions into any modern policy gradient algorithm by a mere modification of the loss function and by implementing projected gradient descent optimizer.
2

---

Published as a conference paper at ICLR 2022
2 PRELIMINARIES
We consider the framework of Multi-Objective Markov Decision Process (MOMDP) M = (S, .. 7.,, 22, fn), where S and A denote the (discrete) state and action spaces, P S x A x [0. is the transition kernel representing the environment dynamics, r(s, a) = R S A I is function mapping state-action tuples to reward vectors, Ω is the space of preferences and fw RM + R a preference function which produces a scalar utility using a preference
Ωt To declutter notation, we assume that there exists given initial state. We consider linear preferences (w((s,,)) a)) = w"r(s,a) A fixed preference vector w E Ω al- lows to directly compare the vectorized reward and value function by comparing their scalarized utilities, and, therefore, the MOMDP collapses to a standard MDP. A stationary control policy S A - P(A), where P(A) the Borel set of A. is a function that assigns probability distribution over actions for all states. The value y= : S + RM of a policy π is defined as v (s) := Ex ( 7 (sr(st, at) 8o = s), where E (0, 1) is discount factor and the expectation taken under the distribution over trajectories
(su. ag. $1, a, ...)... obtained by starting from and following policy x thereafter. The discounted state-action occupation measure under policy π defined as dd (8,a) a) == Ei= P(s 8, 04 = a π) for all states 8 E S and allows us to write the value function compactly as (s) = Rx- (r(s 80 s).
Definition 1. Considering all possible returns F -t= - az), we define the Reward Space Pareto Coverage Set (RS-PCS) or Pareto frontier Fr+ : (F $ E RM F' >r f), where the symbol Pp denotes Pareto dominance: greater or equal in all objectives and strictly greater in at least one objective. For all possible preferences in Q2 and under the linear preference assumption, we define the Reward Space Convex Coverage Set (RS-CCS) as
CR : { F R, 3w E s.t. w 7 WT', RR.
The set CR contains all discounted reward vectors that could be optimal for some preference vector in Q. Anything in F but not in R.. cannot be useful under linear preferences. In that case, it suffices to restrict our analysis to CR, Also, we assume, without loss of generality, that Em Wm 1, 1. 2 0, 1,2... M for all w E Ω i.e., the preference vector is convex combination of the objectives. We extend the definition of Pareto optimality from the reward space to the policy space.
Definition 2. A policy x dominates policy x' if and only if vz >p v² A policy x¹ is called Pareto optimal if there exists no other policy π that dominates '². The set II of all Pareto optimal policies is called Pareto policy set and its image F= = {v" TrEN is called Pareto policy frontier. The set
Ch.a = ( II, Be E s.s.t. wt o² 2 w Va' E III, Σ Jum = 1, 2 0}
(2)
m
is called the Policy Space Convex Coverage Set (PS- -CS).
Lemma 1. Let E Ci be policy. Then, have E,, [F] CR i.e., the expected return vector under policy π belongs in the RS-CCS.
Lemma (proof given in the Appendix) connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later. This very convenient because searching for Pareto optimal vectors in the reward space is not practically efficient (e.g., due to the sparsity of reward signal) or useful (e.g., because they cannot be easily mapped values or actions). By finding a policy π E Ci.o- Lemma guarantees that the expected return vector under belongs in CR.a and, therefore, maps the problem of reward space exploration to policy space exploration. Additionally, Lemma 1 imposes restrictions on the class of policies and allows us to adopt the common prac- tice of parameterizing the policy by nigh-dimensional vector e θ (c.g., a neural network). The definition of PS-CCS given Eq. equation 3 carries over to this parametric class as is, i.c. we can define the PS-CCS in the policy parameter space as follows
Ca = {0 0 E O, Fe E Ω s.t. T
To 2 WTT 40 E E
3 LEARNING WITH UNKNOWN PREFERENCES
In this section, we propose a new policy gradient algorithm for MORL under unknown preferences. The key idea of our approach is to infer the preference vector w, use that scalarization coeffi-
3

---

Published as a conference paper at ICLR 2022
cient for the multi-objective criterion and ascent the gradient of the scalarized value. To search for a Pareto optimal policy, we leverage a fist-order condition known as Pareto stationarity.
policy To is called Pareto stationary and only if Σm= Wm Ve Eg I'm (s, a)] = 0 for some wis E [0, m = M..
Definition 3 (Pareto stationarity).
Pareto stationarity implies that there exists vanishing convex combination of the objectives" gra- dients and is necessary condition for Pareto optimality (15). The following theorem connect the PS-CCS with Pareto stationarity.
Theorem 1. Let πTe E Ca be a policy in the PS-CCS. Then, we Pareto stationary.
Proof. Let Ty E Ca. be a policy and w the corresponding preferences, i.e. we have wa aT > wTv"er V0' E O. The last relation be written w" [x] 2 w" Eng [F], V0' O. This implies that global maximizer of wt Err [F] or, equivalently, that its gradient at 0 vanishes. Therefore, we have
Vew"
Edro
Wm Vo Ed ('m a)] =
which corresponds the necessary condition for Pareto stationarity as per Def. 3.
Theorem 1 essentially states that Pareto stationarity is a necessary condition for policy to belong to PS-CCS. Additionally, as stated in the proof, the convex combination the objectives that leads to vanishing gradients coincides with the preferences that make the underlying policy part of the PS-CCS. This presents us with a strong necessary condition for recovering the unknown preferences and for identifying optimal policy. This is because the problem of finding Pareto stationary is tractable via gradient methods. In fact, a policy is not Pareto stationary then there exists a common ascent direction for all objectives. To find this descent direction, use the following theorem.
Theorem 2 ((15), Theorem 1). Consider the following quadratic optimization problem
{
M Σ m==1
M
min Wi
wm Vo (Tm(s
5.t
Σ
Wm =, wm 2 O, Vm
(PA)
m=m1
Let w be the optimal solution of Problem PA and μ* the underlying minimal-none nnrm objective. Then:
1. either μ= and TT- Pareto stationary policy
2. ΣM wm V Eggo [Tm a)) is common ascent direction for all objectives.
Problem PA is essentially equivalent to the formulation of the famous Winimum-Norm Point prob- lem, which arises many fields such as optimal control (39), submodular minimization (40) and portfolio optimization (41). The MNP problem has been well studied and several combinatorial and recursive algorithms have been proposed (42; 43; 44). Such methods search for the exact solution and typically have exponential run-time (45), which prevents their application in nigh-dimensional control policies. Therefore, we adopt convex optimization approach for Problem 2. We start by
noting that the constrain set ع = {w E RM, EM Wm = 1, wm 2 0, Vm = 1.2...M is essen- tially the unit simplex and the projection problem, i.e., (T( = arg min ||ا - ø|1. can be effi- ciently solved. In fact, there exists a unique dual variable IR such that
(w) = (w)
(5)
where (x)+ max(x, (z. 0). To find the dual variable, we enforce the equality constraint, i.e., 1. This non-linear equation of single variable be solved very efficiently
Em Em(um - -)+ using the Newton method. This suggests a projected gradient descent solver for solving Problem PA and recovering the unknown preferences.
Our method (Algorithm 1) extends the classical REINFORCE algorithm (46) to take gradient steps in the direction given by the weighted combination of the multi-objective gradient, with the weights being the preferences recovered by the projected gradient descent solver. Lemma 1, Theorem and 2 essentially state that Algorithm learns arbitrary policies in the Convex Coverage Set along with
4

---

Published as a conference paper at ICLR 2022
their underlying preference vectors and provide theoretical justification for method. Algorithm is simple to implement, compatible with all policy gradient and actor-critic methods and provides practical algorithm for learning Pareto optimal policies under unknown preferences. However, it suffers from a subtle drawback: we have no control over the identified preferences and they cannot be optimized account for any given preference distribution. This is the problem of policy adaptation which we address in the next section.
is worth comparing our projected gradient descent solver with (38), where the authors used the Frank-
Algorithm 1: Multi-Objectivg Objective Policy Gradient
Inputs: Initial parameters 0o, learning rate a foreach k == 0, 1,... do
Wolfe method for Problem PA. albeit
in the context of supervised learning. Both algorithms have the same con- vergence rate of O(1/N) and since the projection to the constraint set can be easily calculated, we opted for the projected gradient descent optimizer. The overhead added by solving for the dual variable is minimal be- cause the Newton's method is known to enjoy quadratic convergence rate and is very efficient in practice. The trade-off of using Frank-Wolfe is that 12
: Collect trajectories D = [Ti], T3 ~ Te.
Compute rewards-to-go
9m + [D] Σ rı log mo, (aelst)
(wi,...W WM) t PGDSolver(gn... -MM a)
ac+1 07 Em=1 Wm9m
M
end
procedure PGDSolver (91, ................................ 9M. a):
10 Initialize feasible w = (W1, WM) while not converged do
Find t solving Σm (wm - +)+ =
requires to obtain the feasible mini- mizer of the linear approximation of the objective around the current iter- ant, which is typically solved approx- imately and worsens the convergence rate by a multiplicative constant (47). Nonetheless, we expect that both methods can be used inter- changeably in practice.
Wm + Wm + agin 2n engnn m. = 1.... ....N M
w (w (w-T)) +)+
end
return (wi,...WW.) WAs)
4 PARETO POLICY ADAPTATION
The policy gradient algorithm presented in the previous section finds policies that are optimal (i.e.. Pareto stationary) for the inferred preferences. In this section, we show how to address the problem of adapting the policy to be optimal for a given distribution Da over preferences. We start by expressing the optimal solution of Problem PA as function of the policy parameters w- = (0) = (0)) (O).. We introduce the Pareto Policy Adaptation loss func-
tion follows:
(()) Exe
A
Σ (0)r - A) wl (0) w
(6)
where A E 1) I] is weight that trades off between the two losses. The first term of PPA is the scalarized reward, with the scalarization coefficient being the inferred preferences. The second term penalizes the expected deviation between the given and inferred preferences. To further mo- tivate the PPA loss, consider the special case of a single, given preference vector w". In this case, the MOMDP collapses to standard MDP, with wh being the scalarization coefficient for the re- ward and value functions. Additionally, assume that the policy Ty has been adapted to that pref- erence vector, i.c., it achieves w (0) = w for some parameter vector 0. Then, Eq. 6 reduces to L(0) = Earo [IM "m,nms (s, a)). Maximization of this loss corresponds taking ascent steps along En, w Vo Eggo [rmmss, a)). which, per Theorem 2, is an ascent direction guaranteed to in- crease all objectives. This simplified scenario essentially states the if we exactly match the pref- erences learned from Problem PA to the given preference vector w then not only the PPA loss function reduces to the loss function for the scalarized reward, but also the ascent steps along its gradient lead to concurrent improvements of all objectives. We show a geometric illustration of the PPA loss function in Fig. 2.
Despite its simplicity, one important obstacle in using the PPA loss to train agents in practice is that the optimal solution to the Problem PA is analytically known only for M 2 (solution is presented
5

---

Published as a conference paper at ICLR 2022
0.6
1.1
15
xg
(a)
(b)
(c)
Figure 2: (a) The Convex Coverage Set (CCS) for 2-dimensional value function. Algorithm I identi- fies arhitrary solutions the Pareto front. (h) Illustration of the PPA loss function for single preference the solution converges to the optimal solution cor- responding preference w. (c) PPA loss function for two preference vectors w w the minimizer of (0) (0) w - (0) w')l (also called the geometric median) gives the optimal solution because maximizes the projections onto the preference vectors.
vector wh: by minimizing the distance ||ا (0) - ='I|, ||
the Appendix). For M > 2, it can rather only be approximated by the iterative projected gradient descent solver introduced in Algorithm 1. This makes the backward pass challenging because the gradients need to be back pooo-propagated through the (perhaps thousands) iterations of the projected gradient descent solver. Since back-propagation is known to be very expensive operation compared to the forward pass, this could make our approach inefficient for practical problems. The problem of back propaapppgggting through solutions of numerical solvers arises in several deep learning architectures such as deep equilibrium models (48) and optimization-ball bbaeed neural network layers (49). Luckily, the implicit value theorem allows projected gradient descent solver.
back-propagate the gradient bypassing the operations of the
Theorem 3. Let := I(w" (0)) be a generic loss function that depends explicitly on the the optimal solution w" (0) of Problem PA. Then, the gradient of with respect the policy parameters
(0)
G(0)
-TM (0)
TT
d!
des
Vol (T(()) 0 0
D(p (0))
-
0
0
0
where (G(0) No Ex I'mm (s, a)), μ (B) the optimal dual variable, e and O are the identity and zero vectors of appropriate dimensions, IM is the identity matrix of dimension M and D(.) create a diagonal matrix from a vector. When applied on vector quantity, the operator Vo denotes the Jacobian matrix and the operator V denotes the transpose of the underlying gradient or Jacobian.
29 (0) (0), [H(0) - En w (0) (gTh (0))m(()) (0) =
The proof of Theorem 3 (given in the Appendix) leverages the implicit value theorem which en- ables differentiate the KKT optimality conditions of Problem PA with respect to the policy parameters. Observe that matrix (f) is essentially of dimension M x dg. de being the dimension of 0. This dependence on de makes the solution of eq:ımatrix inefficient, and store a matrix of size (2M + 1)da. However, in practice, we are not interested in knowing the full gradient of w" (0), but rather in back-propag the gradient of the loss function; Eq. has an appealing form which allows us to use vector-Jacobian products (VJPs) to evaluate the gradient Val one row time. This frees us from having to store matrix (()) and compute its product with another matrix. We observe that the solution in Eq. 7 requires knowledge of the optimal dual variable 1 (0), which is not directly available. However, the optimal primal solution w a0) given by the projected gradient descent solver (Algorithm I) can be utilized to obtain μ u²()) by substi- tuting it into the KKT optimality conditions of Problem PA (details in the Appendix) and solving the resulting linear system. That system has M variables, therefore its solution adds minimal computational overhead.
we need to compute
It is worth discussing similarities and differences between PPA and EQL (26). PPA retrieves and adapts to preferences during policy and then retrieves preferences using combination of stochastic search and policy gradient.
the training phase, whereas EQL first learns a

---

Published as a conference paper at ICLR 2022
Also, the loss functions are partially similar: the penalty term in PPA is the MSE loss between the given and inferred preferences, whereas the EQL loss function is the MSE loss of vector Q-values. Finally, the optimal policy obtained under a given preference distribution is not, in general, optimal under a different distribution and the model has to he fine-tuned. We expect that we will not need to re-train the network entirely and that we can leverage the previous policy to find the new one. This is an interesting direction for future work.
5 EXPERIMENTAL EVALUATION
In this section, we evaluate the performance of our proposed method. In our implementation, we extended the popular Proximal Policy Optimization (PPO) (50) algorithm that it can handle our PPA loss function. We implemented all experiments on Py PyTorch using modified versions of OpenAI Gym environments. We give the details of our experimental setup in the Appendix.
Domains: We evaluate on 4 environments (details given in the Appendix): (a) Multi- Mujti-Objecttive GridWorld (MOWG): variant of the classical gridworld, (b) Deep Sea Treasure (DST): slightly modified version of classical multi-objective reinforcement learning enviroment (51), (c) Mulit- Objective SuperMario (MOSM): modified, multi-objective variant of the popular video game that has 5-dimensional reward signal and (d) Multi-Objective MuJoCo (MOMU): a modified version of the MuJoCo physics simulator, focusing on locomotion tasks.
To highlight the extent to which our results are driven by the PPA loss function, we perform an ablation study. In more details, we consider the following three agents: (a) Pareto Policy Adaptation (PPA): This is our original loss function as introduced in Sec. 4. (b) non-adaptive PPA (na-PPA): Same as PPA but the preferences are static, i.e., the policy is not adapted to the preference distribution. This agent is essentially Algorithm I without the PPA loss function.
Ablation Study and Baselines:
(c) Fixed Preferences: Same as na- PPA but the preferences are given. This corresponds to optimizing the scalarized objective. Additionally, we compare the performance of our method against a range of state-of-the-art baselines. In particular, we compare against: the Pareto Follow- ing Algorithm (PFA) (33), the Conditional Neural network with Optimistic Linear Support (CN- (2S)(24: 25), the Conditioned Network with Diverse Experience Replay (CN-DER) (25), the En- velope Q-Learning (EQL.) (26) and the Drediction-Guidee Multi-Objective Reinforcement Learning (PGMORL) (36).
Evaluation Metrics: To evaluate the performance of our method and compare against the baselines, we use the following metrics:
1. HyperVolume (HV): Let F be a reward space Pareto front approximation and ro be reference vector. Then, we define the HyperVolume as HV := [H(P)(2)dz where H(P) = {= E RM Fr E F ro <p <p r and 1-) is the indicator function.
2. Pareto Dominance (PD): Let r!, r? be the reward vectors trajectories sampled from agent and 2, respectively. We define the Pareto Dominance metric as PD1.2 := + Ei= 1(r] > rz), where 1(-) is the indicator function. This is essentially the fraction of time when the reward vector under agent Pareto dominates the reward vector under agent 2. Note that PD1,2 + PD2,1 S 1.
3. Utility (UT): For reward vector r; sampled from any agent, we define the Utility metric as UT := EwIt E rr), where the expected value is evaluated by sampling uniformly from the preference space Ω or from the preference distribution Dw.
Figure shows the results of the ablation study conducted on the MOGW and DST domains. We compare against the non-adaptive and the fixed-preferences agents. For the later case, we pick three different preference vectors: (wi, 42) = (0.5. 0.5), (w), w2) (0.75, 0.25) and (wI,w2) = (0.25, 0.75). These plots facilitate the visual identification of Pareto dominant agents. Focusing on the MOGW (upper row), we observe that our PPA agent converges to a policy that yields a higher reward and lower time-penalty than any other agent, i.e, PPA Pareto dominates all other agents. While a similar statement cannot be made for the DST domain, we note that PPA is not Pareto dominated by any other agent and the only agent that it does not dominate is the fixed- preference one for (w)) = (0.5, 0.5). On the other hand, the non-adaptive PPA (na-PPA) seems to perform relatively poor. However, there exists some merit in using it because, in contrast to the
Main Results:
7

---

Published as a conference paper at ICLR 2022
fixed- prefeereeeeence agents, na-PPA does not rely on being given preference vector. It also worth noting that in the DST domain the preference vector (W1, (2) = (0.25, 0.75) gives very poor results, which implies that handpicking the preferences might lead performance collapse. Overall, these results suggest that PPA steadily achieves better performance and is superior than the other agents. It is interesting observe that the reward and time penalty curves quite similar in the MOWG domain. This agrees with our intuition because in the MOGW domain the reward and time penalty have "inverse" relationship (i.e., the sooner reach the bottom right, the higher the reward).
Humandio-v
Hopper-u3
HollCheetah-4/3
Ant-433
0.8
0.7-
0.7-
0.5
2.4
0.3
0.3
8.2
611
0.4
3.6
0.5
1.2
0.0
0.1
0.0
02
5.0
1.0
5.0
0.4
0.4
1.0 3e6
Famrss
1sc
famis
fiamss
1ed
framic
Humanaid v3
Vo
Harlhhettah v3
Ant
0.4
0.4
6..0
-0.3
-6.3
0.2
Let
105
106
Figure 4: Learning curves on MuloCo environments. We show the Pareto Dominance (PD) and Average Utility (UT) metrics. Experiments are run for 10 random seeds. We compare against the non-adaptive PPA and the fixed ficed-pefferences agents.
PFA
CN-OLS
CD-DER
EQL
PGMORL
nA-PPA
PPA
EV
2.52
2.36
2.65
3.10
3.15
2.35
3.32
FD
0.52
0.53
UT
0.23
0.19
-0.12
0.18
0.28
0.08
0.28
EV 36.32
35.36
35.65
35.12
39.35
34.20
40.12
DST
FD 0.46
0.55
0.39
0.48
0.52
0.38
0.75
UT 0.19
0.20
BV
1146.14
1364.14
1134.12
1401. 14
1125.
1442. 14
MOSM
pn
0.47
0.41
0.50
0.59
0.42
0.64
UT
0.15
-0.07
0.24
0.26
0.14
0.27
FV
4.39
5.18
4.95
4.65
4.02
5.15
Humanold-v3
FD
0.41
0.46
0.54
0.52
0.33
UT
0.19
0.22
0.41
0.30
0.28
0.48
3.18
2.98
3.44
3.74
2.23
3.73
Hopper-v3
FD
0.43
0.49
0.45
0.52
0.45
0,70
U1
EV
5.177
5.38
5.39
5.77
4.18
5.68
HalfChectah-v3
FD
0.37
0.38
0.51
0.55
0.45
0.62
UT
-0.10
0.11
0.22
0.20
0.18
0.31
EV
5.95
5.47
6.30
6.35
5.25
6.45
Ant-v3
ED
0.26
0.30
0.44
0.41
0.32
U1
0.36
0.31
0.34
0.55
0.39
0.55
Comparison of MORL agents using the HyperVolume (HV), Pareto Dominance (PD), and Average Utility (UT) metrics. Possible values for each metric lie the range [0, oo), [0, 1]. and [-1, 1], respectively. In all cases, higher values indicate better performance. We evaluate the agents using 10 random seeds.
Table 1:
In the next line of experiments, we evaluate the performance of PPA continuous control tasks. We modified the MuloCo Gym interface output 2-dimensional rewards each step. The first
8

---

Published as a conference paper at ICLR 2022
one is the commonly used reward in such environments, i.e., the difference between two sequential positions of the robot. The second one is the energy consumed by the robot. We show the results on Fig. 4. We compare against the ablated agents and show the Pareto Dominance (PD) and Average Utility (UT) metrics. The only case where our method does not attain the best performance is the Hopper-v3 environment under the UT metric, where the performance of the exed-preferences agent (w), (0.75, 0.25) slightly surpasses the performance of PPA. However, observe that the same agent attains very poor performance in the HalfCheetah-v3 enviroment. This suggests that, given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector far from optimal and preference vector that yields good performance in one environment may perform poorly in a different environment judged under a different metric.
In the last line of experiments, we com- pare our method against state-of-the-art MORL algorithms (Table 1). Since PD is pair-wise metric, we first compute the pair-wise PDj.j values for all agents and then average-out the j component to ob- tain a single value for each agent. Note that this metric is specific to the ensemble of agents that we consider and needs to be re-calculated if we compare a different set of agents. The UT metric is calculated by sampling preferences from the preference space of each domain. We assume assume a uniform preference distribution and sam- ple 1000 points for the 2-dimensional pref- erence case (MOGW, DST, MUMO) and 100 across each dimension for the MOSM case, where the preference space has di- mensions. Additionally,
tpeode S0Aa1)
5
DeepSia Treasum Epiade Reword
2000 Sea Troparr tosode Time Feraty
200000 JI00CE 40000
50000
30000
20000
3000
40300
5KKC
Figure 3: Results on the MOGW and DST environments. wards in the range [- across each dimension. This scaling is important when calculating the average utility but has particular influence the Pareto dominance or hypervolume. Observe that some agents have negative utility, which is indeed possible given the aforementioned reward scaling. Our method systematically outperforms all other baselines across all domains, which is another strong empirical finding for its performance.
scale the re-
6 CONCLUSION
We introduced policy gradient algorithm for Multi-Objective Reinforcement Learning (MORL) under linear preferences. By approximating the Pareto front via a first-order necessary condition, our method learns a single policy that can be adapted to any preference distribution. The two pillars of our method is projected gradient descent solver that searches for common ascent direction for all objectives and novel Pareto Policy Adaptation (PPA) loss function that leverages the solutions of that solver to adapt the policy to any preference distribution. Our method enjoys the advantages of both single and multiple policy MORL algorithms while being simple to implement and compatible with all policy gradient and actor-critic methods. We demonstrated the effectiveness of our method in several reinforcement learning tasks.
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the support by the National Science Foundation under the Career Award CPS/CNS- 1453860, the NSF award under Grant Numbers CCF- 1837131, MCB- JCBB-193675, CNS- CNN-1993620, and CMMI I 1936624 and the DARPA Young Faculty Award and DARPA Director's 's Fellowship Award, under Grant and Northrop Grumman grant. The views, opinions, and/or findings contained
Number 166001-17-1-40444 this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied by the Defense Advanced Research Projects Agency, the Department of Defense or the National Science Foundation. This work was also supported by the National Science Foundation under the CAREER Award SHF- 2048044, FMitF award 1CF-1837 CPS award CNS-1932620. Toyota RD, and Northro- Grumman Acrospace Systems.

---

Published as a conference paper at ICLR 2022
REFERENCES
[ Mingxi Cheng, Chenzhong Yin, Junyao Zhang, Shahin Nazarian, Jyotirmoy Deshmukh, and Paul Bogdan. A general trust framework for multi-agent systems. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 332-340, 2021.
[2] Panagiotis Kyriakis, Jyotirmoy V Deshmukh, and Paul Bogdan, Specification mining and robust design under uncertainty: A stochastic temporal logic approach. ACM Transactions on Emhedded Computing Systems (TECS), 18(5s):1-21, 2019.
[3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. CoRR, abs/I 707 .63477,37 2017.
[4] Xiaoxiao Guo, Satinder Singh, Honglak Lec, Richard L. Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offiine monte-carlo tree search planning. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014.
[5] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489 Jan 2016.
[6] Gabriel Dulac-Arold, Nir Levine. Daniel J. Mankowitz. Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, Apr 2021.
[7] I. Y. Kim and O. L. de Week. Adaptive weighted method for multiobjective optimization: new method for pareto front generation. Structural and Multidisciplinar Optimization, (1 116, Feb 2006.
[8] Timothy Ward Athan and Panos Y. Papalambros. A note on weighted criteria methods for compromise solutions in multi-objective optimization. Engineering Optimization, 27(2):155-176, 1996.
[9] P. L. Yu and Leitmann. Compromise solutions, domination structures, and salukvadze's solution. Journal of Optimization Theory and Applications, 13(3):362-333 338,, Mar 1974.
[10] Ralph E. Steuer and Eng-Ung Choo. An interactive weighted tchebychefl procedure for multiple objective programming. Mathematical Programming, 26(3):326-34, Oct 1983.
[11] F. Waltz, An engineering approach: Hicrarchical optimization criteria. IEFE Transactions on Automatic Control, 12(2):179-180) 1967.
[12] Das and J. E. Dennis. A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural optimization, 14(1):63-6)) Aug 1997.
[13] A. Messac and A. Yahail- Multiohjective robust design using physical programming. Structural and Multidisciplinary Optimization, 23(5):357-371, Jun 2002.
[14] Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathematical Methods of Operations Research, 51(3):479-494, Aug 2000.
[15] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313 318, 2012.
[16] S. Schäffler, R. Schultz, and K. Weinzierl. Stochastic method for the solution of unconstrained vector optimization problems. Journal of Optimization Theory and Applications, 114(1):20-222,, Jul 2002.
[17] Kristof Van Moffacrt, Madalina M. Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEFE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 191-199, 2013.
[18] Sriraam Natarajan and Prasad Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In Proceedings of the 22nd International Conference Machine Learning, ICML '05, page 601-608 New York, NY, USA, 2005. Association for Computing Machinery.
[19] Andrea Castelletti, Gúorgio Corani, Andrea-Fmilio Rizzoli, Rodolfo Soncini-Sessa, and E. Weber. Rein- forcement learning in the operational management of a water system, pages 325- 01 2002.
10

---

Published as a conference paper at ICLR 2022
[20] Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey Kephart, David Levine, Freeman Rawson and Charles Lefurgy. Managing power consumption and performance of computing systems using reinforeement learning. In J. Platt, D. Koller, Y. Singer, and S. Rowcis, editors, Advances in Neural Information Pro- cessing Systems, volume 20. Curran Associates, Inc., 2008.
[21] Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-criitria reinforvement learning. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML '98. page 197-205, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.
[22] Shie Mannor and Nahum Shimkin, The steering approach for multi-criteria reinforcement learning. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems, volume 14. MIT Press, 2002.
[23] Shie Mannor and Nahum Shimkin. A geometric approach to multi-criterion reinforcement learning. J. Mach. Learn. Res., 5:325-360, December 2004.
[24] Hossam Mossalam, Yannis M. Assael, Diederik Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. CoRR, abs/ abs/1610.02707, 2016.
[25] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Stcckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Muchine Leurning Research, pages 11-20. PMLR, 09-15 Jun 2019.
[26] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[27] Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zam- belli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objectivv policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Re- search, pages 1 11-22. PMLR, 13-18 Jul 2020.
[28] Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, page 41 47, New York, NY, USA, 2008. Association for Computing Machinery.
[29] Daniel J. Lizotte, Michael Bowling, and Susan A. Murphy. Linear fitted-q iteration with multiple reward functions. Journal of Machine Learning Research, 13(105):323333255, 2012.
[30] Daniel Lizotte, Michael Bowling, and Susan A. Murphy. Efficient reinforcement learning with multi- ple reward functions for randomized controlled trial analysis. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, page 695-702, Madison, WI, USA, 2010. Omnipress.
[31] Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Tree-based fitted q-iteration for multi- objective markov decision problems. In The 2012 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2012.
[32] Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Multi-ohjective fitted q-iteration: Pareto frontier approximation in one single run. In 2011 International Conference on Networking, Sensing and Control, pages 260-265, 2011.
[33] Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli. Policy gradient approaches for multi-objective sequential decision making. In 2014 International Joint Conference on Neural Networks (IJCNN), pages 2323-2330, 2014.
[34] Simone Parisi, Matte Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through continuous pareto manifold approximation. J. Artif. Int. Res., 57(1):187722 September 2016.
[35] Simone Parisi, Mattco Pirotta, and Jan Peters. Manifold-based multi-ohjective policy search with sample reuse. Neurocomputing, 263:3-14, 2017.
[36] Jie Xu, Yunsheng Tian, Pingchuan Ma. Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction- guided multi- oujti-objecttve reinforcement learning for continuous robot control. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10607-10616. PMI.R, 13-18 Jul 2020.
11

---

Published as a conference paper at ICLR 2022
[37] Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-makin J. Artif. Int. Res., 48(1):67-113,, October 2013.
[38] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[39] Hongwei Jiao and Yonggiang Chen. A global optimization algorithm for generalized quadratic program- ming. Journal of Applied Mathematics, 2013:215312, Oct 2013.
[40] Satoru Iwata. Submodular function minimization. Mathematical Programming, 112(1):45. Jan 2007.
[41] Satoru Fujishige, Hiroko Sato, and Ping Zhan. An algorithm for finding the minimum-norm point in the intersection convex polyhedron and a hyperplane. Japan Journal of Industrial and Applied Mathematics, 11(2)225, Jun 1994.
[42] P. Wolfe. Finding the nearest point in polytope. Mathematical Programming, 11:128-149, 1976.
[43] Satoru Fujishige and Ping Zhan. A dual algorithm for finding the minimum-norm point in polytope. Journal of the Operations Research Society of Japan, 33, 01 1990.
[44] Kazuyuki Sekitani and Yoshitsugu Yamamoto. A recursive algorithm for finding the minimum norm point in polytope and pair closest points in two polytopes. Mathematical Programming, 61())33332249 Aug 1993.
[45] Jesús Loera, Jamie Haddock, and Luis Rademacher. The minimum euclidean-norn point in a convex polytope: Wolfe's combinatorial algorithm is exponential. In Proceedings of the 50th Annual ACM SIGACT Symposiun on Theory of Computing, STOC 2018, page 545-553 553, New York, NY, USA, 2018. Association for Computing Machinery.
[46] Ronald J. Williams. Simple statistical moddient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-255 -56. May 1992.
[47] Martin Jaggi. Revisiting Frank- Wolfe: Projection-fie freee sparse convex optimization. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference Machine Learning. volume of Proceedings of Machine Learning Research, pages 427 4335, Atlanta, Georgia, USA, 17-19 Jun 2013. PMI.R
[48] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Alché-Buc, Emily B. Fox, and Roman Gareett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver BC, Canada, pages 688-699, 2019.
[49] Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as layer in neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 136-145. PMLR, 06-11 Aug 2017.
[50] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. arXiv preprint arXiv:1707.06344) 2017.
[51] Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84(1):51-80, Jul 2011.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A THEORETICAL FRAMEWORK
the first part of the Appendix, give the proofs omitted in Sections 2 and 4 well the analytical solution Problem Pa the special case of 2-dimensional reward signals. For ease of reference, we re-state underlying theorem.
A.1 PROOFS
Lemma 1. Ler x E Ci.o be policy. Then, we have E=(p) CR. i.e. the expected return vector under policy x belongy in the RS-CCS.
Proof. Consider a policy x E Ci.a and policy n' E II such that d' (s, = wherever d' (s,a) 0. Then, we can write
d' (s, (s,
Ed' (s)a)r(s.
(s, Ed d= r(s. Es- (r((s,)]
(8)
Aia
where we used importance sampling identity and defined (s, == d' a)/" (a, (),a)) a). Per Def. there exists w such that w v" 2 wn " Leveraging fact that the value function can be expressed as v" = E. [F] as well as Eq. equation 8, we have w" E. 2 w" E, [F']. This implies that E. [f] E CR. and concludes the proof.
Remark 1. Under the additional assumption of normalized rewards, we can prove the converse of Lemma 1. However, since we are are concerned with approximating CR by Ci,s (and not the other way around), we refrain from making that additional assumption to prove the converse of Lemma 1.
Theorem 3. Let == I(w" (0)) be a generic loss function that depends explicitly on the the optimal solution w (0) of Problem Pa. Then, the gradient of with respect to the policy parameters is
G(O)
-TM (Ow
Vol = H (0) O 0)
(0)
1
deo O
where [G(0) 29 gr. (0). [H]
m. = - En wi ( T (0) (0)), and O are the identity and
zero vectors of appropriate dimensions, Is is the identity matrix of dimension M and D(.) create diagonal matrix from vector.
Prouf. We start by writing the Lagrangian Problem PA
M
Il,,A,1))
Σ Wm Grr (0)
+
Σ
Wm 1)
Σ m0=1
(10)
where gen (0) V Egro [mm( a)). Let (w (0),
(0) (0) 14" (0)) policy-dependan optimal primal and
dual variables. The KKT conditions for stationarity, primal feasibility and complementary slackness are
ØL
= 2 >
(1)g (0)gm (0) + (0) + pin = 0, m 1.2... M
(11a)
Ownn (0)
0L
X" (0)
(0) 1)
(11b)
0x (0)
aL
Opei(0 (0) pmm (0) (0) 0, m 1. M,
(11c)
By taking the gradient of the above equations with respect to the policy parameters, obtain
M
M
Vow, (0) (0)gmm)))))) +2 Σ
n=1
VoA"(0) +
Veli = m 1,2...
(12a)
M
Vonn (0)=
(12h)
Vopin (f)w (1))""(("" (0) (0) m 1,2., M.
(12c)
13

---

Published as a conference paper at ICLR 2022
Using matrix notation, we rewrite the above system of equations follows:
G( -Di (0))
Tur - DD(w (0))
V w (0)
(0)
DOD
1 (0)
(13)
0
V X (0)
By inverting Eq. equation 13 and using the chain rule Vol(w(()) Vol obtain the claimed result.
We observe that the solution of Eq. equation 13 requires knowledge of the optimal dual variable μ 40)), which is not directly available when we use the projected gradient descent solver of Algorithm I; that solver only gives the optimal primal variable w" (0). However,
can substitute that optimal w (0) into the KKT conditions given by Eq. (1la-1 11c) and solve the resulting linear system of M + variables find the optimal dual variables. The number of variables in that system depends only on the dimension of the reward signal M and not on the dimension of f, therefore it adds minimal computational overhead. This step would not be necessary if we chose primal-dual method for solving Prohlem PA- However, primal-dual methods may add to the complexity of the implementation, which may be worthy given the simplicity of Problem PA- Exploring such a trade-off is an interesting direction for future work.
A.2 PARETO STATIONARITY: 2- DIMENSIONAL CASE
In general, the problem of finding common ascent directions (Problem Pa) can be solved either exactly using the methods from the minimum norm points literature (as discussed in Sec. 3) or approximately using convex optimization methods. Due to the dimensionality of the gradient vector, the former class of methods are not applicable in our case. Therefore, This solver is essentially an internal optimizer which we run hefore every parameter update. Luckily, Problem Pa is convex and the gradient descent optimizer converges rapidly in practice with little tuning. Additionally, for the special case of two objectives. can leverage the analytical solution to achieve further performance improvements. We re-state Problem PA for convenience bellow:
resorted the projected gradient descent solver presented in Algorithm
M
M
min
Σ 1==1
Em Va (s.
1, cm 2 0, Vm
(PA)
m=1
For M 2, the problem reduces min | V Egre Ini (s, a)] + (1 V) Eare [r> (s, a)) which is one-dimensional quadratic optimization problem of single variable and has the following analytical solution:
Voo Eava (r2(s, a)] Vo Ers (s, a))) Vo Vo Ere [ra(s, a)]
(14)
V Eare (rz a)] V Bare [ri (s. a)ll
where [+]+ represents the clipping operator, i.e., [x]+ = max (min (xc, 1). 0). Geometrically, Eq. equation 14 implics that minimal-norm point lics either on the boundary on the convex hull (e O = 1) and the desired descent direction is parallel to corresponding gradient is equal to the vector perpendicular of the gradients. The former one a corner case and occurs when the ascent direction for one of the objectives a non-decreasing direction for the other one. While this is only applicable to 2, it enables us to replace the inner projected gradient descent optimizer in Algorithm with the analytical solution in Eq. equation 14. Additionally, this special case is applicable to several real-world cases since there several problems that have the reward-cost structure.
the difference
B EXPERIMENTAL DETAILS
B.1 ENVIRONMENTS
Our implementation uses modified versions of 4 OpenAl Gym enviroments (Fig. 5). The first two synthetic domains and act as prooe-of-concept for our method, while the other two are complex, real domains. The details are listed below:
Multi- Objeei-beeetivee GridWorld (MOGW): A multi-objective variant the classic gridworld environment. The agent starts from the top-left corner, moves in 16 x 16-grid and receives a reward equal to 1 when it reaches the bottom-right corner. Each action results in a small time penalty of -0.1. Any policy that results in trajectory that minimizes the taxicab distance between the agent's starting position and the goal is Pareto optimal. Our implementation uses gym-minngriin enviroment, which we modify to output the aforementioned 2- dimensional reward.
hhtpps:/ thub. com wtt.m/awwimmmmmmmmmmmmmmmmm htweee/c////m//mmmmeeeeeee nigrid
14

---

Published as a conference paper at ICLR 2022
Figure 5: OpenAl Gym enviroments used in our experimental results.
Deep Sea Treasure (DST): A classic oulti-objective reinforvement learning environment. The agent is located in 10 11-grid and controls submarine searching for treasures. There multiple treasures the value of which increases as their distance increases. The agent receives time penalty of - for each action. The Pareto front of non-dominated policies can be casily extracted (51) and is also shown on Fig. 1. As the MOGW case, we use the gym-minigr package and extend it to the reward map shown in Fig. 5.
Multi-Ojjective SuperMario (MOSM): We use modified, multi-objective variant of the popular video game. There 5 scalar rewards: x-pos: the difference in Mario's horizontal position between current and time point, time: a small time penalty of -0.1, deaths: a large negative penalty - -100 each time Mario dies, coin: rewards for collecting coins, and enemy: rewards for eliminating an enemy. We use the gym- super-mario-bros package and the multi- obbjectvee extension developed in (26).
Multi-Objectiv MuJoCo (MOMU): We focus locomotion tasks and modify the environments provided by the MuloCo physics simulator to output vector rewards. To do so, create custom wrapper for the built-in OpenAl Gym interface. There are sealar rewards: x-pox: the difference the robot's current horizontal position, r-pos: a positive reward for moving in circular trajectories.
B.2 IMPLEMENTATION DETAILS
We implement our algorithm in PyTorch building on top of the torch-ac ac package". Our implementation extends the Proximal Policy Optimization (PPO) algorithm to handle our PPA loss function (Eq. equation 6). We use the clipped version of PPO with the clip ratio threshold set to 0.2. We set the GAE parameter 0.95 and the discount factor to y 0.99.
use consistent network architecture across all our simulation for both the actor and the critic: Two hidden layers of 64 and 64 neurons each and a ReLU non-linearity. In the discreet action case (MOGW, DST, MOSM), we append the actor network with an output layer of dimension equal to the number of actions. In the MuJoCo enviroments, where the actions parametrize a multi-dimensions actions. Each non-terminal layer has ReLU non-linearity. Even though the hyperbolic tangent known to he preferred for reinforcement learning applications, we experimentally observed that it leads to vanishing gradients and slows down learning. This may be due to the fact that the PPA loss function requires second- differentiation. In the SuperMario enviroment, we worked the pixel domain and two convolutional layer of 32 and layers as feature extractors, which were shared between actor and the critic.
Network Architecture:
continous, we that output layer mean and variance, assuming uncorrelated
Gaussian distribution,
Hyperparameters and Training: To train neural network, we use the Adam optimizer (81 = 0.9, (3 = 0. 0.999) with a learning rate of 0,0001. In the MOGW, DST and MOMU domains, we optimize the parameters of the actor and critic separately. In the MOSM domain, given that the actor and critic share the convolutional layers,
their losses (using weights I and 0.5, respectively) and our simulations in Google Cloud Platorm using 48 vCores and one NVIDIA Tesla GPU. For MOGW and DST environments, we train the agent for 0.5M frames and for the MOSM and MOMU for 1M frames. For cach 512 frames we perform one update of the network parameters iterating over 10 epochs of the collected data and using mini-bater size of 64.
common optimizer. We all of
htw.:/atteennaaoooouuueeeaaaaaaa 'https://ee wtwp.ccoo/1cs.iiiiccccc. ems terch-ac
15


Reviews:
[{"review_id": "jSFcbVVKaRo", "reviewer": "Reviewer_1Zn3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is written well and easy to follow. I didn't find major drawbacks in the paper, and overall the theoretical results seem correct (yet not very novel or fundamental). Overall the work seems solid to me. A few items I would like the authors to address:\n1. In the proof of Lemma 1, $d^\\pi(s,a)$ can be 0. This would make the lemma incorrect I believe.\n2. In Thm 1, $E_{\\pi_\\theta}$ should be $E_{d^{\\pi_\\theta}}$\n3. In Thm 3, $g_m$ and not defined. I'm assuming these are $g_m(\\theta) = \\nabla_\\theta E_{d^{\\pi_\\theta}} r_m(s,a)$.\n4. The authors wrote: \"We start by interpreting the optimal solution of Problem PA as a function of the policy parameters $\\omega^* = \\omega^*(\\theta)$\". Can you elaborate on this? Why should the policy and preference parameters have the same parameters?", "summary_of_the_paper": "This paper introduces a policy gradient approach to multi objective RL. The authors learn show that an optimal solution for a specific reward vector can be found by minimizing a constrained quadratic optimization problem. They propose to solve this problem using projected gradient descent. Then, the authors offer an efficient solution for optimizing the Pareto front through \"Pareto Policy Adaptation\" loss function, which trades off learning the Pareto front and the policy. The authors conduct experiments and ablations studies on four benchmarks, validating their method with improvement over current sota approaches for MORL.", "summary_of_the_review": "See review", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SEc8E98-HAs", "reviewer": "Reviewer_T2hX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the idea is interesting, my main concern is about the novelty and the appropriate reference of previous work. As far as I know, the first-order necessary stationary condition was already used together with REINFORCE in previous work (see [31]). You mentioned [31] in the intro but I think a more detailed comparison is due. In [31], the authors already discussed the existence of a common ascent direction and provided the associated quadratic programming approach for the computation. They also presented gradient-based approaches for the computation of first-order stationary policies. I think a deeper discussion of the differences between your and their work is necessary. Given this, I think it is also necessary to add their algorithms to the empirical comparison. \n\nThe domains considered in the experiments are standard but I would have appreciated a more extended comparison. For example, why do you focus on domains with very few objectives (3 domains with 2 objectives and one with 5)?\n\nFinally, there are a few typos in the paper that makes the reading complicated, especially in section 3.\n- The definition of the value function $v : |S| \\to \\mathbb{R}^|S|$ is not precise. You should use the notation $v \\in \\mathbb{R}^{|S|}$ or $v : S \\to \\mathbb{R}$. Same for the multi-objective case.\n- Why are you considering finite state space and continuous actions?\n- Eq. 3 is not clear. In particular, the first equality is not correct. There is an expectation w.r.t. the initial distribution missing and probably an additional $w^T$. Please clarify it.\n- In the proof of lemma 1, you refer to definition 2. It should be Def. 1\n- Based on what you mentioned at the end of section 2.2, there is a constraint missing in definition 2 ($w$ should belong to the simplex).", "summary_of_the_paper": "The paper presents a multi-objective method for linear preferences. The key concept used for the design of the approach is that there always exists a common ascent direction for all the objectives. The authors leverage this property for introducing a novel loss function that can be integrated into different DeepRL methods. The proposed approach is simple and proved to be effective in the proposed experiments.", "summary_of_the_review": "My score is justified by the doubts I have about the novelty, comparison with previous work, and clarity of the writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7CYQK92MM0-", "reviewer": "Reviewer_hSRE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the method is well presented. The introduction covers well prior work. However, I think that the authors do not properly compare their method to the existing ones discussed in the introduction, and I find it hard to distinguish between what is really novel and what has already been done. Furthermore, there are some other issues that should be clarified.\n\n1) The idea of convex coverage set (CCS) is well known in MORL since years. Roijers et al. have published several papers on the subject, e.g., \"A Survey of Multi-Objective Sequential Decision-Making\" (2013) and \"Computing Convex Coverage Sets for Faster Multi-objective Coordination\" (2015). Therefore everything up to line 150 is already known (correct me if I am wrong). The novel part should be the gradient descent leveraging on the fist-order condition, resulting in a method that can be applied to a large variety of RL algorithms. I think it would be better to put the CCS part in \"Preliminaries\", and keep the novel part in Section 3 to highlight it.\n\n2) The first-order condition is a necessary but not sufficient condition for Pareto optimality. You discuss this but only at the very end, as well as the other limitation (PPA can find only convex frontiers). It would be better to write a short paragraph at the end of Section 4 to highlight this. Similarly, you should mention that the Deep Sea Treasure version used for the experiments is not the original version [1]. The original has a convex frontier, which PPA cannot solve. \n\n[1] Vamplew et al., \"Empirical evaluation methods for multiobjective reinforcement learning algorithms\" (2011)\n\n3) Third, the adaptation part is highly related to Envelope Q-Learning, which you use as baseline in the experiments. You should discuss similarities and differences more in-depth, given that both PPA and EQL retrieve unknown preferences given a solution, and their losses are very similar. For instance, the first term of your Eq. 8 is the same of EQL Eq. 6, while the second term differs from EQL Eq. 7 (I am referring to EQL equations in its paper).\n\n4) It is a bit misleading to claim in the abstract that the final PPA method learns under “unknown preferences” since the objective in (8) does require $\\omega$ to evaluate the loss.\n\n5) In Figure 2(b), why PPA is guaranteed to converge to in this example?\n\n6) For the MuJoCo experiments, the paper only considers the more basic baselines (nA-PPA and fixed-preference agents). It would be helpful to compare PPA with other stronger baselines, such as PGMORL (Xu et al., ICML 2020).\n\n7) You should discuss PPA limitation more in details. E.g., PPA directly maximizes the expected return for a given preference or a preference distribution, which means when the environmental preference changes, one has to finetune the model with some extra cost. However, other value-based baselines align the optimal policies with preferences during a single training procedure and can respond with the corresponding optimal policy to any given preference without fine-tuning.\n\nFinally, some questions regarding the experiments.\n\n* Why did you not use the hypervolume as evaluation criterion? It is probably the most common and accurate criterion in MORL literature and could easily replace UT. UT is computed over some samples preferences (how many?), and you could use the same preferences to compute the hypervolume of the corresponding solutions. For MOSM you could use a Monte-Carlo approximation of the hypervolume.\n* Why did you not compute PD and UT against baselines for MuJoCo experiments as well?\n* Why did you choose to reward the agent for moving in circular paths as second objective for MuJoCo? Usually, the first reward is the distance from a goal, while the second is the energy consumption (see Abdomaleki et al., \"A Distributional View on Multi-Objective Policy Optimization\" (2020)).\n* I am a bit confused by the results. In Table 1, PPA does well under PD and UT criterion. In the plots, results are mixed. PPA does well in MOGW, but is not the best in DST. Then in MuJoCo experiments plots are very noisy, and performance curves sometimes increase and decrease multiple times. For instance, why does w=0.5 increase and then decrease in Hopper? The same curve then performs terribly in Cheetah, why? And why w=0.25 show opposite behavior (decent on Cheetah, terrible in Hopper and Ant)? I am not convinced by your intuition that \"given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector is far from optimal\". The rewards are the same for all domains, why would Cheetah be different?\n", "summary_of_the_paper": "The paper presents a new method for solving MORL problems. The main contributions are: a generic gradient descent MORL algorithm for finding multiple solutions, and another algorithm to retrieves preferences from the given solutions. The authors also present theoretical proofs of their method.", "summary_of_the_review": "The idea of the paper is interesting, but there are many things that the authors should clarify, both in regards to limitations of the methods and evaluation (metrics, domains, comparison against other baselines).\nFor these reasons, I vote for its rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While this paper has divergent reviews, reviewer hSRE has by far the most detailed review, seems clearly the most informed on the subject, and is the least supportive.  The main issues with the paper seems to be the degree of novelty and reviewer hSRE's feeling that the results on MojuCo are unclear and not explained in the paper.  But this alone does not seem like an adequate reason for rejection and hSRE seems happy with the other aspects of the paper.  Some of hSRE's original complaints do not concern me, such as the fact that first-order stationarity does not imply Pareto-optimality.  I am recommending a poster.
Decision: Accept (Poster)