Distributional Pareto-Optimal Multi-Objective Reinforcement Learning
Authors: Xin-Qiang Cai
Pushi Zhang
Li Zhao
Jiang Bian
Masashi Sugiyama
Ashley Juan Llorens
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-Objective Reinforcement Learning

Abstract: Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account for distributional preferences over the multi-variate returns, which are particularly important in real-world scenarios such as autonomous driving. To address this issue, we extend the concept of Pareto-optimality in MORL into distributional Pareto-optimality, which captures the optimality of return distributions, rather than the expectations. Our proposed method, called Distributional Pareto-Optimal Multi-Objective Reinforcement Learning~(DPMORL), is capable of learning distributional Pareto-optimal policies that balance multiple objectives while considering the return uncertainty. We evaluated our method on several benchmark problems and demonstrated its effectiveness in discovering distributional Pareto-optimal policies and satisfying diverse distributional preferences compared to existing MORL methods.

The paper has been revised address the identified inaccuracies the references.
Distributional Pareto-Optimal Multi-Objective Reinforcement Learning
Xin-Qiang Cail*, Pushi Zhang²", Li Zhao", Jiang Bian',
Masashi Sugiyama*. 3.1 Ashley J. Llorens"
1 The University of Tokyo, Tokyo, Japan
Microsoft Research Asia, Beijing, China
RIKEN AIP, Tokyo, Japan
Abstract
Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account for distributional preferences over the multi-variate returns, which are particularly important in real- world scenarios such as autonomous driving. To address this issue, extend the concept of Pareto-optimality in MORL into distributional Pareto-optimality, which captures the optimality of return distributions, rather than the expectations. Our proposed method, called Distributional Fareto-Optimal Multi-Objectiin Reinforce- ment Learning (DPMORL), is capable of learning distributional Pareto-optimal policies that balance multiple objectives while considering the return uncertainty. We evaluated our method on several benchmark problems and demonstrated its effectiveness in discovering distributional Pareto-optimal policies and satisfying diverse distributional preferences compared existing MORL methods.
1
Introduction
Apart from most Reinforcement Learning (RL) works that consider the scalar reward of task I 4, Multi-OOjective Reinforcement Learning (MORL) has recently received extensive attention due to adeptness at managing intricate decision-making issues with multiple conflicting objectives 6 7 8]. In many multi-objective tasks, the relative preferences of users over different objectives are typically indeterminate a priori. Consequently, MORL's primary aim is to learn a variety of optimal policies under different preferences to approximate the Pareto frontier of optimal solutions. has been demonstrated that MORL can significantly reduce the reliance on scalar reward design for objective combinations and dynamically adapt to the varying preferences of different users.
2
However, numerous real-world situations involve not only unknown relative preferences across multi- ple objectives, but also uncertain preferences in return distributions. These may include preference risk 99] 10, safety conditions 11. and non-linear user's utility 122 Consider, for instance, autonomous driving scenarios where agents need to strike a balance between safety and efficiency ob- jectives. Different users might possess varying levels of risk tolerance. Some may demand high safety performance, tolerating lower expected efficiency, while others may seek more balanced perfor- mance between safety and efficiency. Current MORL methods, by focusing exclusively on expected values with linear preferences, may not adequately capture multi-objective risk- sensieeniivve preferences, hence being unable to deliver diverse policies catering to users with varied risk preferences.
In this work, we broaden the concept of Pareto- optimality in MORL to encompass distributional Pareto-optimality, which prioritizes the optimality of return distributions over mere expectations, as
"Equal contribution. Work done during an internship Microsoft Research Asia.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

f2
fa
fi
fi
(a) MORL's Parcto-front
(b) Distributional MORL's Parcto- front
Figure 1:
The comparison of learning targets between the traditional MORL tasks and distributional MORL tasks, in which f1 and f2 are two (conflicting) objectives.
depicted in Figure From a theoretical perspective, we define Distributional Pareto-Optimal (DPO) policies, which capture the optimality of multivariate distributions through stochastic dominance [3 [4]. Distributional Prveto-Optimality delineates the set of policies with optimal return distributions, which we formally establish as an extension of Pareto-optimalle in MORL.
On the practical side, we propose a novel method named Distributional Pareto-Optimal Multi- Objective Reinforcement Learning (DPMORL which aims learn of DPO policies. We demonstrate that a policy achieving the highest expected utility for a given strictly increasing utility function DPO policy. To this end, we first learn diverse non-linear utility functions and then optimize policies under them [15]. Experimental outcomes on several benchmark problems attest DPMORL's effectiveness in optimizing policies that meet preferences on multivariate return distributions. We posit that our proposed method significantly addresses the challenge of managing multiple conflicting objectives with unknown preferences on multivariate return distributions in complex Decision-making situations.
Our main contributions are listed as follows:
1. We introduce the concept of Distributional Pareto-Optimal (DPO) policies. This expands the notion of Pareto-optimality in MORL to include preferences over the entire distribution of returns, not just their expected values. This enables agents to express more nuanced preferences over policies and align their decisions more closely with their actual objectives.
2. We propose DPMORI, functions. This algorithm accommodates a broad range of distributional preferences, thus offering more flexible and expressive approach MORL.
algorithm for learning DPO policies under non-linear utility
3. We execute extensive experiments various MORL tasks, demonstrating the effectiveness our approach in learning DPO policies. Our findings show that our algorithm consistently surpasses existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences, underscoring its practical benefits.
2 Related Work
Oulti-Objective Reinforcement Learning. MORL has emerged vibrant research area in the reinforcement learning community owing to its adeptness in managing intricate decision-making scenarios involving multiple conflicting objectives (5 [6] A plethora of MORL algorithms have been put forth in the literature. For instance, [66] proposed the utilization of Generalized Policy Improvement (GPI) to infer set of policies, employing Optimistic Linear Support (OLS) for dynamic reward weight exploration 17 This GPI- learning process was further enhanced by [7] through the incorporation of world model to augment sample efficiency. [8] proposed evolutionary approach to learn policies with diverse weights. However, our proposed approach diverges from these by leveraging utility functions to guide policy learning [18 19.. The urility-based paradigm [6] accentuates the user's utility in decision-making problems, capitalizing on known information about the user's utility function and permissible policy types. Such methods scalarize the multi- dimensional objectives into single reward function, enabling traditional RL algorithms to infer
"The code is available on httph:://// htapttttww..ccoepppcce/////P//00

---

desirable policies, while also respecting axiomatic principles where necessary. Several studies have utilized non-lineat utility functions guide policy learning 120 [2], catering to more intricate preferences compared to linear ones but still in view of the expectation of the Pareto front. In contrast, our work extends the Pareto-optimality to distributional Pareto-optimality by introducing a set of non-linear utility functions with distributional attributes to guide policy learning. Some of the recent works [2I] [22 [3] in MORL defines Distributional Undominated Set or ESR set which is both very close to Distributionally Pareto-Optimal policies in this paper. These works propose novel designs to enhance policy improvement in distributional reinforcement learning algorithms to find the Distributional Undominated Set. Compared to these works, we build novel theoretical results on the conditions of DPO policies and a novel algorithm (DPMORL) of learning DPO policies based on reward shaping that can be built on the top of any online RL algorithm.
Distributional Reinforcement Learning. Distributional RI. extends traditional RI. by modeling the entire distribution of returns, rather than just their expected values [24 25. This approach has been shown to improve both learning efficiency and policy quality in a variety of single-objective RL. tasks 226 27 Key algorithms in this area include Categorical DQN (C51) 24), Quantile Regression DQN (QRDDNN) [26). and Distributional MPO 27 Despite the success of these distributional RL algorithms in single-objective settings, their direct extension to MORL has been limited due to the added complexity of handling multiple conflicting objectives and expressing preferences over the distribution of returns. On the other hand, the insights and techniques developed in distributional RL can provide a valuable foundation for incorporating distributional preferences into MORL, as demonstrated by our proposed approach.
Risk-Sensitive and Safe Reinforcement Learning. Risk and safe RL are special cases of MORL that accentuate specific facets of reward distributions [o. Risk-sensitive RL primarily considers reward variance, striving to optimize policies that negotiate the trade-off between expected Conversely, safe RL prioritizes constraints on the agent's behavior,
return and risk 22
29 30
ensuring adherence to specified safety criteria throughout the learning process [31 32 [3) Some
studies have proposed the integration of constraints into the state space, constructing new Markov Decision Processes [T]]. Others have explored the distributional aspects of rewards, investigating the implications of distributional RL on risk- sensitive and -riente decision- making [26] 24).
However, our proposed setting is more general in terms of objectives, as it considers a broader range of user preferences and captures the entire distribution of rewards, rather than focusing solely on specific aspects such as risk, variance, or constraints. By extending MORL to incorporate distributional properties, our approach enables the learning of distributional Pareto-optimal policies that cater to diverse user preferences and offer better decision-making in a wide range of real-world applications.
3 Preliminaries: Multi-Ohjective Reinforcement Learning
35 in MORL, the agent interacts with an environment modeled as a Multi-Objective Markov Decision Process (MOMDP) with multi-dimensional reward functions. Multi-Objective MDP is defined by a tuple (S, A, P. Po, R, y, T), where S is the state space. A is the action space, P is the state transition function, Po(s) is the initial state distribution, R x A x - R a vectored reward function and K is the number of objectives, y is the discount factor, T is the total timesteps"] The goal of the agent is to learn set of Pareto-optimal policies, which represent the best possible trade-offs among the different objectives.
Similar to the procedure of RL 34
One popular methodology for MORL problems is the utility-based method, which combines the multi-dimmnsional reward functions into single scalar reward function using a weighted sum or another aggregation method [36]. The intuition is to map the agent's 's preferences over different (wi, W2, ... wk)
objectives to a scalar value for the agent training. Given a weight vector with wy representing the importance of the i-th objective, the scalarized reward function is defined as Tscalar(s, a) Σ w;ri(s, a). The agent then solves the scalarized MDP by optimizing its policy to maximize the expected scalar reward, using standard reinforcement learning algorithms like Q-learning or policy gradient methods. This approach can be straightforward to implement and has been shown to be effective in various MORL settings under expected preferences [37]. However, such a linear combination of each dimension of the reward cannot deal with the preferences with distributional considerations.
We assume that current time f is part of state to accommodate for the finite horizon setting.
3

---

4 Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning
In this section, we introduce our proposed theoretical framework and algorithms for extending MORL to handle distributional preferences.
4.1 Distributionally Pareto-Optimal Policies
In this work, we consider the reward K K-dimensional vector, where each element represents the reward for specific objective. Given the MOMDP and a policy л, we define the random variable of multi-objective return as ( = Ei= y're, where T2 is the multi--iimessionll reward at step t. and the states So, $1. 8T and actions du, a..... or sampled from the MOMDP and the policy respectively. The return distribution of policy æ, denoted as (T)),, represents the joint distribution of the returns Z(x) following policy и. The utility function f is non-decreasing function that maps a K-ddimensional return into a scalar utility value for the agent [38 (3, 44), capturing the user's distributional preferences over the different objectives. The expected utility of policy under the utility function f, represented as Ex>u((>) (z), is the expected value of applying f to the return distribution (()z).
Our goal is to learn a set of policies that are distributionally Pareto-optimal, which means that their return distributions of each policy cannot dominate that of another. To measure such a relationship, here we first introduce the concept of stochastic dominance:
Definition 1 (Stochastic Dominance for Multivariate Distribution). A multivariate distribution μr dominates another distribution μ2z, denoted as μn SD p2, if and only if μr * p2 and for any non non-decreasing utility function TRK + IR, pn has greater expected utility than paz i.e. Egena [z) 2 E
Definition 2 (Stochastic Dominance for Policies). policy Tı stochastically dominates another policy T2, denoted as an SDD T2, if and only if p(m)) SD p(Tz), indicating N1 has greater expected utility than n2 under any decreassgg utility function f.
300
122
(a) Stochastic Dominance Example (b) Stochastic Dominance Example 2 (c) Stochastic Dominance Example
Figure 2: Three 2D examples of stochastic dominance given synthetic returns of two policies. (a) T2 > SD Tı, since any non-dccreasing utility function satisfies (T()) 2 (f(Z((T)) (b) m and T2 cannot dominate each other. For example, there exists f(z) ReLu(20 5.0), such that (Tz2) > (f())) thus Ti cannot dominate n2. Similarly, T2 cannot dominate TT. Thus there no dominant relationship between the two policies. (c) ग and 72 cannot dominate each other, which is similar to (b).
The definition of stochastic dominance extends univariate first-order stochastic dominance [13 into multivariate cases. Figure provides some examples of stochastic dominance given 2D returns of two policies. The definition of stochastic dominance in policies allows for comparing the optimality of distributions between policies, which allows to define the Distributionally Pareto-optimal policy:
Definition 3 (Distributionally Pareto-Optimal Policy). Formally, कi is Distributionally Pareto- Optimal Policy if there does not exist policy T2 such that (u(T2) >D (TT))..
In other words, mi is considered Distributionally OpttmalOttimal (DPO) if not stochastically dominated by any other policy. DPO policies are essential in our framework as they represent the most desirable policies for users with different distributional preferences. To find such policy, we have the following theorem:

---

Theorem 1. If policy a has optimal expected utility under some non non-decreasing utility function f, where either utility function f is strictly increasing, or π is the only optimal policy with greatest expected uiility under function function f, then is a Distributionally Pareto-Optima policy, Also, any Pareto-Optimal policy is a Distributionally Pareto- Optimal policy.
The proof of Theorem is provided in Appendix A which utilizes form of optimal transport theory for stochastic dominance. In practice, this theorem guarantees that the optimal policy of a strictly increasing non-linear utility function is DPO policy, making it suitable candidate for deployment in MORL problems. The strict increasing property of utility function f in Theorem is crucial the theoretical analysis for proving the distributional aareto-optimalit.. In the next section, we base the result of this theorem to find optimal policies for a diverse set of utility functions, in order to learn the set of DPO policies. This result also shows that our definition of Distributional Pareto-Optimal policies is an extension of Pareto-optimal policies, allowing for a more diverse set of optimal policies to be captured.
We also formally prove that optimal policies for risk sisk-itnsitvve and safe constraint objectives belong to the set of Distributional Pareto-Optimal policies. This proves that DPO policies can successfully cover policies with diverse distributional preferences. The detailed proof is provided by Theorem in Appendix A
4.2 Distributionally Pareto-Optimal Multi-Ohjective Reinforcement Learning
We now present our proposed algorithm, Distributionally Pareto-Optimal Multi-Objective Reinforce- ment Learning (DPMORL). The main idea of DPMORL is to learn a set of non-linear utility functions that can guide the agent to discover distributional Pareto-optimal policies. The algorithm proceeds in two stages: (I) generating the utility functions and (2) training the policies.
4.2.1 Utility Function Generation with Diversity-basee Objective
The first component of our algorithm focuses on generating diverse set of plausible utility functions, upon which we find the optimal policies to find a diverse set of optimal policies. This is essential to ensure our method can accommodate various distributional preferences and adapt to different problem settings. To achieve this, we propose (1) Non-decreasing Neural Network for parameterizing a diverse set of non-linear utility functions (2) an objective function of minimum distance which encourages generating a diverse set of utility functions.
Non-decreasing Neural Network. We employ non-decra neural network to parameterize the utility function. The use of a neural network allows us to represent complex, non-linear, and arbitrary continuous utility functions, while the non-decreasing constraint ensures that the utility function satisfies the desired properties for multi-objective problems. We ensure the non-decreasing property in neural networks by constraining the weight matrix be non-negative and the activation function to be non-decreasing following existing work in convex neural networks [41] and QMIX which can approximate any multivariate non-decreasing function with arbitrary small errors The implementation details of the Non-decreasing Neural Network are provided in Appendix B
42
43
Diversity-basse Objective Function. We propose objective function based diversity for learning a diverse set of plausible utility functions. Specifically, we define fe,; fe.. fe., to be the set of candidate utility functions, parameterized by 81, .fas. For any given utility function fo., the learning objective is defined as
jualue min E E2. (f) (z) (2)] jti
So. (Ez) - (z)) || - z1ll =
(22) fo, (=1)
Jgrad (0x) = min E, -(11,,1)) j+i
(2)
|| 21
J(O:) Jvalue (0:) + aa--aa)Jee Jsradd((
(3)
Optimizing this objective function can encourage diversity among the generated utility functions within the normalized range [0, 1 in both value and derivative, leading to more comprehensive coverage of potential user preferences. We minimize the objective function by gradient descent on non-decreasing neural networks to generate set of N utility functions. To make the learned utility function strictly increasing for satisfying the constraints in Theorem [
we add an additional term to the learned utility function as fo. (x) = fo. (x) + # Σk= 25 in our experiment with a = 0.01, and use fa (z), i 1,..., .M as the strictly increasing utility function to learn DPO policies.
5

---

4.2.2 Optimizing Policy with Utility-based Reinforcement Learning
Once we have generated a diverse set of utility functions, the second component of our algorithm focuses optimizing policies to maximize the expected utility. This process, which we call utility- based RL, leverages the generated utility functions to guide the optimization of policies. By focusing on the expected utility, our method can efficiently balance the trade-offs between multiple objectives and distributions, ultimately generating policies that are more likely to align with user preferences.
We show that the following utility- based reinforcement learning algorithm can effectively optimize the policy with respect given utility function.
Algorithm 1 Utility-based based Reinforcement Learning
Input: policy म, an environment .M- = =S A, P. R, Output: new policy n'
7, T",, utility function f
1: Augment state space with $ x 2, where 2 is the space of cumulative multivariate returns.
2: Let transition Po(.) and P(-I(s;, ((-1(s),z)), ax) with su ~ P(so),
0, and S1+1 P(-Ist, az),
Zt+1 24 + T-T
3: Let scalar reward function Rt as R( (sr, =t); drs (8t41, =++1)) " [(x))
4: Optimize policy under environment M (S, A, P. Po. R, 7, 7, under sfflthe-shelf RL
algorithm (such as PPO or SAC) to x'.
Briefly, Algorithm [ augments the state space with the cumulative multi-objective returns, and transforms the multi-dimensional rewards into scalar reward by the difference in the utility function f. The following result shows that the new sealar-reward environment M generated by Algorithm has the same optimal policy as the optimal policy under utility function f:
Theorem 2. The optimal policy n² under environment M = (8, A, P, Po R. T,,, with scalar reward function
R((st, 2i), at (8t41, Z+41)) = ((=())))))) - f(z))
is the optimal policy under the utility function f. i.e.
Ezsn(rr) [f)] = maxEE [f())]
An advantage of Algorithm T is that can directly utilize off-the-shed RL algorithms, such as PPO 44 and SAC [45] without any modification, which makes the algorithm easy to implement using widespread existing implementations of online RL algorithms.
is also important to note that our algorithm simplifies to optimizing the weighted sum of rewards in MORL when the utility function is linear. This implies that method is generalization of the linear utility function MORL approaches by accommodating a wide range of non-linear utility functions. This flexibility makes our algorithm particularly suited for problems where the user's preferences may not be adequately captured by a linear utility function.
In the next experimental section, DPMORL. only undergoes a single iteration: we initially generate set of N utility functions as per the methodology detailed in Section 4.2.T Subsequently, optimizes set of N policies using these generated utility functions, as outlined in Section 4.2.2
5 Experiments
In this section, we conducted several experiments under the setting of MORL. Through the experi- ments, want to investigate the following questions:
1. Can Utility-based RL effectively learn policies with diverse distributional preferences?
2. Can DPMORL generate a set of diverse non-linear utility functions?
3. Can DPMORL obtain promising performance compared with state-of-the-art MORL meth- ods in view of expected and distributional preferences?

---

5.1 Case Study of Utility Functions
To answer the first question, we train policies with a diverse set of utility functions on DiverseGoal, DeesSeaTreasure and Reacher. Here, we focus showing the effectiveness of the srility-based We select several different non-linear functions f. each in favor of distributions at one target. We train policy with utility function Si with Algorithm| 1 for each target, and show the policy's trajectory and return samples.
RL algorithm (Algorithm D.
Diverse Goal Envronment
Roward Distribution on Each Goal
-4-
00
10
20
(a) The map of the DiverseGoal envi-
(b) Reward distribution within each goal position.
ronment.
Figure 3: Illustrations of the DiverseGoal environment.
DiverseGoal is a MORL, environment in Figure with multiple goals that the agent can take multiple steps to reach, where each goal has its unique reward function. Upon reaching particular goal, the agent secures a 2D reward. This reward sampled from the specific normal distribution associated with that goal, illustrated in Figure 3b Conversely, if the agent reaches the boundary of the map, it incurs a negative 2-dimensional reward. The environment requires the agent to navigate the trade-offs between various reward distributions, underscoring the complexity and nuance inherent to the task.
In this study, we investigate if the learned policy can reach the goal that yields the highest expected utility under the utility function. The results are illustrated in Figure 4 Under different types of utility functions and return distributions, the utility- based RI algorithm able to find the optimal policy with the highest expected utility, which shows the effectiveness of the utility- RL algorithm.
TreaSueTreasure Treasure and Reacher are among the most widely used MORL environments. Under the TreaSuaTreasure Treasure environment and Reacher environment, we show the return distributions of the policies optimized with utility-based RL under different utility functions. In Figure Ia Figure use utility functions generated by DPMORL in Section I I.2.Jamd and a set of manually constructed utility functions for different types of distributional preferences respectively. From the result Figure 5 and Figure 6 we can see that our algorithm effectively finds policies that maximize the expected utility for different non-line ooneirn utility functions. Meanwhile, the DeepSeaTreasure environment aligns well
BY REdEEt
IT SEREERL
6200
Utility functions in DiverseGoal environment used for training policies.
Saturn Citritatennd
Tebun Catra
Betam off
Dettmmtirtttdeee ot
Deturn Cotrbutotttttttooo
(b) Return distribution policy to maximize expected utility for each utility function.
Telectir d'dyy
Trtedtoy NEEE
Tajetto df
Tajectuo ofddey
(c) The urajectory of policies leamed by maximizing the expected utility for each utility function. Figure 4: The case study results of DPMORL under DiverseGoal environment.
7

---

function
-
21
15
20
15
(a) Utility functions generated by our method in Scction 4.2.1 under DeepSeaTreasure Treasure environment.
(b) Return distribution policy learned by DPMORL. to maximize expected utility each utility function.
Figure 5: The study results of DPMORL under DeepSeaTTreauurg environment.
Utilty funcion
Uditt functin 5
Usility function
Uoity function
Utility function 4
Utily function
Udily
25
114
12
(a) Utility functions with different distributional preferences under Reacher environment.
Noloy It
Policy ft.
Palky
Polcy
Nix
Nix
Palky
7
(b) Return distribution of policy learned by DPMORL maximize expected utility for each utility function.
Figure 6: The case study results of DPMORL under Reacher environment.
with safe RL settings, where the two returns time penalty and task reward respectively. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint.
5.2 Main Experiment
In this section, we illustrate the performance of DPMORL on five environments based on MO- Gymnasium 46 to answer the latter two questions.
Environments. We conducted experiments across five environments based on MO-Gymnasium 46 to evaluate the performance of our proposed method, DPMORL. These environments represent diverse range of tasks, from simple toy problems more complex continuous control tasks, and cover various aspects of multi-objective reinforcement learning:
DeepSeaTreasurr A classic MORL benchmark that requires exploration in gridworld to find treasures with different values and depths.
FruitTree: A multi-objective variant of the classic gridworld problem, where agent has to collect different types of fruits with varying rewards and penalties.
HalCheetah, Hopper, MountainCar: Three continuous control tasks that require controlling different agents for task solving and minimizing energy usage.
More details about the environments are gathered in Appendix C.I
Baselines. We compare DPMORL with four state-of-the-ar baselines in the context of distributional preferences of MORL: Optimistic Linear Support (OLS) 47 [6]; Prediction-Guidde Guided Multi-Objective Reinforcement Learning (PGMORL) 88) Generalized Policy Improvement with Linear Support (GPI-LS) 77: Generalized Policy Improvement with Prioritized Dyna (GPI-PD) 7
Training Details. For all methods, including DPMORL and the baselines, each policy was trained for 10 steps. We learn a set of N 20 policies for DPMORL. and all the baselines to ensure fair comparison. Finally, we use the learned N 20 policies in each method for evaluations.

---

Lo
Lo
Laf
Loe
1a/
Lo Mo
Loftt by 1ugttal
Staaf
5291
Lo
foud Stuul
Leaff
Sayal of uo La
212
Lo 1o
Lo "0
235
to
Loffitt fa ja alt fo fo Seaa
2221
an
10
14
20 85 1o 50 12
6300
Figure 7: Illustration of 2D utility functions learned by our methods in Section 4.2.1
Implementation Details. We use PPO algorithms implemented in Stable Baselines 3
48 in
Algorithm|] 1 We use a normalization technique by linearly mapping the return into scale 0. 1
without modifying the optimal policies. Detailed implementations provided in Appendix B
Evaluation Metrics. To thoroughly evaluate the performance of DPMORL and compare it with the baseline methods, we employed four distinct metrics. These comprise two existing MORL metries, HyperVolune and Expected Utility, addition to two novel metrics developed specifically for assessing distributional preference, i.e., Constraint Satisfaction and Variance Objective. The latter two are designed to underscore the optimality of multivariate distributions associated with the learned policies. In terms of Constraint Satisfaction, we randomly generate M 100 constraints for the policy set produced. The Constraint Satisfaction metric is then computed by considering the highest probability within the policy set that satisfies each individual constraint. The Variance Objective metric, on the other hand, involves generating M = 100 random linear weights. These weights applied to both the expected returns and the standard deviations of returns in each dimension. This objective encourages attaining greater expected returns while simultaneously reducing variance, thereby catering to dynamic preferences. Further details about the implementation of these evaluation metrics are provided in Appendix C.2 and Appendix C
5.3 Results
Generation of Utility Functions. In accordance with the methodology detailed in Section 4.2.
we employ non-decreasing neural networks conjunction with diversity- objectives to generate diverse assortment of utility functions. The generated functions are visually represented in Figure 7 The outcomes clearly indicate that optimizing diversity-based objective functions allows for generating broad range of non-linear utility functions, thereby encompassing expansive array of preferences with respect to returns. Subsequently, we utilize the first N = 20 utility functions depicted in Figure to train equivalent number of policies under DPMORL.
Standard MORL Metrics. The results under standard MORL metries are shown in Table Focusing on the Expected Utility, the performance of DPMORL is the highest in the Hopper, HalfCheetah, MountainCar, and DeepSeaTreasee others in terms of expected utility. While for FruitTree environment, DPMORL also obtains consistent performance with the best contender, GPI-LS. terms the HyperVolume, DPMORL also performs the best in the Hopper, HalfCheetah, MountainCar, and Fruit'Tree environments (4/5). The results show that DPMORL can yield better performance across both metrics and most environments (Hopper, HalfCheetah, MountainCar, and DeepSea Treasuree, meanwhile delivering robust and consistent results on the other one (FruitTree). Since PGMORL works on continuous action spaces, here we omit the results of PGMORL on environments with discrete action space, DeepSeaTTreasure and FruitTree.
environments (4/5), indicating that our method outperforms
Distributional MORL Metrics. The results under distributional metrics are shown in Table 2 DPMORL outperforms other methods in most environments on both Constraint Satisfaction (5/5) and Variance Objective (3/5), indicating its strong ability to handle the distributional multi-objective reinforcement learning problem. For the rest environments, DPMORL also obtained comparable results compared with the best contender, GPI-LS. The results show that the policies learned by

---

Table 1: Experimental results of each method on each standard MORL metric on all five environments. "EU" stands for Expected Utility, and "HV" stands for Hypervolume.
Envirumment Metric
Hopper
HallCheetah HV
MountainCar
DeepSea Treasure
FruitTree
EU
HV
EU
HV
EU
HV
GPL-PD
2374.44
5033458.
412.62
1083227.57
-55.44
7367.59
GPI-LS
1398.00
1446705.10 10
98.31
2839051 1.99051.00 66
-37.37
8052.58
5.04
9.36
3.84
1.49
OLS
175.07
3298.13
-580.88
1420270.13 13
-470.00
2.45
4.64
10.28
6.27
7.49
PGMORL
300.19
32621.25
111.30
383681.22
-429.48
94.45
DPMORL (Ours)
3492,93
12154967.99
1189.68
8593769.62
-29.89
8663.80
6.70
8.68
6.89
16.39
Table 2: Experimental results of each method Distributional metric all five environments. "Constraints" stands for Constraints Satisfaction, and "Var" stands for Variance Objective.
Environment
Hopper
HalfChectah
MountainCCr
TrepSeaTreeasure
FruitTree
Metric
Constraints
Var
Construint
Var
Constraint
Constraint
Var
Constraint
Var
GPL-PD
0.47
979.74
0.64
83.49
1,00
-31.15
0.85
2.59
0.65
3.42
GPI-LS
0.25
607.47
0.60
$1.67
1.00
-22733
0.85
1.99
0.33
1.35
OLS
0.05
75.48
0.43
-311.27
0.05
-244.21
0.50
1.86
0.50
2.98
PGMORL 0.05
98.47
0.50
45.48
0.11
-249.69
DPMORL (Ours)
0.76
1645.89
0.82
431.26
1,00
-16.32
0.90
2.54
0.67
3.21
DPMORL have a higher probability of satisfying randomly generated constraints, and can better balance the trade-off between expectations and variances.
Return Distributions of Learned Policies by DPMORL. We provide visualizations of the multi- dimensional return distributions of each policy learned by DPMORL and four baseline methods (PGMORL, OLS, GPI-LS and GPI-PD) on different MORL environments, as shown Appendix [ CC.4
DPMORL's learned policy set demonstrates higher diversity with respect to return distributions compared to baseline methods.
DPMORL on More Than 2-dimensional Return Space. We run experiments with objectives under Hopper and FruitTree environment, and show the learned return distribution and standard MORL evaluation metrics in Appendix CC5] The results demonstrate that DPMORL can effectively learn DPO policies under more than 3-dimensional reward functions, and achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.
Ablation Analysis with Variable Number of Policies. We provide the ablation studies of DPMORL with a varying amount of learned policies N in Appendix C.6 As the number of policies increases, both expected utility and hypervolume increase, which means DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only policies, which verify the effectiveness of DPMORL in all environments.
Conclusion
In this work, we initialized the study of distributional MORL, specifically when preferences different objectives and their return distributions are uncertain. We introduced the concept of Distributional Pareto- Oortmal (DPO) policies with rigorous theoretical analysis, which extend the notion Pareto-optimallity
MORL to include preferences over the entire distribution of returns, not just their expected values. To obtain such desirable policies, we proposed new algorithm, DPMORL, designed to learn DPO policies with non- non-linear utility functions. DPMORL allows for expressing wide range of distributional preferences, providing flexible and expressive approach MORL. Experiment results showed that DPMORL consistently outperformed existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences.
Acknowledgments
This research was supported by the Institute for and Beyond, UTToyoo; JST SPRING, Grant Number JPMJSP2108 and Microsoft Research Asia D-CORE program. The authors would like to thank Soichiro Nishimori, Kazuki Ota, and the anonymous reviewers for their insightful comments and suggestions.
10

---

References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.560,, 2013.
[2] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[3] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10961-1097)) PMLR, 2021.
[4] Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptively- selected demonstrations, In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pages 3155-3161. ijcai.org, 2021.
[5] Diederik Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley, A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
[6] Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstracten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide multi-objective reinforcement learning and planning. Autonomous Agents and Aueti-Agent Systems, 36(1):26, 2022.
[7] Jie Xu, Yunsheng Tian, Pingchuan Ma. Danicla Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guidee multi-objective reinforcement learning for continuous robot control. pages 10607-10616, 2020.
[8] Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guider multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10607-10616. PMLR, 2020.
[9] Rainer Schlosser. Risk-sensitive control of markov decision processes: A moment-based approach with target distributions. Computers & Operations Research, 123:104997, 2020.
[10] Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. In Advances in Neural Information Processing Systems, 2022.
[11] Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pages 20423-20443. PMLR, 2022.
[12] Mathieu Reymond, Conor Hayes, Denis Steckelmacher, Diederik M Roijers, and Ann Nowé. Actor-critic multi-objective reinforcement learning for non-linear utility functions. Autonomous Agents and Multi-Agent Systems, 37(2):23, 2023.
[13] Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management science, 3844)5555-59, 1992.
[14] Moshe Shaked and J George Shanthikumar. Stochastic orders. Springer Science Business Media, 2007.
[15] Daniel J I Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis. and Michael R Kosorok. Estimating dynamic treatment regimes mobile health using v-learning. Journal of the american statistical association, 2020.

---

[16] Lucas Nunes Alegre, Ana L. C. Bazzan, and Bruno C. da Silva. Optimistic linear support and successor features as a basis for optimal policy transfer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 994-413. PMLR, 2022.
[17] Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pages 11-20. PMLR, 2019.
[18] Matthicu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022. pages 489 487. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
[19] Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. Multi-objective reinforcement learning with non-linear scalarization. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages 9-17. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
[20] Conor F Hayes, Mathieu Reymond, Diederik M Roijers, Enda Howley, and Patrick Mannion. Distributional monte carlo tree search for risk-aware and multi-objective reinforcement learning. In Proceedings of the 20th international conference on autonomous agents and multiagent systems. pages 1532, 2021.
[21] Conor F Hayes, Diederik M Roijers, Enda Howley, and Patrick Mannion. Multi-objective distributional value iteration. In Adaptive and Learning Agents Workshop (AAMAS 2022), 2022.
[22] Conor F Hayes, Timothy Verstraeten, Diederik M Roijers, Enda Howley, and Patrick Mannion. Expected scalarised returns dominance: new solution concept for multi-objective decision making. Neural Computing and Applications, pages 1-21, 2022.
[23] Willem Röpke, Conor F. Hayes, Patrick Mannion, Enda Howley, Ann Nowé, and Diederik M. Roijers. Distributional multi-objective decision making. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 571 1-5719. ijcai.org, 2023.
[24] Mare G Bellemare, Will Dabney, and R'emi Munos. A distributional perspective on reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 449-458. JMLR. org, 2017.
[25] Pushi Zhang, Xiaoyu Chen, Li Zhao, Wei Xiong, Tao Qin, and Tie- Liu. Distributional reinforcement learning for multi-dimensional reward functions. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1519-1529, 2021.
[26] Will Dabney, Mark Rowland, Mare G Bellemare, and Remi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[27] Gabriel Barth-Maron Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional de- terministic policy gradients. In 6th International Conference on Leurning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018
12

---

[28] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning. 49:267-290, 2002.
[29] Wenjie Huang and William B Haskell. Stochastic approximation for risk- markov decision processes, IEEE Transactions on Automatic Control, 66(3):1314-1320, 2020.
[30] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49:267-290, 2002.
[31] Javier Gare'ia ia and Fernando Fern'andez, Safe exploration in markov decision processes. arXiv preprint arXiv: 111,,111,10111 2012.
[32] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abheel. Constrained policy optimization. arXiv preprint arXiv: 7705. 10528, 2017.
[33] Yinlam Chow, Ofir Nachum, Edgar A Duenez-Guzman, and Mohammad Ghavamzadeh.
Lyapunov-based safe policy optimization for continuous control. arXiv preprint
1,
[34] Zi-Xuan Chen, Xin-Qiang Cai, Yuan Jiang, and Zhi-Hua Zhou. Anomaly guided policy learning from imperfect demonstrations. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors. 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages
[35] Yunke Wang, Xiyu Wang, Anh-Dung Dinh, Bo Du, and Charles Xu. Learning to schedule in diffusion probabilistic models. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu. Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6- 6-10, 2023, pages
ACM, 2023.
[36] Chumming Liu, Xin Xu, and Dewen Hu. Multi-objective reinforcement learning: A comprehen- sive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-398, 2014.
[37] Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi- objective deep reinforcement learning. arXiv preprint 11Xiv:1610.02707, 2016.
[38] Xin Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou. Imitation learning from pixel-level demonstrations by hashreward. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors, AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021, pages 279-287. ACM, 2021.
[39] Xin-Qiang Cai, Yao-Xiang Ding, Zi-Xuan Chen, Yuan Jiang, Masashi Sugiyama, and Zhi-Hua Zhou. Seeing differently, acting similarly: Heterogeneously observable imitation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[40] Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial imi- tation learning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, SherethSeventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 10262-10270. AAAI Press, 2023.
[41] Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146-155. PMLR, 2017.
[42] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-72) 2020.
13

---

[43] Charles Dugas, Yoshua Bengio, François Bélisle. Claude Nadeau, and René Garcia. Incorporat- ing functional knowledge in neural networks. Journal of Machine Learning Research, 10(6), 2009.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, abs/ abs/1707.06347, 2017.
[45] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmssüüssan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[46] Lucas N. Alegre, Florian Felten, El-Ghazali Talbi, Grégoire Danoy, Ann Nowé, Ana L. C. Bazzan, and Bruno C. da Silva. MO-Gym: A library of multi-objectivve reinforcement learn- ing environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelearn 2022, 2022.
[47] Diederik M Roijers. Multi-objective decision-theoretic planning. Al Matters, 2(4): 2016.
[48] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselinesst Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[49] Takashi Kamihigashi and John Stachurski. Partial stochastic dominance via optimal transport. Operations Research Letters, 4865):884-586, 2020.
[50]
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
[51] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
14

---

A Proof
Lemma 1. If policy x is the only optimal policy under some non-decreasing utility function f, then it is a Distributionally Pareto-Optimal policy.
Proof. If π is the only optimal policy under non non-decreasing utility function f. then for any policy policy n',
T (x)) > Fxz=((xr)
From Definition 1 and Definition 2, any policy n' stochastically dominates я, then for any utility function f. Bz-p((ar) 2 (e)))())) fz), which makes contradition. Therefore, л is not stochastically dominated by any other policies, and a Distributional Pareto-Optimal policy.
Lemma 2. Ifa policy is one of the optimal policy under some strictly increasing utility function then it is a Distributionally Pareto-Optimes policy. The utility function f is strictly increasing if and only if x ž y and + [(x) [yy)
Proof. Following the Equation (2) and (3) of 49 which is derived from Kantorovich duality (Theo- rem 5.10 in [50],, we have
N1 >D #2 if and only if μr * U2 and
E (an))>y(ooz) =
(5)
gell( (ua)
where I1(M1, (2) is the set of all couplings of μe and p42, which is the set of distributions 7(x, v) such that
f (l, v) (ı(2), / (zx, v)de p2(v).
Suppose there exists a policy N' stochastically dominate π, then there exists 7(2, y) e [1(((T)), (())), such that ((x.g)=y(-ovv 1. Since is one of the optimal policy under strictly increasing utility function f, we have
E E Ey)u(T) f()) 0o
I [ - f f(v)
[f - f(u))r(a, d)dddy S Since ((zg))y/oozu i.e. Jaazy 71, vdxdy have
f()))-(())((a) v)drdy = J/ A
and
J| szy
((wx)>(w)))(a gdddd
There are two cases. Case I: there exists e > 0 such that ff (x, ddddd >
Since f is an strictly increasing function, we have
If x Z y and x + then f(x) > f(y).
Then, have:
0 0
[f(x) - f())) y)dad
J[ Juzuto [f(x) f(y))r y) + /1
2
ytorazn [f(x) - f()))r( y)ddd
> /f [f + c) (x,)) ddddy
Jxzyte
15

---

Since > O, everywhere, we have Illzv+elf(y + f y) ydddy O, which yields contradiction.
> 0. Since JTazute 7(2, v) > 0 and 7(2, y) is continuous almost
Case there does not c > 0 such that JJ 70, y)drdy > 0. Then, we have (( y| 1, which contradicts with the assuption that u((²) + p(=)).
Therefore, we have proved that there does exist n' that stochastically dominate n, when has optimal expected utility under some strictly increasing utility function f.
Lemma 3. Any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
Proof. We first show that if distribution μí' stochastically dominates μ., then the expectation Ex [z] dominates LE Ez [z]. Following Kantorovich duality, there exists a coupling 7 of fu' and fe that ((aa)>> [1 szvl 1. Then,
Exww [z] [x-u[=]
(a,y))y [
= () 10] = 0
When ((aay)~/[ y] = O, it requires that that contradicts with the fact that μ SD μ.. Therefore, the expectation Ez-pe [z] dominates the expectation Eggu [z). It follows that if the expectation Ex~u( is not dominated by the expectation Ex [z], then the distribution f4 is not stochastically dominated by μt'
holds where (22 y) > 0, which makes μ μ
If policy ı is Pareto-optimal, then its multidimensional expected returns R ( not dominated by ([Z((') of any other policies T'. Then, the return distribution (T(T) of policy ι is not stochasti- cally dominated by the return distribution (( of any other policies x'. Therefore, policy æ is
Distributionally Pareto-Optimal policy. To conclude, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
Theorem 1. If policy has optimal expected utility under some non-decreasing utility function f. where either
1. utility function f strictly increasing
2. æ is the only optimal policy with greatest expected utility under utility function f
then is a Distributionally Pareto-Optimal policy. Also, any Pareto-Optimal policy is a Distribution- ally Pareto Optimal policy.
Proof. The theorem directly follows from Lemma] Lemma 2 and Lemma
The following theorem proves that the optimal policy over Value at Risk (VaR) and over Conditional Value Risk (CVaR) belong the set of Distributionally Pareto-Optimal policies. Here, the random variable (TT) to represent the negative risk, where smaller value represents larger risks, in order to be aligned with the definition of Pareto-Optimality.
Definition 4. Given distribution μ. the Value at Risk (VaR) is defined as
VaRa(u) = (n [ < 20] 2 a}
Definition 5. Given a distribution μ., the Conditional Value at Risk (CVaR) is defined as
CVaR. (p) = Ez~u[z|z S Va)x
Theorem 3. When K there exists policy with optimal Value at Risk (VaR), and policy with optimal Condition Value at Risk (CVaR), which belongs to the set of Distributional Pareto- Pareto-Optimal Policies.
Proof. We let the policy set TwaR the policy set with optimal value at risk, which zo VaR,, (u(T)) for any æ E TwaR- We let IT to the policy set with non-optimal policy, with VaR, (u(x')) < zo for any n' E Hyyr. Suppose all Distributionally Pareto-Optimal policy belongs to Hy We define
16

---

1,
2 = 2u which is non-decreasing. Then, we have Be 2 a
utility function as f(=) =
0, 2 < 20
for any E NaR, while [z=ju(=")((2)) < a for any x' E Ia Therefore, any policy π E vaR is not stochastically dominated by any policy n' E -Taa- Therefore, there must exists a Distributional Aareto-Optimal Policy in II, which results in a contradiction. Therefore, there must exists a policy with optimal value at risk, which is Distributionally Parcto-Optimal.
Similarly to the case in VaR, we let the policy set IlcaR the policy set with optimal conditional value at risk, which 20 = (J((T) for any я E [CCaR. We let IT'VaR to be the policy set with non-optimal CVaR, with CVaR,, (V(R,((ux=') < 2g for any π I'' Then can prove that any policy VVvR does not stocastically dominate the policy in HevaR- Suppose the opposite case where be the cumulative distribution function of (T(T) and μ(=²), and f(z) 2 (z) must be satisfied to make π <sD n Therefore,
[IcvaR is dominated by n' Iy We let [()),
2g = CVaR(((TT) = E2- (2) S E2- {2) = CVaR(u(TT))
which results in a contradiction. Therefore, any policy I''aaR does not stocastically dominate the policy in IIvaRR. We thus proved that there exists Distributionally Pareto-Optimal policy in I.vaRR which is an optimal policy under conditional value at risk.
L
B Implementation Details
Non-Decreasing Neural Network. We used Von-Decreasing Neural Network to parameterize the utility function. The network architecture consisted of three fully connected layers, with our designed non-decreasing activation activation functions applied to the hidden layers:
f()) concat( -0.55
-0.5), min(x, 0.5), clip(x, 0.5,
This design of activation function is an extension to ReLU activation function, which empirically make the learned utility function more diverse, covering more types of decision boundaries. After each gradient update during training. we clip all the trainable weights in the network to be non-negative ensure that the resulting utility function was non-decreasing. The network's input dimension corresponded to the number of objectives in the environment, and the output dimension was scalar representing the utility value.
Return Normalization. We use a normalization technique by linearly mapping the return into scale [0, 1j without modifying the optimal policies. Specifically, we keep track of the minimum and maximum multivariate returns that the policies have currently achieved, denote by Zmin and mmoz- We let midd = (Zmin 2maz)/2 and d max, [Zmaz,i Zmin.s]. Before the any multivariate return 2 is fed into the utility function, we normalize it by
2norm d (Enorr 2mid)
This maps z linearly to Znorms where Znorm E 00, 1], without affecting the scale of each dimension of returns. In Algorithm 1, the return is normalized before feeding into the utility function to compute the scalar reward.
DPMORL Algorithm. For the implementation of our DPMORL algorithm, we used the Proximal Policy Optimization (PPO) algorithm (44) as the basic RI. algorithm. The learning rate was set to 3 x 10-1, with discount factor of 0.99. For PPO, we used clipping parameter of 0.2. The batch size was set to 256 for all environments and algorithms, with updates performed every 2,048 steps. The utility functions were trained using the Adam optimizer [5]], with learning rate of 3 x 10
C Experimental Details
C.1 Environments
The details of the environments used in the main paper are listed below:
17

---

DeepSeaTreasinn A classic MORL benchmark that requires exploration in a gridworld to find treasures with different values and depths.
FruitTree: A multi-objeccive variant of the classic gridworld problem, where an agent has to collect different types of fruits with varying rewards and penalties.
HalCheetah: continuous control task where simulated cheetah must run and jump over obstacles while minimizing energy consumption.
Hopper: A continuous control task that involves balancing and hopping a simulated one- legged robot while minimizing energy usage.
MountainCar: A classic reinforcement learning problem, extended to include multiple objectives, such as minimizing the time reach the goal and reducing energy consumption.
In the Appendix, we additionally add one environment:
Reacher: A discrete control task where simulated robotic arm must reach target while minimizing energy usage.
C.2
Standard MORL Metries
The details of the standard MORL metrics used in the main paper are listed below:
Expected Utility is similar R-Metrics used in Multi-Objectiig Optimization (MOO). This metric is concerned with the expected utility of the policies on the Pareto front for various weights. It provides means assess the quality of the policies derived by the algorithm in terms their expected performance across different utility functions. A higher Expected Utility indicates that the policies offer better trade-off between the objectives, thereby leading to more desirable outcomes under variety of user preferences:
EU(IT) = -w [max [max fw(z) (z)], well
where fw denotes the user's utility function linearly parameterized by w, and z denotes the returns of episode.
Hypervolume is prominent measure used in Multi- Multi-Ohjective Optimization (MOO) and Multi-Objective Reinforcement Learning (MORL). This metric calculates the volume in the objective space dominated by the Pareto front with respect to reference point. The Hypervolume gives scalar measure of the quality of a Pareto front, with higher values indicating better Pareto front. A Pareto front with larger Hypervolume signifies that the algorithm can yield wider range policies, thus dominating more the objective space, which indicates better exploration of the policy space:
HV r) Leb(y E R 3a E Il z -DD <sD r),
where r is the reference point in the objective space and Leb denotes the Lebesgue measure, which in this context refers to the volume in the objective space. The reference points used in the experiment of the main paper are calculated as the minimal mean returns of each method under evaluation.
C.3 Distributional MORL Metrics
We propose two novel metrics for assessing distributional preference: Constraint Satisfaction and Variance Objective.
For Constraint Satisfaction, we define u(p, TT) to be the probability of return random variance (Tr) that satisfy the constraint p, representing how policy satisfy the given constraint. We randomly generate each constraint defined by n linear constraints represented by (=|Vi, w/ 2 2 ) where w; uniformly sampled from the surface w; 0, 1T w, and c; is sampled from the distribution of (4(iin u.!" max wi 2), where 2 is the set of all return samples of the final policies learned by DPMORL.
For Variance Objective, we define u(p, T) to be a weighted sum of expected return Ez~u(") and standard deviation of return /Varg [z]. We randomly generate w, E 22K where w, 2 0 and
18

---

MountainCar
Reacher
DeepSeaTreasure
0.0
0.0
30.0
-2.5
-10.0
Return
Return
20.0
Return
-5.0
10.0
-20.0
-7.5
0.0
-30.0
-10.0
-1000
-750
-500
-250
30
40
Retum
Return
Return
Hopper
HalfCheetah
FruitTree
4000.0
0.0
3000.0
-100.0
Return
Return
Return
2000.0
-20.00
-300.0
1000.0
2.0
400.0
1000
2000
3000
0
1000 2000
3000
4000
2
6
Retur 1
Return
Return 1
12
17s
1Ts0
R2
fty
T5p
R23
715
Figure 8: Return distribution of the set of policies learned by DPMORL on each environment.
1'w; w = 1. We let w, = [w] wf where wi wº E RRK, and use w! and -wo as the weight for expected return and standard derivation of return respectively.
For both of the metric, we define u(p, ι) to be the satisfaction score of policy a under preference We generate M 100 number of preference P2, PMr For both of the Constraint Satisfaction and Variance Objective metric, the mean score is defined as
M
max u(p:.Ty) JE(L2...,
M
The mean score the above equation measures the best performance satisfying the preference of the user, averaged in M 100 preferences. Higher mean score represents better ability of meeting different preferences. Table the main experiment, all of the numbers are calculated with the above equation.
C.4 Return Distributions of Learned Policies by DPMORL and Baseline Methods
We demonstrate the return distribution of the learned policies by DPMORL in Figure which demonstrates that DPMORL is able learn set of policies with optimal return distributions. We also visualize the return distribution of policies learned by four baseline methods, which is shown Figure DPMORL's learned policy set has higher diversity with respect to return distributions compared to baseline methods.
C.5 DPMORL on More Than 2-Dimensional Return Space
We run experiments with 3 objectives under Hopper and FruitTree environment, where we use the first three dimension of the reward in the MO-Hopper and Fruit-Tree environment the MO-Gymnasium, We show the 3-dimensional return distribution of the set of policies learned by DPMORL and baseline methods in Figure [0. The results demonstrate that DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Also, DPMORL learns better 3-dimensional return distributions than GPI-PD, a competitive MORL baseline method. We also show quantitative evaluation metrics under Hopper and FruitTreee environments with 3 objectives in Table The results demonstrate that DPMORI. achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.
19

---

MountainCar
Hopper
Ha ICChethh
5.0
200.00 0
-40.0
Baturn
100.0
10.0
.1.00
0.0
-00.0
-100.0
22%
-1000
-800
-600
-400
300
400
500
Retur
Return
Return
Rt
Fhee
11g
11
Ags
11s
(a)
MountainCar
Reacher
DespSea Treasure
-10.0
-2.0
Retur:: ?.
-10.0
20.0
Battur:
-1.0
-10.0
0.0
110.0
-940
-935
-930
-925
0
10
20 Retum
30
4.0
0
10
15
Return 1
Retur
Hopper
Half HelfCheetah
FruitTree
140.0
260.0
1.9.0
6.0
-20.00
Degm
120.0
110.0
-300.0-
2.0
150
175
200
225
500
1000
1500
Rctum
(b)
MountainCar
Reacher
DeegSee Treesure
60.0
-2.5
Retum
20.0
10.0
0.0
-30.0
-1030
40
Return
Return
Return
Happer
HalfCheetah
FruitTree
2200.0 0
200.0
2000.0
-250.0
6.0
1800.0 0
Retur
300.0
Retum
330.0
2.0
1400.0 0
400.0
0.0
2000
2500
3000
2000
4000
10
Retur 1
Retur
Return
MountainCar
Reacher
DeepSSa Treasure
40.0
Retur
Detum
4.0
20.0
-1.0
+6.0
0.0
40.0
-100
10
Hopper
HalfCheetah
FruitTren
300.0
1500.0
3,4
1000.0
Neture
-35..0
3.2
3.0
Retum
Retum
Return
(d)
Figure 9: Return distribution of the policy set learned by four baseline methods: PGMORL, OLS, GPI-LS, and GPI-PD.
20

---

Huppor (DPMONII
IOMMORII
Figure 10: DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Left: 3-dimensional return distribution of the set of policies learned by DPMORL. under Hopper and FruitTree environment. Right: 3-dimensional return distribution under the same environment learned by GPI-PD.
Table 3: Experimental results of each method on Hopper and FruitTree environments with 3- dimensional reward function.
Enviroement
Hopper
FruiTres Constrain: Satisfaction
Metric
Expected Uniity
CVaRR
Constraini Satisfaction
Variance Objective
Expected Unilty
CVRR
Variance Objective
LS
1087.77
1286.61
0:15
450.94
4.19
4,55
0.27
1.60
GPLPD
1511.40
1339
0:36
664 (12
4.44
6.76
0:20
2.30
OL.S
441.62
194.56
0.05
75.77
4.72
6.36
0.35
2.36
DPMORT
2107.36
2840.44
0,67
949.96
5.32
7.25
0,38
2.57
C.6 Ablation Analysis with Variable Number of Policies
We also provide the ablation studies DPMORL with a varying amount Iearned policies N, as shown in Table We can that the number of policies increases, both expected utility and hypervolume increase, which indicates that DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only 5 policies, which verify the effectiveness of DPMORL in all environments.
Table 4: Experimental results DPMORL on each standard MORL metric on five environments with various number of policies. "EU" stands for Expected Utility, and "HV" stands for Hypervolume.
Environment
Hopper
HalrCheetah
MountainCar BU
DeepSen/rresuuee
Frui
Metric
EU
HV
EU
HV
HV
EU
HV
EU
HV
DPMORL (N
3274.36
9917663.14
817.61
5676147
29.94
7462.10
6.22
DPMORL (N 10)
3309.52
10512291.02
1244.48
8497055.3 37
-2.994
7465.59
5.86
14.88
6.65
21.64
DPMORL (N 15)
3377.75
11453079.89 82
1245.63
8497055.0 37
-29.94
7465.50
15.02
6.68
23.80
DPMORL (N
3377.75
11453079.82
1749.27
13417685.27
29.94
7465.59
6.53
19.68
6.68
23,80
21


Reviews:
[{"review_id": "1DQOtqjqwj", "reviewer": "Reviewer_y77J", "summary": "In Multi-Objective Reinforcement Learning (MORL), the goal is to learn Pareto optimal policies that achieve a balance among multiple conflicting objectives while considering the user's preferences. Existing MORL methods optimize the Pareto frontier by using a single value obtained from the linear combination of a multi-dimensional reward vector representing the user's preferences. In this paper, instead of the traditional approach, the authors interpret the problem from a return distribution perspective and consider the uncertainty in returns to better capture the user's preference distribution. Furthermore, they propose an algorithm that generates various utility functions, enabling the learning of Pareto optimal policies for diverse preferences.", "questions": "- Can we design utility functions in a real-world setting that consider individual preferences? For instance, in the case of autonomous driving, can we provide information that highlights safety to encourage the creation of utility functions that prioritize safety significantly?\n- I wonder how the algorithm performs even in cases where the multiple objectives in the Continuous Control environment experiments are conflicting. For instance, in the case of HalfCheetah, which has two multi-objectives, \"run\" and \"jump,\" in the experiments, I wonder if the algorithm can still learn Pareto-optimal policies based on preferences when an additional conflicting objective, such as \"run backward,\" is introduced. \n- The authors conducted MORL experiments on various environments in the main experiments. It would be better if they provide more detailed explanation with some examples of what objectives in each environment and how many objectives are there. 2D graphs of Figure 5 and Appendix Figure 1 seem that the environments have two objectives each. I wonder about some results of the experiments with three or more objectives. \n- Appendix Figure 1 depicts the return distribution of the set of policies learned by DPMORL on each environment, and Return 1 and 2 seem to mean returns for two objectives. It would be helpful if authors clarify explanation of what each return means, and what preference each policy has. \n", "limitations": "Author provides the limitations of their works appropriately, and I think there is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Instead of considering the traditional approach of single expected value in MORL, the paper proposes a Distributional Pareto-Optimal (DPO) method by defining the return as a distribution. This approach enables capturing complex preferences and allows for a more nuanced representation of utility functions. \n- The paper defines Stochastic Dominance (SD) and uses it to define Pareto-Optimal policies. Furthermore, it provides a mathematical proof that the pareto-optimal policy is Distributional Pareto-Optimal policy, which means that their return distributions of each policy cannot dominate that of another. \n- The author presents DPMORL algorithm that iteratively generates various non-linear, non-decreasing utility functions in cases where Stochastic Dominance does not exist and then train DPO policies with the generated utility functions. From these utility functions, proposed algorithm identifies Pareto-Optimal policies. By generating various utility functions, the algorithm can achieve to get more flexible and diverse policies set that can accommodate a wide range of preferences. ", "weaknesses": "- The main experiment shows superior performance of the proposed algorithm compared to existing MORL methods but seems to do not fully explain the motivation(ability to capture users' complex preferences and intentions in a more detailed manner). Including additional experiments or toy examples such as diversity of utility function heatmaps between the return distribution approach and the conventional weighted sum method would substantiate the authors' motivation. \n- The proposed loss function aims to generate various utility functions, and it consists of three components: equations (1), (2), and (3). Providing explanations for each of these components would make it more readable, describing their purposes and how they contribute to creating diverse utility functions. \n- It would be beneficial to include an ablation study. For instance, in the main experiment, training 20 utility functions and policies may show differences in evaluation values with changes in the size of N. By conducting experiments with various N sizes and adding the results to the study, it could provide further support for the author's insistence. \n- Additionally, the paper proposes a method to increase the diversity of utility functions by iteratively generating them. In Appendix C Table 1, they applied N^{iter}=2 iterations to create a broader range of utility functions and trained policies based on them. However, the experimental results show that the performance does not exhibit significant differences compared to the experiments where utility functions were generated only once. Increasing N^{iter} is expected to generate a more diverse set of utility functions, enabling the learning of various Pareto-optimal policies. Consequently, this is likely to further enhance the \"Constraints Satisfaction\" score. \n- Minor typos: (1) Line 192 “We maximize” → “We minimize”,  (2) Appendix line 99 “standard derivation” → “standard deviation”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer y77J: \nThanks for your valuable feedback! We provide the response to each of your questions as follows. \n\n**Q1:** “The main experiment shows superior performance of the proposed algorithm compared to existing MORL methods but seems to do not fully explain the motivation”\n\n**A1:** The motivation of the paper is to optimize for a diverse set of policies for optimal return distributions, which is achieved by optimizing policies on a diverse set of non-linear utility functions. In the main paper, we provide case studies in Figure 4 in the DiverseGoal environment, which shows that DPMORL can obtain policies with a diverse set of return distributions with different utility functions to optimize, supporting the motivation on a toy example. Here, we will show the additional experimental results on harder environments in MO-Gymnasium for the same case study, which is shown in Figures 1 and 2 of the global rebuttal PDF file. In the ReacherBullet and DeepSeaTreasure environment, when providing different generated utility functions for the policies, DPMORL is able to optimize the policy to obtain policies with optimal non-linear utility, covering a diverse set of optimal return distributions, which directly supports our motivations. We will add the additional case studies in Figure 1 and Figure 2 to our paper to better support our motivation in the revision. \n\n**Q2:** “equations (1), (2), and (3). Providing explanations for each of these components would make it more readable, describing their purposes and how they contribute to creating diverse utility functions.”\n\n**A2:**  Briefly, equation 1 quantifies the output difference between two utility functions, and accordingly measures the smallest distance from one utility function to any other utility function; equation 2 quantifies the average slope difference between two utility functions, and accordingly measures the smallest distance from one utility function to any other utility function, and equation 3 is a weighted combination of equation 1 and 2, which is optimized to maximize the dissimilarity of the current utility function. We will add the explanation to our paper in the revision. \n\n**Q3:** “conducting experiments with various N sizes and adding the results to the study”\n\n**A3:** We show the experimental results of various N sizes in the Table 2 of the global rebuttal PDF file. As N becomes larger, DPMORL has better Expected Utility, and Hypervolume.\n\n**Q4:** “Increasing N^{iter} is expected to generate a more diverse set of utility functions”\n\n**A4:** In Table 1 in the Appendix, DPMORL in the second iteration has improved performance in 11 of the metrics compared to the first iteration, while having 7 metrics in worse performance. In ReacherBullet, we also find DPMORL in the second iteration having an overall better performance than in the first iteration. We will add the additional experimental results in the revised version of our paper.\n\n**Q5:** “can we provide information that highlights safety to encourage the creation of utility functions that prioritize safety significantly?”\n\n**A5:** In Figures 1 and 2 of the global rebuttal PDF file, we show experimental results demonstrating that DPMORL can effectively optimize policies with utility functions that characterize safety properties. The DeepSeaTreasure environment of Figure 2 aligns well with safe RL settings, where the two returns are time penalty and task reward respectively. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint. Due to the one-page limitation of the rebuttal pdf, we did not provide the comparison results on more environments with safe RL baselines, and we will add more results on this topic in the revision. \n\n**Q6:** “I wonder about some results of the experiments with three or more objectives.” and “in the case of HalfCheetah, which has two multi-objectives, ‘run’ and ‘jump,’ in the experiments, I wonder if the algorithm can still learn Pareto-optimal policies based on\npreferences when an additional conflicting objective, such as ‘run backward,’ is introduced.”\n\n**A6:** The DPMORL algorithm can be applied well for three or more objectives. We have added experiments with 3 objectives under Hopper and FruitTree environment, as illustrated in the Figure 3 and Table 1 of the global rebuttal PDF file. Figure 3 demonstrates that DPMORL learns better 3-dimensional return distributions than GPI-PD, a competitive MORL baseline method; Table 1 demonstrates that DPMORL achieves better performance in different MORL metrics compared to baseline methods under 3 dimensional reward functions. Regarding the HalfCheetah scenario, introducing a diametrically opposing objective, like “running backward”, as an antithesis to “running forward” would render any arbitrary policy as a Pareto-optimal policy. This diverges from the optimization goal of MORL. Hence, we selected widely accepted and pragmatic MORL environments and targets for our experiments.\n\n**Q7:** “Appendix Figure 1 depicts the return distribution of the set of policies learned by DPMORL on each environment, and Return 1 and 2 seem to mean returns for two objectives. It would be helpful if authors clarify explanation of what each return means, and what preference each policy has”\n\n**A7:** We add the brief explanations to the meaning of reward here: for DeepSeaTreasure, MountainCar, and HalfCheetah, the two objectives are respectively task reward (gathering treasure, climb mountain and running) and energy consumption; for FruitTree, Hopper, and Reacher, the two objectives are the reward of reaching two different types of positions. We will add the clarifications in the appendix of our paper. Our case studies (Figure 2) in the global rebuttal PDF file show the preference for a part of learned policies, and we will add preferences for more policies in our revised manuscript. "}, {"review_id": "5RCIYfjxOg", "reviewer": "Reviewer_JBHP", "summary": "This work introduces distributional Pareto-optimality for multi-objective reinforcement learning. Multi-objective RL (MORL) is typically formulated as to find Pareto optimality of all objectives. \n\nTo build the ground, the paper defines stochastic dominance for multivariate distribution and then stochastic dominance for policies. Distributional Pareto-optimality can then be formulated.\n\nThe authors propose a practical algorithm for addressing the distributional MORL problem. First, a non-decreasing NN is used (as the Q network in MARL paper QMIX) as the utility function. Then a new objective function can be defined and creates a reward function for RL. Therefore, we can iteratively train the utility function and policies.\n\nExperiment in multi-objective gymnasium shows the effectiveness of the method.", "questions": "1. The authors list safe RL as a special case to MORL. That would great if we can apply the proposed method and other baselines to safe RL benchmark such as Safety Gym to see the performance and also compared to other safe RL baselines. This is because safe RL is a domain with huge social impact and thus working on those tasks can improve the impact of the proposed method.\n2. In the DiverseGoal environment, how we get the distribution of reward? Is it some kinds of Normal distribution predefined?", "limitations": "The authors do not address the limitation at all. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses the distributional preferences in multi-objective RL, which is an understudied problem.\n2. The paper formulates a framework and propose practical method to address the problem.", "weaknesses": " 1. The paper is self-contained and there is no particular worth-noting weaknesses. It would be great to see if the proposed method can be applied to other more realistic environments, such as safety-aware autonomous driving. I believe driving is a decision-making task that has many objectives that should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer JBHP: \nThanks for your valuable feedback! We provide the response to each of your questions as follows. \n\n**Q1:** “The authors list safe RL as a special case to MORL. That would great if we can apply the proposed method and other baselines to safe RL benchmark such as Safety Gym to see the performance and also compared to other safe RL baselines.”\n\n**A1:** In Figures 1 and 2 of the global rebuttal PDF file, we show experimental results demonstrating that DPMORL can effectively optimize policies with utility functions that characterize safety properties. Specifically, the DeepSeaTreasure environment in Figure 2 resonates with safe RL dynamics, balancing between time penalties and task rewards. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint. Due to the one-page limitation of the rebuttal pdf, we cannot provide the comparison results on more environments with safe RL baselines, but we're committed to delving deeper into this in our revised manuscript.\n\n**Q2:** “In the DiverseGoal environment, how we get the distribution of reward? Is it some kinds of Normal distribution predefined?”\n\n**A2:** Yes, in the DiverseGoal environment, the reward distribution associated with each goal position is a pre-defined Normal distribution. Contrasted with standard environments where the reward functions are deterministic, in our DiverseGoal environment, the reward from distinct objectives is sampled from unique Normal distributions. Consequently, the approach necessitates acquiring a policy set embodying the distributional attributes of returns to cater to diverse user preferences. For more environments in MO-Gymnasium, we provide the same case study results on the effectiveness of our method for optimizing a diverse set of non-linear utility functions under stochastic rewards in Figures 1 and 2 of the global rebuttal PDF file. "}, {"review_id": "KbY4HPtxYS", "reviewer": "Reviewer_GTy6", "summary": "The authors propose extending multi-objective RL (MORL) to policies that are Pareto optimal over distributions of non-negative utility functions. Specifically, they define distributional Pareto optimal (DPO) policies as those whose expected returns are non-dominated for any non-negative utility function. They then propose learning a set of DPO policies by first learning a set of diverse non-negative utility functions over $[0 ~ 1]^K$ bounded objectives, then learning their corresponding optimal policies using PPO. They demonstrate in several experiments that their approach outperforms prior works in both hyper-volume and expected utility of the learned policy set.", "questions": "It would be great if the authors can address the major weaknesses I outlined above. I am happy to increase my score if they are properly addressed, as I may have misunderstood pieces of paper. In addition to those:\n\n- Why is the set of utility functions restricted to non-decreasing ones? Is this a fundamental limitation of the proposed framework?\n\n- Why is the given Diversity-based Objective Function expected to lead to policies that cover significant volume of the Pareto front? The authors should provide a proof of how well their approach approximates the Pareto front, or at the very least a detailed discussion of it.\n\n- How is the number of utility functions to learn ($M$) determined? Is this just an arbitrary hyperparameter? If so, how does its choice affect how well the approach approximates the Pareto front?\n\n- Why is the given definition of DPO policies to be preferred over that of [1]? Can the authors compare against MO-MPO from [1]?\n\n- In the proof of Lemma 3, the authors say \"Suppose π is a Pareto-Optimal policy, then π is the optimal policy under some linear combination of rewards with weight w ≻ 0\". Why is this true? Is the lemma maybe assume linear utility functions (if yes this needs to be made explicit)?", "limitations": "While there is no limitations section, I believe the authors have adequately addressed the limitations of their work sparsely throughout the paper (besides the ones I mentioned in the weaknesses section). It will be preferable if a limitations section is added to discuss the ones the authors feel are most relevant.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written, clearly presented, and tackles the very significant problem of finding policies that are distributionally Pareto-optimal in MORL.\n\n- I like the paper's formal treatment of Distributional MORL. The definitions and theorems given are particularly clear and precise, showing that the proposed framework is sound (not including how to actually find DPO policies, and not saying I agree with said definitions).\n\n- The proposed approach for finding DPO policies is particularly interesting. By learning a set of utility functions using a diversity metric, then converting the learned utilities to a reward function, one can then use any off-the-shelf RL algorithm to learn the DPO policies. The paper then shows that these optimal policies are guaranteed to maximise their respective utilities.\n\n- The paper conducts several experiments demonstrating that the learned utility functions are indeed diverse and non-decreasing. The paper also demonstrates that their approach outperforms prior works in several functional approximation domains.", "weaknesses": "MAJOR:\n\n- Theorem 1 is stated very differently in the Appendix (lines 33-38) compared to the main paper (lines 153-155). In particular, the one in the Appendix adds 2 more assumptions (lines 35-36). This makes the one in the paper severely misleading. This also decreases the strength of this theorem since there almost never is only a single optimal policy in RL (optimal policies are rarely unique), and strictly increasing utility functions further limits the applicability of the proposed framework.\n\n- The paper limits the definition of distributional MORL to non-decreasing utility functions, and the proposed algorithm for finding DPO policies is limited to non-decreasing utilities with objectives bounded by $[0 ~ 1]^K$. While this still covers a wide range of utilities, this severely limits the applicability of the proposed approach.\n\n- No proof is given to show what percentage volume of the Pareto front is covered by the set of optimal policies obtained by the proposed approach. In fact, there doesn't seem to be a reason why we should expect the specific diversity metric given in equation (3) to give us utility functions that lead to DPO policies covering any significant fraction of the Pareto frontier. This is a major missing component of the paper since its operational goal is to extend MORL to DPO policies.\n\n- [1] seems closely related to this paper as it also proposes a distributional view of MORL, and its implementation is not constrained to non-decreasing utilities over objectives bounded by $[0 ~ 1]^K$. However, this paper does not include it in its baselines, nor does it discuss or even cite it.\n\n\nMINOR\n\n- f(z) is missing on line 7 of the proof of Lemma 1\n\n\n[1] Abdolmaleki, Abbas, et al. \"A distributional view on multi-objective policy optimization.\" International conference on machine learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer GTy6: \nThanks for your valuable feedback! We provide the response to each of your questions as follows. \n\n**Q1:** “Why is the set of utility functions restricted to non-decreasing ones? Is this a fundamental limitation of the proposed framework?” and “the proposed algorithm for finding DPO policies is limited to non-decreasing utilities with objectives bounded by $[0, 1]^K$ ”\n\n**A1:** This non-decreasing condition is foundational to MORL, aiming to optimize all objectives simultaneously without diminishing any. This basic premise for non-decreasing utility functions is articulated in works like Definition 7 in [2], Definition 3 and 5 in [3], and Section 3.1 in [4]. The normalization of utility function outputs to [0, 1] is a prevalent technique to stabilize model optimization and policy learning across varied objectives, which might span different numerical ranges. This is not a limitation per se but a design choice.\n\n**Q2:** “there doesn't seem to be a reason why we should expect the specific diversity metric given in equation (3) to give us utility functions that lead to DPO policies covering any significant fraction of the Pareto frontier.”\n\n**A2:** At the methodology level, our method (DPMORL) optimizes policies to maximize the expected utility on a diverse set of non-linear utility functions, where Theorem 1 shows that these policies are DPO policies when they have optimal expected utility. At the experimental level, we show in Figure 1 and Figure 2 of the global rebuttal PDF file that DPMORL can learn optimal policies under a diverse set of non-linear utility functions, where the policies cover a diverse set of optimal return distributions. Therefore, we show DPMORL can learn a diverse set of optimal return distributions at both method and experimental levels. \n\n**Q3:** “How is the number of utility functions to learn (M) determined?”\n\n**A3**: In our methodology, the quantity of utility functions aligns with learned policies. This count is a predetermined parameter for MORL scenarios. For fairness in our experiments, we ensured that all baselines method and our method share the same number of learned policies.\n\n**Q4:** “[1] seems closely related to this paper as it also proposes a distributional view of MORL” and “Why is the given definition of DPO policies to be preferred over that of [1]?”\n\n**A4:** Thank you for highlighting the work in [1]. We acknowledge this work's importance in advancing the MORL field by introducing a distributional perspective. However, we'd like to clarify some misunderstandings. \n\n1.  **Fundamental Conceptual Differences**: Our work introduces the concept of DPO policies to ensure that the policies we derive can accommodate a wide range of distributional preferences in the **return space**, capturing nuances often missed when focusing purely on expected returns. In contrast, [1] proposes a scale-invariant approach to MORL. While this is a valuable perspective, it emphasizes the **action space** and combining objectives in distribution space without necessarily prioritizing the distributional nature of the returns themselves.\n2. **Implementation Details**: Our approach is grounded in the concept of stochastic dominance for multivariate distributions to learn desirable DPO policies. On the other hand, [1] introduces MO-MPO, characterized as a two-tier policy enhancement procedure. It's pivotal to understand that while MO-MPO is a technique, DPO serves as the objective in our research. \nGiven the distinct ambitions and approaches of the two papers, we did not include [1] as one of the baselines in the experiments. Nonetheless, we appreciate the academic merit of comparative discourse. In acknowledgment of the broader academic community's interests, we are glad to enrich our revisions with discussions [1] with our contributions.\n\n**Q5:** “In the proof of Lemma 3, the authors say \"Suppose π is a Pareto-Optimal policy, then π is the optimal policy under some linear combination of rewards with weight w ≻ 0\". Why is this true? Is the lemma maybe assume linear utility functions (if yes this needs to be made explicit)?”\n\n**A5:** This is correct from Definitions 1 and 4 in [3]. For Lemma 3 “Any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy”, we provide a better proof sketch for the lemma from scratch as follows: if policy π is Pareto-optimal, then its multidimensional expected returns are not dominated by any other policies (see Definition 2.1 in [4]). Then, the return distribution mu(π) of policy π is not dominated by the return distribution mu(π’) of any other policies π’. Therefore, policy π is a Distributionally Pareto-Optimal policy. Lemma 3 has no other assumptions, and does not assume linear utility functions. \n\n**Q6:** “Theorem 1 is stated very differently in the Appendix (lines 33-38) compared to the main paper (lines 153-155). In particular, the one in the Appendix adds 2 more assumptions (lines 35-36). This makes the one in the paper severely misleading.”\n\n**A6:** We appreciate your feedback, where the statement in Theorem 1 lacks the required assumptions, and we will add the assumptions in our paper. Meanwhile, we note that either assumption “π is the only optimal policy” or assumption “utility function f is strictly increasing” can make the claim of Theorem 1 correct, which makes the conclusion adaptable to many more scenarios (than the case where both the assumption has to be satisfied). \n\n## References\n[1] A distributional view on multi-objective policy optimization. ICML 2020.\n\n[2] A Survey of Multi-Objective Sequential Decision-Making. Journal of Artificial Intelligence Research, 2013.\n\n[3] A practical guide to multi‑objective reinforcement learning and planning. AAMAS, 2022.\n\n[4] Prediction-guided multi-objective reinforcement learning for continuous robot control, ICML 2020.\n"}, {"review_id": "TFSlQTLztO", "reviewer": "Reviewer_1W6z", "summary": "The paper introduces a new approach for solving Multi-Objective Reinforcement Learning (MORL) problems. Traditional MORL methods aim at optimizing multiple objectives, but generally focus on the expected values of returns, which can be inadequate in real-world scenarios with diverse preferences over returns. The paper addresses this limitation by extending Pareto-optimality to include distributional preferences, called Distributional Pareto-Optimality (DPO). The authors also include discussions of the relationship between Pareto-optimality policies and Distributional Pareto-Optimality policies. Furthermore, the authors propose a novel algorithm, DPMORL, that learns policies considering both the return distributions and their expectations. It captures the optimality of multivariate distributions through stochastic dominance. Through experiments on several benchmarks, DPMORL was found to be effective in learning distributional Pareto-optimal policies and outperformed existing MORL methods.", "questions": "I would appreciate further clarification regarding Table 2. Specifically, what are the constraints being referenced in this table? Could you provide illustrative examples to enhance understanding?", "limitations": "It would be better to delve deeper into the distribution properties of preferences/constraints and to illuminate the limitations therein with greater specificity. Including illustrative examples of such constraints would not only bolster comprehension but also offer valuable insights into the practical implications and considerations of employing DPMORL in real-world scenarios.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper astutely addresses a significant perspective in current MORL research by emphasizing the importance of not only accounting for the expected values of different objectives but also examining the distributional properties of returns in the context of the heterogeneity and diversity inherent in users' preferences. DPMORL stands out for its remarkable flexibility, as it adeptly accommodates a wide spectrum of distributional preferences, making it highly versatile and well-suited for an array of problem scenarios. This attributes to a more nuanced and expressive approach to MORL. Moreover, the algorithm is anchored in a robust theoretical foundation. The paper lucidly delineates Distributional Pareto-Optimal policies and astutely establishes their interrelation with utility functions. This amalgamation of theoretical rigor and empirical validation accentuates the potential of DPMORL to tackle distributional MORL challenges with efficacy. Furthermore, the authors deserve commendation for the inclusion of an intuitive and easily comprehensible case study that effectively demonstrates the utility function of learning.", "weaknesses": "The figures of illustration of 2D utility functions, including the ones in Appendix, are arranged somewhat randomly. It is suggested that the authors arrange the results of the utility function according to some gradual pattern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer 1W6z: \nThanks for your valuable feedback! We provide the response to each of your questions as follows. \n\n**Q1:** “I would appreciate further clarification regarding Table 2. Especially, what are the constraints being referenced in this table? Could you provide illustrative examples to enhance understanding?”\n\n**A1:** The way of constructing the constraints in Table 2 is detailed in the “Evaluation Metrics” paragraph of Appendix B. Briefly, each constraint is a combination of multiple linear constraints for the multidimensional returns. This covers different constraint satisfaction scenarios, including single safety constraints and multiple combinatorial safety constraints. We will elucidate this with specific examples in our revised manuscript.\n\n**Q2:** “It is suggested that the authors arrange the results of the utility function according to some gradual pattern.”\n\n**A2:** We appreciate your feedback. Our initial intent was to portray the diversity of utility functions, hence the randomized arrangement. Based on your suggestion, we will arrange the utility functions by their average slopes for better clarity in the revised version.\n"}]

Author rebuttal:
We express our sincere gratitude to all reviewers for their constructive feedback and insightful suggestions. In this global response, we include a supplementary PDF comprising additional experimental outcomes. We summarize the experiment result as follows: 

### **Figure 1: Enhanced Case Studies for DPMORL on ReacherBullet Environment**

In the case study, we show that DPMORL can effectively learn optimal policies under a diverse set of non-linear utility functions under the complex control environment (ReacherBullet), and the policies cover a wide range of optimal return distributions. We randomly generate 7 different non-linear utility functions with our method, and learn 7 policies by DPMORL to maximize expected utility for each utility function. We use the first two reward functions (two goal-reaching rewards) in MO-Gymnasium. 

The results show that our algorithm effectively finds policies that maximize the expected utility for different non-linear utility functions, and the learned policies cover broader areas on the return distributions compared to the baseline methods.  

### **Figure 2: Enhanced Case Studies for DPMORL on DeepSeaTreasure Environment**

In the case study, we show that DPMORL can effectively learn optimal policies under a diverse set of non-linear utility functions under the DeepSeaTreasure environment. We randomly generate 7 different non-linear utility functions with our method, and learn 7 policies by DPMORL to maximize expected utility for each utility function. We use the same two-dimensional reward functions (two goal-reaching rewards) as in MO-Gymnasium. 

The results show that our algorithm effectively finds policies that maximize the expected utility for different non-linear utility functions. 

The DeepSeaTreasure environment of Figure 2 aligns well with safe RL settings, where the two returns are time penalty and task reward respectively. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint.

### **Figure 3: Return distribution of DPMORL on more-than-two Dimensional Objectives**

In this experiment, we show the 3-dimensional return distribution of the set of policies learned by DPMORL under Hopper and FruitTree environments. The results demonstrate that DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Also, DPMORL learns better 3-dimensional return distributions than GPI-PD, a competitive MORL baseline method. 

### **Table 1: Comparative Analysis for DPMORL on more-than-two Dimensional Objectives**

In this experiment, we compare the performance of our method on more than two dimensions of reward with baseline methods on four MORL evaluation metrics. We run our experiment on Hopper and FruitTree environments with 3-dimensional reward function, following the reward settings in MO-Gymnasium. We measure four MORL metrics: conditional value at risk, constraint satisfaction and variance Objective, and expected utility. These metrics are detailed in Appendix C. DPMORL achieves the best performance compared to all baseline methods under 3d reward functions.

### **Table 2: Ablation Analysis with Variable Policy Counts**

In this experiment, we show the ablation studies of our method with a varying amount of learned policies ($N$). We can see that as the number of policies increases, both expected utility and hypervolume increase, which means DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exists only 5 policies, which verify the effectiveness of DPMORL among all environments.


Meta-review: This paper considers the problem of learning Pareto-optimal policies under risk-sensitive risk criteria. A method for generating diverse monotonic utility functions is proposed, and distinct policies are then trained on these reward functions, as a way of approximating policies at the Pareto frontier with respect to the risk-sensitive criteria considered.

All reviewers recommend acceptance for the paper. There is a general consensus that the authors are tackling an interesting problem, that there is merit in the approach they propose, and that the paper is well written.

Several important concerns were also noted during the review process, including:
 - Reviewer GTy6 notes that the authors introduced additional assumptions into the statement of Theorem 1 in the course of proving it in the appendix. It is crucial that the main paper statement is amended to reflect these assumptions, and some discussion is given as to the strength of the assumptions/applicability of the theorem in light of them. In particular, as Reviewer GTy6 notes, a unique optimal policy is a strong assumption.
 - From a narrative point of view, several reviewers felt that the paper could do more to connect the proposed approach to the stated motivations relating to capturing complex preferences.
 - The focus of the main paper experiments is primarily on qualitative aspects (i.e. visualization of learned utility functions) and performance (Tables 1 and 2). Several reviewers noted that the experimental work in the paper could be significantly strengthened with ablative experiments. I strongly agree with this, and think this is the clearest path to improving the paper. In particular, experiments that aim to develop an understanding of the hyperparameters of the method, as well as situations in which the method may not work as well (such as the case of conflicting objectives, as mentioned by Reviewer y77J) would strengthen the paper considerably.
 - Reviewer GTy6 also notes that there is relatively little theoretical/empirical investigation into the diversity objective in Equation (3), and why this (rather than some other objective) should be expected to provide a wide covering of Pareto-optimal policies.

The authors have posted a detailed rebuttal in response to these questions, including additional experiments in the rebuttal pdf. After discussion, the reviewers believe the strengths of the paper outweigh the concerns described above.

In my own view, there are a few other aspects I would have liked more commentary on, which should be straightforward to address for the authors:
 - The specific types of risk-sensitivity the authors consider are expectations of non-linear functions of the return. I believe this excludes common risk measures such as CVaR, and I would have liked to see this limitation discussed.
 - It would be good to make Definition 1 more precise by being clear about exactly what is meant by non-decreasing f in the multivariate case.
 - Reference [19] for distributional MPO is not correct. A more appropriate reference may be Acme (Hoffman et al., 2020).

The recommendation overall is to accept the paper. I strongly encourage the authors to take on board the constructive feedback provided by the reviewers in the course of preparing the camera-ready version of the paper.
Decision: Accept (poster)