Title: Distributional Pareto-Optimal Multi-Objective Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Objective Reinforcement Learning
Abstract: Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account for distributional preferences over the multi-variate returns, which are particularly important in real-world scenarios such as autonomous driving. To address this issue, we extend the concept of Pareto-optimality in MORL into distributional Pareto-optimality, which captures the optimality of return distributions, rather than the expectations. Our proposed method, called Distributional Pareto-Optimal Multi-Objective Reinforcement Learning~(DPMORL), is capable of learning distributional Pareto-optimal policies that balance multiple objectives while considering the return uncertainty. We evaluated our method on several benchmark problems and demonstrated its effectiveness in discovering distributional Pareto-optimal policies and satisfying diverse distributional preferences compared to existing MORL methods.

Abstract

Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account for distributional preferences over the multi-variate returns, which are particularly important in realworld scenarios such as autonomous driving. To address this issue, we extend the concept of Pareto-optimality in MORL into distributional Pareto-optimality, which captures the optimality of return distributions, rather than the expectations. Our proposed method, called Distributional Pareto-Optimal Multi-Objective Reinforcement Learning (DPMORL), is capable of learning distributional Pareto-optimal policies that balance multiple objectives while considering the return uncertainty. We evaluated our method on several benchmark problems and demonstrated its effectiveness in discovering distributional Pareto-optimal policies and satisfying diverse distributional preferences compared to existing MORL methods.

1 Introduction

Apart from most Reinforcement Learning (RL) works that consider the scalar reward of a task [1, 2, 3, 4], Multi-Objective Reinforcement Learning (MORL) has recently received extensive attention due to its adeptness at managing intricate decision-making issues with multiple conflicting objectives [5, 6, 7, 8]. In many multi-objective tasks, the relative preferences of users over different objectives are typically indeterminate a priori. Consequently, MORL's primary aim is to learn a variety of optimal policies under different preferences to approximate the Pareto frontier of optimal solutions. It has been demonstrated that MORL can significantly reduce the reliance on scalar reward design for objective combinations and dynamically adapt to the varying preferences of different users.

However, numerous real-world situations involve not only unknown relative preferences across multiple objectives, but also uncertain preferences in return distributions. These may include preference in risk [9, 10], safety conditions [11], and non-linear user's utility [12]. Consider, for instance, autonomous driving scenarios where agents need to strike a balance between safety and efficiency objectives. Different users might possess varying levels of risk tolerance. Some may demand high safety performance, tolerating lower expected efficiency, while others may seek a more balanced performance between safety and efficiency. Current MORL methods, by focusing exclusively on expected values with linear preferences, may not adequately capture multi-objective risk-sensitive preferences, hence being unable to deliver diverse policies catering to users with varied risk preferences.

In this work, we broaden the concept of Pareto-optimality in MORL to encompass distributional Pareto-optimality, which prioritizes the optimality of return distributions over mere expectations, as

(a) MORL's Pareto-front

(b) Distributional MORL's Pareto-

front

f2

f2

depicted in Figure 1. From a theoretical perspective, we define Distributional Pareto-Optimal (DPO) policies, which capture the optimality of multivariate distributions through stochastic dominance [13, 14]. Distributional Pareto-Optimality delineates the set of policies with optimal return distributions, which we formally establish as an extension of Pareto-optimality in MORL.

On the practical side, we propose a novel method named Distributional Pareto-Optimal MultiObjective Reinforcement Learning (DPMORL) 2 , which aims to learn a set of DPO policies. We demonstrate that a policy achieving the highest expected utility for a given strictly increasing utility function is a DPO policy. To this end, we first learn diverse non-linear utility functions and then optimize policies under them [15]. Experimental outcomes on several benchmark problems attest to DPMORL's effectiveness in optimizing policies that meet preferences on multivariate return distributions. We posit that our proposed method significantly addresses the challenge of managing multiple conflicting objectives with unknown preferences on multivariate return distributions in complex decision-making situations.

Our main contributions are listed as follows:

We introduce the concept of Distributional Pareto-Optimal (DPO) policies. This expands the notion of Pareto-optimality in MORL to include preferences over the entire distribution of returns, not just their expected values. This enables agents to express more nuanced preferences over policies and align their decisions more closely with their actual objectives.

We propose DPMORL, a new algorithm for learning DPO policies under non-linear utility functions. This algorithm accommodates a broad range of distributional preferences, thus offering a more flexible and expressive approach to MORL.

We execute extensive experiments on various MORL tasks, demonstrating the effectiveness of our approach in learning DPO policies. Our findings show that our algorithm consistently surpasses existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences, underscoring its practical benefits.

2 Related Work

Multi-Objective Reinforcement Learning. MORL has emerged as a vibrant research area in the reinforcement learning community owing to its adeptness in managing intricate decision-making scenarios involving multiple conflicting objectives [5, 6]. A plethora of MORL algorithms have been put forth in the literature. For instance, [16] proposed the utilization of Generalized Policy Improvement (GPI) to infer a set of policies, employing Optimistic Linear Support (OLS) for dynamic reward weight exploration [17]. This GPI-based learning process was further enhanced by [7] through the incorporation of a world model to augment sample efficiency. [8] proposed an evolutionary approach to learn policies with diverse weights. However, our proposed approach diverges from these by leveraging utility functions to guide policy learning [18, 19]. The utility-based paradigm [6] accentuates the user's utility in decision-making problems, capitalizing on known information about the user's utility function and permissible policy types. Such methods scalarize the multidimensional objectives into a single reward function, enabling traditional RL algorithms to infer

desirable policies, while also respecting axiomatic principles where necessary. Several studies have utilized non-linear utility functions to guide policy learning [20, 12], catering to more intricate preferences compared to linear ones but still in view of the expectation of the Pareto front. In contrast, our work extends the Pareto-optimality to distributional Pareto-optimality by introducing a set of non-linear utility functions with distributional attributes to guide policy learning. Some of the recent works [21, 22, 23] in MORL defines Distributional Undominated Set or ESR set which is both very close to Distributionally Pareto-Optimal policies in this paper. These works propose novel designs to enhance policy improvement in distributional reinforcement learning algorithms to find the Distributional Undominated Set. Compared to these works, we build novel theoretical results on the conditions of DPO policies and a novel algorithm (DPMORL) of learning DPO policies based on reward shaping that can be built on the top of any online RL algorithm.

Distributional Reinforcement Learning. Distributional RL extends traditional RL by modeling the entire distribution of returns, rather than just their expected values [24, 25]. This approach has been shown to improve both learning efficiency and policy quality in a variety of single-objective RL tasks [26, 27]. Key algorithms in this area include Categorical DQN (C51) [24], Quantile Regression DQN (QR-DQN) [26], and Distributional MPO [27]. Despite the success of these distributional RL algorithms in single-objective settings, their direct extension to MORL has been limited due to the added complexity of handling multiple conflicting objectives and expressing preferences over the distribution of returns. On the other hand, the insights and techniques developed in distributional RL can provide a valuable foundation for incorporating distributional preferences into MORL, as demonstrated by our proposed approach.

Risk-Sensitive and Safe Reinforcement Learning. Risk-sensitive and safe RL are special cases of MORL that accentuate specific facets of reward distributions [6]. Risk-sensitive RL primarily considers reward variance, striving to optimize policies that negotiate the trade-off between expected return and risk [28, 29, 30]. Conversely, safe RL prioritizes constraints on the agent's behavior, ensuring adherence to specified safety criteria throughout the learning process [31, 32, 33]. Some studies have proposed the integration of constraints into the state space, constructing new Markov Decision Processes [11]. Others have explored the distributional aspects of rewards, investigating the implications of distributional RL on risk-sensitive and safety-oriented decision-making [26, 24]. However, our proposed setting is more general in terms of objectives, as it considers a broader range of user preferences and captures the entire distribution of rewards, rather than focusing solely on specific aspects such as risk, variance, or constraints. By extending MORL to incorporate distributional properties, our approach enables the learning of distributional Pareto-optimal policies that cater to diverse user preferences and offer better decision-making in a wide range of real-world applications.

3 Preliminaries: Multi-Objective Reinforcement Learning

Similar to the procedure of RL [34, 35], in MORL, the agent interacts with an environment modeled as a Multi-Objective Markov Decision Process (MOMDP) with multi-dimensional reward functions. A Multi-Objective MDP is defined by a tuple ( S, A, P, P 0 , R , γ, T ) , where S is the state space, A is the action space, P is the state transition function, P 0 ( s ) is the initial state distribution, R : S × A × S → R K is a vectored reward function and K is the number of objectives, γ is the discount factor, T is the total timesteps 3 . The goal of the agent is to learn a set of Pareto-optimal policies, which represent the best possible trade-offs among the different objectives.

One popular methodology for MORL problems is the utility-based method, which combines the multi-dimensional reward functions into a single scalar reward function using a weighted sum or another aggregation method [36]. The intuition is to map the agent's preferences over different objectives to a scalar value for the agent training. Given a weight vector w = ( w 1 , w 2 , . . . , w K ) , with w i representing the importance of the i -th objective, the scalarized reward function is defined as r scalar ( s, a ) = ∑ K i =1 w i r i ( s, a ) . The agent then solves the scalarized MDP by optimizing its policy to maximize the expected scalar reward, using standard reinforcement learning algorithms like Q-learning or policy gradient methods. This approach can be straightforward to implement and has been shown to be effective in various MORL settings under expected preferences [37]. However, such a linear combination of each dimension of the reward cannot deal with the preferences with distributional considerations.

4 Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning

In this section, we introduce our proposed theoretical framework and algorithms for extending MORL to handle distributional preferences.

4.1 Distributionally Pareto-Optimal Policies

In this work, we consider the reward as a K -dimensional vector, where each element represents the reward for a specific objective. Given the MOMDP and a policy π , we define the random variable of multi-objective return as Z ( π ) = ∑ T t =0 γ t r t , where r t is the multi-dimensional reward at step t , and the states s 0 , s 1 , · · · , s T and actions a 0 , a 1 , · · · , a T are sampled from the MOMDP and the policy π respectively. The return distribution of policy π , denoted as µ ( π ) , represents the joint distribution of the returns Z ( π ) following policy π . The utility function f is a non-decreasing function that maps a K -dimensional return into a scalar utility value for the agent [38, 39, 40], capturing the user's distributional preferences over the different objectives. The expected utility of policy π under the utility function f , represented as E z ∼ µ ( π ) f ( z ) , is the expected value of applying f to the return distribution µ ( π ) .

Our goal is to learn a set of policies that are distributionally Pareto-optimal, which means that their return distributions of each policy cannot dominate that of another. To measure such a relationship, here we first introduce the concept of stochastic dominance:

̸

Definition 1 (Stochastic Dominance for Multivariate Distribution) . A multivariate distribution µ 1 dominates another distribution µ 2 , denoted as µ 1 ≻ SD µ 2 , if and only if µ 1 = µ 2 and for any non-decreasing utility function f : R K → R , µ 1 has greater expected utility than µ 2 , i.e. E z ∼ µ 1 f ( z ) ≥ E z ∼ µ 2 f ( z ) .

Definition 2 (Stochastic Dominance for Policies) . A policy π 1 stochastically dominates another policy π 2 , denoted as π 1 ≻ SD π 2 , if and only if µ ( π 1 ) ≻ SD µ ( π 2 ) , indicating π 1 has greater expected utility than π 2 under any non-decreasing utility function f .

1

0

1

2

3

4

5

6

z

0

1

0

1

2

3

4

5

6

7

z

1

2

SD

1

1

2

5.0

2.5

0.0

2.5

5.0

7.5

10.0

12.5

z

0

4

2

0

2

4

6

8

10

z

1

1 and

2 cannot dominate each other

1

2

2

0

2

4

6

8

10

z

0

4

2

0

2

4

6

8

10

z

1

1 and

2 cannot dominate each other

1

2

Figure 2: Three 2D examples of stochastic dominance given synthetic returns of two policies. (a) π 2 ≻ SD π 1 , since any non-decreasing utility function satisfies E f ( Z ( π 2 )) ≥ E f ( Z ( π 1 )) . (b) π 1 and π 2 cannot dominate each other. For example, there exists f ( z ) = ReLu( z 0 -5 . 0) , such that E f ( Z ( π 2 )) > E f ( Z ( π 1 )) , thus π 1 cannot dominate π 2 . Similarly, π 2 cannot dominate π 1 . Thus there is no dominant relationship between the two policies. (c) π 1 and π 2 cannot dominate each other, which is similar to (b).

The definition of stochastic dominance extends univariate first-order stochastic dominance [13] into multivariate cases. Figure 2 provides some examples of stochastic dominance given 2D returns of two policies. The definition of stochastic dominance in policies allows for comparing the optimality of distributions between policies, which allows us to define the Distributionally Pareto-optimal policy:

Definition 3 (Distributionally Pareto-Optimal Policy) . Formally, π 1 is Distributionally ParetoOptimal Policy if there does not exist a policy π 2 such that µ ( π 2 ) ≻ SD µ ( π 1 ) .

In other words, π 1 is considered Distributionally Pareto-Optimal (DPO) if it is not stochastically dominated by any other policy. DPO policies are essential in our framework as they represent the most desirable policies for users with different distributional preferences. To find such a policy, we have the following theorem:

Theorem 1. If a policy π has optimal expected utility under some non-decreasing utility function f , where either utility function f is strictly increasing, or π is the only optimal policy with greatest expected utility under utility function f , then π is a Distributionally Pareto-Optimal policy. Also, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.

The proof of Theorem 1 is provided in Appendix A which utilizes a form of optimal transport theory for stochastic dominance. In practice, this theorem guarantees that the optimal policy of a strictly increasing non-linear utility function is a DPO policy, making it a suitable candidate for deployment in MORL problems. The strict increasing property of utility function f in Theorem 1 is crucial in the theoretical analysis for proving the distributional pareto-optimality. In the next section, we base on the result of this theorem to find optimal policies for a diverse set of utility functions, in order to learn the set of DPO policies. This result also shows that our definition of Distributional Pareto-Optimal policies is an extension of Pareto-optimal policies, allowing for a more diverse set of optimal policies to be captured.

We also formally prove that optimal policies for risk-sensitive and safe constraint objectives belong to the set of Distributional Pareto-Optimal policies. This proves that DPO policies can successfully cover policies with diverse distributional preferences. The detailed proof is provided by Theorem 3 in Appendix A.

4.2 Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning

We now present our proposed algorithm, Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning (DPMORL). The main idea of DPMORL is to learn a set of non-linear utility functions that can guide the agent to discover distributional Pareto-optimal policies. The algorithm proceeds in two stages: (1) generating the utility functions and (2) training the policies.

4.2.1 Utility Function Generation with Diversity-based Objective

The first component of our algorithm focuses on generating a diverse set of plausible utility functions, upon which we find the optimal policies to find a diverse set of optimal policies. This is essential to ensure our method can accommodate various distributional preferences and adapt to different problem settings. To achieve this, we propose (1) Non-decreasing Neural Network for parameterizing a diverse set of non-linear utility functions (2) an objective function of minimum distance which encourages generating a diverse set of utility functions.

Non-decreasing Neural Network. We employ non-decreasing neural network to parameterize the utility function. The use of a neural network allows us to represent complex, non-linear, and arbitrary continuous utility functions, while the non-decreasing constraint ensures that the utility function satisfies the desired properties for multi-objective problems. We ensure the non-decreasing property in neural networks by constraining the weight matrix to be non-negative and the activation function to be non-decreasing following existing work in convex neural networks [41] and QMIX [42], which can approximate any multivariate non-decreasing function with arbitrary small errors [43]. The implementation details of the Non-decreasing Neural Network are provided in Appendix B.

Diversity-based Objective Function. We propose an objective function based on diversity for learning a diverse set of plausible utility functions. Specifically, we define f θ 1 , f θ 2 , · · · , f θ M to be the set of candidate utility functions, parameterized by θ 1 , · · · , θ M . For any given utility function f θ i , the learning objective is defined as

̸

̸

Optimizing this objective function can encourage diversity among the generated utility functions within the normalized range [0 , 1] K in both value and derivative, leading to more comprehensive coverage of potential user preferences. We minimize the objective function by gradient descent on non-decreasing neural networks to generate a set of N utility functions. To make the learned utility function strictly increasing for satisfying the constraints in Theorem 1, we add an additional term to the learned utility function as ˜ f θ i ( z ) = f θ i ( z ) + α K ∑ K k =1 z k in our experiment with α = 0 . 01 , and use ˜ f θ i ( z ) , i = 1 , · · · , M as the strictly increasing utility function to learn DPO policies.

4.2.2 Optimizing Policy with Utility-based Reinforcement Learning

Once we have generated a diverse set of utility functions, the second component of our algorithm focuses on optimizing policies to maximize the expected utility. This process, which we call utilitybased RL, leverages the generated utility functions to guide the optimization of policies. By focusing on the expected utility, our method can efficiently balance the trade-offs between multiple objectives and distributions, ultimately generating policies that are more likely to align with user preferences.

We show that the following utility-based reinforcement learning algorithm can effectively optimize the policy with respect to a given utility function.

Algorithm 1 Utility-based Reinforcement Learning

Input: policy π , an environment M = ( S, A, P, P 0 , R , γ, T ) , utility function f Output: new policy π ′

1: Augment state space with ˜ S = S × Z , where Z is the space of cumulative multivariate returns.

2: Let transition ˜ P 0 ( · ) and ˜ P ( ·| ( s t , z t ) , a t ) with s 0 ∼ P ( s 0 ) , z 0 = 0 , and s t +1 ∼ P ( ·| s t , a t ) , z t +1 = z t + γ t r t .

3: Let scalar reward function R as R (( s t , z t ) , a t , ( s t +1 , z t +1 )) = γ -t [ f ( z t +1 ) -f ( z t )] .

4: Optimize policy π under environment ˜ M = ( ˜ S, A, ˜ P, ˜ P 0 , R, γ, T ) under off-the-shelf RL algorithm (such as PPO or SAC) to π ′ .

Briefly, Algorithm 1 augments the state space with the cumulative multi-objective returns, and transforms the multi-dimensional rewards into a scalar reward by the difference in the utility function f . The following result shows that the new scalar-reward environment ˜ M generated by Algorithm 1 has the same optimal policy as the optimal policy under utility function f :

Theorem 2. The optimal policy π ∗ under environment ˜ M = ( ˜ S, A, ˜ P, ˜ P 0 , R, γ, T ) , with scalar reward function

is the optimal policy under the utility function f , i.e.

An advantage of Algorithm 1 is that we can directly utilize off-the-shelf RL algorithms, such as PPO [44] and SAC [45] without any modification, which makes the algorithm easy to implement using widespread existing implementations of online RL algorithms.

It is also important to note that our algorithm simplifies to optimizing the weighted sum of rewards in MORL when the utility function is linear. This implies that our method is a generalization of the linear utility function MORL approaches by accommodating a wide range of non-linear utility functions. This flexibility makes our algorithm particularly suited for problems where the user's preferences may not be adequately captured by a linear utility function.

In the next experimental section, DPMORL only undergoes a single iteration: we initially generate a set of N utility functions as per the methodology detailed in Section 4.2.1. Subsequently, it optimizes a set of N policies using these generated utility functions, as outlined in Section 4.2.2.

5 Experiments

In this section, we conducted several experiments under the setting of MORL. Through the experiments, we want to investigate the following questions:

Can Utility-based RL effectively learn policies with diverse distributional preferences?

Can DPMORL generate a set of diverse non-linear utility functions?

Can DPMORL obtain promising performance compared with state-of-the-art MORL methods in view of expected and distributional preferences?

5.1 Case Study of Utility Functions

To answer the first question, we train policies with a diverse set of utility functions on DiverseGoal, DeepSeaTreasure and Reacher. Here, we focus on showing the effectiveness of the Utility-based RL algorithm (Algorithm 1). We select several different non-linear functions f , each in favor of distributions at one target. We train policy π i with utility function f i with Algorithm 1 for each target, and show the policy's trajectory and return samples.

(a) The map of the DiverseGoal envi- ronment.

(b) Reward distribution within each goal position.

DiverseGoalEnvironment

6

Goal 1

Goal 0

Goal 2

4

Goal 3

Start

2

0

-2

-6

-4

-2

0

2

4

6

X

RewardDistributiononEachGoal

15

10

Reward

1

5

0

Goal o

Goal 1

Goal 2

-5

Goal 3

-10

5

0

5

10

15

20

Rewardo

DiverseGoal is a MORL environment in Figure 3 with multiple goals that the agent can take multiple steps to reach, where each goal has its unique reward function. Upon reaching a particular goal, the agent secures a 2D reward. This reward is sampled from the specific normal distribution associated with that goal, as illustrated in Figure 3b. Conversely, if the agent reaches the boundary of the map, it incurs a negative 2-dimensional reward. The environment requires the agent to navigate the trade-offs between various reward distributions, underscoring the complexity and nuance inherent to the task.

In this study, we investigate if the learned policy can reach the goal that yields the highest expected utility under the utility function. The results are illustrated in Figure 4. Under different types of utility functions and return distributions, the utility-based RL algorithm is able to find the optimal policy with the highest expected utility, which shows the effectiveness of the utility-based RL algorithm.

DeepSeaTreasure and Reacher are among the most widely used MORL environments. Under the DeepSeaTreasure environment and Reacher environment, we show the return distributions of the policies optimized with utility-based RL under different utility functions. In Figure 5 and Figure 6, we use utility functions generated by DPMORL in Section 4.2.1 and a set of manually constructed utility functions for different types of distributional preferences respectively. From the result in Figure 5 and Figure 6, we can see that our algorithm effectively finds policies that maximize the expected utility for different non-linear utility functions. Meanwhile, the DeepSeaTreasure environment aligns well

(b) Return distribution of policy to maximize expected utility for each utility function.

20

15

Utility Function fo

11.45

10.18

20

15

Utility Functionf

10.18

11.45

15

20

Utility Function f2

0.9408

0.8371

20

15

Utility Function f

0.9408

0.8371

20

15

UtilityFunction f4

0.9408

0.8371

20

15

Utility Functionfs

0.9408

0.8371

10

8.90

7.63

10

8.90

7.63

10

0.7334

0.6297

10

0.7334

0.6297

10

0.7334

0.6297

10

0.7334

0.6297

5

6.36

5.08

3.81

0

6.36

5.08

3.81

0

0.5259

0.4222

0.3185

0

5

0.5259

0.4222

0.3185

。

0.5259

0.4222

0.3185

0.5259

0.4222

0.3185

-10

-5

-10

0

10

20

2.54

1.26

-0.01

5

0

10

20

2.54

1.26

-0.01

-10

-5

-10

0

10

20

0.2147

0.1110

0.0073

-10

-5

10

0

10

20

0.2147

0.1110

0.0073

-5

0

10

20

0.2147

0.1110

0.0073

-10

-5

-10

0

10

20

0.2147

0.1110

0.0073

20

Return Distribution ofng

20

Return Distribution ofπ

20

Return Distribution ofn2

20

Return Distribution ofn3

20

Return Distribution ofn4

20

Return Distribution ofns

15

15

15

15

15

15

N

10

0

5

N

Return

10

5

0

N

Return

10

5

0

N

10

0

N

Return

10

u

0

N

10

5

0

-5

-5

-5

-10

10

0

Return 1

10

20

-10

10

Q

Return 1

10

20

-10

10

0

Return 1

10

20

-10

10

0

Return 1

10

20

-10

10

0

Return 1

10

20

-10

10

0

Return 1

10

20

Trajectory ofPolicyo

TrajectoryofPolicyn

TrajectoryofPolicyn2

TrajectoryofPolicy3

Trajectory ofPolicyn4

TrajectoryofPolicyI5

5

6ael2

1

Seet

Geal 1

Coal 3

SEarti

Un

Goal

5

Mart

5

Goal 1

Soar

Gsal 2

Gxl 3

4ar1

Q

0

Q

0

Q

0

0

5

5

0

5

0

5

0

5

0

0

5

X

X

X

X

X

Utilityfunctionfo

Utilityfunctionf

Utilityfunctionf2

Utility functionf3

Utilityfunctionf4

Utilityfunctionfs

Utilityfunctionf6

56.18

-20

20

-20

20

20

20

20

49.94

43.70

40

40

40

-40

40

40

40

37.46

31.22

60

60

60

-60

-60

60

-60

24.98

18.74

80

80

80

-80

-80

80

80

12.49

6.25

-100

0

10

20

-100

Q

10

20

-100

Q

10

20

-100

10

20

-100

Q

10

20

-100

Q

10

20

-100

Q

10

20

0.01

(a) Utility functions generated by our method in Section 4.2.1 under DeepSeaTreasure environment.

Return Distribution of ng

Return Distribution of n

Return Distribution of n2

Return Distribution of n3

Return Distribution of n4

Return Distribution of nts

Return Distribution of ne

20

20

20

20

20

20

20

Return

N

40

N

40

N

Return

40

N

E

40

N

40

N

Return

40

N

Return

40

60

60

60

09-

60

60

60

80

80

80

80

80

80

-80

-100

0

5

Return 1

10

15

20

-100

5

Return 1

10

15

20

-100

0

5

Retum 1

10

15

20

-100

0

Return 1

10

15

20

-100

0

5

Retum 1

10

15

20

-100

0

5

Return 1

10

15

20

-100

0

5

Return 1

10

15

20

(b) Return distribution of policy learned by DPMORL to maximize expected utility for each utility function.

(a) Utility functions with different distributional preferences under Reacher environment.

Utilityfunctionfo

Utilityfunctionf

Utilityfunctionf2

Utilityfunctionf3

Utilityfunctionf4

Utilityfunctionfs

Utilityfunctionf6

0.0

75

75

75

75

75

75

75

12.4

24.9

50

50

50

50

50

50

50

49.7

25

25

25

25

25

25

25

62.2

74.6

87.0

0

0

0

0

0

S'66

111.9

50

50

50

50

50

50

50

PolicyIo

PolicyI1

PolicyI2

PolicyI3

PolicyT4

Policys

PolicyI6

75

75

75

75

75

75

75

50

50

50

50

50

50

50

25

25

25

25

25

25

25

0

0

0

0

0

0

0

0

25

50

75

0

25

50

75

0

25

50

75

0

25

50

75

0

25

50

75

0

25

50

75

0

25

50

75

(b) Return distribution of policy learned by DPMORL to maximize expected utility for each utility function.

with safe RL settings, where the two returns are time penalty and task reward respectively. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint.

5.2 Main Experiment

In this section, we illustrate the performance of DPMORL on five environments based on MOGymnasium [46] to answer the latter two questions.

Environments. We conducted experiments across five environments based on MO-Gymnasium [46] to evaluate the performance of our proposed method, DPMORL. These environments represent a diverse range of tasks, from simple toy problems to more complex continuous control tasks, and cover various aspects of multi-objective reinforcement learning:

DeepSeaTreasure: A classic MORL benchmark that requires exploration in a gridworld to find treasures with different values and depths.

FruitTree: A multi-objective variant of the classic gridworld problem, where an agent has to collect different types of fruits with varying rewards and penalties.

HalCheetah, Hopper, MountainCar: Three continuous control tasks that require controlling different agents for task solving and minimizing energy usage.

More details about the environments are gathered in Appendix C.1.

Baselines. We compare DPMORL with four state-of-the-art baselines in the context of distributional preferences of MORL: Optimistic Linear Support ( OLS ) [47, 16]; Prediction-Guided Multi-Objective Reinforcement Learning ( PGMORL ) [8]; Generalized Policy Improvement with Linear Support ( GPI-LS ) [7]; Generalized Policy Improvement with Prioritized Dyna ( GPI-PD ) [7].

Training Details. For all methods, including DPMORL and the baselines, each policy was trained for 1 × 10 7 steps. We learn a set of N = 20 policies for DPMORL and all of the baselines to ensure a fair comparison. Finally, we use the learned N = 20 policies in each method for evaluations.

1.00

UtilityFunctionf

1.00

UtilityFunctionf2

1.00

UtilityFunctionf3

1.00

UtilityFunctionf4

1.00

UtilityFunctionfs

1.00

UtilityFunctionf6

1.00

UtilityFunctionf

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.9408

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.0%.0

00.08.0

0.8371

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

1.00

UtilityFunctionfg

1.00

UtilityFunctionfg

1.00

0.7334

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.6297

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.08.0

0.5

1.00.08.0

0.5

1.0

.08.0

0.5

1.0

00.0.0

0.5

1.0

00.08.0

0.5

1.0

00.08.0

0.5

1.00.00.0

0.5

1.00.08.0

0.5

1.0

0.5259

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.4222

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.3185

0.08.0

1.00.0.0

.08.0

00.08.0

0.08.0

0 0.08.0

0 0.08.0

0.5

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.2147

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.75

0.1110

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.08.0

0.08.0

0.0073

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

0.5

1.0

Implementation Details. We use PPO algorithms implemented in Stable Baselines 3 [48] in Algorithm 1. We use a normalization technique by linearly mapping the return into scale [0 , 1] K without modifying the optimal policies. Detailed implementations are provided in Appendix B.

Evaluation Metrics. To thoroughly evaluate the performance of DPMORL and compare it with the baseline methods, we employed four distinct metrics. These comprise two existing MORL metrics, HyperVolume and Expected Utility, in addition to two novel metrics developed specifically for assessing distributional preference, i.e., Constraint Satisfaction and Variance Objective. The latter two are designed to underscore the optimality of multivariate distributions associated with the learned policies. In terms of Constraint Satisfaction, we randomly generate M = 100 constraints for the policy set produced. The Constraint Satisfaction metric is then computed by considering the highest probability within the policy set that satisfies each individual constraint. The Variance Objective metric, on the other hand, involves generating M = 100 random linear weights. These weights are applied to both the expected returns and the standard deviations of returns in each dimension. This objective encourages attaining greater expected returns while simultaneously reducing variance, thereby catering to dynamic preferences. Further details about the implementation of these evaluation metrics are provided in Appendix C.2 and Appendix C.3.

5.3 Results

Generation of Utility Functions. In accordance with the methodology detailed in Section 4.2.1, we employ non-decreasing neural networks in conjunction with diversity-focused objectives to generate a diverse assortment of utility functions. The generated functions are visually represented in Figure 7. The outcomes clearly indicate that optimizing diversity-based objective functions allows for generating a broad range of non-linear utility functions, thereby encompassing an expansive array of preferences with respect to returns. Subsequently, we utilize the first N = 20 utility functions depicted in Figure 7 to train an equivalent number of policies under DPMORL.

Standard MORL Metrics. The results under standard MORL metrics are shown in Table 1. Focusing on the Expected Utility, the performance of DPMORL is the highest in the Hopper, HalfCheetah, MountainCar, and DeepSeaTreasure environments (4/5), indicating that our method outperforms others in terms of expected utility. While for FruitTree environment, DPMORL also obtains consistent performance with the best contender, GPI-LS. In terms of the HyperVolume, DPMORL also performs the best in the Hopper, HalfCheetah, MountainCar, and FruitTree environments (4/5). The results show that DPMORL can yield better performance across both metrics and most environments (Hopper, HalfCheetah, MountainCar, and DeepSeaTreasure), meanwhile delivering robust and consistent results on the other one (FruitTree). Since PGMORL works on continuous action spaces, here we omit the results of PGMORL on environments with discrete action space, DeepSeaTreasure and FruitTree.

Distributional MORL Metrics. The results under distributional metrics are shown in Table 2. DPMORL outperforms other methods in most environments on both Constraint Satisfaction (5/5) and Variance Objective (3/5), indicating its strong ability to handle the distributional multi-objective reinforcement learning problem. For the rest environments, DPMORL also obtained comparable results compared with the best contender, GPI-LS. The results show that the policies learned by

DPMORL have a higher probability of satisfying randomly generated constraints, and can better balance the trade-off between expectations and variances.

Return Distributions of Learned Policies by DPMORL. We provide visualizations of the multidimensional return distributions of each policy learned by DPMORL and four baseline methods (PGMORL, OLS, GPI-LS and GPI-PD) on different MORL environments, as shown in Appendix C.4. DPMORL's learned policy set demonstrates higher diversity with respect to return distributions compared to baseline methods.

DPMORL on More Than 2-dimensional Return Space. We run experiments with 3 objectives under Hopper and FruitTree environment, and show the learned return distribution and standard MORL evaluation metrics in Appendix C.5. The results demonstrate that DPMORL can effectively learn DPO policies under more than 3-dimensional reward functions, and achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.

Ablation Analysis with Variable Number of Policies. We provide the ablation studies of DPMORL with a varying amount of learned policies N in Appendix C.6. As the number of policies increases, both expected utility and hypervolume increase, which means DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only 5 policies, which verify the effectiveness of DPMORL in all environments.

6 Conclusion

In this work, we initialized the study of distributional MORL, specifically when preferences over different objectives and their return distributions are uncertain. We introduced the concept of Distributional Pareto-Optimal (DPO) policies with rigorous theoretical analysis, which extend the notion of Pareto-optimality in MORL to include preferences over the entire distribution of returns, not just their expected values. To obtain such desirable policies, we proposed a new algorithm, DPMORL, designed to learn DPO policies with non-linear utility functions. DPMORL allows for expressing a wide range of distributional preferences, providing a flexible and expressive approach to MORL. Experiment results showed that DPMORL consistently outperformed existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences.

Acknowledgments

This research was supported by the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108; and Microsoft Research Asia D-CORE program. The authors would like to thank Soichiro Nishimori, Kazuki Ota, and the anonymous reviewers for their insightful comments and suggestions.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 10961-10970. PMLR, 2021.

Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptivelyselected demonstrations. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 3155-3161. ijcai.org, 2021.

Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multiobjective sequential decision-making. Journal of Artificial Intelligence Research , 48:67-113, 2013.

Conor F Hayes, Roxana R˘ adulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems , 36(1):26, 2022.

Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. pages 10607-10616, 2020.

Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 10607-10616. PMLR, 2020.

Rainer Schlosser. Risk-sensitive control of markov decision processes: A moment-based approach with target distributions. Computers & Operations Research , 123:104997, 2020.

Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. In Advances in Neural Information Processing Systems , 2022.

Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning , pages 20423-20443. PMLR, 2022.

Mathieu Reymond, Conor F Hayes, Denis Steckelmacher, Diederik M Roijers, and Ann Nowé. Actor-critic multi-objective reinforcement learning for non-linear utility functions. Autonomous Agents and Multi-Agent Systems , 37(2):23, 2023.

Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management science , 38(4):555-593, 1992.

Moshe Shaked and J George Shanthikumar. Stochastic orders . Springer Science Business Media, 2007.

Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the american statistical association , 2020.

Lucas Nunes Alegre, Ana L. C. Bazzan, and Bruno C. da Silva. Optimistic linear support and successor features as a basis for optimal policy transfer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 394-413. PMLR, 2022.

Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 11-20. PMLR, 2019.

Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022 , pages 489-497. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.

Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. Multi-objective reinforcement learning with non-linear scalarization. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022 , pages 9-17. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.

Conor F Hayes, Mathieu Reymond, Diederik M Roijers, Enda Howley, and Patrick Mannion. Distributional monte carlo tree search for risk-aware and multi-objective reinforcement learning. In Proceedings of the 20th international conference on autonomous agents and multiagent systems , pages 1530-1532, 2021.

Conor F Hayes, Diederik M Roijers, Enda Howley, and Patrick Mannion. Multi-objective distributional value iteration. In Adaptive and Learning Agents Workshop (AAMAS 2022) , 2022.

Conor F Hayes, Timothy Verstraeten, Diederik M Roijers, Enda Howley, and Patrick Mannion. Expected scalarised returns dominance: a new solution concept for multi-objective decision making. Neural Computing and Applications , pages 1-21, 2022.

Willem Röpke, Conor F. Hayes, Patrick Mannion, Enda Howley, Ann Nowé, and Diederik M. Roijers. Distributional multi-objective decision making. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China , pages 5711-5719. ijcai.org, 2023.

Marc G Bellemare, Will Dabney, and R'emi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine LearningVolume 70 , pages 449-458. JMLR. org, 2017.

Pushi Zhang, Xiaoyu Chen, Li Zhao, Wei Xiong, Tao Qin, and Tie-Yan Liu. Distributional reinforcement learning for multi-dimensional reward functions. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1519-1529, 2021.

Will Dabney, Mark Rowland, Marc G Bellemare, and Remi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional deterministic policy gradients. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning , 49:267-290, 2002.

Wenjie Huang and William B Haskell. Stochastic approximation for risk-aware markov decision processes. IEEE Transactions on Automatic Control , 66(3):1314-1320, 2020.

Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning , 49:267-290, 2002.

Javier Garc'ia and Fernando Fern'andez. Safe exploration in markov decision processes. arXiv preprint arXiv:1205.1111 , 2012.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. arXiv preprint arXiv:1705.10528 , 2017.

Yinlam Chow, Ofir Nachum, Edgar A Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1805.08711 , 2018.

Zi-Xuan Chen, Xin-Qiang Cai, Yuan Jiang, and Zhi-Hua Zhou. Anomaly guided policy learning from imperfect demonstrations. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022 , pages 244-252.

Yunke Wang, Xiyu Wang, Anh-Dung Dinh, Bo Du, and Charles Xu. Learning to schedule in diffusion probabilistic models. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023 , pages 2478-2488. ACM, 2023.

Chunming Liu, Xin Xu, and Dewen Hu. Multi-objective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 45(3):385-398, 2014.

Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multiobjective deep reinforcement learning. arXiv preprint arXiv:1610.02707 , 2016.

Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou. Imitation learning from pixel-level demonstrations by hashreward. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors, AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021 , pages 279-287. ACM, 2021.

Xin-Qiang Cai, Yao-Xiang Ding, Zi-Xuan Chen, Yuan Jiang, Masashi Sugiyama, and Zhi-Hua Zhou. Seeing differently, acting similarly: Heterogeneously observable imitation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial imitation learning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 10262-10270. AAAI Press, 2023.

Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning , pages 146-155. PMLR, 2017.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research , 21(1):7234-7284, 2020.

Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating functional knowledge in neural networks. Journal of Machine Learning Research , 10(6), 2009.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1856-1865. PMLR, 2018.

Lucas N. Alegre, Florian Felten, El-Ghazali Talbi, Grégoire Danoy, Ann Nowé, Ana L. C. Bazzan, and Bruno C. da Silva. MO-Gym: A library of multi-objective reinforcement learning environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelearn 2022 , 2022.

Diederik M Roijers. Multi-objective decision-theoretic planning. AI Matters , 2(4):11-12, 2016.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Takashi Kamihigashi and John Stachurski. Partial stochastic dominance via optimal transport. Operations Research Letters , 48(5):584-586, 2020.

Cédric Villani et al. Optimal transport: old and new , volume 338. Springer, 2008.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

A Proof

Lemma 1. If a policy π is the only optimal policy under some non-decreasing utility function f , then it is a Distributionally Pareto-Optimal policy.

Proof. If π is the only optimal policy under non-decreasing utility function f , then for any policy policy π ′ ,

From Definition 1 and Definition 2, if any policy π ′ stochastically dominates π , then for any utility function f , E z ∼ µ ( π ′ ) ≥ E z ∼ µ ( π ) f ( z ) , which makes a contradition. Therefore, π is not stochastically dominated by any other policies, and π is a Distributional Pareto-Optimal policy.

̸

Lemma 2. If a policy π is one of the optimal policy under some strictly increasing utility function f , then it is a Distributionally Pareto-Optimal policy. The utility function f is strictly increasing if and only if x ⪰ y and x = y ⇒ f ( x ) > f ( y ) .

Proof. Following the Equation (2) and (3) of [49] which is derived from Kantorovich duality (Theorem 5.10 in [50]), we have

̸

µ 1 ≻ SD µ 2 if and only if µ 1 = µ 2 and max γ ∈ Π( µ 1 , µ 2 ) E ( x,y ) ∼ γ [ ✶ x ⪰ y ] = 1 (5) where Π( µ 1 , µ 2 ) is the set of all couplings of µ 1 and µ 2 , which is the set of distributions γ ( x, y ) such that

Suppose there exists a policy π ′ stochastically dominate π , then there exists γ ( x, y ) ∈ Π( µ ( π ′ ) , µ ( π )) , such that E ( x,y ) ∼ γ [ ✶ x ⪰ y ] = 1 . Since π is one of the optimal policy under strictly increasing utility function f , we have E x ∼ µ ( π ′ ) f ( x ) -E y ∼ µ ( π ) f ( y ) ≤ 0 ∫ f ( x ) µ ( π ′ )( x ) dx -∫ f ( y ) µ ( π )( y ) dy ≤ 0 ∫∫ [ f ( x ) -f ( y )] γ ( x, y ) dxdy ≤ 0

Since E ( x,y ) ∼ γ [ ✶ x ⪰ y ] = 1 , i.e. ∫∫ x ⪰ y γ ( x, y ) dxdy = 1 , we have and

There are two cases. Case 1: there exists c ≻ 0 such that ∫∫ γ ( x, y ) dxdy > 0 .

Since f x ⪰ y + c is an strictly increasing function, we have

̸

Then, we have:

Since c ≻ 0 , f ( y + c ) -f ( y ) > 0 . Since ∫∫ x ⪰ y + c γ ( x, y ) > 0 and γ ( x, y ) is continuous almost everywhere, we have ∫∫ x ⪰ y + c [ f ( y + c ) -f ( y )] γ ( x, y ) dxdy > 0 , which yields a contradiction.

̸

Therefore, we have proved that there does not exist π ′ that stochastically dominate π , when π has optimal expected utility under some strictly increasing utility function f .

Case 2: there does not c ≻ 0 such that ∫∫ x ⪰ y + c γ ( x, y ) dxdy > 0 . Then, we have E ( x,y ) ∼ γ ( x,y ) ✶ [ x = y ] = 1 , which contradicts with the assuption that µ ( π ′ ) = µ ( π ) .

Lemma 3. Any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.

Proof. We first show that if distribution µ ′ stochastically dominates µ , then the expectation E z ∼ µ ′ [ z ] dominates E z ∼ µ [ z ] . Following Kantorovich duality, there exists a coupling γ of µ ′ and µ that E ( x,y ) ∼ γ [ ✶ x ⪰ y ] = 1 . Then,

When E ( x,y ) ∼ γ [ x -y ] = 0 , it requires that x = y holds where γ ( x, y ) > 0 , which makes µ ′ = µ that contradicts with the fact that µ ′ ≻ SD µ . Therefore, the expectation E z ∼ µ ′ [ z ] dominates the expectation E z ∼ µ [ z ] . It follows that if the expectation E z ∼ µ [ z ] is not dominated by the expectation E z ∼ µ ′ [ z ] , then the distribution µ is not stochastically dominated by µ ′ .

If policy π is Pareto-optimal, then its multidimensional expected returns E [ Z ( π )] are not dominated by E [ Z ( π ′ )] of any other policies π ′ . Then, the return distribution µ ( π ) of policy π is not stochastically dominated by the return distribution µ ( π ′ ) of any other policies π ′ . Therefore, policy π is a Distributionally Pareto-Optimal policy. To conclude, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.

Theorem 1. If a policy π has optimal expected utility under some non-decreasing utility function f , where either

utility function f is strictly increasing

π is the only optimal policy with greatest expected utility under utility function f

then π is a Distributionally Pareto-Optimal policy. Also, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.

Proof. The theorem directly follows from Lemma 1, Lemma 2 and Lemma 3.

The following theorem proves that the optimal policy over Value at Risk (VaR) and over Conditional Value at Risk (CVaR) belong to the set of Distributionally Pareto-Optimal policies. Here, we use the random variable µ ( π ) to represent the negative risk, where smaller value represents larger risks, in order to be aligned with the definition of Pareto-Optimality.

Definition 4. Given a distribution µ , the Value at Risk (VaR) is defined as

Theorem 3. When K = 1 , there exists policy with optimal Value at Risk (VaR), and policy with optimal Condition Value at Risk (CVaR), which belongs to the set of Distributional Pareto-Optimal Policies.

Proof. We let the policy set Π VaR as the policy set with optimal value at risk, which z 0 = VaR α ( µ ( π )) for any π ∈ Π VaR. We let Π ′ VaR to be the policy set with non-optimal policy, with VaR α ( µ ( π ′ )) < z 0 for any π ′ ∈ Π ′ VaR . Suppose all Distributionally Pareto-Optimal policy belongs to Π ′ VaR . We define a

for any π ∈ Π VaR, while E z ∼ µ ( π ′ ) f ( z ) < α for any π ′ ∈ Π ′ VaR . Therefore, any policy π ∈ Π VaR is not stochastically dominated by any policy π ′ ∈ Π ′ VaR . Therefore, there must exists a Distributional Pareto-Optimal Policy in Π , which results in a contradiction. Therefore, there must exists a policy with optimal value at risk, which is Distributionally Pareto-Optimal.

Similarly to the case in VaR, we let the policy set Π CVaR as the policy set with optimal conditional value at risk, which z 0 = CVaR α ( µ ( π )) for any π ∈ Π CVaR. We let Π ′ CVaR to be the policy set with non-optimal CVaR, with CVaR α ( µ ( π ′ )) < z 0 for any π ′ ∈ Π ′ CVaR . Then we can prove that any policy Π ′ CVaR does not stocastically dominate the policy in Π CVaR. Suppose the opposite case where π ∈ Π CVaR is dominated by π ′ ∈ Π ′ CVaR . We let f ( z ) , f ′ ( z ) to be the cumulative distribution function of µ ( π ) and µ ( π ′ ) , and f ( z ) ≥ f ′ ( z ) must be satisfied to make π ≺ SD π ′ . Therefore,

which results in a contradiction. Therefore, any policy Π ′ CVaR does not stocastically dominate the policy in Π CVaR. We thus proved that there exists a Distributionally Pareto-Optimal policy in Π CVaR, which is an optimal policy under conditional value at risk.

B Implementation Details

Non-Decreasing Neural Network. We used Non-Decreasing Neural Network to parameterize the utility function. The network architecture consisted of three fully connected layers, with our designed non-decreasing activation activation functions applied to the hidden layers:

This design of activation function is an extension to ReLU activation function, which empirically make the learned utility function more diverse, covering more types of decision boundaries. After each gradient update during training, we clip all the trainable weights in the network to be non-negative to ensure that the resulting utility function was non-decreasing. The network's input dimension corresponded to the number of objectives in the environment, and the output dimension was a scalar representing the utility value.

Return Normalization. We use a normalization technique by linearly mapping the return into scale [0 , 1] K without modifying the optimal policies. Specifically, we keep track of the minimum and maximum multivariate returns that the policies have currently achieved, denote by z min and z max . We let z mid = ( z min + z max ) / 2 , and d = max i [ z max,i -z min,i ] . Before the any multivariate return z is fed into the utility function, we normalize it by

This maps z linearly to z norm , where z norm ∈ [0 , 1] K , without affecting the scale of each dimension of returns. In Algorithm 1, the return z t is normalized before feeding into the utility function to compute the scalar reward.

DPMORL Algorithm. For the implementation of our DPMORL algorithm, we used the Proximal Policy Optimization (PPO) algorithm [44] as the basic RL algorithm. The learning rate was set to 3 × 10 -4 , with a discount factor γ of 0.99. For PPO, we used a clipping parameter of 0.2. The batch size was set to 256 for all environments and algorithms, with updates performed every 2,048 steps. The utility functions were trained using the Adam optimizer [51], with a learning rate of 3 × 10 -4 .

C Experimental Details

C.1 Environments

The details of the environments used in the main paper are listed below:

DeepSeaTreasure: A classic MORL benchmark that requires exploration in a gridworld to find treasures with different values and depths.

FruitTree: A multi-objective variant of the classic gridworld problem, where an agent has to collect different types of fruits with varying rewards and penalties.

HalCheetah: A continuous control task where a simulated cheetah must run and jump over obstacles while minimizing energy consumption.

Hopper: A continuous control task that involves balancing and hopping a simulated onelegged robot while minimizing energy usage.

MountainCar: A classic reinforcement learning problem, extended to include multiple objectives, such as minimizing the time to reach the goal and reducing energy consumption.

In the Appendix, we additionally add one environment:

Reacher: A discrete control task where a simulated robotic arm must reach a target while minimizing energy usage.

C.2 Standard MORL Metrics

The details of the standard MORL metrics used in the main paper are listed below:

Expected Utility is similar to R-Metrics used in Multi-Objective Optimization (MOO). This metric is concerned with the expected utility of the policies on the Pareto front for various weights. It provides a means to assess the quality of the policies derived by the algorithm in terms of their expected performance across different utility functions. A higher Expected Utility indicates that the policies offer a better trade-off between the objectives, thereby leading to more desirable outcomes under a variety of user preferences:

where f w denotes the user's utility function linearly parameterized by w , and z denotes the returns of one episode.

Hypervolume is a prominent measure used in Multi-Objective Optimization (MOO) and Multi-Objective Reinforcement Learning (MORL). This metric calculates the volume in the objective space dominated by the Pareto front with respect to a reference point. The Hypervolume gives a scalar measure of the quality of a Pareto front, with higher values indicating a better Pareto front. A Pareto front with a larger Hypervolume signifies that the algorithm can yield a wider range of policies, thus dominating more of the objective space, which indicates a better exploration of the policy space:

where r is the reference point in the objective space and Leb denotes the Lebesgue measure, which in this context refers to the volume in the objective space. The reference points used in the experiment of the main paper are calculated as the minimal mean returns of each method under evaluation.

C.3 Distributional MORL Metrics

We propose two novel metrics for assessing distributional preference: Constraint Satisfaction and Variance Objective.

For Constraint Satisfaction, we define u ( p, π ) to be the probability of return random variance z ∼ Z ( π ) that satisfy the constraint p , representing how a policy satisfy the given constraint. We randomly generate each constraint defined by n linear constraints represented by { z |∀ i, w T i z ≥ c i } , where w i is uniformly sampled from the surface w i ⪰ 0 , 1 T w i = 1 , and c i is sampled from the distribution of U (min z ∈Z w T i z , max z ∈Z w T i z ) , where Z is the set of all return samples of the final policies learned by DPMORL.

For Variance Objective, we define u ( p, π ) to be a weighted sum of expected return E z ∼ µ ( π ) z and standard deviation of return √ Var z ∼ µ ( π ) [ z ] . We randomly generate w i ∈ R 2 K , where w i ⪰ 0 and

MountainCar

Reacher

DeepSeaTreasure

0.0

0.0

·.

30.0

A

2

-2.5

220.0

-10.0

Return

Return

Return

-5.0

10.0

-20.0-

-7.5

0.0

-30.0

-10.0

-1000 -750 -500 -250

0

10

20

30

40

0

10

20

Return 1

Return 1

Return 1

Hopper

HalfCheetah

FruitTree

4000.0

0.0

3000.0

-100.0

6.0

2

2

Return

Return

-200.0

Return

2000.0

4.0

-300.0

1000.0

2.0-

-400.0

1000

2000

3000

0

1000200030004000

2

4

6

8

10

Return 1

Return 1

Return 1

TL2

T14

·

T6

IT8

T10

T12

T14

I16

▼

I18

IT1

TT3

·

IT5

·

Tlg

A

T11

▼

I13

▼

▼

I17

▼

I19

1 T w i = 1 . We let w i = [ w 1 i , w 2 i ] , where w 1 i , w 2 i ∈ R K , and use w 1 i and w 2 i as the weight for expected return and standard derivation of return respectively.

For both of the metric, we define u ( p, π ) to be the satisfaction score of policy π under preference p . We generate M = 100 number of preference p 1 , p 2 , · · · , p M . For both of the Constraint Satisfaction and Variance Objective metric, the mean score is defined as

The mean score in the above equation measures the best performance of satisfying the preference of the user, averaged in M = 100 preferences. Higher mean score represents better ability of meeting different preferences. In Table 2 of the main experiment, all of the numbers are calculated with the above equation.

C.4 Return Distributions of Learned Policies by DPMORL and Baseline Methods

We demonstrate the return distribution of the learned policies by DPMORL in Figure 8, which demonstrates that DPMORL is able to learn a set of policies with optimal return distributions. We also visualize the return distribution of policies learned by four baseline methods, which is shown in Figure 9. DPMORL's learned policy set has higher diversity with respect to return distributions compared to baseline methods.

C.5 DPMORL on More Than 2-Dimensional Return Space

We run experiments with 3 objectives under Hopper and FruitTree environment, where we use the first three dimension of the reward in the MO-Hopper and Fruit-Tree environment the MO-Gymnasium. We show the 3-dimensional return distribution of the set of policies learned by DPMORL and baseline methods in Figure 10. The results demonstrate that DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Also, DPMORL learns better 3-dimensional return distributions than GPI-PD, a competitive MORL baseline method. We also show quantitative evaluation metrics under Hopper and FruitTree environments with 3 objectives in Table 3. The results demonstrate that DPMORL achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.

(a)

(b)

(c)

(d)

MountainCar

Hopper

HalfCheetah

0.0

200.0

-40.0

2

-5.0

2

2

eturn

eturn

100.0

eturn

-60.0

-10.0

-80.0

-15.0

0.0

-100.0

-1000-800-600

-400

100

200

300

400

400

500

600

Return 1

Return 1

Return 1

Ilo

I2

T4

Il6

Tl8

10

12

I14

T

I16

I18

I1

3

It5

Ilg

T

11

I13

T

I17

I19

MountainCar

Reacher

DeepSeaTreasure

-10.0

-2.0

Return

N -10.0

Return

2

20.0

Return

2

-4.0

-10.0

0.0

-6.0

-10.0

-20.0

-8.0

-940

-935-930-925

0

10

20

30

0

5

10

15

Return 1

Return 1

Return 1

Hopper

HalfCheetah

FruitTree

140.0

-260.0

Return

2

130.0

120.0

2

Return

-280.0

Return

2

6.0

4.0

110.0

-300.0

2.0

150

175

200

225

500

1000

1500

2

4

6

Return 1

Return 1

Return 1

MountainCar

Reacher

DeepSeaTreasure

40.0

-2.5

Return

2

-5.0

Return

2

20.0

Return

2

-5.0

-7.5

-10.0

0.0

-10.0

-1000

-500

0

20

40

0

10

20

Return 1

Return 1

Return 1

Hopper

HalfCheetah

FruitTree

2200.0

-200.0

8.0

Return

2

2000.0

1800.0

eturn

2

-300.0

-250.0

Return

2

6.0

4.0

R

1600.0

-350.0

1400.0

-400.0

2.0

0.0

2000

2500

3000

0

2000

4000

0

5

10

Return 1

Return 1

Return 1

MountainCar

Reacher

DeepSeaTreasure

40.0

-2.0

-0.8

30.0

Return

2

Return

2

20.0

Return

2

-4.0

-1.0

-6.0

10.0

-8.0

-1.2

0.0

-100

-80

-60

0

10

20

30

0

5

10

15

Return 1

Return 1

Return 1

Hopper

HalfCheetah

FruitTree

-300.0

1500.0

3.4

2

Return

1000.0

Return

2

-350.0

Return

2

3.2

500.0

-400.0

3.0

0.0

400

600

800

1000

2000

2500

3000

1

2

3

4

Return 1

Return 1

Return 1

Hopper (DPMORL)

FruitTree (DPMORL)

Hopper (GPI-PD)

FruitTree (GPI-PD)

800

10

0

1000

Return 1

2000

3000

4000

1000

2000

3000

/4000

200

400

200

800

400

600

0

m

Ret

2

Return1

4

6

8

100

2

Return

4

6

8

10

10

0

2

6

4

Return

m

0

1000

Return 1

2000

3000

400

1000

2000

Return

3000

4000

200

400

400

200

600

0

m

0

2

Return 1

4

6

8

100

2

Returmn 2

4

6

8

10

0

2

4

8

6

m

Ilo

ft4

Ils

Il6

I7

Ilg

If9

fl11

Rlo

R2

Il4

Ils

Il6

fl6

Ilg

10

I↓1

R1

Rl}

C.6 Ablation Analysis with Variable Number of Policies

We also provide the ablation studies of DPMORL with a varying amount of learned policies N , as shown in Table 4. We can see that as the number of policies increases, both expected utility and hypervolume increase, which indicates that DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only 5 policies, which verify the effectiveness of DPMORL in all environments.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach for solving Multi-Objective Reinforcement Learning (MORL) problems. Traditional MORL methods aim at optimizing multiple objectives, but generally focus on the expected values of returns, which can be inadequate in real-world scenarios with diverse preferences over returns. The paper addresses this limitation by extending Pareto-optimality to include distributional preferences, called Distributional Pareto-Optimality (DPO). The authors also include discussions of the relationship between Pareto-optimality policies and Distributional Pareto-Optimality policies. Furthermore, the authors propose a novel algorithm, DPMORL, that learns policies considering both the return distributions and their expectations. It captures the optimality of multivariate distributions through stochastic dominance. Through experiments on several benchmarks, DPMORL was found to be effective in learning distributional Pareto-optimal policies and outperformed existing MORL methods.\n\nSTRENGTHS:\nThe paper astutely addresses a significant perspective in current MORL research by emphasizing the importance of not only accounting for the expected values of different objectives but also examining the distributional properties of returns in the context of the heterogeneity and diversity inherent in users' preferences. DPMORL stands out for its remarkable flexibility, as it adeptly accommodates a wide spectrum of distributional preferences, making it highly versatile and well-suited for an array of problem scenarios. This attributes to a more nuanced and expressive approach to MORL. Moreover, the algorithm is anchored in a robust theoretical foundation. The paper lucidly delineates Distributional Pareto-Optimal policies and astutely establishes their interrelation with utility functions. This amalgamation of theoretical rigor and empirical validation accentuates the potential of DPMORL to tackle distributional MORL challenges with efficacy. Furthermore, the authors deserve commendation for the inclusion of an intuitive and easily comprehensible case study that effectively demonstrates the utility function of learning.\n\nWEAKNESSES:\nThe figures of illustration of 2D utility functions, including the ones in Appendix, are arranged somewhat randomly. It is suggested that the authors arrange the results of the utility function according to some gradual pattern.\n\nQUESTIONS:\nI would appreciate further clarification regarding Table 2. Specifically, what are the constraints being referenced in this table? Could you provide illustrative examples to enhance understanding?\n\nLIMITATIONS:\nIt would be better to delve deeper into the distribution properties of preferences/constraints and to illuminate the limitations therein with greater specificity. Including illustrative examples of such constraints would not only bolster comprehension but also offer valuable insights into the practical implications and considerations of employing DPMORL in real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose extending multi-objective RL (MORL) to policies that are Pareto optimal over distributions of non-negative utility functions. Specifically, they define distributional Pareto optimal (DPO) policies as those whose expected returns are non-dominated for any non-negative utility function. They then propose learning a set of DPO policies by first learning a set of diverse non-negative utility functions over $[0 ~ 1]^K$ bounded objectives, then learning their corresponding optimal policies using PPO. They demonstrate in several experiments that their approach outperforms prior works in both hyper-volume and expected utility of the learned policy set.\n\nSTRENGTHS:\n- The paper is very well written, clearly presented, and tackles the very significant problem of finding policies that are distributionally Pareto-optimal in MORL.\n\n- I like the paper's formal treatment of Distributional MORL. The definitions and theorems given are particularly clear and precise, showing that the proposed framework is sound (not including how to actually find DPO policies, and not saying I agree with said definitions).\n\n- The proposed approach for finding DPO policies is particularly interesting. By learning a set of utility functions using a diversity metric, then converting the learned utilities to a reward function, one can then use any off-the-shelf RL algorithm to learn the DPO policies. The paper then shows that these optimal policies are guaranteed to maximise their respective utilities.\n\n- The paper conducts several experiments demonstrating that the learned utility functions are indeed diverse and non-decreasing. The paper also demonstrates that their approach outperforms prior works in several functional approximation domains.\n\nWEAKNESSES:\nMAJOR:\n\n- Theorem 1 is stated very differently in the Appendix (lines 33-38) compared to the main paper (lines 153-155). In particular, the one in the Appendix adds 2 more assumptions (lines 35-36). This makes the one in the paper severely misleading. This also decreases the strength of this theorem since there almost never is only a single optimal policy in RL (optimal policies are rarely unique), and strictly increasing utility functions further limits the applicability of the proposed framework.\n\n- The paper limits the definition of distributional MORL to non-decreasing utility functions, and the proposed algorithm for finding DPO policies is limited to non-decreasing utilities with objectives bounded by $[0 ~ 1]^K$. While this still covers a wide range of utilities, this severely limits the applicability of the proposed approach.\n\n- No proof is given to show what percentage volume of the Pareto front is covered by the set of optimal policies obtained by the proposed approach. In fact, there doesn't seem to be a reason why we should expect the specific diversity metric given in equation (3) to give us utility functions that lead to DPO policies covering any significant fraction of the Pareto frontier. This is a major missing component of the paper since its operational goal is to extend MORL to DPO policies.\n\n- [1] seems closely related to this paper as it also proposes a distributional view of MORL, and its implementation is not constrained to non-decreasing utilities over objectives bounded by $[0 ~ 1]^K$. However, this paper does not include it in its baselines, nor does it discuss or even cite it.\n\n\nMINOR\n\n- f(z) is missing on line 7 of the proof of Lemma 1\n\n\n[1] Abdolmaleki, Abbas, et al. \"A distributional view on multi-objective policy optimization.\" International conference on machine learning. PMLR, 2020.\n\nQUESTIONS:\nIt would be great if the authors can address the major weaknesses I outlined above. I am happy to increase my score if they are properly addressed, as I may have misunderstood pieces of paper. In addition to those:\n\n- Why is the set of utility functions restricted to non-decreasing ones? Is this a fundamental limitation of the proposed framework?\n\n- Why is the given Diversity-based Objective Function expected to lead to policies that cover significant volume of the Pareto front? The authors should provide a proof of how well their approach approximates the Pareto front, or at the very least a detailed discussion of it.\n\n- How is the number of utility functions to learn ($M$) determined? Is this just an arbitrary hyperparameter? If so, how does its choice affect how well the approach approximates the Pareto front?\n\n- Why is the given definition of DPO policies to be preferred over that of [1]? Can the authors compare against MO-MPO from [1]?\n\n- In the proof of Lemma 3, the authors say \"Suppose π is a Pareto-Optimal policy, then π is the optimal policy under some linear combination of rewards with weight w ≻ 0\". Why is this true? Is the lemma maybe assume linear utility functions (if yes this needs to be made explicit)?\n\nLIMITATIONS:\nWhile there is no limitations section, I believe the authors have adequately addressed the limitations of their work sparsely throughout the paper (besides the ones I mentioned in the weaknesses section). It will be preferable if a limitations section is added to discuss the ones the authors feel are most relevant.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces distributional Pareto-optimality for multi-objective reinforcement learning. Multi-objective RL (MORL) is typically formulated as to find Pareto optimality of all objectives. \n\nTo build the ground, the paper defines stochastic dominance for multivariate distribution and then stochastic dominance for policies. Distributional Pareto-optimality can then be formulated.\n\nThe authors propose a practical algorithm for addressing the distributional MORL problem. First, a non-decreasing NN is used (as the Q network in MARL paper QMIX) as the utility function. Then a new objective function can be defined and creates a reward function for RL. Therefore, we can iteratively train the utility function and policies.\n\nExperiment in multi-objective gymnasium shows the effectiveness of the method.\n\nSTRENGTHS:\n1. The paper addresses the distributional preferences in multi-objective RL, which is an understudied problem.\n2. The paper formulates a framework and propose practical method to address the problem.\n\nWEAKNESSES:\n1. The paper is self-contained and there is no particular worth-noting weaknesses. It would be great to see if the proposed method can be applied to other more realistic environments, such as safety-aware autonomous driving. I believe driving is a decision-making task that has many objectives that should be considered.\n\nQUESTIONS:\n1. The authors list safe RL as a special case to MORL. That would great if we can apply the proposed method and other baselines to safe RL benchmark such as Safety Gym to see the performance and also compared to other safe RL baselines. This is because safe RL is a domain with huge social impact and thus working on those tasks can improve the impact of the proposed method.\n2. In the DiverseGoal environment, how we get the distribution of reward? Is it some kinds of Normal distribution predefined?\n\nLIMITATIONS:\nThe authors do not address the limitation at all.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn Multi-Objective Reinforcement Learning (MORL), the goal is to learn Pareto optimal policies that achieve a balance among multiple conflicting objectives while considering the user's preferences. Existing MORL methods optimize the Pareto frontier by using a single value obtained from the linear combination of a multi-dimensional reward vector representing the user's preferences. In this paper, instead of the traditional approach, the authors interpret the problem from a return distribution perspective and consider the uncertainty in returns to better capture the user's preference distribution. Furthermore, they propose an algorithm that generates various utility functions, enabling the learning of Pareto optimal policies for diverse preferences.\n\nSTRENGTHS:\n- Instead of considering the traditional approach of single expected value in MORL, the paper proposes a Distributional Pareto-Optimal (DPO) method by defining the return as a distribution. This approach enables capturing complex preferences and allows for a more nuanced representation of utility functions. \n- The paper defines Stochastic Dominance (SD) and uses it to define Pareto-Optimal policies. Furthermore, it provides a mathematical proof that the pareto-optimal policy is Distributional Pareto-Optimal policy, which means that their return distributions of each policy cannot dominate that of another. \n- The author presents DPMORL algorithm that iteratively generates various non-linear, non-decreasing utility functions in cases where Stochastic Dominance does not exist and then train DPO policies with the generated utility functions. From these utility functions, proposed algorithm identifies Pareto-Optimal policies. By generating various utility functions, the algorithm can achieve to get more flexible and diverse policies set that can accommodate a wide range of preferences.\n\nWEAKNESSES:\n- The main experiment shows superior performance of the proposed algorithm compared to existing MORL methods but seems to do not fully explain the motivation(ability to capture users' complex preferences and intentions in a more detailed manner). Including additional experiments or toy examples such as diversity of utility function heatmaps between the return distribution approach and the conventional weighted sum method would substantiate the authors' motivation. \n- The proposed loss function aims to generate various utility functions, and it consists of three components: equations (1), (2), and (3). Providing explanations for each of these components would make it more readable, describing their purposes and how they contribute to creating diverse utility functions. \n- It would be beneficial to include an ablation study. For instance, in the main experiment, training 20 utility functions and policies may show differences in evaluation values with changes in the size of N. By conducting experiments with various N sizes and adding the results to the study, it could provide further support for the author's insistence. \n- Additionally, the paper proposes a method to increase the diversity of utility functions by iteratively generating them. In Appendix C Table 1, they applied N^{iter}=2 iterations to create a broader range of utility functions and trained policies based on them. However, the experimental results show that the performance does not exhibit significant differences compared to the experiments where utility functions were generated only once. Increasing N^{iter} is expected to generate a more diverse set of utility functions, enabling the learning of various Pareto-optimal policies. Consequently, this is likely to further enhance the \"Constraints Satisfaction\" score. \n- Minor typos: (1) Line 192 “We maximize” → “We minimize”,  (2) Appendix line 99 “standard derivation” → “standard deviation”\n\nQUESTIONS:\n- Can we design utility functions in a real-world setting that consider individual preferences? For instance, in the case of autonomous driving, can we provide information that highlights safety to encourage the creation of utility functions that prioritize safety significantly?\n- I wonder how the algorithm performs even in cases where the multiple objectives in the Continuous Control environment experiments are conflicting. For instance, in the case of HalfCheetah, which has two multi-objectives, \"run\" and \"jump,\" in the experiments, I wonder if the algorithm can still learn Pareto-optimal policies based on preferences when an additional conflicting objective, such as \"run backward,\" is introduced. \n- The authors conducted MORL experiments on various environments in the main experiments. It would be better if they provide more detailed explanation with some examples of what objectives in each environment and how many objectives are there. 2D graphs of Figure 5 and Appendix Figure 1 seem that the environments have two objectives each. I wonder about some results of the experiments with three or more objectives. \n- Appendix Figure 1 depicts the return distribution of the set of policies learned by DPMORL on each environment, and Return 1 and 2 seem to mean returns for two objectives. It would be helpful if authors clarify explanation of what each return means, and what preference each policy has.\n\nLIMITATIONS:\nAuthor provides the limitations of their works appropriately, and I think there is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}