SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin-Qiang Cai $^{1*}$ , Pushi Zhang $^{2*}$ , Li Zhao $^{2}$ , Jiang Bian $^{2}$ , Masashi Sugiyama $^{3,1}$  Ashley J. Llorens $^{2}$
<sup>1</sup> The University of Tokyo, Tokyo, Japan
$^{2}$  Microsoft Research Asia, Beijing, China
$^{3}$  RIKEN AIP, Tokyo, Japan
Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account for distributional preferences over the multi-variate returns, which are particularly important in real-world scenarios such as autonomous driving. To address this issue, we extend the concept of Pareto-optimality in MORL into distributional Pareto-optimality, which captures the optimality of return distributions, rather than the expectations. Our proposed method, called Distributional Pareto-Optimal Multi-Objective Reinforcement Learning (DPMORL), is capable of learning distributional Pareto-optimal policies that balance multiple objectives while considering the return uncertainty. We evaluated our method on several benchmark problems and demonstrated its effectiveness in discovering distributional Pareto-optimal policies and satisfying diverse distributional preferences compared to existing MORL methods.
Apart from most Reinforcement Learning (RL) works that consider the scalar reward of a task [1, 2, 3, 4], Multi-Objective Reinforcement Learning (MORL) has recently received extensive attention due to its adeptness at managing intricate decision-making issues with multiple conflicting objectives [5, 6, 7, 8]. In many multi-objective tasks, the relative preferences of users over different objectives are typically indeterminate a priori. Consequently, MORL's primary aim is to learn a variety of optimal policies under different preferences to approximate the Pareto frontier of optimal solutions. It has been demonstrated that MORL can significantly reduce the reliance on scalar reward design for objective combinations and dynamically adapt to the varying preferences of different users.
However, numerous real-world situations involve not only unknown relative preferences across multiple objectives, but also uncertain preferences in return distributions. These may include preference in risk [9, 10], safety conditions [11], and non-linear user's utility [12]. Consider, for instance, autonomous driving scenarios where agents need to strike a balance between safety and efficiency objectives. Different users might possess varying levels of risk tolerance. Some may demand high safety performance, tolerating lower expected efficiency, while others may seek a more balanced performance between safety and efficiency. Current MORL methods, by focusing exclusively on expected values with linear preferences, may not adequately capture multi-objective risk-sensitive preferences, hence being unable to deliver diverse policies catering to users with varied risk preferences.
In this work, we broaden the concept of Pareto-optimality in MORL to encompass distributional Pareto-optimality, which prioritizes the optimality of return distributions over mere expectations, as
*Equal contribution. Work done during an internship at Microsoft Research Asia.
(a) MORL's Pareto-front
(b) Distributional MORL's Paretofront
Figure 1: The comparison of learning targets between the traditional MORL tasks and distributional MORL tasks, in which  $f_{1}$  and  $f_{2}$  are two (conflicting) objectives.
depicted in Figure 1. From a theoretical perspective, we define Distributional Pareto-Optimal (DPO) policies, which capture the optimality of multivariate distributions through stochastic dominance [13, 14]. Distributional Pareto-Optimality delineates the set of policies with optimal return distributions, which we formally establish as an extension of Pareto-optimality in MORL.
On the practical side, we propose a novel method named Distributional Pareto-Optimal Multi-Objective Reinforcement Learning (DPMORL)<sup>2</sup>, which aims to learn a set of DPO policies. We demonstrate that a policy achieving the highest expected utility for a given strictly increasing utility function is a DPO policy. To this end, we first learn diverse non-linear utility functions and then optimize policies under them [15]. Experimental outcomes on several benchmark problems attest to DPMORL's effectiveness in optimizing policies that meet preferences on multivariate return distributions. We posit that our proposed method significantly addresses the challenge of managing multiple conflicting objectives with unknown preferences on multivariate return distributions in complex decision-making situations.
Our main contributions are listed as follows:
1. We introduce the concept of Distributional Pareto-Optimal (DPO) policies. This expands the notion of Pareto-optimality in MORL to include preferences over the entire distribution of returns, not just their expected values. This enables agents to express more nuanced preferences over policies and align their decisions more closely with their actual objectives.
2. We propose DPMORL, a new algorithm for learning DPO policies under non-linear utility functions. This algorithm accommodates a broad range of distributional preferences, thus offering a more flexible and expressive approach to MORL.
3. We execute extensive experiments on various MORL tasks, demonstrating the effectiveness of our approach in learning DPO policies. Our findings show that our algorithm consistently surpasses existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences, underscoring its practical benefits.
Multi-Objective Reinforcement Learning. MORL has emerged as a vibrant research area in the reinforcement learning community owing to its adeptness in managing intricate decision-making scenarios involving multiple conflicting objectives [5, 6]. A plethora of MORL algorithms have been put forth in the literature. For instance, [16] proposed the utilization of Generalized Policy Improvement (GPI) to infer a set of policies, employing Optimistic Linear Support (OLS) for dynamic reward weight exploration [17]. This GPI-based learning process was further enhanced by [7] through the incorporation of a world model to augment sample efficiency. [8] proposed an evolutionary approach to learn policies with diverse weights. However, our proposed approach diverges from these by leveraging utility functions to guide policy learning [18, 19]. The utility-based paradigm [6] accentuates the user's utility in decision-making problems, capitalizing on known information about the user's utility function and permissible policy types. Such methods scalarize the multidimensional objectives into a single reward function, enabling traditional RL algorithms to infer
2The code is available on https://github.com/zpschang/DPMORL.
desirable policies, while also respecting axiomatic principles where necessary. Several studies have utilized non-linear utility functions to guide policy learning [20, 12], catering to more intricate preferences compared to linear ones but still in view of the expectation of the Pareto front. In contrast, our work extends the Pareto-optimality to distributional Pareto-optimality by introducing a set of non-linear utility functions with distributional attributes to guide policy learning. Some of the recent works [21, 22, 23] in MORL defines Distributional Undominated Set or ESR set which is both very close to Distributionally Pareto-Optimal policies in this paper. These works propose novel designs to enhance policy improvement in distributional reinforcement learning algorithms to find the Distributional Undominated Set. Compared to these works, we build novel theoretical results on the conditions of DPO policies and a novel algorithm (DPMORL) of learning DPO policies based on reward shaping that can be built on the top of any online RL algorithm.
Distributional Reinforcement Learning. Distributional RL extends traditional RL by modeling the entire distribution of returns, rather than just their expected values [24, 25]. This approach has been shown to improve both learning efficiency and policy quality in a variety of single-objective RL tasks [26, 27]. Key algorithms in this area include Categorical DQN (C51) [24], Quantile Regression DQN (QR-DQN) [26], and Distributional MPO [27]. Despite the success of these distributional RL algorithms in single-objective settings, their direct extension to MORL has been limited due to the added complexity of handling multiple conflicting objectives and expressing preferences over the distribution of returns. On the other hand, the insights and techniques developed in distributional RL can provide a valuable foundation for incorporating distributional preferences into MORL, as demonstrated by our proposed approach.
Risk-Sensitive and Safe Reinforcement Learning. Risk-sensitive and safe RL are special cases of MORL that accentuate specific facets of reward distributions [6]. Risk-sensitive RL primarily considers reward variance, striving to optimize policies that negotiate the trade-off between expected return and risk [28, 29, 30]. Conversely, safe RL prioritizes constraints on the agent's behavior, ensuring adherence to specified safety criteria throughout the learning process [31, 32, 33]. Some studies have proposed the integration of constraints into the state space, constructing new Markov Decision Processes [11]. Others have explored the distributional aspects of rewards, investigating the implications of distributional RL on risk-sensitive and safety-oriented decision-making [26, 24]. However, our proposed setting is more general in terms of objectives, as it considers a broader range of user preferences and captures the entire distribution of rewards, rather than focusing solely on specific aspects such as risk, variance, or constraints. By extending MORL to incorporate distributional properties, our approach enables the learning of distributional Pareto-optimal policies that cater to diverse user preferences and offer better decision-making in a wide range of real-world applications.
3 Preliminaries: Multi-Objective Reinforcement Learning
Similar to the procedure of RL [34, 35], in MORL, the agent interacts with an environment modeled as a Multi-Objective Markov Decision Process (MOMDP) with multi-dimensional reward functions. A Multi-Objective MDP is defined by a tuple  $(S, A, P, P_0, R, \gamma, T)$ , where  $S$  is the state space,  $A$  is the action space,  $P$  is the state transition function,  $P_0(s)$  is the initial state distribution,  $R: S \times A \times S \to \mathbb{R}^K$  is a vectored reward function and  $K$  is the number of objectives,  $\gamma$  is the discount factor,  $T$  is the total timesteps<sup>3</sup>. The goal of the agent is to learn a set of Pareto-optimal policies, which represent the best possible trade-offs among the different objectives.
One popular methodology for MORL problems is the utility-based method, which combines the multi-dimensional reward functions into a single scalar reward function using a weighted sum or another aggregation method [36]. The intuition is to map the agent's preferences over different objectives to a scalar value for the agent training. Given a weight vector  $\boldsymbol{w} = (w_{1}, w_{2}, \ldots, w_{K})$ , with  $w_{i}$  representing the importance of the  $i$ -th objective, the scalarized reward function is defined as  $r_{\mathrm{scalar}}(s, a) = \sum_{i=1}^{K} w_{i} r_{i}(s, a)$ . The agent then solves the scalarized MDP by optimizing its policy to maximize the expected scalar reward, using standard reinforcement learning algorithms like Q-learning or policy gradient methods. This approach can be straightforward to implement and has been shown to be effective in various MORL settings under expected preferences [37]. However, such a linear combination of each dimension of the reward cannot deal with the preferences with distributional considerations.
<sup>3</sup>We assume that current time  $t$  is a part of state to accommodate for the finite horizon setting.
4 Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning
In this section, we introduce our proposed theoretical framework and algorithms for extending MORL to handle distributional preferences.
4.1 Distributionally Pareto-Optimal Policies
In this work, we consider the reward as a  $K$ -dimensional vector, where each element represents the reward for a specific objective. Given the MOMDP and a policy  $\pi$ , we define the random variable of multi-objective return as  $Z(\pi) = \sum_{t=0}^{T} \gamma^t r_t$ , where  $r_t$  is the multi-dimensional reward at step  $t$ , and the states  $s_0, s_1, \dots, s_T$  and actions  $a_0, a_1, \dots, a_T$  are sampled from the MOMDP and the policy  $\pi$  respectively. The return distribution of policy  $\pi$ , denoted as  $\mu(\pi)$ , represents the joint distribution of the returns  $Z(\pi)$  following policy  $\pi$ . The utility function  $f$  is a non-decreasing function that maps a  $K$ -dimensional return into a scalar utility value for the agent [38, 39, 40], capturing the user's distributional preferences over the different objectives. The expected utility of policy  $\pi$  under the utility function  $f$ , represented as  $\mathbb{E}_{z \sim \mu(\pi)} f(z)$ , is the expected value of applying  $f$  to the return distribution  $\mu(\pi)$ .
Our goal is to learn a set of policies that are distributionally Pareto-optimal, which means that their return distributions of each policy cannot dominate that of another. To measure such a relationship, here we first introduce the concept of stochastic dominance:
Definition 1 (Stochastic Dominance for Multivariate Distribution). A multivariate distribution  $\pmb{\mu}_{1}$  dominates another distribution  $\pmb{\mu}_{2}$ , denoted as  $\pmb{\mu}_{1} \succ_{SD} \pmb{\mu}_{2}$ , if and only if  $\pmb{\mu}_{1} \neq \pmb{\mu}_{2}$  and for any non-decreasing utility function  $f: \mathbb{R}^{K} \to \mathbb{R}$ ,  $\pmb{\mu}_{1}$  has greater expected utility than  $\pmb{\mu}_{2}$ , i.e.  $\mathbb{E}_{\pmb{z} \sim \pmb{\mu}_{1}} f(\pmb{z}) \geq \mathbb{E}_{\pmb{z} \sim \pmb{\mu}_{2}} f(\pmb{z})$ .
Definition 2 (Stochastic Dominance for Policies). A policy  $\pi_1$  stochastically dominates another policy  $\pi_2$ , denoted as  $\pi_1 \succ_{SD} \pi_2$ , if and only if  $\pmb{\mu}(\pi_1) \succ_{SD} \pmb{\mu}(\pi_2)$ , indicating  $\pi_1$  has greater expected utility than  $\pi_2$  under any non-decreasing utility function  $f$ .
(a) Stochastic Dominance Example 1 (b) Stochastic Dominance Example 2 (c) Stochastic Dominance Example 3
Figure 2: Three 2D examples of stochastic dominance given synthetic returns of two policies. (a)  $\pi_2\succ_{SD}\pi_1$ , since any non-decreasing utility function satisfies  $\mathbb{E}f(\pmb {Z}(\pi_2))\geq \mathbb{E}f(\pmb {Z}(\pi_1))$ . (b)  $\pi_1$  and  $\pi_2$  cannot dominate each other. For example, there exists  $f(z) = \mathrm{ReLu}(z_0 - 5.0)$ , such that  $\mathbb{E}f(\pmb {Z}(\pi_2)) > \mathbb{E}f(\pmb {Z}(\pi_1))$ , thus  $\pi_1$  cannot dominate  $\pi_2$ . Similarly,  $\pi_2$  cannot dominate  $\pi_1$ . Thus there is no dominant relationship between the two policies. (c)  $\pi_1$  and  $\pi_2$  cannot dominate each other, which is similar to (b).
The definition of stochastic dominance extends univariate first-order stochastic dominance [13] into multivariate cases. Figure 2 provides some examples of stochastic dominance given 2D returns of two policies. The definition of stochastic dominance in policies allows for comparing the optimality of distributions between policies, which allows us to define the Distributionally Pareto-optimal policy:
Definition 3 (Distributionally Pareto-Optimal Policy). Formally,  $\pi_1$  is Distributionally Pareto-Optimal Policy if there does not exist a policy  $\pi_2$  such that  $\pmb{\mu}(\pi_2) \succ_{SD} \pmb{\mu}(\pi_1)$ .
In other words,  $\pi_1$  is considered Distributionally Pareto-Optimal (DPO) if it is not stochastically dominated by any other policy. DPO policies are essential in our framework as they represent the most desirable policies for users with different distributional preferences. To find such a policy, we have the following theorem:
Theorem 1. If a policy  $\pi$  has optimal expected utility under some non-decreasing utility function  $f$ , where either utility function  $f$  is strictly increasing, or  $\pi$  is the only optimal policy with greatest expected utility under utility function  $f$ , then  $\pi$  is a Distributionally Pareto-Optimal policy. Also, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
The proof of Theorem 1 is provided in Appendix A which utilizes a form of optimal transport theory for stochastic dominance. In practice, this theorem guarantees that the optimal policy of a strictly increasing non-linear utility function is a DPO policy, making it a suitable candidate for deployment in MORL problems. The strict increasing property of utility function  $f$  in Theorem 1 is crucial in the theoretical analysis for proving the distributional pareto-optimality. In the next section, we base on the result of this theorem to find optimal policies for a diverse set of utility functions, in order to learn the set of DPO policies. This result also shows that our definition of Distributional Pareto-Optimal policies is an extension of Pareto-optimal policies, allowing for a more diverse set of optimal policies to be captured.
We also formally prove that optimal policies for risk-sensitive and safe constraint objectives belong to the set of Distributional Pareto-Optimal policies. This proves that DPO policies can successfully cover policies with diverse distributional preferences. The detailed proof is provided by Theorem 3 in Appendix A.
4.2 Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning
We now present our proposed algorithm, Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning (DPMORL). The main idea of DPMORL is to learn a set of non-linear utility functions that can guide the agent to discover distributional Pareto-optimal policies. The algorithm proceeds in two stages: (1) generating the utility functions and (2) training the policies.
4.2.1 Utility Function Generation with Diversity-based Objective
The first component of our algorithm focuses on generating a diverse set of plausible utility functions, upon which we find the optimal policies to find a diverse set of optimal policies. This is essential to ensure our method can accommodate various distributional preferences and adapt to different problem settings. To achieve this, we propose (1) Non-decreasing Neural Network for parameterizing a diverse set of non-linear utility functions (2) an objective function of minimum distance which encourages generating a diverse set of utility functions.
Non-decreasing Neural Network. We employ non-decreasing neural network to parameterize the utility function. The use of a neural network allows us to represent complex, non-linear, and arbitrary continuous utility functions, while the non-decreasing constraint ensures that the utility function satisfies the desired properties for multi-objective problems. We ensure the non-decreasing property in neural networks by constraining the weight matrix to be non-negative and the activation function to be non-decreasing following existing work in convex neural networks [41] and QMIX [42], which can approximate any multivariate non-decreasing function with arbitrary small errors [43]. The implementation details of the Non-decreasing Neural Network are provided in Appendix B.
Diversity-based Objective Function. We propose an objective function based on diversity for learning a diverse set of plausible utility functions. Specifically, we define  $f_{\theta_1}, f_{\theta_2}, \dots, f_{\theta_M}$  to be the set of candidate utility functions, parameterized by  $\theta_1, \dots, \theta_M$ . For any given utility function  $f_{\theta_i}$ , the learning objective is defined as
$$
J ^ {\text {v a l u e}} \left(\theta_ {i}\right) = \min  _ {j \neq i} \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {U} ([ 0, 1 ] ^ {K})} \left[ f _ {\theta_ {i}} (\boldsymbol {z}) - f _ {\theta_ {j}} (\boldsymbol {z}) \right] ^ {2} \tag {1}
$$
$$
J ^ {\operatorname {g r a d}} \left(\theta_ {i}\right) = \min  _ {j \neq i} \mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2} \sim \mathcal {U} ([ 0, 1 ] ^ {K})} \left[ \frac {f _ {\theta_ {i}} \left(\boldsymbol {z} _ {2}\right) - f _ {\theta_ {i}} \left(\boldsymbol {z} _ {1}\right)}{\| \boldsymbol {z} _ {2} - \boldsymbol {z} _ {1} \|} - \frac {f _ {\theta_ {j}} \left(\boldsymbol {z} _ {2}\right) - f _ {\theta_ {j}} \left(\boldsymbol {z} _ {1}\right)}{\| \boldsymbol {z} _ {2} - \boldsymbol {z} _ {1} \|} \right] ^ {2} \tag {2}
$$
$$
J \left(\theta_ {i}\right) = \alpha J ^ {\text {v a l u e}} \left(\theta_ {i}\right) + (1 - \alpha) J ^ {\text {g r a d}} \left(\theta_ {i}\right) \tag {3}
$$
Optimizing this objective function can encourage diversity among the generated utility functions within the normalized range  $[0,1]^K$  in both value and derivative, leading to more comprehensive coverage of potential user preferences. We minimize the objective function by gradient descent on non-decreasing neural networks to generate a set of  $N$  utility functions. To make the learned utility function strictly increasing for satisfying the constraints in Theorem 1, we add an additional term to the learned utility function as  $\tilde{f}_{\theta_i}(z) = f_{\theta_i}(z) + \frac{\alpha}{K}\sum_{k=1}^{K}z_k$  in our experiment with  $\alpha = 0.01$ , and use  $\tilde{f}_{\theta_i}(z), i = 1,\dots,M$  as the strictly increasing utility function to learn DPO policies.
4.2.2 Optimizing Policy with Utility-based Reinforcement Learning
Once we have generated a diverse set of utility functions, the second component of our algorithm focuses on optimizing policies to maximize the expected utility. This process, which we call utility-based RL, leverages the generated utility functions to guide the optimization of policies. By focusing on the expected utility, our method can efficiently balance the trade-offs between multiple objectives and distributions, ultimately generating policies that are more likely to align with user preferences.
We show that the following utility-based reinforcement learning algorithm can effectively optimize the policy with respect to a given utility function.
Algorithm 1 Utility-based Reinforcement Learning
Input: policy  $\pi$ , an environment  $\mathcal{M} = (S, A, P, P_0, R, \gamma, T)$ , utility function  $f$   
Output: new policy  $\pi'$
1: Augment state space with  $\tilde{S} = S\times \mathcal{Z}$ , where  $\mathcal{Z}$  is the space of cumulative multivariate returns.
2: Let transition  $\tilde{P}_0(\cdot)$  and  $\tilde{P} (\cdot |(s_t,z_t),a_t)$  with  $s_0\sim P(s_0),z_0 = 0$  , and  $s_{t + 1}\sim P(\cdot |s_t,a_t)$ $\pmb{z}_{t + 1} = \pmb {z}_t + \gamma^t\pmb {r}_t$
3: Let scalar reward function  $R$  as  $R((s_t, z_t), a_t, (s_{t+1}, z_{t+1})) = \gamma^{-t}[f(z_{t+1}) - f(z_t)]$ .
4: Optimize policy  $\pi$  under environment  $\tilde{\mathcal{M}} = (\tilde{S}, A, \tilde{P}, \tilde{P}_0, R, \gamma, T)$  under off-the-shelf RL algorithm (such as PPO or SAC) to  $\pi'$ .
Briefly, Algorithm 1 augments the state space with the cumulative multi-objective returns, and transforms the multi-dimensional rewards into a scalar reward by the difference in the utility function  $f$ . The following result shows that the new scalar-reward environment  $\tilde{M}$  generated by Algorithm 1 has the same optimal policy as the optimal policy under utility function  $f$ :
Theorem 2. The optimal policy  $\pi^{*}$  under environment  $\tilde{\mathcal{M}} = (\tilde{S},A,\tilde{P},\tilde{P}_0,R,\gamma ,T)$ , with scalar reward function
$$
R \left(\left(s _ {t}, z _ {t}\right), a _ {t}, \left(s _ {t + 1}, z _ {t + 1}\right)\right) = \gamma^ {- t} \left[ f \left(z _ {t + 1}\right) - f \left(z _ {t}\right) \right]
$$
is the optimal policy under the utility function  $f$ , i.e.