SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\boldsymbol {z} \sim \boldsymbol {\mu} (\pi^ {*})} [ f (\boldsymbol {z}) ] = \max _ {\pi} \mathbb {E} _ {\boldsymbol {z} \sim \boldsymbol {\mu} (\pi)} [ f (\boldsymbol {z}) ].
$$
An advantage of Algorithm 1 is that we can directly utilize off-the-shelf RL algorithms, such as PPO [44] and SAC [45] without any modification, which makes the algorithm easy to implement using widespread existing implementations of online RL algorithms.
It is also important to note that our algorithm simplifies to optimizing the weighted sum of rewards in MORL when the utility function is linear. This implies that our method is a generalization of the linear utility function MORL approaches by accommodating a wide range of non-linear utility functions. This flexibility makes our algorithm particularly suited for problems where the user's preferences may not be adequately captured by a linear utility function.
In the next experimental section, DPMORL only undergoes a single iteration: we initially generate a set of  $N$  utility functions as per the methodology detailed in Section 4.2.1. Subsequently, it optimizes a set of  $N$  policies using these generated utility functions, as outlined in Section 4.2.2.
In this section, we conducted several experiments under the setting of MORL. Through the experiments, we want to investigate the following questions:
1. Can Utility-based RL effectively learn policies with diverse distributional preferences?
2. Can DPMORL generate a set of diverse non-linear utility functions?
3. Can DPMORL obtain promising performance compared with state-of-the-art MORL methods in view of expected and distributional preferences?
5.1 Case Study of Utility Functions
To answer the first question, we train policies with a diverse set of utility functions on DiverseGoal, DeepSeaTreasure and Reacher. Here, we focus on showing the effectiveness of the Utility-based RL algorithm (Algorithm 1). We select several different non-linear functions  $f$ , each in favor of distributions at one target. We train policy  $\pi_{i}$  with utility function  $f_{i}$  with Algorithm 1 for each target, and show the policy's trajectory and return samples.
(a) The map of the DiverseGoal environment.
(b) Reward distribution within each goal position.
Figure 3: Illustrations of the DiverseGoal environment.
DiverseGoal is a MORL environment in Figure 3 with multiple goals that the agent can take multiple steps to reach, where each goal has its unique reward function. Upon reaching a particular goal, the agent secures a 2D reward. This reward is sampled from the specific normal distribution associated with that goal, as illustrated in Figure 3b. Conversely, if the agent reaches the boundary of the map, it incurs a negative 2-dimensional reward. The environment requires the agent to navigate the trade-offs between various reward distributions, underscoring the complexity and nuance inherent to the task.
In this study, we investigate if the learned policy can reach the goal that yields the highest expected utility under the utility function. The results are illustrated in Figure 4. Under different types of utility functions and return distributions, the utility-based RL algorithm is able to find the optimal policy with the highest expected utility, which shows the effectiveness of the utility-based RL algorithm.
DeepSeaTreasure and Reacher are among the most widely used MORL environments. Under the DeepSeaTreasure environment and Reacher environment, we show the return distributions of the policies optimized with utility-based RL under different utility functions. In Figure 5 and Figure 6, we use utility functions generated by DPMORL in Section 4.2.1 and a set of manually constructed utility functions for different types of distributional preferences respectively. From the result in Figure 5 and Figure 6, we can see that our algorithm effectively finds policies that maximize the expected utility for different non-linear utility functions. Meanwhile, the DeepSeaTreasure environment aligns well
(a) Utility functions in DiverseGoal environment used for training policies.
(b) Return distribution of policy to maximize expected utility for each utility function.
(c) The trajectory of policies learned by maximizing the expected utility for each utility function.
Figure 4: The case study results of DPMORL under DiverseGoal environment.
(a) Utility functions generated by our method in Section 4.2.1 under DeepSeaTreasure environment.
(b) Return distribution of policy learned by DPMORL to maximize expected utility for each utility function.
Figure 5: The case study results of DPMORL under DeepSeaTreasure environment.
(a) Utility functions with different distributional preferences under Reacher environment.
(b) Return distribution of policy learned by DPMORL to maximize expected utility for each utility function.
Figure 6: The case study results of DPMORL under Reacher environment.
with safe RL settings, where the two returns are time penalty and task reward respectively. With our method, users can provide different utility functions derived from safety constraints, and optimize the expected utility via DPMORL for obtaining policies that satisfy the constraint.
5.2 Main Experiment
In this section, we illustrate the performance of DPMORL on five environments based on MO-Gymnasium [46] to answer the latter two questions.
**Environments.** We conducted experiments across five environments based on MO-Gymnasium [46] to evaluate the performance of our proposed method, DPMORL. These environments represent a diverse range of tasks, from simple toy problems to more complex continuous control tasks, and cover various aspects of multi-objective reinforcement learning:
- DeepSeaTreasure: A classic MORL benchmark that requires exploration in a gridworld to find treasures with different values and depths.
- FruitTree: A multi-objective variant of the classic gridworld problem, where an agent has to collect different types of fruits with varying rewards and penalties.
- HalCheetah, Hopper, MountainCar: Three continuous control tasks that require controlling different agents for task solving and minimizing energy usage.
More details about the environments are gathered in Appendix C.1.
Baselines. We compare DPMORL with four state-of-the-art baselines in the context of distributional preferences of MORL: Optimistic Linear Support (OLS) [47, 16]; Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL) [8]; Generalized Policy Improvement with Linear Support (GPI-LS) [7]; Generalized Policy Improvement with Prioritized Dyna (GPI-PD) [7].
Training Details. For all methods, including DPMORL and the baselines, each policy was trained for  $1 \times 10^{7}$  steps. We learn a set of  $N = 20$  policies for DPMORL and all of the baselines to ensure a fair comparison. Finally, we use the learned  $N = 20$  policies in each method for evaluations.
Figure 7: Illustration of 2D utility functions learned by our methods in Section 4.2.1.
Implementation Details. We use PPO algorithms implemented in Stable Baselines 3 [48] in Algorithm 1. We use a normalization technique by linearly mapping the return into scale  $[0,1]^K$  without modifying the optimal policies. Detailed implementations are provided in Appendix B.
Evaluation Metrics. To thoroughly evaluate the performance of DPMORL and compare it with the baseline methods, we employed four distinct metrics. These comprise two existing MORL metrics, HyperVolume and Expected Utility, in addition to two novel metrics developed specifically for assessing distributional preference, i.e., Constraint Satisfaction and Variance Objective. The latter two are designed to underscore the optimality of multivariate distributions associated with the learned policies. In terms of Constraint Satisfaction, we randomly generate  $M = 100$  constraints for the policy set produced. The Constraint Satisfaction metric is then computed by considering the highest probability within the policy set that satisfies each individual constraint. The Variance Objective metric, on the other hand, involves generating  $M = 100$  random linear weights. These weights are applied to both the expected returns and the standard deviations of returns in each dimension. This objective encourages attaining greater expected returns while simultaneously reducing variance, thereby catering to dynamic preferences. Further details about the implementation of these evaluation metrics are provided in Appendix C.2 and Appendix C.3.
Generation of Utility Functions. In accordance with the methodology detailed in Section 4.2.1, we employ non-decreasing neural networks in conjunction with diversity-focused objectives to generate a diverse assortment of utility functions. The generated functions are visually represented in Figure 7. The outcomes clearly indicate that optimizing diversity-based objective functions allows for generating a broad range of non-linear utility functions, thereby encompassing an expansive array of preferences with respect to returns. Subsequently, we utilize the first  $N = 20$  utility functions depicted in Figure 7 to train an equivalent number of policies under DPMORL.
Standard MORL Metrics. The results under standard MORL metrics are shown in Table 1. Focusing on the Expected Utility, the performance of DPMORL is the highest in the Hopper, HalfCheetah, MountainCar, and DeepSeaTreasure environments (4/5), indicating that our method outperforms others in terms of expected utility. While for FruitTree environment, DPMORL also obtains consistent performance with the best contender, GPI-LS. In terms of the HyperVolume, DPMORL also performs the best in the Hopper, HalfCheetah, MountainCar, and FruitTree environments (4/5). The results show that DPMORL can yield better performance across both metrics and most environments (Hopper, HalfCheetah, MountainCar, and DeepSeaTreasure), meanwhile delivering robust and consistent results on the other one (FruitTree). Since PGMORL works on continuous action spaces, here we omit the results of PGMORL on environments with discrete action space, DeepSeaTreasure and FruitTree.
Distributional MORL Metrics. The results under distributional metrics are shown in Table 2. DPMORL outperforms other methods in most environments on both Constraint Satisfaction (5/5) and Variance Objective (3/5), indicating its strong ability to handle the distributional multi-objective reinforcement learning problem. For the rest environments, DPMORL also obtained comparable results compared with the best contender, GPI-LS. The results show that the policies learned by
Table 1: Experimental results of each method on each standard MORL metric on all five environments. "EU" stands for Expected Utility, and "HV" stands for Hypervolume.
Environment Metric	Hopper	HalfCheetah	MountainCar	DeepSeaTreasure	FruitTree
EU	HV	EU	HV	EU	HV	EU	HV	EU	HV
GPI-PD	2374.44	5033458.34	412.62	1083227.57	-55.44	7367.59	5.93	9.75	6.98	0.14
GPI-LS	1398.00	1446705.10	98.31	2839051.60	-37.37	8052.58	5.04	9.36	3.84	1.49
OLS	175.07	3298.13	-580.38	1420270.13	-470.00	2.45	4.64	10.28	6.27	7.49
PGMORL	300.19	32621.25	111.30	383681.25	-429.48	94.45	-	-	-	-
DPMORL (Ours)	3492.93	12154967.99	1189.68	8593769.62	-29.89	8663.80	6.70	8.68	6.89	16.39
Table 2: Experimental results of each method on Distributional metric on all five environments. "Constraints" stands for Constraints Satisfaction, and "Var" stands for Variance Objective.
Environment Metric	Hopper	HalfCheetah	MountainCar	DeepSeaTreasure	FruitTree
Constraints	Var	Constraint	Var	Constraint	Var	Constraint	Var	Constraint	Var
GPI-PD	0.47	979.74	0.64	83.49	1.00	-31.15	0.85	2.59	0.65	3.42
GPI-LS	0.25	607.47	0.60	51.67	1.00	-22.73	0.85	1.99	0.33	1.35
OLS	0.05	75.48	0.43	-311.27	0.05	-244.21	0.80	1.86	0.50	2.98
PGMORL	0.05	98.47	0.50	45.48	0.11	-249.69	-	-	-	-
DPMORL (Ours)	0.76	1645.89	0.82	431.26	1.00	-16.32	0.90	2.54	0.67	3.21
DPMORL have a higher probability of satisfying randomly generated constraints, and can better balance the trade-off between expectations and variances.
Return Distributions of Learned Policies by DPMORL. We provide visualizations of the multidimensional return distributions of each policy learned by DPMORL and four baseline methods (PGMORL, OLS, GPI-LS and GPI-PD) on different MORL environments, as shown in Appendix C.4. DPMORL's learned policy set demonstrates higher diversity with respect to return distributions compared to baseline methods.
DPMORL on More Than 2-dimensional Return Space. We run experiments with 3 objectives under Hopper and FruitTree environment, and show the learned return distribution and standard MORL evaluation metrics in Appendix C.5. The results demonstrate that DPMORL can effectively learn DPO policies under more than 3-dimensional reward functions, and achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.
Ablation Analysis with Variable Number of Policies. We provide the ablation studies of DPMORL with a varying amount of learned policies  $N$  in Appendix C.6. As the number of policies increases, both expected utility and hypervolume increase, which means DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only 5 policies, which verify the effectiveness of DPMORL in all environments.
In this work, we initialized the study of distributional MORL, specifically when preferences over different objectives and their return distributions are uncertain. We introduced the concept of Distributional Pareto-Optimal (DPO) policies with rigorous theoretical analysis, which extend the notion of Pareto-optimality in MORL to include preferences over the entire distribution of returns, not just their expected values. To obtain such desirable policies, we proposed a new algorithm, DPMORL, designed to learn DPO policies with non-linear utility functions. DPMORL allows for expressing a wide range of distributional preferences, providing a flexible and expressive approach to MORL. Experiment results showed that DPMORL consistently outperformed existing MORL methods in terms of optimizing policies for multivariate expected and distributional preferences.
This research was supported by the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108; and Microsoft Research Asia D-CORE program. The authors would like to thank Soichiro Nishimori, Kazuki Ota, and the anonymous reviewers for their insightful comments and suggestions.
[43] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating functional knowledge in neural networks. Journal of Machine Learning Research, 10(6), 2009.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[45] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[46] Lucas N. Alegre, Florian Felten, El-Ghazali Talbi, Gregoire Danoy, Ann Nowé, Ana L. C. Bazzan, and Bruno C. da Silva. MO-Gym: A library of multi-objective reinforcement learning environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelearn 2022, 2022.
[47] Diederik M Roijers. Multi-objective decision-theoretic planning. AI Matters, 2(4):11-12, 2016.
[48] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[49] Takashi Kamihigashi and John Stachurski. Partial stochastic dominance via optimal transport. Operations Research Letters, 48(5):584-586, 2020.
[50] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[51] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Lemma 1. If a policy  $\pi$  is the only optimal policy under some non-decreasing utility function  $f$ , then it is a Distributionally Pareto-Optimal policy.
Proof. If  $\pi$  is the only optimal policy under non-decreasing utility function  $f$ , then for any policy policy  $\pi'$ ,
$$
\mathbb {E} _ {\boldsymbol {z} \sim \boldsymbol {\mu} (\pi)} f (\boldsymbol {z}) > \mathbb {E} _ {\boldsymbol {z} \sim \boldsymbol {\mu} \left(\pi^ {\prime}\right)} f (\boldsymbol {z}). \tag {4}
$$
From Definition 1 and Definition 2, if any policy  $\pi'$  stochastically dominates  $\pi$ , then for any utility function  $f$ ,  $\mathbb{E}_{z \sim \mu(\pi')} \geq \mathbb{E}_{z \sim \mu(\pi)} f(z)$ , which makes a contradiction. Therefore,  $\pi$  is not stochastically dominated by any other policies, and  $\pi$  is a Distributional Pareto-Optimal policy.
Lemma 2. If a policy  $\pi$  is one of the optimal policy under some strictly increasing utility function  $f$ , then it is a Distributionally Pareto-Optimal policy. The utility function  $f$  is strictly increasing if and only if  $x\succeq y$  and  $x\neq y\Rightarrow f(x) > f(y)$ .
Proof. Following the Equation (2) and (3) of [49] which is derived from Kantorovich duality (Theorem 5.10 in [50]), we have
$$
\boldsymbol {\mu} _ {1} \succ_ {S D} \boldsymbol {\mu} _ {2} \text {i f a n d o n l y i f} \boldsymbol {\mu} _ {1} \neq \boldsymbol {\mu} _ {2} \text {a n d} \max  _ {\gamma \in \Pi \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\mu} _ {2}\right)} \mathbb {E} _ {(x, y) \sim \gamma} [ \mathbb {1} _ {x \succeq y} ] = 1 \tag {5}
$$
where  $\Pi (\pmb {\mu}_1,\pmb {\mu}_2)$  is the set of all couplings of  $\pmb{\mu}_{1}$  and  $\pmb{\mu}_{2}$ , which is the set of distributions  $\gamma (x,y)$  such that
$$
\int \boldsymbol {\gamma} (x, y) d y = \boldsymbol {\mu} _ {1} (x), \int \boldsymbol {\gamma} (x, y) d x = \boldsymbol {\mu} _ {2} (y).
$$
Suppose there exists a policy  $\pi'$  stochastically dominate  $\pi$ , then there exists  $\gamma(x,y) \in \Pi(\pmb{\mu}(\pi'), \pmb{\mu}(\pi))$ , such that  $\mathbb{E}_{(x,y) \sim \gamma}[\mathbb{1}_{x\succeq y}] = 1$ . Since  $\pi$  is one of the optimal policies under strictly increasing utility function  $f$ , we have
$$
\mathbb {E} _ {x \sim \boldsymbol {\mu} (\pi^ {\prime})} f (x) - \mathbb {E} _ {y \sim \boldsymbol {\mu} (\pi)} f (y) \leq 0
$$
$$
\int f (x) \boldsymbol {\mu} \left(\pi^ {\prime}\right) (x) d x - \int f (y) \boldsymbol {\mu} (\pi) (y) d y \leq 0
$$
$$
\iint [ f (x) - f (y) ] \boldsymbol {\gamma} (x, y) d x d y \leq 0
$$
Since  $\mathbb{E}_{(x,y)\sim \gamma}[\mathbb{1}_{x\succeq y}] = 1$ , i.e.  $\iint_{x\succeq y}\gamma (x,y)dxdy = 1$ , we have
$$
\iint_ {x \not \leq y} [ f (x) - f (y) ] \boldsymbol {\gamma} (x, y) d x d y = 0
$$
$$
\iint_ {x \succeq y} [ f (x) - f (y) ] \gamma (x, y) d x d y \leq 0
$$
There are two cases. Case 1: there exists  $c\succ 0$  such that  $\iint_{x\succeq y + c}\gamma (x,y)dxdy > 0$
Since  $f$  is an strictly increasing function, we have
$$
\text {I f} x \succeq y \text {a n d} x \neq y, \text {t h e n} f (x) > f (y).
$$
$$
\begin{array}{l} 0 \geq \iint_ {x \succeq y} [ f (x) - f (y) ] \gamma (x, y) d x d y \\ \geq \iint_ {x \succeq y + c} [ f (x) - f (y) ] \boldsymbol {\gamma} (x, y) d x d y + \iint_ {y + c \succ x \succeq y} [ f (x) - f (y) ] \boldsymbol {\gamma} (x, y) d x d y \\ \geq \iint_ {x \succeq y + c} [ f (y + c) - f (y) ] \gamma (x, y) d x d y \\ \end{array}
$$
Since  $c\succ 0$ ,  $f(y + c) - f(y) > 0$ . Since  $\iint_{x\succ y + c}\pmb {\gamma}(x,y) > 0$  and  $\pmb {\gamma}(x,y)$  is continuous almost everywhere, we have  $\iint_{x\succ y + c}[f(y + c) - f(y)]\pmb {\gamma}(x,y)dxdy > 0$ , which yields a contradiction.
Case 2: there does not  $c \succ 0$  such that  $\iint_{x \succeq y + c} \gamma(x, y) \, dx \, dy > 0$ . Then, we have  $\mathbb{E}_{(x, y) \sim \gamma(x, y)} \mathbb{1}[x = y] = 1$ , which contradicts with the assumption that  $\mu(\pi') \neq \mu(\pi)$ .
Therefore, we have proved that there does not exist  $\pi'$  that stochastically dominate  $\pi$ , when  $\pi$  has optimal expected utility under some strictly increasing utility function  $f$ .
Lemma 3. Any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
Proof. We first show that if distribution  $\pmb{\mu}^{\prime}$  stochastically dominates  $\pmb{\mu}$ , then the expectation  $\mathbb{E}_{z\sim \mu '}[z]$  dominates  $\mathbb{E}_{z\sim \mu}[z]$ . Following Kantorovich duality, there exists a coupling  $\gamma$  of  $\pmb{\mu}^{\prime}$  and  $\pmb{\mu}$  that  $\mathbb{E}_{(x,y)\sim \gamma}[\mathbb{1}_{x\succeq y}] = 1$ . Then,
$$
\begin{array}{l} \mathbb {E} _ {z \sim \mu^ {\prime}} [ z ] - \mathbb {E} _ {z \sim \mu} [ z ] \\ = \mathbb {E} _ {(x, y) \sim \gamma} [ x - y ] \\ \succeq \mathbb {E} _ {(x, y) \sim \gamma} [ \mathbf {0} ] = \mathbf {0} \\ \end{array}
$$
When  $\mathbb{E}_{(x,y)\sim \gamma}[x - y] = 0$ , it requires that  $x = y$  holds where  $\gamma (x,y) > 0$ , which makes  $\pmb{\mu}^{\prime} = \pmb{\mu}$  that contradicts with the fact that  $\pmb{\mu}^{\prime}\succ_{SD}\pmb{\mu}$ . Therefore, the expectation  $\mathbb{E}_{z\sim \mu '}[z]$  dominates the expectation  $\mathbb{E}_{z\sim \mu}[z]$ . It follows that if the expectation  $\mathbb{E}_{z\sim \mu}[z]$  is not dominated by the expectation  $\mathbb{E}_{z\sim \mu'}[z]$ , then the distribution  $\pmb{\mu}$  is not stochastically dominated by  $\pmb{\mu}'$ .
If policy  $\pi$  is Pareto-optimal, then its multidimensional expected returns  $\mathbb{E}[Z(\pi)]$  are not dominated by  $\mathbb{E}[Z(\pi')]$  of any other policies  $\pi'$ . Then, the return distribution  $\mu(\pi)$  of policy  $\pi$  is not stochastically dominated by the return distribution  $\mu(\pi')$  of any other policies  $\pi'$ . Therefore, policy  $\pi$  is a Distributionally Pareto-Optimal policy. To conclude, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
Theorem 1. If a policy  $\pi$  has optimal expected utility under some non-decreasing utility function  $f$ , where either
1. utility function  $f$  is strictly increasing
2.  $\pi$  is the only optimal policy with greatest expected utility under utility function  $f$
then  $\pi$  is a Distributionally Pareto-Optimal policy. Also, any Pareto-Optimal policy is a Distributionally Pareto-Optimal policy.
Proof. The theorem directly follows from Lemma 1, Lemma 2 and Lemma 3.  $\square$
The following theorem proves that the optimal policy over Value at Risk (VaR) and over Conditional Value at Risk (CVaR) belong to the set of Distributionally Pareto-Optimal policies. Here, we use the random variable  $\mu (\pi)$  to represent the negative risk, where smaller value represents larger risks, in order to be aligned with the definition of Pareto-Optimality.
Definition 4. Given a distribution  $\mu$ , the Value at Risk (VaR) is defined as
$$
V a R _ {\alpha} (\mu) = \min  \left\{z _ {0} | \mathbb {E} _ {z \sim \mu} \mathbb {1} [ z \leq z _ {0} ] \geq \alpha \right\}
$$
Definition 5. Given a distribution  $\mu$ , the Conditional Value at Risk (CVaR) is defined as
$$
C V a R _ {\alpha} (\mu) = \mathbb {E} _ {z \sim \mu} [ z | z \leq V a R _ {\alpha} (\mu) ]
$$
Theorem 3. When  $K = 1$ , there exists policy with optimal Value at Risk (VaR), and policy with optimal Condition Value at Risk (CVaR), which belongs to the set of Distributional Pareto-Optimal Policies.
Proof. We let the policy set  $\Pi_{\mathrm{VaR}}$  as the policy set with optimal value at risk, which  $z_0 = \mathrm{VaR}_{\alpha}(\mu (\pi))$  for any  $\pi \in \Pi_{\mathrm{VaR}}$ . We let  $\Pi_{\mathrm{VaR}}^{\prime}$  to be the policy set with non-optimal policy, with  $\mathrm{VaR}_{\alpha}(\mu (\pi^{\prime})) < z_{0}$  for any  $\pi^{\prime}\in \Pi_{\mathrm{VaR}}^{\prime}$ . Suppose all Distributionally Pareto-Optimal policy belongs to  $\Pi_{\mathrm{VaR}}^{\prime}$ . We define a