SlowGuess/NeurIPS_2023_OCR huggingface dataset:
utility function as  $f(z) = \left\{ \begin{array}{ll} 1, & z \geq z_0 \\ 0, & z < z_0 \end{array} \right.$ , which is non-decreasing. Then, we have  $\mathbb{E}_{z \sim \mu(\pi)} f(z) \geq \alpha$  for any  $\pi \in \Pi_{\mathrm{VaR}}$ , while  $\mathbb{E}_{z \sim \mu(\pi')} f(z) < \alpha$  for any  $\pi' \in \Pi_{\mathrm{VaR}}'$ . Therefore, any policy  $\pi \in \Pi_{\mathrm{VaR}}$  is not stochastically dominated by any policy  $\pi' \in \Pi_{\mathrm{VaR}}'$ . Therefore, there must exists a Distributional Pareto-Optimal Policy in  $\Pi$ , which results in a contradiction. Therefore, there must exist a policy with optimal value at risk, which is Distributionally Pareto-Optimal.
Similarly to the case in VaR, we let the policy set  $\Pi_{\mathrm{CVaR}}$  as the policy set with optimal conditional value at risk, which  $z_0 = \mathrm{CVaR}_{\alpha}(\mu(\pi))$  for any  $\pi \in \Pi_{\mathrm{CVaR}}$ . We let  $\Pi_{\mathrm{CVaR}}'$  to be the policy set with non-optimal CVaR, with  $\mathrm{CVaR}_{\alpha}(\mu(\pi')) < z_0$  for any  $\pi' \in \Pi_{\mathrm{CVaR}}'$ . Then we can prove that any policy  $\Pi_{\mathrm{CVaR}}'$  does not stochastically dominate the policy in  $\Pi_{\mathrm{CVaR}}$ . Suppose the opposite case where  $\pi \in \Pi_{\mathrm{CVaR}}$  is dominated by  $\pi' \in \Pi_{\mathrm{CVaR}}'$ . We let  $f(z), f'(z)$  to be the cumulative distribution function of  $\mu(\pi)$  and  $\mu(\pi')$ , and  $f(z) \geq f'(z)$  must be satisfied to make  $\pi \prec_{SD} \pi'$ . Therefore,
$$
z _ {0} = \mathrm {C V a R} (\mu (\pi)) = \mathbb {E} _ {z \sim \mathcal {U} (0, \alpha)} f ^ {- 1} (z) \leq \mathbb {E} _ {z \sim \mathcal {U} (0, \alpha)} f ^ {\prime - 1} (z) = \mathrm {C V a R} (\mu (\pi^ {\prime}))
$$
which results in a contradiction. Therefore, any policy  $\Pi_{\mathrm{CVaR}}^{\prime}$  does not stochastically dominate the policy in  $\Pi_{\mathrm{CVaR}}$ . We thus proved that there exists a Distributionally Pareto-Optimal policy in  $\Pi_{\mathrm{CVaR}}$ , which is an optimal policy under conditional value at risk.
Non-Decreasing Neural Network. We used Non-Decreasing Neural Network to parameterize the utility function. The network architecture consisted of three fully connected layers, with our designed non-decreasing activation activation functions applied to the hidden layers:
$$
f (x) = \operatorname {c o n c a t} (\max  (x, - 0. 5), \min  (x, 0. 5), \operatorname {c l i p} (x, - 0. 5, 0. 5))
$$
This design of activation function is an extension to ReLU activation function, which empirically make the learned utility function more diverse, covering more types of decision boundaries. After each gradient update during training, we clip all the trainable weights in the network to be non-negative to ensure that the resulting utility function was non-decreasing. The network's input dimension corresponded to the number of objectives in the environment, and the output dimension was a scalar representing the utility value.
Return Normalization. We use a normalization technique by linearly mapping the return into scale  $[0,1]^K$  without modifying the optimal policies. Specifically, we keep track of the minimum and maximum multivariate returns that the policies have currently achieved, denote by  $z_{min}$  and  $z_{max}$ . We let  $z_{mid} = (z_{min} + z_{max}) / 2$ , and  $d = \max_i[z_{max,i} - z_{min,i}]$ . Before the any multivariate return  $z$  is fed into the utility function, we normalize it by
$$
z _ {n o r m} = \frac {1}{d} (z _ {n o r m} - z _ {m i d}) + \frac {1}{2}
$$
This maps  $z$  linearly to  $z_{norm}$ , where  $z_{norm} \in [0,1]^K$ , without affecting the scale of each dimension of returns. In Algorithm 1, the return  $z_t$  is normalized before feeding into the utility function to compute the scalar reward.
DPMORL Algorithm. For the implementation of our DPMORL algorithm, we used the Proximal Policy Optimization (PPO) algorithm [44] as the basic RL algorithm. The learning rate was set to  $3 \times 10^{-4}$ , with a discount factor  $\gamma$  of 0.99. For PPO, we used a clipping parameter of 0.2. The batch size was set to 256 for all environments and algorithms, with updates performed every 2,048 steps. The utility functions were trained using the Adam optimizer [51], with a learning rate of  $3 \times 10^{-4}$ .
C.1 Environments
The details of the environments used in the main paper are listed below:
- DeepSeaTreasure: A classic MORL benchmark that requires exploration in a gridworld to find treasures with different values and depths.
- FruitTree: A multi-objective variant of the classic gridworld problem, where an agent has to collect different types of fruits with varying rewards and penalties.
- HalCheetah: A continuous control task where a simulated cheetah must run and jump over obstacles while minimizing energy consumption.
- Hopper: A continuous control task that involves balancing and hopping a simulated one-legged robot while minimizing energy usage.
- MountainCar: A classic reinforcement learning problem, extended to include multiple objectives, such as minimizing the time to reach the goal and reducing energy consumption.
In the Appendix, we additionally add one environment:
- Reacher: A discrete control task where a simulated robotic arm must reach a target while minimizing energy usage.
C.2 Standard MORL Metrics
The details of the standard MORL metrics used in the main paper are listed below:
- Expected Utility is similar to R-Metrics used in Multi-Objective Optimization (MOO). This metric is concerned with the expected utility of the policies on the Pareto front for various weights. It provides a means to assess the quality of the policies derived by the algorithm in terms of their expected performance across different utility functions. A higher Expected Utility indicates that the policies offer a better trade-off between the objectives, thereby leading to more desirable outcomes under a variety of user preferences:
$$
\operatorname {E U} (\Pi) = \mathbb {E} _ {\mathbf {w} \sim \mathcal {W}} [ \max  _ {\pi \in \Pi} f _ {\mathbf {w}} (\mathbf {z}) ], \tag {6}
$$
where  $f_{\mathbf{w}}$  denotes the user's utility function linearly parameterized by  $\mathbf{w}$ , and  $\mathbf{z}$  denotes the returns of one episode.
- Hypervolume is a prominent measure used in Multi-Objective Optimization (MOO) and Multi-Objective Reinforcement Learning (MORL). This metric calculates the volume in the objective space dominated by the Pareto front with respect to a reference point. The Hypervolume gives a scalar measure of the quality of a Pareto front, with higher values indicating a better Pareto front. A Pareto front with a larger Hypervolume signifies that the algorithm can yield a wider range of policies, thus dominating more of the objective space, which indicates a better exploration of the policy space:
$$
\operatorname {H V} (\Pi , r) = \operatorname {L e b} (\mathbf {y} \in \mathbb {R} ^ {K} \mid \exists \pi \in \Pi : \mathbf {z} \prec_ {S D} \mathbf {y} \prec_ {S D} \mathbf {r}), \tag {7}
$$
where  $r$  is the reference point in the objective space and Leb denotes the Lebesgue measure, which in this context refers to the volume in the objective space. The reference points used in the experiment of the main paper are calculated as the minimal mean returns of each method under evaluation.
C.3 Distributional MORL Metrics
We propose two novel metrics for assessing distributional preference: Constraint Satisfaction and Variance Objective.
For Constraint Satisfaction, we define  $u(p,\pi)$  to be the probability of return random variance  $z\sim Z(\pi)$  that satisfy the constraint  $p$ , representing how a policy satisfies the given constraint. We randomly generate each constraint defined by  $n$  linear constraints represented by  $\{z|\forall i,\pmb {w}_i^T\pmb {z}\geq c_i\}$  where  $w_{i}$  is uniformly sampled from the surface  $\pmb {w}_i\succeq 0,\mathbf{1}^T\pmb {w}_i = 1$ , and  $c_{i}$  is sampled from the distribution of  $\mathcal{U}(\min_{z\in \mathcal{Z}}\pmb {w}_i^T\pmb {z},\max_{z\in \mathcal{Z}}\pmb {w}_i^T\pmb {z})$ , where  $\mathcal{Z}$  is the set of all return samples of the final policies learned by DPMORL.
For Variance Objective, we define  $u(p,\pi)$  to be a weighted sum of expected return  $\mathbb{E}_{z\sim \mu (\pi)}z$  and standard deviation of return  $\sqrt{\operatorname{Var}_{z\sim\mu(\pi)}[z]}$ . We randomly generate  $\boldsymbol{w}_i\in \mathbb{R}^{2K}$ , where  $\boldsymbol{w}_i\succeq 0$  and
Figure 8: Return distribution of the set of policies learned by DPMORL on each environment.
·π0	·π2	·π4	·π6	·π8	·π10	·π12	·π14	·π16	·π18
·π1	·π3	·π5	·π7	·π9	·π11	·π13	·π15	·π17	·π19
$\mathbf{1}^T \pmb{w}_i = 1$ . We let  $\pmb{w}_i = [\pmb{w}_i^1, \pmb{w}_i^2]$ , where  $\pmb{w}_i^1, \pmb{w}_i^2 \in \mathbb{R}^K$ , and use  $\pmb{w}_i^1$  and  $-\pmb{w}_i^2$  as the weight for expected return and standard derivation of return respectively.
For both of the metric, we define  $u(p,\pi)$  to be the satisfaction score of policy  $\pi$  under preference  $p$ . We generate  $M = 100$  number of preference  $p_1,p_2,\dots ,p_M$ . For both of the Constraint Satisfaction and Variance Objective metric, the mean score is defined as
$$
\frac{1}{M}\sum_{i = 1}^{M}\max_{j\in \{1,2,\dots ,N\}}u(p_{i},\pi_{j})
$$
The mean score in the above equation measures the best performance of satisfying the preference of the user, averaged in  $M = 100$  preferences. Higher mean score represents better ability of meeting different preferences. In Table 2 of the main experiment, all of the numbers are calculated with the above equation.
C.4 Return Distributions of Learned Policies by DPMORL and Baseline Methods
We demonstrate the return distribution of the learned policies by DPMORL in Figure 8, which demonstrates that DPMORL is able to learn a set of policies with optimal return distributions. We also visualize the return distribution of policies learned by four baseline methods, which is shown in Figure 9. DPMORL's learned policy set has higher diversity with respect to return distributions compared to baseline methods.
C.5 DPMORL on More Than 2-Dimensional Return Space
We run experiments with 3 objectives under Hopper and FruitTree environment, where we use the first three dimension of the reward in the MO-Hopper and Fruit-Tree environment the MO-Gymnasium. We show the 3-dimensional return distribution of the set of policies learned by DPMORL and baseline methods in Figure 10. The results demonstrate that DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Also, DPMORL learns better 3-dimensional return distributions than GPI-PD, a competitive MORL baseline method. We also show quantitative evaluation metrics under Hopper and FruitTree environments with 3 objectives in Table 3. The results demonstrate that DPMORL achieves better performance in different MORL metrics compared to baseline methods under 3-dimensional reward functions.
π0	π1	π2	π4	π6	π8	π10	π12	π14	π16	π18
π1	π3	π5	π7	π9	π11	π13	π15	π17	π19
Figure 9: Return distribution of the policy set learned by four baseline methods: PGMORL, OLS, GPI-LS, and GPI-PD.
Figure 10: DPMORL can effectively learn DPO policies under more than two-dimensional reward functions. Left: 3-dimensional return distribution of the set of policies learned by DPMORL under Hopper and FruitTree environment. Right: 3-dimensional return distribution under the same environment learned by GPI-PD.
Table 3: Experimental results of each method on Hopper and FruitTree environments with 3-dimensional reward function.
Environment Metric	Expected Utility	CVaR	Hopper Constraint Satisfaction	Variance Objective	Expected Utility	CVaR	FruitTree Constraint Satisfaction	Variance Objective
GPI-LS	1037.77	1286.61	0.15	480.94	4.19	4.55	0.27	1.60
GPI-PD	1511.40	1339.67	0.36	664.02	4.44	6.76	0.20	2.30
OLS	441.62	194.56	0.08	78.77	4.72	6.96	0.35	2.36
DPMORL	2107.26	2840.44	0.67	949.96	5.32	7.25	0.38	2.57
C.6 Ablation Analysis with Variable Number of Policies
We also provide the ablation studies of DPMORL with a varying amount of learned policies  $N$ , as shown in Table 4. We can see that as the number of policies increases, both expected utility and hypervolume increase, which indicates that DPMORL can indeed obtain better performance as the number of policies increases. On the other hand, DPMORL has obtained promising results when there exist only 5 policies, which verify the effectiveness of DPMORL in all environments.
Table 4: Experimental results of DPMORL on each standard MORL metric on five environments with various number of policies. "EU" stands for Expected Utility, and "HV" stands for Hypervolume.
Environment Metric	Hopper	HalfCheetah	MountainCar	DeepSeaTreasure	FruitTree
EU	HV	EU	HV	EU	HV	EU	HV	EU	HV
DPMORL (N=5)	3274.36	9917663.14	817.61	5676147.04	-29.94	7462.10	5.82	8.93	6.22	18.74
DPMORL (N=10)	3309.52	10512291.02	1244.48	8497055.37	-29.94	7465.59	5.86	14.88	6.65	21.64
DPMORL (N=15)	3377.75	11453079.82	1245.63	8497055.37	-29.94	7465.59	5.90	15.02	6.68	23.80
DPMORL (N=20)	3377.75	11453079.82	1749.27	13417685.27	-29.94	7465.59	6.53	19.68	6.68	23.80