SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>12</sup>Our implementation of PGMORL only supports environments with two objectives. This is because the implementation by the original authors relies on different methods for two and three objectives. Hence, PGMORL is only displayed in these two environments.
(a) Evaluation metrics w.r.t. training samples.
Figure 12: Evaluation metrics in environments with continuous actions spaces.
(b) Evaluation metrics w.r.t. training time.
faithful to the original papers instead. The following command was used to generate the plots:
python -m openrlbenchmark.rlops-multi_metric \--filters'?we=openrlbenchmark&wpn=MORL-Baselines&ceik  $\equiv$  env_id&cen  $\equiv$  algo&metrics  $=$  eval/hypervolume&metrics  $\equiv$  eval/sparsity&metrics  $\equiv$  eval/eum'  $'\mathrm{PGMORL?cl} = \mathrm{PGMORL}$  PAPQL?cl  $\equiv$  CAPQL'   
'GPI-LS Continuous Action?cl  $\equiv$  GPI-LS'   
'GPI-PD Continuous Action?cl  $\equiv$  GPI-PD'   
--env-ids mo-halfcheetah-v4 mo-hopper-2d-v4   
--pc.ncols 2   
--pc.ncols-legend 4   
--pc.xlabel 'Training steps'   
--pc.ylabel'   
--pc.max_steps 1000000   
--output-filename morl/morl連續   
--scan-history
(a) Evaluation metrics w.r.t. training samples.
Figure 13: Results on environments with more than two objectives and continuous observation and actions spaces.
The plots in Figure 13a and 13b give the training results for continuous environments with more than 2 objectives. In these environments, there are no clear-cut performance differences between the three algorithms. The plots were generated using the following command:
python -m openrlbenchmark.rlops-multi_metric \--filters'?we=openrlbenchmark&wpn=MORL-Baselinese&ceik  $\equiv$  env_id&cen  $\equiv$  algo&metrics  $=$  eval/hypervolume&metrics  $\equiv$  eval/sparsity&metrics  $\equiv$  eval/eum' \NCAPQL?cl  $\equiv$  CAPQL' \NGPI-LS Continuous Action?cl  $\equiv$  GPI-LS' \NGPI-PD Continuous Action?cl  $\equiv$  GPI-PD' \--env-ids mo-hopper-v4 mo-lunar-lander-continuous-v2 \--pc.ncols 2\--pc.ncols-legend 3\--pc.xlabel 'Training steps' \--pc.ylabel '\--output-filename morl/morl連續ious_more \--scan-history
Continuous observations and discrete actions
Figure 14 and 15 give the training results of MORL algorithms compatible with continuous observations and discrete actions. We can see that GPI-based methods outperform Envelope (Yang et al., 2019) and PCN (Reymond et al., 2022) for EU and Hypervolume. Yet, there is no performance difference between the model-based algorithm (GPI-PD) and the model-free one (GPI-LS) in this environment. It is worth noting that the performances of Envelope could be enhanced by tuning hyperparameters or using modern deep RL tricks such as dropout layers and normalization, as done in Alegre et al. (2023). Indeed, we believe that this algorithm suffers from pathologies that it inherits from DQN (Mnih et al., 2015a), such as overestimation bias. Finally, remember that PCN is designed for deterministic environments and thus is not really fit for the tested domains. Still, it performs
Figure 14: Evaluation metrics on environments with discrete actions and continuous observations w.r.t training samples.
Figure 15: Evaluation metrics on environments with discrete actions and continuous observations w.r.t training time.
(a) Evaluation metrics w.r.t training samples.
Figure 16: Results on minecart-v0; discrete actions, continuous observations and known Pareto front.
(b) Evaluation metrics w.r.t training time.
surprisingly well on mo-lunar-lander. The plots were generated using the following command:
python -m openrlbenchmark.rlops-multi_metric \--filters'?we=openrlbenchmark&wpn=MORL-Baselinese&ceik  $\equiv$  env_id&cen  $\equiv$  algo&metrics  $=$  eval/hypervolume&metrics  $\equiv$  eval/sparsity&metrics  $\equiv$  eval/eum' \ 'Envelope?cl  $\equiv$  Envelope' \PCN?cl  $\equiv$  PCN' \GPI-LS?cl  $\equiv$  GPI-LS' \GPI-PD?cl  $\equiv$  GPI-PD' \--env-ids mo-lunar-lander-v2 mo-highway-fast-v0 mo-reacher-v4 \--pc.ncols 3\--pc.ncols-legend 4\--pc.xlabel 'Training steps' \--pc.ylabel '\--pc.max_steps 400000\--output-filename morl/morl_discrete\--scan-history
The performance of the algorithms on the minecart-  $\nu 0$  environment, which has a known Pareto front enabling the computation of IGD and MUL, is depicted in Figure 16a and Figure 16b. Similar
Figure 17: Evaluation metrics on deterministic environments with discrete actions and observations w.r.t. training samples.
to the previous plot, we can draw the same conclusion: GPI-based methods (especially GPI-PD) appear to be more sample efficient than Envelope and PCN. However, PCN demonstrates an advantage with its efficient implementation, making it indistinguishable from GPI-based methods when evaluating w.r.t. training time (Figure 16b). The following command line generated these plots:
python -m openrlbenchmark.rlops-multi.metrics \
--filters'?we=openrlbenchmark&wpn=MORL-Baselinese&ceik  $\equiv$  env_id&cen  $\equiv$  algo&metrics= eval/hypervolume&metrics  $\equiv$  eval/sparsity&metrics  $\equiv$  eval/eum&metrics  $\equiv$  eval/igd& metrics  $\equiv$  eval/mul' \
'Envelope?cl=Envelope' \
'PCN?cl=PCN' \
'GPI-LS?cl=GPI-LS' \
'GPI-PD?cl=GPI-PD' \
--env-ids minecart-v0 \
--pc.ncols 1 \
--pc.ncols-legend 4 \
--pc.xlabel 'Training steps' \
--pc.ylabel ’\\
--pc.max_steps 400000 \
--output-filename morl/morl_discreteknown_pf \
--scan-history
Discrete action and discrete observations
Figure 17 and 18 illustrate the performance of tabular algorithms trained on environments with discrete actions and observations. Notice that these runs also include more metrics as they involve environments with known Pareto front. In all environments, Pareto Q-Learning (Van Moffaert and
Figure 18: Evaluation metrics on deterministic environments with discrete actions and observations w.r.t. training time.
Nowé, 2014) finds optimal Pareto front, as shown by the IGD and MUL metrics reaching 0. Yet, our implementation of this algorithm is limited to deterministic environments; this could be enhanced by using the work of (Rojers et al., 2021). The MO Q-Learning (Van Moffaert et al., 2013a) variants, described in Section 5, seem to plateau in deep-sea-treasure-concave as they rely on linear scalarization and cannot find the points in the concave part of the front. Finally, the GPI-LS variant of MO Q-Learning shows strong performance in all environments compared to the other variants.
The plots were generated using the following command: python -m openrlbenchmark.rlopsmulti_metricss --filters'?we=openrlbenchmark&wpn=MORL-Baselinese&ceik  $\equiv$  env_id&cen  $\equiv$  algo&metrics  $=$  eval/hypervolume&metrics  $\equiv$  eval/igd&metrics  $\equiv$  eval/sparsity&metrics  $\equiv$  eval/eum& metrics  $\equiv$  eval/mul'   
'Pareto Q-Learning?cl  $\equiv$  Pareto Q-Learning'   
'MultiPolicy MO Q-Learning?cl  $\equiv$  MultiPolicy MO Q-Learning'   
'MultiPolicy MO Q-Learning (OLS)?cl  $\equiv$  MultiPolicy MO Q-Learning (OLS)'   
'MultiPolicy MO Q-Learning (GPI-LS)?cl  $\equiv$  MultiPolicy MO Q-Learning (GPI-LS)'   
--env-ids deep-sea-treasure-v0 deep-sea-treasure-concave-v0 fruit-tree-v0   
--pc.ncols 3   
--pc.ncols-legend 4   
--pc.xlabel 'Training steps'   
--pc.ylabel'   
--pc.max_steps 400000   
--output-filename morl/mor1_deterministic_envs   
--scan-history