SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1 lists the algorithms currently supported by MORL-Baselines and the MORL settings they tackle. Algorithms employing neural networks as function approximators were implemented using PyTorch (Paszke et al., 2019), and tabular algorithms rely on NumPy (Harris et al., 2020). Algorithms in Table 1 are described according to whether they produce a single policy (based on user-provided utility functions) or multiple policies (i.e., to approximate a CCS or a PF). Additionally, notice that some algorithms optimize w.r.t. the ESR while others optimize w.r.t. SER (Section 3). Finally, notice that MORL-Baselines' algorithms may support different observation and action spaces (e.g., images).
Tabular algorithms. Multi-objective Q-learning (MOQL) (Van Moffaert et al., 2013b) is an extension of the classic tabular Q-learning algorithm (Watkins, 1989) that learns and stores the Q-values of each objective separately. A scalarization function is then used to convert these Q-values into a scalar quantity, allowing agents to select an action. Multi-policy MOQL, which consists of running MOQL multiple times with different preferences, can be instantiated using different methods that select which preference weight vector will be optimized next by a corresponding specialized policy. Currently, MORL-Baselines supports randomly generated weight vectors, Optimistic Linear Support (OLS) (Roijers, 2016), and Generalized Policy Improvement Linear Support (GPI-LS) (Alegre et al., 2023). Pareto Q-learning (PQL) (Van Moffaert and Nowé, 2014) aims at simultaneously learning all policies in the Pareto front by storing sets of non-dominated Q-values. These sets are then converted into scalars (using metrics similar to the ones discussed in the next section) that guide the selection of actions during the learning phase. This algorithm is only compatible with deterministic environments.
Deep MORL algorithms. The Expected Utility Policy Gradient (EUPG) algorithm (Roijers et al., 2018b) introduced a policy gradient update capable of taking into account both the return achieved up to the current moment, as well as future returns, in order to determine expected utilities in an ESR setting. However, although it employs a neural network as the policy, it has only been
evaluated in discrete settings. The Envelope algorithm (Yang et al., 2019) uses a single neural network conditioned on a weight vector (Abels et al., 2019) to approximate the CCS. Prediction-Guided MORL (PGMORL) (Xu et al., 2020) is an evolutionary algorithm that maintains a population of policies learned using PPO (Schulman et al., 2017b). This algorithm focuses on predicting, at each iteration, the most promising weight vectors and policies to select for further training in order to more effectively enhance the PF. Pareto Conditioned Networks (PCN) (Reymond et al., 2022) employs a neural network conditioned on a given desired return per objective. This network is trained via supervised learning to predict which actions produce the desired return in deterministic environments. GPI-LS (Alegre et al., 2023) employs GPI (Barreto et al., 2017) to combine policies in its learned CCS and prioritize the weight vectors on which agents should train at each moment. GPI-Prioritized Dyna (GPI-PD) is a model-based extension of GPI-LS that uses a learned model of the environment and GPI to prioritize experiences in the replay buffer. Concave-Augmented Pareto Q-learning (CAPQL) (Lu et al., 2023) employs a multi-objective extension of SAC (Haarnoja et al., 2018) by conditioning the actor and critic networks on the rewards weight vector.
5.2 Evaluation metrics
Recall that in single objective RL settings, policies are evaluated in terms of their corresponding expected returns. In MORL settings, by contrast, they are typically evaluated using multi-objective metrics computed based on Pareto fronts. While these metrics are widely adopted by the multi-objective optimization community, the MORL community has yet to establish a consensus on which metrics should be preferred in each particular problem or setting. For this reason, our framework supports all commonly-used MORL metrics—see below. These metrics can be split into two groups: utility-based metrics, which assume particular properties of the utility function (e.g., linearity), and axiomatic metrics, which do not make assumptions but may produce less informative performance information to users. For a thorough discussion of these metrics, see Hayes et al. (2022).
Expected utility  $(\uparrow)$ . If the utility function  $u$  is linear, it is possible to express the expected utility over a distribution of rewards weights,  $\mathcal{W}$ , via the expected utility (EU) metric (Zintgraf et al., 2015). Let  $\Pi$  be a set of policies and  $\tilde{\mathcal{F}} = \{\mathbf{v}^{\pi}|\pi \in \Pi \}$  be its corresponding approximate PF. Then, the EU metric is defined as:
$$
\operatorname {E U} (\tilde {\mathcal {F}}) = \mathbb {E} _ {\mathbf {w} \sim \mathcal {W}} \left[ \max  _ {\mathbf {v} ^ {\pi} \in \tilde {\mathcal {F}}} \mathbf {v} ^ {\pi} \cdot \mathbf {w} \right].
$$
Maximum utility loss  $(\downarrow)$ . Zintgraf et al. (2015) introduced this metric to quantify the maximum utility loss that results from using an approximate PF,  $\tilde{\mathcal{F}}$ , rather than a reference PF,  $\mathcal{Z}^5$ . It is defined as follows:
$$
\operatorname {MUL}(\tilde{\mathcal{F}},\mathcal{Z}) = \max_{\mathbf{w}\in \mathcal{W}}\bigl(\max_{\mathbf{v}^{*}\in \mathcal{Z}}\mathbf{v}^{\pi^{*}}\cdot \mathbf{w} - \max_{\mathbf{v}^{\pi}\in \tilde{\mathcal{F}}}\mathbf{v}^{\pi}\cdot \mathbf{w}\bigr).
$$
Inverted generational distance  $(\downarrow)$ . This metric characterizes the convergence rate of an approximate PF,  $\tilde{\mathcal{F}}$ , towards a reference PF,  $\mathcal{Z}$  (Coello Coello and Reyes Sierra, 2004). If the reference front is unknown, it is usually defined/constructed by aggregating the best value vectors observed after several executions of the underlying RL algorithm. The inverted generation distance (IGD) is computed as:
$$
\operatorname {I G D} (\tilde {\mathcal {F}}, \mathcal {Z}) = \frac {1}{| \mathcal {Z} |} \sqrt {\sum_ {\mathbf {v} ^ {*} \in \mathcal {Z}} \min _ {\mathbf {v} ^ {\pi} \in \tilde {\mathcal {F}}} \| \mathbf {v} ^ {*} - \mathbf {v} ^ {\pi} \| ^ {2}}.
$$
Sparsity  $(\downarrow)$ . This metric characterizes the diversity of the policies in a given PF. A diverse set of policies allows users to choose from qualitatively different behaviors based on the trade-offs they may wish to optimize (Xu et al., 2020). The sparsity of an approximate PF,  $\tilde{\mathcal{F}}$ , is given by:
$$
\mathrm {S} (\tilde {\mathcal {F}}) = \frac {1}{| \tilde {\mathcal {F}} | - 1} \sum_ {j = 1} ^ {m} \sum_ {i = 1} ^ {| \tilde {\mathcal {F}} | - 1} \left(\mathcal {L} _ {j} (i) - \mathcal {L} _ {j} (i + 1)\right) ^ {2},
$$
where  $\mathcal{L}_j$  is the sorted list of the values of the  $j$ -th objective considering all policies in  $\tilde{\mathcal{F}}$ , and  $\mathcal{L}_j(i)$  is the  $i$ -th value in  $\mathcal{L}_j$ .
5If the true PF of the MOMDP,  $\mathcal{F}$  , is known, then typically  $\mathcal{Z} = \mathcal{F}$
Hypervolume  $(\uparrow)$ . This is a hybrid metric quantifying both a PF's rate of convergence and the diversity of the policies in it. Given an approximate PF,  $\tilde{\mathcal{F}}$ , and a reference point,  $\mathbf{v}_{\mathrm{ref}}$ , the hypervolume metric (Zitzler, 1999) is defined as:
$$
\mathrm {H V} (\tilde {\mathcal {F}}, \mathbf {v} _ {\mathrm {r e f}}) = \bigcup_ {\mathbf {v} ^ {\pi} \in \tilde {\mathcal {F}}} \mathrm {v o l u m e} (\mathbf {v} _ {\mathrm {r e f}}, \mathbf {v} ^ {\pi}),
$$
where  $\mathrm{volume}(\mathbf{v}_{\mathrm{ref}},\mathbf{v}^{\pi})$  is the volume of the hypercube spanned by the reference vector,  $\mathbf{v}_{\mathrm{ref}}$ , and the vector,  $\mathbf{v}^{\pi}$ . The reference point used in the hypervolume computation is typically an estimate of the worst-possible return per objective. For instance, one can use as reference point the vector  $\frac{\mathbf{r}_{\mathrm{min}}}{1 - \gamma}$  where  $\mathbf{r}_{\mathrm{min}}$  is a vector with the minimum value for each objective in any given possible state.
5.3 Additional features
In addition to providing implementations of the algorithms and evaluation metrics discussed above, MORL-Baselines also provides a range of tools to assist researchers in designing new algorithms. These resources include a data structure for storing Pareto fronts and Pareto filtering functions, methods for policy evaluation with automated metric and PF reporting via Weights and Biases (Biewald, 2020), scalarization functions, various neural network architectures, as well as methods for generating videos of policies trajectories (e.g., via GIF files). For a comprehensive discussion of these functionalities, we refer the reader to Appendix C.2.
6 Benchmark dataset
Our final contribution is a comprehensive collection of benchmark results evaluating MORL-Baselines algorithms on a diverse set of MO-Gymnasium environments. These results are compiled in a dataset hosted in and integrated into the openrlbenchmark framework (Huang et al., 2023). Access to such a dataset enables users performing new experiments to combine, compare, and aggregate the corresponding new results with the ones in the dataset, visualizing them in real-time using Weights and Biases (W&B) dashboards. They also allow users to manipulate raw data based on subsequent analyses, compare performance metrics of new algorithms w.r.t. state-of-the-art techniques without having to retrain the latter, and store the associated hyperparameters and command lines for each run. For instance, users can access the approximate Pareto fronts identified by each algorithm in our benchmark via the W&B dashboard, under the panel named 'eval/front'.
6.1 Proof-of-concept experiments using MORL-Baselines and MO-Gymnasium
In this section, we use MORL-Baselines and MO-Gymnasium to perform various proof-of-concept empirical experiments and discuss their corresponding results. Our goal is to highlight the type of experiments, comparisons, and analyses—of different algorithms on various domains, and under different performance metrics—that are possible using our framework. Given the potentially high cost of training algorithms, we performed 10 runs of each algorithm on each environment. Notice that while experimental results with tighter confidence intervals may be conducted (by performing more runs), our goal here is solely to highlight the capabilities of our libraries—not to fully evaluate all existing algorithms. The hyperparameters used on each run can be found in the W&B dashboards. $^{7}$  Examples of the W&B user interface through which our benchmark dataset can be analyzed are shown in the appendices. $^{8}$  Notice that some algorithms were trained over more timesteps than others since they may be fast but not as sample efficient; e.g., PGMORL relies on PPO (Schulman et al., 2017a), which is computationally faster but less sample efficient than GPI-LS, which relies on TD3 (Fujimoto et al., 2018a).
Figure 3: Performance of tabular MORL algorithms on deep-sea-treasure-v0 w.r.t. training samples.
Figure 4: Performance of MORL algorithms on the minecart-v0 domain w.r.t. training samples.
6.2 Results and discussion of the proof-of-concept experiments
Figures 3, 4, 5, and 6 present the performance of various MORL-Baseline algorithms when evaluated on a few representative MO-Gymnasium environments. They show, in particular, the mean and corresponding  $95\%$  confidence intervals with respect to the evaluation metrics introduced in Section 5.2. In the Appendix, we provide the complete set of experimental results and comparisons.
Figure 3 presents the performance of various tabular MORL algorithms available on MORL-Baselines, when evaluated in the classic deep-sea-treasure domain. Notice that the PQL and GPI-LS algorithms have similar performance and achieve near-zero maximum utility loss and Inverted Generational Distance. This indicates that they successfully identify the true Pareto front.
Figure 4 depicts the performance of various MORL algorithms that support discrete action spaces in minecart- $\nu$ 0. GPI-PD significantly outperforms all other baselines (when considering all performance metrics) in terms of sample efficiency and asymptotic performance. This empirical result supports recent observations by Alegre et al. (2022a) about the crucial role of efficient prioritization schemes when constructing PFs and selecting training experiences. Interestingly, notice that even though PCN was designed to tackle deterministic MOMDPs, it still outperforms the Envelope algorithm.
Figure 5: Performance of MORL algorithms on the mo-halfcheetah-  $\nu 4$  domain. Pareto fronts were constructed by identifying (across all runs) the front with the highest Expected Utility after a given training budget (5M steps for PGMORL, 100k for GPI-PD, 200k for the others).
Finally, Figures 5 and 6 present the performance of MORL algorithms that support continuous action spaces in the mo-halfcheetah-  $\nu 4$  and mo-hopper-2d-  $\nu 4$  domains, respectively. In mo-halfcheetah,
These results can be viewed and analyzed at https://wandb.ai/openrlbenchmark/MORL-Baseline.
<sup>7</sup>We also include a description of the high-performance computers at the University of Luxembourg (Varrette et al., 2014) and Vrije Universiteit Brussel on which experiments were conducted. Training all algorithms on all environments, using various random seeds, required approximately 3 months of computation time.
An overview of the command lines used to conduct each experiment can be found at https://github. com/LucasAlegre/morl-baselines/issues/43.
Figure 6: Performance of MORL algorithms on mo-hopper-2d-v4 w.r.t. training time (in minutes).
GPI-based algorithms perform better w.r.t. the hypervolume and expected utility metrics. PGMORL and CAPQL, by contrast, perform better w.r.t. the Sparsity metric—they are able to identify denser Pareto fronts, as shown in the rightmost plot. Recall that sample efficiency often comes at the cost of execution time. To investigate this trade-off, we show, in Figure 6, performance results in the mo-hopper domain with respect to training time (in minutes). These experiments suggest that PGMORL performs well in terms of training time, in contrast to previous experiments where it had a qualitatively different behavior when analyzed with respect to sample complexity. In these experiments, both GPI-based algorithms still dominated all other baselines.
We introduced a comprehensive collection of software libraries for reliable benchmarking and research in MORL. MO-Gymnasium and MORL-Baselines are the first libraries providing standardized APIs and extensible sets of environments and state-of-the-art MORL algorithms. Our framework also includes a thorough set of benchmark results comparing a wide range of algorithms in various environments. These can be used as guidelines by the community, underscoring the properties and limitations of different techniques. By making this toolkit open source and extensible—and thus open to contributions from other researchers—we hope to provide a solid foundation for reproducible research in MORL. This work will allow researchers and end users to seamlessly deploy existing algorithms on various MORL domains, speeding up experiments, facilitating algorithm evaluation, and being more conducive to reproducible experimental results.
Limitations and future work. In the benchmarks we presented in this paper, we did not perform exhaustive hyperparameter tuning. Our main goal with these experiments was to validate our implementations and showcase the learning behavior of the algorithms in our framework with respect to different evaluation metrics. For this reason, such empirical results should be interpreted as proof-of-concept and may still be further improved with the aid of the MORL community. In future work, we will continue maintaining all algorithms in MORL-Baselines, and plan to augment the set of techniques available through our framework. We also plan to introduce new features for automating hyperparameter tuning. We do not anticipate any negative societal impacts of this work.
We would like to thank Willem Röpke for his implementation of PQL, Denis Steckelmacher and Conor F. Hayes for the original implementation of EUPG, and Mathieu Reymond for the original implementation of PCN. Jordan K. Terry, Mark Towers, Manuel Goulão, and the broader Farama Foundation team for supporting MO-Gymnasium. Shengyi Huang, Antonin Raffin, and Quentin Gallouédec for their advice on experimental setup and openrlbenchmark integration. This work was funded by: Coordenação de Aperfeçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Finance Code 001; CNPq (Grants 140500/2021-9, 304932/2021-3); FAPESP/MCTI/CGI (Grant 2020/05165-1); the Fonds National de la Recherche Luxembourg (FNR), CORE program under the ADARS Project, ref. C20/IS/14762457; the Research Foundation Flanders (FWO) [G062819N]; the AI Research Program from the Flemish Government (Belgium); and the Francqui Foundation.
Varrette, S., Bouvry, P., Cartiaux, H., and Georgatos, F. (2014). Management of an academic HPC cluster: The UL experience. In 2014 International Conference on High Performance Computing & Simulation (HPCS), pages 959-967. IEEE. 7, D.1
Watkins, C. (1989). Learning from Delayed Rewards. PhD thesis, University of Cambridge. 5.1
Xu, J., Tian, Y., Ma, P., Rus, D., Sueda, S., and Matusik, W. (2020). Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning (ICML). 1, 2, 4, 1, 5.1, 5.2, D.2
Yang, R., Sun, X., and Narasimhan, K. (2019). A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 14610-14621. 2, 4, 1, 5.1, B.1, B.6, B.16, D.2
Yoo, A. B., Jette, M. A., and Grondona, M. (2003). SLURM: Simple Linux Utility for Resource Management. In Feitelson, D., Rudolph, L., and Schwiegelshohn, U., editors, Job Scheduling Strategies for Parallel Processing, Lecture Notes in Computer Science, pages 44-60, Berlin, Heidelberg. Springer. D.1
Zhu, B., Dang, M., and Grover, A. (2023). Scaling pareto-efficient decision making via offline multi-objective RL. In *In Proceedings of The Eleventh International Conference on Learning Representations*. 2
Zintgraf, L. M., Kanters, T. V., Roijers, D. M., Oliehoek, F. A., and Beau, P. (2015). Quality assessment of MORL algorithms: A utility-based approach. In *Benelearn 2015: Proceedings of the 24th Annual Machine Learning Conference of Belgium and the Netherlands.* 5.2
Zitzler, E. (1999). Evolutionary algorithms for multiobjective optimization: methods and applications. In Ph.D. Dissertation. ETH Zurich, Switzerland. 5.2
This appendix contains supplementary material about MO-Gymnasium and MORL-Baselines. In Section A, we provide general information such as the links to our libraries, licenses, and maintenance plan. In Section B, we describe in detail all environments available in MO-Gymnasium, and in Section C we present all features of MORL-Baselines. Finally, in Section D, we provide in more detail the experimental setting and results of our benchmarks.
A General information
A.1 Links
The MO-Gymnasium documentation is accessible at: https://mo-gymnasium.farama.org and its code repository at: https://github.com/Farama-Foundation/MO-Gymnasium.
The MORL-Baseline documentation is available at: https://lucasalegre.github.io/morl-baseline and its code repository at: https://github.com/LucasAlegre/morl-baseline.
The openrlbenchmark code and early documentation are available at: https://github.com/openrlbenchmark/openrlbenchmark.
Our training metrics and results, hosted by Weights and Biases, are available at: https://wandb.ai/openrlbenchmark/MORL-Baselines.
A.2 Licenses
MO-Gymnasium, MORL-Baseline, and openlbenchmark are under the MIT License. We, the authors, bear all responsibility in case of violation of rights.
A.3 Maintenance
MO-Gymnasium is maintained by the authors, with the support of the Farama Foundation team. MORL-Baseline is currently maintained by the authors, with the support of the community. openrl-benchmark is still under development, yet a large community of researchers is already using it to upload RL training data.
B MO-Gymnasium
Below, we provide a description of each environment currently available in MO-Gymnasium. Figure 7 presents a visualization of each environment. A comprehensive list and documentation of the environments available through the MO-Gymnasium API, along with the definition of their observation and action spaces, can be found on the documentation website: https://mo-gymnasium.farama.org/environments/all-environments. We have plans to introduce more challenging environments in the future.
B.1 deep-sea-treasure-v0
This domain, originally proposed by Vamplew et al. (2011), is a classic MORL problem in which the agent is a submarine that must collect a treasure while taking into account a time penalty. At each step, the agent perceives its current location as  $x, y$  coordinates in a  $11 \times 11$  grid world, and its actions correspond to movement in one of the four cardinal directions. Hence, in this domain, the state space is defined as  $S = \{0, \dots, 10\}^2$ , and the action space is  $\mathcal{A} = \{\text{up}, \text{down}, \text{left}, \text{right}\}$ . We use the same treasures' values as defined by Yang et al. (2019). A version of this domain with a concave Pareto frontier (Vamplew et al., 2011) is also available with identifier deep-sea-treasure-concave-v0. The
$^{9}$ https://farama.org/team.
(a) deep-sea-treasure-v0
(b) resource-gathering-v0
(c) breakable-bottles-v0
(e) four-room-v0
(f) fruit-tree-v0
(g) mo-mountaincar-v0
(d) fishwood-v0
(h)minecart-v0
(1) water-reservoir-vO
(i) mo-lunar-lander-v2
(m) mo-reacher-v4
Figure 7: Visualization of the environments currently available in MO-Gymnasium.
(j) mo-highway-v0
(n) mo-hopper-v4
(k) mo-highway-fast-v0
(o) mo-halfcheetah-v4
(p) mo-supermario-v0
multi-objective reward function of this domain,  $\mathbf{r}(s,a,s^{\prime})\in \mathbb{R}^{2}$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {v a l u e o f t r e a s u r e i n s t a t e} s ^ {\prime},
$$
$$
r _ {2} (s, a, s ^ {\prime}) = - 1.
$$
B.2 resource-gathering-v0
In this domain, an agent must collect the gold or gem available in a  $5 \times 5$  grid world while avoiding being killed by enemies (Barrett and Narayanan, 2008). The state space is defined as  $\mathcal{S} = \{0, \dots, 4\}^2 \times \{0, 1\}^2$  and denotes the agent's current position and two binary variables representing whether the agent carries gold or diamond. Its action space representing movements in the four cardinal directions is  $\mathcal{A} = \{\mathrm{up}, \mathrm{down}, \mathrm{left}, \mathrm{right}\}$ . There are two enemies that have a  $10\%$  chance of killing the agent if it is in the same cell. The multi-objective reward function of this domain,  $\mathbf{r}(s, a, s') \in \mathbb{R}^3$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = - 1 \text {i f k i l l e d b y e n e m y i n} s ^ {\prime}, \text {e l s e} 0,
$$
$$
r _ {2} (s, a, s ^ {\prime}) = + 1 \text {i f r e t u r n e d h o m e w i t h g o l d , e l s e 0},
$$
$$
r _ {3} (s, a, s ^ {\prime}) = + 1 \text {i f r e t u r n e d h o m e w i t h g e m , e l s e 0 .}
$$
B.3 breakable-bottles-v0
This domain consists of a grid corridor with 5 cells. The agents must collect bottles from the source location and deliver them to the destination cell (Vamplew et al., 2021). Its action space is defined as  $\mathcal{A} = \{\text{right, left, pick_up_bottle}\}$ . Every time step, the agent has a  $10\%$  chance of dropping one of the
bottles it is currently carrying. The multi-objective reward function of this domain,  $\mathbf{r}(s,a,s^{\prime})\in \mathbb{R}^{3}$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = - 1,
$$
$$
r _ {2} (s, a, s ^ {\prime}) = + 2 5 \times \# \text {b o t t l e s}
$$