A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning
Authors: Florian Felten
Lucas Nunes Alegre
Ann Nowe
Ana L. C. Bazzan
El Ghazali Talbi
Grégoire Danoy
Bruno Castro da Silva
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Multi-Objective
Reinforcement Learning
Benchmarking
TL;DR: We introduce a toolkit for reproducible research in Multi-Objective RL: (i) a set of environments under a standardized API; (ii) a set of algorithms implementations compatible with our API; and (iii) a benchmark of the implemented algorithms.

Abstract: Multi-objective reinforcement learning algorithms (MORL) extend standard reinforcement learning (RL) to scenarios where agents must optimize multiple---potentially conflicting---objectives, each represented by a distinct reward function. To facilitate and accelerate research and benchmarking in multi-objective RL problems, we introduce a comprehensive collection of software libraries that includes: 
(i) MO-Gymnasium, an easy-to-use and flexible API enabling the rapid construction of novel MORL environments. It also includes more than 20 environments under this API. This allows researchers to effortlessly evaluate any algorithms on any existing domains; (ii) MORL-Baselines, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed to provide a solid foundation for advancing research. Notably, all algorithms are inherently compatible with MO-Gymnasium; and
(iii) a thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms, tested across various challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular state-of-the-art method.

A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning
Florian Felten +1,7 Lucas N. A 223,7 Ann Nowe Ana L. C. Bazzan 2
El-Ghazali Talbi Grégoire Danoy Bruno C. da Silva
SnT, University of Luxembourg lInstitute of Informatics, Federal University of Rio Grande do Sul 3 Artiifiiil Intelligence Lab, Vrije Universiteit Brussel 'CNRS/CRISIAL, University of Lille $FSTM/DCS, University of Luxembourg University of Massachusetts Farama Foundation (florian. felten, gregoire danoy) .uu (Inalegre, bazzan]@ini Janaaegre,bazzan| .finf.frggs br ann now.nowe@vub..o be el el-ghazali, talbi@univ-lille.ee bsilva@cs. umass. edu
Abstract
Multi-objective reinforcement learning algorithms (MORL) extend standard re- inforcement learning (RL) scenarios where agents must optimize multiple- potentially -olietttin-ooeetiiveee..... each represented by distinct reward function. To facilitate and accelerate research and benchmarking multi-objective RL prob- lems, we introduce a comprehensive collection of software libraries that includes: (i) MO-Gymnasium, an easy-to-use and flexible API enabling the rapid construc- tion of novel MORL environments. It also includes more than 20 environments under this API. This allows researchers to effortlessly evaluate any algorithms on any existing domains; (ii) MORL-Baselines, collection of reliable and efficient implementations state-of-the- MORL algorithms, designed to provide a solid foundation for advancing research. Notably, all algorithms inherently compati- ble with H0-Gymnasium; and (iii) a thorough and robust set benchmark results and comparisons of MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations each particular state-of-the-ar method.?
algorithms,, tested across various challenging
Introduction
Research in reinforcement learning (RI.) algorithms (Sutton and Barto, 2018) has gained significant attention in recent years, in great part due to its remarkable success in range of challenging problems (Mnih et al., 2015b; Silver et al., 2017: Bellemare et al., 2020). This led to a substantial increase in the number of papers published in the field every year. The rapid growth of RL research, however, has not necessarily been accompanied by the design of well-thought and reliable tools allowing for appropriate evaluation practices, resulting in reproducibility crisis within the field. This resulted, e.g., in researchers often questioning the validity and reproducibility of results presented in influential papers (Agarwal et al., 2021; Patterson et al., 2023).
There are several reasons why reproducibility is often a challenge in RL. A key factor is the significant amount of time required to train RI. agents, which makes it difficult for researchers to gather sufficient data to perform rigorous statistical analyses of empirical results. As a consequence, some authors may claim superior performance over state-of-the-an techniques without adequate evidence.
Authors contributed equally to this work.
MMO-G Gymnasium and MORL-Bascling are available htttps///githei http://ggthuuu.ccooaaaaaaaaaaa Foundation/ mo- gymnas and httpp:://g/ //s::///ithu com http/LccasAlegre/m/ezzee mosrl-bsselines... respectivlly The benchmark results are available on openrlbenchmark: https:/ htaps:://wanb..... ai ht///paaarbeencaaar/k/aaaa Basel ines.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Additionally, papers often do not provide sufficient ceormattoonssllee as hyperparameter values and implementation optimizations-to allow for the reliable reproducibility of their results. Finally, the use of environment implementations that lack standardization also contributes to this challenge. To address such issues, several libraries and experimentation protocols have been proposed in the RI. ecosystem (Brockman et al., 2016; Agarwal et al., 2021; Raffin et al., 2021; Huang et al., 2022b).
As RL garners increasing interest, its subfield. Multi- Majee-Obeeettvve Reinforcement Learning (MORL), is simultancously attracting notable attention within the RL research community (Hayes et al., 2022). MORL algorithms tackle problems where an agent has to optimize multiple-possibly conflicting objectives. Each of these objectives is represented via a distinct reward function. In this setting, agents typically aim to find optimal decision- making policies defined with respect to different compromises or trade-offs between the objectives. Examples of MORL problems include the widely-used set of Mujoco (Todorov et al., 2012) tasks, which model compromises hetween an agent's objectives. As an example, the reward function of the half-cheetah agent is weighted combination of velocity- and energy- terms, where the weights are pre-determined and kept constant. However, allowing for such weights (i.e., for the relative importance of objectives) to change may induce significantly different optimal behaviors (Xu et al., 2020). While the goal of standard RL algorithms is to learn a single policy, specialized in optimizing a single reward function, MORL algorithms typically search for a set of policies such that given any compromises between objectives, a corresponding optimal (or (par-oltimal)) policy is in the set. Designing and properly evaluating MORL algorithms shares the difficulties encountered when designing and empirically testing RL techniques wwith with additional challenges particular to multi-objective settings. Yet, to the best of our knowledge, there currently no public standard libraries providing reliable implementations of widely-used MORL domains and ofte-offtheear MORL algorithms, designed particularly to facilitate research in the field.
In this paper, we introduce a comprehensive suite of benchmark MORL environments and reliable implementations of widely-used MORL algorithms, designed to facilitate the construction of repro- ducible empirical performance evaluation of existing and novel multi-objective techniques. First, we introduce MO-Gymnasium (Section 4), an easy-to-use and flexible API enabling the rapid construc- tion of novel MORL environments. As of now, this API includes over 20 environments with diverse characteristics. This allows researchers to evaluate, with minimal effort, any algorithms compatible with our extendable API in any existing domains. Secondly, we introduce MORL-Bas BaRelBasllss (Sec- tion 5), collection of reliable and efficient implementations of state-of-the-art MORL algorithms designed to provide a solid foundation for advancing MORL research. Notably, all such algorithms inherently compatible with MO-Gymnasium. Finally, in Section 6 we provide thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms tested across var- ious challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular sute-of-the-as thttarttm method.
2 Related work
The challenge of reproducing experimental results in machine learning research is widely acknowl- edged. The significance of this issue has risen such a degree that top-tier conferences like NeurIPS have implemented reproducibility programs aimed at improving the standards for conducting, com- municating, and evaluating research in the field (Pineau al., 2020). The RL field is directly affected by this issue, as highlighted by different authors. For example, Engstrom et al. (2020) show that optimizations at the implementation level of deep RL. algorithms can be more impactful than ex- perimenting with different algorithms. Similarly, Huang et al. (2022a) investigated and identified the existence of 37 particular code-level tricks necessary to achieve statc-of-the-an results when deploying the PPO algorithm (Schulman et al., 20)7a). Agarwal et al. (2021) point out the lack of statistically significant results in numerous papers published in the field, despite many of them claiming to introduce techniques that outperform the tffeeofftthe-at Finally, an in-depth discussion of issues pertaining to methodologies used to conduct empirical evaluation in RL is presented by Patterson et al. (2023). These issues include, among others, inadequate tuning of the hyperparameters of baselines algorithms when deployed on new environments, averaging performance metrics over a limited number of runs, absence of random seed control, and environment overfitting.
To address these concerns, several RL libraries have been developed to provide more reliable baselines. For example, Gymnasium (Towers et al., 2023) (formerly known as OpenAI Gym (Brockman et al.,

---

2016)) provides a standard API and a collection of reference environments for RL research and experimentation. Furthermore, various libraries have been published that contain well-tested, reliable, and continually maintained implementations of RI. algorithms;; e.g., Kable-Baselliee 3 (Raffin et al., 2021) and cleanRI. (Huang al., 2022b). Finally, recent initiatives such as openrlhenchmark (Huang et al., 2023) facilitate the analysis of various learning metrics, and tackle the reproducibility challenge via experiment tracking software such as Weights and Biases, which makes it easier to, e.g., make hyperparameters of different baselines publicly available (Biewald, 2020). Other subfields of RL, such as multi-agent RL (MARL), have also benefited from standard baseline libraries. In MARL, PettingZoo (Terry et al., 2021) is often used to design novel environments, and EPyMARL (Papoudakis et al., 2021) is commonly used when designing new learning algorithms. Zhu et al. (2023) recently introduced D4MORL, repository that includes datasets specifically designed to evaluate MORI. algorithms in the offline setting. Although these subareas of RI have reaped the benefits from standardized libraries, to the best of our knowledge, there are currently no publicly available (and widely adopted) standard libraries providing reliable implementations of widely-used MORL domains and state-of-the-ar MORL algorithms, designed to facilitate research in the field.
As discussed by Hayes et al. (2022); Cassimon et al. (2022), various benchmark problems have been proposed to evaluate MORL methods. However, these benchmarks have not yet been made available via standardized APIs or centralized repositories. Arguably, this has made the experimental reproducibility of MORL algorithms harder, time-consuming, and error-prone. MORL-GIA (Vam- plew et al., 2017) represents an attempt to establish a centralized repository of MORL benchmarks. However, this library has not been widely adopted due to the fact that it is implemented in Java and targets tabular problems, whilst the community currently focuses on using Python and deep RL. techniques. Our first contribution, MO-GM Symnaamas,,,, addresses this issue by introducing standard API and set of reference environments for MORL.
Although some published works MORL make their code publicly available (Yang et al., 2019; Abels et al., 2019; Xu et al., 2020), these implementations are not often maintained by their authors. This makes reproducing empirical results usually time-consuming and error-prone process. Our second contribution,
BiselnBss,,,ses tackles this challenge. It includes a set of clear, well- structured, regularly maintained, and reliable implementations of MORL algorithms. Importantly, all such algorithms are compatible with MO-Gymnasium's 's API.
Finally, recall that the performance of newly proposed algorithms is generally compared with previously-publishe baselines. However, the methodology employed in conducting experiments is not always scientifically rigorous. To address this issue, we introduce a dataset of training results of MORL-Baselines algorithms when evaluated on all MO-Gymnasium environments. This dataset is available through openrlbenchmark and includes information about all hyperparameters used by each algorithm, in each experiment, thereby allowing researchers to compare new algorithms with such existing baselines without having to retrain models from scratch.
3 Multi-objective reinforcement learning
In MORL, the interaction of an agent with its environments is modeled via multi-objecttive Markov decision process (MOMDP) (Roijers et al., 2013). MOMDPs differ from standard MDPs (Sutton and Barto, 2018) only in that MOMDP's reward function is vector-valued. A MOMDP is defined tuple M A (S, A. p, r. p., 7), where S is state space, A is action space, p(-|ss, a) is the distribution over next states given state s and action a, r S x A x S + RT a multi-objective reward function containing m objectives, μ an initial state distribution, and 7 [0, 1) is a discounting factor. A policy : S + A is a function mapping states to actions. Let St. Ar, and R. r(Sr, Ar, Sz+1) denote the random variables corresponding state, action, and vector reward, respectively, time step l. The multi-objectivv value function of a policy æ in state is defined as
Do
(s)
A E
Σ
Re4s
1 St =
where Ez|-] denotes expectation with respect to trajectories induced by the policy x. We denote v A "sse=[(""(So) as the value vector of policy и. Notice that v an m-dimensional vector whose i-th component is the expected return of under the i-th objective.
3

---

In contrast to singlc-objective RL, comparing the values of two different policies is not straightforward in MORL. For instance, a policy that achieves higher expected return with respect to one objective may, as result, have lower performance with respect to other objectives. Hence, in MORI., policies are typically evaluated and compared in terms of a user utility function (or scalarization function), I + R. which is a mapping from the multi-objective value of policy æ, v", to scalar.
Let be a monotonically increasing utility function; i.e., a function such that the utility increases if the value of one objective can be improved without decreasing the value of other objectives. Then, the value vector v" of an optimal policy with respect to u is in a Pareto frontier (PF). The PF of MOMDP is a set of nondominated multi-objecting value vectors:
(2) In particular, linear utility functions linearly combine the value of a policy under each of the m objectives using a set of weights w E IR:" u(v",, w) _ v" w, where each element of w E R specifies the relative importance of each objective. The space of weight vectors, W, is an m-- dimensional simplex so that Σ; W; 1, w, 0,i 11,=1,... l..... m. Under such utility scheme, the PF corresponds to a convex coverage set (CCS), as shown by Roijers al. (2013).
F [v- m's.tt vT >p v"), where v b v"
(Vi: 27: of (3i
MORL algorithms can be classified in terms of whether they are single-policy (i.e., they learn single policy that optimizes one particular utility u), or multi-policy (i.e., they learn a set of policies with the goal of approximating the PF). When non-linear utility functions are considered, the optimization procedure of a MORL problem can also be defined with respect the expected scalarized do
return (ESR), [[[(> f=o Re)], or the scalarized expected return (SER), u [B] 7 yRR,). For a thorough review of MORL concepts and definitions, we refer the reader to Hayes et al. (2022).
4 MO-Gymnasium
In this section, we introduce our first contribution: MO-Gymnasuum. 3 MO- Gymmasium is an easy-to-une and flex- ible API enabling the rapid construc- tion of novel MORL environments, and a collection of regularly main- tained and thoroughly tested environ- ment implementations.
import mo mo__ymmasium as mo_gym
env mo_gym make (*minecart-v0*) obs, info env ()
for action policy(obe)
range(1000)-
obs, rews, term, trunc, info env. www.steepaccionl
Figure 1: An example of how MO-Gymnasium can be used.
Unlike classical MI. settings, which rely on fixed datasets, RL problems typically do not; this makes replication of experimental results challenging. Indeed, even small differences in the definition of environment can have a significant impact on the performance of algorithms.
To address this issue and facilitate research when it comes the standard RL settings, Gymnasium (Towers et al., 2023) (formerly Gym (Brockman et al., 2016)) introduces an API and collection of versioned environments. With millions of downloads, this is currently the de facto standard library in RL, enabling researchers to easily test their algorithms on a variety of problems. Despite its widespread use, Gymnasium is limited to modeling single-objectivy MDPs. It has since been expanded in various ways; e.g., PettingZoo extends it to MARL settings and D4MORL to offline MORL. To the best of our knowledge, there are currently no publicly available and widely adopted libraries providing reliable implementations of MORL domains and faate-of-the-aee art MORL algorithm,, designed to facilitate research in the field.
We address this limitation by introducing M0-Gymmasiun--previinn al. 2022b). MO-Gymnasiun's 's API is designed to be as similar as possible to Gymnasium's 's API. This allows it to inherit many of the features in Gymnasium, such as wrappers-featuree that allow individual properties of domain be modified-whil. extending the original API only where and when necessary. This makes MO-Gymnasium automatically backward compatible with wide range of MORL benchmark domains. The key difference between these two frameworks is that, in MO-Gymnasium, rewards returned after the execution of an action (i.c., after a call of the step method) are vectors rather than scalars (see Figure 1). MO-Gymnasium is available on PyPI and can be installed via pip install mo- mo-gymnasium Importantly, we highlight that MO-Gymmasium is
known as MO-Gym (Alegre
The documentation of MO M0-Gaamaai is available at https: / /mo- gymnasium. f farama. org.

---

Figure 2: A few of the environments available in MO-Gymnasium. From left right: deep-sea- treasure, mo-halfcheetah, mo-supermario, minecart, and resource-gathering
an official part of the projects maintained by the Farama Foundation, and considered mature and well-supported library by the research community.
Currently, NO-Gymnasium includes over 20 environments commonly used in the MORL literature-Ban incluudin environments with discrete and continuous state and action spaces-such such trep-sea-treasure (Vamplew et al., 201 1). four-rooom (Alegre et al., 2022a). mo-supermario (Yang et al., 2019), 9), minecarl (Abels et al., 2019), and mo-halfcheetah (Xu et al., 2020). In Figure 2 we depict a few of the currently available environments. See Appendix B for detailed description of each environment. This large collection of environments allows designers to thoroughly assess the performance of novel algorithms in different scenarios. In environments in which the true PF is known, it can be accessed via the pareto_from () method available in the MO-Gymmasium' 's API. Additionally, for reproducibility purposes, each environment labeled with a version number; e.g., "-v0". Each time an environment is modified in way that may affect algorithms' performances, the domain's 's version number is incremented.
Environments.
Wrappers. 10-Gymmasium introduces MORL-specific wrappers such as MONormalizeRewardd which normalizes given component of the reward vector; and LinearReward, wrapper that linearly scalarizes the reward function of a MOMDP environment, transforming it into a standard MDP. The latter feature makes MO-Gymmasium directly compatible with widely-used RL libraries compatible 3 (Raflin et al., 2021) and cleanRL (Huang et al., 2022b).
with Gymnasium, such as
MORL-Baselines
In this section, we introduce MORL-Basclines,, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed provide a solid foundation for advancing research in MORL. To the best of our knowledge, this stands as the first open-source repository encompassing multitude of MORL algorithms. Notably, all such algorithms are inherently compatible with MO-G HO-Gymnasium' 's API.
It is well known that the performance of learning algorithms is closely tied their specific implemen- tations (Engstrom et al., 2020). Unfortunately, details such as implementation-speeiee optimizations seldom discussed in research papers, which makes reproducibility and comparisons challenging. To address this issue, some authors provide access to their codebases. While this a positive step, codebases are often not regularly maintained and may become outdated, rendering replication difficult. 3 (Raffin et al., 2021) and cleanRL (Huang et al., 2022b) have
Thus, libraries like Wtall--Baseling been designed with the goal of regularly maintaining state-of-the-art algorithms. These codebases are well documented, thoroughly tested, and offer top-notch performance on the algorithms they imple- ment. This allows researchers to start coding novel methods by extending existing implementations rather than starting from scratch. These libraries also provide useful tools for research purposes, such as efficient replay buffer implementations, performance reports, and evaluation methods.
MORL- includes more than 10 state-of-the-arr MORL. algorithms, all of which are compati- ble with MO-Gymnasium. Up this point, similar libraries were available. MORl-Baselines offers a range of features to aid researchers in designing new algorithms, such as methods to compute and analyze Pareto fronts, perform evaluation w.r.t. various metrics, replay buffers, and experiment tracking tools. Below, we provide a list of the algorithms currently available via MORL-Baselines, along with description of the settings in which they are applicable.
'MRRL-Baselines documentation is available https: https://lucasaaegree github baaeerees...eeeess.....
5

---

Table Algorithms currently implemented in MORL-Baselines. (*) PCN and PQL are designed to tackle deterministic environments, (**) OLS is an algoritm--agostic agnostic method for generating reward weights, or preferences; it does not assume any partieular type of observation or action spaces.
Single or
Utitlity
Observation
Action
Algorithm
multi-policy
function
space
space
MOQL (Van Moffaert et al., 2013a)
Single
Linear
Disc.
Disc.
Non-linear,
EUPG (Roijers et al., 201 8a)
Single
Disc.
Disc.
ESR
MPMOQL (Van Moffaert al., 2013a)
Multi
Linear
Disc.
Disc.
Non-linear,
PQL (Van Moffaert and Nowé, 2014)
Multi
Disc.
Disc.
SER (*)
OLS (Roijers, 2016)
Multi
Linear
/ ((*))
/ (**)
Envelope (Yang et al., 2019)
Multi
Linear
Cont.
Disc.
PGMORL (Xu et al., 2020)
Multi
Linear
Cont.
Cont.
Non-linear,
PCN (Reymond et al., 2022)
Multi
Cont.
Disc.
ESR/SER (*)
GPI-LS
Multi
Linear
Cont.
Any
GPI-PD (Alegre al., 2023)
CAPQL (Lu et al., 2023)
Multi
Linear
Cont.
Cont.
5.1 Implemented algorithms
Table lists the algorithms currently supported by MORL-Baselines and the MORL settings they tackle. Algorithms employing neural networks as function approximators were implemented using PyTorch (Paszke et al., 2019), and tabular algorithms rely on NumPy (Harris et al., 2020). Algorithms in Table are described according to whether they produce single policy (based on user-provided utility functions) multiple policies (i.c., approximate CCS PF). Additionally, notice that some algorithms optimize w.rt. the ESR while others optimize w.r.t. SER (Section 3). Finally, notice that MORL-Baseling algorithms may support different observation and action spaces (c.g., images).
Tabular algorithms. Multi-objective Q-learning (MOQL) (Van Moffaert et al., 2013b) is an extension of the classic tabular Q-learning algorithm (Watkins, 1989) that learns and stores the Q- values of each objective separately. A scalarization function is then used to convert these Q-values into scalar quantity, allowing agents select an action. Multi-policy MOQL, which consists running be instantiated using different methods that
MOQL multiple times with different preferences, select which preference weight vector will be optimized next by corresponding specialized policy, Currently, MORL-Baselines supports randomly generated weight vectors, Optimistic Linear Support (OLS) (Roijers, 2016), and Generalized Policy Improvement Linear Support (GPI-LS) (Alegre et al., 2023). Pareto Q-learning (PQL) (Van Moffaert and Nowé, 2014) aims at simultaneously learning all policies in the Pareto front by storing sets non- non-dnnnd Q-values. These sets are then converted into scalars (using metries similar to the ones discussed in the next section) that guide the selection of actions during the learning phase. This algorithm only compatible with deterministic environments.
Deep MORL algorithms. The Expected Utility Policy Gradient (EUPG) algorithm (Roijers et al., 201 8b) introduced a policy gradient update capable of taking into account both the return achieved up to the current moment, as well as future returns, order to determine expected utilities in an ESR setting. However, although it employs neural network the policy, it has only been

---

evaluated in discrete settings. The Envelope algorithm (Yang et al., 2019) uses single neural network conditioned on weight vector (Abels et al., 2019) to approximate the CCS. Prediction- Guided MORL (PGMORL) (Xu et al., 2020) is an evolutionary algorithm that maintains population of policies learned using PPO (Schulman et al., 2017b). This algorithm focuses on predicting, at each iteration, the most promising weight vectors and policies to select for further training in order to more effectively enhance the PF. Pareto Conditioned Networks (PCN) (Reymond et al., 2022) employs neural network conditioned a given desired return per objective. This network is trained via supervised learning to predict which actions produce the desired return in deterministic environments. GPI-LS (Alegre al., 2023) employs GPI (Barreto et al., 2017) to combine policies in its learned CCS and prioritize the weight vectors on which agents should train at each moment. GPI-Prioriee Prioriioitieed Dyna (GPI-PD) is a model- extension of GPI-LS that uses a learned model of the environment and GPI to prioritize experiences in the replay buffer. Concave- Concave-Augmented Pareto Q-learning (CAPQL) (Lu al., 2023) employs multi-objective extension of SAC (Haamojaa et al., 2018) by conditioning the actor and critic networks on the rewards weight vector.
5.2 Evaluation metrics
Recall that in single objective RL settings, policies are evaluated in terms of their corresponding expected returns. In MORL settings, by contrast, they are typically evaluated using multi oujti-obeecctive metrics computed based on Pareto fronts. While these metrics are widely adopted by the multi- objective optimization community, the MORL community has yet to establish a consensus on which metrics should be preferred in each particular problem setting. For this reason, our framework supports all commonly-un utility-based metries, which assume particular properties of the utility function (e.g., linearity), and axiomatic metrics, which do not make assumptions but may produce less informative performance information users. For a thorough discussion of these metries, see Hayes et al. (2022).
MORL metrics- see below. These metrics can be split into two groups:
Expected utility (t). the utility function u is linear, it is possible to express the expected utility over distribution of rewards weights, W, via the expected utility (EU) metric (Zintgraf et al., 2015). Let 11 be set of policies and F {v-|T |m E I1} be its corresponding approximate PF. Then, the EU metric is defined as:
EU(F)= = Ew max v c
W
Maximum utility loss ()). Zintgraf al. (2015) introduced this metric quantify the maximum utility loss that results from using an approximate PF, F, rather than a reference PF, 2 It is defined as follows:
MUL(F Z) max (max v" WEW V'EZ
max VTEF
w).
Inverted generational distance (1). This metric characterizes the convergence rate of approximate PF, F. towards reference PF, Z (Coello Coello and Reyes Sierra, 2004). If the reference front is unknown, is usually defined/construuteed by aggregating the best value vectors observed after several exccutions of the underlying RL algorithm. The inverted generation distance (IGD) is computed as:
1
IGD(F,
min || v'Ez EF
Σ
1
|z|
Sparsity (L). This metric characterizes the diversity of the policies in a given PF. A diverse set of policies allows users to choose from qualitatively different behaviors based on the trade-offs they may wish to optimize (Xu et al., 2020). The sparsity of approximate PF, F. is given by:
11 Σ
-
S(²) = |
(c,(1) - C, (i
where Lj is the sorted list of the values of the j-th objective considering all policies in F, and (i( is the i-th value in L..
If the true PF of the MOMDP, J. is known, then typically Z = F.
7

---

Hypervolume (). This is a hybrid metric quantifying both PF's rate of convergence and the diversity of the policies in it. Given an approximate PF, F, and reference point, Yee the hypervolume metric (Zitzler, 1999) is defined as:
HV(F, vid)
U VTEf
volume(Viet. v"),
where volume(Vyets v²) is the volume of the hypercube spanned by the reference vector, Ved and the vector, v The reference point used in the hypervolume computation is typically estimate of the worst-possibs return per objective. For instance, one can use as reference point the vector Frie where min is a vector with the minimum value for each objective in any given possible state.
1-y²
5.3 Additional features
In addition to providing implementations of the algorithms and evaluation metrics discussed above, MORL-Baselines also provides range tools to assist researchers in designing new algorithms. These resources include data structure for storing Pareto fronts and Pareto filtering functions, meth- ods for policy evaluation with automated metric and PF reporting via Weights and Biases (Biewald, 2020), scalarization functions, various neural network architectures, well as methods for gener- ating videos of policies trajectories (e.g.. via GIF files). For a comprehensive discussion of these functionalities, refer the reader to Appendix C.2.
6 Benchmark dataset
Our final contribution is a comprehensive collection of benchmark results evaluating MORI-Baselines algorithms diverse set of MO-Gymnasium environments. These results are compiled in dataset hosted in and integrated into the openrlbenchmark framework (Huang et al., 2023). Access to such a dataset enables users performing new experiments combine, compare, and aggregate the corresponding new results with the ones in the dataset, visualizing them in real-time using Weights and Biases (W&B) dashboards. They also allow users manipulate raw data based on subsequent analyses, compare performance metrics of new algorithms w.r.t. state-of-the fi-thhe-rt art techniques without having retrain the latter, and store the associated hyperparameters and command lines for each run. For instance, users can access the approximate Pareto fronts identified by each algorithm in our benchmark via the W&B dashboard, under the panel named 'eval/front'.
6.1 Proof-of-concept experiments using MORI-Basclines and MO-Gymnasium
In this section, we use MORL-Baselines and MO-Gymnasium to perform various proof-of-concept empirical experiments and discuss their corresponding results. Our goal is highlight the type of experiments, comparisons, and analyses- of different algorithms on various domains, and under different performance metries- that are possible using our framework. Given the potentially high cost of training algorithms, we performed 10 runs of each algorithm on each environment. Notice that while experimental results with tighter confidence intervals may be conducted (by performing more runs), our goal here is solely to highlight the capabilities of our libraries not to fully evaluate all existing algorithms. The hyperparameters used on each run can be found the W&B dashboards." Examples of the W&B user interface through which our benchmark dataset can be analyzed are shown the appendices." 8 Notice that some algorithms were trained over more timesteps than others since they may be fast but not sample efficient; e.g., PGMORL relies on PPO (Schulman et al., 201 7a), which is computationally faster but less sample efficient than GPI-LS, which relies on TD3 (Fujimoto et al., 2018a).

---

PQL
MultiPoliyy MO Q-Learning
OLS Hypervolume (1)
GPI-LS
Expected Utility (1)
Max. Utilty Loss (L)
(L)
800
5-
800
0-
0-
0M
0.1M
0.2M
0.1M
0.2M
0M
0.1M
0.2M
0M
0.1M
0.2M
Training steps
Figure 3: Performance of tabular MORL algorithms on deep-seea-rreasseee-O w.r.t training samples.
PCN
Envelope
GPI-15
GPLPD
Expected Utility (t)
Max. Utility Loss (L)
Hypervolume (1)
IGD (L)
1.0
0.73
400
0.70
0.25
0.25
0.04
0M
0.1M
0.2M
0M
0.1M
0.2M
0.1M
0.2M
0M
0.1M
0.2M
Training steps
Figure 4: Performance of MORL algorithms on the minecart-vo domain w.r.t. training samples.
6.2 Results and discussion of the proof-of-concept experiments
Figures 3, 4, 5, and present the performance of various MORL-Baselines algorithms when eval- uated on a few representative MO-Gymnasium environments. They show, in particular, the mean and corresponding 95% confidence intervals with respect the evaluation metrics introduced in Section 5.2. In the Appendix, we provide the complete set of experimental results and comparisons.
Figure 3 presents the performance of various tabular MORL algorithms available MORL-Baselines, when evaluated in the classic deep-sea-treasure domain. Notice that the PQL and GPI-LS algorithms have similar performance and achieve near-zero maximum utility loss and Inverted Generational Distance. This indicates that they successfully identify the true Pareto front.
Figure 4 depicts the performance of various MORL algorithms that support discrete action spaces in minecart-vO. GPI-PD significantly outperforms other baselines (when considering all performance metrics) in terms of sample efficiency and asymptotic performance. This empirical result supports recent observations by Alegre et al. (2022a) about the crucial role of efficient prioritization schemes when constructing PFs and selecting training experiences. Interestingly, notice that even though PCN was designed to tackle deterministic MOMDPs, still outperforms the Envelope algorithm.
PGMORL
CAPQL
Expected Utility (t)
Hypervolume (t)
Sparsity
(L)
Pareto Front
Volue of Chyecioe
200
2000
20000
1000
0M
0.1M
0.2M
0M
0.1M
0.2M
0.1M
0.2M
500
Training steps
Value of Objessivve
Figure 5: Peformance of MORL algorithms on the ma-halfcheetah--44 domain. Pareto fronts were constructed by identifying (across all runs) the front with the highest Expected Utility after given training budget (5M steps for PGMORL, 100k for GPI-PD, 200k for the others).
Finally, Figures 5 and present the performance of MORL algorithms that support continuous action spaces in the mo-halfcheetah-vv and mo-hopper-2d- domains, respectively. In mo-halfcheetah.
Theee results can be viewed and analyzed at https: //wndd..//bb///////////////// ai st//openrlbenchheee www.echmmnnnMo--L---- /MORL- Baselines. We also include a description of the high--ertten performance computers at the University of Luxembourg (Varrette al.. 2014) and Vrije Universiteit Brussel on which experiments environments, using various random seeds, required approximately months computation time.
conducted. Training all algorithms all
An overview of the command lines used to conduct each experiment can be found at https:/ // thub. con/LucasAleggremoorl. maselines// maselines/issues/55 1ssses//s.....

---

PGMORL
CAPQL
GPL-S
GPI-PD
Expected Utility (1)
Hypervolume (1)
Sparsity (J)
Pareto Front
1000
Objective
300
200
3000
100
50000
0
0
300
1000
300
1000
500
1000
200 Oljectivn
250
Time
(m)
Figure 6: Performance of MORL algorithms on mo-hopper-2d-v4 w.r.t training time (in minutes).
GPI-based algorithms perform better w.r.t. the hypervolume and expected utility metrics. PGMORL and CAPQL, by contrast, perform better w.r.t. the Sparsity metric-they are able to identify denser Pareto fronts, as shown in the rightmost plot. Recall that sample efficiency often comes at the cost of execution time. To investigate this trade-off, we show, in Figure 6, performance results in the mo-hopper domain with respect to training time (in minutes). These experiments suggest that PGMORI. performs well in terms of training time, in contrast to previous experiments where had a qualitatively different behavior when analyzed with respect to sample complexity. In these experiments, both GPI-based algorithms still dominated all other baselines.
7 Conclusion
We introduced a comprehensive collection of software libraries for reliable benchmarking and research in MORL. MO-Gym N0-Gymnasium and MORL-Baseling are the first libraries providing standardized APIs and extensible sets of environments and state-of -h-the- MORL algorithms. Our framework also includes thorough set of benchmark results comparing a wide range of algorithms in various environments. These can be used as guidelines by the community, underscoring the properties and limitations of different techniques. By making this toolkit open source and extensible and thus open to contributions from other researchers- we hope to provide a solid foundation for reproducible research in MORI. This work will allow researchers and end users to seamlessly deploy existing algorithms on various MORL domains, speeding up experiments, facilitating algorithm evaluation, and being more conducive to reproducible experimental results.
Limitations and future work. the benchmarks we presented in this paper, we did not perform exhaustive hyperparameter tuning. Our main goal with these experiments was to validate our implementations and showcase the learning behavior of the algorithms our framework with respect to different evaluation metrics. For this reason, such empirical results should be interpreted as proof-of-concept and may still be further improved with the aid of the MORL community. future work, we will continue maintaining all algorithms in MORL-Baselines, and plan to augment the set of techniques available through our framework. We also plan introduce new features for automating hyperparameter tuning. We do not anticipate any negative societal impacts of this work.
Acknowledgments and disclosure of funding
We would like to thank Willem Röpke for his implementation of PQL., Denis Steckelmacher and Conor F. Hayes for the original implementation of EUPG, and Mathieu Reymond for the original implementation of PCN. Jordan K. Terry, Mark Towers, Manuel Goulão, and the broader Farama Foundation team for supporting MO-Gymnasium. Shengyi Huang, Antonin Raffin, and Quentin Gallouédec for their advice on experimental setup and openrIbenchmark integration. This work was funded by: Coordenação de Aperfeigoamento de Pessoal de Nível Superior Brazil (CAPES) Finance Code 001; CNPq (Grants 140500/2021-9, 304932/2021 2020/05 165-1), )); the Fonds National de Recherche Luxembourg (FNR), CORE program under the ADARS Project, ref. C20/IS//4761 14762457; the Research Foundation Flanders (FWO) [G0628199) 19N]; the AI Research Program from the Flemish Government (Belgium); and the Franequi Foundation.
FAPESP/MCTI/CII (Grant
10

---

References
Abels, A., Roijers, D. M., Lenaerts, T., Nowé, A., and Steckelmacher, D. (2019). Dynamic weights in multi- objective deep reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 11-20. International Machine Learning Society (IMLS). 2, 4, 5.1, B.8, C.2
Agarwal, R., Sehwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. (2021). Deep reinforcement learning at the edge of the statistical precipice. In Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems 34 (NeurIPS 2021). 1,2
Alegre, L. N., Bazzan, A. L.. C., and da Silva, B. C. (2022a). Optimistic linear support and successor features as a basis for optimal policy transfer. In Chaudhuri, K., Jegelka, S., Song. L., Szepesvari, C., Niu, G., and Sabato. S., editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 394 413. PMLR. 6.2, B.5, B.13
Alegre, L. N., Bazzan, A. L. C., Roijers, D. M., Nowé, A., and da Silva, B. C. (2023). Sample-efficient multi- objective learning via generalized policy improvement prioritization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, pages 2003-2012, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems. 1, 5.1, 5.1, C.1, C.2, D.2, D.2
Alegre, L. Felten, F., Talbi, E.-G., Danoy, G., Nowé, A., Bazzan, A. L., and da Silva, B. C. (2022b). MO-Gym: Library of Objee-- Reinforvement Learning Environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelear. 4
Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. (2017). Successor features for transfer in reinforcement learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. 5. B.5, B.13, C.I
Barrett, L. and Narayanan, S. (2008). Learning all optimal policies with multiple criteria. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 41 47. New York, NY, USA. Association for Computing Machinery. B.2
Bellemare, M. G., Candido, S., Castro, P. S., Gong. J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang. Z. (2020). Autonomous navigation of stratospheric balloons using reinforeement learning. Nature, 588(7836):77- 82. 1
Biewald, L. (2020). Experiment Tracking with Weights and Biases. 2, 5.3
Blank, J., Deb, K., Dhebar, Y., Bandaru, S., and Seada, H. (2021). Generating Well-Spaced Points on a Unit Simplex for Evolutionary Many-Ohjective Optimization. JEEE Transactions Evolutionary Computation, 25(1):48-66 60. Conference Name: IEEE Transactions Evolutionary Computation. C.2
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAl Gym. arXiv preprint 110.001600.01540. 1, 4
Cassimon, T., Eyckerman, R., Mcrcelis, S., Latré, S., and Hellinekx, P. (2022). A survey on discrete multi- objective reinforcement learning benchmarks. In Proceedings of the Adaptive and Learning Agents Workshop (ALA 2022).?
Castelletti, A., Pianosi, F., and Restelli, M. (2012). Tree-based fitted q-iteration for multi-objective markov decision problems. In The 2012 International Joint Conference on Neural Networks (IJCNN, pages 1-8. B.12
Coello Coello, C. and Reyes Sierra, M. (2004). A Study of the Parallelization of a Coevolutionary Multi- objective Evolutionary Algorithm. In Monroy, R., Arroyo-Figueroy G., Sucar, L. E., and Sossa, H., editors, MICAI 2004: Advances in Artificial Intelligence, Lecture Notes in Computer Science, pages 688 -67,, Berlin, Heidelberg. Springer. 5.2
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. (2020). Implementation Matters in Deep RL: A Case Study on PPO and TRPO. In Proceedings of the Eighth International Conference on Learning Representations. 5
Fujimoto, S., Hoof, H., and Meger. D. (2018a). Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Muchine Learning, pages 1587-1596.
PMLR. ISSN: 2640-3498. 6.1
Fujimoto, S., Hoof, H., and Meger, D. (2018b). Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, pages 1582-159) 1591. D.2
11

---

Haarnoja. T., Zhou, A., Abbeel, P., and Levine. S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy. J. and Krause. A., editors, Proceedings of the 35th International Conference on Machine Learning (ICML). volume of Proceedings of Machine Learning Research, pages
Stockholmsmässan, Stockholm Sweden. PMLR. 5.1, D.2
Harris, C. R., Millman, K. J., Walt, S. J. v. d., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J. Kern, R., Picus, M.. Hoyer, S., Kerkwijk, M. H. v., Brett, M.. Haldane, A., Río, F. d., P., Sheppard. K., Reddy, T.. Weckesser. W., Abbasi, H.. Gohlke, C., and Oliphant, T. R. (2020). Array programming with NumPy. Nature, 585(7825)3357-366 Publisher: Springer Science and Business Media L.C. 5.1
Wiebe, M., Peterson, P., Gérard-Marchant
Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstracten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A. A., Mammion, P., Nowé, A., Ramos, G., Restelli, M., Vamplew, P., and Roijers, D. M. (2022). A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26. 1, 2, 3, 5.2
Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. (2022a). The implementation details of proximal policy optimization. The ICLR Blog Truck 2023. 2
Huang. S., Dossa, R. J.J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. M. (2022b). CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine
Learning Research, 23(274):1-18, 1, 2, 4,5
Huang, S., Gallouédee, Q. Felten, F., Rafin,, A., Dossa, R. F. J., Zhao, Y., Sullivan, R., Makoviychuk, V., Makoviichuk, D., Roumégous, C.. Weng, J., Chen, C., Rahman, M. M. Araújo, G., Quan, G., Tan, D., Klein, T., Charakorn, R., Towers, M., Berthelot, Y., Mehla, K., Chakraborty, D.,, KG, A., Charraut, V., Ye, C., Liu, Z., Alegre, L. N., Choi, J., and Yi, B. (2023). openrlbenchmark. 2, 6, D.2
Kauten, C. (2018). Super Mario Bros for OpenAl Gym. GitHub. B.16
Leurent, F. (2018). An Environment for Autonomous Driving Decision-Making. B.10
Lu, H., Herman, D., and Yu, Y. (2023). oulti-objective reinforeement learning; Convexity, stationarity and Pareto optimality. In Proceedings of The Eleventh International Conference on Learning Representations. 1, 5.1, D.2
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015a). Human-level control through deep reinforcement learning. Nature, 518(7540):52-533
D.2
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015b). Human-level control through deep reinforcement learning. Nature, 518(7540):529 533.
Moore, (1990). Efficient memory-based learning for robot control. Technical report, Carnegie Mellon University, Pittsburgh, PA. B.7
Papoudakis, G., Christianos, F., Schäfer, L.., and Albrecht, S. V. (2021). Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. arXiv:2006008869 [es, stat]. 2
Paszke, A., Gross, S., Massa, F, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, 7.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf. A., Yang, E., DeVito, Z.. Raison, M.. Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F. Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc. 5.1
Patterson, A., Neumann, S., White, M., and White, A. (2023). Empirical Design in Reinforcement Learning. arXiv:2304.01315 [es]. 1,2
Pineau, J.. Vincent-Lamare
Sinha, K., Larivière, V., Beygelzimer, A., d' Alché Buc, F., Fox, E.,, and Larochelle, H. (2020). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003. 12206 [cs, stat]. 2
Raffin, A., Hill. A., Gleave, A., Kanervisto, A., Emestus, M., and Dormann, N. (2021). babbe-baselines3:
Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8. 1, 4,5
12

---

Reymond, M., Bargiacchi. E., and Nowé, A. (2022). Pareto conditioned networks. In Proc. of the 21st International Conference on Autonomous Agents and Multiagent Systems, Virtual. IFAAMAS. D.S
Roijers, D. (2016). Multi-Objective Decision-Theoreti Planning. PhD thesis, University of Amsterdam. 5.1, C.1
Roijers, D., Steckelmacher, D., and Nowe, A. (2018a). Multi-objective Reinforcement Learning for the Expected Utility of the Return. In Proceedings of the ALA workshop at ICMI/AAMASIJCAI 2018.
Roijers, D. M., Röpke, W., Nowé, A., and Rädulescu, R. (2021). On Following Pareto-Optimal Policies in Multi-Objectivn Planning and Reinforcement Learning. In Proceedings of the 1st Multi Objeti-Oeeective Decision Making Workshop (MODeM), page 8. D.2
Roijers, D. M., Steckelmacher, D., and Nowé, A. (2018h). Multi-ohjective reinforcement learning for the expected utility of the return. In Proceedings of the ALA workshop at ICMLAAMAS//JCA, 2018. 5.1, B.4
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. (2013). A survey of multi-objectivee sequential decision-making. J. Artificial Intelligence Research, 48(1))67-113, 3,
Schulman, J., Wolski, F., Dhariwal, P., Radford. A.. and Klimov, O. (2017a). Proximal Policy Optimization Algorithms, arXiv:1707.06447 [es]. arXiv: 1707.06347. 6.1
Schulman, J.. Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017b). Proximal policy optimization
algorithms. CoRR, abs/1707.06347. 5.1, D.2
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap. T., Hui, F,, Sifre, L.,, van den Driessche, G., Graepel, T., and Hassabis, D. (2017). Mastering the game of go without human knowledge. Nature, 550(7676)354-355. 1
Sutton, S. and Barto, A. G. (2018). Reinforcement learning: An introduction. The MIT Press, second edition. 1,3
Terry, J. K., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., Santos, L., Perez, R., Horsch, C., Dieffendahl, C., Williams, N. L., Lokesh, Y., and Ravi, P. (2021). PettingZoo: Gym for Multi-Agent Reinforcement Learning. arXiv:2009.1e
[cs, stat]. 2
Todorov, E.. Erez, I., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. pages 5026-5033, Vilamoura, Portugal. 1, B.13
Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G., Deleu, T., Goulão, M., Kallinteris, A., KG, A.,
Krimmel, M., Perez-Vicente R., Pierré, A., Schulhoff, S., Tai, J.J Tan, A. J. S., and Younis, O. G. (2023).
Gymnasium. 2, 4, B.9
Vamplew, P., Dazeley, R., Berry, A., Issabekov, R., and Dekker, F. (2011). Empiriaal evaluation methods for multiobjective reinforcement learning algorithms. Mach. Learn., 84(1-2):51-80. 4, B.I, B.7
Vamplew, P., Foale, C., Dazeley, R., and Bignold, (2021). Potential-based multiobjective reinforcement
learning approaches to low-impact agents for Al safety. Engineering Applications of Artificial Intelligence, 104166.. B.3
Vamplew, P.. Webb. D... Zintgraf, L. M.. Roijers, D. M.. Dazeley, R., Issabekov, R., and Dekker, E. (2017). MORL-Glue: benchmark suite for multi multi-objective reinforcement learning. In Proceedings of the 29th Benelux Conference on Artificial Intelligence, pages 389-390. 2
Van Moffaert, K., Drugan, M. M., and Nowe, A. (2013a). Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 191-199, Singapore, Singapore. IEEE. 1, D.2
Van Moffaert, K., Drugan, M. M., and Nowé, A. (2013b). Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). pages 191-199. 5.1, C.2
Van Moffacrt, K. and Nowé, A. (2014). Multi-objective reinforeement learning using sets of Parcto dominating policies. J. Mach. Learn. Res., 15(1):348333512. 5.1
Van Moffaert, K. and Nowé, A. (2014). Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):34833355 Publisher: JMLR. org. 1, D.2
13

---

Varrette, S., Bouvry, P., Cartiaux, H., and Georgatos, F. (2014). Management of an academic HPC cluster: The UL experience. In 2014 International Conference High Performance Computing Simulation (HPPCS), pages 959-967. IEEE. D.1
Watkins, C. (1989). Learning from Delayed Rewards. PhD thesis, University of Cambridge. 5.1
Xu, J., Tian, Y., Ma, P., Rus, D., Sueda, S., and Matusik, W. (2020). Prediction-guider multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning (ICML). 2, 4, 1, 5.1, 5.2, D.2
Yang. R., Sun, X.. and Narasimhan, K. (2019). A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Wallach. H., Larochelle, H., Beygelzimer, A., Alché-Bue F., Fox. E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 14610-14621. 2, 2,4, 1, 5.1, B. B.6, B.16, D.2
Yoo, A. B., Jette, M. A., and Grondona, M. (2003). SLURM: Simple Linux Utility for Resource Management. In Feitelson. D., Rudoiph, L.. and Schwiegelshohn, U.. editors, Job Scheduling Strategies for Parallel Processing. Lecture Notes in Computer Science, pages 44-60, Berlin, Heidelberg. Springer. D.1
Zhu, B., Dang, M., and Grover, A. (2023). Scaling pareto pareto-efficient decision making via offline multi culti-obbective RL. In In Proceedings of The Eleventh International Conference Learning Representations. 2
Zintgraf, L. M., Kanters, T. V., Roijers. D. M., Oliehoek, F. A., and Beau, P. (2015). Quality assessment of MORL algorithms: A atility-based approach. In Benelearn 2015: Proceedings of the 24th Annual Machine Learning Conference of Belgium and the Netherlands. 5.2
Zitzler, E. (1999), Evolutionary algorithms for multiobjective optimization: methods and applications. In Ph.D.
Dissertation. ETH Zurich, Switzerland, 5.2
14

---

Appendix
This appendix contains supplementary material about MO-Gymnasium and MORL-Baselines, In Section A, we provide general information such the links to our libraries, licenses, and maintenance plan. In Section B, we describe detail all environments available MO-Gymnasium, and in Section C we present all features of MORL-Baaellin detail the experimental setting and results of our benchmarks.
Finally, in Section D, we provide in more
A General information
A.1 Links
The MO-Gymnasium documentation is accessible at: https://mo-gymmasiil farama org and its code repository at: https://github. httpaaaccoocoaaaaaaaauaaaaaeeee http://aaaaa-uuaaaaaiaoooo---aaa
The MORL-Baselines documentation available at: http:://gtcuub..ommmuaaaagggg
https://hucasalegre github.
io/morl-baselinee and its code repository morl-baseling ines.
The openrlbenchmark code and early documentation are available at: https://github. benchark...
Our training metrics and results, hosted by Weights and Biases, are available at: https://vandb a httpp/rr/benncaaaar//ttttttt Baw.eoonn...aeee.....e.
A.2 Licenses
MO-Gyymasiuum, MORL-Baselines, and openrlbenchmark are under the MIT License. We, the authors, bear all responsibility in case of violation of rights.
A.3
Maintenance
NO-Gymmasium is maintained by the authors, with the support of the Farama Foundation team." MORL-Baselines is currently maintained by the authors, with the support of the community, openrl- benchmark is still under development, yet large community of researchers already using it to upload RL training data.
B MO-Gymnasium
Below, we provide a description of each environment currently available in MO-Gymnasium. Figure 7 presents a visualization of each environment. comprehensive list and documenta- tion of the environments available through the MO-Gymnasium API, along with the definition of their observation and action spaces, / //mo- gymnasium. farama. oraima..rr/eeee org/ prg/enviine ///eevvironmeetttt http//allllvoomeeeeeeee We have plans to intro-
be found the documentation website: https:
duce more challenging environments in the future.
B.1 deep-sea-treasure-vd
This domain, originally proposed by Vamplew et al. (2011), is a classic MORL problem in which the agent submarine that must collect treasure while taking into account time penalty. At each step, the agent perceives its current location as x. y coordinates a 11 x grid world, and its actions corresponn movement in one of the four cardinal directions. Hence, in this domain, the state space defined as S = .0...., 10]2, and the action space is A = {up. down, left, right). We use the same treasures' values defined by Yang et al. (2019). A version of this domain with a concave Pareto frontier (Vamplew et al., 201 1) is also available with identifier deep-sea- deep-eeetttraxxarrccoevvvwe The
"attps://faraaa.orrg//
15

---

Sow
(a) deep-sea- seatrreuuee--
(b) resource-gathering--il
(c) wreakable-bottle--ee
(d) fishwood-v0
four-room-id
i fruit-tree-vv
g mo-mountaincur-vd
(h) minecart-vO
Hevr Fidered 30
Food Croddo
i mo -iiihwwway--0
(k) mo-highway-fastee
weter--eservoii-v0
(m) mo-reacher-v4
(n) mo-hopper-vd
(o) mo-halfeheetah-vv
(p) no-supermario-vv
Figure 7: Visualization the environments currently available MO-Gymnasium.
multi-objective reward function of this domain, r(s, a,,'s,) E IR, is defined as:
(1 a, a,,,,) 8') value of treasure in state s' r a, = -1
B.2 resource-gathering--d
In this domain, agent must collect the gold gem available in 5 grid world while avoiding being killed by enemies (Barrett and Narayanan, 2008). The state space defined as S s= (o, 4]2 {0, 1 and denotes the agent's current position and two binary variables rep- resenting whether the agent carries gold or diamond. Its action space representing movements in the four cardinal directions is A {up, down, left, right). There two enemies that have 10% chance of killing the agent it the same cell. The multi-objective reward function of this domain, r(s, s') E R², is defined as:
ri(s, a, s') 1 if killed by enemy in s', else 0,
r2(8, a, s') = +1 if returned home with gold, else 0, ra(s, a, = +1 if returned home with gem, else O.
B.3 breakable-bottlesvv/
This domain consists of grid corridor with 5 cells. The agents must collect bottles from the source location and deliver them to the destination cell (Vamplew al., 2021). Its action space defined as A fright, left, pick_up. wp_ bottle). Every time step, the agent has 10% chance of dropping one of the
16

---

bottles it is currently carrying. The oulti-objectivve reward function of this domain, a(s,a,s) E R. defined as:
as, -1, a,(s,,,s) +25 x #bottles deliveredd
Ta(8, 8') s(s') -f(s), where f(s) = -11 if dropped any bottles in else 0.
B.4 fishwood-v0
This is simple MORL problem in which the agent controls fisherman who either fish go collect wood (Roijers et al., 201 There are only two discrete states, and the agent can decide between fishing collecting wood, i.e. multi-objective reward function of this domain, r(s, a, 8') E IR, is defined as: ri(s, a, = +1 in the woods, with 90% probability, else 0, r2(s, 8') =+ +1 if in fishing, with 10% probability, else
{fishing, woods), } and = {go_fish,
fish, go_wood]. The
B.5 four-room-v0
This is a classic grid world environment in the literature successor features (SFs) (Barreto et al., 2017). Importantly, Alegre al. (2022a) showed that any problem in the SFs framework can be mapped to a MORL problem. In this domain, the agent can collect three different types items (see Figure 7(e)). Its state space defined by the x. y coordinates of the agent well as binary vector stating which items have been collected, the actions are going into one of the four cardinal directions. Formally, S .O..... 12 {0, 1) 12 and A {up, down, left, right). The multi-objective reward function of this domain, rss, a, s') E R$, is defined as:
a,, s') = +1 if collected blue square (or is goal location), else 0, 12(8, 8') ) if collected green triangle (or goal location), else 0, a3'), = 1 if collected red circle (or in goal location), else
B.6 fruit-tree-v0
This domain is modeled as a full binary tree of parameterized depth (d E {5, 6, 7)) (Yang et al., 2019). The agent starts at the root node and choose to move left right until reaching leaf node. The state space encodes the current position in the tree as a vector, which contains the current depth and the corresponding node: S = {0, ..... d} x (0....,2t-1) The action space is defined as A = (left, right). Every leaf contains a fruit with a different value for the nutrients: protein, carbs, fats, vitamins, minerals, and water. The multi-objective reward function of this domain, r(s, a, 8') E R$, is defined as:
a=(),,,8)) = value of nutrient in 8', for i 1...6.
B.7 mo-mountaincarrvv0
This domain, introduced by Vamplew et al. (2011), is multi-objective version of the classic mountain problem (Moore, 1990). The state space is defined by vector stating the agent's current posítion and its velocity, S = 1.2, 0.6] x [-0.07, 0.07]. Its actions consist in accelerating to the left, right, doing nothing: A (reverse, forward, none). Unlike the single-objective version, the agent is also penalized for applying the reverse and forward actions, which leads to different policies depending on the trade-offss between these penalties. The multi-objective reward function of this domain, r(s, a,) E R², is defined as:
ri a, 8') = -1,
r2(8, a, = -1 if reverse, else O,
r3(8, =- if forward, else
A continuous action version of this domain is also available with identifier mo-mountaincar- continuous-vd. In this version, the action space is a speed vector A = [-1, 1], and the reward function contains two components; time penalty and fuel penalty.
17

---

a1 s') -1, 22(,,a,)) =-laa]?
B.8 minecart-v0
This domain consists of cart that must collect two different ores and return them to the base while minimizing fuel consumption (Abels et al., 2019). The agent perceives its environment through 7-dimensional vector containing the agent z, y position, the current speed of the cart, its orientation (sin and cos), and the percentage of occupied capacity in the cart by each ore: S = [-1, x [0, 1 The agent has the choice between 6 actions: AA { (mine, left, right, accelerate, brake, do nothing). Each mine in Figure 7(h) has different distribution over two types of ores. Fuel is consumed at every time step, and extra fuel consumed when the agent accelerates or selects the mine action. A version of this domain with deterministic values for the ores is also available as minecart- diirecurrdddterminiiiic-dd The objti-objective reward function of this domain, r(s, a, s') E IR is defined as:
ri (i a, s') quantity of ore collected if s' is inside the base, else 0, r2(5, a, 8') quantity of ore 2 collected if s is inside the base, else 0, as(s,a, -0.005 0.0251 {a accelerate) 0.051 (a mine).
B.9 mo-lunar-lander--2
This domain is multi- objectvb version of the lunar lander environment available in Gym- nasium (Towers al., 2023). The state is an 8-dimensional vector: the z, coordinates of the lander, its linear velocities in x & y. its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground not: S [-1.5, 1.5) x [-5, 5] x T, x| x [-5,5] x 10. {do nothing, fire left orientation engine, fire main engine, fire right engine). A continuous action version of this domain is also available with identifier mo-lumag-lande--coniinnss variant, the first coordinate of the action determines the throttle of the main engine, while the second coordinate specifies the throttle of the lateral boosters, A = [11, The multi-objective reward function of this domain, r(s, a, s') E R-, is defined as:" 10
The action space consists of four actions: A
In this
(1 0, s') + 100 if landed successfully, 100 if crashed, else 0,
T2(5, a, s') shaping reward,
r3(s, a, s') fuel cost of the main engine
ra(s,a, s') fuel cost of the side engines.
B.10 mo-highway-r0
This domain is multi-objective version of the autonomous driving environment introduced by Leurent (2018). In this domain, the agent controls vehicle on multilane highway populated with other vehicles, it perceives the state space is given by array of size V x F, where V represents the number of vehicles in the environment and are the features descríbing each vehicle, e.g. velocity, position, angle. The agent's 's actions consist in changing lanes, accelerating decelerating, doing nothing, i.e. A = [lane left, idle, lane right, faster, slower). The multi-objective reward function this domain, r(s, a, 8') E R3, is defined as:
coordinates as well as coordinates from other vehicles. Formally,
ri a, s') = normalized forward speed of the vehicle, (2 a, s') = bonus for driving in the rightmost lane, ra(8, a,,'s 8') = -1 if vehicle crashed, else 0.
All three reward functions above are zeroed when the agent is off the road.
Seeee https: http:///yynaauummm/mm farama. aerraaa.oo//eeeee wtgp://wirroommtt/boo////////a/e lander for details on the shap-
ing reward.
18

---

B.11 mo-highhwa-fest-v00
This is more computationally efficient version of mo-highway-vo with reduced number lanes and vehicles. The reward function, state, and action spaces are kept the same as the original environment.
B.12 water-reserrvirr-d
This domain simulates water reservoir and is implemented defined by Castelletti al. (2012) 2). The agent perceives floating point number corresponding to the current amount of water in the reservoir: S I The agent executes continuous action corresponding to the amount of water the dam will release in a day: A C RT. The action space can also be specified to be normalized, in which case the action is expressed as a percentage of water to release: A [0, 1]. The multi-objective reward function of this domain, r(s, a, s') E R. is defined as:
ri a, s') = cost due to excess level w.r.t. a flooding threshold (upstream),
r2(s, a, s') = deficit in the water supply w.r.. the water demand, ra(8, a, s') deficit in hydroelectric supply wrt hydroelectric demand,
r1(s, a, cost due excess level w.r.t. flooding threshold (downstream).
By default, only rewards and 12 are used. However, the user has the option instantiate the environment using all four reward functions.
B.13 mo-reacher-v4
This domain is also a classic in the SFs literature (Barreto et al., 2017: Alegre et al., 2022a). It consists of two-joint robot arm that must reach different target locations with the tip of its arm, implemented using the Mujoco robotics simulator (Todorov et al., 2012). The agent's 's state space 8 C R consists of the sine and cosine of the angles of the central and elbow joints, as well as their angular velocities. The action space, originally continuous, is discretized using bins per dimension corresponding to maximum positive torque (+1), negative torque (-1), and zero torque for each actuator. This results in total of possible actions: A 13 The multi-objective reward function r(s, a, s') R+ is defined as:
as a, 8') (taaget,,, for 1...4,
where Atargett, is the Euclidean distance between the tip of the robot's 's arm and the i-th target's location.
B.14 mo-hopper-v4
This domain is multi-objective version of the Gymnasium's 's Hopper-v4 environment. The state space S IR' consists in velocities. angles, and position of each part of the robot. An action represents the torques applied at the hinge joints, A = [-,11, 1| In this modified version, the agent must balance optimizing for its forward speed, jumping height, and energy cost. A version this environment with a woo-dimensional reward function is available with identifier mo-hopper-2d-v4. The multi-objective reward function r(s, 0, 8') E R is defined as:
ri a, s') velocity of the agent in the x-axis direction, r2(s, a, s') height of the agent over the z-axis,
ra(s, a, s') -|/a|
B.15 mo-halfcheetah--vt
This domain is a multi-objective version of the Gymnasium's 's HalfCheetah-v4 environment. The state space encodes the velocities, angles, and position of each part of the robot: S C RT Also, the actions are the torque to apply at each joint: AA [- 1]6. The multi-objective reward function r(s, s') E R² is defined as:
(i,s, a,) = forward speed of the agent,
-||a|
19

---

B.16 mo-supermario-v0
This domain consists of the first level of the Super Mario Bros. video game. It is imple- mented using Gym Super Mario (Kauten, 2018). In this environment, the observations con- sist of RGB frames of the game, and the actions are the following combination of buttons: A= (NOOP, right, right A, right B, right A B, A, left]. The multi-objective reward function r(8, a, 8') E R$ defined similarly as defined by Yang et al. (2019):
ri(s, a, s') increase in the x-coordinate of Mario, r2(8, a, s') = time elapsed executing action a,
ra(s, a, s') = 25 if Mario died, else 0,
ra a, s') = 100 x #coins collected,
rs a, 8') increase score by killing an enemy.
C MORL-Baselines
This section describes MORL-Baselines in more detail. MORI-Baselines is comprehensive library that offers dependable, validated, well-documented, and efficient implementations of MORL algorithms. The library goes beyond supporting existing algorithm implementations by equipping researchers with valuable tools streamline the process of designing future MORL algorithms. In the remainder of this section, we describe the algorithms, features, and provide few examples of the user interface brought by this library.
C.1 Algorithms
All the algorithms currently supported in MORL-Baselines are listed in Table 1 of the main paper. A few algorithms also have variants that the user can select by instantiating the agents with different parameters. In particular, our implementation of Multi-Policy MO Q-Learning (Van Moffaert et al., 2013a) can be derivated multiple ways. For instance, the weights can be generated randomly, using OLS (Roijers, 2016), or using GPI-LS (Alegre et al., 2023). Moreover, using GPI (Barreto al., 2017) to select actions can be enabled or not, as well as using model-based learning with GPI-PD (Alegre et al., 2023).
More details on the implementations are available in our documentation and GitHub reposi- tory. For example, see the documentation of PGMORL https://lucasat wewpalllrrreeegggiggeeee github. htt.aaaaeeeeeeaallo/oaaaaataaaae
C.2 Additional features
Aside from the algorithms, MORL-Baselines provides a set of utilities allowing the user to construct novel MORL algorithms. These are listed below.
Pareto frontier filters. Our library provides a set of functions that permit to filter Pareto-dominated points out of a set of points. The signatures and documentation of these functions can be found in our documentation at https://lucasde h/tc///aalaeeeeeeeeeeeeaaaa www.ggit..uu.......... io 1o/morl- bwseaaaeellleeeeee weeeeeiineseeee wttpe///eeattteeeeeeeeeeeeeeeeee /arrtoo.
Policies evaluation. In order to build a Pareto front of policies, it is necessary to assess the values of the policies currently undergoing training the environment. To standardize the evaluation process and enable fairer comparisons, we offer collection of functions specifically designed for evaluating these policies. Furthermore, we have implemented a function that receives Pareto front input and records all metrics to Weights and Biases (W&B). This approach is employed across all multi-policy algorithms to ensure standardized metric logging. Additionally, we provide a method to initialize all components with a seed, including making PyTorch deterministic. The documen- github.
tation related to these can be found at f features/ www.ereieeeaalaiiooos/./
iines/
Performance metrics. As outlined the main paper, there are multiple metrics that can be used to evaluate given Parcto front in order to compare different algorithms. We offer collection of
20

---

metrics implementations for this purpose, including Hypervolume, Inverted Generational Distance, Maximum Utility Loss, Sparsity, and Expected Utility. These implementations accept Pareto front as input and produce scalar value representing the corresponding metric value. More details can be found atttps///uucaalie indicators/
.gi celub.io/mman
Scalarization functions. Although most of the MORL algorithms rely on weighted sum scalar- ization, there are still cases where the user utility can be captured by non-lineat schemes. To solve such an issue, we provide other scalarization schemes such as the Chebyshev scalariza- tion function as used in Van Moffaert et al. (2013b). More details can be found at https: //lucasalegre, gi githb.io/morl. http:/ool-baseeneess/ee wws-baalllseee
Reward weights utils. There are many ways to generate weight vectors to scalarize vectorized rewards. These can, for instance, be randomly sampled or generated as equidistant points from the weight simplex, e.g., via the Riesz s-Energy method (Blank et al., 2021)." These helpers can be found at https:: //aucs/aacgeeeeeeee github.io/mor/ b
Experience replay buffers. In RL algorithms, experience replay buffers play a crucial role as they store experienced tuples collected via interacting with the environment. These are usually used for updating the agent's 's policy and/or value functions via sampling. There are various approaches to designing these buffers. For instance, Abels et al. (2019) propose an experience replay buffer that maintains diverse samples to enable the learning of diverse policies. On the other hand, Alegre et al. (2023) rely on a prioritized experience replay buffer. We make these implementations available so that researchers can incorporate them into novel algorithms. More details can be found at https: //lcs////ccaaarree github. io/morl- waaselinnsffeattie buffees..
Neural networks. Neural networks are the main component used in modern deep MORL algorithms There are many possible architectures and enhancements related to the usage of such function approximators, e.g. Convolutional Neural Networks are usually used environments with image- based observations, and Polyak update is used when the algorithm relies on target networks. We provide few helpers related to neural networks in MORL-Baselinee see https: //lpssss///ac github io/ wtw.oooooooo/eoooooeoooeeoooeele baselineseeseee..e http:///eaaieeeeeeeeeaeeeeeeeeee httr///eeteteeeeerrrreee/
Finally, there are couple of other methods that have been reused in MORL- Baselines. For example, we provide utilities to capture GIFs from the policy executions, or a linearly decaying value for eesilln--reedd exploration, in Mnih al. (2015a). All these utilities listed at https: : //twca/aaaaaaeeeeeeeeeeeee github.......... io/morl- basel wwneenneeeee wates//eatureemmeeeeee
Miscellaneous utils.
C.3 User interface
This section gives some insight into the user interface provided by Weights and Biases to visualize and manipulate the training results.
Figure 8 shows an example of the real-time metrics obtained during the training. In this example we have a view of the multi-objective metries described in the main paper, as well as live view of the Pareto front found by various algorithms runs. Figure 9 illustrates the overview of a run. It provides information on the exact version of the library that was used (via git commit identifier), information regarding the duration of the run, and when it was started. Moreover, information regarding the computing node, OS, and Python version is also available. Hyperparameters are also available for each run, as shown in Figure 10. This, along with the git commit and hardware specifications, eases reproducibility. Finally, W & also automatically logs system metrics along the training process, as shown in Figure 11.
D Benchmarking dataset
This section gives more details on our third contribution. consists of dataset of training results of algorithms in MORL-Baselings applied on various environments available in MO-Gymnasium.
We also support generating weight vectors based OLS and GPI-LS methods.
21

---

Sontppoct
Centerrpee
Pareto front
Hypervolume
Runs (with filters)
concre M0Q-..
dvep treas concare Mollhing M0Q
Sparsity
IGD
trea concee Mull
Muli
Hp Nottzaoo
Figure& 8: Screenshot of the W&B dashboards providing metrics and training information in real-time to the user.
hopper Condinuous Action
Wist mukko tuc spoud
Prayy
Aather
Sade
Feohhed
Ccsaton
T.s sth
openibesc rOc Bescleschenteee
48 dxts 64 slontighost223
25.33
hesthie
Sptar Harowand mast GrUmant
6150
Figure 9: Overview of given run the W&B user interface. Notice that command line, OS and Python version, hardware specifications, and git commits are available for reproducibility.
22

---

Config
Config porameters describe your model's inputs. Learn more
Search keys
Value
algo
"GPI-PD Continuous Action"
alpha
0.6
batch_size
128
buffer_size
400,000
delay_policy_uuddate
2
dyna
true
dynamics_ _ii hta_mmiin_u_uue uncertainty
2
oentacc_ collapsed)
dynamics_rest rraal__r_____________e_ ratio
0.1
dynamics_ rollout. batch size
50,000
drnomics_rollout_fre
250
dynomics_rollout.len
5
dynamics_rollout_start
1,000
ainamics_trainfrees
250
env
'mo-hopper-v4"
gamma
0.99
gradient updates
Figure 10: Hyperparameters of a given run.
18
Passon
Pronee sillation
Figure 11: System usage metrics reported through W&B.
23

---

D.1
Experimental settings
The section provides a more detailed description of the hardware and software settings used to produce our experimental results. For further details, the full specifications of each run can be found W dashboards, as shown in Figure and 10.
Hardware specifications
The experiments conducted on the University of Luxembourg high-performance computer (Varrette et al., 2014) were carried out on the iris cluster. The compute nodes equipped with GPU contain NVidia Tesla v100 SXM2 (16 and 32 GB) and Intel Xeon Gold 6132 @ 2.6GHz CPUs. Each training job has been allocated one GPU and 7 cores using SLURM (Yoo et al., 2003). The experiments conducted on the Vlaams Supereomputer Centrum (VSC) cluster Hydra hosted by the Vrije Universiteit Brussel (VUB) used nodes equipped with NVIDIA 1100-CCIE400GE GPUs.
Training all algorithms across all environments with multiple random seeds required approximately three months of computational resources.
Hyperparameters and code
All experiments have been run on MO-Gymnasium v1.0.0 and MORL-Baselines v1.0.0. All al- gorithms have been run with seeds from 0 to 9 on each supported environment. For most runs, we kept the default hyperparameters which are based on the original papers. However, in certain instances where could not achieve satisfactory results, made minor attempt fine-tune the hyperparameters. As a result, these hyperparameters, such as the Envelope hyperparameters, differ from those specified in the original paper. Finally, for the deterministic environments, we chose to increase the learning rate and exploration as these are the most critical in such domains. A GitHub issue (https:/ //gi::///iii thub. com/LucasAleger wew.lllgeemmmm wereemmollbaaaaeeeeeeeeeeeeeeeae
ines sssuss/sss/33 gives an overview of the command line used to launch each run, along with the values of the hyperparameters.
D.2 Results
The following section presents the experimental results available in our dataset utilizing the command- line interface of openrlbenchmark (Huang et al., 2023). The provided command lines explicitly demonstrate how effortless can be to generate plots to visualize the available evaluation metrics using this tool. It is important to note that openrlbenchmark is still in the developmental phase, and therefore its API is likely to undergo changes. Furthermore, we remind the reader that these results should be interpreted as proof of concept and should not be considered empirical evidence of performance differences among the implemented algorithms. In other words, we have not conducted enough runs to make statistically valid claims and have not fine-tuned the hyperparameters for all instances. Lastly, we would like to highlight that even though we have implemented single-policy algorithms, we have not extensively tested them at this stage.
Continuous observation and continuous actions
Figure 12a and 12b show the training results for environments with continuous actions and two o2 Both graphs show that PGMORL (Xu et al., 2020) seems less efficient than GPI-based algorithms (Alegre et al., 2023) CAPQL (Lu et al., 2023) in terms of EU and Hypervolume. While CAPQL seems less sample efficient than GPI-based algorithms, the time-based plots (Figure 12b) look less categorical, especially for mo-halfcheetah. The sparsity metric is improved for CAPQL and PGMORL, as discussed in the main paper.
As a side note, we remind the reader that PGMORL is based on PPO (Schulman et al., 2017b) while the other algorithms are based on more modern single-objective RL algorithms. We be- lieve that this plays huge role in the final performances of the algorithms, e.g. if we were to reimplement PGMORL. based on SAC (Haarnoja et al., 2018) or TD3 (Fujimoto et al., 2018b) would probably obtain better performance. To make it easier understand, we chose to keep
Ourr implementation of PGMORL only supports environments with two objectives. This is because the implementation by the original authors relies on different methods for two and three objectives. Hence, PGMORL is only displayed in these environments.
24

---

POMORL
CAPOL
GPIL5
PD
POMORL
CAPQL
GPI
GSIPD
mo-bulichwetal-o4
mo-hopper-ld--d
mo-ballcheetah-oo
Hyperokime (1)
100000
30000
30030
50000
2000
50000
0.1M
0.2M
0.3M
500
1000
Sparday (3)
2000
Sponity
2000
1000
0M
0.1M
0.2M
06-
0.3M
0.2M
500
0000
1000
Expected Uittey (1)
Expotal Usitty (f)
100
200
2003
0M
0.1M
0.2M
0M
0.1M
0.2M
500
500
1000
Training steps
Time
(m)
(a) Evaluation metrics w.r.t. training samples.
(b) Evaluation metrics w.r.t. training time.
Figure 12: Evaluation metrics in environments with continuous actions spaces.
faithful to the original papers instead. The following command was used to generate the plots:
python
openrlbenchmark.co www.s_ooopssss_eee tooos_mmllt_nntricss
--filters http:///eeeeeeeeeeeeeaee.eeeeeed
wttp://weeeouuaaaateeeeeeeeeeeee 'CAPQL?c1=CAPQL'
*PCM0LL?c1-PCOORL
*GIILL Continuous Action?c1=GPI-LS' \
*GPI-P Continuous Action?c1=GPI-PD' \
--env-ids mo-halfcheetah- ho-hopper-2d-11
--pc.ncols
--pc.ncols--egem 4
-<pc.xlabel 'Training steps'
--pc.ylabel 1000000
--pc. eec.max_steppe
--output-filemame notl/mort_comtinuuus
25

---

CAPGL
GHILS
GPIPD
CAPQL
GP:45
GPIPDD
meshopperos
4.5
1.5
Hyperakme (?)
4.0
3.5
100K
100K
60
Sparity (1)
10
100
Spersity
10-
0K
100K
200K
LOOK
200K
0K
0.5K
0K
Expected Uniny (1)
15-
Expected Uxilty (f)
150
10
10-
100
100
0K
100K
200K
0K
100K
200K
0K
0.5K
1K
0K
1K
3K
Training steps
Time (m)
(a) Evaluation metrics w.r.t. training samples.
(b) Evaluation metrics w.r.t. training time.
Figure 13: Results environments with than objectives and continuous observation and actions spaces.
The plots in Figure 13a and 13b give the training results for continuous environments with more than 2 objectives. In these environments, there are no clear-cut performance differ- ences between the three algorithms. The plots were generated using the following command:
python -m pparr//bnchhaak.roop....iiii--
--filters http:p/p.......oo..oooo....i.iie
*CAPQL?c1-CAPQL'
GPI-LS Continuous Action?cI-GPI-LS*
GGPIPP Continuous Action?cl-GPI-2
--env-ids mo-hopper-ve mo no--umar-
--pc.ncols
--pc.ncols-legend 3
--pc.xlabel 'Training steps'
--pc.ylabel >>
-output-filenare nott/morl_continnonn
-- --sscn-history
Continuous observations and discrete actions
Figure 14 and 15 give the training results of MORL algorithms compatible with continuous observa- tions and discrete actions. We can that GPI-based methods outperform Envelope (Yang al., 2019) and PCN (Reymond et al., 2022) for EU and Hypervolume. Yet, there is no performance difference between the model-based algorithm (GPI-PD) and the model-free one (GPI-LS) this environment. It is worth noting that the performances of Envelope could be enhanced by tuning hyperparameters or using modern deep
tricks such as dropout layers and normalization, as done in Alegre et al. (2023). Indeed, we believe that this algorithm suffers from pathologies that it inherits from DQN (Mnih et al., 2015a), such as overestimation bias. Finally, remember that PCN designed for deterministic environments and thus is not really fit for the tested domains. Still, it performs
26

---

Envelope
PCN
GPI-S
GPHPDD
mo highway- fast-v0
10 mo-reacher-vi
Hypevolune (c)
1.0
3.5-
20000
0.8
3.0-
0.6
10000
0.4
0.2M
0.4M
0M
0.2M
0K
200K
400K
300
Spansiyy (1)
0.6
MW
100
2-
0.2
0M
0.2M
0.4M
0M
0.2M
0.4M
0K
200K
400K
Expocted Ubllty (1)
24
16.0
12.5
-20-
10.0
-40
18
0M
0.2M
0.4M
0M
0.2M
0.4M
0K
200K
400K
Training steps
Figure 14: Evaluation metrics on environments with discrete actions and continuous observations w.r. training samples.
Envelope
PCN
GPH-S
GPH-D
10 i/fo-llanalladdeeer2
mo- moo-highwwyy-lase last-vo
10
mo-reacher-vl
Hypervolune (t)
3.5
20000
3.0-
0.8
2.5
10000
1.0
Sparsity ())
200
4-
0.5
0.0
0-
0.5K
0K
1K
Expected Usitty (t)
16.0
22.5
0-
12.5
20.0
-20
17.5
0K
0.5K
0K
1K
0K
1K
Time (m)
Figure 15: Evaluation metrics on environments with discrete actions and continuous observations w.r.t training time.
27

---

Envelops
PCN
GPILS
GPIPD
Envelope
PCN
GPLLS
GPIPPD
minocart-vO
minocart-vo0
800
Hyperolume (1)
Hypervelume (t)
600-
000
2000
0M
0.2M
250
300
0.75
1.0-
Sparsity (1)
0.300
WWM
(1)
ww
0.0-
0.00
0M
0.2M
0.4M
250
100
Expected (?)
Expected Utiley (r)
0.2
0.2
0.0
Mayn
0M
0.2M
0.4M
250
300
1.00
0.75
1GD (U)
0.70
0..00
1GD
0.25
0M
0.2M
0.4M
250
10
Max. Utilay Loss (U)
1.0
0.5-
Utiliy
0.5
0.0
0M
0.2M
0.4M
300
Training steps
Time (m)
(a) Evaluation metrics w.r.t training samples.
(b) Evaluation metrics w.r. training time.
Figure 16: Results on minecart-vO; discrete actions, continuous observations and known Pareto front.
surprisingly well mo-lunar-lander.
The plots were generated using the following command:
python -m
-fiiteers hawpappgghoocaccccoooooooooeeeae
http://wwwwwa..ccc.e.ceaaaaaaaea http://pprrcuuumtttrreeeeee/sese >CC
2 Eavellpeecc1-Envellpp
*GPI-LS7C1-GPI-L'
*GPI-PD?C--GPI-DD'
--env-ids mo- lander-an no-highway-fast--u mo no-reacher-ve
--pc.ncols
-ppc.ncols-legend 4
--pc.xlabel 'Training steps'
--pc.ylabel
--pc. -ac.max_steps 400000 \
-outputtffleeaaee htel/morl_discreen
--scan-history
The performance of the algorithms on the minecart-vO environment, which has known Pareto
front enabling the computation of IGD and MUL, is depicted in Figure 16a and Figure 16b, Similar
28

---

Poned Glaaring
Mattilae MO G-Laanin
My MO Q Qeemmnn (G15)
Muttilaly Qeeamngg (GFTLS)
Seep trossre-vd
deep-993 treas are concve-vd
Trah: tree
Hypereline ITC
100
2000
01
0.61
0.2M
0.4M
0.2M
0.4M
10000
Spority [1]
-00
9000
061
0.25
0.4M
034
0.2M
0.4M
0M4
0.2M
0.4M
3M
311
0.2M
0.4M
Loss (1)
10
3.0
9.5
3.2M
Training steps
Figure 17: Evaluation metrics on deterministic environments with discrete actions and observations w.r.t training samples.
to the previous plot, we can draw the same conclusion: GPI-based methods (especially GPI-PD) appear to be more sample efficient than Envelope and PCN. However, PCN demonstrates advan- tage with its efficient implementation, making indistinguishable from GPI-based methods when evaluating w.r.t. training time (Figure 16b). The following command line generated these plots:
python -m hwtpppppwwottaaaaaaiieeeeeeeeeee
--filters ottphheww...eeeeeeeeee///e//reee
metrics-eval/mll Envelopeecc1=Envelopee
PCN7c1==CN>>
.GII-LScC-GPI-13'
*GPI-PD?C--GPI-PD
--env-ids minecart-v0
--pc.ncols
--cc.ncols-eegend 4
--pc.xlabel 'Training steps' \
--pcyylabel
--pc.max_steps 400000
--ouputtffleeaaae nort//metl_dhscrree..hooo
--ann-historyy
Discrete action and discrete observations
Figure 17 and 18 illustrate the performance of tabular algorithms trained on environments with discrete actions and observations, Notice that these runs also include more metrics as they involve environments with known Parcto front. In all environments, Pareto Q-Learning (Van Moffaert and
29

---

Poned Glaaring
Mattiiay MO Q-Laaning
MyniMdey MO Quemmnn (G15)
Muttillyy Qe (GFTL.)
Keep oc tra3uuee/
deep waw.aa--oocce
Hyyerotune (1)
3.00
2300
20000
7.5
11
50
100
25-
2.5
01-
0.5
10000
Sporott 11)
9000
Exeested
Unilty Loss 111
7.5
00
3.0
2.5-
(m)
Figure 18: Evaluation metrics on deterministic environments with discrete actions and observations
w.r.t training time.
Nowé, 2014) finds optimal Pareto front, shown by the IGD and MUL metrics reaching 0. Yet, implementation of this algorithm is limited to deterministic environments; this could enhanced by using the work of (Roijers al., 2021). The MO Q-Learning (Van Moffaert et al., 2013a) variants, described in Section 5, seem to plateau in deep-seaatrrasure-coomee as they rely on linear scalarization and cannot find the points in the concave part of the front. Finally, the GPI-LS variant of MO Q-Learning shows strong performance in all environments compared to the other variants.
The plots
were
generated
using
the following
command:
python -m
--ilteer hawppppprrrtatatttooteeeeeeeaaee
wwt.ppppooaaattteeeee///e.eeeeee netrice=evvll/mii>
Pareto q-Learming'cl-Parrte q-Learning'
"MultiPolicy q-searaingg/l-*uuuttiillii Q-Learning'
"MultiPolicy MO Q-Learning (OLS))cl-=MultiPoii MO Q-Learning (OLS)
"MultiPolicy MO Q-Learning (GP1-L9)7a1=MultiPlll.. MO Q-Learning (GPI-LS)) \
--env-ids teap-sea-treasuree deep-sea- wtw.saaaarrrsure-cccceeeeeee.ee fruit-tree-vo
-<pc.ncols
-ppc.ncols-legend
--pc.xlabel 'Training steps? \
--pc.ylabel "
--pc.max_stepp 400000
--output-fiienaame mot//morl_.ertermnnstt....
--scan-historg
30


Reviews:
[{"review_id": "ga8cXYrMND", "reviewer": "Reviewer_SyY8", "summary": "", "questions": "", "limitations": "-It would be better to explain why these five metrics and 10 state-of-the-art MORL algorithms are included.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "-This paper proposes a toolkit to collect MORL environments and state-of-the-art MORL algorithms. This collection improved the weakness of previous toolkits.\n\n-The motivation and contributions of this paper are clarified. \n\n-The main parts of the proposed toolkit are explained clearly.\n\n-The proposed toolkit can be useful for researchers in MORL.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "U57DDce2ag", "reviewer": "Reviewer_QTsf", "summary": "", "questions": "", "limitations": "The reliability of the implemented algorithms should be clearly discussed.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The MO-Gymnasium and MORL-Baselines are accessible online and well-documented. The benchmarking results are also available online. The paper established a good research platform for multi-objective RL. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sifMtJi8ON", "reviewer": "Reviewer_humL", "summary": "", "questions": "", "limitations": "The authors mention that there is a lack of extensive exploration in terms of hyperparameter tuning. This issue could undermine the trustworthiness of the benchmark results.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Provide a strong library for multi-objective reinforcement learning. This attempt is rarified in the field of reinforcement learning; therefore, the reviewer thinks that this work serves as an opportunity for improvement of MORL. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "F4yVM4mXcl", "reviewer": "Reviewer_WQpx", "summary": "", "questions": "", "limitations": "I think the authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper emphasizes the importance of establishing standardized APIs for MORL, as this would greatly enhance the reproducibility and comparability of MORL research for researchers. It brings attention to the lack of standardized APIs or centralized repositories for various benchmark problems used to evaluate MORL algorithms. Additionally, it notes that while some published works in MORL offer their code, they are not often maintained by their authors. By addressing these issues, the paper aims to provide a more accessible and reliable library, i.e., MO-Gymnasium for advancing MORL research. \n2. The authors introduce a dataset of training results for MORL-Baselines algorithms on various MO-Gymnasium environments. It allows researchers to compare new algorithms with existing baselines without retraining models from scratch, promoting scientific rigor in performance evaluations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xh5hHqOvw5", "reviewer": "Reviewer_LKgu", "summary": "", "questions": "", "limitations": "Limitations have been appropriately addressed, I do not see any negative societal impacts of the work.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper brings several substantial contributions to the RL field.\n\n1. Open-source MO-Gymnasium is introduced, a standardized set of MORL environments, allowing for the rapid construction of novel MORL environments and following the Gymnasium API with versioning, etc., facilitating results comparison; 20 diverse environments are contained.\n\n2. State-of-the-art MORL baseline open-source implementations are introduced to facilitate progress in the field. 10 MORL algorithms implementations are provided within the library. The baseline implementation development is based on the popular Stable-Baselines 3 RL algorithms library. The baseline implementations include various tools facilitating the development and evaluation of new MORL algorithms: methods to compute and analyze Pareto fronts, perform evaluation wr.t. various metrics, replay buffers, and experiment tracking tools. \n\n3. A Thorough benchmark is performed in the openrlbenchmark framework with all training curves, hyperparameters, etc. available through ML-Ops (W&B).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dymy4PIfQ6", "reviewer": "Reviewer_JBtw", "summary": "", "questions": "", "limitations": "The main limitations are listed in the last section. There is no negative societal impact as far as I am concerned.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Multi-objective decision-making itself is an interesting and vital problem since MO is a very common characteristic in the real world. A good benchmark could greatly benefit researchers interested in this field.\n* Implementation and benchmarking of a wide range of algorithms. They provide a good reference and can be a good starting point for future research.\n* An emphasis on reproducibility. The environments and algorithms come with a standardized and clean API. The authors also made efforts to make the experiment results easily reproducible. \n* An well-organized open-source codebase actively developed by the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: I think that this is a good paper. All the six reviewers think that this paper is acceptable. I also think that this paper is acceptable.
Decision: Accept (Poster)