SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Florian Felten\*,1,7 Lucas N. Alegre\*,2,3,7 Ann Nowé³ Ana L. C. Bazzan² El-Ghazali Talbi¹,⁴ Grégoire Danoy¹,⁵ Bruno C. da Silva⁶
$^{1}$ SnT, University of Luxembourg  $^{2}$ Institute of Informatics, Federal University of Rio Grande do Sul  $^{3}$ Artificial Intelligence Lab, Vrije Universiteit Brussel  $^{4}$ CNRS/CRIStAL, University of Lille  $^{5}$ FSTM/DCS, University of Luxembourg  $^{6}$ University of Massachusetts  $^{7}$ Farama Foundation {florian.felten,gregoire.danoy}@uni.lu {lnalegre,bazzan}@inf.ufrgs.brann.nowe@vub.be el-ghazali.talbi@univ-lille.fr bsilva@cs.umass.edu
Multi-objective reinforcement learning algorithms (MORL) extend standard reinforcement learning (RL) to scenarios where agents must optimize multiple—potentially conflicting—objectives, each represented by a distinct reward function. To facilitate and accelerate research and benchmarking in multi-objective RL problems, we introduce a comprehensive collection of software libraries that includes: (i) MO-Gymnasium, an easy-to-use and flexible API enabling the rapid construction of novel MORL environments. It also includes more than 20 environments under this API. This allows researchers to effortlessly evaluate any algorithms on any existing domains; (ii) MORL-Baselines, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed to provide a solid foundation for advancing research. Notably, all algorithms are inherently compatible with MO-Gymnasium; and (iii) a thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms, tested across various challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular state-of-the-art method.[2]
Research in reinforcement learning (RL) algorithms (Sutton and Barto, 2018) has gained significant attention in recent years, in great part due to its remarkable success in a range of challenging problems (Mnih et al., 2015b; Silver et al., 2017; Bellemare et al., 2020). This led to a substantial increase in the number of papers published in the field every year. The rapid growth of RL research, however, has not necessarily been accompanied by the design of well-thought and reliable tools allowing for appropriate evaluation practices, resulting in a reproducibility crisis within the field. This resulted, e.g., in researchers often questioning the validity and reproducibility of results presented in influential papers (Agarwal et al., 2021; Patterson et al., 2023).
There are several reasons why reproducibility is often a challenge in RL. A key factor is the significant amount of time required to train RL agents, which makes it difficult for researchers to gather sufficient data to perform rigorous statistical analyses of empirical results. As a consequence, some authors may claim superior performance over state-of-the-art techniques without adequate evidence.
*Authors contributed equally to this work.
$^{2}$ MO-Gymnasium and MORL-Baselines are available at https://github.com/Farama-Foundation/mo-gymnasium and https://github.com/LucasAlegre/morl-baselines, respectively. The benchmark results are available on openrlbenchmark: https://wandb.ai/openrlbenchmark/MORL-Baselines.
Additionally, papers often do not provide sufficient information—such as hyperparameter values and implementation optimizations—to allow for the reliable reproducibility of their results. Finally, the use of environment implementations that lack standardization also contributes to this challenge. To address such issues, several libraries and experimentation protocols have been proposed in the RL ecosystem (Brockman et al., 2016; Agarwal et al., 2021; Raffin et al., 2021; Huang et al., 2022b).
As RL garners increasing interest, its subfield, Multi-Objective Reinforcement Learning (MORL), is simultaneously attracting notable attention within the RL research community (Hayes et al., 2022). MORL algorithms tackle problems where an agent has to optimize multiple—possibly conflicting—objectives. Each of these objectives is represented via a distinct reward function. In this setting, agents typically aim to find optimal decision-making policies defined with respect to different compromises or trade-offs between the objectives. Examples of MORL problems include the widely-used set of Mujoco (Todorov et al., 2012) tasks, which model compromises between an agent's objectives. As an example, the reward function of the half-cheetah agent is a weighted combination of velocity- and energy-related terms, where the weights are pre-determined and kept constant. However, allowing for such weights (i.e., for the relative importance of objectives) to change may induce significantly different optimal behaviors (Xu et al., 2020). While the goal of standard RL algorithms is to learn a single policy, specialized in optimizing a single reward function, MORL algorithms typically search for a set of policies such that given any compromises between objectives, a corresponding optimal (or near-optimal) policy is in the set. Designing and properly evaluating MORL algorithms shares the difficulties encountered when designing and empirically testing RL techniques—with additional challenges particular to multi-objective settings. Yet, to the best of our knowledge, there are currently no public standard libraries providing reliable implementations of widely-used MORL domains and state-of-the-art MORL algorithms, designed particularly to facilitate research in the field.
In this paper, we introduce a comprehensive suite of benchmark MORL environments and reliable implementations of widely-used MORL algorithms, designed to facilitate the construction of reproducible empirical performance evaluation of existing and novel multi-objective techniques. First, we introduce MO-Gymnasium (Section 4), an easy-to-use and flexible API enabling the rapid construction of novel MORL environments. As of now, this API includes over 20 environments with diverse characteristics. This allows researchers to evaluate, with minimal effort, any algorithms compatible with our extendable API in any existing domains. Secondly, we introduce MORL-Baselines (Section 5), a collection of reliable and efficient implementations of state-of-the-art MORL algorithms designed to provide a solid foundation for advancing MORL research. Notably, all such algorithms are inherently compatible with MO-Gymnasium. Finally, in Section 6 we provide a thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms tested across various challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular state-of-the-art method.
The challenge of reproducing experimental results in machine learning research is widely acknowledged. The significance of this issue has risen to such a degree that top-tier conferences like NeurIPS have implemented reproducibility programs aimed at improving the standards for conducting, communicating, and evaluating research in the field (Pineau et al., 2020). The RL field is directly affected by this issue, as highlighted by different authors. For example, Engstrom et al. (2020) show that optimizations at the implementation level of deep RL algorithms can be more impactful than experimenting with different algorithms. Similarly, Huang et al. (2022a) investigated and identified the existence of 37 particular code-level tricks necessary to achieve state-of-the-art results when deploying the PPO algorithm (Schulman et al., 2017a). Agarwal et al. (2021) point out the lack of statistically significant results in numerous papers published in the field, despite many of them claiming to introduce techniques that outperform the state-of-the-art. Finally, an in-depth discussion of issues pertaining to methodologies used to conduct empirical evaluation in RL is presented by Patterson et al. (2023). These issues include, among others, inadequate tuning of the hyperparameters of baselines algorithms when deployed on new environments, averaging performance metrics over a limited number of runs, absence of random seed control, and environment overfitting.
To address these concerns, several RL libraries have been developed to provide more reliable baselines. For example, Gymnasium (Towers et al., 2023) (formerly known as OpenAI Gym (Brockman et al.,
2016)) provides a standard API and a collection of reference environments for RL research and experimentation. Furthermore, various libraries have been published that contain well-tested, reliable, and continually maintained implementations of RL algorithms; e.g., Stable-Baselines 3 (Raffin et al., 2021) and cleanRL (Huang et al., 2022b). Finally, recent initiatives such as openrlbenchmark (Huang et al., 2023) facilitate the analysis of various learning metrics, and tackle the reproducibility challenge via experiment tracking software such as Weights and Biases, which makes it easier to, e.g., make hyperparameters of different baselines publicly available (Biewald, 2020). Other subfields of RL, such as multi-agent RL (MARL), have also benefited from standard baseline libraries. In MARL, PettingZoo (Terry et al., 2021) is often used to design novel environments, and EPyMARL (Papoudakis et al., 2021) is commonly used when designing new learning algorithms. Zhu et al. (2023) recently introduced D4MORL, a repository that includes datasets specifically designed to evaluate MORL algorithms in the offline setting. Although these subareas of RL have reaped the benefits from standardized libraries, to the best of our knowledge, there are currently no publicly available (and widely adopted) standard libraries providing reliable implementations of widely-used MORL domains and state-of-the-art MORL algorithms, designed to facilitate research in the field.
As discussed by Hayes et al. (2022); Cassimon et al. (2022), various benchmark problems have been proposed to evaluate MORL methods. However, these benchmarks have not yet been made available via standardized APIs or centralized repositories. Arguably, this has made the experimental reproducibility of MORL algorithms harder, time-consuming, and error-prone. MORL-Flue (Vamplew et al., 2017) represents an attempt to establish a centralized repository of MORL benchmarks. However, this library has not been widely adopted due to the fact that it is implemented in Java and targets tabular problems, whilst the community currently focuses on using Python and deep RL techniques. Our first contribution, MO-Gymnasium, addresses this issue by introducing a standard API and set of reference environments for MORL.
Although some published works on MORL make their code publicly available (Yang et al., 2019; Abels et al., 2019; Xu et al., 2020), these implementations are not often maintained by their authors. This makes reproducing empirical results a usually time-consuming and error-prone process. Our second contribution, MORL-Baselines, tackles this challenge. It includes a set of clear, well-structured, regularly maintained, and reliable implementations of MORL algorithms. Importantly, all such algorithms are compatible with MO-Gymnasium's API.
Finally, recall that the performance of newly proposed algorithms is generally compared with previously-published baselines. However, the methodology employed in conducting experiments is not always scientifically rigorous. To address this issue, we introduce a dataset of training results of MORL-Baselines algorithms when evaluated on all MO-Gymnasium environments. This dataset is available through openrlbenchmark and includes information about all hyperparameters used by each algorithm, in each experiment, thereby allowing researchers to compare new algorithms with such existing baselines without having to retrain models from scratch.
3 Multi-objective reinforcement learning
In MORL, the interaction of an agent with its environments is modeled via a multi-objective Markov decision process (MOMDP) (Roijers et al., 2013). MOMDPs differ from standard MDPs (Sutton and Barto, 2018) only in that a MOMDP's reward function is vector-valued. A MOMDP is defined as a tuple  $M \triangleq (\mathcal{S}, \mathcal{A}, p, \mathbf{r}, \mu, \gamma)$ , where  $\mathcal{S}$  is a state space,  $\mathcal{A}$  is an action space,  $p(\cdot | s, a)$  is the distribution over next states given state  $s$  and action  $a$ ,  $\mathbf{r} : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \mapsto \mathbb{R}^m$  is a multi-objective reward function containing  $m$  objectives,  $\mu$  is an initial state distribution, and  $\gamma \in [0,1)$  is a discounting factor. A policy  $\pi : \mathcal{S} \mapsto \mathcal{A}$  is a function mapping states to actions. Let  $S_t, A_t$ , and  $\mathbf{R}_t = \mathbf{r}(S_t, A_t, S_{t+1})$  denote the random variables corresponding to state, action, and vector reward, respectively, at time step  $t$ . The multi-objective value function of a policy  $\pi$  in state  $s$  is defined as
$$
\mathbf {v} ^ {\pi} (s) \triangleq \mathbb {E} _ {\pi} \left[ \sum_ {i = 0} ^ {\infty} \gamma^ {i} \mathbf {R} _ {t + i} \mid S _ {t} = s \right], \tag {1}
$$
where  $\mathbb{E}_{\pi}[\cdot ]$  denotes expectation with respect to trajectories induced by the policy  $\pi$ . We denote  $\mathbf{v}^{\pi}\triangleq \mathbb{E}_{S_0\sim \mu}[\mathbf{v}^{\pi}(S_0)]$  as the value vector of policy  $\pi$ . Notice that  $\mathbf{v}^{\pi}$  is an  $m$ -dimensional vector whose  $i$ -th component is the expected return of  $\pi$  under the  $i$ -th objective.
In contrast to single-objective RL, comparing the values of two different policies is not straightforward in MORL. For instance, a policy that achieves higher expected return with respect to one objective may, as a result, have lower performance with respect to other objectives. Hence, in MORL, policies are typically evaluated and compared in terms of a user utility function (or scalarization function),  $u: \mathbb{R}^m \mapsto \mathbb{R}$ , which is a mapping from the multi-objective value of policy  $\pi$ ,  $\mathbf{v}^\pi$ , to a scalar.
Let  $u$  be a monotonically increasing utility function; i.e., a function such that the utility increases if the value of one objective can be improved without decreasing the value of other objectives. Then, the value vector  $\mathbf{v}^{\pi}$  of an optimal policy with respect to  $u$  is in a Pareto frontier (PF). The PF of a MOMDP is a set of nondominated multi-objective value vectors:
$$
\mathcal {F} \triangleq \left\{\mathbf {v} ^ {\pi} \mid \not \exists \pi^ {\prime} \text {s . t .} \mathbf {v} ^ {\pi^ {\prime}} \succ_ {p} \mathbf {v} ^ {\pi} \right\}, \text {w h e r e} \mathbf {v} ^ {\pi} \succ_ {p} \mathbf {v} ^ {\pi^ {\prime}} \iff (\forall i: v _ {i} ^ {\pi} \geq v _ {i} ^ {\pi^ {\prime}}) \wedge (\exists i: v _ {i} ^ {\pi} > v _ {i} ^ {\pi^ {\prime}}). \tag {2}
$$
In particular, linear utility functions linearly combine the value of a policy under each of the  $m$  objectives using a set of weights  $\mathbf{w} \in \mathbb{R}^m$ :  $u(\mathbf{v}^\pi, \mathbf{w}) \triangleq \mathbf{v}^\pi \cdot \mathbf{w}$ , where each element of  $\mathbf{w} \in \mathbb{R}^m$  specifies the relative importance of each objective. The space of weight vectors,  $\mathcal{W}$ , is an  $m$ -dimensional simplex so that  $\sum_{i} w_i = 1$ ,  $w_i \geq 0$ ,  $i = 1, \dots, m$ . Under such a utility scheme, the PF corresponds to a convex coverage set (CCS), as shown by Roijers et al. (2013).
MORL algorithms can be classified in terms of whether they are single-policy (i.e., they learn a single policy that optimizes one particular utility  $u$ ), or multi-policy (i.e., they learn a set of policies with the goal of approximating the PF). When non-linear utility functions are considered, the optimization procedure of a MORL problem can also be defined with respect to the expected scalarized return (ESR),  $\mathbb{E}[u(\sum_{t=0}^{\infty} \gamma^{t}\mathbf{R}_{t})]$ , or the scalarized expected return (SER),  $u(\mathbb{E}[\sum_{t=0}^{\infty} \gamma^{t}\mathbf{R}_{t}])$ . For a thorough review of MORL concepts and definitions, we refer the reader to Hayes et al. (2022).
4 MO-Gymnasium
In this section, we introduce our first contribution: MO-Gymnasium.3 MOGymnasium is an easy-to-use and flexible API enabling the rapid construction of novel MORL environments, and a collection of regularly maintained and thoroughly tested environment implementations.
import mo_gymnasium as mo_gym  
env = mo_gym.make('minecart-v0')  
obs, info = env.reset()  
for _ in range(1000):  
    action = policy(obs)  
    obs, rews, term, trunc, info = env.step(action)
Figure 1: An example of how MO-Gymnasium can be used.
Unlike classical ML settings, which rely on fixed datasets, RL problems typically do not; this makes replication of experimental results challenging. Indeed, even small differences in the definition of environment can have a significant impact on the performance of algorithms.
To address this issue and facilitate research when it comes to the standard RL settings, Gymnasium (Towers et al., 2023) (formerly Gym (Brockman et al., 2016)) introduces an API and collection of versioned environments. With millions of downloads, this is currently the de facto standard library in RL, enabling researchers to easily test their algorithms on a variety of problems. Despite its widespread use, Gymnasium is limited to modeling single-objective MDPs. It has since been expanded in various ways; e.g., PettingZoo extends it to MARL settings and D4MORL to offline MORL. To the best of our knowledge, there are currently no publicly available and widely adopted libraries providing reliable implementations of MORL domains and state-of-the-art MORL algorithms, designed to facilitate research in the field.
We address this limitation by introducing MO-Gymnasium—previously known as MO-Gym (Alegre et al., 2022b). MO-Gymnasium's API is designed to be as similar as possible to Gymnasium's API. This allows it to inherit many of the features in Gymnasium, such as wrappers—features that allow individual properties of a domain to be modified—while extending the original API only where and when necessary. This makes MO-Gymnasium automatically backward compatible with a wide range of MORL benchmark domains. The key difference between these two frameworks is that, in MO-Gymnasium, rewards returned after the execution of an action (i.e., after a call of the step method) are vectors rather than scalars (see Figure 1). MO-Gymnasium is available on PyPI and can be installed via pip install mo-gymnasium. Importantly, we highlight that MO-Gymnasium is
<sup>3</sup>The documentation of MO-Gymnasium is available at https://mo-gymnasium.farama.org.
Figure 2: A few of the environments available in MO-Gymnasium. From left to right: deep-sea-treasure, mo-halfcheetah, mo-supernario, minecart, and resource-gathering.
an official part of the projects maintained by the Farama Foundation, and is considered a mature and well-supported library by the research community.
**Environments.** Currently, MO-Gymnasium includes over 20 environments commonly used in the MORL literature—including environments with discrete and continuous state and action spaces—such as deep-sea-treasure (Vamplew et al., 2011), four-room (Alegre et al., 2022a), mo-supernario (Yang et al., 2019), minecart (Abels et al., 2019), and mo-halfcheetah (Xu et al., 2020). In Figure 2 we depict a few of the currently available environments. See Appendix B for a detailed description of each environment. This large collection of environments allows designers to thoroughly assess the performance of novel algorithms in different scenarios. In environments in which the true PF is known, it can be accessed via the pareto_front() method available in the MO-Gymnasium's API. Additionally, for reproducibility purposes, each environment is labeled with a version number; e.g., "-v0". Each time an environment is modified in a way that may affect algorithms' performances, the domain's version number is incremented.
Wrappers. MO-Gymnasium introduces MORL-specific wrappers such as  $MONormalizeReward$ , which normalizes a given component of the reward vector; and LinearReward, a wrapper that linearly scalarizes the reward function of a MOMDP environment, transforming it into a standard MDP. The latter feature makes MO-Gymnasium directly compatible with widely-used RL libraries compatible with Gymnasium, such as Stable-Baselines 3 (Raffin et al., 2021) and cleanRL (Huang et al., 2022b).
5 MORL-Baselines
In this section, we introduce MORL-Baselines<sup>4</sup>, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed to provide a solid foundation for advancing research in MORL. To the best of our knowledge, this stands as the first open-source repository encompassing a multitude of MORL algorithms. Notably, all such algorithms are inherently compatible with MO-Gymnasium's API.
It is well known that the performance of learning algorithms is closely tied to their specific implementations (Engstrom et al., 2020). Unfortunately, details such as implementation-specific optimizations are seldom discussed in research papers, which makes reproducibility and comparisons challenging. To address this issue, some authors provide access to their codebases. While this is a positive step, codebases are often not regularly maintained and may become outdated, rendering replication difficult. Thus, libraries like Stable-Baselines 3 (Raffin et al., 2021) and cleanRL (Huang et al., 2022b) have been designed with the goal of regularly maintaining state-of-the-art algorithms. These codebases are well documented, thoroughly tested, and offer top-notch performance on the algorithms they implement. This allows researchers to start coding novel methods by extending existing implementations rather than starting from scratch. These libraries also provide useful tools for research purposes, such as efficient replay buffer implementations, performance reports, and evaluation methods.
MORL-Baselines includes more than 10 state-of-the-art MORL algorithms, all of which are compatible with MO-Gymnasium. Up to this point, no similar libraries were available. MORL-Baselines offers a range of features to aid researchers in designing new algorithms, such as methods to compute and analyze Pareto fronts, perform evaluation w.r.t. various metrics, replay buffers, and experiment tracking tools. Below, we provide a list of the algorithms currently available via MORL-Baselines, along with a description of the settings in which they are applicable.
4MORL-Baseline documentation is available at: https://lucasalegre.github.io/mor1-baseline.
Table 1: Algorithms currently implemented in MORL-Baselines. (*) PCN and PQL are designed to tackle deterministic environments. (** OLS is an algorithm-agnostic method for generating reward weights, or preferences; it does not assume any particular type of observation or action spaces.
Algorithm	Single or multi-policy	Utility function	Observation space	Action space
MOQL (Van Moffaert et al., 2013a)	Single	Linear	Disc.	Disc.
EUPG (Rojers et al., 2018a)	Single	Non-linear, ESR	Disc.	Disc.
MPMOQL (Van Moffaert et al., 2013a)	Multi	Linear	Disc.	Disc.
PQL (Van Moffaert and Nowé, 2014)	Multi	Non-linear, SER (*)	Disc.	Disc.
OLS (Rojers, 2016)	Multi	Linear	/(**)	/(**)
Envelope (Yang et al., 2019)	Multi	Linear	Cont.	Disc.
PGMORL (Xu et al., 2020)	Multi	Linear	Cont.	Cont.
PCN (Reymond et al., 2022)	Multi	Non-linear, ESR/SER (*)	Cont.	Disc.
GPI-LS & GPI-PD (Alegre et al., 2023)	Multi	Linear	Cont.	Any
CAPQL (Lu et al., 2023)	Multi	Linear	Cont.	Cont.
5.1 Implemented algorithms