SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
r _ {3} (s, a, s ^ {\prime}) = f \left(s ^ {\prime}\right) - f (s), \text {w h e r e} f (s) = - 1 \text {i f d r o p p e d a n y b o t t l e s i n} s, \text {e l s e} 0.
$$
B.4 fishwood-v0
This is a simple MORL problem in which the agent controls a fisherman who can either fish or go collect wood (Roijers et al., 2018b). There are only two discrete states, and the agent can decide between fishing or collecting wood, i.e.  $\mathcal{S} = \{\text{fishing, woods}\}$  and  $\mathcal{A} = \{\text{go_fish, go_wood}\}$ . The multi-objective reward function of this domain,  $\mathbf{r}(s,a,s') \in \mathbb{R}^2$ , is defined as:
$$
r_{1}(s,a,s^{\prime}) = +1\text{if in the woods, with} 90\% \text{probability, else} 0,
$$
$$
r _ {2} \left(s, a, s ^ {\prime}\right) = + 1 \text {if in fishing , with} 10 \% \text {probability}, \text {else} 0.
$$
B.5 four-room-v0
This is a classic grid world environment in the literature of successor features (SFs) (Barreto et al., 2017). Importantly, Alegre et al. (2022a) showed that any problem in the SFs framework can be mapped to a MORL problem. In this domain, the agent can collect three different types of items (see Figure 7(e)). Its state space is defined by the  $x, y$  coordinates of the agent as well as a binary vector stating which items have been collected, the actions are going into one of the four cardinal directions. Formally,  $\mathcal{S} = \{0, \dots, 12\}^2 \times \{0, 1\}^{12}$ , and  $\mathcal{A} = \{\text{up}, \text{down}, \text{left}, \text{right}\}$ . The multi-objective reward function of this domain,  $\mathbf{r}(s, a, s') \in \mathbb{R}^3$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = + 1 \text {i f c o l e c t e d a b l e}
$$
$$
r _ {2} (s, a, s ^ {\prime}) = + 1 \text {i f c o l l e c t e d a g r e e n t r i a n g l e (o r i s i n g o a l l o c a t i o n) , e l s e 0 ,}
$$
$$
r _ {3} (s, a, s ^ {\prime}) = + 1 \text {i f c o l e c t e d a r e d c i r c l e (o r i s i n g o a l l o c a t i o n) , e l s e 0 .}
$$
B.6 fruit-tree-v0
This domain is modeled as a full binary tree of parameterized depth ( $d \in \{5,6,7\}$ ) (Yang et al., 2019). The agent starts at the root node and can choose to move left or right until reaching a leaf node. The state space encodes the current position in the tree as a vector, which contains the current depth and the corresponding node:  $S = \{0,\dots,d\} \times \{0,\dots,2^{d - 1}\}$ . The action space is defined as  $\mathcal{A} = \{\text{left},\text{right}\}$ . Every leaf contains a fruit with a different value for the nutrients: protein, carbs, fats, vitamins, minerals, and water. The multi-objective reward function of this domain,  $\mathbf{r}(s,a,s') \in \mathbb{R}^6$ , is defined as:
$$
r _ {i} (s, a, s ^ {\prime}) = \text {v a l u e o f n u t r i e n t} i \text {i n} s ^ {\prime}, \text {f o r} i = 1 \dots 6.
$$
B.7 mo-mountaincar-v0
This domain, introduced by Vamplew et al. (2011), is a multi-objective version of the classic mountain car problem (Moore, 1990). The state space is defined by a vector stating the agent's current  $x$ -position and its velocity,  $S = [-1.2, 0.6] \times [-0.07, 0.07]$ . Its actions consist in accelerating to the left, right, or doing nothing:  $\mathcal{A} = \{\text{reverse, forward, none}\}$ . Unlike the single-objective version, the agent is also penalized for applying the reverse and forward actions, which leads to different policies depending on the trade-offs between these penalties. The multi-objective reward function of this domain,  $\mathbf{r}(s, a, s') \in \mathbb{R}^3$ , is defined as:
$$
r _ {2} (s, a, s ^ {\prime}) = - 1 \text {i f} a = \text {r e v e r s e}, \text {e l s e} 0,
$$
$$
r _ {3} (s, a, s ^ {\prime}) = - 1 \text {i f} a = \text {f o r w a r d}, \text {e l s e} 0.
$$
A continuous action version of this domain is also available with identifier mo-mountaincar-continuous-v0. In this version, the action space is a speed vector  $\mathcal{A} = [-1,1]$ , and the reward function contains two components: a time penalty and a fuel penalty.
$$
r _ {2} (s, a, s ^ {\prime}) = - \| a \| ^ {2}.
$$
B.8 minercart-v0
This domain consists of a cart that must collect two different ores and return them to the base while minimizing fuel consumption (Abels et al., 2019). The agent perceives its environment through a 7-dimensional vector containing the agent  $x, y$  position, the current speed of the cart, its orientation (sin and cos), and the percentage of occupied capacity in the cart by each ore:  $S = [-1,1]^5 \times [0,1]^2$ . The agent has the choice between 6 actions:  $\mathcal{A} = \{\text{mine, left, right, accelerate, brake, do nothing}\}$ . Each mine in Figure 7(h) has a different distribution over two types of ores. Fuel is consumed at every time step, and extra fuel is consumed when the agent accelerates or selects the mine action. A version of this domain with deterministic values for the ores is also available as minecart-deterministic- $\nu O$ . The multi-objective reward function of this domain,  $\mathbf{r}(s,a,s') \in \mathbb{R}^3$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {q u a n t i t y o f o r e 1 c o l l e c t e d i f} s ^ {\prime} \text {i s i n s t e d t h e b a s e , e l s e 0 ,}
$$
$$
r _ {2} (s, a, s ^ {\prime}) = \text {q u a n t i t y o f o r e 2 c o l l e c t e d i f} s ^ {\prime} \text {i s i n s i d e t h e b a s e , e l s e 0 ,}
$$
$$
r _ {3} (s, a, s ^ {\prime}) = - 0. 0 0 5 - 0. 0 2 5 \mathbb {1} \{a = \text {a c c e l e r a t e} \} - 0. 0 5 \mathbb {1} \{a = \text {m i n e} \}.
$$
B.9 mo-lunar-lander-v2
This domain is a multi-objective version of the lunar lander environment available in Gymnasium (Towers et al., 2023). The state is an 8-dimensional vector: the  $x, y$  coordinates of the lander, its linear velocities in  $\mathbf{x} \& \mathbf{y}$ , its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground or not:  $\mathcal{S} = [-1.5, 1.5]^2 \times [-5, 5] \times [-\pi, \pi] \times [-5, 5] \times \{0, 1\}^2$ . The action space consists of four actions:  $\mathcal{A} = \{\text{do nothing, fire left orientation engine, fire main engine, fire right engine}\}$ . A continuous action version of this domain is also available with identifier mo-lunar-lander-continuous- $\nu$ 2. In this variant, the first coordinate of the action determines the throttle of the main engine, while the second coordinate specifies the throttle of the lateral boosters,  $\mathcal{A} = [-1, 1]^2$ . The multi-objective reward function of this domain,  $\mathbf{r}(s, a, s') \in \mathbb{R}^4$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = + 1 0 0 \text {i f l a n d e d s u c c e s s f u l l y , - 1 0 0 i f c r a s h e d , e l s e 0},
$$
$$
r _ {2} (s, a, s ^ {\prime}) = \text {s h a p i n g r e w a r d},
$$
$$
r _ {3} (s, a, s ^ {\prime}) = \text {f u e l c o s t o f t h e m a i n e g r i n e}
$$
$$
r _ {4} (s, a, s ^ {\prime}) = \text {f u e l c o s t}
$$
B.10 mo-highway-v0
This domain is a multi-objective version of the autonomous driving environment introduced by Leurent (2018). In this domain, the agent controls a vehicle on a multilane highway populated with other vehicles, it perceives its coordinates as well as coordinates from other vehicles. Formally, the state space is given by an array of size  $V \times F$ , where  $V$  represents the number of vehicles in the environment and  $F$  are the features describing each vehicle, e.g. velocity, position, angle. The agent's actions consist in changing lanes, accelerating or decelerating, or doing nothing, i.e.  $\mathcal{A} = \{\text{lane left, idle, lane right, faster, slower}\}$ . The multi-objective reward function of this domain,  $\mathbf{r}(s,a,s') \in \mathbb{R}^3$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {n o r m a l i z e d f o r w a r d s p e e d o f t h e v e h i c l e},
$$
$$
r _ {2} (s, a, s ^ {\prime}) = \text {b o u n s f o r d r i v i n g i n t h e r i g h t m o s t l a n e},
$$
$$
r _ {3} (s, a, s ^ {\prime}) = - 1 \text {i f v e h i c l e c r a s h e d , e l s e 0 .}
$$
All three reward functions above are zeroed when the agent is off the road.
<sup>10</sup>See https://gymnasium.farama.org/environments/box2d/lunar_lander for details on the shaping reward.
B.11 mo-highway-fast-v0
This is a more computationally efficient version of mo-highway-v0 with reduced number of lanes and vehicles. The reward function, state, and action spaces are kept the same as the original environment.
B.12 water-reservoir-v0
This domain simulates a water reservoir and is implemented as defined by Castelletti et al. (2012). The agent perceives a floating point number corresponding to the current amount of water in the reservoir:  $S \subset \mathbb{R}^{+}$ . The agent executes a continuous action corresponding to the amount of water the dam will release in a day:  $\mathcal{A} \subset \mathbb{R}^{+}$ . The action space can also be specified to be normalized, in which case the action is expressed as a percentage of water to release:  $\mathcal{A} = [0, 1]$ . The multi-objective reward function of this domain,  $\mathbf{r}(s, a, s') \in \mathbb{R}^4$ , is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {c o s t d u e t o e x c e s s l e v e l w . w r . t . a f l o o d i n g t h r e s h o l d (u p s t r e a m)},
$$
$$
r _ {2} (s, a, s ^ {\prime}) = \text {d e f i c i t}
$$
$$
r _ {3} (s, a, s ^ {\prime}) = \text {d e f i c i t}
$$
$$
r _ {4} (s, a, s ^ {\prime}) = \text {c o s t d u e t o e x c e s s l e v e l w . w . r . t . a f l o o d i n g t h r e s h o l d (d o w n s t r e a m)}.
$$
By default, only rewards  $r_1$  and  $r_2$  are used. However, the user has the option to instantiate the environment using all four reward functions.
B.13 mo-reacher-v4
This domain is also a classic in the SFs literature (Barreto et al., 2017; Alegre et al., 2022a). It consists of a two-joint robot arm that must reach different target locations with the tip of its arm, implemented using the Mujoco robotics simulator (Todorov et al., 2012). The agent's state space  $S \subset \mathbb{R}^6$  consists of the sine and cosine of the angles of the central and elbow joints, as well as their angular velocities. The action space, originally continuous, is discretized using 3 bins per dimension corresponding to maximum positive torque (+1), negative torque (-1), and zero torque for each actuator. This results in a total of 9 possible actions:  $\mathcal{A} = \{-1,0,1\}^2$ . The multi-objective reward function  $\mathbf{r}(s,a,s') \in \mathbb{R}^4$  is defined as:
$$
r _ {i} (s, a, s ^ {\prime}) = 1 - 4 \Delta \left(\text {t a r g e t} _ {i}\right), \text {f o r} i = 1 \dots 4, \tag {3}
$$
where  $\Delta (\mathrm{target}_i)$  is the Euclidean distance between the tip of the robot's arm and the  $i$ -th target's location.
B.14 mo-hopper-v4
This domain is a multi-objective version of the Gymnasium's Hopper-v4 environment. The state space  $S \subset \mathbb{R}^{11}$  consists in velocities, angles, and position of each part of the robot. An action represents the torques applied at the hinge joints,  $\mathcal{A} = [-1,1]^3$ . In this modified version, the agent must balance optimizing for its forward speed, jumping height, and energy cost. A version of this environment with a two-dimensional reward function is available with identifier mo-hopper-2d-v4. The multi-objective reward function  $\mathbf{r}(s,a,s') \in \mathbb{R}^3$  is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {v e l o c i t y}
$$
$$
r _ {2} (s, a, s ^ {\prime}) = \text {h e i g h t o f t h e a g e n t o v e r t h e z - a x i s},
$$
$$
r _ {3} (s, a, s ^ {\prime}) = - | | a | | _ {2} ^ {2}.
$$
B.15 mo-halfcheetah-v4
This domain is a multi-objective version of the Gymnasium's HalfCheetah-v4 environment. The state space encodes the velocities, angles, and position of each part of the robot:  $S \subset \mathbb{R}^{17}$ . Also, the actions are the torque to apply at each joint:  $\mathcal{A} = [-1, 1]^6$ . The multi-objective reward function  $\mathbf{r}(s, a, s') \in \mathbb{R}^2$  is defined as:
$$
r _ {1} (s, a, s ^ {\prime}) = \text {f o r w a r d s p e e d o f t h e a g e n t},
$$
$$
r _ {2} (s, a, s ^ {\prime}) = - | | a | | _ {2} ^ {2}.
$$
B.16 mo-supermario-v0
This domain consists of the first level of the Super Mario Bros. video game. It is implemented using Gym Super Mario (Kauten, 2018). In this environment, the observations consist of RGB frames of the game, and the actions are the following combination of buttons:  $\mathcal{A} = \{\mathrm{NOOP},\mathrm{right},\mathrm{right} + \mathrm{A},\mathrm{right} + \mathrm{B},\mathrm{right} + \mathrm{A} + \mathrm{B},\mathrm{A},\mathrm{left}\}$ . The multi-objective reward function  $\mathbf{r}(s,a,s') \in \mathbb{R}^5$  is defined similarly as defined by Yang et al. (2019):
$r_1(s, a, s') = \text{increase in the x-coordinate of Mario}$ ,
$r_2(s, a, s') = -\text{time elapsed executing action } a,$
$r_3(s,a,s') = -25$  if Mario died, else 0,
$r_4(s, a, s') = 100 \times \# \text{coins collected},$
$r_5(s, a, s') = \text{increase in score by killing an enemy.}$
C MORL-Baselines
This section describes MORL-Baselines in more detail. MORL-Baselines is a comprehensive library that offers dependable, validated, well-documented, and efficient implementations of MORL algorithms. The library goes beyond supporting existing algorithm implementations by equipping researchers with valuable tools to streamline the process of designing future MORL algorithms. In the remainder of this section, we describe the algorithms, features, and provide a few examples of the user interface brought by this library.
C.1 Algorithms
All the algorithms currently supported in MORL-Baselines are listed in Table 1 of the main paper. A few algorithms also have variants that the user can select by instantiating the agents with different parameters. In particular, our implementation of Multi-Policy MO Q-Learning (Van Moffaert et al., 2013a) can be derived in multiple ways. For instance, the weights can be generated randomly, using OLS (Roijers, 2016), or using GPI-LS (Alegre et al., 2023). Moreover, using GPI (Barreto et al., 2017) to select actions can be enabled or not, as well as using model-based learning with GPI-PD (Alegre et al., 2023).
More details on the implementations are available in our documentation and GitHub repository. For example, see the documentation of PGMORL https://lucasalegre.github.io/morl-baselines/algos/multi_policy/pgmorl/.
C.2 Additional features
Aside from the algorithms, MORL-Baselines provides a set of utilities allowing the user to construct novel MORL algorithms. These are listed below.
Pareto frontier filters. Our library provides a set of functions that permit to filter Pareto-dominated points out of a set of points. The signatures and documentation of these functions can be found in our documentation at https://lucasalegre.github.io/morl-baselines/features/pareto.
Policies evaluation. In order to build a Pareto front of policies, it is necessary to assess the values of the policies currently undergoing training in the environment. To standardize the evaluation process and enable fairer comparisons, we offer a collection of functions specifically designed for evaluating these policies. Furthermore, we have implemented a function that receives a Pareto front as input and records all metrics to Weights and Biases (W&B). This approach is employed across all multi-policy algorithms to ensure standardized metric logging. Additionally, we provide a method to initialize all components with a seed, including making PyTorch deterministic. The documentation related to these can be found at https://lucasalegre.github.io/morl-baselines/features/evaluations/.
Performance metrics. As outlined in the main paper, there are multiple metrics that can be used to evaluate a given Pareto front in order to compare different algorithms. We offer a collection of
metrics implementations for this purpose, including Hypervolume, Inverted Generational Distance, Maximum Utility Loss, Sparsity, and Expected Utility. These implementations accept a Pareto front as input and produce a scalar value representing the corresponding metric value. More details can be found at https://lucasalegre.github.io/morl-baselines/features/performance_indicators/.
Scalarization functions. Although most of the MORL algorithms rely on weighted sum scalarization, there are still cases where the user utility can be captured by non-linear schemes. To solve such an issue, we provide other scalarization schemes such as the Chebyshev scalarization function as used in Van Moffaert et al. (2013b). More details can be found at https://lucasalegre.github.io/morl-baselines/features/scalarization/.
Reward weights utils. There are many ways to generate weight vectors to scalarize vectorized rewards. These can, for instance, be randomly sampled or generated as equidistant points from the weight simplex, e.g., via the Riesz s-Energy method (Blank et al., 2021). These helpers can be found at https://lucasalegre.github.io/morl-baselines/features/weights/.
Experience replay buffers. In RL algorithms, experience replay buffers play a crucial role as they store experienced tuples collected via interacting with the environment. These are usually used for updating the agent's policy and/or value functions via sampling. There are various approaches to designing these buffers. For instance, Abels et al. (2019) propose an experience replay buffer that maintains diverse samples to enable the learning of diverse policies. On the other hand, Alegre et al. (2023) rely on a prioritized experience replay buffer. We make these implementations available so that researchers can incorporate them into novel algorithms. More details can be found at https://lucasalegre.github.io/morl-baselines/features/buffers.
Neural networks. Neural networks are the main component used in modern deep MORL algorithms. There are many possible architectures and enhancements related to the usage of such function approximators, e.g. Convolutional Neural Networks are usually used on environments with image-based observations, and Polyak update is used when the algorithm relies on target networks. We provide a few helpers related to neural networks in MORL-Baselines; see https://lucasalegre.github.io/morl-baselines/features/networks/.
Miscellaneous utils. Finally, there are a couple of other methods that have been reused in MORL-Baselines. For example, we provide utilities to capture GIFs from the policy executions, or a linearly decaying value for epsilon-greedy exploration, as in Mnih et al. (2015a). All these utilities are listed at https://lucasalegre.github.io/morl-baselines/features/misc.
C.3 User interface
This section gives some insight into the user interface provided by Weights and Biases to visualize and manipulate the training results.
Figure 8 shows an example of the real-time metrics obtained during the training. In this example we have a view of the multi-objective metrics described in the main paper, as well as a live view of the Pareto front found by various algorithms runs. Figure 9 illustrates the overview of a run. It provides information on the exact version of the library that was used (via git commit identifier), information regarding the duration of the run, and when it was started. Moreover, information regarding the computing node, OS, and Python version is also available. Hyperparameters are also available for each run, as shown in Figure 10. This, along with the git commit and hardware specifications, eases reproducibility. Finally, W&B also automatically logs system metrics along the training process, as shown in Figure 11.
D Benchmarking dataset
This section gives more details on our third contribution. It consists of a dataset of training results of algorithms in MORL-Baselines applied on various environments available in MO-Gymnasium.
<sup>11</sup>We also support generating weight vectors based on OLS and GPI-LS methods.
Figure 8: Screenshot of the W&B dashboards providing metrics and training information in real-time to the user.
Figure 9: Overview of a given run in the W&B user interface. Notice that command line, OS and Python version, hardware specifications, and git commits are available for reproducibility.
View Raw Data
Config parameters describe your model's inputs. Learn more
Figure 10: Hyperparameters of a given run.
Figure 11: System usage metrics reported through W&B.
D.1 Experimental settings
The section provides a more detailed description of the hardware and software settings used to produce our experimental results. For further details, the full specifications of each run can be found in W&B dashboards, as shown in Figure 9 and 10.
Hardware specifications
The experiments conducted on the University of Luxembourg high-performance computer (Varrette et al., 2014) were carried out on the iris cluster. The compute nodes equipped with GPU contain NVidia Tesla v100 SXM2 (16 and 32 GB) and Intel Xeon Gold 6132 @ 2.6GHz CPUs. Each training job has been allocated one GPU and 7 cores using SLURM (Yoo et al., 2003). The experiments conducted on the Vlaams Supercomputer Centrum (VSC) cluster Hydra hosted by the Vrije Universiteit Brussel (VUB) used nodes equipped with NVIDIA A100-PCIE-40GB GPUs.
Training all algorithms across all environments with multiple random seeds required approximately three months of computational resources.
Hyperparameters and code
All experiments have been run on MO-Gymnasium v1.0.0 and MORL-Baselines v1.0.0. All algorithms have been run with seeds from 0 to 9 on each supported environment. For most runs, we kept the default hyperparameters which are based on the original papers. However, in certain instances where we could not achieve satisfactory results, we made a minor attempt to fine-tune the hyperparameters. As a result, these hyperparameters, such as the Envelope hyperparameters, differ from those specified in the original paper. Finally, for the deterministic environments, we chose to increase the learning rate and exploration as these are the most critical in such domains. A GitHub issue (https://github.com/LucasAlegre/morl-baselines/issues/43) gives an overview of the command line used to launch each run, along with the values of the hyperparameters.
D.2 Results
The following section presents the experimental results available in our dataset utilizing the command-line interface of openrlbenchmark (Huang et al., 2023). The provided command lines explicitly demonstrate how effortless it can be to generate plots to visualize the available evaluation metrics using this tool. It is important to note that openrlbenchmark is still in the developmental phase, and therefore its API is likely to undergo changes. Furthermore, we remind the reader that these results should be interpreted as proof of concept and should not be considered empirical evidence of performance differences among the implemented algorithms. In other words, we have not conducted enough runs to make statistically valid claims and have not fine-tuned the hyperparameters for all instances. Lastly, we would like to highlight that even though we have implemented single-policy algorithms, we have not extensively tested them at this stage.
Continuous observation and continuous actions
Figure 12a and 12b show the training results for environments with continuous actions and two objectives $^{12}$ . Both graphs show that PGMORL (Xu et al., 2020) seems less efficient than GPI-based algorithms (Alegre et al., 2023) or CAPQL (Lu et al., 2023) in terms of EU and Hypervolume. While CAPQL seems less sample efficient than GPI-based algorithms, the time-based plots (Figure 12b) look less categorical, especially for mo-halfcheetah. The sparsity metric is improved for CAPQL and PGMORL, as discussed in the main paper.
As a side note, we remind the reader that PGMORL is based on PPO (Schulman et al., 2017b) while the other algorithms are based on more modern single-objective RL algorithms. We believe that this plays a huge role in the final performances of the algorithms, e.g. if we were to reimplement PGMORL based on SAC (Haarnoja et al., 2018) or TD3 (Fujimoto et al., 2018b) we would probably obtain better performance. To make it easier to understand, we chose to keep