# Multi-Objective Categorical Deep Q-Networks

Farès Chouaki, Aurélie Beynier, Nicolas Maudet, Paolo Viappiani

► To cite this version:

Farès Chouaki, Aurélie Beynier, Nicolas Maudet, Paolo Viappiani. Multi-Objective Categorical Deep Q-Networks. 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), May 2026, Paphos, Cyprus. hal-05517093

HAL Id: hal-05517093

https://hal.science/hal-05517093v1

Submitted on 18 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Multi-Objective Categorical Deep Q-Networks

Farès Chouaki

LIP6, Sorbonne Université
Paris, France
Fares.chouaki@lip6.fr

Nicolas Maudet

LIP6, Sorbonne Université
Paris, France
nicolas.maudet@lip6.fr

Aurélie Beynier

LIP6, Sorbonne Université
Paris, France
aurelie.beynier@lip6.fr

Paolo Viappiani

LAMSADE, PSL
Paris, France
paolo.viappiani@lamsade.dauphine.fr

## ABSTRACT

Motivated by recent advances in distributional reinforcement learning on the one hand and Multi-Objective Reinforcement Learning (MORL) on the other, we propose MO-CDQN, a value based algorithms that, given a possibly non-linear scalarization function, learn the policy with maximal expected scalarized return. Leveraging the Kantorovich-Rubenstein duality we prove the theoretical validity of our method for Lipschitz-continuous scalarization function. We establish that the state-action return distributions learned by our algorithm converge to a fixed point whose expected scalarized return is optimal. Our approach is then extended to propose a first value-based multi-policy algorithm for solving MORL problems under the expected scalarized return criterion. The proposed algorithms are tested on several environments from the MO-gymnasium benchmark. The results are promising and show that, on the one hand, our algorithm learns policies better than those obtained by existing approaches in the literature while requiring fewer interactions with the environments. On the other hand, given a set of scalarization functions, our multi-policy takes advantage of its off-policy nature to successfully optimize several policies concurrently and efficiently provide a set of policies each one optimal for a given scalarization function.

## ACM Reference Format:

Farès Chouaki, Aurélie Beynier, Nicolas Maudet, and Paolo Viappiani. 2026. Multi-Objective Categorical Deep Q-Networks. In Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), Paphos, Cyprus, May 25 – 29, 2026, IFAAMAS, 18 pages.

## 1 INTRODUCTION

Many real world sequential decision making problems such as electric plant control and autonomous driving involve compromising between several possibly conflicting objectives. To solve such problems, algorithms from the Multi-Objective Reinforcement Learning (MORL) sphere can be used. On the one hand, when no information on the preferences over the objectives is available, multi-policy algorithms can be used to learn a set of policies that ensure non-dominated returns. On the other hand, when such preferences are available in the form of a scalarization function, single-policy algorithms can learn a policy maximizing the scalarized returns.

A majority of single-policy algorithms assume the scalarization function to be a weighted sum. Such a simplification allows for using state-of-the-art algorithms from mono-objective reinforcement learning to solve MORL problems. However it is often too restrictive and unrealistic (especially it requires to set the appropriate weights which is quite difficult in practice and limits expressivity of the scalarization). Some single-policy algorithms address this expressivity issue through the use of more complex non-linear scalarization functions. These algorithms can be categorized according to the criterion they optimize: some use the Scalarized Expected Return (SER) criterion [18] aiming to identify a policy that maximizes a score defined as the aggregation of the obtained average return vectors, while others [8, 16, 17] choose to optimize the Expected Scalarized Return (ESR) and learn the policy that maximizes the average of the aggregated returns.

Unlike SER, ESR ensures more balanced performances among the objectives over a single execution. ESR achieves this by penalizing policies that prioritize different objectives inconsistently across executions. Moreover, the distinction between ESR and SER becomes crucial when dealing with non-linear preferences such as those commonly exhibited by humans. Under such preferences, ESR is particularly well-suited for scenarios where a policy is executed only once due to the nature of the application. For example, rescue rovers or exploration robots typically operate over a single mission in a given environment and must perform effectively during that one deployment. In other cases, the cost of implementing a policy may be prohibitively high, making repeated execution impractical. Government budget allocation problems highlight a use case where repeated policy execution is not possible as funding is non-recoverable once a policy is implemented. It is also the desirable way to solve scenarios where the implementation of a policy is irreversible. Medical treatment problems illustrate this use-case. Another more recent example of such scenarios is the choice of response to the current global environmental crisis, where greenhouse gas emissions have to be reduced while still ensuring that the global economy stays stable. It is clear that no matter what global policy is chosen it will have lasting impacts on humanity. Under such circumstances, a single execution of the policy determines its value. Given the critical nature of these tasks, developing efficient algorithms to determine the optimal ESR policy is crucial.
**Contributions** This paper addresses two issues of the MORL literature. First, we address the lack of a value based MORL algorithm for ESR-optimization. Building on categorical deep-Q-networks algorithm, we propose Multi-Objective Categorical Deep-Q-Networks (MO-CDQN), a value-based algorithm that finds the optimal policy for a given scalarization function. We prove that the return distribution learned by our algorithm for each state-action pair has the same expected scalarized return as the true distribution of the returns. The lack of ESR optimizing multi-policy algorithms is then addressed. We extend MO-CDQN by leveraging off-policy experience and we provide Distributed-MO-CDQN, a first solution to multi-policy learning under ESR. Comparison of our single policy algorithm against existing state-of-the-art algorithms proves that ours learns better policies more efficiently. Moreover, our multi-policy algorithm is even more efficient when provided with a portfolio of scalarization functions. Through the sharing of experience across learners our algorithm is able to parallelize the policy optimization process and finds the optimal policies for each scalarization function in the given portfolio.

# 2 BACKGROUND

This section provides background related to our approach.

## 2.1 Multi-Objective Markov Decision Processes

Multi-objective sequential decision-making problems can be modeled using Multi-Objective Markov Decision Processes (MO-MDPs) [12], which generalize the classical Markov Decision Process framework. An MO-MDP is defined as a tuple <d, S, A, T, μ, R, γ>, where d denotes the number of objectives in the decision problem, S is the state space, and A is the action space. The transition model T: S × A → P(S) maps each state-action pair (s, a) to a probability distribution over the next states, specifying the likelihood of transitioning to each state s' ∈ S. The initial state distribution given by μ: P(S) represents the probability distribution over the starting states. The reward function R: S × A → ℝ^d assigns a d-dimensional real-valued reward vector to each state-action pair, capturing the multi-objective nature of the problem. Finally, γ ∈ [0, 1)^d is a vector of discount factors, one for each objective.

## 2.2 Optimization criteria in Multi-Objective Reinforcement learning

Multi-objective RL algorithms aim to either approximate the Pareto front or optimize a given aggregation of objectives. When an aggregation function is provided, single-policy methods seek the policy that maximizes the aggregated return. Under non-linear scalarization functions, two distinct optimization criteria have to be distinguished:

* **Scalarized Expected Return (SER):** The value of a policy is obtained by applying the scalarization function over the expected return vector it obtains. Thus, given a scalarization function $u: ℝ^d → ℝ$, the value $V^π$ of a policy $π$ is given by $V_u^π = u(\mathbb{E}[\sum_{t=0}^∞ γ^t r_t | π, s_0 \sim μ])$.

* **Expected Scalarized Return (ESR):** The value of a policy is the expected value of the scalarized returns it obtains. Therefore, given a scalarization function $u: ℝ^d → ℝ$, the value $V^π$ of a policy $π$ is given by $V_u^π = \mathbb{E}[u(\sum_{t=0}^∞ γ^t r_t) | π, s_0 \sim μ]$.

In practice, SER-maximizing policies are suited for settings where performance is averaged over many executions, while ESR-maximizing policies are preferred when each individual execution matters or must meet specific criteria.

## 2.3 Categorical reinforcement learning

This section discusses issues related to the representation of the value function, the action selection, and the update procedure in categorical RL for multi-objective problems.

**State-action value function representation** Given a policy $π$, categorical RL aims to learn, for each state-action pair (s, a), the distribution of returns $Z^π(s, a)$ obtained by taking action a in state s and then following $π$. To approximate this distribution, a categorical representation is used. It is parameterized by $N, V_{MIN}$, and $V_{MAX}$. The support of this distribution consists of $N$ evenly spaced points $S = \{z_i = V_{MIN} + i\Delta z | i = 0, ..., N-1\}$ where $\Delta z = \frac{V_{MAX}-V_{MIN}}{N-1}$, and each point $z_i$ is assigned a probability mass $p_i$.

**Action selection** In each state s each action a has an associated categorical return distribution. The Q-value of each action is computed as the expected value of each return distribution i.e. $Q(s, a) = \mathbb{E}[Z(s, a)] = \sum_{z_i \in S} z_i p_i$. The action with the maximum Q-value is taken.

**State-action value function update** Bellemare et al. [3] extended the Bellman equation to the distributional setting as:

$$Z(s, a) \stackrel{D}{=} R(s, a) + \gamma Z(S', A') \quad (1)$$

This expresses the return distribution for (s, a) as the sum of the reward distribution $R(s, a)$ and the discounted return distribution at the next state-action pair $Z(S', A')$. When all three of these distributions are represented with a categorical representation with the same support, scaling $Z(S', A')$ by $γ$ shifts its support to $\text{shrinked}_S = \{\gamma z_i | z_i \in S\}$. It must then be projected back onto the original support $S$ to be combined with $R(s, a)$. Bellemare et al. [3] introduced a projection operator for this purpose, enabling learning with fixed-support categorical distributions.

# 3 LITERATURE REVIEW

This section introduces the existing research and algorithms on topics related to the ones addressed in the paper.

## 3.1 Distributional reinforcement learning

Distributional reinforcement learning can be done using either categorical or quantile-based representations. C51-DQN [3], the first distributional RL algorithm, extends DQN by modeling returns as a categorical distribution and updating them using a projected Bellman operator. However, it requires prior knowledge of the return range, limiting its flexibility. QR-DQN [7] addresses this by representing returns using fixed quantiles, which improves accuracy and performance. Still, fixing the number of quantiles is suboptimal, as shown by IQN [6] which learns the full quantile function dynamically. Recently, [29] proposed MD3QN, allowing to learn the joint return distribution in multi-reward settings. However, their algorithm focuses on settings where a primitive reward function can be decomposed into a sum of several components.
## 3.2 Multi-objective reinforcement learning

Two main types of MORL algorithms can be distinguished:

**Single-policy algorithms** assume a known preference over objectives via a scalarization function and learn a single policy that maximizes the chosen optimization criterion.

**Multi-policy algorithms** assume no prior knowledge of preferences and instead learn a diverse set of potentially optimal policies, allowing later adaptation to user preferences.

When no scalarization function is provided, *inner-loop algorithms* learn multiple policies simultaneously, while *outer-loop algorithms* assume a parametrized model of the scalarization function and learn optimal policies for different parameter settings. These settings are varied systematically to generate a set of optimal policies.

Most MORL research has focused on maximizing SER [1, 2, 15, 20, 23, 28]. Recently, more attention has been given to optimizing ESR, introducing novel single-policy and multi-policy algorithms:

**Single-policy algorithms for optimizing ESR:** The first ESR-oriented algorithm for non-linear scalarizations, *Expected Utility Policy Gradient (EUPG)* [17], extends policy gradient methods by conditioning policies on both observations and accumulated returns, and incorporating the accumulated returns into action value estimation. To improve sample efficiency and stability, [16] proposed *Multi-Objective Categorical Actor-Critic (MOCAC)*, which uses a distributional critic to model return distributions, enabling mid-episode bootstrapping and more frequent updates. [10] introduced *Non-Linear Utility MCTS (NLU-MCTS)* and *Distributional MCTS*, which extend Monte Carlo Tree Search to optimize policies under ESR. To promote fairness across objectives, [8] proposed *NSW-Q-learning*, a Q-learning variant optimizing under the Nash Social Welfare scalarization.

**Multi-policy algorithms for optimizing ESR:** [11] extended value iteration to learn a set of ESR-optimal policies, while [19] built on [23] to find undominated policies under a new dominance criterion. Both are inner-loop methods restricted to discrete state and action spaces. To address this, [4] introduced a two-step outer-loop approach: it first samples *N* diverse non-linear scalarization functions, then solves the corresponding *N* scalarized MDPs using policy-gradient methods to obtain *N* ESR-optimal policies.

## 3.3 Research Gap

Despite the bulk of research in multi-objective RL focused on SER, interest for optimizing ESR is growing. However, existing solutions still suffer from several limitations highlighted below:

(1) Policy-based algorithms like EUPG and MOCAC greedily exploit experiences, cannot learn deterministic policies or generalize to other utility functions and are unable to leverage novel research in value based reinforcement learning.

(2) Single-policy value-based algorithms like NSW-QL are only useful for tasks with small state and action spaces.

(3) Single-objective algorithms like [3, 14, 21] cannot treat multi-objective problems and require adaptation.

(4) Inner-loop multi-policy algorithms optimizing ESR can only be used on discrete state and action spaces.

(5) Outer-loop multi-policy algorithms optimizing ESR like [4] require retraining the agent several times with different

scalarization functions to achieve a decent approximation of all the possibly optimal policies.

In this paper, we first address limitations 1–3 by introducing MO-CDQN, the first value-based multi-objective distributional algorithm extending C51-DQN to optimize policies under ESR given a scalarization function. To tackle limitations 4 and 5, we then propose a multi-policy variant that leverages MO-CDQN’s off-policy nature to simultaneously optimize multiple non-linear scalarization functions in large state spaces, reusing experience across functions.

# 4 OPTIMIZING A SINGLE SCALARIZATION FUNCTION UNDER ESR

We propose the Multi-Objective Categorical Deep-Q-Networks (MO-CDQN) algorithm which extends the C51-DQN algorithm to the multi-objective case. We adapt both the action selection and update rules of the C51-DQN algorithm while also modifying the network architecture to predict multivariate categorical distributions. In this section, we first describe the representation of return distributions in our proposed method. We then outline the action-selection rule employed by our algorithm and finally detail how the policy network is updated to predict accurate estimates of the return distributions.

## 4.1 Multi-Objective Categorical Deep-Q-Networks

We first present the inner-workings of the proposed solution and show how it derives the ESR-maximizing policy for a chosen scalarization function.

**Return distribution representation** Our MO-CDQN algorithm learns to predict the entire multivariate return distribution for each state-action pair in the environment. For each objective, we propose to represent the return distribution as a categorical representation. In the multi-objective context, since the return distribution is a multivariate distribution, our algorithm uses a multivariate categorical representation to store the return distribution. The support of this distribution is the set obtained from the cartesian product of the supports of the categorical representations of the different objectives $S = \times_{j \in \{0..d\}} S_j$ where $S_j = \{V_{MIN_j} + \Delta z_j \cdot i | 0 \le i \le N_j - 1\}$. $V_{MIN_j}$ is the minimal value of the return the agent can achieve on the *j*-th objective and $\Delta z_j = \frac{V_{MAX_j} - V_{MIN_j}}{N_j - 1}$ is the size of the intervals of the categorical representation of the *j*-th objective. Note that each item of *S* is a tuple of size *d*, each tuple in *S* is called *atom* and we write $S = \{f_i | 0 \le i \le \prod_{j \in \{0..d\}} N_j\}$. In order to estimate these distributions our algorithm exploits two neural networks: a *policy network* conditioned by a set of parameters $\theta$ and a *target network* conditioned by the parameters $\theta'$. The policy network is used at decision time to estimate the return distributions $Z_{\theta}(s, a)$ obtained by performing action *a* from state *s*. The target network is only used when updating the policy. It allows for stabilizing learning and also enables to estimate the target return distribution $\hat{Z}(s, a)$.

**Action selection** [17] showed that, to learn optimal policies for ESR, the agent’s policy needs to be conditioned by the return it has accumulated until the time of decision. To choose which action to perform, our algorithm takes into account the current state $s_t$ and
the accumulated return $\mathbf{r}_{acc}$ of the agent to estimate for each action $a$ its utility $Q(s_t, a)$. Algorithm 2 in Appendix A<fnref n="1" /> details how the action selection procedure is performed by MO-CDQN. For each action, the algorithm starts by predicting the multivariate future return $Z(s, a)$ (Line 1). The accumulated return of the agent $\mathbf{r}_{acc}$ is used to compute the new support $S'$ of the predicted distributions. The new support is computed by translating the initial support $S$ with the vector $\mathbf{r}_{acc}$ and is given by: $S' = \times_{j \in \{0..d\}} S'_j$ where $S'_j = \{\mathbf{r}_{acc_j} + V_{MIN_j} + \Delta z_j \cdot i | 0 \le i < N_j\}$ (Line 2). The atoms of the new support are then scalarized (Line 3) and multiplied by their probability, to compute the Q-values of the actions (Line 4). The action performed by the agent is finally selected by sampling from the softmax distribution obtained from the predicted Q-values (Lines 5-7).

**Update Rule** In order to train the neural network to accurately predict state-action values we use temporal difference learning [22]. At each update step, a batch of transitions is sampled from the replay buffer to compute the temporal difference loss. The temporal difference error is computed for each sample transition $(s, a, r, s')$. The target return distribution for the sample is first computed as $\hat{T}Z(s, a) = r + \gamma \cdot Z'_{\theta}(s', a^*)$ (Line 5). This is performed by shrinking the support of $Z'_{\theta}(s', a^*)$ by a factor of $\gamma$ and then translating said support by $r$ (cf. [3, 16]). The target distribution is then projected to the original support $S$ by distributing the probability mass of each atom of the support of $\hat{T}Z(s, a)$ to its immediate neighbors in $S$ giving us the projected target distribution $\Phi\hat{T}Z(s, a)$ (Line 6). The cross entropy between these two distributions $Z_{\theta}(s, a)$ and $\Phi\hat{T}Z(s, a)$ is computed and used to estimate the gradient of the parameter set $\theta$. Gradients are accumulated through all the sample transitions in the batch, the policy and the target network are updated with the aggregated gradient (Lines 12-13). Algorithm 3 in Appendix A presents the update procedure of the policy and target network.

**Validity of the algorithm** This section discusses the theoretical validity of our MO-CDQN algorithm. Consider a policy $\pi$ and a return function $Z$ are given, with $Z$ assigning a distribution over returns to each state-action pair. [3] introduced the distributional Bellman operator $\mathcal{T}^{\pi}$, defined as:

$$ \mathcal{T}^{\pi} Z(s, a) = r(s, a) + \gamma Z(s', a') \quad (2) $$

where $s' \sim P(\cdot | s, a)$, $a' \sim \pi(s')$.

It has been shown by [3, 29] that, for any initial return function, the iterative application of $\mathcal{T}^{\pi}$ converges to a unique fixed point $Z^{\pi}$, which corresponds to the true return distribution induced by policy $\pi$. Formally,

$$ \lim_{n \to \infty} (\mathcal{T}^{\pi})^n Z = Z^{\pi}, \quad (3) $$

with

$$ \begin{gathered} Z^{\pi}(s, a) \stackrel{d}{=} \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \qquad s_0 = s, a_0 = a \\ s_{t+1} \sim P(\cdot | s_t, a_t) \quad a_{t+1} \sim \pi(s_{t+1}) \qquad \forall t \ge 0, \end{gathered} $$

where $d$ denotes equality in distribution. Furthermore, [29] showed that the distributional Bellman optimality operator admits a

fixed point corresponding to an optimal return distribution. This fixed point has the property that its expected return—when scalarized using a linear scalarization function—coincides with the optimal value function obtained through classical (scalar) Bellman optimality. While prior work (e.g., [29]) established this property, our work extends this result to a broader class of scalarization functions. We first define two operators necessary to the analysis:

**Definition 4.1 (Distributional Bellman Optimality Operator).** Let $Z$ be a return function, a function assigning a probability distribution over $\mathbb{R}^d$ to each state-action pair $(s, a)$. Define the distributional Bellman optimality operator $\mathcal{T}$ applied in Algorithm 3 as follows:

$$ \mathcal{T}Z(s, a) = \mathbf{r} + \gamma Z(s', a^*), $$

where $a^* = \arg\max_{a'} \mathbb{E}_{z \sim Z(s', a')} [f(\mathbf{r} + \gamma \mathbf{z})]$, $s' \sim P(\cdot | s, a)$, and $\mathbf{r} \sim R(\cdot | s, a)$.

**Definition 4.2 (Expected scalarized Operator).** Given a distribution $Z(s, a)$ over vectors in $\mathbb{R}^N$, the expected scalarized operator $E_u$ is defined by:

$$ E_u Z(s, a) = \mathbb{E}_{z \sim Z(s, a)} [u(\mathbf{z})] $$

where $u: \mathbb{R}^N \to \mathbb{R}$ is a possibly nonlinear scalarization function that satisfies the Lipschitz continuity and has a Lipschitz constant of $K$.

Theorem 4.3 states that the composition of these two operators is a contraction on MO-MDPs with a discount factor of $\gamma$ and $K$ Lipschitz continuous scalarization when $\gamma K < 1$.

**THEOREM 4.3.** Let $Z_1$ and $Z_2$ be two return functions. Assume the scalarization function $u: \mathbb{R}^N \to \mathbb{R}$ is $K$-Lipschitz with respect to the $\ell_1$ norm. Then, the composed operator $E_u \mathcal{T}$ satisfies:

$$ |E_u \mathcal{T} Z_1 - E_u \mathcal{T} Z_2| \le \gamma K |E_u Z_1 - E_u Z_2| $$

In particular, if $\gamma K < 1$, the operator is a contraction.

Using this result, we derive Corollary 4.4 ensuring that the policy learned by our algorithm achieves the same expected scalarized return as an optimal policy.

**COROLLARY 4.4 (OPTIMALITY OF THE FIXED POINT).** Let $Z^*$ be the unique fixed point of the composed operator $E_u \mathcal{T}$ and suppose the scalarization function $u$ is $K$ Lipschitz, such that $\gamma K < 1$. Then, the expected scalarized return of $Z^*$ corresponds to the optimal scalarized value function:

$$ \forall (s, a), E_u Z^*(s, a) = \max_{\pi} \mathbb{E}_{\pi} \left[ u\left( \sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t \right) \middle| s_0 = s, a_0 = a \right] $$

The proofs of Theorem 4.3 and Corollary 4.4 are provided in Appendix D.

# 5 OPTIMIZING SEVERAL SCALARIZATION FUNCTIONS WITH OFF-POLICY EXPERIENCE UNDER ESR

In the previous section we assumed that the scalarization function was known and fixed throughout the learning process. However, in many applications, it is desirable or even necessary to optimize multiple scalarization functions simultaneously to capture different

<footnote n="1">¹A version of the paper containing the appendices can be found on</footnote>
trade-offs or user preferences. This is often the case in decision-support scenarios [12], where designers can optimize various utility functions to better understand user preferences. It also applies when users aim to enforce a desired behavior through a family of parameterized scalarization functions but are unsure of the best parameter values. For example, a user seeking fairness across objectives might use the Generalized Gini Index (GGI), yet still struggle to select parameters that balance fairness and efficiency. More practically algorithms like [4] rely on this strategy to learn a diverse set of policies. In fact, the algorithm proposed by [4] is a two-step algorithm where the first step consists in generating a set of diverse scalarization functions that are later used to provide a diverse set of solutions.

This section extends the MO-CDQN algorithm to learn optimal policies for multiple scalarization functions. The Distributed MO-CDQN we propose (Algorithm 1) addresses the gap in multi-policy optimization under Expected Scalarized Return (ESR) by reusing experience across several learning instances, each targeting a different scalarization function.

The intuition behind Distributed MO-CDQN resides in the off-policy nature of our algorithm. Suppose we are given two scalarization functions $u, u' : \mathbb{R}^d \rightarrow \mathbb{R}$ and let $\pi_u^{t_i}$ and $\pi_{u'}^{t_i}$ be the policies of two agents optimizing $u$ and $u'$ respectively. It is possible to use experience collected under $\pi_u$ to update $\pi_u^{t_i}$ and vice versa. With this in mind, we assume that we are given a finite set of possible scalarization functions that can be considered online by the decision-making agent. Our approach aims at learning policies that can fit to different scalarization functions without requiring re-training. We thus extend our single-policy algorithm MO-CDQN to account for a set of $M$ scalarization functions $U = \{u_1, ..., u_M\}$ by introducing $M$ artificial agents (referred to as learners) where each agent $l_i$ is assigned the scalarization function $u_i$ and searches for the policy $\pi_{u_i}^* = \underset{u \in \mathcal{U}}{\arg\max_{\pi \in \Pi}} \mathbb{E}[u_i(\sum_{t=0}^{\infty} \gamma^t r^t) | \pi, s_0 \sim \mu]$. However, these learners are not trained separately. Instead, we propose to share environment experiences between the learners. More precisely, we explore two distinct action selection strategies to balance exploration and exploitation across the set of $M$ learners. In the first strategy, referred to as *per-step action sampling*, each learner $l_i$ proposes an action $a_i^t \sim \pi_{u_i}(s_t)$ based on its policy at every environment timestep $t^2$. A single action is then randomly sampled from the set of proposed actions $a_1^t, ..., a_M^t$ and executed in the environment, yielding the experience tuple $(s_t, a_t, r_t, s_{t+1})$, which is shared across all learners for policy updates. In the second strategy, termed *per-episode learner selection*, a single learner is randomly selected at the beginning of each episode, and its policy $\pi_{u_i}$ is followed exclusively for the entire episode. This means all actions within that episode are sampled from the chosen learner's policy, generating a sequence of experience tuples that are again shared across all learners. The policy of each learner is then updated using the rule described in Algorithm 3.

**Validity of the algorithm** We briefly discuss the theoretical validity of the multi-policy algorithm. Specifically, we address the following question: how can the policy of learner $l_i$ in charge of

**Algorithm 1** Distributed MO-CDQN

**Require:** Number of objectives $d$, Set of scalarization functions to optimize $U = \{u_1, ..., u_M\}$, Training budget $B$,

1. **for** $i = 1$ to $M$ **do**

2. Create learner $l_i$ with replay buffer $D_i$, value network $Z_{\theta}^i$ and target $Z_{\theta'}^i$.

3. **end for**

4. **for** $t = 1$ to $B$ **do**

5. $acc_r \leftarrow \vec{0}$; $c \leftarrow 0$

6. Initialize environment and receive initial state $s_0$

7. $s_t \leftarrow s_0$

8. **while** $\text{done}$ **do**

9. proposed\_actions $\leftarrow \{\}$

10. **for** $i = 1$ to $M$ **do**

11. $a_i \leftarrow \text{Dist\_Action\_Selection}(s_t, \text{acc\_r}, Z_{\theta}, u_i)$

12. Add $a_i$ to proposed\_actions

13. **end for**

14. $a_t \leftarrow \text{RandomChoice}(\text{proposed\_actions})$

15. Perform action $a_t$ in the environment and observe reward $r_t$, next state $s_{t+1}$, and done flag

16. **for** $i = 1$ to $M$ **do**

17. Add $(s_t, \text{acc\_r}, a_t, r_t, s_{t+1}, c)$ to $D_i$

18. Update $Z_{\theta}^i$ using distributional Bellman update described in Algorithm 3

19. **end for**

20. $acc_r \leftarrow acc_r + \gamma^c r_t; s_t \leftarrow s_{t+1}; c \leftarrow c + 1$

21. **end while**

22. **end for**

optimizing the scalarization function $u_i$ converge to its corresponding optimal policy while exploiting experience collected from other learners?

The answer to this question resides in the off-policy nature of the algorithm, meaning an agent's policy can be updated from experience that was collected from any other policy. A popular example of this is the use of $\epsilon$-greedy policies to improve exploration in Q-learning [27]. With this in mind, when updating the policy of the $i$-th learner using experience tuple $(s_t, acc_r, a_t, r, s'_t, t)$ we can distinguish two cases. If the algorithm picked the action proposed by $l_i$ (or by another learner with the same greedy action) then, $a_t = \pi_{u_i}$ is a greedy action from $l_i$'s perspective. When the algorithm picks an action different from the one proposed by $l_i$ i.e. $a_t \neq \pi_{u_i}$ then the action $a_t$ can be considered as an exploratory action from $l_i$'s point of view. Thus, the action selection procedure of the algorithm does not hinder its convergence properties but also improves each learners exploration. The algorithm also allows for concurrent policy updates, making it more interesting than applying a single policy algorithm sequentially to learn several policies one by one. Algorithm 1 uses this property to reduce the number of environment interactions needed to learn multiple policies.

## 6 EXPERIMENTS

In this section we compare the performances of the proposed algorithms to state-of-the-art algorithms proposed in [16, 17] on several

<footnote n="2">²The action of learner $l_i$ is sampled from the distribution $\pi_{u_i}(a|s_t)$ as described in Algorithm 2.</footnote>
environments of the MO-Gymnasium suite of benchmarks<fnref n="3" />. This section briefly describes those environments and the baselines and then dives into analyzing the obtained results. The experimental protocol followed to train and evaluate the different algorithms is described in the following section, meanwhile, Appendix C gives an empirical argument for the validity of our algorithm.

## 6.1 Evaluation Scenarios and Baselines

We evaluate the algorithms on the Fruit-Tree-Navigation (FTN), Deep-Sea-Treasure (DST), Fishwood, Four-room and Minecart environments from the MO-Gymnasium benchmark suite [9], commonly used to assess ESR optimization in multi-objective RL. Full environment details are provided in Appendix C.

MO-CDQN and its multi-policy extension, distributed MO-CDQN, are compared to state-of-the-art multi-objective RL algorithms that learn optimal policies for ESR. Specifically, the EUPG [17] and MOCAC [16] algorithms are used as baselines. Additionally, we compare our algorithm to the Distributionally Pareto-Optimal Multi-Objective Reinforcement Learning approach proposed in [4] while using the state-of-the-art single objective RL algorithm PPO. For each scalarization function, this approach transforms the MO-MDP into a scalarized MDP with the same optimal policy, then applies PPO on this MDP. To demonstrate the limitations of the approach proposed by [8], their algorithm is tested on a small toy MO-MDP, with results presented in Appendix B.

## 6.2 Experimental protocol

### 6.2.1 Single-policy algorithms.
We describe the experimental protocol used to evaluate single-policy algorithms, including training budgets, scalarization functions, and key hyperparameters (learning rate, batch size, number of atoms per objective).<fnref n="4" /> All policy-based methods use a neural network with one hidden layer of 64 units to represent the policy (and value function for MOCAC), while MO-CDQN relies on a dueling architecture [26] with the same backbone. Algorithms are evaluated periodically by freezing the current policy, executing it multiple times in the environment, and reporting the expected scalarized return. To alleviate the impact of random initialization of the policies, experiments are repeated over 11 seeds for short tasks (Fruit-Tree-Navigation, Deep-Sea-Treasure) and 5 seeds for longer tasks (Fishwood, Minecart, Four-Room, MountainCar) and the median score is reported.

**Fruit-Tree-Navigation:** The objective is to maximize the product of three objectives. Our method is trained for 500k environment steps, while baselines use 1M steps. Each objective's value is normalized between [0, 1]. Policies are evaluated every 5k steps over 100 episodes. Policy-based methods use a learning rate of 5 · 10⁻⁵, MO-CDQN uses 10⁻<fnref n="4" />, and both MOCAC and MO-CDQN discretize objectives using 21 atoms in [0, 1].

**Deep-Sea-Treasure:** We follow the motivating scenario from [16], using the same piecewise scalarization function. Our method is trained for 100k steps, while baselines are trained for 200k steps, with evaluations every 1k steps. All algorithms use a learning rate

of 10⁻<fnref n="4" />, and MOCAC and MO-CDQN discretize objectives using 21 atoms with ranges $r_0 \in [0, 125]$ and $r_1 \in [-25, 0]$.

**Fishwood:** We adopt the setting of [17] with scalarization $u(r_1, r_2) = \min(r_1, [r_2/2])$. Policy-based methods are trained for 1M steps, while MO-CDQN uses 500k steps. Policies are evaluated every 5k steps over 100 episodes. EUPG and MOCAC use a learning rate of 10⁻<fnref n="3" />, while MO-CDQN uses 10⁻<fnref n="4" />. MOCAC and MO-CDQN use 11 atoms with objective ranges $r_1 \in [0, 20]$, $r_2 \in [0, 100]$.

**Four-Room:** Agents aim to maximize the product of three objectives. Policy-based methods are trained for 1M steps, MO-CDQN for 500k steps. All methods use a learning rate of 10⁻<fnref n="4" />, with MO-CAC and MO-CDQN discretizing objectives using 11 atoms in [0, 4]. Policies are evaluated every 5k steps over 100 episodes.

**Minecart:** We focus on the two ore-collection objectives, normalized to [0, 1.5], and use their product as scalarization. MO-CDQN is trained for 2.5M steps, while policy-based baselines are trained for 5M steps. Policies are evaluated every 10k steps over 100 episodes. EUPG uses a learning rate of 10⁻<fnref n="3" />, others use 10⁻<fnref n="4" />, and MOCAC and MO-CDQN use 21 atoms.

**MountainCar:** This continuous-state task uses three objectives: total time $t$, forward actions $f$, and backward actions $b$, with scalarization $u(t, f, b) = -(t^2 + f \cdot b)$. The evaluation protocol and hyperparameters match those of Minecart, except that MOCAC and MO-CDQN use 5 atoms per objective.

### 6.2.2 Multi-policy algorithm.
Our multi-policy algorithm is evaluated on three of the previously presented tasks namely the Fruit-Tree-Navigation task, the Four-Room task and the MineCart task. The FTN and Minecart tasks are limited to 2 objectives with 5 hand picked non-linear scalarization functions ($u_1$ to $u_5$). To showcase the scalability of the distributed with respect to the number of objectives we consider the Four-Room task with 5 other hand picked scalarization functions ($u_6$ to $u_{10}$). The considered scalarization functions for these tasks are the following. Note that $r^\uparrow$ is the vector obtained from reordering the components of the reward vector increasingly and $I$ is ideal point for each task and takes the values (9.50, 8.78) for the FTN task, (1.5, 1.5) for the MineCart task and (4, 4, 4) for the Four-Room task.

$$ 
\begin{align*}
u_1(r_0, r_1) &= \max(r_0, r_1) & u_6(r_0, r_1, r_2) &= r_0 r_1 r_2 \\
u_2(r_0, r_1) &= \min(r_0, r_1) & u_7(r_0, r_1, r_2) &= -\frac{(r_0 - I_0)^2 + (r_1 - I_1)^2}{(r_0 - I_0)^2 + (r_2 - I_2)^2} \\
u_3(r_0, r_1) &= r_0 r_1 & u_8(r_0, r_1, r_2) &= -\frac{(r_0 - I_0)^2 + (r_2 - I_2)^2}{(r_1 - I_1)^2 + (r_2 - I_2)^2} \\
u_4(r_0, r_1) &= w_0 r_0^\uparrow + w_1 r_1^\uparrow & u_9(r_0, r_1, r_2) &= -\frac{(r_1 - I_1)^2 + (r_2 - I_2)^2}{(r_1 - I_1)^2 + (r_2 - I_2)^2} \\
u_5(r_0, r_1) &= -\frac{(r_0 - I_0)^2 + 4(r_1 - I_1)^2}{5} & & \\
u_{10}(r_0, r_1, r_2) &= -\frac{(r_0 - I_0)^2 + (r_1 - I_1)^2 + (r_2 - I_2)^2}{3}
\end{align*}
$$

The performances of our multi-policy algorithm are compared to the ones obtained by single-policy algorithms namely: MO-CDQN, EUPG and MOCAC, and to the ones obtained by DPMORL [4], the only existing multi-policy algorithm. The single policy algorithms and the multi-policy baseline (DPMORL) are allowed a budget of $B = 100,000$ steps to optimize each one of $m = 5$ scalarization functions independently, meanwhile the proposed multi-policy algorithm is allowed a budget of $B' = B * m = 500,000$ to optimize all 5 scalarization functions in parallel. Each experiment is repeated

<footnote n="3">³The code used to perform the experiments can be found on the following GitLab repository https://gitlab.com/chouakifares/multi-objective-categorical-deep-q-networks</footnote>

<footnote n="4">⁴Additional hyperparameters and implementation details are provided Appendix C.</footnote>
5 times. We then compare the best scores obtained by the median policy found by each algorithm.

## 6.3 Experimental results

This section presents the experimental results that were obtained by our MO-CDQN algorithm and the considered baselines.

**Comparison to baselines** We now turn our focus into showing how our single-policy algorithm can yield better results than existing state-of-the-art algorithms optimizing ESR. In all the experiments, the baselines were allowed twice as many environment steps than the proposed algorithm to showcase the superiority of our algorithm even when the baselines are given a higher budget. Figures 1b,1a, 1c, 1d and 1e show the evolution of the median discounted scalarized return obtained by each algorithm. The shaded regions show the fourth and sixth decile of the returns obtained by the agents. The figures clearly illustrate the superiority of our algorithm compared to the baselines. For instance, from Figure 1a, it can be seen that our algorithm requires less timesteps to converge to a better policy than the one found by the baselines. Figure 1b shows that our algorithm learns policies with higher returns even on tasks with a higher number of objectives. Figure 1c outlines the superiority of our algorithm in heavily stochastic environments. From the figure, we can clearly see that our algorithm converges to a better policy after less than 100,000 training timesteps while MOCAC requires more than 200,000 timesteps to achieve similar returns. Figures 1d and 1e illustrate the superiority of our algorithm on tasks with large and continuous state spaces. Figure 1d clearly shows the superiority of our algorithm that learns a satisfying policy after only 150,000 steps whereas EUPG and MOCAC require many more environment interactions and are unable to solve this task even when their training budget is double the one given to MO-CDQN. Figures 1e shows the same tendency although the baselines perform better on this task our algorithm is still the only one able to converge to the optimal policy. Indeed, both baselines fail to optimally solve the task even after 5,000,000 steps. Finally, 1f shows the total domination of MO-CDQN over the baselines on continuous state-space task with a denser reward function where our agent learns to solve the task after just about 50,000 steps and then start focusing on finding the optimal policy that balances the use of forward and backward actions. The rest of the baselines are not even able to efficiently solve the task. Indeed, the policies learned by the baselines need more than 160 environment steps to achieve the top of the mountain compared to less than 110 environment step for the policy learned by MO-CDQN.

**Results of the multi-policy algorithm** Tables 1 and 2 show the best expected scalarized return obtained by the policy found using each algorithm on the FTN and FourRoom tasks respectively. From the tables, we can observe that for each scalarization function the proposed single-policy algorithm (MO-CDQN) matches or outperforms the baselines confirming the results of the previous section. It can be observed that the multi-policy variant of our algorithm is not only competitive with the rest of the baseline. It also manages to outperform them on 9 out of the 10 considered scalarization functions without requiring re-training. A very interesting observation is that the multi-policy variant of MO-CDQN obtains better results than the single-policy counterpart, this is due to the

higher exploration ratio and experience sharing across the learners. This increased diversity of experience can help the algorithm escape local optima faster where the single-policy algorithm might get trapped for longer, ultimately leading to better overall performance. Finally, we can also observe that both action-selection strategies result in similar returns even in the four-room environment. We argue that further research is needed to explore better action selection strategies or policy re-usability across scalarization functions under ESR.

## 7 DISCUSSION

In this section, we discuss the performance of our algorithm and the limitations that our approach still suffers from.

Our results demonstrate that our algorithm achieves superior sample efficiency compared to state-of-the-art policy-based methods such as EUPG and MOCAC. As shown in Figure 1b, our approach converges to better policies than the ones learned by MOCAC and EUPG while requiring less timesteps. Moreover, our algorithm is the only one allowing to learn both deterministic and stochastic policies contrarily to existing approaches that can only learn stochastic policies. Furthermore, MO-CDQN is able to leverage recent research in value based RL such as [13] and [24]. Moreover, our multi-policy algorithm illustrates how experience can be re-used to train policies optimizing different scalarization functions further improving the sample-efficiency of our approach. Both algorithms highlights the advantage of our method. However, the categorical return representation can become computationally expensive as the number of objectives increases since the size of the support of the representation increases exponentially with the number of objectives in the environment. Another limitation of our algorithm is its inability to effectively handle continuous action spaces. Since our approach relies on a discrete set of actions, applying it to continuous control problems would require discretization, which may lead to suboptimal performance due to the loss of fine-grained control. In contrast, policy-based methods like EUPG and MOCAC inherently support continuous actions through parametrized policies.

## 8 CONCLUSION

This paper tackles the multi-objective sequential decision making problems where the goal is to optimize non-linear scalarisation functions under ESR. A novel value-based algorithm based on learning a categorical representation of the multivariate return distribution is proposed. We provide a theoretical argument for the validity of our algorithm. Comparision results between our algorithm and other existing solutions on several benchmark MORL problems prove that the proposed solution can learn better policies faster. The algorithm is then extended to propose a first value-based multi-policy algorithm able to optimize several non-linear scalarization functions under ESR. The contributed algorithms still suffer from scaling issues that will be addressed in future works. Moreover, the multi-policy algorithm suffers from a limitation in that it assumes the scalarization functions are known and explicitly specified by the users. To alleviate this issue, we plan on extending our algorithm to a two-step approach like [4] where a first step consists in generating a set of diverse non-linear scalarization functions that are then fed to the multi-policy algorithm to optimize each
<figure><img src="image_2.png" /><figcaption>Figure 1: Comparison of the performances of the proposed algorithms to the baselines on the validation and evaluation tasks</figcaption></figure>

<table><caption>Table 1: Performance of each algorithm on each scalarization function for the FTN task (median ± standard deviation over 5 seeds).</caption><thead><tr><th rowspan="2"></th><th rowspan="2">MO-CDQN</th><th rowspan="2">MOCAC</th><th rowspan="2">EUPG</th><th colspan="2">Dist MO-CDQN</th><th rowspan="2">DPMORL</th></tr><tr><th>per-step</th><th>per-episode</th></tr></thead><tbody><tr><th>u<sub>1</sub></th><td>8.55 ± 0.18</td><td>8.05 ± 2.51</td><td>8.06 ± 2.29</td><td>8.52 ± 0.05</td><td>8.43 ± 0.14</td><td><strong>8.94 ± 0.00</strong></td></tr><tr><th>u<sub>2</sub></th><td>4.11 ± 0.92</td><td>2.04 ± 0.37</td><td>3.07 ± 0.42</td><td><strong>4.81 ± 0.22</strong></td><td>4.77 ± 0.16</td><td>2.99 ± 0.29</td></tr><tr><th>u<sub>3</sub></th><td>24.78 ± 5.06</td><td>10.64 ± 1.88</td><td>23.98 ± 6.46</td><td><strong>36.04 ± 1.37</strong></td><td><strong>36.04 ± 2.65</strong></td><td>24.80 ± 2.42</td></tr><tr><th>u<sub>4</sub></th><td>4.76 ± 0.19</td><td>4.47 ± 0.87</td><td>4.43 ± 1.00</td><td><strong>4.89 ± 0.17</strong></td><td>4.87 ± 0.12</td><td>4.78 ± 0.08</td></tr><tr><th>u<sub>5</sub></th><td>-6.84 ± 0.09</td><td>-23.04 ± 4.40</td><td>-22.23 ± 3.87</td><td>-4.51 ± 1.46</td><td>-4.39 ± <strong>1.91</strong></td><td>-16.88 ± 1.20</td></tr></tbody></table>

<table><caption>Table 2: Performance of each algorithm on each scalarization function for the Four-Room task (median ± standard deviation over 5 seeds).</caption><thead><tr><th rowspan="2"></th><th rowspan="2">MO-CDQN</th><th rowspan="2">MOCAC</th><th rowspan="2">EUPG</th><th colspan="2">Dist MO-CDQN</th><th rowspan="2">DPMORL</th></tr><tr><th>per-step</th><th>per-episode</th></tr></thead><tbody><tr><th>u<sub>6</sub></th><td>10.12 ± 4.88</td><td>0.57 ± 0.35</td><td>0.61 ± 0.26</td><td><strong>24.64 ± 8.46</strong></td><td>20.19 ± 10.52</td><td>1.85 ± 0.31</td></tr><tr><th>u<sub>7</sub></th><td>-1.69 ± 0.87</td><td>-5.67 ± 0.53</td><td>-7.37 ± 0.74</td><td><strong>-0.64 ± 2.58</strong></td><td>-0.82 ± 0.46</td><td>-3.22 ± 0.21</td></tr><tr><th>u<sub>8</sub></th><td>-1.03 ± 0.77</td><td>-5.01 ± 0.46</td><td>-7.02 ± 0.56</td><td>-0.99 ± 3.06</td><td><strong>-0.71 ± 1.53</strong></td><td>-2.89 ± 0.08</td></tr><tr><th>u<sub>9</sub></th><td>-1.96 ± 0.25</td><td>-5.27 ± 0.28</td><td>-7.00 ± 0.57</td><td>-0.87 ± 1.29</td><td><strong>-0.80 ± 1.78</strong></td><td>-3.27 ± 0.32</td></tr><tr><th>u<sub>10</sub></th><td>-1.24 ± 1.87</td><td>-3.63 ± 0.30</td><td>-4.72 ± 0.40</td><td><strong>-0.40 ± 0.68</strong></td><td>-0.56 ± 2.30</td><td>-2.46 ± 0.16</td></tr></tbody></table>

function and generate a diverse front of possibly optimal policies under ESR.

REFERENCES

[1] Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. 2022. Multi-Objective Reinforcement Learning with Non-Linear Scalarization. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems* (Virtual Event, New Zealand) (AAMAS '22). International Foundation for Autonomous
Agents and Multiagent Systems, Richland, SC, 9-17.

[2] Toygun Basaklar, Suat Gumussoy, and Umit Y. Ogras. 2023. PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm. arXiv:2208.07914 [cs.LG] https://arxiv.org/abs/2208.07914

[3] Marc G. Bellemare, Will Dabney, and Rémi Munos. 2017. A Distributional Perspective on Reinforcement Learning. In *Proceedings of the 34th International Conference on Machine Learning - Volume 70* (Sydney, NSW, Australia) (ICML'17). JMLR.org, 449-458.

[4] Xin-Qiang Cai, Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley J. Llorens. 2023. Distributional pareto-optimal multi-objective reinforcement learning. In *Proceedings of the 37th International Conference on Neural Information Processing Systems* (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 686, 21 pages.

[5] Krzysztof Ciesielski. 2007. On Stefan Banach and some of his results. *Banach Journal of Mathematical Analysis* [electronic only] 1, 1 (2007), 1–10. http://eudml.org/doc/53684

[6] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. 2018. Implicit Quantile Networks for Distributional Reinforcement Learning. In *Proceedings of the 35th International Conference on Machine Learning* (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 1096–1105. https://proceedings.mlr.press/v80/dabney18a.html

[7] Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. 2018. Distributional reinforcement learning with quantile regression. In *Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence* (New Orleans, Louisiana, USA) (AAAI'18/IAAI'18/EEAAI'18). AAAI Press, Article 353, 10 pages.

[8] Ziming Fan, Nianli Peng, Muhang Tian, and Brandon Fain. 2023. Welfare and Fairness in Multi-objective Reinforcement Learning. In *Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems* (London, United Kingdom) (AAMAS '23). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1991-1999.

[9] Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. 2023. A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning. In *Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023)*.

[10] Conor F. Hayes, Mathieu Reymond, Diederik M. Roijers, Enda Howley, and Patrick Mannion. 2023. Monte Carlo tree search algorithms for risk-aware and multi-objective reinforcement learning. *Autonomous Agents and Multi-Agent Systems* 37, 2 (April 2023), 37. https://doi.org/10.1007/s10458-022-09596-0

[11] Conor F. Hayes, Diederik M. Roijers, Enda Howley, and Patrick Mannion. 2022. Decision-Theoretic Planning for the Expected Scalarised Returns. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems* (Virtual Event, New Zealand) (AAMAS '22). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1621-1623.

[12] Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. 2022. A practical guide to multi-objective reinforcement learning and planning. *Autonomous Agents and Multi-Agent Systems* 36, 1 (April 2022). https://doi.org/10.1007/s10458-022-09552-y

[13] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. 2018. Rainbow: combining improvements in deep reinforcement learning. In *Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence* (New Orleans, Louisiana, USA) (AAAI'18/IAAI'18/EEAAI'18). AAAI Press, Article 393, 8 pages.

[14] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. *Nature* 518 (2015), 529-533. https://apisemanticscholar.org/CorpusID:205242740

[15] Mathieu Reymond, Eugenio Bargiacchi, and Ann Nowé. 2022. Pareto Conditioned Networks. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems* (Virtual Event, New Zealand) (AAMAS '22). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1110-1118.

[16] Mathieu Reymond, Conor Hayes, Denis Steckelmacher, Diederik Roijers, and Ann Nowé. 2023. Actor-critic multi-objective reinforcement learning for nonlinear utility functions. *Autonomous Agents and Multi-Agent Systems* 37 (04 2023). https://doi.org/10.1007/s10458-023-09604-x

[17] Diederik M. Roijers, Denis Steckelmacher, and Ann Nowé. 2020. Multi-objective reinforcement learning for the expected utility of the return. 2018 Adaptive Learning Agents, ALA 2018 - Co-located Workshop at the Federated AI Meeting, FAIM 2018; Conference date: 14-07-2018 Through 15-07-2018.

[18] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley. 2013. A Survey of Multi-Objective Sequential Decision-Making. *Journal of Artificial Intelligence Research* 48 (Oct. 2013), 67-113. https://doi.org/10.1613/jair.3987

[19] Willem Röpke, Conor F. Hayes, Patrick Mannion, Enda Howley, Ann Nowé, and Diederik M. Roijers. 2023. Distributional multi-objective decision making. In *Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence* (Macao, P.R.China) (IJCAI '23). Article 634, 9 pages. https://doi.org/10.24963/ijcai.2023/634

[20] Manuela Ruiz-Montiel, Lawrence Mandow, and José-Luis Pérez de-la Cruz. 2017. A temporal difference method for multi-objective reinforcement learning. *Neurocomputing* 263 (2017), 15-25. https://doi.org/10.1016/j.neucom.2016.10.100 Multiobjective Reinforcement Learning: Theory and Applications.

[21] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized Experience Replay. CoRR abs/1511.05952 (2015). https://apisemanticscholar.org/ CorpusID:13022595

[22] Gerald Tesauro. 1995. Temporal difference learning and TD-Gammon. *Commun. ACM* 38, 3 (March 1995), 58-68. https://doi.org/10.1145/203330.203343

[23] Kristof Van Moffaert and Ann Nowé. 2014. Multi-objective reinforcement learning using sets of pareto dominating policies. *J. Mach. Learn. Res.* 15, 1 (Jan. 2014), 3483-3512.

[24] Nino Vieillard, Olivier Pietquin, and Matthieu Geist. 2020. Munchausen reinforcement learning. In *Proceedings of the 34th International Conference on Neural Information Processing Systems* (Vancouver, BC, Canada) (NIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 356, 12 pages.

[25] Cédric Villani. 2008. *Optimal Transport: Old and New*. Gründlehren der mathematischen Wissenschaften, Vol. 338. Springer-Verlag, Berlin Heidelberg.

[26] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando De Freitas. 2016. Dueling network architectures for deep reinforcement learning. In *Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48* (New York, NY, USA) (ICML'16). JMLR.org, 1995-2003.

[27] Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. *Machine Learning* 8, 3 (1992), 279-292. https://doi.org/10.1007/BF00992698

[28] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. 2019. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Curran Associates Inc., Red Hook, NY, USA.

[29] Pushi Zhang, Xiaoyu Chen, Li Zhao, Wei Xiong, Tao Qin, and Tie-Yan Liu. 2021. Distributional reinforcement learning for multi-dimensional reward functions. In *Proceedings of the 35th International Conference on Neural Information Processing Systems* (NIPS '21). Curran Associates Inc., Red Hook, NY, USA, Article 117, 11 pages.
A ALGORITHMS

**Action selection** The procedure of Algorithm 2 describes how to select an action given a state and an accumulated return using a scalarized distributional representation of multi-objective returns.

**Algorithm 2 DistActionSelection**

**Require:** State s, accumulated return $\mathbf{r}_{acc}$, value network $Z_{\theta}$, scalarization function $f: \mathbb{R}^n \to \mathbb{R}$, discount factor $\gamma$, time of the decision t.

1: Retrieve categorical distribution $Z_{\theta}(s, a) = \{(p_i, s_i)\}_{i=0}^{\prod_{j}(N_j+1)}$ for each action a in state s using the value network.

2: Compute the support of the new distribution:

$$S' = \bigtimes_{j \in \{0..d\}} \{\mathbf{r}_{acc_j} + \gamma^{t+1}(V_{MIN_j} + i\Delta z_j) | 0 \le i \le N_j\}$$

3: Apply scalarization to the atoms of $S'$:

$$\text{scalarized } S' = \{f(\mathbf{s}), \quad \forall \mathbf{s} \in S'\}$$

4: Compute expected scalarized value for each action:

$$Q(s_t, a) = \sum_{\text{atom} \in \text{scalarized } S'} p_i * \text{atom}$$

5: Compute action probabilities using softmax:

$$\pi(a|s) = \frac{\exp(Q(s, a))}{\sum_b \exp(Q(s, b))}$$

6: Sample action $a^* \sim \pi(a|s)$.

7: Return selected action $a^*$.

**Update** Algorithm 3 outlines the update step of the distributional value-based learning algorithm. It computes the projected target distribution for each transition in the sampled batch and minimizes the cross-entropy loss between this target and the current estimated distribution.

**Algorithm 3 Distributional Update**

**Require:** Batch size b, Replay buffer $\mathcal{D}$, value network $Z_{\theta}$, target network $Z_{\theta'}$, learning rate $\alpha$, discount factor $\gamma$, update rate $\tau$.

1: Sample batch of size b from $\mathcal{D}$

2: **for** $j = 1$ to $b$ **do**

3:   $(s, r_{acc}, a, r, s', t) \leftarrow \text{batch}[j]$

4:   Select action $a^* = \arg\max_a \mathbb{E}_{z \sim Z(s',a)} f(r + \gamma z)$

5:   Compute target distribution using $Z_{\theta'}(s', a^*)$, $\gamma$ and $r$.

$$\hat{T}Z(s, a) = r + \gamma \cdot Z_{\theta'}(s', a^*)$$

6: Using Algorithm 4, project $\hat{T}Z(s, a)$ into $S$ resulting in $\Phi\hat{T}Z(s, a)$.

7: Retrieve current action distribution from $Z_{\theta}(s, a)$

8: Compute cross-entropy loss between the projected target distribution $\Phi\hat{T}Z(s, a)$ and the current distribution $Z_{\theta}(s, a)$

9: Compute total loss: $L_j = L_{CE}$

10: Accumulate gradient update $\Delta \leftarrow \Delta + \nabla_{\theta} L_j$

11: **end for**

12: Update network parameters $\theta \leftarrow \theta - \eta\Delta$,

13: Soft update target network: $\theta' \leftarrow \tau\theta + (1 - \tau)\theta'$

**Projection** Algorithm 4 describes the projection step used in multi-objective distributional reinforcement learning. For each transition, it computes how the target return distribution should be projected onto the fixed support of the value distribution, separately along each objective.

**Algorithm 4** Multi-objective projection

**Require:** reward $\mathbf{r}$, done flag $dw$, discount factor $\gamma$, atoms, atom spacing $\Delta z$, min/max atom values $v_{min}, v_{max}$, number of atoms per objective $N$, number of objectives $d$, next distribution $Z'$

1: Compute target values: $t_z \leftarrow \mathbf{r} + \gamma \cdot \text{atoms} \cdot \neg dw$

2: Clamp target values: $t_z \leftarrow \text{clamp}(t_z, v_{min}, v_{max})$

3: **for** each objective $j \in \{1, 2, \dots, d\}$ **do**

4:   **for** each atom $i$ in objective $j$ **do**

5:   **if** $t_z[i] \le v_{min}[j]$ **then**

6:   lower_neighbors[j, i] $\leftarrow v_{min}[j]$

7:   upper_neighbors[j, i] $\leftarrow v_{min}[j]$

8:   weight_low[j, i] $\leftarrow 1$

9:   weight_high[j, i] $\leftarrow 0$

10: **else if** $t_z[i] \ge v_{max}[j]$ **then**

11:   lower_neighbors[j, i] $\leftarrow v_{min}[j] + (N - 1) \cdot \Delta z[j]$

12:   upper_neighbors[j, i] $\leftarrow v_{min}[j] + (N - 1) \cdot \Delta z[j]$

13:   weight_low[j, i] $\leftarrow 0$

14:   weight_high[j, i] $\leftarrow 1$

15: **else**

16:   $l \leftarrow \left[ \frac{t_z[j,i] - v_{min}[j]}{\Delta z[j]} \right]$

17:   $h \leftarrow \frac{t_z[j,i] - (v_{min}[j] + l \cdot \Delta z[j])}{\Delta z[j]}$

18:   lower_neighbors[j, i] $\leftarrow v_{min}[j] + l \cdot \Delta z[j]$

19:   upper_neighbors[j, i] $\leftarrow v_{min}[j] + (l + 1) \cdot \Delta z[j]$

20:   weight_low[j, i] $\leftarrow 1 - h$

21:   weight_high[j, i] $\leftarrow h$

22: **end if**

23: **end for**

24: **end for**

25: **for** each atom $k$ in current distribution support **do**

26:   Generate all $2^d$ neighbor combinations for atom $k$

27:   **for** each $(c_1, c_2, \dots, c_d)$ where $c_j \in \{\text{low, high}\}$ **do**

28:   neighbor $\leftarrow ()$

29:   **for** $j = 1$ to $d$ **do**

30:   **if** $c_j = \text{low then}$

31:   neighbor[j] $\leftarrow lower_neighbors[j][k[j]]$

32:   weight $\leftarrow$ weight $\times$ weight_low[j][k[j]]

33:   **else**

34:   neighbor[j] $\leftarrow upper_neighbors[j][k[j]]$

35:   weight $\leftarrow$ weight $\times$ weight_high[j][k[j]]

36:   **end if**

37:   **end for**

38:   $\text{prob\_mass} \leftarrow Z'[\text{neighbor}] \cdot \text{weight}$

39:   $m[\text{neighbor}] \leftarrow m[\text{neighbor}] + \text{prob\_mass}$

40: **end for**

41: **end for**
## B EMPIRICAL VALIDATION

In this section, we aim to empirically assess the quality of the return distributions learned by our algorithm.

Figure 2 shows the categorical representations of the return distributions the algorithm converges to for each action in state $s_0$. Each square identified with the coordinates of its bottom left point ($x_{min}, y_{min}$) and its upper right point ($x_{max}, y_{max}$) in the figure represents the probability of getting a vector return $r = (r_0, r_1)$ such that $r_0 \in [y_{min}, y_{max}] \land r_1 \in [x_{min}, x_{max}]$ while its color indicates the intensity of the probability. For example, the yellow square at the top left side of Figure 2b indicates that the probability of getting a vector return of (1, 11) is ≈ 0.5. We can see that for both actions the predicted return distribution is accurate since the agent predicts that it will receive a return of (5, 2) with probability ≈ 1 when taking action $a_0$ (cf. Figure 2a) and predicts that it will receive returns (11, 1) and (1, 11) with equal probability ≈ 0.5.

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 2:** Predicted return multivariate distributions for different actions in state $s_0$. Y axis shows returns on the first objective and X axis returns on the second objective</figcaption></figure>

### B.1 Validation scenario

To validate our algorithm and confirm that the return distribution the algorithm converges to is the true distribution of returns, we train our algorithm to solve a toy MO-MDP illustrated on Figure 4. A detailed description of this task is given in Appendix C.3.

<figure><img src="image_5.png" /><figcaption>**Figure 3:** Evolution of Mean Discounted Scalarized Return of each algorithm on the toy MO-MDP</figcaption></figure>

**Validation** We present the results obtained by our algorithm on the validation task. Figure 3 shows the results of training our single-policy algorithm, MOCAC, EUPG and NSW-QL on the toy MO-MDP used for validation with the same scalarization described in C.1. We can see that our algorithm and both policy-based algorithms are able to solve the task whereas NSW-Qlearning is unable to. This proves the novelty of our approach and shows that representing the entire multivariate distribution is necessary to design a value-based MORL algorithm for ESR optimization.

## C EXPERIMENTS DETAILS

### C.1 Validation

On the validation task, the agent is trained over 1250 episodes in the environment, and the average scalarized return of the policy is recorded after each episode. At the end of the training the categorical representation of the return distribution the agent converged to is presented. The scalarization function used to collect experience is the product (i.e. $f(\bar{r}) = \prod_i r_i$). To ensure statistical robustness of our results, the experiments are performed across 11 different seeds.

### C.2 Single policy algorithms

In this section, we describe the experimental protocol followed to generate the testing results presented in section 6. We detail the number of training steps, the scalarization function used and important hyperparameters of each algorithm<fnref n="5" /> (learning-rate, batch-size, number of atoms per objectives). In all the experiments, the policy based algorithms use a neural network with a hidden layer of 64 units to represent the policy (and state values in the case of MOCAC), whereas the MO-CDQN algorithm uses a dueling network architecture [26] with same network architecture to represent the state and advantage values. Throughout all the experiments, the algorithms are evaluated periodically, by taking the policy and

<footnote n="5">⁵The rest of the hyperparameters and other minor implementation details can be found in the code attached to the supplementary material</footnote>
executing a certain number of times on the task and logging the expected scalarized return obtained by that policy. Experiments were repeated across 11 different seeds for the shorter tasks (Fruit-Tree-navigation, Deep-Sea-Treasure) and across 5 different seeds on the longer tasks (Fish-wood, Minecart, Four-room).

**Fruit-Tree-Navigation:** On this task, the goal of the agent is to learn a policy maximizing the product of the objectives. Our algorithm is trained over 500 000 environment steps, whereas baselines algorithms are allowed twice that budget. The policy of each agent is evaluated on the same task periodically every 5 000 steps in the environment. Since the policy learned by the agent is stochastic every evaluation steps consists of running the agent's policy on the tree-navigation task for 100 episodes. After each evaluation step, the average scalarized return achieved by the policies of each agent is reported. The policy based agents are trained using a learning rate of $5 \cdot 10^{-5}$ whereas MO-CDQN uses a learning rate of $10^{-4}$. MOCAC and MO-CDQN both uses 21 atoms to discretize all three objectives. Each objective's value is normalized between [0, 1]. To alleviate the impact of random initialization of the policies, all agents are trained 11 times and the median and fourth and sixth decile scores are reported.

**Deep-Sea-Treasure:** To motivate the use of this environment, we use the same scenario as the one studied in [16]. Consider that the submarine crew is in debt $d$ and that they are given a certain amount of time $t$ to fully reimburse their debt. If the crew manages to find a treasure that allows them to fully reimburse the debt on time then they are set free and get to keep the rest of the treasure. If they are late but still find a treasure that allows them to reimburse the debt then they get to keep the rest of the treasure but they have to pay a late fee $f$ and are penalized proportionally to the number of days they were late by. Finally, if the crew comes back with a treasure that does not fully cover the debt then they are imprisoned in which case they receive the maximal penality $e$. This scenario is formalized by the following scalarization function:

$$u(r_0, r_1) = \begin{cases} r_0 - d & \text{if } r_0 \ge d \quad \wedge r_1 \le t \\ r_0 - d - ((r_1 - t)^2 + f) & \text{if } r_0 \ge d \quad \wedge r_1 > t \\ e & \text{if } r_0 < d \end{cases}$$

To highlight the efficiency of our algorithm, it is only allowed a budget of 100000 timesteps while the rest of the baselines are trained for 200000 timesteps. The policy of each algorithm is periodically evaluated after 1000 steps. In these tasks, all agents use a learning rate of $10^{-4}$. Algorithms MO-CDQN and MOCAC both use the same number of atoms, 21 to discritize the objective space. The first objective values are in the interval [0, 125] and the second are in the interval [-25, 0].

**Fishwood:** We place ourselves in the same context as the one described in [17]. We use a scalarization function $f$ defined as $f(r_1, r_2) = \min(r_1, \lfloor \frac{r_2}{2} \rfloor)$. The policy based agents are trained for a million steps while our MO-CDQN agent is trained for 500,000 steps. The algorithms are evaluated periodically every 5000 steps over 100 execution of the learned policy. The algorithms were evaluated every 5000 steps by taking the learned policy at that instant and evaluating it over 100 executions and logging the average scalarized return obtained over these executions. The policy based algorithms are allowed a million environment step to train a policy while our

value based algorithm is given half that budget. EUPG and MOCAC used a learning rate of $10^{-3}$ to train their policy while ours used a learning rate of $10^{-4}$. MOCAC and MO-CDQN used 11 atoms to discretize each of the two objectives and the objective ranges that were considered by these algorithms are $r_1 \in [0, 20]$ and $r_2 \in [0, 100]$.

**Fourroom:** In these experiments, each agents aims to maximize the product of the three available objectives. The Policy based agents are allowed a budget of one million environment step, whereas the single-policy MO-CDQN agent is allowed half that budget. All algorithms were trained using a learning rate of $10^{-4}$ and MOCAC and MO-CDQN used 11 atoms to discretize each objective. Each objective takes values between $v_{min} = 0$ and $v_{max} = 4$. The agents' policies are frozen every 5000 steps and evaluated on 100 different seeds.

**Minecart:** This environment presents a challenging multi-objective task with a continuous, high-dimensional state space and delayed, stochastic rewards. The agent must learn to collect and return ores while managing fuel consumption. In this setting, we restrict our attention to the two ore collection objectives, which are both normalized to lie within the interval [0, 1.5]. The scalarization function used is the product of these two objectives, encouraging policies that collect a balanced amount of both ores. Our MO-CDQN agent is trained for 2.5 million environment steps, while the policy-based baselines (EUPG and MOCAC) are each trained for 5 million steps. Every 10,000 steps, the current policy is frozen and evaluated over 100 episodes, and the average scalarized return is recorded. EUPG uses a learning rate of $10^{-3}$ while the rest of the algorithms are trained with a learning rate of $10^{-4}$. MOCAC and MO-CDQN both use 21 atoms to discretize each of the two objectives.

### C.2.1 multi-policy algorithm.
To evaluate our multi-policy algorithm, the Fruit-Tree-Navigation task limited to 2 objectives is used with 5 hand picked non-linear scalarization functions. The considered scalarization functions are the following. Note that $r^\uparrow$ is the vector obtained from reordering the components of the reward vector increasingly.

$$\begin{align*} u_1(r_0, r_1) &= \max(r_0, r_1) & u_3(r_0, r_1) &= r_0 r_1 \\ u_2(r_0, r_1) &= \min(r_0, r_1) & u_4(r_0, r_1) &= w_0 r_0^\uparrow + w_1 r_1^\uparrow \\ u_5(r_0, r_1) &= -(w_0(i_0 - r_0) + w_1(i_1 - r_1))^2 \end{align*}$$

The performances of our multi-policy algorithm are compared to the ones obtained by the single-policy variant of MO-CDQN, EUPG and MOCAC. The on-policy single policy algorithms are allowed a budget of 1000000 environment timesteps while the single policy MO-CDQN algorithm is given 500000 environment timesteps to optimize a single scalarization function. The multi-policy algorithm is provided the same budget to find the optimal policy for all the scalarization functions at once. Each experiment is repeated 5 times. We then compare the scores obtained by the best policies found by each algorithm.

## C.3 MO-MDP used for validation

A description of the validation environment is given in this section. The problem consists of a bi-objective MO-MDP illustrated in Figure 4. In this problem, a single agent has to solve a bi-objective stochastic problem. Each action leads to vector of rewards: one reward per objective hence, in this example a couple of values. The
<figure><img src="image_6.png" /><figcaption>Figure 4: Toy MO-MDP used for validating the algorithm</figcaption></figure>

agent starts at state $s_0$ and has to choose between two actions $a_1$ and $a_2$ leading to states $s_1$ and $s_2$ and receiving rewards (2, 0) and (1, 1) respectively. From state $s_1$, the agent can perform action $a_1$ which leads to terminal state $s_3$ and gives reward (3, 3). From state $s_2$, the agent can perform action $a_1$ which is stochastic and leads the agent to states $s_4$ and $s_5$ with probability 0.5 respectively. When the agent arrives at state $s_4$ it receives a reward of (10, 0) and when it arrives at state $s_5$ it receives reward (0, 10).

## C.4 Testing environments description

This section provides a detailed description of the environments used to evaluate the performance of the proposed algorithms. These environments are taken from the MO-Gymnasium benchmark suite [9] and are designed to assess the ability of multi-objective reinforcement learning algorithms to learn under complex reward trade-offs.

*Fruit-Tree-Navigation (FTN)*. The Fruit-Tree-Navigation environment is structured as a binary tree of depth *d*, where each internal node corresponds to a decision point for the agent. At each step, the agent chooses whether to move left or right. Once a leaf node is reached, the episode ends and the agent receives a reward vector associated with the chosen leaf.

In our experiments, the tree has a fixed depth of 7, resulting in $2^7 = 128$ possible terminal nodes. Each leaf node is associated with a *d*-dimensional reward vector where *d* is the number of objectives. The reward vectors are assigned such that different paths through the tree lead to different trade-offs between the objectives, requiring the agent to make strategic decisions early in the episode to reach favorable leaves.

While the original FTN environment supports up to 6 objectives, we limit ourselves to the first 3 objectives for the single-policy experiments and to the first 2 objectives for the multi-policy setting in order to ensure scalability of the categorical algorithms.

*Deep-Sea-Treasure (DST)*. Deep-Sea-Treasure is a classic multi-objective grid world problem that models the trade-off between finding valuable treasures and minimizing resource consumption. The agent controls a submarine and navigates a two-dimensional grid to reach a cell containing a treasure.

Each cell in the grid either contains no treasure (yielding a reward of (0, -1)) or a treasure of some positive value *v* (yielding a reward of (*v*, -1)). The second component represents a fixed cost per action due to fuel consumption. The episode terminates when the agent reaches a cell containing a treasure.

The treasure map used in our experiments is the standard one proposed in [16], where several treasures of varying values are

<figure><img src="image_7.png" /><figcaption>Figure 5: Treasure map used in our experiments</figcaption></figure>

placed at increasing depths. This setup introduces a trade-off between quickly reaching low-value treasures and spending more fuel to reach higher-value ones. The treasure locations and their corresponding values are shown in Figure 5.

The values of the parameters of the scalarization function used for this environment are the following: $d = 20$; $t = 19$; $e = -100$

*Fishwood*. The Fishwood environment models a simple resource-gathering problem where the agent alternates between fishing and collecting wood. At each step, it chooses between two actions: go fishing or go collect wood.

The reward is a 2-dimensional vector: the first component corresponds to successfully collecting wood, and the second to successfully catching a fish, both with fixed probabilities. The episode lasts for a fixed number of steps.

In our experiments, this environment is used to evaluate the agent's ability to balance between stochastic resource yields under limited time.

*Fourroom*. The Fourroom environment is a gridworld split into four rooms separated by walls with narrow passageways. The agent navigates the grid to collect specific items and then reach a fixed goal location.

The reward is a 3-dimensional vector: each component corresponds to whether a particular object—blue square, green triangle, or red circle—was collected during the episode. The episode terminates once the agent reaches the goal.

*Minecart*. The Minecart environment simulates a resource collection task where the agent controls a cart that can move, mine, and return to a base to deposit minerals. The environment features two types of minerals and introduces a trade-off between collecting valuable resources and minimizing fuel consumption.

The agent receives a 3-dimensional reward vector corresponding to the quantities of each mineral retrieved and the fuel used. The episode ends when the cart returns to the base.

While the original Minecart environment includes a fuel consumption component, we limit ourselves to optimizing fairly the two objectives related to resource collection in order to focus on learning diverse but balanced policies. The scalarization function used in this case is a *Product*, encouraging policies that perform well across both resource-related objectives.
*MountainCar*. The MO-MountainCar-v0 environment is a multi-objective extension of the classic *MountainCar* task from Gymnasium, designed to evaluate trade-offs between multiple conflicting objectives in a continuous control setting. The *action space* is discrete with three possible actions: reverse (0), neutral (1), and forward (2). Each *observation* consists of a two-dimensional vector representing the car's position and velocity, bounded respectively between [-1.2, 0.6] and [-0.07, 0.07]. The *reward* is a three-dimensional vector by default, capturing separate penalties for time, reverse movement, and forward movement. Specifically, each time step incurs a time penalty of -1.0, and additional -1.0 penalties are applied whenever the agent chooses to move in reverse or forward. These objectives encourage the agent to reach the top of the hill efficiently while balancing the cost of movement.

In the experiments, the rewards are aggregated using the following non-linear scalarization function:

$$u(r) = -(r_0^2 + r_1 \cdot r_2)$$
# D PROOFS

In this section we aim to prove the theorem and corollary presented in the paper used to justify the validity of the proposed algorithm. The proof follows the following structure:

(1) Define a distance between return functions based on the Wasserstein distance between probability distributions.

(2) Prove that, given a scalarization function $u$, the composition of operators $E_u$ and $\mathcal{T}$ is a contraction under certain conditions.

(3) Prove that the fixed point distribution towards which the these operators converge has optimal expected scalarized utility.

**Definition D.1 (Earth mover distance between probability distributions).** Let $Z_1(s, a)$ and $Z_2(s, a)$ be two probability distributions over $\mathbb{R}^d$. We define the Wasserstein-1 distance (also called Earth Mover Distance) between $Z_1(s, a)$ and $Z_2(s, a)$ as:

$$W_1(Z_1(s, a), Z_2(s, a)) := \inf_{\pi \in \Pi(Z_1(s,a), Z_2(s,a))} \int_{\mathbb{R}^d \times \mathbb{R}^d} \|z_1 - z_2\| d\pi(z_1, z_2)$$

where $\Pi(Z_1(s, a), Z_2(s, a))$ is the set of all couplings of $Z_1(s, a)$ and $Z_2(s, a)$, i.e., joint distributions $\pi$ over $\mathbb{R}^N \times \mathbb{R}^N$ such that the marginals of $\pi$ are $Z_1(s, a)$ and $Z_2(s, a)$,

**Definition D.2 (Distance Between Return Functions).** Let $Z_1$ and $Z_2$ be two return functions mapping each state-action pair $(s, a)$ to a probability distribution over $\mathbb{R}^d$. We define the distance between $Z_1$ and $Z_2$ as:

$$\bar{d}(Z_1, Z_2) := \sup_{(s,a)} W_1(Z_1(s, a), Z_2(s, a))$$

where $W_1$ is the Wasserstein-1 distance defined in Definition 1.

**LEMMA D.3.** The distance $\bar{d}(Z_1, Z_2)$ defines a metric on the space of return functions $Z$.

**PROOF.** Although this result was proved by [3] (who focused mainly on the triangle inequality), we provide a full proof here for completeness.

To show that $\bar{d}$ defines a metric, we verify that it satisfies the following four properties:

(1) **Non-negativity:** $\bar{d}(Z_1, Z_2) \ge 0$.

(2) **Identity of indiscernibles:** $\bar{d}(Z_1, Z_2) = 0 \iff Z_1 = Z_2$.

(3) **Symmetry:** $\bar{d}(Z_1, Z_2) = \bar{d}(Z_2, Z_1)$.

(4) **Triangle inequality:** $\bar{d}(Z_1, Z_3) \le \bar{d}(Z_1, Z_2) + \bar{d}(Z_2, Z_3)$.

We now verify each of these properties individually:

1. **Non-negativity.** For any $(s, a)$, the norm $\|z_1 - z_2\|$ is non-negative, and so is its expectation over any coupling. Therefore,
$$W_1(Z_1(s, a), Z_2(s, a)) \ge 0,$$

and hence

$$\bar{d}(Z_1, Z_2) = \sup_{(s,a)} W_1(Z_1(s, a), Z_2(s, a)) \ge 0.$$

2. **Identity of indiscernibles.**

* If $Z_1 = Z_2$, then $Z_1(s, a) = Z_2(s, a)$ for all $(s, a)$, so

$$W_1(Z_1(s, a), Z_2(s, a)) = 0 \quad \forall (s, a),$$

and thus $\bar{d}(Z_1, Z_2) = 0$.

* Conversely, if $\bar{d}(Z_1, Z_2) = 0$, then $W_1(Z_1(s, a), Z_2(s, a)) = 0$ for all $(s, a)$, and since $W_1$ is a metric on distributions, this implies $Z_1(s, a) = Z_2(s, a)$ for all $(s, a)$, i.e., $Z_1 = Z_2$.

3. **Symmetry.** Since the Wasserstein-1 distance is symmetric:

$$W_1(Z_1(s, a), Z_2(s, a)) = W_1(Z_2(s, a), Z_1(s, a)), \quad \forall (s, a),$$

we have:

$$\bar{d}(Z_1, Z_2) = \sup_{(s,a)} W_1(Z_1(s, a), Z_2(s, a)) = \bar{d}(Z_2, Z_1).$$

4. **Triangle inequality.** For any return functions $Z_1, Z_2, Z_3$ and any $(s, a)$, we have:

$$W_1(Z_1(s, a), Z_3(s, a)) \le W_1(Z_1(s, a), Z_2(s, a)) + W_1(Z_2(s, a), Z_3(s, a)),$$

by the triangle inequality for $W_1$. Taking the supremum over $(s, a)$, we get:

$$\bar{d}(Z_1, Z_3) \le \bar{d}(Z_1, Z_2) + \bar{d}(Z_2, Z_3).$$

Hence, $\bar{d}$ satisfies all the properties of a metric.

□
**THEOREM D.4.** Let $Z_1$ and $Z_2$ be two return functions. Assume the scalarization function $u: \mathbb{R}^N \to \mathbb{R}$ is K-Lipschitz with respect to the $\ell_1$ norm. Then the composed operator $E_u \circ \mathcal{T}$ satisfies:

$$|E_u \mathcal{T} Z_1 - E_u \mathcal{T} Z_2| \le \gamma K |E_u Z_1 - E_u Z_2|$$

In particular, if $\gamma K < 1$, the operator is a contraction.

**Proof.** For any state-action pair $(s, a)$, we have:

$$E_u \mathcal{T} Z_i(s, a) = \mathbb{E}_{z \sim \mathcal{T} Z_i(s, a)} [u(z)]$$

From the definition of $\mathcal{T}$:

$$\mathcal{T} Z_i(s, a) = \mathbf{r}(s, a) + \gamma Z_i(s', a_i^*)$$

where

$$a_i^* = \arg\max_{a' \in A} \mathbb{E}_{z \sim Z_i(s', a')} [f(\mathbf{r}_{\text{acc}} + \gamma^{t+1} \mathbf{z})]$$

Thus

$$E_u \mathcal{T} Z_i(s, a) = \mathbb{E}_{\mathbf{r} \sim R(\cdot | s, a), s' \sim P(\cdot | s, a), z_i \sim Z_i(s, a_i^*)} [u(\mathbf{r} + \gamma \mathbf{z}_i)]$$

Applying this substitution we get

$$\begin{aligned}
|E_u \mathcal{T} Z_1(s, a) - E_u \mathcal{T} Z_2(s, a)| &= |\mathbb{E}_{\mathbf{r},s',z_1 \sim Z_1(s',a_1^*)} u(\mathbf{r} + \gamma \mathbf{z}_1) - \mathbb{E}_{\mathbf{r},s',z_2 \sim Z_2(s',a_2^*)} u(\mathbf{r} + \gamma \mathbf{z}_2)| \\
&= |\mathbb{E}_{\mathbf{r},s'}[\mathbb{E}_{z_1 \sim Z_1(s',a_1^*)} u(\mathbf{r} + \gamma \mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s',a_2^*)} u(\mathbf{r} + \gamma \mathbf{z}_2)]| \quad (4) \\
&= |\mathbb{E}_{\mathbf{r},s'}[\mathbb{E}_{z_1 \sim Z_1(s',a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s',a_2^*)} f(\mathbf{z}_2)]| \quad (\text{with } f(\mathbf{z}) := u(\mathbf{r} + \gamma \mathbf{z}))
\end{aligned}$$

We focus on the difference of expectation term $|\mathbb{E}_{z_1 \sim Z_1(s',a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s',a_2^*)} f(\mathbf{z}_2)|$.

We suppose without loss of generality that $\mathbb{E}_{z_1 \sim Z_1(s',a_1^*)} f(\mathbf{z}_1) \ge \mathbb{E}_{z_2 \sim Z_2(s',a_2^*)} f(\mathbf{z}_2)$

Thus

$$|\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)| = \mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) \quad (5)$$

We add and subtract $\mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)$

$$\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) = \mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) + \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) \quad (6)$$

Since $a_2^* = \arg\max_a \mathbb{E}_{z_2 \sim Z_2(s', a)} f(\mathbf{z}_2)$ then $\mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) \ge \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)$

$$0 \le \mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) \le \mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2) \quad (7)$$

Thus

$$|\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)| \le |\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)| \quad (8)$$

Using the Kantorovich-Rubenstein duality theorem [25] and knowing that $f$ is $\gamma K$ Lipschitz continuous we have

$$|\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)| \le \gamma K \cdot W_1(Z_1(s', a_1^*), Z_2(s', a_1^*)) \quad (9)$$

and since the right hand side of that equation does not depend on $r$ we can combine equations (5), (9) and (10) to obtain

$$|\mathbb{E}_{\mathbf{r},s'}[\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)]| \le \mathbb{E}_{s', \gamma K} W_1(Z_1(s', a_1^*), Z_2(s', a_1^*)) \quad (10)$$

$$\begin{aligned}
|\mathbb{E}_{\mathbf{r},s'}[\mathbb{E}_{z_1 \sim Z_1(s', a_1^*)} f(\mathbf{z}_1) - \mathbb{E}_{z_2 \sim Z_2(s', a_2^*)} f(\mathbf{z}_2)]| &\le \gamma K \cdot \mathbb{E}_{s', W_1(Z_1(s', a_1^*), Z_2(s', a_1^*))} \quad (\text{Using the linearity of expectation}) \\
&\le \gamma K \cdot \sup_{s'} W_1(Z_1(s', a_1^*), Z_2(s', a_1^*)) \quad (\text{By definition of the sup}) \quad (11) \\
&\le \sup_{(s,a)} W_1(Z_1(s, a), Z_2(s, a)) = \bar{d}(Z_1, Z_2)
\end{aligned}$$

Hence, we have shown that

$$|E_u \mathcal{T} Z_1(s, a) - E_u \mathcal{T} Z_2(s, a)| \le \gamma K \cdot \bar{d}(Z_1(s, a), Z_2(s, a))$$

for all $(s, a)$, which implies

$$|E_u \mathcal{T} Z_1 - E_u \mathcal{T} Z_2| \le \gamma K \cdot \bar{d}(Z_1, Z_2).$$

Thus, if $\gamma K < 1$, the operator $E_u \circ \mathcal{T}$ is a contraction, and by Banach's fixed-point theorem [5], it admits a unique fixed point. ■
The condition $\gamma K < 1$ ensures that the operator $E_u \circ \mathcal{T}$ is a contraction. This condition can be relaxed by using n-step bootstrapping instead of 1-step updates. In that case, the contraction factor becomes $\gamma^n K$, and thus a scalarization function with a larger Lipschitz constant can still yield a contraction as long as $\gamma^n K < 1$.

**COROLLARY D.5 (OPTIMALITY OF THE FIXED POINT).** Let $Z^*$ be the unique fixed point of the composed operator $E_u \mathcal{T}$ and suppose the scalarization function $u$ is Lipschitz with constant $K$, such that $\gamma K < 1$. Then the expected scalarized return of $Z^*$ corresponds to the optimal scalarized value function:

$$ \forall (s, a), \quad E_u Z^*(s, a) = \max_{\pi} \mathbb{E}_{\pi} \left[ u\left(\sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t\right) \middle| s_0 = s, a_0 = a \right] $$

**PROOF.** Let $Q^*(s, a) := \max_{\pi} \mathbb{E}_{\pi} [u(\sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t) | s_0 = s, a_0 = a]$ denote the optimal scalarized value function.

**We first prove that**

$$ E_u Z^*(s, a) \geq Q^*(s, a) $$

Since $Q^*(s, a)$ is the maximum expected scalarized return, there exists an optimal policy $\pi^*$ such that:

$$ Q^*(s, a) = \mathbb{E}_{\pi^*} \left[ u\left(\sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t\right) \middle| s, a \right] $$

Now, consider the return distribution $\tilde{Z}(s, a)$ induced by following policy $\pi^*$ from each state-action pair $(s, a)$. That is, $\tilde{Z}(s, a)$ is the distribution of $\sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t$ when starting from $(s, a)$ and following $\pi^*$.

By construction:

$$ E_u \tilde{Z}(s, a) = \mathbb{E}_{\pi^*} \left[ u\left(\sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t\right) \middle| s, a \right] = Q^*(s, a) $$

Now, $\tilde{Z}$ satisfies a policy-evaluation version of the distributional Bellman equation ([3], [29]):

$$ \tilde{Z}(s, a) = \mathbf{r} + \gamma \tilde{Z}(s', \pi^*(s')) \quad \forall (s, a) $$

where $s' \sim P(\cdot|s, a)$ and $\mathbf{r} \sim R(\cdot|s, a)$.

However, since $Z^*$ is the fixed point of the **optimality** operator $\mathcal{T}$, we have:

$$ Z^*(s, a) = \mathbf{r} + \gamma Z^*(s', a^*) \quad \forall (s, a) $$

where $a^* = \arg \max_{a'} \mathbb{E}_{z \sim Z^*(s', a')} u(\mathbf{r} + \gamma \mathbf{z})$.

By definition of $a^*$, we must have:

$$ \mathbb{E}_{z \sim Z^*(s', a^*)} u(\mathbf{r} + \gamma \mathbf{z}) \geq \mathbb{E}_{z \sim \tilde{Z}(s', \pi^*(s'))} u(\mathbf{r} + \gamma \mathbf{z}) $$

Thus

$$ E_u Z^*(s, a) \geq E_u \tilde{Z}(s, a) \quad \forall (s, a) $$

In conclusion

$$ E_u Z^*(s, a) \geq Q^*(s, a) \quad \forall (s, a) $$

**Furthermore, we prove that** For all state-action pairs $(s, a)$, the expected scalarized return under the optimal return distribution is bounded above by the optimal value:

$$ E_u Z^*(s, a) \leq Q^*(s, a) $$

**PROOF.** Let $Z^*(s, a)$ denote the return distribution obtained by taking action $a$ in state $s$ and following an optimal policy thereafter. Then $E_u Z^*(s, a)$ is the expected scalarized return under this specific policy.

By definition, $Q^*(s, a)$ is the maximal expected return achievable from $(s, a)$, i.e.,

$$ Q^*(s, a) = \max_{\pi} \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t \mathbf{r}_t \middle| s_0 = s, a_0 = a \right] $$

Since the expectation under a fixed policy (in this case, the one inducing $Z^*$) cannot exceed the maximum over all policies, we have:

$$ E_u Z^*(s, a) \leq Q^*(s, a) $$

□
**Conclusion:** We proved that

$$E_u Z^*(s, a) \le Q^*(s, a) \wedge E_u Z^*(s, a) \ge Q^*(s, a) \quad \forall (s, a) \implies E_u Z^*(s, a) = Q^*(s, a) \quad \forall (s, a)$$

This establishes that the expected scalarized return of the distributional fixed point corresponds exactly to the optimal scalarized value function. □