Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning

## Abstract

Abstract In many real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines.
To address these considerations, we propose a constrained multi-objective RL algorithm named Co nstrained M ulti- O bjective G radient A ggregator (CoMOGA) .
In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima.
It is also essential to efficiently handle safety constraints for stable training and constraint satisfaction.
We address these challenges straightforwardly by treating the maximization of multiple objectives as a constrained optimization problem (COP), where the constraints are defined to improve the original objectives.
Existing safety constraints are then integrated into the COP, and the policy is updated using a linear approximation, which ensures the avoidance of gradient conflicts.
Despite its simplicity, CoMOGA guarantees optimal convergence in tabular settings.
Through various experiments, we have confirmed that preventing gradient conflicts is critical, and the proposed method achieves constraint satisfaction across all tasks.

## 1 Introduction

Many real-world reinforcement learning (RL) applications involve multiple objectives and have to consider safety simultaneously.
For instance, locomotion tasks for legged robots focus on maximizing goal-reaching success rate and energy efficiency while avoiding collisions with the environment, as illustrated in Figure 2.
To effectively tackle these considerations, constrained multi-objective RL (CMORL) has been introduced [].
CMORL aims to find a set of constrained-Pareto (CP) optimal policies [], which are not dominated by any others while satisfying safety constraints, rather than finding a single optimal policy.
This allows users to choose and utilize their preferred policy from the set without additional training.
In order to express a set of CP optimal policies, it is common to use a concept called preference [, ], which encodes the relative importance of the objectives.
Then, a set of policies can be represented by a preference-conditioned policy, a mapping function from a preference space to a policy space.

In most multi-objective RL (MORL) algorithms [, , ], the preference-conditioned policy is trained by maximizing a scalarized reward, computed as the dot product of a given preference and rewards of the multiple objectives.
These approaches, called linear scalarization (LS), offer the benefits of straightforward implementation and ensure Pareto-optimal convergence in tabular settings [].
However, when applied to deep RL with function approximators, LS tends to converge to local optima due to the nonlinearity of the objective functions, as shown in a toy example in Figure 2.
In the field of multi-task learning (MTL) [, , ], this issue has been addressed by avoiding conflicts between the gradients of multiple objective functions.
We hypothesize that similar improvements can be achieved in CMORL by avoiding gradient conflicts.

It is also crucial to handle constraints as well as multiple objectives in CMORL.
A straightforward approach is treating the constraints as other objectives and concurrently adjusting the preferences corresponding to those objectives to maintain them below thresholds, as done in [].
This approach exactly aligns with the Lagrangian method, which solves the Lagrange dual problem by updating policy and multipliers alternatively.
However, the Lagrangian method can make training unstable due to the concurrent update of policy and multipliers [, ].
As the updates of the policy and multipliers influence each other, any misstep in either can make the training process diverge.
Thus, it is critical to manage constraints without introducing additional optimization variables.

In order to avoid gradient conflicts and efficiently handle safety constraints, we introduce a novel but straightforward algorithm named Constrained Multi-Objective Gradient Aggregator (CoMOGA).
CoMOGA treats a multi-objective maximization problem as a constrained optimization problem (COP), where constraints are designed to enhance the original objective functions in proportion to their respective preference values.
Existing safety constraints are then incorporated into the constraint set of the COP.
By explicitly preventing the given objectives from decreasing, CoMOGA successfully avoids gradient conflicts.
The policy gradient is then calculated by solving this transformed problem using linear approximation within a local region, thereby eliminating the need for extra variables to handle safety constraints.
While the proposed method is intuitive and straightforward, we have also demonstrated that it converges to a CP optimal policy in a tabular setting.

The proposed method has been evaluated across diverse environments, including multi-objective tasks with and without constraints.
The experimental results support our hypothesis that avoiding gradient conflicts is effective for preventing convergence to local optima.
This is particularly essential in environments with unstable dynamics, such as bipedal and humanoid locomotion tasks, since they are prone to get stuck in suboptimal behaviors, such as frequent falling.
Furthermore, the absence of additional variables enables stable training with constraint satisfaction across all tasks.

Figure: Figure 1: Example of CMORL. The robot aims to maximize energy efficiency and velocity while maintaining its balance to avoid falling. In order to consider such safety and multiple objectives, CMORL finds a set of feasible policies that are not dominated by other policies and satisfy constraints, which are indicated by the dashed line.
Refer to caption: extracted/5633925/imgs/overview.png

## 2 Related Work

Multi-Objective RL.
MORL aims to find a set of Pareto-optimal policies that are not dominated by any others.
Given that Pareto-optimal policies can be obtained with appropriate reward scalarization [], many MORL algorithms utilize scalarization techniques.
Based on linear scalarization, Yang et al. [] introduced a new Bellman operator for MORL algorithms, which estimates a target Q value for a given preference by combining Q values from other preferences.
Basaklar et al. [] presented a similar Bellman operator and extended it to an actor-critic framework to enable its application in continuous action spaces.
For the continuous action spaces, Lu et al. [] proposed a SAC-based MORL algorithm, theoretically demonstrating that elements of the Pareto-optimal policy set can be achieved through linear scalarization.
Not only the linear scalarization, Van Moffaert et al. [] proposed a nonlinear scalarization method, called Chebyshev scalarization, which can cover a non-convex policy set.
To further improve performance, the evolution strategy (ES) can be applied as done in multi-objective optimization criteria [].
Chen et al. [] and Xu et al. [] proposed methods that repetitively update a policy for each population using linear scalarization and produce a new generation of policy parameters.
There is also a different approach from scalarization.
Abdolmaleki et al. [] have proposed a heuristic method of applying preference in the action distribution space instead of the reward space.
It is further extended to a CMORL algorithm in [].

Constrained RL.
Constrained RL is developed to explicitly consider safety or constraints in RL.
Based on the approach to handling constraints, constrained RL algorithms can be categorized into two types: primal-dual methods and primal methods.
The primal-dual methods, also called Lagrangian methods, are designed to address Lagrange dual problems.
Ding et al. [] introduced a natural policy gradient-based primal-dual method and demonstrated its convergence to an optimal policy at a specified convergence rate.
Another primal-dual method, proposed by Bai et al. [], ensures that a trained policy results in zero constraint violations during evaluation.
While these methods are straightforward to implement and can be integrated with standard RL algorithms, the training process can be unstable due to additional optimization variables (Lagrange multipliers) [].
In contrast to the primal-dual method, the primal method addresses the constrained RL problem directly, so no additional optimization variables are required.
Achiam et al. [] introduced a trust region-based primal method. This approach linearly approximates a constraint within a trust region and updates the policy through a line search.
Xu et al. [] presented a natural policy gradient-based algorithm and demonstrated its convergence rate towards an optimal policy.
In multi-constrained settings, Kim et al. [] proposed a primal method to handle infeasible starting cases.

## 3 Background

Constrained Multi-Objective MDP.
We define a constrained multi-objective Markov decision process (CMOMDP) as a tuple represented by $\langle S$, $A$, $\mathcal{P}$, $R$, $C$, $\rho$, $\gamma\rangle$, where $S$ is a state space, $A$ is an action space, $\mathcal{P}:S\times A\times S\mapsto[0,1]$ is a transition model, $R:S\times A\times S\mapsto\mathbb{R}^{N}$ is a vectorized reward function, $C:S\times A\times S\mapsto\mathbb{R}^{M}$ is a vectorized cost function, $\rho$ is an initial state distribution, and $\gamma$ is a discount factor.
The reward and cost function are bounded by $R_{\mathrm{max}}$.
In the CMOMDP setting, a policy $\pi(\cdot|s)\in\Pi$ can be defined as a state-conditional action distribution.
The objective function of the $i$th reward is defined as:

$$ $J_{R_{i}}(\pi):=\mathbb{E}\left[\sum\nolimits_{t=0}^{\infty}\gamma^{t}R_{i}(s_ {t},a_{t},s_{t+1})\right],$ $$

where $s_{0}\sim\rho$, $a_{t}\sim\pi(\cdot|s_{t})$, and $s_{t+1}\sim\mathcal{P}(\cdot|s_{t},a_{t})$ $\forall t$.
We also define the constraint function $J_{C_{k}}(\pi)$ by replacing the reward with the cost function.
A constrained multi-objective RL (CMORL) problem is defined as:

$$ $\mathrm{maximize}_{\pi}\;J_{R_{i}}(\pi)\;\forall i\in\{1,...,N\}\quad\mathbf{s .t.}\;J_{C_{k}}(\pi)\leq d_{k}\;\forall k\in\{1,...,M\},$ (1) $$

where $d_{k}$ is the threshold of the $k$th constraint.
Here, it is ambiguous to determine which policy is optimal when there are multiple objectives.
To address this, we instead define a set of optimal policies using the following notion, called *constrained dominance* []:

###### Definition 3.1 (Constrained Dominance) .

Given two policies $\pi_{1},\pi_{2}\in\{\pi\in\Pi|J_{C_{k}}(\pi)\leq d_{k}\;\forall k\}$, $\pi_{1}$ is dominated by $\pi_{2}$ if $J_{R_{i}}(\pi_{1})\leq J_{R_{i}}(\pi_{2})\;\forall i\in\{1,...,N\}$.

A policy $\pi$ which is not dominated by any policy is called a *constrained-Pareto (CP) optimal* policy, and the set of all CP optimal policies is called *constrained-Pareto (CP) front* [].
Additionally, a condition for gradient conflicts is defined as follows:

###### Definition 3.2 (Gradient Conflict) .

Let $g_{i}$ be the gradient of the $i$th objective function, and let $g$ be the gradient used for policy updates.
We say there is a gradient conflict if $\exists i$ such that $g_{i}^{T}g<0$.

As studied in MTL [, ] and shown in Figure 2, gradient conflicts can cause convergence to local optima.
Thus, we aim to find the CP front while updating policies without gradient conflicts.

Preference.
In order to express a subset of the CP front, many existing MORL works [, ] introduce a preference space, defined as: $\Omega:=\{\omega\in\mathbb{R}^{N}|\;\omega_{i}\geq 0,\;||\omega||_{\infty}=1\}$,(^1^11It is common to define the preference space using the norm $||\omega||_{1}$, but for convenience, we employ $||\omega||_{\infty}$. Additionally, an one-to-one conversion exists between these definitions.) where a preference $\omega\in\Omega$ serves to specify the relative significance of each objective.
Subsequently, a specific subset of the Pareto front can be characterized by a function mapping from the preference space to the policy space.
We denote this function as a universal policy $\pi(\cdot|s,\omega)$, which is characterized as an action distribution contingent upon both the state and preference.
Ultimately, our goal is to train the universal policy that covers the CP front as extensively as possible.

###### Definition 3.1 (Constrained Dominance) .

###### Definition 3.2 (Gradient Conflict) .

## 4 Proposed Method

In this section, our discussion is organized into four parts: 1) building a constrained optimization problem to handle the multiple objectives with safety constraints, 2) calculating a policy gradient by aggregating gradients of the objective and constraint functions for a single preference, 3) introducing a method for training a universal policy covering a spectrum of preferences, and 4) concluding with an analysis of the convergence properties of the proposed method.
Before that, we first parameterize the policy and define gradients of the objective and constraint functions, as well as a local region in the parameter space.
By representing the policy as $\pi_{\theta}$ with a parameter $\theta\in\Theta$, the gradients of the objective and constraint functions in (1) are expressed as:

$$ $g_{i}:=\nabla_{\theta}J_{R_{i}}(\pi_{\theta}),\;b_{k}:=\nabla_{\theta}J_{C_{k} }(\pi_{\theta}).$ $$

For brevity, we will denote the objectives as $J_{R_{i}}(\theta)$ and the constraints as $J_{C_{k}}(\theta)$.
The local region is then defined using a positive definite matrix $H$ as: $||\Delta\theta||_{H}\leq\epsilon$, where $||x||_{H}:=\sqrt{x^{T}Hx}$, and $\epsilon$ is the local region size, a hyperparameter.
$H$ can be the identity matrix or the Fisher information matrix, as in TRPO paper [].

### 4.1 Transformation to Constrained Optimization

We introduce a process of treating the CMORL problem as a constrained optimization problem (COP).
To provide insight, we first present the process in a simplified problem where only the $i$th objective exists and then extend it to the original problem.
The problem of maximizing the $i$th objective within the local region is expressed as follows:

$$ $\displaystyle\mathrm{max}_{\Delta\theta}\;J_{R_{i}}(\theta_{\mathrm{old}}+ \Delta\theta)\quad\mathbf{s.t.}\;||\Delta\theta||_{H}\leq\epsilon.$ (2) $$

Assuming that $J_{R_{i}}(\theta)$ is linear with respect to $\theta$ within the local region, the above problem has the same solution as the following problem:

$$ $\displaystyle\mathrm{min}_{\Delta\theta}\;\Delta\theta^{T}H\Delta\theta\quad \mathbf{s.t.}\;e_{i}\leq J_{R_{i}}(\theta_{\mathrm{old}}+\Delta\theta)-J_{R_{i }}(\theta_{\mathrm{old}}),$ (3) $$

where $e_{i}:=\epsilon||g_{i}||_{H^{-1}}$.
It is shown that the solutions to (2) and (3) are identical in Appendix B.1, and this finding confirms that objectives can be converted into constraints in this way.
Consequently, the CMORL problem can be transformed as follows:

$$ $\mathrm{min}_{\Delta\theta}\Delta\theta^{T}H\Delta\theta\quad\mathbf{s.t.}\; \omega_{i}e_{i}\leq J_{R_{i}}(\theta_{\mathrm{old}}+\Delta\theta)-J_{R_{i}}( \theta_{\mathrm{old}})\;\forall i,\;J_{C_{k}}(\theta_{\mathrm{old}}+\Delta \theta)\leq d_{k}\;\forall k,$ (4) $$

where $e_{i}$ in (3) is scaled to $\omega_{i}e_{i}$ to reflect the given preference $\omega$.
We will show that the policy updated using (4) converges to a CP optimal policy of (1) in Section 4.4 under mild assumptions.

### 4.2 Gradient Aggregation

Now, we calculate a policy gradient by solving (4) through a linear approximation and update a policy within the local region to reduce approximation errors.
First, the problem (4) can be linearly approximated as the following quadratic programming (QP) problem:

$$ $\bar{g}_{\omega}^{\mathrm{ag}}:=\mathrm{argmin}_{\Delta\theta}\Delta\theta^{T} H\Delta\theta\quad\mathbf{s.t.}\;\omega_{i}e_{i}\leq g_{i}^{T}\Delta\theta\; \forall i,\;b_{k}^{T}\Delta\theta+J_{C_{k}}(\theta_{\mathrm{old}})\leq d_{k}\; \forall k,$ (5) $$

where the gradients of the objective and constraint functions, $g_{i}$ and $b_{k}$, are aggregated into $\bar{g}_{\omega}^{\mathrm{ag}}$.
The gradient is then clipped to ensure that the policy is updated within the local region, as follows:

$$ $g_{\omega}^{\mathrm{ag}}:=\min(1,\epsilon/||\bar{g}_{\omega}^{\mathrm{ag}}||_{ H})\bar{g}_{\omega}^{\mathrm{ag}},$ $$

where the policy will be updated by $\theta_{\mathrm{old}}+g_{\omega}^{\mathrm{ag}}$.
However, due to the linear approximation, the updated policy can violate the safety constraints.
To address this issue, we take a recovery step by solving the following QP problem, which only consists of the safety constraints, when the current policy violates any safety constraints:

$$ $\bar{g}_{\omega}^{\mathrm{ag}}:=\mathrm{argmin}_{\Delta\theta}\Delta\theta^{T} H\Delta\theta\quad\mathbf{s.t.}\;b_{k}^{T}\Delta\theta+J_{C_{k}}(\theta_{ \mathrm{old}})\leq d_{k}\;\forall k.$ (6) $$

As a result, the process for obtaining the policy gradient can be summarized as follows, which is named *constrained multi-objective gradient aggregator (CoMOGA)*:

$$ $\displaystyle\bar{g}_{\omega}^{\mathrm{ag}}$ $\displaystyle=\begin{cases}\mathrm{argmin}_{\Delta\theta}\Delta\theta^{T}H \Delta\theta\;\mathbf{s.t.}\;\omega_{i}e_{i}\leq g_{i}^{T}\Delta\theta,\;b_{k} ^{T}\Delta\theta+J_{C_{k}}(\theta_{\mathrm{old}})\leq d_{k}&\text{if}\;J_{C_{k }}(\theta_{\mathrm{old}})\leq d_{k}\;\forall k,\\ \mathrm{argmin}_{\Delta\theta}\Delta\theta^{T}H\Delta\theta\;\mathbf{s.t.}\;b_ {k}^{T}\Delta\theta+J_{C_{k}}(\theta_{\mathrm{old}})\leq d_{k}&\text{otherwise ,}\\ \end{cases}$ $\displaystyle g_{\omega}^{\mathrm{ag}}$ $\displaystyle=\min(1,\epsilon/||\bar{g}_{\omega}^{\mathrm{ag}}||_{H})\bar{g}_{ \omega}^{\mathrm{ag}},$ $$

and it is illustrated in Figure 3.
Note that the aggregated gradient satisfies $0\leq g_{i}^{T}g_{\omega}^{\mathrm{ag}}$ $\forall i$ due to the transformed constraints in (5), which ensures that the aggregated gradient does not conflict with any objective functions.

Figure: Figure 3: Process of CoMOGA. We visualize the process in the parameter space, and the gray areas represent constraints. (Linear approximation) CoMOGA linearly approximates the original CMORL problem in (1). The gradients of the objective and constraint functions are visualized as black and blue arrows, respectively. (Transformation) The objectives are converted to constraints as described in (5). The intersection of all constraints is shown as the red area. (Scaling) The solution of the transformed problem, $\bar{g}^{\mathrm{ag}}$, is scaled to ensure that the updated policy remains within the local region.
Refer to caption: extracted/5633925/imgs/CoMOGA2.png

### 4.3 Update Rule for Universal Policy

We have introduced a gradient aggregation method named CoMOGA for updating a policy for a given single preference.
This section presents a method for updating a universal policy that can cover the entire preference space.
Given that a policy is updated to $\theta_{\mathrm{old}}+g_{\omega}^{\mathrm{ag}}$ when a preference $\omega$ is specified, the ideal universal policy should satisfy the following conditions:

$$ $\bar{\pi}(a|s,\omega)=\pi_{\theta_{\mathrm{old}}+g_{\omega}^{\mathrm{ag}}}(a|s ,\omega)\quad\forall a\in A,\;\forall s\in S,\;\text{and}\;\forall\omega\in\Omega,$ (7) $$

where we denote $\pi_{\theta_{\mathrm{old}}+g_{\omega}^{\mathrm{ag}}}$ as an intermediate policy $\bar{\pi}_{\omega}$.
This can be interpreted as the KL divergence between the universal policy and the intermediate policy being zero.
Based on this property, we introduce a practical method to achieve the universal policy by minimizing the following loss:

$$ $\displaystyle\mathrm{min}_{\theta}\;\mathbb{E}_{(s,\omega)\sim\mathcal{D}} \left[D_{\mathrm{KL}}(\bar{\pi}_{\omega}(\cdot|s,\omega)||\pi_{\theta}(\cdot|s ,\omega))\right],$ (8) $$

where $\mathcal{D}$ is a replay buffer.
By minimizing the above loss, we can combine policies for each preference into a single universal policy.
After the training is completed, users can utilize various policies by providing preferences to the trained universal policy without additional training.

### 4.4 Convergence Analysis

In this section, we analyze the convergence properties of the proposed method.
Specifically, it can be shown that CoMOGA achieves CP optimal convergence with minor modifications if $H$ is the Fisher information matrix.
To this end, we first introduce the necessary conditions for the policy gradient method to ensure CP optimal convergence.
We then show that the modified CoMOGA satisfies these necessary conditions.
Before that, we first introduce the following assumption:

###### Assumption 4.1 (Slater’s condition) .

There exists a feasible policy $\pi_{f}$ such that $J_{C_{k}}(\pi_{f})\leq d_{k}-\eta$ $\forall k$, where $\eta>0$ is a positive number.

The Slater’s condition is a common assumption in safe RL to ensure optimal convergence [, , ].
We also assume that the state and action spaces are finite.
Now, we introduce a generalized version of the policy gradient method that guarantees optimal convergence.

###### Theorem 4.2 .

Assume that sequences $\nu^{a}_{t,i}$, $\nu^{b}_{t,i}$, $\lambda^{a}_{t,k}$, $\lambda^{b}_{t,k}\in[0,\lambda_{\mathrm{max}}]$ are given for all $i$ and $k$, where $\sum_{k}\lambda^{b}_{t,k}=1$, $\lambda^{b}_{t,k}(J_{C_{k}}(\theta_{t})-d_{k})\geq 0$, $\sum_{i}\nu^{a}_{t,i}=1$, and $\nu^{a}_{t,i}$ converges to a specific point $\bar{\nu}^{a}_{i}$. Then, if a policy is updated according to the following rule, it converges to a CP optimal policy of (1):

$$ $\theta_{t+1}=\theta_{t}+\begin{cases}\alpha_{t}F^{\dagger}(\theta_{t})\left( \sum_{i=1}^{N}\nu^{a}_{t,i}g_{i}-\alpha_{t}\sum_{k=1}^{M}\lambda^{a}_{t,k}b_{k }\right)&\text{if}\;J_{C_{k}}(\theta_{t})\leq d_{k}\;\forall k,\\ \alpha_{t}F^{\dagger}(\theta_{t})\left(\alpha_{t}\sum_{i=1}^{N}\nu^{b}_{t,i}g_ {i}-\sum_{k=1}^{M}\lambda^{b}_{t,k}b_{k}\right)&\text{otherwise},\\ \end{cases}$ (9) $$

where $g_{i}=\nabla J_{R_{i}}(\theta_{t})$, $b_{k}=\nabla J_{C_{k}}(\theta_{t})$, $\alpha_{t}$ is a step size satisfying Robbins-Monro condition [], and $F^{\dagger}$ represents the pseudo-inverse of the Fisher information matrix.

The proof is provided in Appendix B.2.
The sequences $\nu^{a}_{t,i}$, $\nu^{b}_{t,i}$, $\lambda^{a}_{t,k}$, and $\lambda^{b}_{t,k}$ in Theorem 4.2 represent the weights assigned to the gradients of the objective and constraint functions at each update step, and a new CMORL algorithm can be developed based on how the sequences are set.
Similarly, we will show that CoMOGA can be formulated as (9) by identifying these sequences with minor modifications.
Since the quadratic programming holds strong duality, we can express the solution of (5) as
$\bar{g}_{\omega}^{\mathrm{ag}}=H^{-1}(\sum_{i=1}^{N}\nu^{*}_{i}g_{i}-\sum_{k=1
}^{M}\lambda^{*}_{k}H^{-1}b_{k})$, where $\nu^{*}_{i}$ and $\lambda_{k}^{*}$ are the optimal Lagrange multipliers of the following dual problem:

$$ $\displaystyle\nu^{*},\lambda^{*}=\underset{\nu\geq 0,\lambda\geq 0}{\mathrm{ argmax}}$ $\displaystyle-\frac{1}{2}\left(\sum\nolimits_{i}\nu_{i}g_{i}-\sum\nolimits_{k} \lambda_{k}b_{k}\right)^{T}H^{-1}\left(\sum\nolimits_{i}\nu_{i}g_{i}-\sum \nolimits_{k}\lambda_{k}b_{k}\right)+\sum\nolimits_{i}\nu_{i}\omega_{i}e_{i}$ $\displaystyle+\sum\nolimits_{k}\lambda_{k}(J_{C_{k}}(\theta_{\mathrm{old}})-d_ {k}).$ $$

In case of the dual problem has no solution, we set $\nu_{i}^{*}$ and $\lambda_{k}^{*}$ to be $\bar{g}_{\omega}^{\mathrm{ag}}=0$.
For the solution of (6), the same process is applied to achieve $\lambda_{k}^{*}$.
Then, the CoMOGA is modified as follows:

$$ $\displaystyle\bar{g}_{\omega}^{\mathrm{ag}}=H^{-1}\cdot\begin{cases}\sum_{i=1} ^{N}\frac{\nu_{i}^{*}}{\sum_{j}\nu_{j}^{*}}g_{i}-\epsilon_{t}\sum_{k=1}^{M} \min(\frac{\lambda_{k}^{*}}{\epsilon_{t}\sum_{j}\nu_{j}^{*}},\lambda_{\mathrm{ max}})b_{k}&\!\!\text{if}\;J_{C_{k}}(\theta_{t})\leq d_{k}\;\forall k,\\ -\sum_{k=1}^{M}(\lambda_{k}^{*}/(\sum_{j}\lambda_{j}^{*}))b_{k}&\!\!\text{ otherwise},\\ \end{cases}$ (10) $\displaystyle g_{\omega}^{\mathrm{ag}}=\epsilon_{t}\bar{g}_{\omega}^{\mathrm{ ag}}/\min(\max(||\bar{g}_{\omega}^{\mathrm{ag}}||_{H},g_{\mathrm{min}}),g_{ \mathrm{max}}),$ $$

where $g_{\mathrm{min}}$, $g_{\mathrm{max}}$, $\lambda_{\mathrm{max}}\in\mathbb{R}_{\geq 0}$ are hyperparameters, and $\epsilon_{t}$ is the local region size satisfying Robbins-Monro condition.
Finally, we can show optimal convergence of the modified CoMOGA.

###### Theorem 4.3 .

If $H$ is the Fisher information matrix and a policy $\pi_{\theta_{t}}$ is updated as $\theta_{t+1}=\theta_{t}+g_{\omega}^{\mathrm{ag}}$, where $g_{\omega}^{\mathrm{ag}}$ is defined in (10), it converges to a CP optimal policy of (1).

Theorem 4.3 can be proved by identifying sequences $\nu_{t,i}^{a}$, $\nu_{t,k}^{b}$, $\lambda_{t,i}^{a}$, $\lambda_{t,k}^{b}$ that satisfy the conditions mentioned in Theorem 4.2, and details are provided in Appendix B.3.

###### Assumption 4.1 (Slater’s condition) .

###### Theorem 4.2 .

###### Theorem 4.3 .

## 5 Practical Implementation

We need to calculate the gradient of the objective and constraint functions to perform CoMOGA and the universal policy update.
To do that, we use reward and cost critics $V_{R,\psi_{R}}^{\pi}$, $V_{C,\psi_{C}}^{\pi}$, where $\psi_{R}$ and $\psi_{C}$ are the critic network parameters, to estimate the objectives and constraint function.
The critics are trained by minimizing the following loss functions:

$$ $\displaystyle\mathcal{L}(\psi_{R})$ $\displaystyle:=\underset{(s,a,s^{\prime},\omega)\sim\mathcal{D}}{\mathbb{E}} \left[\sum\nolimits_{i=1}^{N}(Y_{R_{i}}-V^{\pi}_{R_{i},\psi_{R}}(s,a,\omega))^ {2}\right],$ (11) $$

where the target $Y_{R_{i}}$ can be calculated either as $R_{i}(s,a,s^{\prime})+\gamma V^{\pi}_{R_{i},\psi_{R}}(s^{\prime},a^{\prime},\omega)$ with $a^{\prime}\sim\pi(\cdot|s^{\prime},\omega)$ or using Retrace($\lambda$) [], and the same process is applied to the cost critics.
Given a preference $\omega$, the objective and constraint functions can be estimated as follows:

$$ $\displaystyle J_{R_{i}}(\theta)$ $\displaystyle\approx\underset{\begin{subarray}{c}s\sim\mathcal{D},\\ a\sim\pi_{\theta}(\cdot|s,\omega)\end{subarray}}{\mathbb{E}}\left[V_{R_{i}, \psi_{R}}^{\pi}(s,a,\omega)\right],\;J_{C_{k}}(\theta)\approx\underset{\begin{ subarray}{c}s\sim\mathcal{D},\\ a\sim\pi_{\theta}(\cdot|s,\omega)\end{subarray}}{\mathbb{E}}\left[V_{C_{k}, \psi_{C}}^{\pi}(s,a,\omega)\right].$ (12) $$

Then, we can compute the policy gradients using the critic networks through the reparameterization trick, as done in the SAC paper [].
Finally, the proposed method is summarized in Algorithm 1.

Figure: Algorithm 1 Policy Update Using CoMOGA

## 6 Experiments

This section evaluates the proposed method and baselines across various tasks with and without constraints.
First, we explain how methods are evaluated on the tasks and then present the CMORL baselines.
Subsequently, each task is described, and the results are analyzed.
Finally, we conclude this section with ablation studies of the proposed method.

### 6.1 Evaluation Metrics and Baselines

Metrics.
Once the universal policy is trained, an estimated CP front, denoted as $P\subset\mathbb{R}^{N}$, can be obtained by calculating the objective values $(J_{R_{1}}(\pi),...,J_{R_{N}}(\pi))$ for a range of preferences.
Given that CMORL aims to approximate the ground truth CP front, it is essential to assess how close the estimated CP front is to the ground truth, and it can be realized by measuring the coverage and density of the estimated front.
To this end, we use the hypervolume (HV) for coverage measurement and *sparsity* (SP) for density estimation, which are commonly used in many MORL algorithms [, ].
Given an estimated CP front $P\subset\mathbb{R}^{N}$ and a reference point $r\in\mathbb{R}^{N}$, the hypervolume (HV) is defined as follows []:

$$ $\displaystyle\mathrm{HV}(P,r):=\int_{\mathbb{R}^{N}}\mathbf{1}_{Q(P,r)}(z)dz,$ (13) $$

where $Q(P,r):=\{z|\exists p\in P\;\text{s.t.}\;r\preceq z\preceq p\}$, and the metric represents the volume of the area surrounding the reference point and the estimated CP front. For a detailed explanation of this metric, including a visual description and how the reference point is set, please refer to Appendix C.2.
Sparsity (SP), on the other hand, measures how uniformly the estimated CP fronts are distributed and is defined as the average of the squared distances between elements of the CP fronts [].
However, the original definition exhibits a correlation with HV, where a larger HV corresponds to a larger SP.
To eliminate this inherent correlation, we propose to use the following normalized version of SP:

$$ $\overline{\mathrm{SP}}(P):=\frac{1}{|P|-1}\sum_{j=1}^{N}\sum_{i=1}^{|P|-1} \left(\frac{\tilde{P}_{j}[i]-\tilde{P}_{j}[i+1]}{\mathrm{max}_{k}\tilde{P}_{j} [k]-\mathrm{min}_{k}\tilde{P}_{j}[k]}\right)^{2},$ (14) $$

where $\tilde{P}_{j}:=\mathrm{Sort}(\{p[j]|\forall p\in P\})$, and $\tilde{P}_{j}[i]$ is the $i$th element in $\tilde{P}_{j}$.
By normalizing the distance using the minimum and maximum values of the CP fronts, we can remove the correlation with HV while preserving the ability to measure sparsity.

Baselines.
We aim to compare the proposed method with various existing CMORL algorithms; however, to the best of our knowledge, LP3 [] is the only existing CMORL algorithm.
To provide more comprehensive baselines, we include MORL algorithms that are based on policy gradient approaches, which can be adapted to handle constraints using the Lagrangian method.
There are two state-of-the-art policy gradient-based MORL algorithms: PD-MORL [], which is based on TD3 [], and CAPQL [], which is based on SAC [].
We have extended them to CMORL by handling constraints using the Lagrangian method.
For implementation details, please refer to Appendix C.3.1.

### 6.2 Legged Robot Locomotion

The legged robot locomotion tasks [] are to control a quadrupedal or bipedal robot to follow randomly given commands while satisfying three constraints: keeping 1) body balance, 2) CoM height, and 3) pre-defined foot contact timing.
Since the original tasks were designed to have a single objective, we have modified the tasks to have two objectives: 1) matching the current velocity with the command and 2) minimizing energy consumption.
Please see Appendix C for more details.

The evaluation results are presented in Figure 4, and the estimated CP fronts are visualized in Appendix D.
Across all tasks, CoMOGA achieves the highest HV and the lowest SP while satisfying all constraints.
Notably, CoMOGA surpasses the baselines in the bipedal task.
This suggests that preventing gradient conflicts is particularly effective for dynamically unstable robots, such as bipedal robots, because these robots require a careful balance of multiple objectives to attain desired behaviors.
LP3 shows performance similar to the proposed method in the quadrupedal task but underperforms in the bipedal task.
This may be attributed to its less effective handling of gradient conflicts.

Figure: Figure 4: Evaluation results of the legged robot locomotion tasks. The upper row shows results for the bipedal robot, while the lower row is for the quadrupedal robot. All algorithms are evaluated at every $10^{5}$ steps. The bold lines and shaded areas represent the mean and quarter-scaled standard deviation of results from five random seeds, respectively. The black dotted lines in constraint graphs indicate the thresholds.
Refer to caption: extracted/5633925/imgs/legged_performance.png

### 6.3 Safety Gymnasium

We utilize single-agent and multi-agent goal tasks in the Safety Gymnasium environments [].
These tasks require navigating robots to designated goals while avoiding obstacles.
The single-agent goal tasks have two objectives: 1) reaching goals as many times as possible within a time limit and 2) maximizing energy efficiency, along with a single constraint to avoid collisions with obstacles.
The multi-agent goal tasks have two objectives and two constraints, which are to maximize goal achievement and avoid collisions for two robots, respectively.
In both tasks, point and car robots are used.
For details on the tasks and hyperparameter settings, please refer to Appendix C.

The results are presented in Figure 5.
CoMOGA shows the highest HV in all tasks and the lowest SP in two out of four tasks, while satisfying all constraints.
Additionally, the low volatility in the evaluation graph indicates that the proposed method can stably handle both constraints and objectives.
LP3 and PD-MORL violate constraints in at least one task, and CAPQL satisfies all constraints but shows weak performance in the single agent tasks, implying the difficulty of handling constraints and objectives simultaneously.

Figure: (a) Single-agent tasks. The left shows the results for the point robot, and the right is for the car robot.
Refer to caption: extracted/5633925/imgs/safetygym_performance1.png

### 6.4 Multi-Objective Gymnasium

We conduct experiments in the Multi-Objective (MO) Gymnasium [], which is a well-known MORL environment, to examine whether the proposed method also performs well on unconstrained MORL tasks.
We use the MuJoCo tasks with continuous action spaces in the MO Gymnasium, and there are six tasks available: Hopper, Humanoid, Half-Cheetah, Walker2d, Ant, and Swimmer.
Each task has three, two, two, two, three and two objectives, respectively.
Details are provided in Appendix C.

The results are presented in Figure 6, and the proposed method achieves the highest HV in three tasks and the lowest SP in five out of six tasks.
Especially, CoMOGA exhibits outstanding performance in the hopper and humanoid tasks, which suggests that the conflict-averse strategy is effective for dynamically unstable robots even in unconstrained MORL settings.
However, all algorithms except LP3 show high HV results in the walker2d task, which is also dynamically unstable.
It can be considered that they have reached the global optimum due to the well-designed reward functions.

Figure: Figure 6: Evaluation results of the MO Gymnasium tasks. Each column shows the results of the task corresponding to its title.
Refer to caption: extracted/5633925/imgs/mogym_performance.png

### 6.5 Ablation Study

Some might suggest that conflict aversion in CMORL can be achieved by simply integrating MTL algorithms with the Lagrangian method.
To answer this question, we compare a method that combines a conflict-averse MTL algorithm, CAGrad [], and the Lagrangian approach against CoMOGA in the legged locomotion tasks.
The results are shown in Figure 7.
CAGrad+Lagrangian violates the second constraint in the quadrupedal robot, and while it satisfies all constraints in the bipedal robot, HV and SP metrics are significantly lower than CoMOGA.
These results demonstrate that the proposed method can handle both constraints and gradient conflicts effectively.

Figure: Figure 7: Results of the ablation study in the legged robot locomotion tasks.
Refer to caption: extracted/5633925/imgs/ablation.png

## 7 Conclusions and Limitations

We have introduced a CMORL algorithm called CoMOGA to maximize multiple objectives while satisfying safety constraints.
The proposed method is based on a novel transformation process that converts objectives into constraints, which enables the avoidance of gradient conflicts among multiple objectives and the handling of constraints without additional optimization variables.
We show that the proposed method converges to a CP optimal policy in tabular settings and demonstrate that it achieves outstanding performance in HV and SP metrics with constraint satisfaction through various experiments.
Since the proposed method updates the policy in a manner that prevents any objective functions from decreasing, the set of trained policies may cover a narrow portion of the CP front.
Future research can address this limitation by mitigating the condition of gradient conflicts to better balance multiple objectives and expand coverage.
Specifically, by satisfying the conditions of the proposed generalized policy update rule in Theorem 4.2, various CMORL algorithms can be developed with convergence guarantees, which can accelerate future work.

## Appendix A Toy Example

Figure: Figure 8: Landscape of the objective functions of the toy example.
Refer to caption: extracted/5633925/imgs/L1.png

In this section, we describe the details of the toy example, which is initially defined in [].
Objective functions of the toy example are formulated as follows:

$$ $\displaystyle L_{1}(x_{1},x_{2})$ $\displaystyle:=c_{1}(x_{2})f_{1}(x_{1},x_{2})+c_{2}(x_{2})g_{1}(x_{1},x_{2}),$ $\displaystyle L_{2}(x_{1},x_{2})$ $\displaystyle:=c_{1}(x_{2})f_{2}(x_{1},x_{2})+c_{2}(x_{2})g_{2}(x_{1},x_{2}), \;\text{where}$ $\displaystyle c_{1}(x)$ $\displaystyle:=\max(\mathrm{tanh}(0.5x),0),$ $\displaystyle c_{2}(x)$ $\displaystyle:=\max(\mathrm{tanh}(-0.5x),0),$ $\displaystyle f_{1}(x_{1},x_{2})$ $\displaystyle:=\log(\max(|0.5(-x_{1}-7)-\mathrm{tanh}(-x_{2})|,0.000005))+6,$ $\displaystyle f_{2}(x_{1},x_{2})$ $\displaystyle:=\log(\max(|0.5(-x_{1}+3)+\mathrm{tanh}(-x_{2}+2)|,0.000005))+6,$ $\displaystyle g_{1}(x_{1},x_{2})$ $\displaystyle:=((-x_{1}+7)^{2}+0.1(-x_{2}-8)^{2})/10-20,$ $\displaystyle g_{2}(x_{1},x_{2})$ $\displaystyle:=((-x_{1}-7)^{2}+0.1(-x_{2}-8)^{2})/10-20,$ $$

where $L_{1}$ and $L_{2}$ are the objective functions.
The landscape of the objective functions are presented in Figure 8.
In addition, we define a constraint function, which is defined as follows:

$$ $C(x_{1},x_{2}):=x_{1}^{2}+0.3(x_{2}-10)^{2}-10.5^{2}.$ $$

Finally, the toy example is formulated as follows:

$$ $\min_{x_{1},x_{2}}L_{1}(x_{1},x_{2})\;\text{and}\;L_{2}(x_{1},x_{2})\quad \textbf{s.t.}\;C(x_{1},x_{2})\leq 0.$ $$

We set the preference as $(0.5,0.5)$ for both algorithms, LS and CoMOGA, and use the following four initial points: $(-10,0)$, $(-10,7.5)$, $(0,7.5)$, and $(10,10)$.
The optimization trajectories for each method are shown in Figure 2.
For the initial point $(-10,7.5)$, the gradient of $L_{1}$ is much larger than $L_{2}$.
Thus, addressing the toy example with LS can be significantly influenced by $L_{1}$ and eventually converges to a local optimal point of $L_{1}$.
To resolve this issue, avoiding gradient conflicts can prevent policy gradients from being dominated by a specific objective, which helps them converge to an optimal point.

## Appendix B Proofs

### B.1 Proof of Transformation Process

In this section, we prove that the solutions of (2) and (3) are equivalent.
Under the linear assumption, we can express the $i$th objective function as $g_{i}^{T}\Delta\theta+J_{R_{i}}(\theta_{\mathrm{old}})$.
Then, (2) can be rewritten as follows:

$$ $\max_{\Delta\theta}\;g_{i}^{T}\Delta\theta+J_{R_{i}}(\theta_{\mathrm{old}})\; \;\mathrm{s.t.}\;\Delta\theta^{T}H\Delta\theta\leq\epsilon^{2}.$ $$

As the strong duality holds, we will find the solution by solving the following Lagrange dual problem:

$$ $\max_{\lambda\geq 0}\min_{\Delta\theta}\;-g_{i}^{T}\Delta\theta+\lambda(\Delta \theta^{T}H\Delta\theta-\epsilon^{2})=:L(\Delta\theta,\lambda).$ $$

Then, $\Delta\theta^{*}(\lambda)=\mathrm{argmin}_{\Delta\theta}L(\Delta\theta,\lambda
)=H^{-1}g_{i}/(2\lambda)$.
By replacing $\Delta\theta$ with $\Delta\theta^{*}(\lambda)$ in the dual problem, we get $\lambda^{*}=\mathrm{argmax}_{\lambda}L(\Delta\theta^{*}(\lambda),\lambda)=
\sqrt{g_{i}^{T}H^{-1}g_{i}}/(2\epsilon)$.
Finally, we obtain the solution, $\Delta\theta^{*}=\Delta\theta^{*}(\lambda^{*})=\epsilon H^{-1}g_{i}/\sqrt{g_{i
}^{T}H^{-1}g_{i}}$.

Now let us find the solution of (3).
Under the linear assumption, we can rewrite (3) as follows:

$$ $\min_{\Delta\theta}\;\Delta\theta^{T}H\Delta\theta\;\;\mathrm{s.t.}\;e_{i}\leq g _{i}^{T}\Delta\theta.$ (15) $$

Similar to the above process, the Lagrange dual problem is derived as follows:

$$ $\max_{\lambda\geq 0}\min_{\Delta\theta}\;\Delta\theta^{T}H\Delta\theta+\lambda (e_{i}-g_{i}^{T}\Delta\theta)=:L(\Delta\theta,\lambda).$ (16) $$

Then, the solution of the dual problem is obtained as $\Delta\theta^{*}(\lambda)=\mathrm{argmin}_{\Delta\theta}L(\Delta\theta,\lambda
)=\lambda H^{-1}g_{i}/2$.
Using $\Delta\theta^{*}(\lambda)$, we can get the optimal Lagrange multiplier as $\lambda^{*}=\mathrm{argmax}_{\lambda}L(\Delta\theta^{*}(\lambda),\lambda)=2e_{
i}/(g_{i}H^{-1}g_{i})=2\epsilon/\sqrt{g_{i}H^{-1}g_{i}}$.
Then, the solution is $\Delta\theta^{*}=\Delta\theta^{*}(\lambda^{*})=\epsilon H^{-1}g_{i}/\sqrt{g_{i
}^{T}H^{-1}g_{i}}$.
As a result, the solutions of (2) and (3) are the same as $\epsilon H^{-1}g_{i}/\sqrt{g_{i}^{T}H^{-1}g_{i}}$.

### B.2 Proof of Theorem 4.2

We first define the following notions for a policy $\pi$:

$$ $\displaystyle d_{\rho}^{\pi}(s)$ $\displaystyle:=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\mathrm{Pr}(s_{t}=s)\Big {|}s_{0}\sim\rho,a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P(\cdot|s_{t},a_{t})\; \forall t,$ $\displaystyle V^{\pi}(s)$ $\displaystyle:=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t +1})\Big{|}s_{0}=s,a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P(\cdot|s_{t},a_{t}) \;\forall t\right],$ $\displaystyle Q^{\pi}(s,a)$ $\displaystyle:=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t +1})\Big{|}s_{0}=s,a_{0}=a,s_{t+1}\sim P(\cdot|s_{t},a_{t}),a_{t+1}\sim\pi( \cdot|s_{t+1})\;\forall t\right],$ $\displaystyle A^{\pi}(s,a)$ $\displaystyle:=Q^{\pi}(s,a)-V^{\pi}(s).$ $$

We also simplify several notations for brevity as follows:

$$ $\pi_{\theta_{t}}\rightarrow\pi_{t},\;d_{\rho}^{\pi_{\theta_{t}}}\rightarrow d_ {\rho}^{t},\;A_{R_{i}}^{\pi_{\theta_{t}}}(s,a)\rightarrow A_{R_{i}}^{t}(s,a), \;A_{C_{k}}^{\pi_{\theta_{t}}}(s,a)\rightarrow A_{C_{k}}^{t}(s,a).$ $$

Since it is assumed that the state and action spaces are finite, a policy can be parameterized using a softmax parameterization as follows []:

$$ $\pi_{\theta}(a|s):=\exp(\theta(s,a))\Big{/}\sum_{a^{\prime}\in A}\exp(\theta(s ,a^{\prime})),$ $$

where $\theta\in\mathbb{R}^{S\times A}$ is a trainable parameter, and $\theta(s,a)$ denotes the value corresponding to state $s$ and action $a$.
Through Lemma 4 in [], the following equations are satisfied when a policy is updated according to (9):

$$ $\displaystyle\theta_{t+1}=\theta_{t}(s,a)+\frac{\alpha_{t}}{1-\gamma}\cdot \begin{cases}(\sum_{i=1}^{N}\nu_{t,i}^{a}A_{R_{i}}^{t}-\alpha_{t}\sum_{k=1}^{M }\lambda_{t,k}^{a}A_{C_{k}}^{t})&\text{if}\;J_{C_{k}}(\theta_{t})\leq d_{k}\; \forall k,\\ (\alpha_{t}\sum_{i=1}^{N}\nu_{t,i}^{b}A_{R_{i}}^{t}-\sum_{k=1}^{M}\lambda_{t,k }^{b}A_{C_{k}}^{t})&\text{otherwise},\end{cases}$ $\displaystyle\pi_{t+1}(a|s)=\pi_{t}(a|s)\frac{\exp(\theta_{t+1}(s,a)-\theta_{t }(s,a))}{Z_{t}(s)},$ $\displaystyle\text{where}\;Z_{t}(s):=\sum_{a\in A}\pi_{t}(a|s)\exp(\theta_{t+1 }(s,a)-\theta_{t}(s,a)).$ $$

Now, we introduce a lemma showing that the LS approach guarantees to converge a CP optimal policy, followed by the proof of Theorem 4.3.

###### Lemma B.1 .

Given a preference $\omega$, let us define an LS optimal policy as follows:

$$ $\pi_{\omega}^{*}:=\underset{\pi}{\arg\max}\sum_{i=1}^{N}\omega_{i}J_{R_{i}}( \pi)\quad\mathbf{s.t.}\;J_{C_{k}}(\pi)\leq d_{k}\;\forall k.$ $$

Then, the LS optimal policy is also a CP optimal policy.

###### Proof.

Let us assume that there exists a policy $\mu$ that dominates the LS optimal policy $\pi_{\omega}^{*}$.
By definition of the constrained dominance, the following is satisfied:

$$ $J_{R_{i}}(\mu)>J_{R_{i}}(\pi_{\omega}^{*})\;\forall i\;\text{and}\;J_{C_{k}}( \mu)\leq d_{k}\;\forall k.\Rightarrow\sum_{i=1}^{N}\omega_{i}(J_{R_{i}}(\mu)-J _{R_{i}}(\pi_{\omega}^{*}))>0.$ $$

However, this equation contradicts the fact that $\pi_{\omega}^{*}$ is an LS optimal policy.
Consequently, $\pi_{\omega}^{*}$ is not dominated by any others, it is also a CP optimal policy.
∎

See 4.2

###### Proof.

We first consider the case $J_{C_{k}}(\theta_{t})\leq d_{k}\;\forall k$, in which the policy is updated as follows:

$$ $\theta_{t+1}=\theta_{t}+\frac{\alpha_{t}}{1-\gamma}(\sum_{i=1}^{N}\nu_{t,i}^{a }A_{R_{i}}^{t}-\alpha_{t}\sum_{k=1}^{M}\lambda_{t,k}^{a}A_{C_{k}}^{t}).$ $$

Using Lemma 1 in [], we can derive the following inequality:

$$ $\displaystyle\sum_{i}\nu_{t,i}^{a}(J_{R_{i}}(\theta_{t+1})-J_{R_{i}}(\theta_{t }))-\alpha_{t}\sum_{k}\lambda_{t,k}^{a}(J_{C_{k}}(\theta_{t+1})-J_{C_{k}}( \theta_{t}))$ (17) $\displaystyle=\frac{1}{1-\gamma}\mathbb{E}_{s\sim d_{\rho}^{t+1}}\left[\mathbb {E}_{a\sim\pi_{t+1}(\cdot|s)}\left[\sum_{i=1}^{N}\nu_{t,i}^{a}A_{R_{i}}^{t}(s, a)-\alpha_{t}\sum_{k=1}^{M}\lambda_{t,k}^{a}A_{C_{k}}^{t}(s,a)\right]\right]$ $\displaystyle=\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{t+1}}\left[ \mathbb{E}_{a\sim\pi_{t+1}(\cdot|s)}\left[\theta_{t+1}(s,a)-\theta_{t}(s,a) \right]\right]$ $\displaystyle=\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{t+1}}\left[ \mathbb{E}_{a\sim\pi_{t+1}(\cdot|s)}\left[\log\frac{\pi_{t+1}(a|s)Z_{t}(s)}{ \pi_{t}(a|s)}\right]\right]$ $\displaystyle=\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{t+1}}\left[D_{ \mathrm{KL}}(\pi_{t+1}(\cdot|s)||\pi_{t}(\cdot|s))+\log Z_{t}(s)\right]$ $\displaystyle\geq\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{t+1}}\left[ \log Z_{t}(s)\right]\overset{\text{(i)}}{\geq}\frac{1-\gamma}{\alpha_{t}} \mathbb{E}_{s\sim\rho}\left[\log Z_{t}(s)\right],$ $$

where (i) follows from the fact that $d_{\rho}^{\pi}=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\mathrm{Pr}(s_{t}=s)\geq
(1-\gamma)\rho(s)$.
By Lemma 5 in [],

$$ $\displaystyle|J_{R_{i}}(\theta_{t+1})-J_{R_{i}}(\theta_{t})|$ $\displaystyle\leq\frac{2R_{\mathrm{max}}}{1-\gamma}||\theta_{t+1}-\theta_{t}|| _{2}\;\forall i,$ $\displaystyle|J_{C_{k}}(\theta_{t+1})-J_{C_{k}}(\theta_{t})|$ $\displaystyle\leq\frac{2R_{\mathrm{max}}}{1-\gamma}||\theta_{t+1}-\theta_{t}|| _{2}\;\forall k.$ $$

$$ $\displaystyle\Rightarrow$ $\displaystyle\Big{|}\sum_{i}\nu_{t,i}^{a}(J_{R_{i}}(\theta_{t+1})-J_{R_{i}}( \theta_{t}))-\alpha_{t}\sum_{k}\lambda_{t,k}^{a}(J_{C_{k}}(\theta_{t+1})-J_{C_ {k}}(\theta_{t}))\Big{|}$ $\displaystyle\leq\frac{2R_{\mathrm{max}}}{1-\gamma}(N+\alpha_{t}M)\lambda_{ \max}||\theta_{t+1}-\theta_{t}||_{2}$ $\displaystyle=\frac{2R_{\mathrm{max}}}{1-\gamma}(N+\alpha_{t}M)\lambda_{\max} \Big{|}\Big{|}\frac{\alpha_{t}}{1-\gamma}(\sum_{i=1}^{N}\nu_{t,i}^{a}A_{R_{i}} ^{t}-\alpha_{t}\sum_{k=1}^{M}\lambda_{t,k}^{a}A_{C_{k}}^{t})\Big{|}\Big{|}_{2}$ $\displaystyle\leq\frac{2\alpha_{t}R^{2}_{\mathrm{max}}}{(1-\gamma)^{3}}(N+ \alpha_{t}M)^{2}\lambda_{\max}^{2}\sqrt{|S||A|}.$ $$

Then, (17) can be rewritten as follows:

$$ $\frac{2\alpha_{t}R^{2}_{\mathrm{max}}}{(1-\gamma)^{3}}(N+\alpha_{t}M)^{2} \lambda_{\max}^{2}\sqrt{|S||A|}\geq\frac{1-\gamma}{\alpha_{t}}\mathbb{E}_{s \sim\rho}\left[\log Z_{t}(s)\right],$ (18) $$

which is satisfied for any $\rho$.
Finally, we derive the following inequality for a policy $\mu$ as follows:

$$ $\displaystyle\sum_{i}\nu_{t,i}^{a}(J_{R_{i}}(\mu)-J_{R_{i}}(\theta_{t}))- \alpha_{t}\sum_{k}\lambda_{t,k}^{a}(J_{C_{k}}(\mu)-J_{C_{k}}(\theta_{t}))$ (19) $\displaystyle=\frac{1}{1-\gamma}\mathbb{E}_{s\sim d_{\rho}^{\mu}}\left[\mathbb {E}_{a\sim\mu(\cdot|s)}\left[\sum_{i=1}^{N}\nu_{t,i}^{a}A_{R_{i}}^{t}(s,a)- \alpha_{t}\sum_{k=1}^{M}\lambda_{t,k}^{a}A_{C_{k}}^{t}(s,a)\right]\right]$ $\displaystyle=\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{\mu}}\left[ \mathbb{E}_{a\sim\mu(\cdot|s)}\left[\log\frac{\pi_{t+1}(a|s)Z_{t}(s)}{\pi_{t}( a|s)}\right]\right]$ $\displaystyle=\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d_{\rho}^{\mu}}\left[D_{ \mathrm{KL}}(\mu(\cdot|s)||\pi_{t}(\cdot|s))-D_{\mathrm{KL}}(\mu(\cdot|s)||\pi _{t+1}(\cdot|s))+\log Z_{t}(s)\right]$ $\displaystyle\overset{\text{(i)}}{\leq}\frac{1}{\alpha_{t}}\mathbb{E}_{s\sim d _{\rho}^{\mu}}\left[D_{\mathrm{KL}}(\mu(\cdot|s)||\pi_{t}(\cdot|s))-D_{\mathrm {KL}}(\mu(\cdot|s)||\pi_{t+1}(\cdot|s))\right]$ $\displaystyle\quad+\frac{2\alpha_{t}R^{2}_{\mathrm{max}}}{(1-\gamma)^{3}}(N+ \alpha_{t}M)^{2}\lambda_{\max}^{2}\sqrt{|S||A|},$ $$

where (i) is from (18) by replacing $\rho$ with $d_{\rho}^{\mu}$.
For brevity, we denote $\mathbb{E}_{s\sim d_{\rho}^{\mu}}\left[D_{\mathrm{KL}}(\mu(\cdot|s)||\pi_{t}(
\cdot|s))\right]$ as $D_{\mathrm{KL}}(\mu||\pi_{t})$.
Now, we consider the second case where the constraints are violated.
In this case,

$$ $\theta_{t+1}=\theta_{t}+\frac{\alpha_{t}}{1-\gamma}(\alpha_{t}\sum_{i=1}^{N} \nu_{t,i}^{b}A_{R_{i}}^{t}-\sum_{k=1}^{M}\lambda_{t,k}^{b}A_{C_{k}}^{t}),$ $$

which is symmetrical to the first case, achieved by replacing $\nu_{t,i}^{a}\rightarrow\alpha_{t}\nu_{t,i}^{b}$ and $\alpha_{t}\lambda_{t,k}^{a}\rightarrow\lambda_{t,k}^{b}$.
Using this symmetry property, we can obtain the following inequality from (19):

$$ $\displaystyle\alpha_{t}\sum_{i}\nu_{t,i}^{b}(J_{R_{i}}(\mu)-J_{R_{i}}(\theta_{ t}))-\sum_{k}\lambda_{t,k}^{b}(J_{C_{k}}(\mu)-J_{C_{k}}(\theta_{t}))$ (20) $\displaystyle\leq\frac{1}{\alpha_{t}}(D_{\mathrm{KL}}(\mu||\pi_{t})-D_{\mathrm {KL}}(\mu||\pi_{t+1}))+\frac{2\alpha_{t}R^{2}_{\mathrm{max}}}{(1-\gamma)^{3}}( \alpha_{t}N+M)^{2}\lambda_{\max}^{2}\sqrt{|S||A|}.$ $$

Now, we define a set of time steps, $\mathcal{N}:=\{t|J_{C_{k}}(\pi_{t})\leq d_{k}\;\forall k\}$.
Then, by summing (19) and (20) over several time steps, the following is satisfied for a policy $\mu$:

$$ $\displaystyle\sum_{t\in\mathcal{N}}\left(\alpha_{t}\sum_{i}\nu_{t,i}^{a}(J_{R_ {i}}(\mu)-J_{R_{i}}(\theta_{t}))-\alpha_{t}^{2}\sum_{k}\lambda_{t,k}^{a}(J_{C_ {k}}(\mu)-J_{C_{k}}(\theta_{t}))\right)$ (21) $\displaystyle+\sum_{t\notin\mathcal{N}}\left(\alpha_{t}^{2}\sum_{i}\nu_{t,i}^{ b}(J_{R_{i}}(\mu)-J_{R_{i}}(\theta_{t}))-\alpha_{t}\sum_{k}\lambda_{t,k}^{b}(J _{C_{k}}(\mu)-J_{C_{k}}(\theta_{t}))\right)$ $\displaystyle\leq D_{\mathrm{KL}}(\mu||\pi_{0})+\sum_{t}\alpha_{t}^{2} \underbrace{\frac{2R^{2}_{\mathrm{max}}}{(1-\gamma)^{3}}\max(\alpha_{0},1)^{2} (N+M)^{2}\lambda_{\max}^{2}\sqrt{|S||A|}}_{=:\kappa_{1}}.$ $$

If $t\notin\mathcal{N}$, $\lambda_{t,k}^{b}=0$ for $J_{C_{k}}(\theta_{t})\leq d_{k}$, which results in $\sum_{k}\lambda_{t,k}^{b}(J_{C_{k}}(\pi_{f})-J_{C_{k}}(\theta_{t}))\leq-\eta$.
By using this fact and replacing $\mu$ in (21) with $\pi_{f}$,

$$ $\displaystyle\sum_{t\in\mathcal{N}}\left(\alpha_{t}\sum_{i}\nu_{t,i}^{a}(J_{R_ {i}}(\pi_{f})-J_{R_{i}}(\theta_{t}))-\alpha_{t}^{2}\sum_{k}\lambda_{t,k}^{a}(J _{C_{k}}(\pi_{f})-J_{C_{k}}(\theta_{t}))\right)$ $\displaystyle+\sum_{t\notin\mathcal{N}}\left(\alpha_{t}^{2}\sum_{i}\nu_{t,i}^{ b}(J_{R_{i}}(\pi_{f})-J_{R_{i}}(\theta_{t}))+\alpha_{t}\eta\right)\leq D_{ \mathrm{KL}}(\pi_{f}||\pi_{0})+\kappa_{1}\sum_{t}\alpha_{t}^{2}.$ $$

$$ $\displaystyle\Rightarrow\sum_{t\in\mathcal{N}}\left(\alpha_{t}\sum_{i}\nu_{t,i }^{a}(J_{R_{i}}(\pi_{f})-J_{R_{i}}(\theta_{t}))\right)+\sum_{t\notin\mathcal{N }}\alpha_{t}\eta$ $\displaystyle\leq D_{\mathrm{KL}}(\pi_{f}||\pi_{0})+\kappa_{1}\sum_{t}\alpha_{ t}^{2}+\sum_{t\in\mathcal{N}}\alpha_{t}^{2}\sum_{k}\lambda_{t,k}^{a}(J_{C_{k}} (\pi_{f})-J_{C_{k}}(\theta_{t}))$ $\displaystyle\quad-\sum_{t\in\mathcal{N}}\alpha_{t}^{2}\sum_{i}\nu_{t,i}^{b}(J _{R_{i}}(\pi_{f})-J_{R_{i}}(\theta_{t}))$ $\displaystyle\leq D_{\mathrm{KL}}(\pi_{f}||\pi_{0})+\sum_{t}\alpha_{t}^{2} \underbrace{\left(\kappa_{1}+\frac{2R_{\mathrm{max}}\lambda_{\mathrm{max}}}{1- \gamma}\max(N,M)\right)}_{=:\kappa_{2}}.$ $$

Since $\sum_{t\notin\mathcal{N}}\alpha_{t}\eta=\sum_{t}\alpha_{t}\eta-\sum_{t\in
\mathcal{N}}\alpha_{t}\eta$,

$$ $\displaystyle\sum_{t\in\mathcal{N}}\left(\alpha_{t}\left(\sum_{i}\nu_{t,i}^{a} (J_{R_{i}}(\pi_{f})-J_{R_{i}}(\theta_{t}))-\eta\right)\right)+\sum_{t}\alpha_{ t}\eta\leq D_{\mathrm{KL}}(\pi_{f}||\pi_{0})+\sum_{t}\alpha_{t}^{2}\kappa_{2}.$ $$

Due to the Robbins-Monro condition, the right term converges to real number.
Since $\sum_{t}\alpha_{t}\eta=\infty$ in the left term, the following must be satisfied:

$$ $\sum_{t\in\mathcal{N}}\left(\alpha_{t}\left(\sum_{i}\nu_{t,i}^{a}(J_{R_{i}}( \pi_{f})-J_{R_{i}}(\theta_{t}))-\eta\right)\right)=-\infty,$ $$

which results in $\sum_{t\in\mathcal{N}}\alpha_{t}=\infty$.
Now, let us define a preference $\omega$, where $\omega_{i}:=\bar{\nu}^{a}_{i}$.
By replacing $\mu$ in (21) with $\pi_{\omega}^{*}$,

$$ $\displaystyle\sum_{t\in\mathcal{N}}\left(\alpha_{t}\sum_{i}\nu_{t,i}^{a}(J_{R_ {i}}(\pi_{\omega}^{*})-J_{R_{i}}(\theta_{t}))\right)-\sum_{t\notin\mathcal{N}} \left(\alpha_{t}\sum_{k}\lambda_{t,k}^{b}(J_{C_{k}}(\pi_{\omega}^{*})-J_{C_{k} }(\theta_{t}))\right)$ $\displaystyle\leq D_{\mathrm{KL}}(\pi_{\omega}^{*}||\pi_{0})+\kappa_{2}\sum_{t }\alpha_{t}^{2}.$ $$

$$ $\displaystyle\Rightarrow\sum_{t\in\mathcal{N}}\left(\alpha_{t}\sum_{i}\nu_{t,i }^{a}(J_{R_{i}}(\pi_{\omega}^{*})-J_{R_{i}}(\theta_{t}))\right)\leq D_{\mathrm {KL}}(\pi_{\omega}^{*}||\pi_{0})+\kappa_{2}\sum_{t}\alpha_{t}^{2}.$ $$

Since the right term converges to a real number and $\sum_{t\in\mathcal{N}}\alpha_{t}=\infty$, the following must be satisfied:

$$ $\lim_{t\to\infty}\sum_{i}\nu_{t,i}^{a}(J_{R_{i}}(\pi_{\omega}^{*})-J_{R_{i}}( \theta_{\mathcal{N}_{t}}))=0,$ $$

where $\mathcal{N}_{t}$ is the $t$th element of $\mathcal{N}$.
Consequently,

$$ $\sum_{i}\bar{\nu}_{i}^{a}J_{R_{i}}(\pi_{\omega}^{*})=\sum_{i}\bar{\nu}_{i}^{a} \lim_{t\to\infty}J_{R_{i}}(\pi_{\mathcal{N}_{t}}),$ $$

which means that $\lim_{t\to\infty}\pi_{\mathcal{N}_{t}}$ is also a LS optimal policy for the preference $\omega$.
Due to Lemma B.1, the policy converges to a CP optimal policy.
∎

###### Lemma B.1 .

###### Proof.

###### Proof.

### B.3 Proof of Theorem 4.3

See 4.3

###### Proof.

This theorem can be proved by identifying sequences $\nu_{t,i}^{a},\nu_{t,i}^{b},\lambda_{t,k}^{a},\lambda_{t,k}^{b}$ that satisfy all conditions mentioned in Theorem 4.2.
By examining (10), $\nu_{t,i}^{a}$, $\nu_{t,i}^{b}$, $\lambda_{t,k}^{a}$, $\lambda_{t,k}^{b}$, and $\alpha_{t}$ can be deduced as follows:

$$ $\displaystyle\nu_{t,i}^{a}=\frac{\nu^{*}_{i}}{\sum_{j=1}^{N}\nu^{*}_{j}},\nu_{ t,i}^{b}=0,\lambda_{t,k}^{b}=\frac{\lambda^{*}_{k}}{\sum_{j=1}^{M}\lambda^{*}_ {j}},$ $\displaystyle\lambda_{t,k}^{a}=\min(\max(||\bar{g}_{\omega}^{\mathrm{ag}}||,g_ {\mathrm{min}}),g_{\mathrm{max}})\cdot\min\left(\frac{\lambda_{k}^{*}}{ \epsilon_{t}\sum_{i=1}^{N}\nu_{i}^{*}},\lambda_{\mathrm{max}}\right),$ $\displaystyle\alpha_{t}=\frac{\epsilon_{t}}{\min(\max(||\bar{g}_{\omega}^{ \mathrm{ag}}||,g_{\mathrm{min}}),g_{\mathrm{max}})}.$ $$

Substituting the deduced sequences into (10), we can obtain the same formulation as (9).
The deduced sequences satisfy that $\sum\nu_{t,i}^{a}=1$, $\sum\lambda_{t,k}^{b}=1$, $\nu_{t,i}^{b}\in\mathbb{R}_{\geq 0}$ is bounded, $0\leq\lambda_{t,k}^{a}\leq g_{\mathrm{max}}\lambda_{\mathrm{max}}$ is also bounded, and $\alpha_{t}\in[\epsilon_{t}/g_{\mathrm{max}},\epsilon_{t}/g_{\mathrm{min}}]$ follows the Robbins-Monro condition.
Additionally, since $\lambda_{k}^{*}$ is an optimal point of the dual problem, it satisfies the KKT condition, which results in $\lambda_{k}^{*}=0$ if $J_{C_{k}}(\theta_{t})<d_{k}$.
Consequently, $\lambda_{t,k}^{b}(J_{C_{k}}(\theta_{t})-d_{k})\geq 0$.
Now, we need to check whether the $\nu_{t,i}^{a}$ converges to a real number.
Given that the learning rate follows the Robbins-Monro condition, the policy converges to a specific policy.
Since (5) becomes invariant at this point of convergence, the solution of the dual problem is also fixed.
Consequently, $\nu_{t,i}^{a}$ converge to a specific value.
∎

###### Proof.

## Appendix C Experimental Details

In this section, we describe the details of each task, including Safety Gymnasium [], legged robot locomotion [], and Multi-Objective Gymnasium []. We then provide a detailed explanation of the performance metrics named hypervolume and sparsity.
Finally, we present implementation details, such as network architectures and hyperparameter settings.

### C.1 Task Details

#### C.1.1 Safety Gymnasium

Figure: (a) Single-Point-Goal
Refer to caption: extracted/5633925/imgs/single_point.png

In the Safety Gymnasium [], we utilize the single-agent safe navigation tasks: SafetyPointGoal1-v0 and SafetyCarGoal1-v0, and the multi-agent safe navigation tasks: SafetyPointMultiGoal1-v0 and SafetyCarMultiGoal1-v0.
Snapshots of each task are shown in Figure 9.

In the single-agent tasks, the observation space includes velocity, acceleration, and Lidar of the goal and hazards, and the dimensions are 60 and 72 for the point and car robots, respectively.
The action space for both robots is two-dimensional.
There are eight hazard areas and one goal, and hazard areas are randomly spawned at the beginning of each episode, while goals are randomly placed whenever the agent reaches the current goal.
These tasks originally had a single objective: maximizing the number of goals reached.
However, in order to have multiple objectives, we have modified them to include a second objective: minimizing energy consumption.
Therefore, the reward is two-dimensional, and we use the original definition for the first reward (please refer to Ji et al. []).
The second reward is defined as follows:

$$ $R_{2}(s,a,s^{\prime}):=-\frac{1}{|A|}\sum_{i=1}^{|A|}(a_{i}/10)^{2}.$ (22) $$

Also, we use the original definition of the cost function, which gives one if the agent enters hazardous areas and zero otherwise.

The multi-agent tasks have two agents and two goals, and we need to control both agents to reach goals as much as possible while avoiding hazardous areas.
The original implementation uses a dictionary type for observations and actions, so we have modified them to be an array type.
Additionally, in the original setup, each goal is pre-assigned to a specific agent, reducing the difficulty of solving multi-agent tasks.
Therefore, we have modified the implementation so that each goal is not pre-assigned, allowing each agent to compete to achieve goals.
The observation space contains the same information from the single-agent tasks for the two agents, as well as additional information: Lidar of the first and second goals.
The dimensions of observation space are then 152 and 176 for the point and car robots, respectively.
The action space is four-dimensional, which is doubled from the single-agent tasks to control both agents.
These tasks have two objectives: maximizing the number of goals reached by each agent.
Also, there are two constraints: avoiding hazardous areas for each agent.
Since there are no additional objectives or constraints, the original definitions of the reward and cost functions are used without modification.

#### C.1.2 Legged Robot Locomotion

Figure: (a) Quadruped
Refer to caption: extracted/5633925/imgs/quadruped.png

The legged robot locomotion tasks [] aim to control bipedal or quadrupedal robots so that their velocity matches a randomly sampled command.
This command specifies the target linear velocities in the x and y-axis directions
and the target angular velocity in the z-axis direction, denoted by ($v_{x}^{\mathrm{cmd}},v_{y}^{\mathrm{cmd}},\omega_{z}^{\mathrm{cmd}}$).
Snapshots of these tasks are shown in Figure 10.
The quadrupedal robot has 12 joints and 12 motors, and the bipedal robot has 14 joints and 10 motors.
Each robot is operated by a PD controller that follows the target position of the motors, and the target position is given as an action.
Hence, the number of motors corresponds to the dimension of action space.
In order to provide enough information for stable control, observations include the command, linear and angular velocities of the robot base, and the position and velocity of each joint.
As a result, the dimensions of the observation space are 160 for the quadruped and 132 for the biped.
Originally, these tasks have a single objective of following a given command and three constraints: 1) maintaining body balance, 2) keeping the height of CoM (center of mass) above a certain level, and 3) adhering to pre-defined foot contact timing.
Therefore, we use the original implementation for the observation, action, and cost functions but modify the reward function to have multiple objectives.
The modified version has two objectives: 1) reducing the difference between the current velocity and the command and 2) minimizing energy consumption.
Then, the reward function is defined as follows:

$$ $\displaystyle R_{1}(s,a,s^{\prime}):=1-(v_{x}^{\mathrm{base}}-v_{x}^{\mathrm{ cmd}})^{2}-(v_{y}^{\mathrm{base}}-v_{y}^{\mathrm{cmd}})^{2}-(\omega_{z}^{ \mathrm{base}}-\omega_{z}^{\mathrm{cmd}})^{2},$ (23) $\displaystyle R_{2}(s,a,s^{\prime}):=1-\frac{1}{J}\sum_{j=1}^{J}\left(\frac{ \tau_{j}}{M_{\mathrm{robot}}}\right)^{2},$ $$

where $J$ is the number of joints, $\tau_{j}$ is the torque applied to the $j$th joint, and $M_{\mathrm{robot}}$ is the mass of the robot.
However, in the simulation of the bipedal robot, obtaining valid torque information is difficult due to the presence of closed loops in the joint configuration.
To address this issue, we modify the second reward for the bipedal robot to penalize the action instead of the joint torque as follows:

$$ $R_{2}(s,a,s^{\prime}):=1-\frac{1}{|A|}\sum_{i=1}^{|A|}a_{i}^{2}.$ (24) $$

#### C.1.3 Multi-Objective Gymnasium

**Table 1: Specifications of MO-Gymnasium tasks.**
|  | Observation Space | Action Space | # of Objectives (Entries) |
| --- | --- | --- | --- |
| Half-Cheetah | $\subseteq\mathbb{R}^{17}$ | $\subseteq\mathbb{R}^{6}$ | 2 (velocity, energy) |
| Hopper | $\subseteq\mathbb{R}^{11}$ | $\subseteq\mathbb{R}^{3}$ | 3 (velocity, height, energy) |
| Humanoid | $\subseteq\mathbb{R}^{376}$ | $\subseteq\mathbb{R}^{17}$ | 2 (velocity, energy) |
| Ant | $\subseteq\mathbb{R}^{27}$ | $\subseteq\mathbb{R}^{8}$ | 3 ($x$-velocity, $y$-velocity, energy) |
| Walker2d | $\subseteq\mathbb{R}^{17}$ | $\subseteq\mathbb{R}^{6}$ | 2 (velocity, energy) |
| Swimmer | $\subseteq\mathbb{R}^{8}$ | $\subseteq\mathbb{R}^{2}$ | 2 (velocity, energy) |

We utilize the Multi-Objective (MO) Gymnasium [] to evaluate the proposed method in MORL tasks which have no constraints.
Among several tasks, we use the MuJoCo tasks with continuous action space, and there are six tasks available: mo-hopper-v4, mo-humanoid-v4, mo-halfcheetah-v4, mo-ant-v4, mo-walker2d-v4, and mo-swimmer-v4.
Description of the observation space, action space, and the number of objectives for these tasks are summarized in Table 1, and for more details, please refer to Felten et al. [].

### C.2 Metric Details

Figure: (a) Hypervolume
Refer to caption: extracted/5633925/imgs/hypervolume.png

In this section, we explain in detail the performance metrics used in the main text: hypervolume (HV) and normalized sparsity ($\overline{\mathrm{SP}}$).
First, we need to prepare an estimated CP front to calculate such metrics.
To do this, we pre-sample a fixed number of equally spaced preferences and roll out the trained universal policy for each pre-sampled preference to estimate objective and constraint values.
If the constraint values do not exceed the thresholds, we store the objective values, $(J_{R_{1}}(\theta),...,J_{R_{N}}(\theta))$, in a set.
Once this process is completed for all preferences, we can construct an estimated CP front by extracting elements from the collected set that are not dominated by any others.
Now, we introduce details on each metric.

HV is the volume of the area surrounding a given reference point, $r$, and an estimated CP front, $P$, as defined in (13) and is visualized in Figure 11(a).
However, as seen in Figure 11(a) or (13), the metric value varies depending on what reference point is used.
Therefore, we do not set the reference point arbitrarily but use a method of determining the reference point from the results of all algorithms, as shown in Figure 11(b).
For a detailed explanation, suppose that we have a set of estimated CP fronts, $\{P_{i}\}_{i=1}^{K}$, where $P_{i}$ is the estimated front obtained from the $i$th algorithm.
Then, we obtain the union of $\{P_{i}\}_{i=1}^{K}$, denoted by $P^{\prime}$, and extract elements that are not dominated by any others in $P^{\prime}$ to find the Pareto front of the entire set, $\mathrm{PF}(P^{\prime})$, whose elements are circled in red in Figure 11(b).
Finally, we get the reference point whose $i$th value is $\mathrm{min}(\{p_{i}|\forall p\in\mathrm{PF}(P^{\prime})\})$.
Through this process, the reference point can be automatically set after obtaining results from all algorithms for each task.
In addition, the HV value can be zero for some algorithms whose elements are entirely dominated by $\mathrm{PF}(P^{\prime})$.
An example of this is Algorithm 3 in Figure 11(b).

As mentioned in the main text, sparsity (SP) [] has a correlation with HV is defined as follows:

$$ $\mathrm{SP}(P):=\frac{1}{|P|-1}\sum_{j=1}^{N}\sum_{i=1}^{|P|-1}(\tilde{P}_{j}[ i]-\tilde{P}_{j}[i+1])^{2},$ (25) $$

where $\tilde{P}_{j}:=\mathrm{Sort}(\{p[j]|\forall p\in P\})$.
An increase in HV implies that the elements of the CP front are moving further away from each other, which increases SP.
To remove this correlation, we provide a normalized version as defined in (14), and the calculation process is illustrated in Figure 11(c).
Given an estimated CP front, $P$, we normalize each estimated objective value so that the minimum value corresponds to zero and the maximum to one.
The normalized objectives are denoted as $\overline{J_{R_{i}}}(\theta)$.
We then calculate the average of the squares of the distances between elements in this normalized space.
This normalized version still produces a large value when a CP front is densely clustered in particular areas while being sparsely distributed overall.
Therefore, it maintains the ability to measure sparsity while removing the correlation with HV.

### C.3 Implementation Details

In this section, several implementation details including hyperparameter settings and network structures are provided.
Before that, we report information on the computational resources.
In all experiments, we used a PC equipped with an Intel Xeon CPU E5-2680 and an NVIDIA TITAN Xp GPU.
The average training time in the single point goal task is presented in Table 2.

**Table 2: Training time on the point single goal task.**
|  | CAPQL | PD-MORL | LP3 | CoMOGA (ours) |
| --- | --- | --- | --- | --- |
| Time | 16h 49m 29s | 16h 40m 9s | 24h 4m 34s | 17h 27m 58s |

#### C.3.1 MORL to CMORL using the Lagrangian method

In order to apply linear scalarization-based MORL methods to CMORL, it is required to solve the following constrained optimization problem given a preference $\omega$:

$$ $\mathrm{max}_{\theta}\sum_{i=1}^{N}\omega_{i}J_{R_{i}}(\theta)\;\mathrm{s.t.} \;J_{C_{k}}(\theta)\leq d_{k}\;\forall k\in\{1,..,M\}.$ (26) $$

As explained in the main text, the constraints of the above problem can be handled by converting it into a Lagrange dual problem, and the dual problem is written as follows:

$$ $\mathrm{max}_{\lambda\geq 0}\mathrm{min}_{\theta}-\sum_{i=1}^{N}\omega_{i}J_{R _{i}}(\theta)+\sum_{k=1}^{M}\lambda_{k}\cdot(J_{C_{k}}(\theta)-d_{k}),$ (27) $$

where $\lambda_{k}$ are Lagrange multipliers.
The multiplies should be learned separately for each preference, but preferences have continuous values.
Therefore, we instead parameterize the multiplier using neural networks as $\lambda_{\phi}(\omega)$, where $\phi$ is a parameter.
Then, the above problem can be solved by concurrently updating the policy parameter and the Lagrange multipliers using the following loss functions:

$$ $\displaystyle\mathrm{min}_{\theta}L(\theta)$ $\displaystyle:=-\sum_{i=1}^{N}\omega_{i}J_{R_{i}}(\theta)+\sum_{k=1}^{M} \lambda_{k}J_{C_{k}}(\theta),$ (28) $\displaystyle\mathrm{min}_{\phi}L(\phi)$ $\displaystyle:=-\sum_{k=1}^{M}\lambda_{k,\phi}(\omega)(J_{C_{k}}(\theta)-d_{k} )\;\mathrm{s.t.}\;\lambda_{k,\phi}(\omega)\geq 0,$ $$

where the multipliers can be forced to non-negative values using some activation functions, such as softplus and exponential functions.
The policy loss in (28) is constructed by adding $\sum\lambda_{k}J_{C_{k}}(\theta)$ to the loss from the unconstrained MORL problem.
Therefore, the implementation of the existing MORL algorithms can be easily extended to a CMORL algorithm by adding $\sum\lambda_{k}J_{C_{k}}(\theta)$ to the original policy loss.

#### C.3.2 Network Architecture

The proposed method and baselines are based on the actor-critic framework, requiring policy and critic networks.
Additionally, since the baselines use the Lagrangian method to handle the constraints, they use multiplier networks, as mentioned in Appendix C.3.1.
We have implemented these networks as fully connected networks (FCNs), and their structures are presented in Table 3.

**Table 3: Network structures.**
|  | Parameter | Value |
| --- | --- | --- |
| Policy network | Hidden layer | (512, 512) |
|  | Activation | LeakyReLU |
|  | Normalization | ✘ |
| Critic network | Hidden layer | (512, 512) |
|  | Activation | LeakyReLU |
|  | Normalization | ✘ |
| Quantile distributional | Hidden layer | (512, 512) |
| critic network | Activation | LeakyReLU |
|  | Normalization | LayerNorm |
|  | # of quantiles | 25 |
| Multiplier network | Hidden layer | (512,) |
|  | Activation | LeakyReLU |
|  | Normalization | ✘ |

In the unconstrained tasks, MO-Gymnasium, we use standard critic networks that output scalar values for given observations and preferences.
However, the standard critic networks usually estimate the objective or constraint functions with large biases, making it challenging to satisfy the constraints.
Therefore, we use quantile distributional critics [] to lower the estimation biases in the Safety-Gymnasium and legged robot locomotion experiments.
The quantile distribution critic outputs several quantile values of the discounted sum of the reward or cost functions for given observations and preferences.
Then, the objective or constraint functions can be estimated by averaging the outputted quantiles.
For the proposed method and LP3 that have similar frameworks to TRPO [], TD($\lambda$) method [] is used to train the distributional critic networks.
For PD-MORL and CAPQL, the distributional critic networks are trained by reducing the Wasserstein distance between the current quantiles and the truncated one-step TD targets, as in TQC [].
The policy network, similar to other RL algorithms dealing with continuous action spaces [, , ], outputs the mean and diagonal variance of a normal distribution.
However, in the locomotion tasks, the policy networks are modified to output only the mean value by fixing the diagonal variance in order to lower the training difficulty.

#### C.3.3 Hyperparameter Settings

We report the hyperparameter settings for the CMORL tasks (Safety-Gymnasium, Locomotion) in Table 4 and the settings for the MORL tasks (MO-Gymnasium) in Table 5.

**Table 4: Hyperparameters for CMORL tasks.**
|  | CAPQL | PD-MORL | LP3 | CoMOGA (Ours) |
| --- | --- | --- | --- | --- |
| Discount factor | 0.99 | 0.99 | 0.99 | 0.99 |
| Length of replay buffer | 1000000 | 1000000 | 100000 | 100000 |
| Steps per update | 10 | 10 | 1000 | 1000 |
| Batch size | 256 | 256 | 10000 | 10000 |
| Optimizer | Adam | Adam | Adam | Adam |
| Policy learning rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ | - | $3\times 10^{-4}$ |
| Critic learning rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ | $3\times 10^{-4}$ | $3\times 10^{-4}$ |
| Multiplier learning rate | $1\times 10^{-5}$ | $1\times 10^{-5}$ | $3\times 10^{-4}$ | - |
| Soft update ratio | 0.005 | 0.005 | - | - |
| # of quantiles to truncate | 2 | 2 | - | - |
| # of HER samples | - | 3 | - | - |
| Angle loss coefficient | - | 10 | - | - |
| Explore and target action noise scale | - | (0.1, 0.2) | - | - |
| # of action samples | - | - | 20 | - |
| TD($\lambda$) factor | - | - | 0.97 | 0.97 |
| # of target quantiles | - | - | 50 | 50 |
| # of preference samples | - | - | 10 | 10 |
| Local region size | - | - | 0.05 | 0.05 |
| $g_{\mathrm{min}}$, $g_{\mathrm{max}}$ | - | - | - | $(0,\infty)$ |
| $\lambda_{\mathrm{max}}$ | - | - | - | $\infty$ |
| $H$ matrix | - | - | Fisher information | Fisher information |

**Table 5: Hyperparameters for MORL tasks.**
|  | CAPQL | PD-MORL | LP3 | CoMOGA (Ours) |
| --- | --- | --- | --- | --- |
| Discount factor | 0.99 | 0.99 | 0.99 | 0.99 |
| Length of replay buffer | 1000000 | 1000000 | 100000 | 1000000 |
| Steps per update | 1 | 1 | 1000 | 10 |
| Batch size | 256 | 256 | 10000 | 256 |
| Optimizer | Adam | Adam | Adam | Adam |
| Policy learning rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ | - | $3\times 10^{-4}$ |
| Critic learning rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ | $3\times 10^{-4}$ | $3\times 10^{-4}$ |
| Soft update ratio | 0.005 | 0.005 | - | 0.005 |
| # of HER samples | - | 3 | - | - |
| Angle loss coefficient | - | 10 | - | - |
| Explore and target action noise scale | - | (0.1, 0.2) | - | - |
| # of action samples | - | - | 20 | - |
| TD($\lambda$) factor | - | - | 0.97 | - |
| # of preference samples | - | - | 10 | 10 |
| Local region size | - | - | 0.14 | 0.05 |
| $g_{\mathrm{min}}$, $g_{\mathrm{max}}$ | - | - | - | $(0,\infty)$ |
| $\lambda_{\mathrm{max}}$ | - | - | - | $\infty$ |
| $H$ matrix | - | - | Fisher information | Identity matrix |

## Appendix D Additional Experimental Results

In this section, estimated CP fronts for tasks with two objectives are visualized.
As mentioned in the appendix C.2, the estimated CP front can be obtained by rolling out a trained universal policy with different preferences, and in this section, 20 equally spaced preferences are used.
In Figure 12, the CP fronts for the legged robot locomotion tasks are visualized, Figure 13 presents the CP fronts for the Safety Gymnasium tasks, and Figure 14 presents the CP fronts for the MO-Gymnasium tasks.
The hopper and ant tasks in the MO-Gymnasium are excluded from this visualization as they have three objectives.

Figure: (a) Biped
Refer to caption: extracted/5633925/imgs/cassie-v0_pareto.png

Figure: (a) Single-Point-Goal
Refer to caption: extracted/5633925/imgs/mo-safetypointgoal1-v0_pareto.png

Figure: (a) Humanoid
Refer to caption: extracted/5633925/imgs/mo-humanoid-v4_pareto.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I think there is a gap from (3) to (4) where the authors generalize to multiple objectives. In the multiobjective case (without 
constraints), are (2) and (3) equivalent only when a solution exists?  If so, is it the case that the norm constraint can always  
be satisfied, while the monotonic improvement constraints are not? 
 
2. The scaling of the gradient essentially adds the norm constraint back, but if this is the case, why not start with that 
constraint and use the form (2)? Like max_d d^\top g_i, s.t., b^\top d + J < d_k, ||d||_H \leq \epsilon? Is it because this form is not 
easily solvable due to the norm constraint (difficult to estimate H^{1/2})? If so, the authors could explicitly mention this. 
 
3. The authors claim that using (6) can help correct the policy when it avoids any constraint due to solving (5) which uses linear 
approximation. But isn't (6) also using linear approximation, and the solution to (5) should also satisfy constraints in (6) if there 
exists any feasible solution? Can the authors elaborate on what this means exactly? 
 
4. The first sentence of the abstract asserts that an RL agent needs to consider multiple objectives.  That's not necessarily the  
case - the reward hypothesis states that an agent's objective can always be summarized with a single scalar reward function.  How  
would you motivate this paper to someone who fundamentally disagrees that an agent needs to consider multiple objectives separately 
(as opposed to aggregating them to explicitly encode their tradeoffs)?
soundness: 3
strengths: 1. The problem of interest has broad application. 
 
2. The proposed method is straightforward yet well motivated. Specifically, the transformation of objective to constraint is 
convenient. 
 
3. The presentation is clear, and the figures illustrate the main ideas well. 
 
4. The empirical results demonstrate that the proposed method consistently outperforms existing methods.
summary: The paper presents an algorithm for multi objective reinforcement learning under constraints. The authors first identified that 
maximizing a specific return within the trust region is equivalent to minimizing $||\Delta \theta||^2_H$$ under a constraint that the 
return increases (if the solution exists). Therefore, the multi objective RL under constraints can be formulated as a problem that 
minimizes the induced norm of $\Delta \theta$ under a bunch of constraints that correspond to the multiple task returns and 
constraints. By using a first-order approximation of the constraints, the problem reduces to a quadratic programming problem. To avoid 
the violation of the constraint due to the inaccurate linear approximation, the authors propose to remove the return constraints once 
the violation happens. 
 
The above can be generalized to multiple objectives with preferences, and then to learn a universal policy (i.e., a policy that 
is conditioned on any given preference $w \in \R^N$), the authors propose to learn a policy that minimize the expected KL divergence 
between the learned policy conditioned on a specific preference and the updated policy that solves the above quadratic programming. In 
other word, it amortizes the optimization into a learned conditional policy. 
 
The authors conduct experiments on both synthetic toy and constrained RL problems in simulation, demonstrating that their method 
results in superior performance (in terms of optimizing the objective and satisfying constraints) compared against prior methods. The 
authors also provide a convergence guarantee for the proposed method.
weaknesses: One main weakness is the computational efficiency of the proposed method, which might hinder the practical usage of the proposed 
method. First it requires computing/storing gradients to constraints/rewards, which can be quite large if N and M are large. Second, it 
requires estimating terms like $x^\top H x$ where x is the fisher information matrix, while this does not in general require computing 
$H$ explicitly, it sill requires computing the inner product like $x^\top \nabla \log \pi$ (which might need extra space). The authors 
did not elaborate on this aspect, and there is no time complexity evaluation in the experiments or analysis sections.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Why does CoMOGA perform significantly better than the combination of CAGrad and Lagrangian in Section 6.5? Could the authors expand on this?
- Why is the combination of CAGrad and Lagrangian treated as an ablation study rather than a baseline?
- How are preferences sampled in Line 328, for example, are they uniformly sampled? How does this affect the distribution of results?
- Are the critics trained using the same targets, such as Retrace, across all methods in the experiments?
soundness: 3
strengths: - The problem addressed is highly relevant and has practical real-world applications.
- The paper is well-written and easy to understand.
- The experiments are thorough, and the evaluation metrics are clearly defined and articulated.
summary: This paper presents CoMOGA, an algorithm designed to optimize multiple objectives while satisfying constraints. CoMOGA converts objectives into constraints and combines the gradients from both objectives and constraints. Theoretical analysis provided in the paper demonstrates that CoMOGA can achieve a local Pareto optimal policy. Empirical tests show that CoMOGA surpasses competing methods in achieving better coverage of the constrained-Pareto front.
weaknesses: - The section on gradient aggregation could be improved by including comparisons with similar MORL methods like CAGrad and PCGrad.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) Can the authors analyze the computational complexity and the rate of convergence of the algorithm after converting multiple objectives into constraints?

2) Does the algorithm converge in continuous space?
soundness: 2
strengths: ## Originality and Significance
1) This paper presents a constrained multi-objective optimization algorithm, CoMOGA, which transforms objectives into constraints. The idea appears to be novel.
2) Multi-objective tasks and constraint optimization are critical issues in reinforcement learning, significantly enhancing the safety of reinforcement learning and its applicability in real-world scenarios.

## Quality and Clarity
1) The paper is well-written and easy to read.
2) This paper includes both theoretical analysis and detailed empirical results.
3) Multiple figures are provided, which are helpful in understanding the main concepts presented in the paper.
summary: This paper presents the Constrained Multi-Objective Gradient Aggregator (CoMOGA) algorithm for constrained multi-objective reinforcement learning (CMORL). It transforms the CMORL problem into a constrained optimization framework and addresses the optimization objective within a local region, using constraints designed to enhance the original objectives in proportion to preference values. The authors also provide a convergence guarantee in the tabular setting. Experimental results across several environments demonstrate that CoMOGA achieves constraint satisfaction.
weaknesses: ## Framework
1) The core idea of this paper is to avoid gradient conflicts by converting multi-objectives into constraint terms. However, during the optimization process, due to factors such as the linear approximation of the objectives and bias in the value function estimation, it is difficult to avoid constraint violations by the constraint terms of the objective, thus causing gradient conflicts. This may undermine the authors' core contribution.
2) Transforming multiple objectives into constraint terms can complicate the constraint boundaries, thereby increasing the difficulty of finding an optimal policy that satisfies these constraints.
3) It is not clear whether this transformation yields positive gains in computational complexity and convergence rate.

## Experiments
1) There is a lack of sensitivity analysis concerning hyperparameters.
2) The fairness of the ablation experiments is questionable. The constraint optimization algorithm employed for the CAGrad algorithm in the ablation experiments is the Lagrangian method, which is inconsistent with the optimization algorithm used in CoMOGA. The discrepancy in the experimental results may arise from the optimization algorithm itself, thus failing to accurately assess the effectiveness of the core components in CoMOGA.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. **Limited Practical Implementation Details in Constrained Optimization Problem (COP)**: The authors propose solving constrained multi-objective reinforcement learning (CMORL) as a constrained optimization problem (COP), which theoretically offers strong convergence guarantees. However, the paper lacks detailed insights into the practical implementation of this approach. In contrast, methods like TRPO provide clear derivations of practical algorithms from theoretical foundations, taking into account finite sample sizes and arbitrary parameterizations in deep reinforcement learning. Can you include a similarly detailed derivation to bridge the theoretical framework with practical implementation?

2. **Inclusion of Pareto-Dominated Points in Figure 7**: In Figure 7, numerous points are shown that are Pareto-dominated by others. This raises questions about the relevance of including these dominated points, as they do not contribute to optimal trade-offs and may complicate the interpretation of the results. What is the rationale behind the inclusion of these Pareto-dominated points, and how do they add value to the analysis presented in this figure?

3. **Inconsistencies in Hypervolume (HV) Reporting in Figure 8**: In Section 6.4, Why do the results for Hopper and HalfCheetah in Figure 8 differ significantly from the original results reported in PD-MORL?

4.***Practical Implementation Challenge: High Dimensionality and Computational Expense:***
In the theoretical work, Equation (9) utilizes the inverse Fisher information matrix over the policy parameter theta, potentially resulting in high dimensionality and significant computational expense in practical applications. How does CoMOGA handle this challenge in its practical implementation?
soundness: 3
strengths: 1.***Theoretical Contribution:***

The paper presents an interesting theoretical contribution, providing a comprehensive convergence analysis.

2.***Experimental Coverage:***

The study includes a wide range of experiments within the MORL domain, offering an extensive evaluation of the proposed method.
summary: This paper introduces a novel multi-objective reinforcement learning (RL) approach that incorporates safety constraints. The authors present a gradient conflict management technique, which formulates multi-objective RL as a constrained optimization problem. This formulation effectively prevents policies from getting trapped in local optima, particularly in tabular settings. Additionally, it enables the seamless integration of safety constraints and facilitates an efficient policy update mechanism.
weaknesses: 1.***Constrained Multi-Objective RL as a Constrained Optimization Problem:***

The authors propose addressing constrained multi-objective reinforcement learning (CMORL) by formulating it as a constrained optimization problem (COP). While COP offers robust theoretical guarantees for convergence, the paper provides limited insights into its practical implementation, which could impact the reproducibility and real-world application of the proposed method.

2.***Inconsistencies in Hypervolume (HV) Scale and Reporting:***
Figure 8 presents hypervolume (HV) results on a significantly smaller scale compared to previous work. For example, in PD-MORL, HV values are reported across training steps up to 10 $e^6$, while in this study, the results are shown only from 1$e^6$ up to 3$e^6$ steps, Which may not reach the convergency. 

3.***Different unit to benchmark in HV:***
 With PD-MORL reporting HV values in the unit of 1$e^6$ , whereas this paper uses a unit of 0.1 for HV. These inconsistencies create confusion and lead to the perception that PD-MORL performs worse in this comparison than reported in the original study.

Meta-review: This paper studies multi-objective reinforcement learning under constraints, by transforming the problem into a constrained optimization framework and addressing the optimization objective within a local region, with constraints designed to enhance the original objectives in proportion to preference values. Both theoretical and experimental results were provided. The problem studied is well-motivated and significant, the algorithmic idea is novel, and the paper is very well-written and easy to follow. There were some concerns regarding the high computational complexity and implementability of the proposed approaches, and the sufficiency of some experimental results. Overall the merits of the paper outweigh the drawbacks, and I suggest the authors incorporate the feedback from this round in preparing the camera-ready version of the paper.

Decision: Accept (Poster)