Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning
Authors: Dohyeong Kim
Mineui Hong
Jeongho Park
Songhwai Oh
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Objective Reinforcement Learning
Safe Reinforcement Learning
Gradient Aggregation
Primary area: reinforcement learning
TL;DR: We present a constrained MORL (CMORL) algorithm which transforms the CMORL problem into a constrained optimization.

Abstract: In real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines.
To address these considerations, we propose a constrained multi-objective RL algorithm named constrained multi-objective gradient aggregator (CoMOGA).
In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima.
It is also essential to efficiently handle safety constraints for stable training and constraint satisfaction.
We address these challenges straightforwardly by treating the maximization of multiple objectives as a constrained optimization problem (COP), where the constraints are defined to improve the original objectives.
Existing safety constraints are then integrated into the COP, and the policy is updated by solving the COP, which ensures the avoidance of gradient conflicts.
Despite its simplicity, CoMOGA guarantees convergence to global optima in a tabular setting.
Through various experiments, we have confirmed that preventing gradient conflicts is critical, and the proposed method achieves constraint satisfaction across all tasks.

Published as a conference paper at ICLR 2025
CONFLICT-AVERSE GRADIENT AGGREGATION FOR CONSTRAINED MULTI-OBIECTIVE REINFORCEMENT LEARNING
Dohyeong Kim, Mineui Hong, Jeongho Park, and Songhwai Oh" Dep. of Electrical and Computer Engineering and ASRI, Scoul National University
ABSTRACT
In real-world applications, reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines. To address these considera- tions, we propose a constrained multi-objective RL algorithm named Constrained Multi-Objective Grudient A Aggregator (CoMOGA). In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima. It is also essential efficiently handle safety constraints for stable training and constraint satisfac- tion. We address these challenges straightforwardly by treating the maximization of multiple objectives as constrained optimization problem (COP), where the constraints are defined to improve the original objectives. Existing safety con- straints are then integrated into the COP, and the policy updated by solving the COP, which ensures the avoidance of gradient conflicts. Despite its simplicity, CoMOGA guarantees convergence to global optima in a tabular setting. Through various experiments, we have confirmed that preventing gradient conflicts is criti- cal, and the proposed method achieves constraint satisfaction across all tasks.
INTRODUCTION
Real-world reinforcement learning (RL) applications involve multiple objectives and have to con- sider safety simultaneously. For instance, legged robot locomotion tasks focus maximizing the success rate of reaching goals and energy efficiency while avoiding collisions, as illustrated in Fig- ure[] To effectively tackle these considerations, constrained multi-objective RL (CMORL) has been introduced (Huang et al.] 2022). CMORL aims to find a set of constrained Parrret (CP) optimal poli- cies (Hayes al. [022), which are not dominated by any others while satisfying safety constraints, rather than finding a single optimal policy. This allows users to choose and utilize their preferred policy from the set without additional training. In order to express a of CP optimal policies, is common to use a concept called preference (Cai et al., 2023 Kyriakis & Deshmukh 2022). which encodes the relative importance of the objectives. Then, a set of policies can be represented by policy, a mapping function from a preference space to a policy space.
In most multi-objective RL (MORL) algorithms (Désidéri 2012 Yang et al. 2019 He al. 2024), the preference-condiiee dot product of given preference and rewards of the multiple objectives. These approaches, called linear scalarization (LS), offer the benefits of straightforward implementation and ensure Pareto- optimal convergence tabular settings (Lu et al. 2023). However, when applied to deep RL with function approximators, LS tends to converge to local optima due to the nonlinearity of the objective functions, as shown in a toy example in Figure 2 In the field multi-task learning (MTL) YYu et al. 2020 Liu et al.) [202] 1: Navon et al. 2022). this issue has been addressed by avoiding conflicts between the gradients of multiple objective functions. Building on these advancements, we extend these gradient-conflict avoiding techniques to CMORL, anticipating analogous benefits.
policy is trained by maximizing a scalarized reward, computed as the
It is also crucial to handle constraints as well as multiple objectives in CMORL. A straightforward approach is treating the constraints as other objectives and concurrently adjusting the preferences corresponding to those objectives to maintain them below thresholds, as done in (Huang et al.)
"Corresponding author: songhwai snu.ac..

---

Published as a conference paper at ICLR 2025
10 LS
CoMOGA
Energy efficiency
Foasible
Space
-10
Velocity
Figure 1: Example of CMORL. The robot aims maximize energy efficiency and velocity while maintaining its halance to remain within the feasible space. In or- der to consider such safety and multiple objectives, CMORL finds ble policies that are dominated by other policies and satisfy constraints, which are indicated by the dashed line.
Figure 2: Comparison between linear scalarization (LS) and the proposed method (CoMOGA). Optimization trajectories are shown in orange, with initial points marked by black eircles 'x' markers for the reached points. Contours and shaded ar- eas represent average values of the two objective functions and regions where constraints are violated, respectively. The grey line indicates the CP optimal set. Unlike LS, CoMOGA con- sistently reaches the optimal set from any starting position. For more details, see Appendix
of feasi-
2022 This approach exactly aligns with the Lagrangian method, which solves the Lagrange dual problem by updating policy and multipliers alternatively. However, the Lagrangian method can make training unstable due to the concurrent update policy and multipliers (Stooke et al. 2020 Kim & Oh 2022). As the updates of the policy and multipliers influence each other, any misstep in either can make the training process diverge. Thus, it is critical to manage constraints without introducing additional optimization variables.
In order to avoid gradient conflicts and efficiently handle safety constraints, we introduce a novel but straightforward algorithm named Constrained Multi-Objecttvv Gradient Aggregator (CoMOGA). CoMOGA treats a multi-objective maximization problem as a constrained optimization problem (COP), where constraints their respective preference values. Existing safety constraints are then incorporated into the con- straint of the COP. By explicitly preventing the given objectives from decreasing, CoMOGA successfully avoids gradient conflicts. The policy gradient is then computed by solving this refor- mulated problem within a local region, in TRPO (Schulman et al.) 2015). thereby eliminating the need for extra variables to handle safety constraints. While the proposed method is intuitive and straightforward, we have also shown that it converges a CP optimal policy in a tabular setting.
designed to enhance the original objective functions in proportion to
The proposed method has been evaluated across diverse environments, including multi-objective tasks with and without constraints. The experimental results confirmed that avoiding gradient con- flicts is effective in preventing convergence to local optima. This is particularly essential in envi- ronments with unstable dynamics, such as bipedal and humanoid locomotion tasks, since they are prone to get stuck in suboptimal behaviors, such frequent falling. Furthermore, the absence of additional variables facilitated stable training while ensuring constraint satisfaction across all tasks.
2 RELATED WORK
Multi-Objective RL. MORL aims find set of Pareto-optimal policies that not dominated by any others. Given that Pareto-optimal policies can be obtained with appropriate reward scalariza- tion (Hayes al. 2022). many MORL algorithms utilize scalarization techniques. Based linear scalarization, Yang et al. 2019) introduced new Bellman operator for MORL algorithms, which estimates target value for given preference by combining Q values from other preferences. Basaklar et al. (2023) presented a similar Bellman operator and extended it to an actor-critic frame- work to enable its application in continuous action spaces. For the continuous action spaces, Lu et al. (2023) proposed SAC-based MORL algorithm, theoretically demonstrating that elements of the Pareto-optimal policy set can be achieved through linear scalarization. Not only the linear scalarization, Van Moffaert et al. (2013) proposed a nonlinear scalarization method, called Cheby- shev scalarization, which can cover a non-convex policy set. To further improve performance, the evolution strategy (ES) can be applied as done in multi-objective optimization criteria (Deb al.) 2000). [Chnn et al. 2020) and Xu et al. (2020) proposed methods that repetitively update a policy for each population using linear scalarization and produce generation of policy parameters.
2

---

Published as a conference paper at ICLR 2025
There is also different approach from scalarization. Abdolmaleki et al. (2020) have proposed heuristic method of applying preference in the action distribution space instead of the reward space. It is further extended to a CMORL algorithm in (Huang et al. 2022).
Constrained RL. Constrained RL is developed to explicitly consider safety or constraints in RL. Based on the approach to handling constraints, constrained RL algorithms be categorized into two types: primal-dual methods and primal methods. The primal-dual methods, also called La- grangian methods, are designed to address Lagrange dual problems. [Ding et al. (2022) introduced a natural policy gradient-based primal-dual method and demonstrated its convergence to an opti- mal policy at a specified convergence rate. Another primal-dual method, proposed by |aii et al.) 2022), ensures that a trained policy results in zero constraint violations during evaluation. While these methods are straightforward to implement and can be integrated with standard RI. algorithms, the training process can be unstable due to additional optimization variables (Lagrange multipliers) (Stooke et al.) 2020). In contrast to the primal-dual method, the primal method addresses the con- strained RL problem directly, so additional optimization variables are required. Acchimm et al. (2177) introduced a trust region-based primal method. This approach linearly approximates a con- straint within a trust region and updates the policy through line search. Xu et al. (2021) presented a natural policy gradient- based algorithm and demonstrated its convergence rate towards an opti- mal policy. In multi-constrained settings, Kim al. (2023) proposed primal method handle infeasible starting cases.
BACKGROUND
Constrained Multi-Ohjective MDP. We define constrained multi-objective Markov decision pro- cess (CMOMDP) as a tuple represented by (S, A, P, R, C, p. 7),, where S is a state space, A is an action space, P: : S A x S + [0. 1] is transition model, R S x A S + RN is vectorized reward function, C² S' x A x S + I M is a vectorized cost function, P is an initial state distribution, and y is discount factor. The reward and cost functions are bounded by Rmax- In the CMOMDP setting, policy <--ss) E II can be defined as caae-condiiional action distribution. The objective function of the ith reward is defined as:
Jn. (T) :
> Ri Ct, S(+1)
where 80 Ps dg T(-1s)), and 8141 P(-|se, at) VI. We also define the constraint function Jc, (Tr) by replacing the reward with the cost function. A constrained multi-ohjective RI. (CMORL) problem is defined as:
(1) where de is the threshold of the kth constraint. Here, it is ambiguous to determine which policy is optimal when there are multiple objectives. To address this, we instead define a set of optimal policies using the following notion, called constrained dominance (Miettinen) 1999):
maximize, Jn, (T) Vi E (1,..,NN s.t. Jc, (T) S de Vk E (1,
Definition 3.1 (Constrained Dominance). Given two policies Tı, T2 E { E IIJo,, (T) s de Vk}, T is dominated by π2 if JR, (T1) S JR. (T2) Vi E N).
A policy æ which not dominated by any policy called constrained-Par Pareto (CP) optimal policy, and the set of all CP optimal policies is called constrained-Parte (CP) front (Miettinen 1999.
Additionally, a condition for gradient conflicts is defned as follows:
Definition 3.2 (Gradient Conflict). Let 9i be the gradient of the ith objective function, and let denote the policy update direction.
We say there is a gradient conflict 3i such that of g < O.
As studied in MTL (Liu et al.) 2021 Yu et al. 2020) and shown in Figure 2 can cause convergence to local optima. Thus, we aim to find the CP front while updating policies without gradient conflicts.
gradient conflicts
Preference. In order to express a subset of the CP front, prior MORL works (Xu al. 2020 Alegre et al. 2023) introduce a preference space, defined as: Ω := (w E N wi 2 0, | 1}. where preference E Ω serves to specify the relative significance of each objective. Subsequently, specific subset of the Pareto front can be characterized by a function mapping from the preference space to the policy space. We denote this function as universal policy ((-ls,w), characterized by an action distribution that depends on both the state s and preference w variables. Ultimately, our goal is to train the universal policy that covers the CP front extensively as possible.
3

---

Published as a conference paper at ICLR 2025
4 PROPOSED METHOD
We propose constructing a sub-problem designed to aggregate gradients without conflicts. Building sub-problems is widely used strategy in conflict-averse MTL approaches (Navon al. 2022 Yu et al. 2020 Liu et al. 2021), but applying this strategy in the context of CMORL requires considering preferences of multiple objectives and ensuring the satisfaction of constraints. This section outlines the proposed method in four distinct parts: 1) building a constrained optimization problem to handle the multiple objectives with safety constraints, 2) calculating policy gradient by aggregating gradients of the objective and constraint functions for single preference, 3) introducing a method for training a universal policy covering a spectrum of preferences, and 4) concluding with first parameterize
an analysis of the convergence properties of the proposed method. Before that, the policy and define gradients of the objective and constraint functions, as well as a local region in the parameter space. By representing the policy To with parameter 0 E O the gradients of the objective and constraint functions in
[ are expressed as:
9i == VoJR. (TO), by := VoJc, (To).
For brevity, we will denote the objectives Jn, (f) and the constraints as Jc, (0). The local region then defined using a positive definite matrix H as: ||~Ø||и I c, where || la == vx Hæ, and e is the local region size, a hyperparameter. H can be the identity matrix or the Fisher information matrix, as in TRPO (Schulman et al. 2015).
4.1 REFORMULATION AS CONSTRAINED OPTIMIZATION
We introduce a process of treating the CMORL problem constrained optimization problem (COP). To provide insight, we first present the process in a simplified problem where only the ith objective exists and then extend it the original problem. The problem of maximizing the ith objective within the local region is expressed as follows:
max JR. (Oold + A0) s.t. ||00|| S.
Assuming that JR. (0) is linear with respect to 0 within the local region, the above problem has the same solution as the following problem:
min A HAR s.t. Jn. (0.t + A0) - JR. (P.ta) 2 el|9:1l =: e₁.
(3)
It is shown that the solutions 2 and 3 are identical in Appendix B. and this finding confirms that objectives can be converted into constraints in this way. Consequently, the CMORL problem can be reformulated as follows:
A8 HA0 s.t. wie, JR. (Bold A0) JR. (Pota) Vi, Jc, (fold + A0) < de Vk, (4)
where in is scaled to wie; to reflect the given preference w. We will show that the policy updated using converges CP optimal policy of I in Section 4.4 under mild assumptions.
4.2 GRADIENT AGGREGATION
Now, we compute policy gradient by solving (4) within the local region using a linear approxima- tion, similar to TRPO (Schulman et al.] 2015). First, the problem (4) can be linearly approximated
as the following quadratic programming (QP) problem:
Ar HAO s.t. wie, S o Vi, 6% 65 Jo, (Dota) S d, Vk,
(5)
where the gradients of the objective and constraint functions, gi and be are aggregated into g6 The gradient is then clipped to ensure that the policy is updated within the local region, as follows:
g = min(1, e/110" |1 1
where the policy will be updated by Oold + n Note that the aggregated gradient satisfies 0 S 9 ges Vi due to the reformulated constraints in (5) This ensures that the aggregated gradient does not conflict with any of the objective functions. However, when both the constraints and objectives are reflected in the updates, the updated policy can violate the constraints by overly focusing on objective improvements. To address this issue, we take recovery step which updates the policy by
4

---

Consequently, the ideal Published as a conference paper at ICLR 2025
Linear approximation
Refomulation
Scaling
Halll
bg
2ai 8/60
4/30
Ras
min| || I mys S 8 4e. 6/20+ 1r,(0n) s
maxg/ 26 Vist. of 50 Je (0
pig
Figure 3: Process of CoMOGA. We visualize the process in the parameter space, and the gray areas represent constraints. (Linear approximation) CoMOGA linearly approximates the original CMORL problem in [ The gradients of the objective and constraint functions are visualized as black and blue arrows, respectively. (Reformulation) The objectives are converted to constraints as described in (5). The intersection of all con- straints is shown as the red area. (Scaling) The solution of the transformed problem, geh, is scaled to that the updated policy remains within the local region.
considering only the safety constraints when the current policy violates them, which is motivated by safe RI algorithms (Kim et al.] 2023 Xu et al. [202]] ZZang et al.) 2022). We will show that this step helps the policy converge to global optima in Section [4.4] The recovery step involves solving the following problem, which solely consists of the safety constraints:
ag :- argmin A0" HAO s.t. by A0 + Jc, (Osta) S dx Vk.
(6)
As result, the process for obtaining the policy gradient is depicted in Figure can be formalized as follows, which is named constrained multi objett-o gradient aggregator (CoMOGA):
g aargmin go AO HQ0 s.t. wye; S of 20, by A0 + Jc, (Dola) S d+ Jc, (Osia) S du Vk, otherwise.
argmin Ao A0T H s.t. D Jc. (foia) $ d2
gh = minn1,,c/|l5 1
We also provide a detailed explanation of the differences between CoMOGA and existing conflict- averse MTL algorithms in Appendix
4.3 UPDATE RULE FOR UNIVERSAL POLICY
We have introduced the gradient aggregation method named CoMOGA for updating policy for given single preference. This section presents a method for updating universal policy that can cover the entire preference space. If a preference 'w is given, the policy should be updated to 0 + g"w Similarly, if the preference is ww, the policy should be updated to 0 + Izw universal policy is required satisfy the following conditions:
ag
Towe (a|s, w) = Toaut (als,w) Va E A, E S, and Vw E Q
where we denote Feig+ as an intermediate policy for w. This can be interpreted as the KL diver- gence between the universal policy and the intermediate policies being zero. Based on this property, we introduce practical method to achieve the universal policy by minimizing the following loss:
ming (T,W)/D [EeT [DKL (Tewu (-1s, [ [+ (+Is,w))]]
(7)
where D is a replay buffer, and F is a trajectory. By minimizing the above loss, we can combine poli- cies for each preference into single universal policy. After training is completed, users utilize diverse policies by providing personalized preferences to the trained universal policy, eliminating the need for additional training.
4.4 CONVERGENCE ANALYSIS
In this section, we analyze the convergence properties of the proposed method. Specifically, it can be shown that CoMOGA achieves CP optimal convergence with minor modifications H is the Fisher information matrix. To this end, we first introduce the necessary conditions for the policy gradient method to ensure CP optimal convergence. We then show that the modified CoMOGA satisfies these necessary conditions. Before that, we first introduce the following assumption:
Assumption 4.1 (Slater's condition). There exists a feasible policy n such that Jc, (TT) S d₁ - rk, where y) > 0 is positive number.
5

---

Published as a conference paper at ICLR 2025
The Slater's condition is common assumption in safe RL ensure optimal convergence (Kim et al.] 2023 Bai et al.) 2022; Ding et al.] 2022). We also assume that the state and action spaces fimite. Now, we introduce generalized version of the policy gradient method that guarantees optimal convergence.
Theorem 4.2. Assume that sequences fVe, vle Ni, N. S [0, Amaxl are given for all i and k. where Σx Na,x L, X (0,) dx) 2 O. E, = and Vi, converges to specific point Di. Then, if a policy is updated by the following rule, converges to a CP optimal policy of 47p
(t..)
EE Li9 Ex= beebb
Jc. (0t) S dg Vk,
0++1- - 02 +
(8)
(i,) a4 Ei U/,9: - ER be,
otherwise,
where 9: J. ), by: V... a₁ step size satisfying Robbins-Mond condition (Rob bins & Monro 1951). and P represents the pseudo-inverse of the Fisher information matrix.
The proof is provided in Appendix B.2 The sequences Ui,,s ui, Ni, and Ni, in Theorem 4.2
represent the weights assigned the gradients of the objective and constraint functions at each update step, and new CMORL algorithm can be developed based on how the sequences are set. Similarly, we will show that CoMOGA can be formulated as (8) by identifying these sequences with minor modifications. Since the quadratic programming holds strong duality, we can express -i=
the solution of (5) as Tht H -H vig, M Xūbk), where 21t and Xg. the optimal Lagrange multipliers of the following dual problem:
X argmax 220,220
1
Σ Vig: -E >abR)
H-1
Vigi E Aalis) + Σ Dswee
2
+ +E, Au(Ja, (Dola) - dx).
In case of the dual problem has no solution, we set 21 and X to be gnt O. For the solution of (6). the same process is applied to achieve X1- Then, the CoMOGA is modified as follows:
Ag EN 3 - E EN min( E, A )6k if Jc. (0.) S de Ak,
ghd = H
otherwise,
9 = - min (max " Il Imnnn) Imax),
where Imin Imax. Amax E R>o are hyperparameters, and e is the local region size satisfying Gobbins-Mont condition. Finally, we can show optimal convergence of the modified CoMOGA.
Theorem 4.3. If H is the Fisher information matrix and policy ma. is updated as 8t+1 = 0₃ g0x
where g is defined in ( it converges to a CP optimal policy of (7)
Theorem 4.3 can be proved by identifying sequences V9 v/, X Nhe that satisfy the conditions
mentioned in Theorem 4.2 and details are provided in Appendix B.3
PRACTICAL IMPLEMENTATION
We need to compute the gradient of the objective and constraint functions to perform CoMOGA and the universal policy update. To do that, we use reward and cost critics Vi,an Ve where u/ and do are the critic network parameters, to estimate the objective and constraint functions. The critics trained by minimizing the following loss functions:
Σn (Yn. - V VR (s,,
((bn)) = (s.a.s' WwDDD
E
(10)
where the target YR, can be caleulated either as as, +75) Rati (s', a',w) with a' ~ T(-Is,,w) or using Retrace(A) (Munos al., 20166), and the same process is applied to the cost crities. Given a preference w, the objective and constraint functions can be estimated as follows:
JR. R
E
VV (s, a, w.] Jc. (-
E
VoL (s, a. w)]
(11)
Then, compute the policy gradients using the critic networks through the reparameterizztion
trick, as done in the SAC paper (Haamoja et al. 2018). For the details of solving COP, please refer to Appendix C.3.1 Finally, the proposed method is summarized in Algorithm

---

Published as a conference paper at ICLR 2025
Algorithm 1 Policy Update Using CoMOGA
Input:
Policy parameter 0, reward critic parameter Wn, cost critic parameter vec, replay buffer D.
for epochs = to E do
for rollouts to L do
Sample preference w
Collect trajectory T {s: oe, Yt, Ct 8141) by using ne(+),,w) and store ()) in D. end for
Update reward and cost critic networks using Equation for porp=1 to P do
TT
Sample (T, ~ D, and estimate the objectives and constraints using Equation
Calculate the aggregated gradient 9h" using Equation (). and set O2, = Oola + 9-"
Store an intermediate policy TTa..
end for
Update the universal policy using Equation end for
EXPERIMENTS
This section evaluates the proposed method and baselines across various tasks with and without constraints. First, we explain how methods are evaluated on the tasks and then present the CMORL baselines. Subsequently, each task is described, and the results are analyzed. Finally, we conclude this section with ablation studies of the proposed method.
6.1 EVALUATION METRICS AND BASELINES
(0)
(0)
Jx(00)
Algrithin
Panilo fron fellenence point
Alporihmm
Agorithm
(0.1)
d
d
SF
[P|
44
d5
(0)
(0)
(0,0)
(1.0)
(0)
(a) Hypervolume
(b) Reference Point
(c) Normalized Sparsity
Figure 4: Visualization of metrics. (a), the hypervolume represented by the gray area. In (b), star indicates the reference point, determined from the entire Pareto front, whose elements are circled in red. In (c), the normalized sparsity is calculated by averaging the squared distances between elements in the normalized spacc.
Metrics. Once the universal policy trained, an estimated CP front, denoted C R.V. can be obtained by calculating the objective values (JR, (r...... Jaw (r)) for range of preferences. Given that CMORL aims to approximate the ground truth CP front, essential assess how close the estimated CP front is to the ground truth, and it can be realized by measuring the coverage and density of the estimated front. To this end, we use the hypervolume (HV) for coverage measurement and sparsity (SP) for density estimation, which are commonly used in many MORL algorithms (Xu) cet al. 2020 Basaklar et al. 2023). The visualization
metrics is shown Figure 4 and detailed
descriptions are provided in Appendix C.2 Given an estimated CP front P C R and reference
point r E N the hypervolume (HV) is defined as follows (Zizzler Thiele )
HV(P.r) / (2)d,)(2)ddz
(12)
where Q(P,r) = {=}3p E P S.L. r 32232 p). and the metric represents the volume of the area surrounding the reference point and the estimated CP front Sparsity (SP), on the other hand, measures how uniformly the estimated CP fronts distributed and is defined as the average of the squared distances between elements of the CP fronts (Xu et al.) 2020 However, the original definition exhibits correlation with HV, where larger HV corresponds larger SP. To eliminate this inherent correlation, we propose to use the following normalized version of SP:
IP--
2
P P
SP(P) Σ
(13)
max, maxxP,[k] min+ Py [k]

---

Published as a conference paper at ICLR 2025
Hypervolume
Sparsity
Constrant 1
64
Corstraint
Constraint 3
Biped
02
Sidn
100
Stape
Look
Staps
Sage
106
Suga
109
Lec1 hyypervluuee
Sparsity
Constra of
Constraint
Constraint
Quatruped
0.3
3,0
1.0 Sigh
5105
Slon>
Stepo
Siao
CAPQL
PD MORLL
LP3
CAG 0d
COMOOG (ours)
Threshold
Figure 5: Evaluation results of the legged robot locomotion tasks. The upper row shows results for the bipedal robot, while the lower row is for the quadrupedal robot. algorithms are evaluated at every 10 steps. The bold lines and shaded areas represent the mean and quarter-scaled standard deviation of results from five random seeds. respectively. The black dotted lines in constraint graphs indicate the thresholds.
where P, == (prl((p e Pt). and Pyld] [i] is the ith element Pj. By normalizing the distance using the minimum and maximum values of the CP fronts, we can remove the correlation with HV while preserving the ability to measure sparsity. Additionally, to reduce the influence of the scale of each objective, normalize each objective within the CP front using the minimum and maximum values from the entire dataset.
Baselines. For comparison with the proposed method, we first employ LP3 (Huang et al. 2022 an existing CMORL algorithm, as a baseline. This method extends the scale-invariant MORL algo- rithm, MOMPO (Abdolmaleki et al. 2020), to CMORL by handling constraints in similar way to the Lagrangian method. To provide more comprehensive baselines, we include MORL algorithms that are based on policy gradient approaches, which can be adapted to handle constraints using the I Lagrangian method. There are two state-of-the-art policy gradient-based MORI. a algorithms: PD- MORL (Basaklar et al.] 2023). which is based on TD3 (Fujimoto al., 2018). and CAPQL (Lu) et al. 2023), which is based on SAC (Haarnoja et al.] 2018)) Also, we include conflict-averse MTL
algorithms, CAGrad (Liu et al. 2021) We have extended them to CMORL by handling constraints
using the Lagrangian method. For implementation details, please refer to Appendix C.3.2
6.2 LEGGED ROBOT LOCOMOTION
The legged robot locomotion tasks (Kim et al., 2023 are to control quadrupedal or bipedal robot to follow randomly given commands while satisfying three constraints: keeping 1) body balance, 2) CoM height, and 3) pre-defined foot contact timing. Since the original tasks were designed to have single objective, we have modified the tasks to have two objectives: 1) matching the current velocity with the command and 2) minimizing energy consumption. Please see Appendix [oor more details.
The evaluation results are presented in Figure and the estimated CP fronts are visualized in Figure 7 Across all tasks, CoMOGA achieves the highest HV and the lowest SP while satisfying all con- straints. Notably, CoMOGA surpasses the baselines the bipedal task. This suggests that prevent- ing gradient conflicts is particularly effective for dynamically unstable robots, such bipedal robots, because these robots require a careful balance of multiple objectives to attain desired behaviors. LP3 shows performance similar to the proposed method in the quadrupedal task but underperforms the bipedal task.
6.3 SAFETY GYMNASTUM
We utilize single-agent and multi-agent goal tasks in the Safety Gymnasium (Ji al. 2023 These tasks require navigating robots to designated goals while avoiding obstacles. The single-agent goal tasks have two objectives: 1) reaching goals many times as possible within time limit and 2) maximizing energy efficiency, along with a single constraint to avoid collisions with obstacles. The multi-agent goal tasks have two objectives and two constraints, which are to maximize goal achievement and avoid collisions for two robots, respectively. In both tasks, point and car robots are used. For details on the tasks and hyperparameter settings, please refer Appendix C
The training curves are presented in Figure 6 and the estimated CP fronts are shown in Figure CoMOGA shows the highest HV in all tasks and the lowest SP in two out of four tasks, while
8

---

Published as a conference paper at ICLR 2025
Hypervolume
Sparsity
Constraint
Hypervolume
Sparsity
Constraint
Singlle Colle-PPintee Goint-GGoa
26-
0/9
20
0/99
wnc-
16
0.44
0.
0.22
05
0.00
000
(a) Single-agent tasks. The left shows the results for the point robot, and the right is for the robot. Constraint 2
Mults-Paint Gool
Hypervolume
Sparsity
Constraint 1
000
845
Days
Hypervolume
Sparsity
Constraint
Constraint
MultiCCa-dco
0012
0000
0557
2435
0.5
3.5
2.0
0000
0.
1.0
1.5
2.3
0.002
04
1.0
703
CAPCL
MORL
LP3
CAGrad
CoMOCA (ours)
Thrashold old
(b) Multi-agent tasks. The upper row shows the results for the point robot, and the lower is for the car robot.
Figure 6: Evaluation results of the Safety Gymnasium tasks.
CAFC)
Fale:
CAPOL
Rator Far::
CAROE
LP3
CoMOGA
0
dlpo
3.00
033
8:0
0.23
1255
825-
-0.2
1.0
012
2.4
0.5
1.0
322
(a) Biped
(b) Quadruped
(c) Single- Point-
CARO 7D- MO3L
LP3 Chcred
CoHCOA Fafte Fart
CAFe PD MORL
LF3 CAGrad
COM roun) Refer Point
CAFOk PD MO1L
LF3 CkGrad
CoMoDA
Bether Point
1.00
2.00
1:00
0.75
2.75
2.75
1.00
2:00
Gool
Agert
Agon:
(d) Single-Car-Ce Car-Goal
(e) Multi--Pintt-Gal
(f) Multi-Car-Goal
Figure 7: Visualization of estimated CP fronts for CMORL tasks.
satisfying all constraints. Additionally, the low volatility in the evaluation graph indicates that the proposed method can stably handle both constraints and objectives. LP3 and PD-MORL violate constraints in at least one task, and CAPQL satisfies all constraints but shows weak performance in the single agent tasks, implying the difficulty of handling constraints and objectives simultancously. CAGrad shows a similar HV in the Multi-Cat Car-Goal task, but it has a large sparsity.
6.4 MULTI-OBJECTIVE GYMNASIUM
We conduct experiments in the Multi- Oujtii-bbjeetive (MO) Gymnasium (Alegre et al. (022), which is a well-known MORL environment, to examine whether the proposed method also performs well on unconstrained MORI, tasks. We use the MuJoCo tasks with continuous action spaces in the MO Gymnasium, and there are six tasks available: Hopper, Humanoid, Half-Cheetah, Walker2d, Ant, and Swimmer. Each task has three, two, two, two, three and two objectives, respectively. Details are provided in Appendix C
The training curves are presented in Figure g the estimated Pareto fronts are detailed in Appendix D The proposed method achieves the highest HV in three tasks and the lowest SP in five out of six tasks. Especially, CoMOGA exhibits outstanding performance in the hopper and humanoid tasks, which suggests that the conflict-averse strategy is effective for dynamically unstable robots even in
9

---

Published as a conference paper at ICLR 2025
Hopper
Numanold
Hhll-Chhetth
Waiter2d
Ant
Swimmer
Hyparosture
5aps
Staps
13
Sporste
Sparsty
Spursty
Sparaity
S2ogs
CAPOL
PD NORL
CoNAOGA rours)
Figure 8: Evaluation results of the MO Gymnasium tasks. Each column shows the results of the task corre sponding to its title.
Hypervelume
Sparsity
Constrant nt 1
Constraint 2
Constraint
Biped
25
Staps
Steps
Steps
100
1e-1 Hypervalume
Sparsity
Constraint:
Constraint
21
25
1.5
20
0.5
1-0
2.0
00
0.5
2.0
1.2
1.5
0.5
1.5
3.0
CLAACIA
CoROGA-LS
CaMMORA
--- Threshold
Figure 9: Results of the ablation study in the legged robot locomotion tasks.
unconstrained MORL settings. However, algorithms except LP3 show high HV results in the walker2d task, which is also dynamically unstable. It can be considered that they have reached the global optimum due the well-designed reward functions.
6.5 ABLATION STUDY
This section evaluate the effectiivneess of the core components of the proposed method. The core components include (1) transforming objectives into constraints to avoid gradient conflicts and (2) handling safety constraints by solving a QP problem, where the constraints are linearly approximated within the local region. To evaluate the efficiency of these components, we designed two ablation methods: 1) CoMOGA+LS, which excludes (1) and uses linear scalarization to handle multiple ob- jectives, and 2) CoMOGA+ Lag, which excludes (2) and handles constraints using the Lagrangian method. The training results on the legged locomotion tasks are presented in Figure [ The re- sults indicate that CoMOGA+Lag suffers from instability in handling safety constraints due to the use of Lagrange multipliers. Meanwhile, CoMOGA+LS ensures stable satisfaction of safety con- straints but achieves lower HVs compared to CoMOGA, confirming the effectiveness of CoMOGA in avoiding gradient conflicts. Additionally, we present the training results for the quadruped task across various hyperparameter values in Appendix to evaluate sensitivity.
CONCLUSIONS AND LIMITATIONS
We have introduced a CMORL algorithm called CoMOGA maximize multiple objectives while satisfying safety constraints. The proposed method is based on a novel reformulation process that converts objectives into constraints, which enables the avoidance of gradient conflicts among mul- tiple objectives and the handling of constraints without additional optimization variables. We show that the proposed method converges to a CP optimal policy in tabular settings and demonstrate that achieves outstanding performance in HV and SP metrics with constraint satisfaction through various experiments. Since the proposed method updates the policy in a manner that prevents any objective functions from decreasing, the set of trained policies may cover narrow portion of the CP front. Future research can address this limitation by mitigating the condition of gradient conflicts to bet- ter balance multiple objectives and expand coverage. Specifically, by satisfying the conditions of the proposed generalized policy update rule in Theorem 4.2 various CMORL algorithms can be developed with convergence guarantees, which can accelerate future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01 190, [SW Star Lab] Robot Learning: Efficient, Safe, and AecepllyAAcepptable Machine Learning).
REFERENCES
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. Proceedings of International Conference on Ma- chine Learning, pp. 11-22, 2020.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of International Conference on Machine Learning, pp. 22-31, 2017.
Alekh Agarwal, Sham Kakade, Jason Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4500 2021.
Lucas Alegre, Florian Felten, El-Ghazali Talbi, Grégoire Danoy, Ann Nowé, Ana L. C. Bazzan, and Bruno C. da Silva. MO-Gym: A library of multi-objective reinforcement learning environ- ments. In Proceedings of Benelux Conference on Artificial Intelligence BNAIC/Benelearn, 2022.
Lucas N Alegre, Ana LC Bazzan, Diederik Roijers, Ann Nowé, and Bruno C Silva. Sample- efficient multi-objective learning via generalized policy improvement prioritization. In Proceed- ings of International Conference on Autonomous Agents and Multiagent Systems, pp. 2003-2012, 2023.
Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal approach. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 3682-3689, 2022.
Toygun Basaklar, Suat Gumussoy, and Umit Ogras. PD-MORL: Preference-driven oujti-objective reinforcement learning algorithm. In Proceedingy of International Conference on Learning Rep- resentations, 2023.
Xin-Qiang Cai, Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens. Distribu- tional pareto-optimal multi-objective reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Stéphane Caron, Daniel Arnström, Suraj Bonagiri, Antoine Dechaume, Nikolai Flowers. Adam Heins, Takuma Ishikawa, Dustin Kenefake, Giacomo Mazzamuto, Donato Meoli, Brendan O'Donoghue, Adam A. Oppenheimer, Abhishek Pandala, Juan José Quiroz Omaña, Nikitas Rontsis, Paarth Shah, Samuel St-Jean, Nicola Vitueci, Soeren Wolfers, Fengyu Yang, @bdel- haisse, @ @MeindertHH, @rimaddo, @urob, @shaoanlu, Ahmed Khalil, and Lev Kozlov. qp- solvers: Quadratic Programming Solvers in Python, 2024. URL https: / /ttg//gtttu/btte com/
Diqi Chen, Yizhou Wang, and Wen Gao. Combining gradient-based method and an evolution strategy for multi-objective reinforcement learning, Applied Intelligence, 50:3301-3317, 2020.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018
Kalyanmoy Deb. Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In Proceedings of Parallel Problem Solving from Nature PPSN VI, pp. 849-858. 2000.
Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Başar, and Mihailo R. Jovanovié. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint 2:Xxv:2206.02344 2022.
11

---

Published as a conference paper at ICLR 2025
Jean-Antoine Désidéri. multiple-gradien descent algorithm (MGDA) for multiobjectiive optimiza- tion. Comptes Rendus Mathematique, 350(5):313-1138 2012.
Florian Felten, Lucas Nunes Alegre, Ann Nowe, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno Castro da Silva. toolkit for reliable benchmarking and research in multi- objectíve reinforcement learning. In Proceedings of Conference Neural Information Process- ing Systems Datasets and Benchmarks Track, 2023.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of International Conference on Machine Learning, pp. 1587- 1596, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of Inter- national Conference Machine Learning, pp. 1861-1870, 2018.
Conor Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
Xiangkun He. Zhongxu Hu, Haohan Yang, and Chen Lv. Personalized robotic control via con- strained multi-objective reinforcement learning. Neurocomputing, 565: 665:12690 16986,, 2024.
Sandy Huang, Abbas Abdolmaleki, Giulia Vezzani, Philemon Brakel, Daniel J. Mankowitz, Michael Neunert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, and Raia Hadsell. A constrained multi-objective reinforcement learning framework. In Proceedings of Conference Robot Learning. pp. 883-893, 2022.
Jiaming Ji, Borong Zhang, Xuehai Pan, Jiayi Zhou, Juntao Dai, and Yaodong Yang. Safety- gymnasium. GitHub repository, 2023.
Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters, 7(3):7644-6611, 2022.
Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust eggion-based safe distributional reinforce- ment learning for multiple constraints. In Advances in Neural Information Processing Systems, 2023.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of International Conference on Machine Learning, pp. 5556-5566, 2020.
Panagiotis Kyriakis and Jyotirmoy Deshmukh. Pareto policy adaptation. In Proceedings of Interna- tional Conference on Learning Representations, 2022.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, pp. 18878-18890, 2021.
Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convexity, stationarity and Pareto optimality. In Proceedings of International Conference on Learning Rep- resentations, 2023.
Kaisa Micttinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-tase learning as a bargaining game. In Proceedings of International Confer- on Machine Learning, pp. 16428-16444) 2022.
12

---

Published as a conference paper at ICLR 2025
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Sabble-bsselinss:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://jmlz htlpj/l.rr/////e.mmmmmeomoo pazpg//pps///2////// 20- 1364. html
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pp. 400 4070-407,000 1951.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of International Conference on Machine Learning, pp. 1889- 1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety reinforcement learning by PID Lagrangian methods. In Proceedings of International Conference on Machine Learning, pp. 9133 9143, 2020.
Kristof Van Moffaert, Madalina Drugan, and Ann Nowé. Scalarized multi-objective reinforce- ment learning: Novel design techniques. In Porceedings of IEEE Symposium Adaptive Dy- namic Programming and Reinforcement Learning (ADPRL), pp. 191-199, 2013.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Pro- ceedings of International Conference on Machine Learning, pp. 10607-10616, 2020.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of International Conference on Machine Learning, pp. 11480-1144 1491, 2021.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pp.
2020.
Linrui Zhang, Li Shen, Long Yang, Shi-Yong Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In Proceedings of Inter- national Joint Conference on Artificial Intelligence, 2022.
E. Zitzler and Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257-271, 1999.
13

---

Published as a conference paper at ICLR 2025
A
TOY EXAMPLE
Landscape of (x)
Landscape of L,(x;
10
10
5
-10
-10
-5
-10
10
X1
X1
Figure 10: Landscape of the objective functions of the toy example.
In this section, we describe the details of the toy example, which is initially defined (Liu et al,
2021 Objective functions of the toy example formulated follows:
[1(21.02) == c1 ((z))))))
(2()))22) = e1 [ (2(2)))))))))) where
+
Ci (x) := max(tanh(0.5xx)00)
c2(x) == max(tanl((-0.522.0))
[1 = log(max(10.5(-m) - tanh( (.000000))) + 6,
(2(21, = (og(max([005(--11 + + Lanli-222 + 2)), 0.000005) + 6,
91 (211 x22 = -(--1 + 77 0.1( 8)3)/10- 1 20,
92 (2(21,02) = ((-)) - 7)2 0.1 8)3)/10 20,
where L, and L2 are the objective functions. The landscape the objective functions are presented in Figure 10 In addition, define constraint function, which is defined as follows:
C(r,, == 2 + 0.3 10)2 10.5².
Finally, the toy example is formulated as follows:
min Ln (Z1, x2) and L2(2),02) s.t. C(r1,
21
We the preference (0.5, 0.5) for both algorithms, LS and CoMOGA, and use the following four initial points: (-10, 0). (-10, 7.5), (0, 7.5). and (10, 10). The optimization trajectories for each method are shown in Figure 2 For the initial point (-10, 7.5), the gradient of I is much larger than T.2. Thus, addressing the toy example with LS can be significantly influenced by I and eventually converges
local optimal point of L1. To resolve this issue, avoiding gradient conflicts can prevent policy gradients from being dominated by a specific objective, which helps them converge an optimal point.
14

---

Published as a conference paper at ICLR 2025
B PROOFS
B.1 PROOF OF REFORMULATION PROCESS
In this section, prove that the solutions of (2) and (3 are equivalent. Under the linear assumption,
we can express the ith objective function as 9: A Jn. (9ota)- Then, (2) can be rewritten as follows:
max g A0 + Jn,(Oouu (foia) s.t. A HA0 Se..
A6
As the strong duality holds, we will find the solution by solving the following Lagrange dual prob- lem:
max min - 0 A0 + A(A0TAO 2 = L()0, 220 28
Then, (A) argmin ( A) = H 9; By replacing AO with (A) in the dual
problem, we get X = argmax [( (A), v (26)) Finally, we obtain the solution,
AO" (x*) = eH- 9// V H- --
Now let us find the solution of (3 Under the linear assumption, we can rewrite 3 as follows:
min A HAO s.t. e, S 9 A0.
(14)
Similar to the above process, the Lagrange dual problem is derived as follows:
maxmin A HA0 + X(e, - 9. A0) =: L( A).
(15)
A20 A
Then, the solution of the dual problem is obtained as A0 (A) argmin so L(A0, A) XH- 92 Using A (A), we can get the optimal Lagrange multiplier as X = argmax, T()00(
20./(g (9 g.) 2// H-H-
g:. Then, the solution is A6" = AA" cH ge/ V g H-1
As result, the solutions of and (3) are the same c 1- V 9 H-
B.2 PROOF OF THEOREM
We first define the following notions for policy m:
(s) = (1 'P(ss: = =) ~ p. ((1s))((l)))ss...mmmm))))))))))) $ ~ P(-(s;,)) Vt,
(s()) = E
? R(st, 02, S441) s, - - 8141 ~ P(. ar) 4A
Q* a) = E
? RR(s,, az St+1)
0, = a, ~ P(-(s,, art1 ~ x(-122+1) Vt
A (8, a) = Q" - (x).
We also simplify several notations for brevity as follows:
Te. + NT, d,t + de, A (s,a) + A',, (s, a), A (s, + A', Since it is assumed that the state and action spaces finite, policy can be parameterized using softmax parameterization as follows (Agarwal et al. 202T):
(aass))
(s)
Σ
exp(O(s,aa),
a'EA
where 0 S a trainable parameter, and ( (s, a) denotes the value corresponding to state and action a. Through Lemma 4 in et 202T). the following equations are satisfied when a policy updated according to {
0 = as + Cr
(EN Ul ar Σ k= Na AG. A/
if Jc, S da Ak, otherwise,
1-7
(ar DN Ah
M
Rt
k=
exp(04+1(6) (8, a) 0c( (((88 a))
(a41( = (als))
(,(8)
where Zs(s) Σ
(T(ass) exp(0x41 a) (s)s)))
aEA
15

---

Published as a conference paper at ICLR 2025
Now, introduce lemma showing that the LS approach guarantees to converge CP optimal
policy, followed by the proof of Theorem 4.3
Lemma B.I. Given a preference w, let us define an LS optimal policy as follows:
T = arg IIXX Σ (T) s.t. Jc.(T) S dg Vk.
Then, the LS optimal policy is also a CP optimal policy.
Proof. Let us assume that there exists a policy μ that dominates the LS optimal policy Tue By definition of the constrained dominance, the following is satisfied:
N
JR. (p) JR, (TL) Vi and Jc, (n) S de Ak. = Σ wJ.(Jn. (m) JR. (Tā)) > 0.
However, this equation contradicts the fact that Th is LS optimal policy, Consequently, Th is not dominated by any others, it also a CP optimal policy.
Theorem 4.2. Assume that sequences {uE, u8.30 Xi, X 1ir S [0, Amax] are given for all and k. where Σx Nº.x 1, Ni. (Jc. - dx) 2 0. E, 1, and vi, converges to specific point Dg. Then, if a policy is updated by the the following rule, it converges to a CP optimal policy of [7
(t.)
EE Ui EEE Xeeabe
Jc. (0t) S dg Vk,
- = 0
(8)
(,Fi(()) a Dim vL.99 - -E Ni.bbs)
otherwise,
where 9: = JR (6.), by = V.c. (Be). ag is step size satisfying Robbins-Mon Mooro condition Rob-
bins & Monro 1951). and F represents the pseudo-inverse of the Fisher information matrix.
Proof. We first consider the case Jc, (0:) S dg Vk, in which the policy is updated as follows:
N
M
1c+1 = 0, +
ARk
Ac.)
A=
Using Lemma in Schulman et al.
2015
we can derive the following inequality:
(R.)).)
- az Σ Xin(Jen (c.(
N
M
Baw
(s, - -a -s) >
N (s)
Rm
= Band!!
E
(1 (0,1 - (ss
Eawness
at
(16)
1
(als(
E
Egnd
11
BaNte log (a(as))
at
1
E
Bodd
+ log (s))
at
1
(x)) i 1 7 1-?BE Eanp [log (x)),
z 00 endtth
(log
where (i) follows from the fact that d (1 - 7) - ''Pr(se =s) 2 (1 ())ss. By Lemma in (Xu et al. 2021.
(0++1))1) - Jn,((.)) 2Rmax
|10+1 0r112 Vi,
1- ?
2R
(o,(0.41) -Jc, (0,)
|1 0.112 Vk.
1
16

---

Published as a conference paper at ICLR 2025
Σ -Ja
-J. (JR()) at Ne, (Jc -o
2Rmax (N
0.ll2
1-7
2Rmax (N or M Amax
02
N
Vi,
An,
2g 04
M
Acx
1-7
2
k=1
20x max (N + or 2 max
(1 (1-
max
/
Then, 16 can rewritten follows:
20x Ro
1 1-77 Ex-p 0t
(1 7)3 (N + a, VIS|IA|
Esmp [log (e(s)
(17)
which is satisfied for any Finally, we derive the following inequality for policy μ as follows:
Σ vi (Jn,( (n.) - azt Σ (Jc. - Jc, (0.)
N Σ
M
E snud
Basp(-Is)
Ui (s, a Σ
N Ag (s,a)
k=1
Bywdf
Fanuilo))
log
0z
(a(s)
(18)
E Bg
[DKL(M(-ss]]lag
(o)du(((s)]l(x41(111 + log [x))
E
(DKL(u((s))]m (
+
20,
R2
7)3 (N a v /S|IA
max
where (i) is from 17 by replacing P with dy. For brevity, we denote Ex-ay (=) as (KI((ulTT)) Now, we consider the second case where the constraints are violated. In this case,
0++1 0, +
(at
Ni.
1=1
which is symmetrical to the first case, achieved by replacing uf, - and and og - Mr. Using this symmetry property, we can obtain the following inequality from (18):
v (JR - (O.) > (La (R) Jcx
(19)
(DKL(Hl|TT)) DKL 204 R2 71- (a, 2
Po
Xmax v V
04
Now, we define set of time steps, N = [L]Jc, (mT) S dg Vk). Then, by summing 18 and over several time steps, the following is satisfied for a policy μ::
19
Σ IEN
a
Σ
Jn, (0.)) -af
af Σ
Jca
+ VEN
Σ
at Σ (1) -R) - at Σ (e -cc (0c))
(20)
ai 2R
DKL(H((TTO)
Σ
(1- (1- max(cu, 1)2(N 2) maxx /SIA
=0K
17

---

Published as a conference paper at ICLR 2025
If t & N, Xi.5. = 0 for Jc, (0.) < dg. which results in Ex (e,e (6.)) -n. By using this fact and replacing in (20 with Ty.
Σ INN
- oZ Σ (in - Jci (01( (J. - JR. (0a)) -Ta Σ
Ot
Σ
at Σ (RR - + a.1)
- (T +K| Σ og.
EE
Σ
04 Σ Ve (T, - Jnn(0.))
Σ 0u!
iEN
LEN
DRL(Ty||TUU) + K1 Las+
a/ Σ a Σ
Jc, (0.))
sEN
Σ a Σ (JR, (TT) JR. (0r)
IEN
2 2R Amax max (N, M)
S DKL(Ty|(TO) + Σ ar Kl +
Since EMEN Emam- am ELEN a07
at (ai AN
JR. (0e) 17 1) -
Σ
+Σ am S (Ky Σ o Kgg
Due to the Robbins-Monro condition, the right term converges real number. Since E, 0:77 = 00 in the left term, the following must be satisfied:
11 11) =
Σ an EEN
( Jn.
which results in EIEN a: 0o. Now, let us define a preference w, where wi == D.. By replacing in 20) with To.
Σ iEN
a4 Σ
Jn. (0:))
EE EEN
at Σ (J.. (m Jc. (0:)
I DKL(FullTT) + +12 Σ at
Σ at
JR. (b.)) (TL(TE K2 Σ of.
TEN
Since the right term converges to a real number and ELEN 0: = oo, the following must be satisfied:
lim Σ t
JR. -
where N₁ is the tth element of N. Consequently,
Σ Jn (m
Σ
De lim Jn. (Tww)),
which means that lim,-soo TT, is also a LS optimal policy for the preference w. Due to Lemma B.1 the policy converges to CP optimal policy.
18

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF THEOREM 4.3
Theorem 4.3. If H is the Fisher information matrix and a policy To. is updated as O2+1 = O2 + gNs, where 9 is defined in (9. it converges CP optimal policy of (7)
Proof. This theorem can be proved by identifying sequences VL,s Ve, A. Xi, that satisfy all con- ditions mentioned Theorem 4.2 follows:
By examining o
pf Ni k and as can be deduced as
4 u/e = 0, 0,A
X
دو v3
Σラ X3
X 2i=1 V
X1 = min (man ||"h||| Imin) ) Imax) min
Amax
min(max( fw I1. Imin Imax
Substituting the deduced sequences into ( can obtain the same formulation sequences satisfy that "T, = 1, Σ A 1, v/, E R>0 is bounded, 0 S > S Imax Xmax is also bounded, and E [<e//ruxx <e/ Itnin] follows the Robbins-Monro condition. Additionally, since Xg is an optimal point of the dual problem, it satisfies the KKT condition, which results X Jc, (0t) < dg- Consequently, >², (Jc. (0.) - d&) 2 0. Now, we need to check whether the vf, converges real number. Given that the learning rate follows the Robbins-Monro condition, the policy converges to a specific policy. Since (5) becomes invariant at this point of convergence, the solution of the dual problem is also fixed. Consequently, vG, converge to specific value.
(8) The deduced
B.4 CONVERGENCE RATE
In order to derive the convergence rate, we adopted the approach presented in the CRPO paper Xu jt al. (02T). To apply this approach, we slightly modified the policy update rule from Theorem 4.2
as follows:
(O.)
(IN Uf a Ex= bi L N 14.99 Ei= M
if Jc, (0.) S ds + +1 Vk,
8.41 = 0.
(21)
aF
a
bb,bb))
otherwise,
where 17 is called slack variable, and the key differences are that the recovery step only taken when the constraints are violated by more than the slack variable, and the learning rate is fixed over time steps. With these slight modifications, we can derive the following convergence rate.
Theorem B.2. Let us set the learning rate a and the slack variable n as follows:
a= ==1/TT, 17 2( ( (m))) + R2)/ VT,
where w = . x) K2 = K1 + 2Rmax
(N, Max(N,M)/(L - 7):
K1 2R max(a 1) 2 /ISIAA/(1 (1-1))1-9)
Then, the followings satisfied:
< DKL(TullTU)) + +
Enn
- (O.)) {
VT
E (0.) de 2 DKL( + +N2 yk.
VT
Proof. By substituting |JR, (mī) - JR. (0e) (mttt - Jcs (0i) < 2R 7) into (0).
Σ a
Σ
(J. - (0.)) -2a² Amax Pmnxx/ - 7)
Σ
(-2a2 AmaxtBmax + +an) < DRI(FalIT)) + 2T.
EN
19

---

Published as a conference paper at ICLR 2025
ΣΣ (J (mET) Jn, (0:))
E am DKI.FFllFF)) +oTaa.
(22)
IEN By using the definitions of and 7,, the following is satisfied:
tEN
anT/2 2 (KL(T + K2a
If we assume that N = 0, from 22 we have:
anl - (T)l Kga I ani'
which leads to contradiction. As a result, N of 0. Additionally, if we assume that [M| < T/2 and LIEN Ei (JR. (T#)
(0.) O, we have:
anT/2
Jz. Σ an E - KE
Σan /n
IEN
VEN
which leads to contradiction. As result, of the followings satisfied: 1) W T/2 2) ELEN (Jn, JJn Jn.(0)) & 3. If T/2.
ExeN Σ Ui (JR - JR((0.1)
ELEN E, Vg (R (TL) (O.))
W
DKL DKL (Ta (I + K2TA 2 PKI(Fallng + A2
T
VT
(T+) If EIEN (T)) JR. (0)) < 0, the above inequality also holds. As result, the perfor- mance gap between the LS optimal policy τ and the current policy m is bounded by O(1/VT) For constraint satisfaction, we derive the following bound:
DKL( (T mo) + X2
[,.w(ce, d; S n
VT
which implies that the constraint violation is also bounded by O(1/VT).
C EXPERIMENTAL DETAILS
In this section, we describe the details of each task, including Safety Gymnasium (Ji et al. 2023
legged robot locomotion (Kim et al. 2023 and Multi-Objective Gymnasium (Felten et 2023 We then provide detailed explanation of the performance metrics named hypervolume and spar- sity. Finally, we present implementation details, such as network architectures and hyperparameter settings.
C.I TASK DETAILS
C.1.1 SAFETY GYMNASIUM
(a) Single-Point-Gan Point-Goal
(b) Single-Car-Goal
(c) Multi-Point Goal
(d) Multi-Car-Goal
Figure 11: Snapshots of the Sefeet-Gymmasium tasks. There are two types of robots: point and car. These tasks aim to control the robots to reach goals while avoiding hazardous areas, colored purple. In the Single-Poim Paile--PoinGGroo and Single-Carr Car-Goal tasks, single goal, shown in green, is ran- domly spawned. Conversely, the Multi-Point Goal and Multi Car--ar--rra Car-Goal tasks have two goals, shown in blue and red, and if an agent reaches any of them, a reward given the agent.
In the Safety Gymnasium (Ji et al. 2023). we utilize the single-agent safe navigation tasks: Sa SafetyyPointGGal 1-v0 and Sa SafetyCarGos nGoa 1-v0, and the multi-agent safe navigation
20

---

Published as a conference paper at ICLR 2025
tasks:
and SafetyCar/aultiGGal1111 Snapshots of each
task are shown Figure
In the single-agent tasks, the observation space includes velocity, acceleration, and Lidar of the goal and hazards, and the dimensions are 60 and 72 for the point and car robots, respectively. The action space for both robots is two-dimensional. There eight hazard areas and one goal, and hazard areas are randomly spawned at the beginning of each episode, while goals are randomly placed whenever the agent reaches the current goal. These tasks originally had single objective: maximizing the number of goals reached. However, in order to have multiple objectives, we have modified them include a second objective: minimizing energy consumption. Therefore, the re- ward is two-dimensional, and we use the original definition for the first reward (please refer to Ji et al. 2023 The second reward is defined as follows:
IA Σ (a:/10)2.
R2(s, a, s') = LA
(23)
Also, we use the original definition of the cost function, which gives one if the agent enters hazardous areas and zero otherwise.
The multi-agent tasks have two agents and two goals, and we need to control both agents to reach goals as much as possible while avoiding hazardous areas. The original implementation uses dictionary type for observations and actions, so we have modified them to be an array type. Addi- tionally, in the original setup, each goal is pre-assigned to a specific agent, reducing
difficulty of solving multi-agent tasks. Therefore, we have modified the implementation so that each goal is not pre-assigned, allowing each agent to compete to achieve goals. The observation space contains the same information from the single-agent tasks for the two agents, well as additional information: Lidar the first and second goals. The dimensions of observation space are then 152 and 176 for the point and robots, respectively. The action space is four-dimensional, which is doubled from the single-agent tasks to control both agents. These tasks have two objectives: maximizing the number of goals reached by each agent. Also, there are two constraints: avoiding hazardous areas for each agent. Since there are no additional objectives or constraints, the original definitions of the reward and cost functions are used without modification.
C.1.2 LEGGED ROBOT LOCOMOTION
(a) Quadruped
(b) Biped
Figure 12: Snapshots of the legged robot locomotion tasks. Robots aim to follow a given com- mand while ensuring they do not fall over.
The legged robot locomotion tasks (Kim et al. 2023) aim to control bipedal quadrupedal robots so that their velocity matches a randomly sampled command. This command specifies the target linear velocities in the and y-axis directions and the target angular velocity in the z-axis direction, denoted by (wgmd, cm wedd). ). Snapshots of these tasks are shown in Figure 12 The quadrupedal robot has 12 joints and 12 motors, and the bipedal robot has 14 joints and 10 motors. Each robot is operated by PD controller that follows the target position of the motors, and the target position is given as an action. Hence, the number of motors corresponds to the dimension of action space. In order to provide enough information for stable control, observations include the command, linear and angular velocities of the robot base, and the position and velocity each joint. As a result, the dimensions of the observation space are 160 for the quadruped and 132 for the biped. Originally, these tasks have single objective of following a given command and three constraints: 1) maintaining body balance, 2) keeping the height of CoM (center of mass) above certain level, and 3) adhering to pre-defined foot contact timing. Therefore, we use the original implementation for the observation. action, and cost functions but modify the reward function to have multiple objectives. The modified version has
21

---

Published as a conference paper at ICLR 2025
two objectives: 1) reducing the difference between the eurrent velocity and the command and 2) minimizing energy consumption. Then, the reward function is defined as follows:
Ri(s,a, == (2)
base
Uxr
.mmdy
(u)
base
by
gmdd
2
base wmnd)?
(24)
R2(s,a,8)) 8):
T3 Mrohot
where the number of joints, Tj is the torque applied to the jth joint, and Mrobot is the mass of the robot. However, in the simulation of the bipedal robot, obtaining valid torque information is difficult due to the presence of closed loops in the joint configuration. To address this issue, we modify the second reward for the bipedal robot to penalize the action instead of the joint torque as follows:
A4
8=(
0
(25)
A
C.1.3 MULTI--BIECTIVE GYMNASIUM
Table I: Specifications of MO-Gymnasium tasks.
Observation Space
Action Space
# of Objectives (Entries)
S
R 17
S I
2 (velocity, energy)
Hopper
C R
C R
3 (velocity, height, energy)
Humanoid
R-376
C R17
2 (velocity, energy)
Ant Walker2d Swimmer
C RR27
C 8R:8
3 (rr-elooityy y-velocity, energy)
C R
C TR
2 (velocity, energy)
C R
IR2
2 (velocity, energy)
We utilize the Multi-Ohjective (MO) Gymnasium (Felten et al. 2023 to evaluate the proposed method in MORL tasks which have constraints. Among several tasks, we use the MuJoCo tasks with continuous action space, and there are six tasks mo humanoid-vv, mo hawhhheetahhhhaaeeeeeeeeeaaaaaae and mo mo-swin wwvimmer-vv Description of the observation space, action space,
available: mo-hopper-va mo mo--aa so--wakker2d and the number of objectives for these tasks are summarized in Table and for more details, please refer to Felten et al. (2033..
mo mo-ant-vv -v4,
C.2 METRIC DETAILS
In this section, we explain in detail the performance metrics used in the main text: hypervolume (HV) and normalized sparsity (SP). First, we need prepare estimated CP front to caleu- late such metrics. To do this, we pre-sample a fixed number of equally spaced preferences and roll out the trained universal policy for each pre-sampled preference to estimate objective and con- straint values. If the constraint values do not exceed the thresholds, we store the objective values, (Jn, J... (f)), set. Once this process completed for all preferences, we can construct an estimated CP front by extracting elements from the collected set that are not dominated by any others. The objective values are then normalized using the minimum and maximum values from the entire dataset, effectively minimizing the impact of scale of each objective. Now, introduce details on each metric.
HV is the volume of the surrounding given reference point, and estimated CP front, P. as defined in (12) and is visualized in Figure However, as seen in Figure 4(a () or (12). the metric value varies depending arbitrarily but use a method of determining the reference point from the results of all algorithms, as shown in Figure 4b). For detailed explanation, suppose that we have a of estimated CP fronts, (P.) (.. where P. is the estimated front obtained from the ith algorithm. Then, we obtain the union K denoted by P', and extract elements that are not dominated by any others in P' to find the Pareto front of the entire set, PF(P'), whose elements are circled in red in Figure 4b). Finally,
what reference point is used. Therefore,
do not set the reference point
of {P:)
22

---

Published as a conference paper at ICLR 2025
get the reference point whose ith value is min((palVp PF(P)))) Through this process, the reference point can be automatically set after obtaining results from all algorithms for each task. In addition, the HV value can be zero for some algorithms whose elements are entirely dominated by PF'P)). An example of this is Algorithm 3 in Figure bb).
As mentioned in the main text, sparsity (SP) [u al. as follows:
2020 has correlation with HV is defined
N PP--1
SP(P) = P
Σ
Σ
(P
(26)
where P, Sprop(pppii E P}). An increase in HV implics that the elements of the CP front are moving further away from each other, which increases SP. To remove this correlation, provide normalized version as defined in (33). and the calculation process is illustrated in Figure(4c). Given an estimated CP front, P, normalize each estimated objective value so that the minimum value corresponds to zero and the maximum to one. The normalized objectíves are denoted as JR. (0). We then calculate the average of the squares of the distances between elements in this normalized space. This normalized version still produces large value when CP front is densely clustered in particular areas while being sparsely distributed overall. Therefore, it maintains the ability to measure sparsity while removing the correlation with HV.
C.3 IMPLEMENTATION DETAILS
In this section, several implementation details including hyperparameter settings and network struc- tures provided. Before that, report information the computational resources. In all exper- iments, we used a PC equipped with an Intel Xeon CPU ES-2680 and an NVIDIA TITAN Xp GPU. The average training time the single point goal task presented in Table
Table 2: Training time on the point single goal task.
CAPQL
PD-MORL
LP3
CoMOGA (ours)
Time
16h 49m 29s
16h 40m 9s 24h 4m 34s
17h 27m 58s
C.3.1
IMPLEMENTATION DETAILS OF CONSTRAINED OPTIMIZATION PROBLEM
CoMOGA aggregates the gradients of the objective and constraint functions by solving the following quadratic programming (QP) problem:
g argmin 28 TT HA0 s.t. wie AAA.
A0, by A0 + Jc, (fola) S de-
As we mentioned in Section 4.4 the QP problem holds strong duality, so the solution can be ex- pressed follows: T H- Di= vi Σd Aāba), where v," and XØ the optimal La-
grange multipliers of the following dual problem:
Σ Vigi Σ, Aabe) H-1 Σ Vigi Σ AabE)
n²
argmax v20,220
2
(27)
+ Σ mwrce + E, An(Jo, (fota) - dx).
we first compute 9: and by. which can be obtained by backpropa- gation using (11). Next, we need to calculate H-19, If H is an identity matrix, H- simplifies to gi itself, and no additional computation is required. When H is the Fisher Information Matrix (FIM), this calculation aligns with the policy gradient method used in the TRPO algorithm (Schulman al., 2015). Consequently, we employ the conjugate gradient (CG) method, as in TRPO, to approximate H H 9.- The CG method iteratively approximates H-1 9: within finite number of steps, and in implementation, we used 10 iterations. For further details about the implementation, we refer to the TRPO code in the Stable-Basee Baselines3 repository (Raffin et al.) 2021)
In order solve this problem
For simplicity of notation, let gk+N = -b, for l = 1.... M. Once 9; and gi are computed, construct (N + M) (N M) matrix P {pij), where Pij g H H-1 gj. We also compute
23

---

Published as a conference paper at ICLR 2025
(N + M) dimmeensio vector where 9: -wye; if $ N, d2 -c, otherwise. Then,
27 can be rewritten as:
= Vi, XE N4 Vk, where x= = arg min = #20 2
Px Px. + q"
(28)
Since this problem is formulated in low-dimensional space with N M variables, it can be efficiently solved using a QP solver (Caron et al. 2024 employed in numerical optimization. After solving the optimal dual variables, the aggregated gradient be calculated g E
Regarding the complexity of solving the constrained optimization problem (COP), the QP solver highly efficient and well-established, allowing to solve most problems within milliseconds. Thus, the time required for the QP solver can be assumed to be constant. The dominant factor in computational complexity is therefore determined by the process setting up (28) This involves calculating the inner products between gradients, which performed (N M) times. Conse- quently, the complexity is proportional to (N + 2 where P is the number of policy parameters. Additionally, the Conjugate Gradient (CG) method involves finite number of iterations. As such, its computational cost can be treated as a constant relative to the gradient calculations. In summary, the overall complexity of solving the COP is O(N P)
C.3.2 MORL TO CMORL USING THE LAGRANGIAN METHOD
In order to apply linear scalarization- based MORL methods CMORL, required solve the following constrained optimization problem given preference wt
maxa J.JR s.t. Jc. (0) < d Ak E {I....
(29)
As explained the main text, the constraints of the above problem can handled by converting into a Lagrange dual problem, and the dual problem is written follows:
M
maxxzoming
WiJ (0)
A -Jc.JJ (0) dx).
(30)
1=1
where A are Lagrange multipliers. The multiplies should be learned separately for each preference, but preferences have continuous values. Therefore, instead parameterize the multiplier using neural networks as )d(w), where & is parameter. Then, the above problem can be solved by concurrently updating the policy parameter and the Lagrange multipliers using the following loss functions:
M
ming Σ W.Jn, (0)
Σ (0),
k=1
(31)
M
mingT = Σ An.
dr) s.t. (x, 2
where the multipliers can be forced to non-negative values using some activation functions, such as softplus and exponential functions. The policy loss in (31) is constructed by adding Σ Ag-Jc, (#) to the loss from the unconstrained MORL problem. Therefore, the implementation of the existing MORL algorithms can be easily extended to a CMORL. algorithm by adding Σ AAJJc, (0,,,,,)) (0) to the original policy loss.
Additionally, the PD-MORL algorithm [Basaklar et al. 2023) uses a preference alignment tech- nique However, for fair comparison, we did not employ this technique it necessitates 1) pre-training to identify the objective direction corresponding to key preferences and 2) additional evaluation steps during the main training process adapt the objective direction. Nevertheless, to enrich our discussion, we provide experimental results of the PD-MORL algorithm with preference alignment in Appendix
24

---

Published as a conference paper at ICLR 2025
C.3.3 NETWORK ARCHITECTURE
The proposed method and baselines are based on the actor-critic framework, requiring policy and critic networks. Additionally, since the baselines use the Lagrangian method to handle the con- straints, they use multiplier networks, as mentioned in Appendix C.3.2 We have implemented these networks as fully connected networks (FCNs), and their structures are presented in Table B In
Table 3: Network structures.
Parameter
Value
Policy network
Hidden layer
(512, 512) LeakyReLU
Activation
Normalization
Critic network
Hidden layer Activation
(512, 512) LeakyReLU
Normalization
X
Quantile distributional
Hidden layer Activation
(512, 512)
critic network
LeakyReLU
Normalization of quantiles
LayerNorm
25
Multiplier network
Hidden layer Activation
(512,)
LeakyReLU
Normalization X
the unconstrained tasks, MO-Gymnasium, for given observations and preferences. However, the standard critic networks usually estimate the objective constraint functions with large biases, making it challenging satisfy the constraints. Therefore, we use quantile distributional critics (Dabney et al.) (018) to lower the estimation biases in the Sifety--ymmmsiium and legged robot locomotion experiments. The quantile distribution critic outputs several quantile values of the discounted sum of the reward or cost functions for given obser- vations and preferences. Then, the objective or constraint functions can be estimated by averaging the outputted quantiles. For the proposed method and LP3 that have similar frameworks to TRPO [Schulman al. 2015), TD(A) method (Kim al. 2023) is used train the distributional critic networks. For PD-MORL and CAPQL, the distributional critic networks are trained by reducing the Wasserstein distance between the current quantiles and the truncated one-step TD targets, as in TQC (Kuznetsov et al. 2020). The policy network, similar other RL algorithms dealing with continu- ous action spaces (Schulman et al., 2015) Haarnoja et al., 2018) Fujimoto et al.] 2018), outputs the mean and diagonal variance of normal distribution. However, in the locomotion tasks, the policy networks are modified to output only the mean value by fixing the diagonal variance order to lower the training difficulty.
standard critic networks that output scalar values
C.3.4 HYPERPARAMETER SETTINGS
We report the hyperparameter settings for the CMORL tasks Sifett--Gymmsiumm,, Locomotion) in Table and the settings for the MORL tasks (OO-Gymnasium) in Table 5
25

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameters for CMORL tasks.
CAPQL
PD-MORL
LP3
CoMOGA (Ours)
Discount factor
0.99
0.99
0.99
0.99
Length of replay buffer
1000000
1000000
100000
100000
Steps per update
10
10
1000
1000
Batch size
256
256
10000
10000
Optimizer
Adam
Adam
Adam
Adam
Policy lcarning rate
3 x 10-4
10-4
3 x 10-4
Critic learning rate
3 10
x 10
10-4
3 10-4
Multiplier learning rate
1 10-5
x 10-5
3 x 10-4
Soft update ratio # of quantiles to truncate
0.005
0.005
2
# of HER samples
3
Angle loss coefficient
10
Explore and target action noise scale
(.21
# of action samples
20
TD(A) factor
0.97
0.97
# of target quantiles
50
50
of preference samples
10
10
Local region size
0.05
0.05
Imin Swwce
(0, oo)
Amax
00
H matrix
Fisher information
Fisher information
Table Hyperparameters for MORL tasks.
CAPQL
PD-MORL
LP3
CoMOGA (Ours)
Discount factor
0.99
0.99
0.99
0.99
Length of replay buffer
1000000
1000000
100000
1000000
Steps per update Batch size
1000
10
256
256
10000
256
Optimizer
Adam
Adam
Adam
Adam
Policy learning rate
3 10-4
x 10
3 10-4
Critic learning rate
3 10-4
3 x 10-4
3 x 10-4
3 10
Soft update ratio # of HER samples
0.005
0.005
0.005
3
Angle loss coefficient
10
Explore and target action noise scale
(0.1, 0.2)
# of action samples
20
TD(A) factor
0.97
A of preference samples Local region size
10
10
0.14
0.05
Juins 9wax
(0)
Amax
00
H matrix
Fisher information
Identity matrix
D ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results for CAGrad (Liu et al. 1021 and PD-
MORL with preference alignment (Basaklar al. 2023), which is mentioned in Appendix C.3.2 and denoted by PD-MORL-NOrm, on the MO Gymnasium tasks. The training curves are shown in Figure
I3 Additionally. Figure 14 presents the Hypervolume results without normalization. The estimated
CP fronts for tasks with two objectives are visualized in Figure [5] where the hopper and ant tasks are excluded as they have three objectives. As mentioned in the appendix C.2 the estimated CP front can be obtained by rolling out trained universal policy with different preferences, and for the MO Gymnasium tasks, 20 equally spaced preferences are used. Figure 15 show that CAGrad performs similarly to our method, CoMOGA, on the overall tasks. However, as seen in Figure 8 it is evident that CAGrad is significantly outperformed by our proposed method. This observation confirms that extending existing conflict-averse MORL algorithms to CMORL is non-trivial. Additionally, PD- MORL-Norm performs better than PD-MORL in the ant and hopper tasks but worse in the humamoid and walker tasks.
26

---

Published as a conference paper at ICLR 2025
Hopper
Humanold
Hell-Chhetth
Waiter2d
Ant
Swimmer
Hyperokine
Hyporvuuue
Saps
1.2
Sparsity
Spursdy
Sparetty
S20pa
CAROL
MORL
COMOGA COurS
FD MORL Norm
Figure 13: Additional evaluation results of the MO Gymnasium tasks. Each column shows the results of the task corresponding to its title. All algorithms are evaluated at every 10 steps. The bold lines and shaded areas represent the mean and quarter-scaled standard deviation of results from five random seeds, respectively.
Hopper
Humanaid
Hall-Cheetah
Soo Hyporetmme
hyperselmee
04
34
Soa By
Spo
Figure 14: The same results as Figure 13
but with Hypervolume calculated without normalization.
CAPOL
LP3 CoMOGA
cD MORL-Norm CAGrad
CAPOL
LI3
PD-MORL-Nom
PD MORL
CoMOGA (ours)
CAGred
1.2
1.2
1.0
0.8
0.8
Energy
0.6
0.8
0.4
0.4
0.7
0.7
-0.2
-0.2
-.2
22
Velocity
Velocity
(a) Humanoid
Half-Cheetah
CAPOI
CAPOI
PD MORL
COMOGA (ours)
CAGrad
PD MORL
CoMOEA (ours)
CAGred
1.2
12
1.0
0.8
0.8
Energy
0.4
0.4
0.0
0.0
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
-0.2
-.2
0.0
0.2
0.4
0.5
0.8
1.0
2.
Velacity
Velocity
(c) Walker2d
(d) Swimmer
Figure 15: Visualization of estimated CP fronts for MO Gymnasium tasks.
27

---

Published as a conference paper at ICLR 2025
D.1 SENSITIVITY ANALYSIS OF HYPERPARAMETERS
Our method introduces two additional hyperparameters: the local region size and the number of preference samples P. as specified Algorithm [ To provide insights into the sensitivity the pro- posed method to these hyperparameters, we conducted experiments on the quadrupedal locomotion task using various values. The results are presented Figure T6
For the local region size experiments were conducted with values of 0.01, 0.02, 0.05, and 0.1. The results show that increasing the size accelerates the convergence of HV performance. However, as the size increases, the constraint curves fluctuate, suggesting that handling constraints might become less stable. Despite this, CoMOGA achieves optimal performance and satisfies the constraints in all cases, demonstrating robustness to variations in the local region size.
For the number of preference samples, experiments were conducted with values of P 1, 5, 10, and 20. The performance was lowest at P = 1, while the results for P = 5, 10, and 20 were con- sistently satisfactory. However, increasing does not always lead to better performance. This can be attributed the fact that larger P reduces the trajectory size corresponding to each preference, potentially limiting effectiveness.
Alci Hyporvelume
lec1
Sparsity
0.00
Quadrupad
0.00
0.04
6.00
0.0
52
0.5
1.
2.0
05
3.0
1.5
3.0
2.00
0.5
3.0
1.5
0.00
0.5
3.0
1.5
0.5
15
2.0
E-0.01
E-b
E-0.05
E-0.1
Tirastald
(a) Local region size,
Hypervalume
Constrant
Constraint
Constraint
0m1
25
20
Zaga
369
Saps
2xps
Steps
Stass
Threshold
(b) The number of preference samples.
Figure 16: Sensitivity experiments on the quadrupedal locomotion task.
D.2 COMPUTATIONAL COMPLEXITY ANALYSIS
In this section, we experimentally analyze the GPU memory usage and computational complexity of solving the constrained optimization problem (COP) in the proposed CoMOGA algorithm. We use Single-Point-Goal Multi-Point- Point-Gooll, and Quadruped tasks, with respective + M values of three, four, and five. The results are summarized in Table The first two rows of the table report the GPU memory usage dedicated to solving the COP, while the remaining rows present the computation times. Specifically, solving COP involves the following steps: 1) calculating the gradients of the objective and constraint functions, 2) performing gradient operations related to the Hessian matrix, and 3) solving the COP. We report the time consumed for each of these steps. The results indicate that COP-related computations contribute minimally to both GPU memory usage and computation time. However, Hessian-relaten operations account for approximately 10% of the total computation time. This suggests that future research could benefit from adopting an approximation approach similar to PPO (Schulman et al. 2017), which avoids explicit Hessian computations, to achieve faster training time.
D.3 PREFERENCE SAMPLING STRATEGIES
In this section, we provide a detailed explanation of the implementation of w ~ Ω as specified in Algorithm [ The most straightforward approach for sampling preferences to draw them from an N d'ddimensional Dirichlet distribution. Another approach is sample one-hot vectors represent strong preferences for a specific objective. In this case, the dimension with a value 1 can be selected uniformly at random. Lastly, hybrid strategy be employed, where preferences
28

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of GPU memory usage and computational time across tasks.
Multi-Point-Gori Single-Point-Goal
Quadruped (N M 5)
(N M
(N M
COP GPU Memory (MB)
20 (0.91 %) 2189 (100 %)
40 (1.71 %) 2333 (100 %)
40 (1.32 %)
Total GPU Memory (MB)
3037 (100 %)
Gradient Computation (sec)
0.097 (1.11 %)
0.156 (1.40 %)
0.192 (1.94 %)
Hessian Computation (sec)
1.271 (14.50 %)
1.975 (17.67 %)
1.324 (13.41 %)
QP Solving Time (sec)
0.012 (0.14 %)
0.014 (0.13 %)
0.014 (0. 14 %)
Training Time per Update (sec)
8.763 (100 %)
11.174 (100 %)
9.870 (100 %)
1e-1 hypervolume
Sparsity
Constrant nt
Constraint 2
Constrain: 3
Biped
0.
Staps
S0gg
Staps
160
Siar
346
Mtys
169
Ae.1 :yypervolume
Ak:3
Sparsity
Constra
Constraint
Constrairs
0.22
0.
0.5
3.0
6.5
1.0
3.0
155
0.00
1.0
0.2
100
35
200
Dinchlet
Hyfrid
Threchold
Figure 17: Training results of different preference sampling approaches on the locomotion tasks.
sampled from the Dirichlet distribution with 50% probability and as one-hot vectors with the remaining 50%.
In our implementation, we adopted the hybrid sampling strategy. To investigate the impact of differ- ent sampling methods on performance, we compared the results of training legged locomotion tasks using all three strategies. The results are summarized in Figure 17
As shown in the figure, sampling preferences from the Dirichlet distribution yields the best perfor- mance. The hybrid strategy performs almost as well as the Dirichlet sampling, while the one-hot sampling method suffers from severe performance degradation. These findings suggest that the Dirichlet distribution, which enables uniform sampling across the preference space, is the most ef- fective approach.
E COMPARISON WITH CONFLICT-AVERST MTL ALGORITHMS
To compare CoMOGA with existing conflict-averse multi-task learning (MTL) algorithms, we se- lected MGDA (Désidéri 2012). PCGrad YY al.) 2020), and CAGrad (Liu al. 2021). The aggregation of multiple gradients each method can be formulated as follows and is illustrated as in Figure 18]
(MGDA)
max min dd s.t. |[d]] S 1,
d
T 9i
(PCGrad)
d Σ of where gf 9:
gj.
11,
(32)
(CAGrad)
max max of s.t.. ||d goll s cllgoll.
d
(CoMOGA)
min |d] s.t. weellgill < of d,
where CoMOGA is a special case where no safety constraints are presented, and H is the identity matrix. Below, detail the differences between these methods and the proposed CoMOGA.
MGDA. This method determines gradient direction that maximizes the improvement of the objective with the smallest increase. However, MGDA has a limitation in that it does not allow the incorporation of task preferences.
29

---

Published as a conference paper at ICLR 2025
81
81
82
82
82
82
MGDA
PCGrad
CAGrad
CoMOGA
Figure 18: Visualization of gradient aggregation process in conffict-avesse MTL algorithms.
PCGrad. This method projects each gradient remove components in conflict with other gradients before aggregating the projected gradients. However, like MGDA, PCGrad does not support the incorporation of task preferences.
CAGrad. This method be viewed of task preferences by setting go = >; Wigi
extension of MGDA that enables the inclusion
CoMOGA. Finally, the proposed method, CoMOGA, transforms each objective into con- straints and solves quadratic programming (QP) problem obtain the gradient.
30


Reviews:
[{"review_id": "ANmevG2IpE", "reviewer": "Reviewer_w9r3", "summary": "The paper presents an algorithm for multi objective reinforcement learning under constraints. The authors first identified that \nmaximizing a specific return within the trust region is equivalent to minimizing $||\\Delta \\theta||^2_H$$ under a constraint that the \nreturn increases (if the solution exists). Therefore, the multi objective RL under constraints can be formulated as a problem that \nminimizes the induced norm of $\\Delta \\theta$ under a bunch of constraints that correspond to the multiple task returns and \nconstraints. By using a first-order approximation of the constraints, the problem reduces to a quadratic programming problem. To avoid \nthe violation of the constraint due to the inaccurate linear approximation, the authors propose to remove the return constraints once \nthe violation happens. \n \nThe above can be generalized to multiple objectives with preferences, and then to learn a universal policy (i.e., a policy that \nis conditioned on any given preference $w \\in \\R^N$), the authors propose to learn a policy that minimize the expected KL divergence \nbetween the learned policy conditioned on a specific preference and the updated policy that solves the above quadratic programming. In \nother word, it amortizes the optimization into a learned conditional policy. \n \nThe authors conduct experiments on both synthetic toy and constrained RL problems in simulation, demonstrating that their method \nresults in superior performance (in terms of optimizing the objective and satisfying constraints) compared against prior methods. The \nauthors also provide a convergence guarantee for the proposed method.", "questions": "1. I think there is a gap from (3) to (4) where the authors generalize to multiple objectives. In the multiobjective case (without \nconstraints), are (2) and (3) equivalent only when a solution exists?  If so, is it the case that the norm constraint can always  \nbe satisfied, while the monotonic improvement constraints are not? \n \n2. The scaling of the gradient essentially adds the norm constraint back, but if this is the case, why not start with that \nconstraint and use the form (2)? Like max_d d^\\top g_i, s.t., b^\\top d + J < d_k, ||d||_H \\leq \\epsilon? Is it because this form is not \neasily solvable due to the norm constraint (difficult to estimate H^{1/2})? If so, the authors could explicitly mention this. \n \n3. The authors claim that using (6) can help correct the policy when it avoids any constraint due to solving (5) which uses linear \napproximation. But isn't (6) also using linear approximation, and the solution to (5) should also satisfy constraints in (6) if there \nexists any feasible solution? Can the authors elaborate on what this means exactly? \n \n4. The first sentence of the abstract asserts that an RL agent needs to consider multiple objectives.  That's not necessarily the  \ncase - the reward hypothesis states that an agent's objective can always be summarized with a single scalar reward function.  How  \nwould you motivate this paper to someone who fundamentally disagrees that an agent needs to consider multiple objectives separately \n(as opposed to aggregating them to explicitly encode their tradeoffs)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of interest has broad application. \n \n2. The proposed method is straightforward yet well motivated. Specifically, the transformation of objective to constraint is \nconvenient. \n \n3. The presentation is clear, and the figures illustrate the main ideas well. \n \n4. The empirical results demonstrate that the proposed method consistently outperforms existing methods.", "weaknesses": "One main weakness is the computational efficiency of the proposed method, which might hinder the practical usage of the proposed \nmethod. First it requires computing/storing gradients to constraints/rewards, which can be quite large if N and M are large. Second, it \nrequires estimating terms like $x^\\top H x$ where x is the fisher information matrix, while this does not in general require computing \n$H$ explicitly, it sill requires computing the inner product like $x^\\top \\nabla \\log \\pi$ (which might need extra space). The authors \ndid not elaborate on this aspect, and there is no time complexity evaluation in the experiments or analysis sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lmeaPBwnER", "reviewer": "Reviewer_BUGD", "summary": "This paper presents CoMOGA, an algorithm designed to optimize multiple objectives while satisfying constraints. CoMOGA converts objectives into constraints and combines the gradients from both objectives and constraints. Theoretical analysis provided in the paper demonstrates that CoMOGA can achieve a local Pareto optimal policy. Empirical tests show that CoMOGA surpasses competing methods in achieving better coverage of the constrained-Pareto front.", "questions": "- Why does CoMOGA perform significantly better than the combination of CAGrad and Lagrangian in Section 6.5? Could the authors expand on this?\n- Why is the combination of CAGrad and Lagrangian treated as an ablation study rather than a baseline?\n- How are preferences sampled in Line 328, for example, are they uniformly sampled? How does this affect the distribution of results?\n- Are the critics trained using the same targets, such as Retrace, across all methods in the experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem addressed is highly relevant and has practical real-world applications.\n- The paper is well-written and easy to understand.\n- The experiments are thorough, and the evaluation metrics are clearly defined and articulated.", "weaknesses": "- The section on gradient aggregation could be improved by including comparisons with similar MORL methods like CAGrad and PCGrad.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EyPldNmGdf", "reviewer": "Reviewer_em3T", "summary": "This paper presents the Constrained Multi-Objective Gradient Aggregator (CoMOGA) algorithm for constrained multi-objective reinforcement learning (CMORL). It transforms the CMORL problem into a constrained optimization framework and addresses the optimization objective within a local region, using constraints designed to enhance the original objectives in proportion to preference values. The authors also provide a convergence guarantee in the tabular setting. Experimental results across several environments demonstrate that CoMOGA achieves constraint satisfaction.", "questions": "1) Can the authors analyze the computational complexity and the rate of convergence of the algorithm after converting multiple objectives into constraints?\n\n2) Does the algorithm converge in continuous space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## Originality and Significance\n1) This paper presents a constrained multi-objective optimization algorithm, CoMOGA, which transforms objectives into constraints. The idea appears to be novel.\n2) Multi-objective tasks and constraint optimization are critical issues in reinforcement learning, significantly enhancing the safety of reinforcement learning and its applicability in real-world scenarios.\n\n## Quality and Clarity\n1) The paper is well-written and easy to read.\n2) This paper includes both theoretical analysis and detailed empirical results.\n3) Multiple figures are provided, which are helpful in understanding the main concepts presented in the paper.", "weaknesses": "## Framework\n1) The core idea of this paper is to avoid gradient conflicts by converting multi-objectives into constraint terms. However, during the optimization process, due to factors such as the linear approximation of the objectives and bias in the value function estimation, it is difficult to avoid constraint violations by the constraint terms of the objective, thus causing gradient conflicts. This may undermine the authors' core contribution.\n2) Transforming multiple objectives into constraint terms can complicate the constraint boundaries, thereby increasing the difficulty of finding an optimal policy that satisfies these constraints.\n3) It is not clear whether this transformation yields positive gains in computational complexity and convergence rate.\n\n## Experiments\n1) There is a lack of sensitivity analysis concerning hyperparameters.\n2) The fairness of the ablation experiments is questionable. The constraint optimization algorithm employed for the CAGrad algorithm in the ablation experiments is the Lagrangian method, which is inconsistent with the optimization algorithm used in CoMOGA. The discrepancy in the experimental results may arise from the optimization algorithm itself, thus failing to accurately assess the effectiveness of the core components in CoMOGA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYxw7potXy", "reviewer": "Reviewer_6dPa", "summary": "This paper introduces a novel multi-objective reinforcement learning (RL) approach that incorporates safety constraints. The authors present a gradient conflict management technique, which formulates multi-objective RL as a constrained optimization problem. This formulation effectively prevents policies from getting trapped in local optima, particularly in tabular settings. Additionally, it enables the seamless integration of safety constraints and facilitates an efficient policy update mechanism.", "questions": "1. **Limited Practical Implementation Details in Constrained Optimization Problem (COP)**: The authors propose solving constrained multi-objective reinforcement learning (CMORL) as a constrained optimization problem (COP), which theoretically offers strong convergence guarantees. However, the paper lacks detailed insights into the practical implementation of this approach. In contrast, methods like TRPO provide clear derivations of practical algorithms from theoretical foundations, taking into account finite sample sizes and arbitrary parameterizations in deep reinforcement learning. Can you include a similarly detailed derivation to bridge the theoretical framework with practical implementation?\n\n2. **Inclusion of Pareto-Dominated Points in Figure 7**: In Figure 7, numerous points are shown that are Pareto-dominated by others. This raises questions about the relevance of including these dominated points, as they do not contribute to optimal trade-offs and may complicate the interpretation of the results. What is the rationale behind the inclusion of these Pareto-dominated points, and how do they add value to the analysis presented in this figure?\n\n3. **Inconsistencies in Hypervolume (HV) Reporting in Figure 8**: In Section 6.4, Why do the results for Hopper and HalfCheetah in Figure 8 differ significantly from the original results reported in PD-MORL?\n\n4.***Practical Implementation Challenge: High Dimensionality and Computational Expense:***\nIn the theoretical work, Equation (9) utilizes the inverse Fisher information matrix over the policy parameter theta, potentially resulting in high dimensionality and significant computational expense in practical applications. How does CoMOGA handle this challenge in its practical implementation?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.***Theoretical Contribution:***\n\nThe paper presents an interesting theoretical contribution, providing a comprehensive convergence analysis.\n\n2.***Experimental Coverage:***\n\nThe study includes a wide range of experiments within the MORL domain, offering an extensive evaluation of the proposed method.", "weaknesses": "1.***Constrained Multi-Objective RL as a Constrained Optimization Problem:***\n\nThe authors propose addressing constrained multi-objective reinforcement learning (CMORL) by formulating it as a constrained optimization problem (COP). While COP offers robust theoretical guarantees for convergence, the paper provides limited insights into its practical implementation, which could impact the reproducibility and real-world application of the proposed method.\n\n2.***Inconsistencies in Hypervolume (HV) Scale and Reporting:***\nFigure 8 presents hypervolume (HV) results on a significantly smaller scale compared to previous work. For example, in PD-MORL, HV values are reported across training steps up to 10 $e^6$, while in this study, the results are shown only from 1$e^6$ up to 3$e^6$ steps, Which may not reach the convergency. \n\n3.***Different unit to benchmark in HV:***\n With PD-MORL reporting HV values in the unit of 1$e^6$ , whereas this paper uses a unit of 0.1 for HV. These inconsistencies create confusion and lead to the perception that PD-MORL performs worse in this comparison than reported in the original study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)