Efficient Discovery of Pareto Front for Multi-Objective Reinforcement Learning
Authors: Ruohong Liu
Yuxin Pan
Linjie Xu
Lei Song
Pengcheng You
Yize Chen
Jiang Bian
Venue: ICLR 2025 Poster (2025)
Keywords: multi-objective reinforcement learning
constrained reinforcement learning
Primary area: reinforcement learning

Abstract: Multi-objective reinforcement learning (MORL) excels at handling rapidly changing preferences in tasks that involve multiple criteria, even for unseen preferences. However, previous dominating MORL methods typically generate a fixed policy set or preference-conditioned policy through multiple training iterations exclusively for sampled preference vectors, and cannot ensure the efficient discovery of the Pareto front. Furthermore, integrating preferences into the input of policy or value functions presents scalability challenges, in particular as the dimension of the state and preference space grow, which can complicate the learning process and hinder the algorithm's performance on more complex tasks. To address these issues, we propose a two-stage Pareto front discovery algorithm called Constrained MORL (C-MORL), which serves as a seamless bridge between constrained policy optimization and MORL. Concretely, a set of policies is trained in parallel in the initialization stage, with each optimized towards its individual preference over the multiple objectives. Then, to fill the remaining vacancies in the Pareto front, the constrained optimization steps are employed to maximize one objective while constraining the other objectives to exceed a predefined threshold. Empirically, compared to recent advancements in MORL methods, our algorithm achieves more consistent and superior performances in terms of hypervolume, expected utility, and sparsity on both discrete and continuous control tasks, especially with numerous objectives (up to nine objectives in our experiments).

Published as a conference paper at ICLR 2025
EFFICIENT DISCOVERY OF PARETO FRONT FOR MULTI- OBJECTIVE REINFORCEMENT LEARNING
Ruohong Liu
Yuxin Pan
University of Oxford Oxford, UK siohong.1lu@eng.. ac.ukk
The Hong Kong University of Science and Technology Hong Kong, China yuxin. pane connect. ust.
Linjie Xu
Lei Song & Jiang Bian Microsoft Research Asia Beijing, China {lei. song, jiang.bian) wwiiaoootmorooottt.oooooooooooom
Queen Mary University of London London, UK
linjie. xu@qmul.c.cm ac. uk
Pengeheng You
Yize Chen
Peking University
University of Alberta
Beijing, China pcyouêpku.edus edu.
Edmonton, Canada yize. ceazececceeualbrra..cc
ABSTRACT
Multi-objective reinforcement learning (MORL) excels at handling rapidly chang- ing preferences in tasks that involve multiple criteria, even for unseen preferences. However, previous dominating MORL methods typically generate a fixed policy set or preference-conditienn
policy through multiple training iterations exclu- sively for sampled preference vectors, and cannot ensure the efficient discovery of the Pareto front. Furthermore, integrating preferences into the input of policy value functions presents scalability challenges, in particular as the dimension of the state and preference space grow, which can complicate the learning process and hinder the algorithm's performance on more complex tasks. To address these issues, we propose a two-stage Pareto front discovery algorithm called Constrained MORL (C-MÖRL), which serves seamless bridge between constrained policy optimization and MORL. Concretely, set of policies are trained in parallel in the initialization stage, with each optimized towards its individual preference over the multiple objectives. Then, to fill the remaining vacancies in the Pareto front, the constrained optimization steps are employed to maximize one objective while constraining the other objectives to exceed predefined threshold. Empirically, compared to recent advancements MORL methods, our algorithm achieves more consistent and superior performances in terms of hypervolume, expected utility, and sparsity on both discrete and continuous control tasks, especially with numerous objectives (up to nine objectives our experiments). Our code available https::/qithuut com Sttp://oooLLii/CMMM00
INTRODUCTION
In many real-world control and planning problems, multiple and sometimes even conflicting ob- jectives are getting involved. Such situations necessitate striking a better trade-off among these decision-making goals (Roijers et al. 2013 Hayes et al.) (022). For instance, in industrial control sce- narios (Salvendy) 2001 Wang et al. 2023). , maximizing utility and minimizing energy consumption
are of particular interest as objectives to be optimized. Since different decision makers have heteroge- neous preferences over these objectives, there may exist multiple Pareto-optimal policies (Roijers) et al. 2014), Classical reinforcement learning (RL) methods typically involve training individual policies exclusively to align with each preference weight vector over multiple rewards (Nagabandi
et al. 2018 Gupta et al.) 2018). Yet it may lead to enormous computational burden due to the overly dependence on the model retraining and fine-tumin meee-tunng stages. Moreover, such policies are hard
"Corresponding author.

---

Published as a conference paper at ICLR 2025
to directly generalize transfer to newer tasks (Cobbe al. 2019 Taiga et al. 2022). Therefore, the multi-objective reinforcement learning (MORL) paradigm has drawn significant attention by reformulating these tasks for optimizing towards multiple criterion (Xu et al. 2020; Basaklar et al..
2022 /hhu et al. 2023). MORI. aims to obtain either a single policy readily adapted to different
set of policies (Zhao & Grover 2024 Felten et al. 2024; Röpke et al. 2024 Kim et al. 2024) aligned with their respective preferences.
preferences (Felten al. 2023b 2022 [Th et al., 2017)
One prevalent category of MORL approaches is to train a single preference-condiiioned policy (Yang) ct al. 2019 [Basaklar et al. 2022). They utilize a weight vector to quantify preferences for different objectives and incorporate this weight vector as part of the input to the policy network. However, such approaches often struggle with scalability, since for nigh-dimensional environments the weight space extends exponentially as the number of objectives increases. By comparison, training a restricted set of policies to align with the set of sampled preference vectors can circumvent the scalability issue to some extent. Yet could be scarcely possible to fulfill the demand of covering the entire Pareto frontier. In addition, although some works focus on boosting sample efficiency (Wiering) ct al. 2014 Alegre et al. 2023). , they may still rely on learning the environment dynamics with extra training time, while inaccurate learned dynamics easily affect multiple RL objectives' performance. Evolutionary approach and user feedback are also integrated to find the Pareto set approximation (Xu) al. 2020 Shao et al. 2024), while additional prediction models are needed to guide the learning process. Although these methods demonstrate promising performance in tasks with simpler dynamics or a limited number of objectives (typically two to five), they often struggle to scale effectively with respect to a greater number of objectives or larger state and action spaces.
In summary, existing works mainly suffer from three aspects: (i) low training efficiency; (ii) hard to cover the complete Pareto front; (iii) inability to maximize utility for any given preference. To address these challenges, we propose novel constrained optimization formulation for improved computation complexity and stronger coverness of the Pareto front. To be specific, we adapt the constrained policy optimization algorithms (Achiam et al. 2017: Liu et al. 2020), and reformulate the MORL problem with designed constraints on policy performance over multiple objectives. Our approach of training the policy set consists of two stages. In the Pareto initialization stage, we train several initial policies in parallel based on fixed preferences until convergence. In the Pareto extension stage, we first select diverse policies based on their crowd distance and employ constrained update steps to the initial policies individually. In each optimization step, optimize a specific objective while constraining the expected returns of other objectives. Through this approach, we can extend the Pareto front in various objective directions. For the third research gap, we introduce Policy assignment, which ensures that for any given preference, we assign a policy from the Pareto set that maximizes its utility.
Our algorithm achieves favorable time complexity characteristics and derives a high-quality Pareto front, as indicated by the following observations. Firstly, training several initial policies based on fixed preferences is efficient. In addition, the adoption of constrained update steps allows for rapid adjustment of the initial policy, leading to the derivation of new solutions on the Pareto front. Regarding the second research gap concerning the complete Pareto front, different from the meta-policy approach that relies on single initial policy (Chen et al. 2019), our method can extend multiple policies selected from initial policies based on their crowd distance to enhance diversity while promoting better performances. Intuitively, a larger crowd distance indicates that the corresponding policy appears on a sparser area on the Pareto front, therefore, extending such a policy is more likely to fill the Pareto front. While sharing similarities with our formulation, one category of multi-objective optimization (MOO) approach, epsilon-constreen
methods solves a MOO problem by converting it several single-objective constrained optimization problems (Laumanns) et al. 2006; Van Moffaert & Nowé 20044.. However, the running time of such a method is exponential in problem size, rendering it impractical for MOO problems with numerous objectives. In contrast, cee--ddstaneeebaee
policy selection eases such burden, exhibiting linear complexity. and can efficiently solve MORL tasks with numerous objectives. Our main contributions are as follows:
We propose C-MORL, a two-stage policy optimization algorithm that enables rapid and complete discovery of the Pareto front. By taking a novel constrained optimization perspective for MORL, our Pareto front extension method can easily handle complex discrete or continuous MORL tasks with multiple objectives (as demonstrated with up to nine objectives in our experimental evaluations).
To empirically solve C-MORL without extra computation such as in the epsilon-constraint method, we propose an efficient interior-poin cortor--oin--bssee approach for finding the solution of a relaxed formula- tion, which can guarantee the derivation of Pareto-optimal policies under specified conditions.
2

---

Published as a conference paper at ICLR 2025
To validate the efficacy of C-MORL, employ array of MORL benchmarks for both continuous and discrete state/action spaces across various domains, such as robotics control and sustainable energy management. C-MORL consistently achieves up to 35% larger hypervolume and 9% higher expected utility in MORI. benchmarks than the state-of-the-ar baselines, indicating the discovery of broader Pareto front given any preferences.
2 RELATED WORK
Prior trials on tackling multi-objective RL fall into training single preference-condttonne policy or multi-policy. Typical preferences as part of network inputs and utilizes the weighted-sum scalarization of the value functions (or advantage functions) to optimize the policy (Van Moffaert et al. 2013; Parisi et al. 2016 Yang
policy approaches adopt policy that takes
et al. 2019
Zhang & Golovin 2020 Basaklar et al. 2022 Lu et al. 2022 Hung et al. 2022 Zhu
et al.
2023 Lin et al.) 2024). In the evaluation stage, agents can execute corresponding solutions
based on users' desired preferences (Yang et al. 2019 Basaklar et al. 2022). With the shared neural networks, gradients from different tasks can interfere negatively, making learning unstable, leading to
imbalanced performance across the entire preference space, and sometimes even less data efficient.
Instead of training a single policy agent, multi-policy approaches train finite set of policies to approximate the Pareto front (Abels et al.) 2019; Xu et al. 2020; Alegre et al. 2022). In the evaluation stage, the policy in policy set that maximizes the utility of the user (i.e., the weighted sum of objectives) is chosen (Friedman & Fontaine 2018). Rather than evenly sampling from preference space, (Xu et al., 2020) proposes an efficient evolutionary learning algorithm to find the Pareto set approximation along with prediction model for forecasting the expected improvement along vector- valued objectives. Yet the performance highly depends on prediction model's accuracy, and it is challenging to recover the Pareto front due to the long-term local minima issue. This method also suffers from low training efficiency due to the exponential complexity involved in repeatedly calculating the quality of the virtual Pareto front during the training process. (Kyriakis & Deshmukh 2022) designs a policy gradient solver to search for a direction that is simultaneously an ascent direction for all objectives. (Alegre et al. 2022) firstly trains set of policies whose successor features form a e-CSS (convex coverage set), then utilizes the generalized policy improvement (GPI) algorithm to derive a solution for a new preference. (Alegre et al. 2023) further introduces novel Dyna-style MORL method that significantly improves sample efficiency while relying on accurate learning of dynamics. (Röpke et al., 2024) discovers the Pareto front by bounding the search space that could contain value vectors corresponding to Pareto optimal policies using an approximate Pareto oracle, and iteratively removing sections from this space. Our proposed C-MORL distinguishes from their crridd-and-conquee scheme by a novel policy selection procedure along with explicitly solving a principled constrained optimization.
Our approach bears connections with (Chen et al. 2019) and (Huang et al., 2022), while distinguishing itself in terms of both the training process and objectives. Compared to the meta-policy approach, adaptation process is not necessary in the evaluation phase of our method. When presented with an unseen preference, the policy in the Pareto front with the highest utility is chosen as the surrogate execution policy. In contrast to (Huang et al., 2022 Kim et al. 2024), we do not aim solving a constrained RL problem for training the working policy under specific preference. Rather, we propose leverage the constrained optimization steps to fill the complete Pareto front with enhanced flexibility (Liu et al. 2020; Xu et al., 2021). and design extension and selection algorithms to explicitly promote diverse policies. Constrained optimization techniques are widely adopted to solve multi-objective optimization (MOO) problems. Apsilon-constraing methods are a category of MOO techniques that involve pre-defining a virtual grid the objective space and solving single-ohjective problems for each grid cell, where the optimum of each problem corresponds to Pareto-optimal solution (Laumanns et al.] (2006). Instead of pre-defining virtual grid, this work proposes crowd distance based policy selection to address the exponential complexity in the epsilon-constraint method.
3 PRELIMINARIES
3.1 MORL SETUP
In this work, we adopt the general framework of a multi-objective Markov decision process (MOMDP), which is represented by the tuple < S,A, P, R1 f.r >. Similar to standard MDP, at each timestep t, the agent under current state s, S takes an action at a A, and transits into a new state 8,+1 with probability 1(s a.). One notable characteris- tie of MOMDP is that for n different objectives, the reward is n-dimensional vector r.
3

---

Published as a conference paper at ICLR 2025
[R1 (st, at), R,),,,,))...,,,1(((,..... E R". For any policy 9, it is associated with vector of expected return, given as G [G C2...., G+] where the expected return of the gth objective is given as G. = Bat such returns are observable.
1>, R (st, at); for some predefined time horizon. We assume
The goal of MORL is to find a policy so that each objective's expected return in G can be optimized. In practice, since training RL typically requires scalar reward to interact with the training agent, and co-optimizing multiple objectives is hard to achieve an ideal tradeoff, especially in situation where objectives are competing against each other. To that end, denote Q = {w E Q2 -i= Wi = 1, 2 0} as the preference vector. We use the preference function (r) to map reward vector r(s, a) to scalar utility given w fw (r(s, a)) = w rss, Our goal is then to find a multi-objective policy (alss, w) such that the expected scalarized return w Ge is maximized.
3.2 PARETO OPTIMALITY
In MORL, policy that simultancously optimizes all objectives does not exist. This holds whenever either of the two objectives is not fully parallel to each other. Thus, set of non-dominated solutions is desired. We say policy x is dominated by policy x' when there is no objective under which x' is worse than i.e., G. < GT for Vi E (1, n). not dominated by any other policies. The Pareto set is composed of non-dominated solutions, denoted as pIp The corresponding expected return vector GT of policy π E III forms Pareto front P. In this paper, an element in solution set Xpp refers to a policy along with its corresponding expected return vector (T, G However, in many complex real-world control problems, obtaining the optimal Pareto set is challenging, and is becoming even more difficult considering the sequential decision- makinn nature of RL problems. Thus the goal of MORL is obtain a Pareto set P to approximate the optimal Pareto front.
A policy is Pareto-optimal if and only if it is
With finite Pareto set, we can define the Set Max Policy (SMP) (Zahavy et al. 2021 of given preference vector:
Definition 3.1. (Set Max Policy). Denote IIp to be Pareto set. Then, Set Max Policy (SMP) is the best policy in the set II for a given preference vector w: T SMP = max (G
The notion of SMP enables the direct assigmment of a surrogate execution policy given an unseen preference in the evaluation phase. Fig. [illustrates the determination of SMP.
To evaluate a Pareto set pp, there are three evaluation metrics introduced to compare the quality of the Pareto front and utility achieved by the underlying algorithm: (i) hypervolume, (ii) expected utility, and (iii) sparsity (Hayes et al. 2022). The details of these evaluation metrics are provided in Appendix E.3
Definition 3.2. (Crowd Distance). Let P be a Pareto front approximation in an n-dimensional objective space. Also denote G3 as the ascending sorted list with a length of [P| for the jth objective values in P. Given the jth solution, and suppose the sort sequence of P P()) in G3 is k then the Crowd
Distance of solution P(j) is D(P(j)) Σn=
-
G2 mas Gammin
The crowd distance is a measure of how close an individual is its neighbors (Deb et al. 2002). OPTIMIZING CONSTRAINED MORL In this section, we start from converting the MORL problem as constrained MDP (CMDP) while guaranteeing the local optimality under such formu- lation. Next, we present the conditions under which a feasible solution to the CMDP problem qualifies as a Pareto-optimal solution. Then, to effectively prove that under
Piaue From
Parca Sakuiz
Reformec
Figure 1: Visualization of metrics. (a) Hypervol- ume, reference point, and example of crowd distance calculation. As example, the crowd distance of ma calculated based on the expected return its neighbors т. and w.. as well as the extreme solutions on the two objectives. (b) Given a preference vec- tor, the corresponding expected return is calculated by selecting the set max policy from Pareto front solutions.
solve the CMDP problem, primal-dual formulation, despite its non-convexity, the CMDP problem has zero duality gap, i.e., it can be solved exactly in the dual domain, where it becomes convex (Paternain al. 2019).
Constrained RL problems are typically formulated as constrained MDPs (CMDP) (Altman 2021). A
4

---

Published as a conference paper at ICLR 2025
CMDP is defined by the tuple < A. P. R1:n- Y the reward function Rs( ar) of the 1th objective that is being optimized, and the constraint- specific reward functions (s, a.) a.) (1=jy of other objectives. d == {di (1=))) represents the corresponding thresholds of constraint-specifee reward functions.
>. The reward function of CMDP includes
The optimal solution to CMDP is a policy that maximizes expected return of the objective being optimized, while ensuring the expected returns of other constraints satisfy their baseline thresholds. We adapt to the following constrained RL formulation to ensure when we optimize π for G7. the returns of other objectives are not seriously hampered:
max Gi
s.t. G. 2 d, i= l=.,..., n, i I.
(I)
Assumption 4.1. The value (s),,,,=,,..11 ,..,, i + is bounded for all policies E Ilp.
Assumption 4.2. Every local minima of GJ is a feasible solution.
Assumption justiiiiss the use of gradient-based algorithm for solving CMDP can converge with general constraints. As long as the threshold d are chosen beforehand such that for initializations, there are G7 2 d, i + then starting points for problem Eq. i always feasible.
Note that implementing the Pareto extension aims to expand the Pareto front in various objective directions, starting from the initialization points of problem Eq. [ In this regard, the solution derived from Eq. [ should contribute to the Pareto front, meaning it must be a Pareto-optimal solution. Therefore, it is crucial to set proper constraint values d;, i = 1,..., n, i * L. Accordingly, we present the following proposition that formalizes the criteria for specifying appropriate constraint values:
Proposition 4.3. Let G, denote the ascending sorted for the ith objective values in P, and suppose the sorted sequence of the initial point P. in P. is k. If 2 -k for all i ...... n, + t, then the optimal solution of problem Eq. 7 is Pareto-optimal solution.
See Proof in Appendix B Proposition 4.3 provides the condition of which the feasible solution of
problem Eq. iss Pareto optimal solution. Fig. 2 visualizes the constraint value setting criteria. In the next section, we will further propose a more practical method of specifying constraint values d: for all i I..... n, i i L.
Having established the criteria for selecting appropriate constraints, we now turn our attention the challenges associated with solving the constrained formulation Eq. [ which is untractable due to the nonconvex and multiple-step formulation. To address such issue, constrained CMDP formulation of MORL problem Eq. can be solved by using the Lagrangian relaxation technique once converted to an unconstrained problem. Denote GT 1mml = [G,, ... G.] ; + 1. And define the Lagrangian :: = GT A GT,)))) where X 2 0, 1,....n,, n, i * I the Lagrange multiplier. The resulting equivalent problem is the dual problem
D a max max L(A,
(2)
Denote the projection operator as 1'x, I'z, which projects A and π to compact and convex sets respectively. can be shown that at optimization iteration step r, for our CMDP formulation of the underlying MORL problem, iteratively working on X,, H, with step size 1/1, 772 is guaranteed to reach the local minima of the unconstrained problem Eq. (Bokkr 2009: Tessler et al. 2018):
Proposition 4.4. Under mild conditions and by implementing the update rules as followed for X and T: Art1, = [x(Ar, T) x
Tr+1 = = V= TT )]; such iterates (An, Tn) converge to a fixed point of MORL policy almost surely.
See proof in Appendix C Indeed, (Paternain et al.) 2019) shows that there is zero duality gap between the original CMDP formulation and its dual formulation. Motivated by such property. if we denote the optimal solution of the original MORL problem Eq. aas P*, can alternatively work on finding the solution of the dual formulation Eq. 2 to derive the optimal MORL policy.
Further, justify the use of interior point method adopted by our C-MORL-IPO described in Section 4 the following Theorem helps connect the solution procedure via the log barrier method with the optimal solution of the original C-MORL problem:
Theorem 4.5. Suppose that R4 is bounded for all i = 1, 1,..., n and that Slater's condition holds for problem Eq. [ Then, strong duality holds for 7 i.e., P = D* Define the logarithmic barrier function for each constraint ((G;) log(G? -BG with t as a hyperparameter. If the optimal policy for C-MORL is strictly feasible, the maximum gap between Eq. Mand solving it via the interior point method is bounded by 7
n-1
5

---

Published as a conference paper at ICLR 2025
The above Theorem indicates that safely solve C-MORL problem with tunable parameter t the log barrier. With such properties, we are ready to show our design strategies to more efficiently discover the Pareto front via solving C-MORL involving multiple constraints on the return of heterogeneous preferences.
Sapeard Zarfice
Whirbe E m4
Figure 2: Visualization of criteria for specifying con-
5 CONSTRAINED
HORL-BASED PARETO FRONT EXTENSION
straint values. Tr denotes initial point, The expected return GT (G**h) of solution Pa (P)) in objective (2) is the (k- - 1)th value list G1 (G2). respectively. Therefore, specifying constraints values di 2 G
In this section, introduce our two stage-design for the construction of the Pareto set via the pro- posed C-MORL along with the selection scheme for the second extension stage.
and d2 2 G is sufficient for the feasible solution of corresponding Eq. too be Pareto-optimal solution.
5.1 PARETO INITIALIZATION
Nustin 5.3
Suttes 51
Suctin 5.4
Pally Solection
Figure 3: Procedure of two-stage C-MORL. Pareto initialization: training several initial policies to derive the initial solution set Ninit. Pureto extension: iteratively implementing policy selection and Pareto extension with constrained policy optimization toward desired Pareto extension directions the objective space. Policy assignment: given preference w, the surrogate execution policy selected from the Pareto set based on Eq. 3.]
As shown in Fig. 3 during the Pareto initialization stage, we first train and collect set of initial policies. Each policy π corresponds to a pre-known preference vector w, as we train policy и using the objti-obeective policy gradient algorithm (Xu et al., 2020) to maximize the return under this particular w. Specifically, we also extend the value function to a vectorized version with vectorized target value (s). Then in the policy update iterations, we utilize the vectorized advantage function A (S;, ar) to implement policy gradient updates: VoJ(O,w) E wi V (0) T Σ w (s; at) V log To (ar(()))
By sampling diverse preference vectors to guide the training of initial policies, we obtain an initial solution set Xinit-
important note that the preference vector is solely used for guiding the training of these initial policies, and the resulting solutions are sereerence-irrleevnne initialized policies can be further trained without specifying preference. Any policy in the set X can also be assigned to and evaluated by any new preference in the policy selection stage. Note that in the Pareto initialization stage, we also maintain a solution buffer to enhance policy diversity.
meaning the
5.2 POLICY SELECTION
Direct implementation of the Pareto extension based on policies from Xenit could encounter several limitations. First, some policies may not lie on the Pareto front, making their extension less effective for discovering new Pareto-optimal solutions. Furthermore, the distribution of Pareto-optimal policies along the Pareto front may be uneven. Random selection for extension could result in trajectory overlap and subsequent inefficiencies, or leaving regions of the Pareto front unexplored. Inspired by multi- mujti-objeccive optimization algorithms (Deb et al. (2002), we design policy selection schemes before and during the Pareto extension stage respectively.
The Pareto-optimal policies are selected based their crowd distance. A larger crowd distance indicates that the corresponding region on the Pareto front is relatively unexplored, and therefore, may hold greater potential for augmenting the Pareto front during the iteration of the Pareto extension. We sort all Pareto-optimal policies according to their crowd distance value and select the top N policies with the greatest crowding distance to construct Xextenssio for the subsequent Pareto extension iteration. C-MORL selects policies after the Pareto initialization stage and every K'
к steps
during the Pareto extension stage, where K is the total number of Pareto extension step, K" the number of constrained optimization steps between two iterations of policy selection. The detailed policy selection procedure is summarized as Algorithm iin Appendix A

---

Published as a conference paper at ICLR 2025
5.3 PARETO EXTENSION
In this Pareto extension stage, achieve the goal of filling the Pareto front from selected solutions Nestension toward different directions by solving constrained optimization on the selected policies, as is shown in the Pareto Extension stage in Fig. In each constrained optimization step, we optimize one of the objectives, denoted as I. By performing such optimization on the initial policy for all listed objectives, we are able to obtain a set of policies that approximate the Pareto front towards various directions while fully utilizing currently collected initial policies.
In Section we present Proposition 4.3 as a sufficient condition for specifying constraint values that ensures the feasible solution of Eq. is Pareto-optimal solution. However, this Proposition impractical for several reasons. First, the condition is too strict and may lead the exclusion of discovering some potential Parcto-optimal solutions. Second, it necessitates evaluating the expected return of all policies for non non-dominated sorting at each step of the constrained optimization, which is inefficient. Therefore, we propose alternative constraint specification method that utilizes only the expected return of the policy from the most recent step. Specifically,, we consider solving the following problem which return constraints are controlled by a hyperparameter B3 E (0, 1):
#1,1, arg max | G. 2 "G." i 1i,,,..,,,, 4} xElls
(3)
where GT is the expected return of the 5th objective in the last constrained optimization step. which is indexed by r for the current iteration. Next, we introduce practical methods to solve Eq. S C-MORL-IIPO Constrained Policy Optimization (CPO) is a widely used general-purpose policy search algorithm for constrained RI. (Achiam al. 2017). While the CPO algorithm ensures monotonic policy improvement and guarantees constraint satisfaction throughout the training process, inner-loop optimization is required when there are multiple constraints. Therefore, CPO hard to handle more than two MORL objectives. To overcome this issue, inspired by (Xu et al. 2021 Liu
jtt al. 2020), we also propose find a satisfiable solution efficiently for Eq. [3] via the interior point method (IPO), which is a primal type approach and holds the promise of finding a solution closer to
the original C-MORL as shown Theorem 4.5 Given G. and the the defined log barrier in 4.5 we
then convert problem Eq. D into unconstrained optimization problem:
max GT
(GG))
(4)
The detailed procedures of solving Eq are provided in Appendix F In practice, IPO is more robust in stochastic environments, and larger t would guide to a solution with higher rewards yet with more computation costs. Without the requirement of calculating the second-order derivative, C-MORL-IPO more computationally efficient than trust-region- based method for CPO updates (Achiam et al. 2017), which we term as C-MORL-CPO and compare it in Appendix F
We note that C-MORL can achieve a better approximation of the Pareto front by design, which is different from previous CMDP approaches (Abdolmaleki et al., 22200] Huang et al. 2022). Such type policy improvement stage to
of methods typically have an explicit learning- or optimization solve for particular preferences. Implementing C-MORL helps directly move away from dominated solutions in the Pareto front, so that the hypervolume can be directly optimized, while resulting policies are also more generalizable. The following proposition analyzes the time complexity of C-MORL, demonstrating its linear time complexity with respect to the number of objectives, making it suitable for MORL tasks with high-dimensional objective spaces:
Proposition 5.1. (Time complexity.) Given that the number of objectives is the number of extension policies is N, assume the running time of each optimization problem is upper bounded by a constant, and the number of Pareto extension steps is K, the expected running time of Algorithm 2 is OnKNN)
Compared to PG-MORL (Xu et al., 2020 which needs to solve knapsack problem with K x N candidate points to evaluate and need O(K Nm-2) steps to solve it exactly, the proposed C-MORL excels when there are growing number of objectives or steps.
5.4 PARETO REPRESENTATION VIA POLICY ASSIGNMENT
Policy Assignment During the Pareto extension stage, we store the policies with their corresponding expected return on the Pareto front. Therefore, after the Pareto extension stage, we derive an approximated Pareto front with Pareto set policies, as shown in the Policy assignment process in With Pareto set policies, given unseen preference, we can select its SMP = SMM by solving
Fig. 3
7

---

Published as a conference paper at ICLR 2025
Eq. 3.1 Such policy assignment process is efficient and does not require any retraining of the new policy. Algorithm 2 in Appendix [A presents the complete workflow of C-MORL.
Traeeato
MD-Hopper-2d
MO-Arr-
Figure 4: Pareto front comparison on two-obeective MO- MO-MuJoCo benchmarks.
SAME
Seaame
Figure 5: Pareto front comparison on MO-Ant-3d benchmark.
6 EXPERIMENTS
In this Section, validate the design of our proposed algorithm using both popular discrete and continuous MORL benchmarks from MO-Gymnasium (Felten et al. 2023a and SustainGym (Yeh)
et al. 2024). These benchmarks include five comprehensive domains: (i) Grid World includes Fruit- Tree, discrete benchmark with six objectives. (ii) Classic Control includes MO-Lunar-I.and Lander, a discrete benchmark with four objectives. (iii) Miscellaneous includes Minecart, discrete benchmark
with four objectives. (iv) Robotics Control includes five MuJoCo tasks with continuous action space based on MuJoCo simulator (Todorov et al. 2012] Xu et al. (020). (v) Sustainable Energy Systems includes two building heating supply tasks. These benchmarks pose significant challenges for MORL due to complex state (up to 3488/action (up to 23) spaces and large objectives spaces (up
to objectives). More details of experiment settings are listed in Appendix E
We benchmark our algorithm against five competitive baselines under the metries of hypervolume (HV), expected utility (EU), and sparsity metrics (SP). Higher HV and EU, lower SP indicate better performance. Each of the baselines are trained for x 10a time steps for discrete benchmarks. Continuous benchmarks with two, three, and nine objectives are trained for 1.5 x 10 2 10 and
2.5 x 10 steps, respectively. The baselines involve: (i)Eneelope (Yang et al. 2019 (ii) CAPQL (Lu et al., 2022). (iii Q-Pensieve (Hung al., 2022). (iv) PG-MORL (Xu al. 2020). (v) GPI-
LS (Alegre et al. 2023). (vi) MORL/D (Felten et al. 2024). Among these baselines, Envelope is
developed for discrete control tasks, GPI-LS is suitable for both discrete and continuous control. The
other basclines are specifically tailored for continuous control.
Overall Results. To assess the performance of MORL, we begin by comparing the quality of the Pareto front. It can be observed from Table D and Table that the proposed C-MORL achieves the highest hypervolume all benchmarks. This indicates that C-MORL successfully discovers high- quality Pareto front across various domains, particularly in benchmarks with large state and action spaces MMO-Humanodd-2d and substantial number of objectives (Building -d). Figures and illustrate the Pareto front for MO-MuJoCo benchmarks with two and three objectives, respectively. C-MORL exhibits more comprehensive coverage the Pareto front across benchmarks, whereas other baselines fail to encompass the entire front. For instance, in the MO-Ant-2d benchmark. Q- Pensieve does not cover the upper-left portion of the Pareto front, indicating insufficient exploration of the Y velocity objective, which results in lower utility for preference pairs prioritizing this objective. Table 1: Evaluation of HV, EU, and SP for discrete MORL tasks.
Environments
Metries
Envelope
GPI-LS
C-MORL
HV(10")
1.9920.00
6.05=00377
6.7=20.88
Minecart
EU(10
-2.72+11011
2.29:10.32
2.12=20.66
SP(10
5.11:2211
0.10::0.00
0.05=20.22
HV(10")
0.3340.18
1.06:0.16
1.12:0.03
MO-Luna-LLnder
EU(10*
-2.844:4.06
1.8120.34
2.35-20.18
SP(10")
0.19::0.16
0.13-20.01
1.441.224
HV(
3.66-20.33
3.57=00.5
3.67=0.14
Fruit-Tree
6.1540.00
6.15:20.00
6.53--0.03
SP(10
5.46:10.15
5.29::0.21
0.42:10.03
While C-MORL leads the highest hypervolume by a large margin in almost all cases, it does not always attain the best sparsity in some benchmarks. This occurs because, in certain cases,
8

---

Published as a conference paper at ICLR 2025
Liningg wwwe %0
Land
Selected sdd
Seleced sokd
1%
K-0
K-10
K-60
Figure 6:
Pareto extension stage of MO-Ant-2d benchmark. The blue points are Pareto-optimal
solutions, and the deep-blue points are selected solutions.
algorithm may identify only a few similar Pareto-optimal solutions, leading to low sparsity. As illustrated in Table along with Figures and both CAPQL and GPI-LS cover only specific portion of the Pareto front, resulting a reduced sparsity. In contrast, C-MORL generates dense and comprehensive Pareto front in these benchmarks. C-MORI. also demonstrates the highest expected utility in nine of the ten benchmarks, which indicates that C-MORL can better maximize user utility given various user preferences. Appendix G.3 further illustrates that as a multi-policy approach, C-MORL can align corresponding Pareto-optimal solution given any preference. This is in contrast to single preference-condition policies, which may yield dominated solutions for unseen preferences.
Regarding time complexity, as stated in Section 4 C-MORL is linear time complexity with respect to the number of objectives, and is therefore generalizable to benchmarks with more than three objectives. We also note from Table 2 that for the Building-9d benchmark with nine objectives, the training times for PG-MORL and GPI-LS excessively long, exceeding the time limitation we set. This finding aligns with the time complexity analysis of PG-MORL discussed in Section Additionally, in both the Pareto initialization and extension stages, the policies can be trained in parallel, which further reduces the total running time. Given that the expected running time is O(nKN), the Pareto extension steps K and the number of selected policies N can be adjusted to balance performance with running time.
Pareto extension analysis. In Fig. 6 we showcase how the Pareto front is filled using set of selected policies with our Pareto extension approach. During the initialization stage of the Pareto set, we uniformly sample preference pairs from the preference space. Training a single policy with a fixed preference vector facilitates convergence to a Pareto-optimal solution. Additionally, we present an ablation study on the number of selected extension policies Appendix G.I
Building on this foundation, the subsequent Pareto set extension stage further populates the Pareto front. As shown in Fig. [6] gaps remain on the Pareto front following the initialization stage. Our effective policy selection algorithm targets Pareto-optimal solutions in these gap regions, which exhibit higher crowding distances, thereby selecting them for the ensuing extension stage. During the Pareto extension phase, as the number of extension steps K increases, C-MORL employs constrained policy optimization to adjust the selected policies toward various objective directions, effectively filling the gaps in the Pareto front.
Next, to better understand the influence of hyperparameters and key components of C-MORL, we perform an in-depth analysis and conduct ablation studies focusing on these aspects.
Parameter study of C-MORL for Pareto Initialization. As mentioned in Section 5.1 during Pareto initialization stage, C-MORL aims to derive a few Pareto optimal solutions by training a set of initial policies. To study the impact of the number of initial policies M, we con- duct experiments while keeping the total number of training steps fixed 1.5 x
Table 3: Parameter study of C-MORL for Policy Initial- ization on MO-Hopper-24 benchmark.
HV(10*)
EU(10P)
SP(10P)
M=3 steps)
1,3220.12
2.53.0.13
0.584 0.43
M=6 (1.5M steps)
1.39:10.04
2.55:0.004
0.16:10.11
M=11 (1.5M steps)
1.32-10.03
2.7110.33
0.44.0.33
M-6 (3M steps)
1.45:10.05
2.63:10.05
0.22-10.10
10 steps (including 1 x 10 steps for initialization stage) to ensure fair comparison. Specifically, we
uniformly sample MM = 3, 6, 11 preference vectors, as shown in Table For example, with sampling interval of 0.2, the preference vectors are = j0, 1], [0.2. 0.8), 00.4, 0.6], [0.6, 0.4], (0.8. 0.2]. [1, Oj. Intuitively, increasing M can enhance diversity among Pareto-optimal solutions, which benefits the subsequent extension phase. However, since the number of total time steps is fixed, increasing M reduces the training steps allocated to each initial policy. The trade-off is evident as the performance for M 11 is worse than for M 6, indicating that the increasing of M does not always guarantee
9

---

Published as a conference paper at ICLR 2025
Table Evaluation of HV, EU, and SP for continuous MORL tasks. T/O indicates that the training time exceeded the maximum limit of 100 hours.
Fnvironments
Metrics
CAPQI
Q-Pensieve
PG-MORT
GPI-T
MORL/D
C-MORT.
HV(10")
1.15
1.26::0.01
1.20:1:009
1.19:0.00
L11::0003
1.99:00.01
MO Ho.perpe
BU(10")
2.28
2.28100.11
2.34.10.10
2.33_10.10
2.19.10.04
2.56.10.02
SP(10")
0.46
1.31
5.134:5.81
0.49=0.37
2:72:4:2055
0.33-10.28
HV(10')
1.65
1.66:1122
1.59::0.45
1.70=0.29
1.44::0005
2.29-10.01
MO- MO-Hopper-33
EU(10")
0.53
0.26.10.79
1.7:10.25
162,10.10
1.68
1.8010.01
SP(10")
2311
0.774.088
0.7611091
0.74
0.84:10117
0.28:10.09
HV(10")
1.11
2.55+0.54
0.35:40008
3.10=0.25
1:03:1:0266
3.13:10.20
MO-Amt22
BU(10")
2.16+0.94
3,14=0.49
0,81
4,28=0.19
2.22-+:049
4.29++0.19
SP(10")
0.18=0007
3.63+2271
2.20:4:3488
3.61=2.13
2.90-:22.1
1.67-20.885
HV(10')
0.22
3.8220043
0.94::0112
0.85=0.81
I 1.0000.17
1.09:10.13
MO-Ant-3d
EU(10")
0.30=0,29
2.18:0041
1.07:00007
2.41=0.20
1.11
2.57-400.66
SP(10*)
0.17+0.09
0.8300.07
0.02:1:001
196=0.79
0.85:20.57
0.03:1:0001
10"
0.30
0,90:0.62
2.62:40.32
0.8=0002
2.81
3.43:1-0.06
MO-Humanoid-2d
EU(10?)
4.75=0.04
0.89
0.00
3.67=0.02
4 4.32-40.06
4.78++0.05
SP(10*)
0.00=0.00
1.09
0.13:+0.17
0.00=0.00
0.00=10.00
0.18=+0.27
HV(I0
0.33
0.00
0.83:10.02
0.26=00044
0.87-100.88
1.15:10.00
Building-3d
EU(10*)
0.75=0.09
0.96=0.00
0.93++0.01
0.74=0.01
0.95++0.00
L.0200.000
SP(10*)
0.18=0.08
0.92=0078
0.04
0.07=0.09
7.31+2.20
0.07
HV(10
0.29
0.28
T/O
T/O
T/O
7.93-10.07
Building-9d
EU(1O")
3.31
3.46-0003
TIO
T/O
T/O
3.50-10.00
SP(10")
4.34#3.72
1.04=0.38
TIO
TXO
T/O
2.79:00.40
better performance. To further investigate, we increase the number of training steps to 3 x 10 (with 2 x 10" steps allocated initialization stage) while keeping M 6. The results demonstrate that proportionally increasing the total training steps and the number of initial policies leads to improved performance, highlighting the importance of balancing training resources with policy diversity.
Parameter Study for Return Constraint Hyperparameter B. In Section 5.3 and Section 4 we
develop criteria and practical methods for specifying constraint values for problem Eq. I We conduct parameter study for return constraint hyperparameter 3 in Eq. 1o examine how constraint values influence the generation of Pareto-optimal solutions. Table presents the results of C-MORL with various 3 on the MO-Ant-24 benchmark. When 8 0.9, C- C-MORL presents the best performance all metrics. Intuitively, a higher 3 ensures that when optimizing a specific objective, the expected returns other objectives do not decrease significantly, thereby facilitating adaptation new Pareto-optimal solutions.
Table 4: Evaluation of HV. EU, and SP for MO-Ant-20 benchmark with different B.
Ablation Study on Policy Selection. To evaluate the effectiveness of the crowding distance-based policy se- lection method, we compare the Pareto extension using this approach with random selection. [5 the comparison results, where Random and Crowd refer random policy selection and crowd distance based
IIV(10")
EU(10")
SP(10")
0.5
3.05
4.24
2,54 2.41
0.7
3.074:0.26
4.23-10.24
2.06 1.51
0.9
3,08+
4.27=00.2
1.71=0.21
policy selection. Each method is tested across three runs with the same seeds to ensure consistent Pareto initialization and uses the same hyperparameters for the Pareto extension. The experimental results show that C-MORL with crowd distance based policy selection outperforms the random selection across all metrics, highlighting the effectiveness of the proposed policy selection method. Intuitively, adjusting the policies in sparse areas facilitates better filling of the gaps in the Pareto front.
CONCLUSIONS AND DISCUSSIONS
This paper introduces a novel formulation and efficient solution procedure for multi- objective reinforcement learning problems.
Table 5: Evaluation of HV, EU, and SP for MO-Ant benchmark with different policy selection methods.
Selection method
HV(10")
BU(10
SP
leverages a two-stage approach con- struct the Pareto front efficiently. By em-
Random
3.91=0.11
2.31=0.133
8.69-3502
Crowd
4.10
2.57
8.34
ploying a precise selection of sparse regions on the Pareto front that require further exploration and the utilization of a customized extension method, C-MORL not only achieves superior performance in terms of training efficiency but also excels in terms both Pareto front quality and user utility compared to state-of-the-ar baselines on both discrete and continuous benchmarks. In future work, we plan to develop a more effective extension method so as to more efficiently discover the unexplored Pareto front, especially for continuous environments. We are also interested in designing transfer schemes of learned MORL policies, as well as coordination of agents for mastering real-world MORL tasks with agent-specific reward signals.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International conference on machine learning, Pp. 11-22. PMLR, 2020.
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning, pp. 11-20. PMLR, 2019.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Lucas N Alegre, Ana LC Bazzan, Diederik M Roijers, Ann Nowé, and Bruno da Silva. Sample- efficient multi-objective learning via generalized policy improvement prioritization. arXiv preprint arXiv:2301.07784. 2023.
Lucas Nunes Alegre, Ana Bazzan, and Bruno C Da Silva. Optimistic linear support and successor features as a basis for optimal policy transfer. In International Conference Machine Learning, pp. 394 413. PMLR, 2022.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. arXiv preprint arXiv:2208.00 079114 2022.
Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48. Springer, 2009.
Xi Chen, Ali Ghadirzadeh, Mårten Björkman, and Patric Jensfelt. Meta-learing for multi-objective reinforcement learning. In 2019 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 977-983. IEFE, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International conference on machine learning, pp. 1282-1289, PMLR, 2019.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiob- jective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary compulation, 6(2):182-197, 2002.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
Florian Felten, Grégoire Danoy, El-Ghazali Talbi, and Pascal Bouvry. Metaheuristics-based ex- ploration strategies for multi-objective reinforcement learning. In ICAART (2), pp- 662-673, 2022.
Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno Castro da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Proceedings of the 37th Conference Neural Information Processing Systems (NeurIPS 2023), 2023a.
Florian Felten, Daniel Gareev, El-Ghazali Talbi, and Grégoire Danoy. Hyperparameter optimization for multi-objective reinforcement learning. arXiv preprint arXiv:2310. 16487, 2023b.
Florian Felten, El--Ghazali Talbi, and Grégoire Danoy. Multi-objective reinforcement learning based on decomposition: A taxonomy and framework. Journal of Artificial Intelligence Research, 79: 679-723, 2024.
Eli Friedman and Fred Fontaine. Generalizing across multi multi-objective reward functions in deep reinforcement learning. arXiv preprint 1rXiv:/809.000 2018.

---

Published as a conference paper at ICLR 2025
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
Conor Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane,
Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Mulli-Agent Systems, 36(1)))nn666))))))))) 36(1):26, 2022.
Sandy Huang, Abbas Abdolmaleki, Giulia Vezzani, Philemon Brakel, Daniel Mankowitz, Michael Neunert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, et al. A constrained multi-ojeective reinforcement learning framework. In Conference on Robot Learning, pp. 883-893, PMLR, 2022.
Wei Hung. Bo-Kai Huang, Ping-Chin Fing-Chun Hsich, and Xi Liu. Q-pensieve: Boosting sample efficiency of multi-objective rl through memory sharing of q-snapshots. arXiv preprint 1rXiv:2212.03/17, 2022.
Dohyeong Kim, Mineui Hong. Jeongho Park, and Songhwai Oh. Scale-invariant gradient aggregation for constrained multi-objective reinforcement learning. arXiv preprint arXiv:2403.002282, 2024.
Ilya Kostrikov. Pytorch implementatinn of reinforcement learning algorithms. https:///ittuu:
com/
a2c- ppo acktr-gail 2018.
Panagiotis Kyriakis and Jyotirmoy Deshmukh. Pareto policy adaptation. In International Conference on Learning Representations, volume 2022, 2022.
Marco Laumanns, Lothar Thiele, and Eckart Zitzler. An efficient, adaptive parameter variation scheme for metaheuristics based on the apsilon-constrain method. European Journal of Operational Research, (69:93932-9944 2006.
Qian Lin, Chao Yu, Zongkai Liu, and Zifan Wu. Policy-regularized offline multi-objective reinforce- ment learning. arXiv preprint arXiv:2401.022 0222444, 2024.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-poie point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4940-49411 2020.
Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convex- ity, stationarity and pareto optimality. In The Eleventh International Conference Learning Representations, 2022.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv:1803.1 11347, 2018.
Simone Parisi, Mattco Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through continuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227, 2016.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019.
R Tyrrell Rockafellar. Convex Analysis, volume 18. Princeton University Press, 1997.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Diederik M Roijers, Shimon Whiteson, and Frans A Oliehoek. Linear support for multi-objective coordination graphs. In AAMAS' 14: PROCEEDINGS OF THE 2014 INTERNATIONAL CON- FERENCE ON AUTONOMOUS AGENTS & MULTIAGENT SYSTEMS, volume 2, pp. 1297-1304. IFAAMAS/ACM, 2014.
12

---

Published as a conference paper at ICLR 2025
Willem Röpke, Mathieu Reymond, Patrick Mannion, Diederik M Roijers, Ann Nowé, and Rox- ana Rädulescu. Divide and conquer: Provably unveiling the pareto front with multi-objective reinforcement learning. arXiv preprint arXlv:2402.07182, 2024.
Gavriel Salvendy. Handbook of industrial engineering: technology and operations management. John Wiley & Sons, 2001.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063777 2017.
Han Shao, Lee Cohen, Avrim Blum, Yishay Mansour, Aadirupa Saha, and Matthew Walter. Eliciting user preferences for personalized multiiobjective decision making through comparative feedback. Advances in Neural Information Processing Systems, 36, 2024.
Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Mare G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. The Eleventh International Conference on Learning Representations, 2022.
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:/805. /074, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483-355 3512- 2014.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE symposium adaptive dynamic programming and reinforcement learning (ADPRL), pp. 191-199. IEEE, 2013.
Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, and Russ Tedrake. Fleet policy learning via weight merging and an application to robotic tool-use. arXiv preprint arXiv:2310.0136) 2023.
Marco Wiering, Maikel Withagen, and Mädälina M Drugan. Model-based multi-objective reinforce- ment learning. In 2014 IEEE symposium adaptive dynamic programming and reinforcement learning (ADPRL), pp. 1-6. IEEE, 2014.
Ronald J Williams. Simple statistical cradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 2266, 1992.
Jie Xu, Yunsheng Tian, Pingchuan Ma. Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guide multi-objective reinforcement learning for continuous robot control. In In- ternational conference on machine learning, pp. 10607-10616. PMLR, 2020.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp. 1180011499. PMLR, 2021.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
Christopher Yeh, Victor Li, Rajeev Datta, Julio Arroo,, Nicolas Christianson, Chi Zhang, Yize Chen, Mohammad Mehdi Hosseini, Azarang Golmohammadi, Yuanyuan Shi, et al. Sustaingym: Rein- forcement learning environments for sustainable energy systems. Advances in Neural Information Processing Systems, 36, 2024.
Tom Zahavy, Andre Barreto, Daniel Mankowitz, Shaobo Hou, Brendan O' O'Donoghue, lurii Kemaev, and Satinder Singh. Discovering a set of policies for the worst case reward. arXiv preprint arXiv:2102.04323, 2021.
Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi-objective black box optimization. In International Conference on Muchine Learning, pp. 11096-11105. PMLR, 2020.
Siyan Zhao and Aditya Grover. Decision stacks: Flexible reinforcement learning via modular generative models. Advances in Neural Information Processing Systems, 36, 2024.
Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized natural policy gradient for multi-objective reinforcement learning. Advances in Neural Information Processing Systems, 001135844135900 2022.
Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline multi-objective rl. arXiv preprint arXiv:2305.000 00567, 2023.
14

---

Published as a conference paper at ICLR 2025
A POLICY SELECTION AND C-MORL ALGORITHM
This section provides details of policy selection and C-MORL algorithms.
In our C-MORL implementation, during the policy initialization stage, employ parallel training of M policies for pre-known preference vector w. We maintain a policy buffer, meaning that in addition the final policy obtained in the Pareto initialization stage, intermediate policies are also saved in the buffer. As result, the number of candidate policies in the policy selection stage is usually larger than M, incorporating the policies stored in the solution buffer.
We conduct policy selection in both after the Pareto initialization stage and during the Pareto extension stage. Specifically, the number of extension policies N, the number of Pareto extension steps and the number of constrained optimization steps K" are predefined. During the Pareto extension stage, C-MORI, selects Pareto optimal solutions based on their crowd distance every K" that the sparse areas on Pareto-front have a higher potential to be explored.
K steps to make sure
Furthermore, in the policy selection process, the extreme policies on the Pareto front (for each objective, the extreme solution on the Pareto front refers to the solution that achieves the maximum value for that particular objective) are selected by default. The other Pareto-optimal solutions are selected based on their crowd distance. The policy selection process continues until predetermined number of N policies are selected or until all Pareto-optimal policies have been selected. This procedure is detailed in the following Algorithm!!
Algorithm 1 Policy Selection
Require: Number of extension polices N, solution set X.
1: Initialize extension solution set Nextension = {t. number of selected policies Nselected = O.
2: Solution set X <- filter Pareto-optimal solutions in solution set X.
3: Sort solutions in solution set X' by crowd distance in descending order.
4: while N Nettected < N and Neelected * | do
5: Add the Nth
selected solution in solution set X' into extension solution set Keztensionn
Nselected Naclected +
7:
end while
Ensure:
extension solution set Nextension.
In Algorithm 2 we describe the whole procedure of our proposed C-MORL algorithm.
15

---

Published as a conference paper at ICLR 2025
Algorithm 2 C-MORL
Require:
Number of initial polices M, Number of extension polices N,
initial policy task set I {(Wsist Tenst) init=1 solution set X' (},
initial solution set Xnnt = (}. number of objectives n, number of Pareto extension step K,
number of constrained update steps K", preference set Ω number of evaluation preferences E.
1: Paareoo Initialization.
2: for (Wanit: Tinit) E I do
3: Solution (Tinstt Fointt + solve task (Winit, Tinst).
4: Store solution (Tinit, G ) in solution set Xinit
5: end for
6: |Policy Selection. (Algorithm I
7: Keatension +- PolictySllettioN(N (ant))
8: \\ Parreoo Extension.
9: for iter 1.... K/K' do
10: |Constrained Policy Optimization.
for (T,, G Nextension do
12: for i 1....: do
Initialize 11,0 n.
14:
for r ro 1, K" do
15:
#rr+1 solve optimization problem in Eq.
16:
Store solution (Nr,+1, G in X.
end for end for
18:
end for
20:
Policy Selection. (Algorithm 1 Nentension f- PolicySSlection
21:
22:
end for
23:
|Polic Assignment.
24: Sample E preferences from ?.
25: for w E f2 do
26: Solve and find SMP policy via S SMP maxxell, fo (G*).
27: Derive solution (I SMP
G for w.
SMP
28: end for
16

---

Published as a conference paper at ICLR 2025
B PROOF OF PROPOSITION 4.3
Proof. We prove by contradiction. Suppose that the feasible solution P' = (n', G of problem Eq. iis not a Pareto-optimal solution. By the definition of Pareto-optimal solution, there exists solution
P P= Ga) in dominates P', i.e., CT S G for Vi E [1, 2, ....,.. n). Assume P. (m,, G.) is the initial point of solving problem Eq. Therefore, Gf 2 Gf 2 Gf Because both P. and P and Pareto-optimal solutions, they do not dominate each other. Therefore, given that Gp G there exists j E (1.2..... ml.jj j #
(#t
that G 2 Gj.
Now consider the values of d, and Gj.. Note that dj C If GJ Gj dy. then G7 2 G Gj(k 1), which is conflicting with the condition that (1ck-1) Gy is the (A - 1)th objective value in Gy. If G I dy, it conflicts with the assumtion that P dominates P'. Therefore, P does not exist, and p' is Pareto-optimal solution.
1).
C PROOF OF PROPOSITION 4.4
In this Section, we provide brief proof for the convergence of iterates majorly inspired by (Tessler al. 2018). For detailed proof, refer to Chapter 6 of (Borkar 2009).
Proof. At iteeaaiion the update steps for XE and are follows:
1 =I (Ar, 7n LArra mr)];
(5a)
+r = Tr. (2( L [(A,,
(5b)
Using the log-likelihooe trick (Williams 1992) for the policy update, we have the gradient step
Vx, ((, (d, 3. (T;));
(6a)
V (A, π) = =V (s, a; R(s) A G...ll
(6b)
In the above, 71, 72 are step-sizes which ensure that the policy update is performed on faster timescale than that of the penalty coefficient '₁ We also make the following assumption:
Σ (2(k)) xc, Σ (n)) mz(k)²) < oo and 17 (k) k=
(b) K=
-
(2
Then for the update of policy for any given A with the slowest timescale, we can show the following ODE governs the updates:
# [V
(8)
Similarly, for the update of A, denote π (A₂,) as the limiting point of the 7-recursion corresponding to de. we have the following ODE
Å.. ((x (T))))
(9)
Then following the characterization made by (Tessler al. 2018) the internally chain transitive invariant sets of the ODE Eq. [ we can conclude the convergence for the two-timescale stochastic approximation processes.
D PROOF OF THEOREM 4.5
We reuse the notations for the primal problem Eq. [ and its dual problem Eq. 2 For the nonconvex primal formulation of MORL, solving its dual problem can only provide an upper bound on P Thus it is of interest to evaluate how close the policy obtained by solving the dual is compared to P*. Further, as we resort to interior point method to empirically solve the unconstrained problem, the second part of Proposition 4.5 characterizes the distance between the solution from Eq. and Pt.
17

---

Published as a conference paper at ICLR 2025
Proof. To show the first part of the Theorem, majorly relies the well-established Fenchel-Moreau theorem and the concavity of the perturbed function defined as the following P(E):
Denote the duality gap A D P To show A 0, we first define the perturbation function of problem equation|
P(() max G7
(10)
s.t. G 2 +i. i= n4 +
Then, the conditions under which problem Eq. I Tass zero duality gap are as follows:
Lemma D.1. (Fenchel-Moraau) If (i) Slater's 's condition holds for (PI) and (ii) its perturbation
function P(g) is concave, then strong duality holds for (PI).
In D.I Slater's condition requires that there exists feasible policy a such that all incquality constraints are strictly satisfied for problem (1), i.e.
G > di,
Vi 1i f
See proof of D.I in the Corollary. 30.2.2 of (Rockafellar 1997). For the proof of concavity of the perturbed C-MORL formulation P(E). suffices to show for every E", E R", and for any J (0, 1), the following equation holds:
[uC] + (1-+) 2 (P((") + (1-1))((E))
(11)
We refer the detailed proof of Eq. II in the Proposition 1 of (Paternain 2019
To show the second part, we make use of the optimality conditions of the dual problem. As the Lagrangian function of Eq. is L(A, T) = G X (d - G.my)) and denote G = d; - G.
Without loss of generality, we follow the standard minimization problem formulation in optimization, and we can write out the dual function as
n
D(A:) = min GT +
Σ 1=,,
G
(12)
When the problem is strictly feasible, there exists optimal ' such that for each objective other than the objective 1, have G2 d,. Then the first-order optimality condition holds:
-VTT
Σ
VG 0.
(13)
t x (GT)
1=,
We then set -tx(oy) and plug it back into Eq,
13
we can obtain
-VTT
Σ
XVO 0.
This shows that under the optimal policy x' and dual variables A & 1, we find the optimal value for the dual function as
1 t
I(A, = GF
G GJ"
(14)
Using the first-part result that the C-MORL formulation has zero duality gap, P = D(X*), we thus have
11 -
p
G7
(15)
t
18

---

Published as a conference paper at ICLR 2025
E EXPERIMENT SETUP DETAILS
E. BENCHMARK
To evaluate the performance of method and baselines. collect benchmarks from MO- Gymnasium Felten et al. (2023a) and SustainGym Yeh et al.] (2024). MO-Gymnasium is an open source Python library that includes more than 20 environments from diverse domains. SustainGym provides standardized benchmarks for sustainable energy systems, encompassing electric vehicles, data centers, electrical markets, power plants, and building energy systems. Among these environ- ments, the building thermal control tasks involve large commercial buildings with multiple zones, aiming to regulate temperature while minimizing energy costs, making them suitable for developing multi-objective control tasks. Additionally, incorporate the objective of demand response (re- ducing peak demand) and extend its multi-objective version. The details of all benchmarks are as follows:
Minecart: A discrete MORL benchmark with a cart that collects two different and must return them to the base while minimizing fuel consumption. The states of the cart include its and y position, the current speed, the sin and cos orientations, and the percentage of its occupied capacity by each ore. The agent is allowed choose between six actions: {mine, left, right, accelerate, brake, do nothing). Let If denote Dirac delta function. The reward of three objectives is defined as:
R1 = quantity of ore 1 collected if s is inside the base, else 0;
R₂ = quantity ore collected 8 inside the base, else 0;
R3 -0.005 - 0.025l {a accelerate 0.05H {a
mine }.
MO--anrr-Lander: A discrete MORL benchmark with classic rocket trajectory optimization problem. The state is an eight-dimensional vector that includes the x, coordinates of the lander, its linear velocities in x and y. its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground or not. The action is six-dimensional vector: {do nothing, fire left orientation engine, fire main engine, fire right engine). The reward of three objectives is defined as:
R₂ = + 100 if landed successfully, - 100 if crashed, else 0;
R2 shaping reward;
R3 fuel cost of the main engine;
R4 fuel cost of the side engines.
Fruit-Tree: A discrete MORL benchmark with full binary tree of depth d with randomly assigned vectorial reward r E _ on the leaf nodes Yang al. 2019). These rewards are related to six objectives, showing the amounts of six different nutrition facts of the fruits on the tree: (Protein, Carbs, Fats, Vitamins, Minerals, Water]. The goal of the MORL agent is to find path on the tree from the root to a leaf node that maximizes utility for given preference. The reward of six objectives defined as:
R, value of nutrient in for i 1...6. 6.
MO-Hopper-2dt The observation and action space are defined as:
S S A
This control task has two conflicting objectives: forward speed and jumping height. The first objective forward speed:
R1 U2 CC
The second objective jumping height:
R₂ = 10(h h himit) + C.
where C' = -0.001 E, af is composed of extra bonus and energy efficiency, U. is the speed toward x direction, / is the current height, himit is the initial height, a, the action of each actuator.
MO-Hopper-3d: The observation and action space are defined as:
S SC R
3
19

---

Published as a conference paper at ICLR 2025
This control task has three conflicting objectives: forward speed, jumping height, and energy effi- ciency.
The first objective is forward speed:
R1 = Ur-
The second objective jumping height:
R2 10(h hinit) )
The third objective is energy efficiency:
R3 = - a
where va is the speed toward x direction, the current height, histt is the initial height, a; action of each actuator.
MO-Ant-2d: The observation and action space are defined as:
S S R A
This control task has two conflicting objectives: x-axis velocity and y-axis velocity.
The first objective x-axis velocity:
R1 Ur-
The second objective is y-axis velocity:
R₂ By.
where us is x-axis speed, Uy y-axis speed, a; the action of each actuator.
MO-Ant-3d: The observation and action space are defined as:
S AC R
This control task has three conflicting objectives: x-axis speed, y-axis speed, and control cost. The first objective is x-axis velocity:
R1 Ug.
The second objective is y-axis velocity:
R2 = By-
The third objective control cost:
R3
-2 -2 Σ aa
where Us is x-axis speed, vy is y-axis speed, a; is the action of each actuator.
Mo-uunnnidd-2d: The observation and action space defined
S C R 376
A 17
This control task has two conflicting objectives: forward speed and control cost. MO-Humanoid was chosen because has one of the most complex state space among all Mujoco environments, with 348 states.
The first objective forward speed:
R1 Uz-
The second objective is control cost:
R₂ = -10 1000 af
where us is the speed in æ direction, a; is the action of each actuator.
20

---

Published as a conference paper at ICLR 2025
Building-3d: A building thermal control environment that controls the temperature of commercial building with 23 zones. The states contain the temperature of multiple zones, the heat acquisition from occupant's 's activities, the heat gain from the solar irradiance, the ground temperature, the outdoor environment temperature, and the current electricity price. The actions set the controlled heating supply of the zones. The conflicting objectives minimizing energy cost, reducing the temperature difference between zonal temperatures and user-set points, and managing the ramping rate of power consumption:
R1 -0.55 Σ
[T(tt] T Tuuer [t]l:
R₂ M - 0.05 * Σ c|t||
R3 M - [.[ Σ 1])--
where .M = 23 is the number of zones; T; and T Tusee are indoor temperature and user setting point of zone i, respectively; e electricity price: P, heating supply power of zone i; is the index of environment time step.
Building-9d: modified version of Building-3d: Instead calculating the reward based on all zones collectively, this version evaluates the reward for each of the three floors of the commercial building separately. Consequently, this results total of 3 9 objectives.
Table Environment details of continuous control benchmarks
Environments
Contiquous (Y/N) Number of Objectives
State Space
Action Space
Minecart
N
S C
R
R.
Lu-LLunr--ander
N
S C
A C 1e4
Fruit-Tree
N
8 C I
A 18
MO-Hopper-2d
R
A C RR-
MO-Hopper-3d
Y
3
S C R
A C R
MO-Ant-2d
Y
S C TR 10
A C
R
MO-Ant-3d
Y
3
$ C 10
A C R
MO- 1-0mHundd
Y
8 C 323
A C
RD7
Building-3d
Y
3
$ C R
A C R
Building-9d
Y
C R
A C R
E.2 TRAINING DETAILS
We run all the experiments cloud server including CPU Intel Xeon Processor and GPU Tesla T4. In the Pareto initialization stage, we use PPO algorithm implemented by Kostrikov (2018). The PPO parameters reported in Table and Table For constrained optimization, method. For the baseline implementations, Q-Pensieve (Hung et al. 2022) and PG-MORL (Xu al. 2020) reproduced using the official code provided by their respective papers, ensuring consistency with the original experiments. For Envelope (Yang et al., 2019), CAPQL (Lu et al. (022). GPI- LS (Alegre al. 2023), and MORL/D (Felten et 2024). utilize the implementations available in the MORL-baselines library (Felten et al. 2023a), adapting them as necessary to align with our experimental setup. The hyperparameters of M-MOLL- include:
adopt C-MOLL-IPO
Number of initial policy M: the number of initial policies. This parameter is also related
the Pareto initialization stage.
Number of extension policy N: the number of policies selected in the Pareto extension
stage. This parameter is also related
Pareto extension stage.
Log barrier coef t: the tunable parameter the log barrier.
Constraint relax coef 3: the constraint relaxing coefficient 8 in Eq. 3
Extension steps K; the number of Pareto extension steps the Pareto extension stage.
Time step: the total time step contains initialization steps and extension steps. To be specific, time step = timesteps per actorbatch x (M + N x n), where timesteps per actorbatch is PPO parameter.
21

---

Published as a conference paper at ICLR 2025
The parameters we used are provided in Table 9 and Table I0
In policy initialization stage, the preference vectors for training initial policies are uniformly sampled from the preference space Ω. For example, in the case of MO-Ant-2d, we set the sampling interval to 0.2, with the minimum and maximum values for each dimension being 0 and 1, respectively. This results in the following preference vectors: [0, 1], [0.2, 0.8]. (0.4, 0.6], [0.6, 0.4), [0.8, 0.2), and [1, for a total of 6 preference vectors.
Table 7: PPO Hyperparameters for discrete baselines.
parameter name
MinncarMO-LLmrr-Laddder Tree
sleps per batch
512
512
512
LR(x10 %)
3
LR decay ratio
gamma
0.995
0.995
0.995
gac lambda
0.95
0.95
0.95
batch
epuch
10
10
10
entropy
value loss coef
0.5
0.5
0.5
mux grad norm
0.5
0.5
clip param
0.2
0.2
0.2
Table 8: PPO Hyperparameters for continuous baselines.
parameter
Hopper
2dMO Hupper- Ant
Ant- 3dM---dM---a Humanoid BuBldddn 3dBuilding
stops per actor batch
512
512
512
512
512
512
512
IRO 10
3
3
3
3
3
3
LR decny ratio
0.995
0.995
0.995
0.995
0.995
0.995
0.995
gue lambda
0.95
0.95
0.95
0.95
0.95
0.95
0.95
mini batch
32
32
32
32
32
32
ppo epech
10
10
10
10
10
10
10
entropy coef
value loss coef
0.5
0,5
0.5
0.5
0.5
0.5
0.5
max grad norm
0.5
0.5
0.5
0.5
0.5
clip
0.2
0,2
0.2
0.2
0.2
0.2
0.2
Table 9: C-MORL-IPO Hyperparameters for discrete baselines.
parameter
LanesarM--LLnaaLLmnniiii Tree
M
6
21
6
log burtier uoef
20
20
constraint relax aoef
0.90
0.90
0.90
Pareto extension stop
60
time step (x10))
5
E.3 EVALUATION
We evaluate the quality Pareto front with the following metrics:
Definition E.1. (Hypervolume). Let P be a Pareto front approximation in an n-dimensional objective space and Go be the reference point. Then the hypervolume metric H(P) is calculated as H(P)- = f If H(( ()) d22 where H(P) {z 7 - < |PI: Go < 3 P()). P()) is the jth solution in P. = is the relation operator of objective dominance, and If (P)) is Dirac delta function that equals if 2 E ((P) and 0 otherwise. A higher hypervolume is better.
Definition E.2. (Expected Utility). Let P be Pareto front approximation in n-dimensional objective space and II be the policy set. The Expected Utility metric is U(P) U(P) = Ew-R G SMP A higher expected utility is better.
Definition E.3. (Sparsity). Let P be Pareto front approximation in an n-dimensional objective space. Then the Sparsity metric S(P) is
PP-1 (Q.(k)-
S(P) = P
Σ
Σ
G,(k
(16)
A=
where G. is the sorted list for the tt objective values in P. and (A)) is the 4t value in this sorted list. Lower sparsity is better.
22

---

Published as a conference paper at ICLR 2025
Table 10: C-MORL-IPO Hyperparameters for continuous baselines.
parameter name
6
6
6
9
N
5
log barrier coef
20
20
20
20
20
20
20
construint
Pureto extension step
60
100
60
100
60
100
100
time step [x 10%)
2.0
1.5
2.0
1.5
2.0
2.5
For metrics evaluation, we evenly generate an evaluation preference set in a systematic manner with specified intervals A 0.01, A = 0.1, and A 0.5 for benchmarks with two objectives, three or four objectives, and six or nine objectives, respectively. For example, for benchmarks with two objectives, we sample preference vectors covering range of preferences from =00, 1] to w = [1,0] 0] with specified interval A 0.01, totally 1,0) preference pairs. For C-MORL and other multi-policy baselines, the calculation of metrics is the same. For single preference-conditione policy methods, the calculation of metrics slightly different. Specifically, since in a multi-pplice setting, the policies in the Pareto set are iefference-ireleeanne Pareto front to compute hypervolume and sparsity. For single veference-conditione first evaluate the solutions using all the preferences in the evaluation preference set. Then, the non-dominated solutions that form the Pareto front are used to compute hypervolume and sparsity. Expected utility is evaluated for multi-policy baselines based the evaluation preference set, and the execution results on all preferences from the evaluation preference set are utilized for single seeference-condition policy methods.
we directly use all solutions in the
policy methods,
F PROCEDURE OF SOLVING C-MORL-CPO AND C-MORL-IPO
C-MORL-CPO
C-MORL-CPO Rather than applying sampling-bass approaches (Duan et al. 2016) for find- ing policy updates in the relaxed formulation, empirically
follow the adapted trust region method (Schulman et al. 2015) for CPO updates (Achiam et al. 2017):
4r+1, = arg max (GT GT 2 BGG i= 1, n, ((T,, 53. #Ells
(17)
The trust region set is defined as {To E II. DKL(T||TT) S 8), where DEL denotes the estimated mean of Kl.-divereence given state 8, and serves as a surrogate function of the original distance function. Such trust region optimization updates hold guarantees of monotonic performance improve- ment and constraint satisfaction. It noteworthy that the original CPO algorithm requires a feasible initialization, which by itself can be very difficult, especially with multiple, general constraints involving policy returns (Zhou et al.) 2022). While in our formulation for solving MORL, we can almost guarantee the initial policy is always feasible for the solving process of extended policy with properly selected 3 along with a well-initialized policy set.
However, solving this problem requires evaluation of the constraint functions to determine whether a proposed point æ is feasible. Therefore, follow (Achiam et al.) 2017). we replace the objectives functions with surrogate functions, which are easy estimate from samples collected m,- To be specific, we solve the following optimization problem to approximately solve Eq. 17
Tr = arg max E deadrraco aarm [A [AT (s)a)) rEllo
s.t. G.
Bawavr and
w (s, 2 BG. i= 1.,,...,,,1a (18)
1-
(KI(T||TT)) S.
However, Eq. 18 is impractical to be solved directly especially when the policy is parameterized
neural network with high-dimensional parameter spaces. We follow (Achiam al. (017) to
23

---

Published as a conference paper at ICLR 2025
implement approximation to the update Eq. that can be computed efficiently:
0 0 arg max (0 0-)
s.L. Gj + (0 0,) < 01 n,...,,,i
(19)
1
2 (0 0,) "0- H 0,) 5.
where g the gradient of the 1th objective that is chosen be optimized, b; denotes the gradient of the jth objective served as constraint, // is the Hessian of the KL-divergence, and G = 3G G The dual problem of Eq. 19 can be expressed as:
max A20
H 2rv Sv) + vF c
(20)
2X
2
0
where = [bi..... by]. [cI...., Cnl Cnl", r --TH-11 and S = =B" B"H-'B, We derive X", by solving the dual, then the solution to the primal Eq. equation is
1 X- (g -B="")
0- = +
(21)
In our implementation, we adopt C-MORL-CPO to solve MORL tasks with two objectives, i.e.,
constrained optimization problem that only involves single constraint. Therefore, directly compute the dual variables X and y* with the analytical solution as follows Achiam et al. (2017):
X' x--rr)
+
(22)
8 - 9) (A())= = (1 X6)
fa(A) = 2A
2
+
n2 -
if Ac - 0 otherwise,
X arg max
A20
with q = 9.= the constrained optimization towards the corresponding objective direction in the current Pareto extension iteration will be terminated.
and b" b. If the constrains in Eq 18 are not satisfied,
F.2 C-MORL-IPO
we augment the objective function of the objective that is being optimized with logarithmic barrier functions for other constrained objectives. Note that the logarithmic barrier functions can be integrated with any other policy optimization methods, in this paper, we follow (Liu jet al. (2200) to integrate with PPO (Schulman et al. 2017) for training our policies. Therefore, the surrogate objective becomes:
Recall in Section 5.3
max CLIP (0)
+ Σ (G)))
(23)
where ((GT) Log(G]-001> and is hyperparameter; CLIP the clipped surrogate objective of PPO. As approaches co, the approximation becomes closer to the indicator function.
G ADDITIONAL RESULTS
G. PARAMETER STUDY FOR NUMBER OF EXTENSION POLICIES
Table 11: Ablation study C-MORL for the number of extension policies MO-Ant-3d benchmark.
N=6
N=12
N=18
HV(10")
4.03=1017
4.21
4.20=0.27
(U0))
2.59+0.08
2.65+10.08
2.63+1.111
(P(
2.91=0.90
2.00
1.53=0.23
is more difficult to derive Pareto-optimal policies to cover the entire Pareto front for the continuous MORL tasks with more than two objectives. In this subsection we look into different settings for
24

---

Published as a conference paper at ICLR 2025
C-MORL and associated impacts the algorithm performances. Table presents the results of the ablation study for the number extension policies on the MO-Ant-30 benchmark, which is one of the continuous MORL tasks with numerous objectives. We can observe that when N 12. C-MORI. can derive a Pareto front with much higher hypervolume and lower sparsity than that of sufficient to be extended to fill the Pareto front for this task. Fig. 7 further illustrates the Pareto extension process with varying values of the number of extension policies (N 6, 12, 18) on the MO-Ant-3d benchmark. It can be observed that as the number of policies selected to be extended increases, the hypervolume also increases. Notably, Fig. highlights that when N 18, there is more comprehensive exploration of the objectives of Y velocity. This observation suggests more thorough exploration of sparser areas on the Pareto front.
N 6. When N 18, the result similar, indicating that 12 policies
K 0
K 30
K 60
Peretu frunt
Paretu front
Pereto front
Selected solution
Selected solution
5 Control cost
Control
Control
N = 6
100 velocity
o 100 200 300 400 s0600 x velority
veloci
100 200 300 400 0
0 100 200 300 400 50600
velecty
velacity
X velocity
so600
Pareto front
Pareto front
Pereto front
solution
100
100
Control cost
000
Control cose
Cantrol
N = 12
20
100 200 go0 velocity
valoc
100
20p
veloc
100
100 200 go0 400 velacity
200
vetocht
100
300
200 velacity
30g
30g
Pereto front
Pareto front
Selected solution
Selected solution
Control cost
N = 18
Control
6
Control cost
100 velocity
100 velocity
0 100 y00 you 400 sodo0 x velocity
0 100 200 you 400 s0600
20g
100 200 you aco x volscay
velocity
Figure Ablation study of C-MORL for the number of extension policies on MO-Hopper-v3 benchmark. Number of extension policies on MO-Ant-3d benchmark N 6, 12, 18 and number of Pareto extension step K 0, 30, 60, respectively.
G.2 ABLATION STUDY ON BUFFER SIZE.
In order better understand how Buffer size can influence the performance of C-MORL, we provide the comparison of the best baseline and C-MORL with various buffer sizes, as shown in Table 12 and Table 13 These studies show that even with reduced buffer sizes, C-MORL maintains highly competitive performance. Compared to PG-MORL, our C-MORL achieves superior results while requiring significantly fewer policies. Another notable observation is that, some benchmarks, the results remain consistent regardless of the buffer size. This consistency indicates that the number of Pareto optimal policies in these cases does not exceed the buffer size.
G.3 EXPECTED UTILITY RESULTS
C-MORL outperforms other methods in terms of expected utility across nine out of ten benchmarks, which can be attributed to two significant advantages. The first advantage is related to the inherent characteristies of the multi-policy approach. As illustrated in Fig. 8 single preference-conditionee
25

---

Published as a conference paper at ICLR 2025
Table 12: Evaluation of HV, EU, and SP in discrete MORI. tasks under varying buffer sizes, alongside a comparison with the best-performing baseline.
HV(10*)
GPPLLS)))
6.22:4:.70
6.57:10:22
6.63:0.89
6.77::0.88
Minecart
BU(10
G.
1888+00599
2.05::0.67
2.09:0.65
2.12::0.66
SP(10
0. 0,10=0000GGPLLSS
0,64+0.17
0.19-0.02
0.09=0.03
0.05=10.02
IVV(10))
1.0610.16GGPLLL.
1.011.0.07
1.08
1.12 0.03
1.12:10.03
MO-Lanar-LLnde
RU(10'
1.81 1.1
1.841:0.31
2.21
0.15
2.35-10.18
SP(10*)
1.14:12.14
1.04=1.24
1.04::1.24
HV(10*)
2.34::0.29
2.86=00119
3.17=0.20
3.67=0.14
Fruit-Tree Tree
EU
6. 6.15 055500000Fieeeoee//PGP....
6.144:0.13
6.38-10.00
6.46=0008
6.53:10.03
0:336002/GPI-LSS)
2.62:4:049
1.65
0.81=0.12
Table 13: Evaluation of HV, EU, and SP in continuous MORL tasks under varying buffer sizes, alongside comparison with the hest-performing baseline.
Enviromments
Metrics
Best Baseline
B=20
B=50
B=100
B=200
IV(30%))
1.26.10.0010 Pensieve)
1.39.10.01
1.39.10.01
1.39.10.01
MO-Hopper-2d
EU(102)
2.34-0. 10PG (ORI)
2.55:10.01
2.56::0.22
2.56:11.02
SP(10
0.HGL0.10(CAPQL)
2.68+1.06
IV(10')
L.70.0.2%KGPILL))
2.20110.44
MO-Hopper-3d
EU(10")
1.62:10. IXGPPILL
1.72.40.88
1.78-1.02
1.80:1.001
1.80:10.01
SP(10")
1,74=1.22(GPI11
7.61
2.74
0.86-20.27
0.28-10.09
IV((10))
3.10 (.26...)5)
3.08
3.13
1.22
MO-Ant-2d
RU(10))
(.2...))))))I-LL))
4.27100.19
4.2911..1
4,29 1.291
SPi10")
IIV(10))
3.82.1..33IQ. Pemsieve)
3.52.10.16
3.8311.117
4.00:10.12
1.01110.11
MO-Ant-3d
EU(10")
2:41+0.000PP--LS)
2.48:00099
2.55=00.8
0.57
0.57=10.06
SP(10")
0.0:10.0111G-MORL)
1.56:20.67
0.30::0.19
0.08:10.05
0.03:10001
HV(10")
0500:O)00CCAPQL)
3.43:0.06
3.43=20.0
0.6
MO-Humanold-24
EU(10")
4.75:0..(CCPOLL)
4.88+0.05
4.8==0005
4.78:10.05
SPi10*)
(LO.MICCAPLLGGPPLLL
0.18
0.18:20.7
HV(10*=)
1.00H00000-Pemiieee)
1,1420.00
1.1420.00
L1S:10.00
1.15:10.00
Building-3d
EU(10*)
0.96:000000PPnnsieee
1.0200.000
1.0200.00
1.0000.000
1.02:1:0.00
SP(10*)
0,3710221GGMMORL
10420.16
199820.38
0.69:10.62
0.69:1:062
HV(IO
7.6420.17
1.93=00.07
0.93:10.07
7.93:100007
Building-9d
EU(10*)
3,50+0.00
3.52-00.00
3.52:10.00
3.52:1-0000
SP(10*)
0.10=
1116:0.19
0.28:20.03
0.28:10.04
0.28:10.04
Retums
Pareto optimal
Pareto optimal
100
Control cost
1000
Control cost
60
0
40
50
20
100
200 velacity
Y velocity
200
200 X velocity
400
200 X velocity
40000
(a) Q-Pensieve
(b) C-MORL
Figure 8: Evaluation results of sampled preferences on MO-Ant-3d benchmark. (a) Q-Pensieve evaluation. Returns of evaluated preference pairs with interval 0.1 are marked with orange points, while Pareto-optimal solutions are marked with blue points. (b) C-MORL evaluation. Pareto-optimal solutions are marked with blue points.
26

---

Published as a conference paper at ICLR 2025
policy approach does not guarantee that each preference sampled to be evaluated corresponds to a solution on the Pareto front. The Pareto front exclusively encompasses solutions that are Pareto optimal, while the majority of preferences do not lead such optimal solutions. This limitation arises from the inherent difficulty of achieving Pareto optimal for every preference in this kind of approach. In contrast, the multi-policy approach yields Pareto solution set that exclusively comprises a Pareto-optimal solution set that is irrelevant of specific preferences. Consequently, when presented with unseen preference, one can simply select the solution with the utility (i.e. the SMP in Eq. 3.1) from this pre-existing set.
H DISCUSSION AND LIMITATIONS OF C-MORL
With the novel design of Pareto initialization, policy selection, and Pareto extension, the proposed method gives a novel and systematic approach on exploring the Pareto front and optimizing the policies. Despite the capability of our approach to effectively populate the Pareto front, we observe that some benchmarks, the current constrained policy optimization method fails to adequately extend the policies toward certain objective directions. Consequently, there still exist unexplored areas on the Pareto front. To address this issuc, we plan to develop more effective extension method. Additionally, since our implementation is based on PPO, its training efficiency is outstanding, but its sample efficiency is relatively low. In the future, we plan to develop more sample-efficies methods for MORL.
27


Reviews:
[{"review_id": "AfR0ysPAF4", "reviewer": "Reviewer_LYRe", "summary": "This paper proposes a two-stage algorithm, Constrained Multi-objective Reinforcement Learning (C-MORL), to efficiently discover the Pareto front in multi-objective reinforcement learning (MORL) problems.  The first stage, Pareto initialization, trains a set of policies in parallel, each with a specific preference vector.  The second stage, Pareto extension, selects policies from the initial set based on their crowd distance and performs constrained optimization to maximize one objective while constraining others.  This approach aims to address limitations in existing MORL methods, such as difficulty covering the complete Pareto front.  The authors demonstrate the effectiveness of C-MORL on various MORL benchmarks, including discrete and continuous control tasks with up to nine objectives, showing superior performance in hypervolume, expected utility, and sparsity compared to baselines.", "questions": "How difficult is to determine the specific values for hyperparameters like the number of initial policies, the number of extension policies, and the constraint relaxation coefficient in the experiments? Does this require problem-specific knowledge such as the range of rewards?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "--Originality: The paper introduces a novel perspective by combining constrained policy optimization with MORL, leading to a unique two-stage algorithm for Pareto front discovery. \n\n--Quality: The paper provides a rigorous theoretical analysis of the proposed C-MORL algorithm. The experimental evaluation covers a diverse set of MORL benchmarks with varying state/action spaces and objective dimensions.   \n\n--Significance: The proposed C-MORL algorithm provides a more efficient and effective way to discover the Pareto front, especially for complex tasks with high-dimensional objective spaces.", "weaknesses": "Clarity: While the core idea is clear, in general it's difficult to follow this paper. The paper could be made more accessible. The algorithm has many stages (Pareto initialization, policy selection, Pareto extension) with different hyperparameters (number of initial policies, number of extension policies, constraint relaxation coefficient) and design choices (C-MORL-CPO vs. C-MORL-IPO, crowd distance vs. random selection). The paper could benefit from a more streamlined presentation of the algorithm and a clearer explanation of the implications of these different choices. Many of these design choices are not studied and the implication of them is not clear. I would suggest the authors move some of the theoretical analysis to the appendix and use the space to discuss these design choices and hyperparameters better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IoYBomMPr5", "reviewer": "Reviewer_JTtx", "summary": "This paper introduces a novel approach for solving multi-objective reinforcement learning (MORL) problems using a two-stage algorithm called Constrained MORL (C-MORL). The authors proposes to frame the MORL problem as a Constrained MDP. Specifically, in the first stage, C-MORL trains a diverse population of policies across various preference vectors, initializing a set of initial solutions. The second stage then improves these policies via constrained optimization, guaranteeing a targetted expansion of the covered Pareto front.  The authors addressed challenges with existing methods, including limited Pareto front coverage and low training efficiency. The authors show that C-MORL achieves state-of-the-art results on multiple MORL benchmarks in terms of hypervolume and expected utility metrics.", "questions": "**1. How are comparisons between GPI-LS/CAPQL vs PGMORL/C-MORL made?** \n\nAs far as I know, CAPQL and GPI-LS are linear scalarization methods and conventionally they are evaluated by sampling preference vectors uniformly across the unit simplex, to form their approximate Pareto front. For evolutionary methods however, they usually keep an archive of policies and their associated vector performances, and then the approximate Pareto front is made during evaluation by filtering the non-Pareto dominated policies. However, this can be concerning during evaluations, especially for metrics like sparsity. If you allow CAPQL/GPI-LS only 100 preference vectors for evaluation, but allow PGMORL/C-MORL and unbounded archive of policies for evaluation, the metrics are naturally going to gravitate in favor of the latter group. Clarification w.r.t. to this aspect needs to be made for me to be more certain about the results of the paper.\n\n**2. How would C-MORL react to concave Pareto fronts?** \n\nIn the end of the paper, you mentioned that \n> *\"...observe that in some benchmarks, the current constrained policy optimization method fails to adequately extend the policies toward certain objective directions\"*. \n\nI am curious as to how C-MORL would react to concave pockets on the Pareto fronts? Could the constrained optimization cause the policy to prefer easier paths in convex regions?\n\n**3. Please answer questions in weaknesses**", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Good presentation** \n\nFirstly, the presentation of the paper is well-done, motivations and preliminaries are well set-up from the beginning, and the flow of the paper was coherent. It was easy to read.\n\n**2. Novelty** \n\nThe proposed idea is novel in the *MORL* field. It seems to be inspired by approaches from the MOO field, such as Epsilon-constraint methods, but the crossover is done in a creative way that addresses the unique challenges of sequential decision-making setting in MORL. Most contrained MORL problems are more focussed on the aspects of safety constraints, so formalizing the evolution of policies as a constrained optimization problem is novel in MORL, to my best knowledge.\n\n**3. Theoretical rigor** \n\nThe paper provides sound propositions and proofs; albeit built upon theoretical foundations already established in CMDPs. One of the more interesting contributions is Proposition 4.3, which provides a practical guideline and sufficient condition for a constrained optimization problem to yield Pareto-optimal solutions. This aspect is a novel adaptation tailored to the requirements of MORL. The condition provided by the proposition is not overly restrictive or computationally intensive to verify. The complexity proof (proposition 5.1) is compelling, and provides strong support for using C-MORL over PGMORL\n\n**4. Extensive experiments** \n\nThe authors did a *very* extensive set of experiments across *10* distinct benchmarks, both discrete and continuous. The results are clearly indicative of significant improvements over existing approaches.", "weaknesses": "**1. Comparisons against utility-based/preference-conditioned methods can be improved** \n\nThe paper mentions that preference-conditioned methods approaches struggle with scalability, I don't fully agree with that. Evolutionary approaches like C-MORL face significant scalability limitations too, for e.g., you can't evolve a population of CNN-based policies (agents) without significant memory overhead. This is worrisome especially if we want to deploy agents to real-world setting with image-based observation spaces. Next, evolutionary approaches seem ideal because you can learn many densely-packed solutions w.r.t. tracing the true optimal Pareto front but they present practical challenges when it comes to policy selection. It is unclear how different users can select the policies they desire for evolutionary approaches whereas utility-based/preference-conditioned methods places these preferences at the forefront. These concerns do not critically impact my overall assessment of this paper, but addressing these points would help clarify the limitations and appropriate use cases of C-MORL for the research community.\n\n**2. No comparisons against the MORL/D algorithm** \n\nThe MORL/D algorithm by [1] is very similar to PGMORL and C-MORL, all three of which are evolutionary approaches with populations initialized using preference vectors sampled across the unit simplex. It has state-of-the-art results across many benchmarks, and demonstrates a vast improvement over PGMORL. A significant drawback of PGMORL is perhaps the reliance on prediction models to guide the training, which MORL/D does not depend on. The algorithm is provided in the same code repository as GPI-LS, i.e. [2]. I can imagine the differences between C-MORL and MORL/D in terms of performances being much tighter. Comparisons against this method would be necessary for the results to be more compelling. Specifically, testing against MORL/D with PSA weight adaptation would be very much desired.\n\n[1] Felten, Florian & Talbi, El-Ghazali & Danoy, Grégoire. (2024). Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework. Journal of Artificial Intelligence Research. 79. 679-723. 10.1613/jair.1.15702. \n\n[2] Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El-Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. 2024. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1028, 23671–23700.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oboKDyjozj", "reviewer": "Reviewer_dWup", "summary": "This paper introduces a novel method for learning a Pareto set of policies for solving multi-objective reinforcement learning (MORL) problems in which the reward is a vector composed of multiple conflicting objectives. The paper introduces Constrained MORL (C-MORL), an algorithm that maintains a population of policies to approximate the Pareto front. At each iteration, it learns novel policies that are optimized to solve a constrained MDP such that the solution to it lies in the Pareto front. The method was evaluated in several benchmarks with varying numbers of objectives and characteristics in terms of different MORL evaluation metrics.", "questions": "Below, I have questions and constructive feedback to the authors:\n\n1) “(Alegre et al., 2023) further introduce a novel Dyna-style MORL method that significantly improves sample efficiency while relying on accurate learning of dynamics”.\nThey also introduce a model-free version of their algorithm (GPI-LS), which parallelizes the training of the CCS.\n\n2) Minor comment about the notation: The letter $G$ is usually used in the RL literature to denote the random variable of the return, and not its expected value. The expected value is usually denoted with the letter $J$ or by $V(S_0)$. I would suggest considering changing this notation to avoid confusion.\n\n3) In the definition of the space of preference weights, $\\Omega$, you should also restrict $w_i\\geq0$. \n\n4) It is not clear how the projection operators $\\Gamma_\\lambda$ and $\\Gamma_\\pi$ are defined. What does it mean to project $\\lambda$ and $\\pi$ into a compact and convex set?\n\n5) In Proposition 4.4, it is not clear how the policy update works. Are you assuming $pi_r$ is a parameterized policy? Is this equation updating the parameters of the policy? What is a “fixed point of MORL policy”?\n\n6) In Theorem 4.5, what is the Slater’s condition?\n\n7) “we also maintain a solution buffer to enhance policy diversity.” What is a solution buffer, and how does it enhance policy diversity? Tihs is not explained in this section.\n\n8) In Eq. 3, how is the hyperparameter $\\beta$ selected? How can it impact the optimality of the method, in theory?\n\n9) In Eq. 5, how is the parameter $t$ of the function $\\phi$ selected?\n\n10) I am not sure Proposition 5.1 is making a fair statement. Although the algorithm running time is linear in the number of extension policies $N$, it is expected that the number of policies $N$ required to cover the Pareto front would probably increase exponentially with the number of objectives. Of course, a user can use the same value of $N$ independently of the number of objectives $n$, but that would probably be sub-optimal. That is, it is expected that one should increase $N$ as $n$ increases.\n\n11) It is not very clear in the paper when C-MORL-CPO or C-MORL-IPO are used. Which one is used in the proposed algorithm?\n\n12) Why does PG-MORL have to solve a knapsack problem with $K$x$N$ points? This is not mentioned in the paper of Xu et al., 2020. \n\n13) In Table 2, the authors denote in bold some metrics which are overlapping with the competitors. For instance, in MO-Ant-2d, HV of $3.10\\pm0.25$ and $3.13\\pm0.20$ and EU $4.28\\pm0.19$ and $4.29\\pm0.19$ overlap.\n\n14) There are many experimental details missing in the experimental section:\n- How many random seeds were used in each experiment? Are the metrics averaged over how many runs?\n- Which version of CAPQL and GPI-LS did the authors use? How were the hyperparameters of the competitors chosen?\n- CAPQL and GPI-LS are based on SAC and TD3, while C-MORL is based on PPO. This could significantly impact the results, and it makes it difficult to infer whether the performance gains come from SAC vs. PPO or the expansion technique introduced in the paper.\n- Alternatively, did the authors reimplement CAPQL and GPI-LS to use PPO as the underlying policy learning technique?\n- How were the PFs in Figure 4 generated? Are those points average over how many episodes? How many weight vectors were used to evaluate the learned policies for different utilities?\n\n15) The authors mention that “each method is tested across three\nruns with the same seeds”. Three runs are very unlikely to be sufficient to provide significant statistical results. \n\n16) I suggest including the ablation of the Pareto Extension procedure and comparing it with solving for random linear weights since this procedure is arguably more important than the Policy Selection procedure. It would be important to observe if the overhead of employing CPO or IPO is really providing performance gains.\n\n17) The idea of the paper of solving constrained MDPs to identify new policies in the Pareto front is similar to that of (Willem et al., 2024, “Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning”). Please add a discussion explaining the similarities and differences.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The problem of deciding which directions in the objective space are more relevant to learn when approximating a Pareto front is very relevant to the MORL community.\n- The evaluation considers many challenging benchmarks with discrete and continuous action spaces and up to 9 objectives.", "weaknesses": "- The paper can significantly be improved in terms of clarity (see below).\n- The method requires learning a separate neural network for each policy, which does not scale in terms of memory. This is in contrast with previous MORL algorithms that train a single model conditioned on the weight vector $w$.\n- There are many experimental details missing in the paper that need to be addressed (see Questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYFSKoQqDq", "reviewer": "Reviewer_EBHH", "summary": "This paper introduces C-MORL, a two-stage policy optimization algorithm aimed at efficiently discovering the Pareto front through a constrained optimization framework for multi-objective reinforcement learning (MORL). C-MORL effectively manages complex tasks with multiple discrete and continuous objectives, as demonstrated in experiments involving up to nine objectives. To achieve solutions without additional computational costs, the paper presents an interior-point-based approach for solving a relaxed formulation, ensuring the derivation of Pareto-optimal policies under specified conditions. Extensive evaluations across diverse MORL benchmarks, including robotics control and sustainable energy management, highlight C-MORL’s superior capability to explore a broader Pareto front under unseen preferences.", "questions": "1. **Selection of Mujoco Environments:**\n   Why were only certain environments from the Mujoco suite selected as baselines for demonstration while other significant environments were omitted?\n\n2. **Unclear Initialization and Preference Distribution:**\n   The paper does not clearly explain how many policies are selected during initialization, how many corresponding weight preferences are chosen? And how they are distributed?\n\n3. **Efficiency and Resource Intensity:**\nThe main idea of selecting policies for training based on crowding distance and storing all intermediate policies in a buffer is straightforward. However, constructing the CCS through a vast number of policies seems highly resource-intensive and inefficient. In each MO-problem, How many policies do you store, and how many are used to construct the CCS? Compared to PGMORL, do you utilize more or fewer policies in this process?\n\n4. **Selection of Extreme Solutions:**\n   Since extreme solutions have already reached performance bottlenecks on particular objectives, selecting them by default does not facilitate the exploration of the potential Convex Coverage Set (CCS). Why are they selected as default for next training round?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper builds a solid foundation for its optimization approach by employing extensive constraint theories. The experimental section thoroughly evaluates state-of-the-art MORL methods on multi-objective optimization problems, with a comprehensive analysis specifically on the newly introduced Building-9D benchmark. The proposed two-stage method partially addresses certain challenges in multi-objective optimization (MOO). The conclusions align well with the research questions posed, demonstrating consistency between the methodology and findings.", "weaknesses": "1. **Lack of Innovation and Convincing Results:**\n   The structure of the paper is reasonable, but the main idea and innovation seem insufficient. The contribution appears to be limited to encapsulating PGMORL or any single RL algorithm by using crowding distance to select policies for constrained optimization. For preferences that have not been trained, the method merely matches them to an existing best policy without discovering new policies corresponding to those preferences. This results in policies that are locally optimal rather than truly globally optimal in the CCS, which is unconvincing.\n\n2. **Selection of Extreme Solutions:**\n   Since extreme solutions have already reached performance bottlenecks on particular objectives, selecting them by default does not facilitate the exploration of the potential Convex Coverage Set (CCS). This approach may limit the discovery of more diverse and optimal policies.\n\n3. **Efficiency and Resource Intensity:**\n   The main idea—selecting policies to train based on crowding distance and storing all intermediate policies in a buffer—is straightforward. However, constructing the CCS through a vast number of policies is highly resource-intensive and inefficient. This method may not be practical for large-scale applications due to its computational demands.\n\n4. **Time Complexity Comparison:**\n   Given that the work uses PGMORL for initialization, comparing the time complexity to PGMORL is not entirely fair since it is already part of the proposed algorithm. A more appropriate comparison would isolate the additional computational overhead introduced by the proposed method.\n\n5. **Unclear Initialization and Preference Distribution:**\n   The paper does not clearly explain how many policies are selected during initialization, how many corresponding weight preferences are chosen, and how they are distributed—an important aspect of the methodology. In subsequent policy selection and optimization, the distribution of preferences remains unchanged. This could lead to policies converging around certain preferences, hindering further exploration of the CCS. While PGMORL addresses this issue effectively with a utility prediction method, your paper does not mention the problem. Consequently, it is unclear whether the filling of CCS gaps in the experimental results belongs to the same cluster.\n\n6. **Overuse of Formulas and Theorems:**\n   The paper includes a large number of formulas and theorems, which detracts from the reading experience. These mathematical details contribute little to the core algorithm, especially regarding policy selection and assignment. Simplifying or reducing the emphasis on these formulas could improve clarity and accessibility.\n\n7. **Inconsistency in Method Presentation:**\n   Similar to the previous point, C-MORL-IPO and C-MORL-CPO are defined with extensive formulas and theorems, but only C-MORL is used in the experimental results. It is unclear which method is employed in the policy extension process of C-MORL (i assume IPO?). These two methods do not appear in the baselines, nor are there corresponding ablation studies to support them, i.e., there is no evidence in the main results support the claim 'C-MORL-IPO is more computationally efficient than C-MORL-CPO'. Their reappearance in the supplementary materials without integration into the main results creates a fragmented reading experience and causes confusion.\n\n8. **Statistical Significance in Comparison Results:**\n   In the comparison results table for the MO-Ant-2d environment under HV and EU metrics, the error of the standard deviation between C-MORL and GPI-LS exceeds the error of mean difference. Therefore, it cannot be considered as a solid surpassing to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)