Hypervolume Maximization: A Geometric View of Pareto Set Learning
Authors: Xiaoyuan Zhang
Xi Lin
Bo Xue
Yifan Chen
Qingfu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: multiobjective optimization;multitask learning;hypervolume maximization;Pareto set learning
TL;DR: This paper build the connection between Pareto set learning and hypervolume optimization

Abstract: This paper presents a novel approach to multiobjective algorithms aimed at modeling the Pareto set using neural networks. 
Whereas previous methods mainly focused on identifying a finite number of solutions, our approach allows for the direct modeling of the entire Pareto set. 
Furthermore, we establish an equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis of hypervolume (as a new metric) for Pareto set learning. 
Specifically, our new analysis framework reveals the connection between the learned Pareto solution and its representation in a polar coordinate system. 
We evaluate our proposed approach on various benchmark problems and real-world problems, and the encouraging results make it a potentially viable alternative to existing multiobjective algorithms. Code is available at \url{https://github.com/xzhang2523/hvpsl/tree/master}.

Hypervolume Maximization: A Geometric View of Pareto Set Learning
Xiaoyuan Zhang", Xi Lin", Bo Xue Yifan Chen', Qingfu Zhang"
Department of Computer Science, City University of Hong Kong;
City University of Hong Kong Shenzhen Research Institute. Departments of Mathematics and Computer Science, Hong Kong Baptist University.
Abstract
This paper presents a novel approach to multiobjective algorithms aimed at mod- eling the Pareto set using neural networks. Whereas previous methods mainly focused on identifying a finite number of solutions, our approach allows for the direct modeling of the entire Pareto set. Furthermore, Ience between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis hypervolume metric) for Pareto set learning. Specifically, our new analysis framework reveals the connection between the learned Pareto solution and representation in a po- lar coordinate system. We evaluate our proposed approach on various benchmark problems and real-world problems, and the encouraging results make a poten- tially viable alternative to existing multiobjective algorithms. Code is available at
establish equiva-
new
Introduction
Pareto solutions [1] effectively balance multiple objectives, making optimal tradeoffs among all objectives. For Pareto solution, improving one objective without adversely affecting others is not possible. The Pareto set (PS) refers to the set all Pareto solutions, while the Pareto front (PF) denotes the image of the PS in the objective space. Over the years, numerous methods have been developed to generate a single Pareto optimal solution [2, 3, 4], or a finite set of Pareto solutions [5, 6, 7,8,9) 9].
Under mild conditions on continuous optimization problems with m objectives, the Pareto be- comes (m-1)-D continuous manifold [10, 11]; given that, the finite set learned from classical methods might not accurately approximate the continuous manifold. To overcome this issue, Pareto set learning (PSL) has been recently proposed in [12, 13], with the hope of training neural net- work model capable of approximating the entire Pareto set. The significant advantage learning the complete Pareto set is that trained model can generate the optimal solution for any preference (conceptually the weight of different objectives, see the definition Lemma real time, allowing for comprehensive decisions.
The training approach utilized in prior PSL techniques can be viewed as an extension of decomposition- multiobjective optimization strategies [5] to an infinite preference scenario. However, several unresolved issues existed prior PSL methods. Firstly, the interpretation of the sum of aggregation functions under different preferences lacks clarity, simply taking all preferences as equally important and thus leading to partial Pareto front. Secondly, aside from the EPO-based
Correspondence to: Yifan Chen <yifanc www.b..duu............ Qingfu Zhang Sqiinfu.zzaagg @ edt.u.edu.hk>>>
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

PSL proposed in [3], there has been limited investigation into the connection between a preference and its corresponding Pareto solution. Despite the effectiveness the EPO-based PSL approach, its computational latency has been identified as significant obstacle in its ability to handle large- scale problems. Thirdly, theoretical guarantees of PSL, are lacking, and therefore the underlying mechanism of PSL. remains unknown.
After identifying the challenges regarding existing PSL techniques for solving multiobjective opti- mization problems, we propose a new approach that addresses all these issues. Our method utilizes a geometric perspective on Pareto learning, which takes the form of hypervolume maximization problem. This approach brings several benefits. Firstly, it provides clear interpretation of Pareto set learning as a hypervolume maximization problem, allowing us build a theoretical bound be- tween the PSL results and the hypervolume, one of the most important indicators in multiobjective optimization. Secondly, our approach establishes a clear correspondence between specific pref- erence and the resulting Pareto solution in polar coordinate system. Lastly, the new perspective implies several important techniques in hypervolume-
PSL. learn an entire Pareto set.
The contributions of this work can be summarized as follows:
1. We provide novel geometric perspective for Pareto set learning, which recognizes its equivalence to hypervolume maximization. As a result, the Pareto solutions derived from this method are aligned precisely with the polar angle under mild conditions, which is the input of the neural network in use.
2. We provide the first theoretical analysis for Pareto set learning, specifically examining the gap between the estimated hypervolume and the true hypervolume of the Pareto set. Our analysis enhances better understanding of Pareto set learning.
3. We have developed techniques to learn a more complete Pareto front, surpassing previous methods that obtain partial solution set. Our approach achieves promising results various benchmark problems and real-world problems.
Related works
Multiobjective Optimization via Hypervolume Maximization. The hypervolume (hv) indica- tor measures the quality of multiobjective optimization solution sets and consistent with Pareto dominance. Maximizing hypervolume is basic principle in multiobjective optimization algorithm design, and several popular algorithms like SMS-EMOA [14], SIBEA [15, 16], and MO-CMA-ES [17] use it to generate finite number of optimal solutions. A new method named multiobjectiive learning using hy maximization was proposed in [18], which leverages hypervolume maximization while training neural networks to obtain a finite set of Pareto solutions. This method differs from our approach as it does not rely on a model to approximate the Pareto set. Instead, it adopts the gradient search method from [19] for hypervolume maximization. A major drawback of this approach is its high computational complexity in obtaining the hypervolume gradient.
Pareto Set Learning and Conditional Models. Pareto set learning involves using neural net- work conditioned on user preference to learn the entire Pareto set. This concept was first introduced in [12, 13] and has gained popularity in various areas like drug design [20]. multitask image classifi- cation| 22], and multiobjective neural combinatorial optimization [23]. In addition, some studies have used similar conditional model to generate set of diverse solutions, including the works ad- dressing multiobjective reinforcement learning [24, 25, 26), and the works generating policies under different conditional levels 227.
3 Background
multiobjective optimization (MO0) problem with m objectives can be stated as follows:
min f(x) = fz). fm. (x)),
(1)
ZEX'R"
where is the decision variable, X' is compact decision space, and f(x) X' Rm is the objective
function. The objective domain y f X' Given two candidate solutions (aa)) (b) X,
TThe function f applied to set A produces set B, denoted B (vlf(x), E A).

---

say z(a) dominates 2²1h) if and only if i [m] 2 f((x(1)) I f((x(1)) and 3j E [m] \ such that (xla)) < fz(x(()). x" is Pareto solution if no other solution E N' dominates x x";its image (a*)) is called a Pareto objective. The set of all Pareto solutions is called the Pareto set (PS). And the image of PS is called the Pareto front T == f o PS. In separate note, solution x' called weakly Pareto optimal if there exists no solution E X' such that fz(z) < fz(2') A Pareto solution is weakly Pareto solution, but weakly Pareto solution is not necessarily Pareto solution.
The nadir/ideal point of a multiobjective problem is con- structed by the worst/best objective values of the Pareto set. yrndir = suPyeT J. and similarly, y inf vi E [m]] The hypervolume indicator is de- fined metric of the optimality of of objective vectors A, detailed as follows.
Definition (The hypervolume (HV) indicator [29]). The hypervolume indicator of set A is defined as:
Hypervolume
Paneto freet
(A) == A({a 3p E A 3 and 1 r),, where A(-) denotes the Lebesgue measure, and is a ref- erence vector. We require that Ž yradir
(2)
Parato selutions
Dominated solutions
12
14
Figure depicts the hypervolume indicator of the Pareto front, represented by the yellow region.
Pareto solutions, weakly Pareto solutions, dominated solutions, and the HV.
Figure 1:
Pareto Set Learning via Hypervolume Maximization
As shown in Figure 2, Pareto set learning (PSL) aims to learn the whole Pareto set by a model, which differs from population-based MOEAs [5, 6]. One straightforward way is learn Pareto neural model z-(())
+ F"
10
that translates a polar coordinate 0 E θ into Pareto so- lution x R", where θ = 00 2 g-mm1 After the train- ing, for any 9 O, the Pareto neural model z-{()) can directly generate approximate Pareto solution in real- time (<1s).
Masping
In this paper, we present a novel geometry view to for- mulate PSL based on hypervolume maximization. Firstly, we notice the following equivalence holds:
Poier atara
0.0
-02
0.5 f200
0.9
1.5
Proposition 1 (Equivalence between PSL and HV max-
Figure 2: PSL learns the whole Pareto set. For coordinate 0 model xx learns cor- responding Pareto solution (0).
imization). When is reference point dominating the whole Pareto front T, and A is non-dominated solution set, we have T arg m H,, (A).
Proposition shows that (A) achieves the maximal value if the model xz(+) learns all Pareto solutions. So, it is natural to convert the Pareto set learning problem with a parameter as hyper- volume maximization problem:
max H. (f f o xr))))).
(3)
B
In Equation (3), the subscript r refers to a fixed reference point satisfying 1 r, Vy E T. The objective function measures the quality of the Pareto neural model zz(()) and this objective achieves optimality only when the model learns all Pareto solutions.
4.1 Pareto Front Hypervolume as an Expectation
2 [m]] denotes the index set [1....,m) 'The symbol "> / > is used to compare two m-D vectors and 6. 2 b when a, 2 b,, Vi E [m]. and similarly. b when > b,, Vi [m]]
3

---

In this section, we are devoted to casting the optimal solution of Equation (3) (i.e., the hypervolume of the Pareto front) into a polar coordinate form. To accomplish this, we leverage an existing tech- nique to transform the hypervol- ume calculation a finite set solutions into expectation prob- lem, which formulated as the fol- lowing lemma,
Hypervouree
plx. B
pet........
Hypervolume
(0
Weatlly
(a) HV (weakly) Pareto front.
(b) HV of finite points.
Figure 3: Pareto front hypervolume calculation in the polar coordi- nate. (xx(0) is the distance from the reference point to the Pareto front along angle 9. p(z, 0) (used Equation (6)) is the projected distance at angle 6.
Lemma (Hypervolume scalariza- tion of finite set [30, 31, 32]). Let A = {y y ..... y { be set of finite objective vectors and is
reference point, 2 y.) The hy-
pervolume of set A with r can be expressed terms of an expectation,
{ max min yEA sE(ma
yb
})
(A) = C. Eg
(4)
{
(0)
where we specify 0 follows Uni the uniform distribution on O. x(0) Sm-1 is the prefer- ence vector: it is the Cartesian coordinate of f on the positive unit sphere s with >
sin 01 sin .2... sin m 1, (0 sin 0₁ sin 02... COs mm-11 ... dm mm) cos 01. Cm (m/(mm a constant that only dependent on m and I()) is the Gamma function.
1/
We notice that, it is not "economii"" to estimate the hypervolume H. (A) in Lemma 1: numerous samples from Un) are needed, while the expensive Monte Carlo estimation only provides the hypervolume for finite set A. Even when A consists of only one point, the calculation of (A) still relies on the expectation over Unif(@). However, extending the scope of the input set in Lemma 1, we discover that the hypervolume of (weakly) Pareto front containing infinite number of objective vectors can as well be expressed as an expectation.
(5) with proof provided in Appendix B.2. The distance function denoted as (xx as illustrated in Figure 3(a), represents the Euclidean distance between a reference point denoted as and the Pareto front at the coordinate of f. The distance function px (0) can be precisely evaluated using the equation provided, with the proof available in Appendix B.4.
(T) Cm Eo [px
rr (--((())) Ax
(0) max p(x, 0) max min ZEX
(6)
xe sEfm)
where p(x, 0) min
(w)
represents the projected distance of an objective vector f(x)
A,(0)
coordinate 0, illustrated in Figure 3(a). Let be the shorthand of X(0), have the following
proposition of the optimal solution of Equation (6).
Proposition 2. Given an objective vector that satisfies TI-M = T' - Min Am
D: If is weakly
'1 Pareto optimal, then y is one of the optimal solutions of g(, A) max minje(m) If y is Pareto optimal, then is the only optimal solution of gly. A).
Tr-yi A;
2:
Proof. D: If there exists a y' such that ğ(v', A) ÿ(y, A). then yf + y. contradicting the weakly Pareto optimality of y. 2 If there exists a y/ * y such that ğ(y', A) = ÿ(y, A), then y - y. and there exists at least one index where v/ < Mi- This contradicts the Pareto optimality y.
Remark 1 (On the "exact" Pareto solution). Proposition 2 establishes connection between the polar coordinate and the resulting Pareto solution. Once the Pareto neural model is well-trained, and assuming the absence of weakly Pareto solutions and the existence of exact solution [3], the

---

Pareto neural model can predict the "exact" Pareto solution aligned with the polar angle 8. Since we only utilize an efficient aggregation function (Equation (6)), this property reveals that there is no need to solve the optimization problem (Eq. (24) in (3J) in order to achieve the "exact" Pareto solution.
We close the subsection with note the case of disjointed Pareto fronts. With disjointed Pareto front T, the function (x( can serve to measure the distance between the reference point r and the attainment surface [33]. In this disjointed scenario, the expectation form of the Pareto front hypervolume by Px (0), as specified by Equation (5), still holds (c.f. Appendix B.4).
4.2 Alternative Forms of Hypervolume
In this section, train the Pareto neural model (:)()) by maximizing the alternative forms of hyper- volumes. The optimization of the value of H, o (xr()) as defined in Equation (1) is challenging. However, by utilizing Equation (6), we can choose to instead optimize an casy-to-compute form of hypervolume, denoted by (B). This surrogate hypervolume function (B), called PSL-HV1, is defined as follows:
(B) = c. (o)) Po( (0)) 0)T if p(zz( 2 0 (())(()) 0) otherwise
Here, p(xx(0)), 0) < px) is the projected distance at angle 0 as defined by Equation (6). When the Pareto neural model is supposed learn the whole Pareto set, p(zz(0),0) - px((), vO E e and thus (3) + (T).
We also provide a second surrogate hypervolume function, denoted as PSL-HV2 (derivation in Ap- pendix B.5). The Pareto hypervolume can be estimated the volume difference between the regions dominating and those that dominate the Pareto front. PSL-HV2 maximizes the following objective,
HL) [I
ideal Eo
where ps(0) px (x))) 0)m and (0) is alternative projected distance function Px (x))) max [0)= Although the Pareto neural model z²(-) theoretically has the ability to represent the complete Pareto set [34] with both surrogate hypervolume functions, empirical results show that the quality of the learned solutions is sensitive to the specific choice:
(,)
Equation (7) and Equation (8) give different performances on various tasks.
Angle 0
Neural model
Solution
Objective
Hypervolume
Xp (+)
xy
(())
TF(PF)
(R) 0g
Chain rule
(xp
()))
atT,()) (f(x)(0))
0xp(0)
Figure 4: The gradient graph above used for computing the estimated gradient of the hypervolume.
Algorithm. The goal of HV-based PSI. is to maximize the objective function, max, (3). Both Equation (7) and Equation (8) can be effectively optimized using gradient descent approaches. The empirical gradient V (B) can be obtained efficiently through backpropagation, as illustrated in Figure 4. Their analytical forms are provided in Appendix B.5. For practical algorithms, the empirical gradient is estimated from a batch of N angles ((1) 2 ...., ((N) through sampling from Un). Then the neural network is updated by + 3 + v/V 9 (B), where 77 is learning rate.
5

---

4.3 Discussion on Different PSL Approaches
PSL-HV1 and PSL-HV2 have different designs for their projected distance function p(z, 0), with
PSL-HV2 exhibiting significant growth as À approaches boundary preferences such as [0.1] or [1,0] (taking the m 2 case as an example). This property brings both pros and cons compared to PSL-HV1. The steep gradient may present numerical difficulties, but it also permits PSL to adopt larger weights and learn boundary solutions more effectively. We have observed that this feature of PSL-HV2 is particularly useful for learning the complete Pareto set, as opposed to previous PSL. methods that only learned a partial Pareto front.
To avoid only learning partial Pareto set, we propose simple strategy for PSL-HVI that involves using a larger reference point than the nadir point. This approach, which is common in hypervolume-bee based optimization methods [35,30] gives larger weights to boundary preferences and allows for a more complete Pareto set.
p(0)
>, 0.
Before our study, two primary approaches for PSL opti- mization were followed [12, 37]: optimizing the expec- tation using EPO/LS and optimizing using Tchebycheff functions. However, EPO-based PSL is slow for complex multiobjective problems and solely learns partial Pareto set, and LS-based PSL only finds the convex portion of Pareto front. Our work extends the approach in [37] by introducing a thperroluuee interpretation for PSL. We establish clear relationship between preferences and so- lutions and develop a theoretical analysis framework for PSL. Additionally, we propose methods to recover more complete Pareto set, thus improving upon the existing ap- proaches.
(x0
fz00
15
The point-wise maximum of quasi-convex functions preserves quasi- convexity. A werttve- lincar com- bination of quasi-convex functions can result in non-quasi-convex function.
Figure 5:
Statistical Guarantees on Hypervolume Convergence
This section first establishes the bounds for the generalization error of the Pareto front when using the exact distance function (0) in Section 5.1. Subsequently, we discuss the generalization bound of proposed HV-based Pareto set learning in Section 5.2.
5.1 Convergence of the Pareto Front Hypervolume
Proposition 3. Dr plz, 0) is concave when fü 's are convex. p(z, 0) is quasi-concave when f; are quasi-convex. D: Pa (0) defined in Equation (7) is quasi-concave w.r.t (0) when the objective function ((x) is quasi-conver.
Proof. The statements ! and 2 are consequences of the preservation of convexity/qqasi-coneexii by taking the point-wise maximum of convex/quasi-convex functions. The statement 3 is justified by the fact that a non-decreasing composition of a quasi-concave function preserves quasi-concavity. Additionally, it can be observed that the function fx:) xm for 2 0 (and f(x) x otherwise) a non non-decreasing function.
Remark 2. To better understand Proposition 3, analyze modified bi-objective VLMOP2 problem from [38]. In this example, let f1(x) 1.2(1 exp(-(x exp(-(x-1)
+)3)) and fa(x) 1
-1))) ) It shows that the LS-based PSL may yield multiple local optimas when using non-negative linear
Figure 5 illustrates the limitation of the linear scalarization (LS) approach [39].
combination of quasi-convex functions fi and fz, resulting local optimal Pareto models. However, it is worth noting that the function -p(x, 0) considered remains quasi-convex [40].
Assuming that b <;; S B. Vi E [m]] y E T. Let Z(0) = cm) then supp Z(0) < Cm .d m22 Let (T) = EN' Z((()) denote the empirical estimation of H,,(T) with N

---

samples. Via Hoeffding inequality, similar to [32], we have the following inequality,
2N
Pr(I7) - (T) -7) c) S 2 exp
c, B2mmm
5.2 PSL Generalization Bound
In this section we show that the proposed hypervolume metric can fit into the regular neural net- work studies and similarly enjoy the statistical guarantees error ()) - HT((T) for PSL-HV1. Specifically, the term can be interrupted as the difference between the metric of vg computed and the metric of ground truth x", indicating the quality of the MLP map z-g- We will shortly show the error above will converge with regular SGD optimizers and large enough sample size N, as expected.
We can first decompose the aforementioned error as follows,
(3) H.((T)
(1.))--.(T)) -
Hr (3) (3)
(3) -Hr
(9)
H,(8") - (5*)
+ Hr((") - (T)
in which we define (B) = So [ (0(0), mm)) as the empirical hypervolume estimation associated with parameter B. 3 is a set of reference parameters for the MLP map zg. Those errors respectively correspond to generalization enror (51), optimization eπror (E2), generalization error (E3), and approximation error (E,)) We can conceptually set the reference parameter 3 as good local maxima (in the local region of (3) for hypervolume maximization so that the approximation enror is small due to the universal approximation character of the MLP model family [41]. In analysis, assume that regular SGD technique brings 3 close enough to the local maxima 8*, resulting in a small optimization error E2.
The rest steps to show convergence is to control the generalization error E1 and E3- We present the following theorem to give the uniform convergence rate of the generalization error, which applies to both E and Eg.
Theorem (Generalization error). Let the functioon fi all be Ly-Liscchitz, and let a. be an L-layer MLP or (Wio01-1 (---oi (W10)....) with 1-Lipschitz positive homogeneous activation as's and |[Wil]p S Bw, vi E [L] (under this setting 3 is the set of W,'s). The design matrix Xa denotes the N collected samples 0,'s. We further set the reference point r so that TÜ f((x)) [b. B), Vi, for x E T U f xz (9) U xz(()))()))))))))))))))())) c R With probability at least 1 5 (8 < 1.). for
8 = 3" or B, we have
(.) (L) S Cm mm.
(222mm
Nb T,g B Xolle (1 + 22. + +3 2N
In(4/0)
Remark 3. The results above imply the generalization errors E and E3 can converge at the rate of 1/VN, considering for each row in Xa its 12-norm bounded from above by (mm-1)772 Obtaining the results require adaptations of classicul Rademacher complexity techniques. The dif- ficulty comes from the minimum form of p(x, minns[m]]
TC- (xlf(la) which differs from usual X,(0)
loss funntiins. We provide the detailed derivation in Appendix B.6. We further remark the scale assumption on r, fx(x)) is mild. o zoz. (9) and f forg(O) are supposed approach T, the Pareto Front, under proper optimization procedures; we are thus able select a reference point r far away enough from the set T U x. (9) U ff x (() to make the assumption hold.
6 Experiments
Testing problems. This section demonstrates that our method can generate high-quality contin- uous Pareto solutions by the Pareto neural model for multiobjective synthetic, design, and control
7

---

problems. We evaluated our approach's performance against established methods using well-known benchmark problems such as ZDT1-2 (m=2) [42]. VLMOP1-2 (m=2) [38], and real-world design problems like Four Bar Truss Design (RE21, m=2), Hatch Cover Design (RE24, m=2), and Rocket Injector Design (m=3) [43] as well as MO-LQQR (m=2,3) [44]. MO-L.QRs serve as multiobjective reinforcement learning problems. For simplicity, except MO-LQR, all objectives are normalized to [0.1]. Problem details are described in Appendix A.3.
Neural model architecture and feasibility guarantees. We employed 4-layer fully connected neural network, similar to [37], to construct our Pareto neural model (.)()) The network is optimized using Stochastic Gradient Descent (SGD) with a batch size of 256 4. For constrained problems, to conform to the lower bound (T) and upper bound (u) constraints, sigmoid activation function mapped the previous layer's output to these boundaries. For unconstrained problems, the output solution was derived from linear combination of Tmid. Further architecture details are available in Appendix A.2.
Baseline methods. Our approaches are compared to existing hypervolume-basee multiobjective evolutionary algorithms (MOEAs) SMS-EMOA [14] and various Pareto set learning (PSL) methods. Before our approach was developed, three PSL approaches were developed, namely, EPO-based and Tchebycheff-beee Baseline methods are listed as follows,
1. One hypervolume-based multiobjective evolutionary algorithm: SMS-MOEA [14], which uses evolutionary algorithms to maximize the hypervolume of a finite set of solutions.
2. The LS-based PSL [12]. optimizing min, Ex- Am) |= (z (0))) where A is the
(m - -imsmmpleex Since linear scalarization can only find the convex part the Pareto front [39], theoretically, this method cannot recover the full Pareto front.
3. The EPO-based PSL [12]. optimizing min, E---Unif Am m-
( [f (0)), the expectation of EPO loss [3]. The EPO-based PSL is slow since each solving for each EPO subproblem, which
involves expensive optimization problem.
4. The Tehebycheff-basee based PSL [23, 37], optimizing min; Ba [aax][[m] {.f((zz(0))-
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(e) LS
(1 SMS-EMOA
Figure 6: Result comparisons VIMOP2.
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(c) LS
(f) SMSSEMOA
Figure 7: Result comparisons on the Four Bar Truss Design problem.
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(e) LS
(0 SMS- EMS-EMOA
Figure Result comparisons the Rocket Injector Design prohlem.
(Metrics.) To evaluate our learned Pareto solutions, we use three indicators. These are: (1) the Hy- pervolume (HV), which measures the covered objective space, where a high value indicates greater
'Batch size is 128 for EPO-based PSL on LQRs since EPO is extremely slow.

---

Table 1: The training/preddiction time of PSL-HV, and running time of SMS-EMOA in Rocket Injector Design.
SMS-EMOA
MOEA/D
Ours (Train)
Ours (Predict)
Time
6.2m
1.5m
40s
<Is
coverage; (2) the Range indicator, which measures the extent of the Pareto front learned by the model, where a larger Range is preferred; and (3) the Spars prsitt metric, which evaluates the de- gree of sparsity in the learned Pareto front, where lower sparsity value indicates a denser Pareto front. See Appendix A. for formal metric definitions.
Key findings from experiments are summarized as follows.
Table 2: PSL results on all problems. For PSL-IIVI, r 0.6 y" For Tche-based PSL. the reference point 1.1 yd as claimed in their original paper.
ZDT1
2DT2
VLMOPI
Method
11V1
Ranget
Spunity
Time(s)_
Runge
Spunsity
Time(s)
Range
Sparsity
Timets)
PSL-EPO
10.82
0.94
0.14
90.7
11.21
1.43
0.86
91.1
11:55
1.18
0.11
90.79
PSL-LS
8.75
PSt--Tahe
11,49
1.37
0.33
21.11
11.19
1.42
0.66
21.45
11.83
1.42
0.22
21.61
PSL-HVI
11.79
1.52
0.72
31.79
11.3
1.46
0.73
36.68
12,07
1.57
0.84
32.24
PSL-HIV2
11.77
1.52
0.7
25.71
11.91
1.55
25.8
12.02
1.54
0.46
25,45
VLMOP2
Four Bar Truss Design
Hatch Cover Design
PSL-EPO
11.37
1.49
0.63
90.95
11.2
1.18
0.21
255.44
11.96
1,16
0.07
649.12
PSL-LS
11.51
1.57
7.22
9.57
11.89
1.56
0.94
41.35
12.14
0.27
113.52
PSL-Tehe
11.5
1.37
1.37
125.43
PSL-HVI
11.57
1.57
0.98
32.52
11.9
1.56
0.81
66.6
12.18
1,53
0.7
131,36
PSL-HIV
11.56
1.56
0.66
25.78
11.82
1.53
0.65
60.00
12.15
1.51
0.37
130.4
MO-LQR2
Rocket Injecter Design
MO-LQRS
PSL-HPO
3,42
0.3
0.23
546.81
37.23
0.68
0.81
133.03
2.41
0.31
2.33
1149,4
PSL-LS
3.59
0.5
0.84
296.98
35.55
0.19
0.88
21.35
2.58
0.37
1.9
408.59
PSL-Tehe
314.75
37.73
0.19
1.34
0.33
424.15
310.01
3.31
14.7
430.72
PSL-HIV2
3.61
0.59
2.19
312.95
40.94
0.7
3.54
36.63
2.6
0.55
8.93
424.51
I (Comparison with SMS-EMOA.) PSL and HV-based evolutionary algorithms serve as different purposes. PSL is designed find infinite of solutions using neural network model, while EMOAS aim to find a finite solution set. This difference becomes evident in three-objective prob- lems, shown in Figure where 100 evenly distributed solutions by traditional MOEAs fall short of approximating the full Pareto set/front. PSL requires only several seconds of training time and can quickly predict highly approximated Pareto solutions for other unknown preferences. In trast, SMS-EMOA typically takes a much longer time, especially for objectives larger than three. The advantage of evolution-based SMS-EMOA is its ability to find global optimal solutions, while current PSL models still struggle with handling local optimas.
Table 2 reports the mean results for various PSL approaches under five random seeds. The standard deviation values can be found in Appendix A.4. All experiments were conducted with 1000 itera- tions for PSL-HV1, PSL-HV2, LS-based PSL, and Tehebycheff-boe beabbychff--bseedd PSL, while EPO-based PSL limited to 100 iterations due to time limitation.
② (Comparison with LS-based PSL.) It is well-recognized that LS-based PSL [12] recovers only convex part of a Pareto front [39], resulting finding a small portion of Pareto solutions for concave problems. For example, in the case of 7DT2,, LS-based PSL could only find two extreme points. We would like to mention another less-discussed limitation of linear scalarization: the non- uniform distribution of Pareto objectives under uniformly distributed preferences. This issue is evident in problems like ZDTI and VLMOP2 where Pareto objectives are densely distributed at the front's 's margins, but fewer objectives are distributed around the center of the front.
3 (Comparison with EPO-based PSL.) As mentioned in Remark 1, both EPO and our proposed method (under mild conditions) can find the exact Pareto objective aligned with the preference vec- tor. However, our proposed approach approximately 40+ times faster than EPO-based PSL. Unlike the EPO-based PSL method, our approach only requires calculating the gradient of one objective function, avoiding the need to compute gradients for all objectives and solve complex optimiza- tion problems as described in their work [3]. This efficiency advantage is particularly evident in

---

MO-LQR problems, where EPO-based PSL takes significantly more time to execute. Furthermore, we have observed that EPO-based PSL does not perform well on tasks ZDTI and Rocket Injector Design.
4 (Balancing range and sparsity of a PE.) When simply running PSL., observed a tendency for the Pareto objectives to concentrate on the central portion of the Pareto front, neglecting solutions that correspond to boundary preferences, such as f [0. 1]/[1,]]]]]]__]1 [1, 0] for bi-objective problems. To address this observation, a simple technique is to assign higher weights to the boundary objectives. For PSL-HVI, a larger reference point suggested be used in comparison to the nadir point, and therefore more coordinates will correspond to boundary Pareto objectives (refer to Remark 4 in Appendix B.4). However, setting the reference point be excessively large can result learned Pareto front; we empirically set this offset to be 0.6 (the empirical study is in Figure 10 and 11). In PSI.-HV2, the preference is placed in the denominator, resulting in greater weighting for the boundary Pareto objectives when the preference value approaches zero.
sparse
PSL-HV2 is related to the modified Tchebycheff (mTche) scalarization function [45], but the latter has different sampling techniques and lacks a hypervolume interpretation. When À approaches [0.1] or [1,0], both mTcce and PSL-HV2 faces numerical challenges resulting in a large gradient, which can be effectively addressed the gradient clipping technique [46, 47] with a clipping norm of a certain value (e.g.. 4.0 is used). Figure shows that the learning process is unstable without gradient clipping but is stabilized when gradient clipping is applied. In contrast, PSL-HV1 involves minimization rather than maximization its inner optimization problem and does not suffer from gradient explosion.
7 Conclusion, Limitation, and Future Work
This paper presented a method to learn the set of all Pareto solutions through hypervolume maximization. The method was motivated by previous lemma, which allows for hypervol- ume estimation of infinite solutions. We further gave a precise geometric perspective Pareto set learning; we studied the generalization gap between estimated and true hypervolumes, and discussed the key techniques to train a Pareto neural model that outperformed previous meth- ods. The proposed method will also be incor- porated into the Moon library [48].
(a) With gradient clipping. (b) W.o. gradient clipping.
Figure Learning curves PSL-HV2 on VLMOP2.
Limitation and future works. The acknowledged constraint of our work is the reliance on gradient-based methods, which can result locally optimal solutions when the objectives fa are non-convex. To overcome this issue and provide more solid statistical guarantee, one feasible direc- tion is explore classical nonparametric techniques that can improve the robustness of the method. Another way is to use some evolutionary methods such as [49, 50, 51, 52] to skip local optimal solutions. Additionally, we plan to apply the more robust Pareto neural models to large-scale multi- objective problems, such molecular design and deep reinforcement learning.
Acknowledgement
This paper is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China [GRF Project No: CityU 11215622 and by Natural Science Foundation of China [Project No: 62276223]. In the last, we sincerely appreciate the helpful insights and feedbacks provided by Liang Dake Bu; Yingying Yu (CityU Hong Kong) and Dr. Jingda Deng (XITU).
10

---

References
[1] Kaisa Miettinen. Nonlinear Noniinar multiobjecttive optimization, volume 12. Springer Science & Busi- ness Media, 2012.
[2] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biand niaa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Con- ference Neural Information P Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 525-536, 2018.
[3] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages
PMLR, 2020.
[4] Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep multi- objective learning. In 2021 IEEE International Conference Data Mining (ICDM), pages 1311. IEEE, 2021.
[5] Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decom- position. IEEE Transactions on evolutionary computation, 11(6):712-731, 2007.
[6] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions evolutionary computation, 6(2):182-197, 2002.
[7] Xingchao Liu, Xin Tong, and Qiang Liu. Profiling pareto front with multi-objective stein varia- tional gradient descent. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 14721-14733 2021.
[8] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buce Emily B. Fox, and Roman Garnett, editors, Advances Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12037-12047, 2019.
[9] Liang Zhao and Qingfu Zhang. Hypervolume- guided decomposition for parallel expensive multiobjective optimization. IEEE Transactions on Evolutionary Computation, 2023.
[10] Timothy Marler and Jasbir S Arora. Survey of multi-objective optimization methods for engineering. Structural and multidisciplinary optimization, 26:369-395, 2004.
[11] William Karush. Minima of functions of several variables with inequalities side constraints.
M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
[12] Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[13] Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learn- ing. ArXiv preprint, abs/2010.063 13, 2020.
[14] Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 181(3))1653- 1669, 2007.
[15] Eckart Zitzler, Dimo Brockhoff, and Lothar Thiele. The hypervolume indicator revisited: On the design of pareto-compliant indicators via weighted integration. In Evolutionary Multi- Criterion Optimization: 4th International Conference, EMO 2007, Matsushima, Japan, March 5-8, 2007. Proceedings 4, pages 862-876. Springer, 2007.

---

[16] Michael Emmerich, Nicola Beume, and Boris Naujoks. An emo algorithm using the hyper- volume measure as selection criterion. In Evolutionary Multi Criterion Optimization: Third International Conference, EMO 2005, Guanajuato, Mexico, March 9-11, 2005. Proceedings
3, pages 62-76. Springer, 2005.
[17] Christian Igel, Nikolaus Hansen, and Stefan Roth. Covariance matrix adaptation for multi- objective optimization. Evolutionary computation, 15(1):1-28, 2007.
[18] Timo M Deist, Monika Grewal, Frank JWM Dankers, Tanja Alderliesten, and Peter AN Bosman. Multi-objective learning using hv maximization. In Evolutionary Multi-Criterion Optimization: 12th International Conference, EMO 2023, Leiden, The Netherlands, March 20-24, 2023, Proceedings, pages 103-117. Springer, 2023.
[19] Michael Emmerich and André Deutz. Time complexity and zeros of the hypervolume indi- cator gradient field. In EVOLVE-a bridge between probability, set oriented numeries, and evolutionary computation III, pages 193.193. Springer, 2014.
[20] Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. ArXiv preprint, abs/2210.12765, 2022.
[21] Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy- Chowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10955-10964, 2022.
[22] Alan Q Wang, Adrian V Dalca, and Mert R Sabuneu. Computing multiple image reconstruc- tions with a single hypernetwork. ArXiv preprint, abs/2202.11009 2022.
[23] Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective com- binatorial optimization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[24] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc,, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14610-14621, 2019.
[25] Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dy- namic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceed- ings of Machine Learning Research, pages 11-20. PMLR, 2019.
[26] Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference-driven multi- objective reinforcement learning algorithm. ArXiv preprint, abs/2208.077 00080007944.. 14, 2022.
[27] Joey Hong, Aviral Kumar, and Sergey Levine, Confidence-condttionee value functions for offline reinforcement learning. ArXiv preprint, abs/2212.04607,, 2022.
[28] Phillip Swazinna, Steffen Udluft, and Thomas Runkler. User-interactive offline reinforcement learning, ArXiv preprint, abs/2205.10629 2022.
[29] Andreia P Guerreiro, Carlos M Fonseca, and Luis Paquete. The hypervolume indicator: Prob- lems and algorithms. ArXiv preprint, abs/2005 005515, 2020.
[30] Ke Shang, Hísao Ishibuchi, Min-Ling Zhang, and Yiping Liu. A new 12 indicator for better hypervolume approximation. Proceedings of the Genetic and Evolutionary Computation Conference, pages 745-752, 2018.
12

---

[31] Jingda Deng and Qingfu Zhang. Approximating hypervolume and hypervolume contributions using polar coordinate. IEEE Transactions on Evolutionary Computation, 23(5):913-918, 2019.
[32] Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi- objective black box optimization. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1196-11 10--11005 105. PMLR, 2020.
[33] Carlos M Fonseca and Peter Fleming- On the performance assessment and comparison of stochastic multiobjective optimizers. In International conference on parallel problem solving from nature, pages 384-593. Springer, 1996.
[34] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
[35] Hisao Ishibuchi, Ryo Imada, Naoki Masuyama, and Yusuke Nojima. Use of two reference points in hypervolume- based evolutionary multiohjective optimization algorithms. In Parallel Problem Solving from Nature-PPSN XV: 15th International Conference, Coimbra, Portugal, September 8- 2018, Proceedings, Part 15, pages 384-396. Springer, 2018.
[36] Hisao Ishibuchi, Ryo Imada, Yu Setoguchi, and Yusuke Nojima. How to specify reference point in hypervolume calculation for fair performance comparison. Evolutionary computation, 26(3):411-440, 2018.
[37] Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. arXiv e-prints, pages arXiv-2210, 2022.
[38] David A Van Veldhuizen and Gary B Lamont. Multiobjective evolutionary algorithm test suites. In Proceedings of the 1999 ACM symposium on Applied computing, pages 351-357, 1999.
[39] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[40] Harvey J Greenberg and William P Pierskalla. A review of quasi-convex functions. Operations research, 19(7):1553-1577) 1971.
[41] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6231-6239, 2017.
[42] Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary compulation, 802) 2000.
[43] Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89:106078, 2020.
[44] Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing, 263:3-14, 2017.
[45] Xiaoliang Ma, Qingfu Zhang, Guangdong Tian, Junshan Yang, and Zexuan Zhu. On tcheby- cheff decomposition approaches for multiobjective evolutionary optimization. IEEE Transac- lions on Evolutionary Computation, 22(2):226-244 2017.
[46] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
13

---

[47] Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding gradient clipping in pri- vate SGD: geometric perspective. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
[48] Xiaoyuan Zhang, Ji Cheng, Liang Zhao, Song Lai, Cheng Gong, Weiduo Liao, Longcan Chen, Xi Lin, Ke Shang, Zhichao Lu, Yifan Chen, Tao Qin, and Qingfu Zhang. Moon: A standard- ized/flexible framework for multiobjective optimization. 01 2024.
[49] Hongzong Li and Jun Wang. From soft clustering hard clustering: A collaborative annealing fuzzy e-means algorithm. JEEE Transactions on Fuzzy Systems, 2023.
[50] Hongzong Li and Jun Wang. Capacitated clustering via majorizatoon-miniizztooo minimization and col- laborative neurodynamic optimization. IEEE Transactions on Neural Networks and Learning Systems, 2022. in press.
[51] Charu Aggarwal and Karthik Subbian. Evolutionary network analysis: A survey, ACM Com- puting Surveys (CSUR), 47(1))1-36, 2014.
[52] Xin Yao and Yong Liu. A new evolutionary system for evolving artificial neural networks. IEEE transactions neural networks, 8(3):694-713, 1997.
[53] Jie Xu, Yunsheng Tian, Pingehuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10607- 10616. PMLR, 2020.
[54] Kaisa Miettinen. Nonlinear ness Media, 1999.
multiohjective monlinear
optimization, volume 12. Springer Science Busi-
[55] Matus Telgarsky. Deep learning theory lecture notes, 2020.
[56] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297-299. PMLR, 2018
[57] Andreas Maurer. A vector-contraction inequality for rademacher complexities, In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3-17. Springer, 2016.
14

---

A Experiment Details
A.1 Metries
We assess the performance of Pareto neural model, denoted zz)(), by examining its output with the input of é. Here, θ represents set of N uniformly discrete angles. The resulting output solution set is denoted as A and can be expressed as A = " ,(N), where = f zg( The three metrics are:
1. The Hypervolume indicator [29], which measures both the diversity and convergence of A;
2. The Range indicator, which measure the angular span of A;
3. The Sparsity indicator [53], which measures the average distances between neighbor- hood points.
(The Hypervolume Indicator.) The hypervolume indicator [29] measuring A is standard, which has been defined in the main paper,
(A) A(fa 3p A and r)),
where r is reference vector, 2 y For bi-objective problems, the reference point is set to [3.5, 3.5], whereas for three-objective problems, the reference point is set to [3.5, 3.5,
(The Range Indicator.) The range indicator of of solutions A { ..... y (A)
measures the angular span of A in their polar coordinates. Let (p()), ((0) be the polar coordinate of objective vectors (i) with reference point r. The relationship between the Cartesian and polar coordinates is,
y1 T1 - p sin 01 sin 0g... sin 6m- y22 = T2 - sin 0₁ sin Oz... cos 0m-1
(10)
3m Tm cos
The Range indicator can be defined as the minimal angle span among all angles.
Range(A) = min max selm 1| ISuSEEN
(11)
(The Sparsity Indicator.) The sparsity indicator first introduced in [53]. A small sparsity indicator preferred, implying that the learned objective set is dense.
N
0f )1
Sparsity(A) = N
(12)
where vf6) is the i-th solution, and the j-th objective values in the sorted list by the non-dominating sorting algorithm [6]. For better illustration, the unit of the Sparsity indicator is 10-3 for bi-objective problems and 10-7 7 for three objective problems.
A.2 Neural Model Architecture and Feasibility Guarantees
We use 4-layer fully connected neural network similar to [37] for the Pareto neural model ma(-).
We optimize the network using Stochastic Gradient Descent (SGD) optimizer with a batch size of 256. The first three layers are,
x (.): : 0 Lincar(m m - 14) + ReLU
Linear(644 64) + ReLU
(13)
Linear 64) + ReLU - 2mid
15

---

For constrained problems, to satisfy the constraint that the solution ( must fall within the lower bound (1) and upper bound (u), sigmoid activation function used to map the previous layer's output to these boundaries,
miid Linear(64,,) n) - Sigmoid O - 1) + 1 Output (A).
(14)
For unconstrained problems, the output solution is obtained through linear combination of Tmid-
mid
Linear(64, m) Output (A))
(15)
A.3 Benchmark Multiobjective Problems
Standard Multiobjective Optimization (MOO) problems. ZDT1-2 [42] and VLMOP1-2 [38] are widely recognized as standard multi-objective optimization (MOO) problems and are commonly employed in mudieent-based MOO methods. ZDTI exhibits convex Pareto front described by
(//2 v(n, S y1 S 1). ZDT2 presents non-convex Pareto front defined by (y/2
1- vi, S 91 S 1), and the LS-based PSL approach can only capture a small number of Pareto solutions.
Real world designing problem. Three real-world design problems with multi-objective optimiza- tion are the Four Bar Truss Design (RE21), Hatch Cover Design (RE24), and Rocket Injector Design (RE37). In order to simplify the optimization process, the objectives have been scaled to a range of zero to one.
Multiobjective Linear Quadratic Regulator: The Multiobjective Linear Quadratic Regulator (MO-L-QR) problem is first introduced in [44]. MO-LQR is regarded specialized form of multi-objective reinforcement learning, where the problem is defined by a set of dynamics presented through the following equations:
88-1 Ast + Ba, 04 ~ ~N(KKQR St, E).
(16)
In accordance with the settings discussed in the aforementioned work by Parisi et al. [44], A, B, and Σ are adopted identity matrices. The initial state for the bi-objective problem is set to so [10, 10], whereas for the three-objective problem, it is set to so = [10, 10, 10j. The reward function defined rr(st, az), where i represents the respective objective. The function is formulated as follows:
ra(s;, a.) = -(1 ( + Σ (
Eld,, +
(17)
Here, is the hyperparameter value that has been set to 0.1. The ultimate objective of the MO- LQQ problem
optimize the total reward while simultaneously taking into account the discount factor of 0.9. The control matrix KLOR is assumed to be diagonal matrix, and the diagonal elements of this matrix are treated as decision variables. The objectives are scaled with 0.01 for better illustration purposes.
Table 3 shows the number of decision variables and objectives used in this paper.
A.4 Results on All Problems
Results for all the examined problems are depicted in Figures 12-20, and combined with the results tabulated in Table 2 of the main paper, several conclusions can be made.
Behavior of LS-based PSL. The linear scalarization function can only learn the convex part of Pareto front [39]. This fact validated by Figure 13(e), where L-S-based PSL can only learn several discrete solutions.
16

---

Table Problem information for multiobjective synthetic, design, and LQR problems.
Problem
m
ZDTI
2
5
ZDT2
2
VLMOP1
2
5
VLMOP2
2
LQR2
2
2
Four Bar Truss Design
2
Hatch Cover Design
2
2
Rocket Injector Design
3
LQR3
3
3
(a) r y mdir
+ 0.1 (b) r
mader
0.25 (c) r m y madir
+ 0.4 (d) r y padrr
+ 0.55 (c) r = -V addr
0.70
Figure 10: Hyperparameter study ZDTI.
However, it is crucial to note that the connection between solution and its corresponding preference vector, X(0), non-uniform. This observation was rarely discussed in previous literature. Therefore, uniform sampling of preferences will not result in uniform sampling of Pareto objectives. This observation is supported by the results depicted in Figures 12(e), 15(e), and 17(e), where the learned Pareto objectives by LS-based PSL are not uniformly distributed. And result, the sparsity indicators are rather high, which indicates the learned front sparse some regions.
Time Consumption of EPO-based PSL. In comparison our approach, the Exact Pareto Op- timization [3] algorithm, which serves as the foundation for EPO-based PSL [12], exhibits low efficiency due two factors.
1. To execute the Exact Pareto Optimization (EPO) algorithm, it is necessary to compute the gradients of all objectives, (f('''ss. This prerequisite entails performing m back- propagations in pytorch, resulting in higher computational costs. In contrast, approach relies just one propagaoaan operation, rendering it a more efficient option in compari- son EPO.
2. For each iteration, the Exact Pareto Optimization (EPO) algorithm entails solving plicated optimization problem based on the specific value of fi 's, utilizing the respective gradients of fi()''s. In contrast, method does rely solving optimization prob- lems for each iteration.
Trade-off' between Range and Sparsity. We find that the reference point plays a crucial role in the performance of the final leamned Pareto front for PSL-HVI. To demonstrate this effect, consider different values of from the set [0.1, 0.25, 0.4, 0.55, 0.7] in addition to the nadir point of each problem. The results are shown in the following two problems on the ZDTI and ZDT2 problems.
Based on our empirical findings, emphasizing boundary Pareto objectives is crucial for recovering a complete Pareto set. From Figure 10 and 11, we observe that treating all coordinates 0 equally (small nadir point) typically results in partial recovery of the Pareto set. When exceeds a certain threshold (e.g., 0.4), raising its value directs Pareto objectives towards the boundary of the Pareto front, leading to poor uniformity of Pareto objective distribution.
17

---

(a)
0.25 (c) r y adir
0.4 (d)
+ +..5 (c)
y nadrr
0.70
Figure 11: Hyperparameter study on ZDT2.
( HVI
(b) HV2
(c) Tche
(d) EPO
(e) LS
Figure 12: ZDTI.
A.5
Pseudocode
For completeness, we provide the pseudocode of the proposed method. PSL-HVI is selected as an example. 1) positive learning rate.
Algorithm 1: PSL-HVI.
Input: The multiobjective function f(-) and initial Pareto neural model parameter 3. for I= Nuer do
Sample N preference angles (02), .-... ((N) from the positive unit sphere Sm-1
Calculate the preference , ((N) by Equation (26).
Calculate the empirical PSL gradient V V by Equation (28) with N sampled preferences.
Update by: +- sH,
end
Output: The trained Pareto neural model r-)())
B Characters of Hypervolume Maximization
B.1
The Notation Table
To enhance the clarity of the paper, we have included summary of the main notations in Table 4.
B.2 Hypervolume Calculation in the Polar Coordinate
The idea of calculating the hypervolume is similar to the Lemma 5 proved in Appendix of [32]. The difference is that, we use distance function px(0 (0) as defined in Equation (6). For completeness, we provide the proof for Equation (5) in this section.
(a) HVI
(b) HV2
(c) Tche
(d)
(e) LS
Figure 13: ZDT2.
18

---

(a) HVI
(b) HV2
(c) Tchc
(d) EPO
(c) LS
Figure 14: VLMOPI.
h.
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(6) LS
Figure 15: VLMOP2.
(a) HVI
HV2
(c) Tche
(d) EPO
(c) LS
Figure 16: RE21.
%
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(e) I.S
Figure RE24.
(a) HVI
(b) HV2
(c) Tche
(d) EPO
(c) LS
Figure 18: MO-LQR2.
(a) HVI
(b) HV2
(c) Tche
(d) EPO
(e) L.
Figure MO-LQR3.
19

---

(a) HV1
(b) HV2
(c) Tche
(d) EPO
(c) LS
Figure 20: RE37.
Table 4: The notation table.
Variable
Definition
2
The decision variable.
X/52
The decision/objective space.
n
The number of the decision variables.
N
The number of samples.
The number of objectives.
The angular polar coordinate.
A(0),
m--dimensional preference vector.
3
The model parameter.
y nadii Jydda
The nadir/ideal point of given MOO problem.
T
The Pareto front.
(A)
The hypervolume set A w.r.t a reference.
ST 8m
The (m-1)-D positive unit sphere.
Proof. HT) can be simplified by the following equations. Here St denoted the dominated regions by the Pareto front, i.e. (T) A(22
fa
f
Figure 21: The hypervolume equals to the integral of dv.
20

---

H,(T)
Indy1 ... dym
1mm
("dv" denoted the infinitesimal sector area. )
du
m-
(dv equals the angle ratio multiplied by p" multiplied by the unit for volume. )
px)
Cm
2π m=
d0: ... d0m-
(18)
(Re-arranging the terms.)
Cy
(x)
2 m-
m-
Cm
Eo(px(())'
2m
Bo (0)
We specify ~ Unif(e) (n, -)m=! in Equation (18).
Line 2 holds since it represents the integral of J expressed in polar coordinates, wherein the element de corresponds to the volume associated with a segment obtained by varying d0.
Line 3 calculates the infinitesimal volume of de by noticing the fact that the ratio of du to Cm is (0) Line is simplification of Line And Line 5 and 6 express the integral in its expectation 20-mm-2 form.
B.3 MOO Theory
In this section, we give the definitions and give proofs of the propositions used the main paper.
We first formally define the term of aggregation function.
Definition (Aggregation function (modified from the Definition 2.6.1 in [54])). A function
g(,2) Rm R representing the preferences of the decision maker among the objective called an aggregation function. g(,,)) 1 IR decreasing, indicating that g(y A) () ,)) when yla) (b)
This definition is slightly different from the original definition in Definition 2.6.1 in [54]), as modify the condition from "strictly decreasing" to "decreasing". The considered aggregation func- tion p(x. ()) does not satisfy the "strictly decreasing" condition in the original definition. When an aggregation function is only decreasing, the optimal objective can be weakly Pareto optimal. For this modified version of aggregation function, we have the following Lemma modified from Theorem 2.6.2 in [54].
Lemma 2. Let y belongs to the optimal solution set of g(, A), then y is weakly Pareto optimal or Pareto optimal. Further, yf be the only optimal solution of g(,, A), then y is Pareto optimal.
We also provide the upper bound of the distance function (x) in this section by the following lemma.
Lemma (The bound of in px Equation (6)).
(x < Bm B/2
(19)
when b - f((z) < B, Vz e R.ViE [m] and [|(()]]
21

---

Proof. We show that the following inequalities hold,
Pi - Sí(Ø)
} (x(
Px(0)= -
max
min [E|m)
= =
(r: f(( S B)
)) (:(0)
B
(20)
< max
min
|A(0)|I
(E(m)
B
1/2
m 1/2
The transition from line one to line two is due to the fact that the inequality тa fi(z) Σ B holds for all x X' and for all i E [m]. The transition from line two to line three is m [m)) [mm] (A.(0) }) is an optimization problem under the constraint ||\(0]| = 1.
The upper bound for this optimization is when A; ... Am -m-2/_
B.4 (x) as a Max-Min Problem
In this section, we first present the proof of Equation (6) from the main paper. To begin, we de- fine the attainment surface AT [33] to handle disjointed PFs. Intuitively, the attainment surface continuous, containing the weakly Pareto objectives and Pareto objectives (see Figure 22).
Definition (Attainment surface [33]). The altainment surface AT induced by the Pareto front (PF)
is defined as the boundary set of the PF.
A7 (y E Ry E T, 3. R))
(21)
With this definition, the distance p. (0) between the reference point r and the point coordinated at be calculated using the following equation:
r f((x)
px (0) max p(x, 0) max min =EX
#EA .E(m)
d
where A; are shorthand's for (0).
Proof. We first define the point P.
Definition 4, P is defined as the intersection point of the ray from r along angle 9 and the attainment surface.
There are two cases to consider:
I P is Pareto objective, i.e., P E T. 2T.2P. 2 belongs to the attainment surface but not to the Pareto front, i.e., P (AT TT).
Let x" be one of the optimal solutions of Problem max,ex p(x, 0). Next, we will separately demon- strate the validity of the above-mentioned statements.
! When P is Pareto objective, the according to Proposition 2, we have that = P f(x)) is the only objective of the problem ō(y, A) maxyey mingejm] X }. And therefore, px px
r
n f((')) Vi E [m]]
② When y P (AT \ T). Then, according Proposition y is one of the optimal ob- A: f. In such case, according to the defi-
jective of problem ő(y, A) maxyey min:e(m] re-ws
nition attainment surface (Equation (21)), we have that there exist at least one index j. where
j arg min 13- -=²)(=") such that px maxxex p(x, max minge|m) X
(r- ((())
2,
f(1=") (x') Ti Pa. i [m].
min;e|m]]
A
22

---

1.0
f2 -2
p(0)
0.6
f
Solution
Attainment surface
0,0
0.2
0.0
0.4
0.8
1.2
f (x)
Figure 22: Illustration of the attainment surface case of disjointed Pareto front.
The previous discussion explained that, in the of disjointed Pareto front, the distance be- tween and the attainment surface corresponding to angle 0 is denoted by (x) In Figure 22, the black dot represents the solution for this scenario. is worth noting that the integral of the distance function px(0) yields the hypervolume of a disjointed Pareto front, i.e., H H, (A7). This property aligns with the purpose of paper.
In closing this section, we note that the absence of a (weakly) Pareto objective aligned with the preference angle O.
Remark 4. When P (in Definition 4) is not (weakly) Pareto objective, The optimal solution
w.rl the problem g X) = maxyey minse[m]
Ti-Ms Ms
belongs to the boundary of the Pareto front
OT. Further y" = arg min [m]] V B Pt
B.5 Pareto Front Hypervolume Calculation (Type2 and gradients)
In this section, we provide another form to cal- culate the Pareto front hypervolume. We first define region A as the set of points dominating the Pareto front.
A= {q 3p E T p I q and g 2 idcal) 1
Hypervolume
(22)
We use the notation A(-) to represent the Lebesgue measure of set. Geometrically, as illustrated Figure 23, can be observed that:
p())
A
0.4
02
+ +
II
(r. di
ideal ).
(23)
0.0
Ideal point
0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
The volume of can calculated a polar coordinate as follows,
(x)
Figure 23: The hypervolume calculation (Type2).
A(A) = Cm /
(0, (0) d0,
(24)
where Cy is a constant and 7x (f) represents the distance from the ideal point to the Pareto front angle This distance function p obtained by solving the optimization problem assuming that any ray from 0 intersects with the Pareto front.
Problem 1.
f((x)= y: jiheal
K m-1 (0,
(x = min (0, min max #EX
E
(25)
ZEX (E(m]
A:(0)
2
23

---

The relationship between preference À and the polar angle 0 is as follows:
A (0) sin 01 sin f2... sin m
A2(0) sin 0 sin 02..... cos 0m-
(26)
Am()) cos 81.
Combining Equation (24) and Equation (25) implies that (3) can be estimated an expectation problem,
1L, (r, Vi jdeal) C. (oneolpx(zaa())00)
(27)
Remark 5. Compured to PSL-HVI, PSL-HVV hus weaker theoretical results due to its reliance on a stronger assumption. This assumption requires the presence of a (weakly) Pareto objective aligned the ray originating from any coordinate 0. An additional challenge with PSL-HV2 that the Lipschitz constant cannot be bounded, making generalization theory for PSL-HV2 significantly more difficult (Equation (30)).
We conclude this section by presenting the analytical expression for V sr, The gradient for PSL-HV1 can be computed using the chain rule, which yields:
mc mCm (olmEEo m m-1 @p(x;((0,,0) 0)
p(z3 (0), 0)
2
0x0(0)
0
V V9H
1xn
axd
(28)
@p(zz(0),0) 0)
(z)
Cm
Otherwise.
(0)
0,
1xn
axd
where we specify satisfy the law of Unif(O). The gradient of PSL-HV2 can be calculated by,
Opx (2))))((000)) a
VJH, = - -m (m
(29)
0
1xn
nxd
B.6 Proof of Theorem
Definitions and preliminaries. The proof will heavily utilize the existing results on Rademacher complexity of MLPs. We will first provide some useful definitions and facts. We start with the definition of Rademacher complexity follows:
Definition (Rademacher complexity, Definition 13.1 in [55]). Given set of vectors C R" define the (unnormalized) Rademacher complexity as
URad(V) == E sup(e, weV
where each coordinate e; is an i.i.d. Rademacher random variable, meaning Pr(e: +1] = Pr [e; -1]. Furthermore, we can accordingly discuss the behavior of a function class on S N { i= by using the following set:
9 =
and its Rademacher complexity is
URad (9|s) E sup (e, = Esup
Σ e:g (z+)
e uedis
gEG
Utilizing Rademacher complexity, we can conveniently bound the generalization error via the fol- lowing theorem:
24

---

Theorem 2 (Uniform Generalization Error, Theorem 13.1 and Corollary 13.1 in [55]). Let G be given with g(2) b) a.s. Vgg g. We collect i..dd samples {= from the law of random variable Z. With probability 2 1
2 N N E 9 (zi) (a)) sup (2(2) SEG
In(2/6)
URad (9|s) + +3b w)
2N
Specifically, the Rademacher complexity in using MLP is provided by the following theorem:
Theorem 3 (Rademacher complexity of MLP, Theorem I in [56]). Let 1-Lipschitz positive homo- geneous activation a; be given, and
gMLP = 10 + on (WLoL-1 (W10) ::)): | s B
Then
URad
(9
B X1 + (2)).
We can then utilize the following composition character of Rademacher complexity, to help induce the final Rademacher complexity of hypervolume.
Lemma 4 (Rademacher complexity of compositional function class, adapted from Lemma 13.3 in [55]). Let θ + R² be vector of n multivariate functions g¹ (())g(2), .g.".....g." G denote the function class of g. and further g()) be the function class of g(a), Vj. We have "partially Lipschitz continuous" function ((g(0), 0) that |E(g) (0), (0), 0}l I Lellgn (0) 92(0)] for all
91. 92 E G and a certain Le > O; the associated associatedfuncciio function class of f is denoted as G We then have
URad (gfs) S 22
URad (0())
Proof. This proof extends Lemma 13.3 in [55] for vector-valued y and "partially Lipschitz continu- ous" l. We first similarly have
URad (gss)
Esu gEg
Σ (g((g((,))0))
N
= E sup ezNN fike
((f(Oi))0)) - -((x(1))) 01) + Σ ())( + E(N(O)))
< E
sup
TellS((1) m(OI) + Σ
(E() 0.) E(N(O:),
2 E2.N f.eE
1=2
< E sup
L Σ
(P((0)())) (
2 y0) AUEEGG)
1=1
E (C((()))))) +(((0)))00++((00000))))
2=
w (f) ' are new i.i.d. Rademacher variables; the last inequality comes from Proposition in [57] (see Equations (5)-(10) in [57] for more details). We can then get rid of the absolute value by
25

---

2 N considering swapping f and h,
V2L
Σ
-
Σ
G (((T)))))))++((())))..). +(((())))))))
sup (UGD
V2L Σ c -((0()))))))))))))))))))))))))))
Σ (e(f(O.)) 0.) (200)))))))000000))
max
sup (ogge)))
1=1
V2L Σ
1 (hbi)
(0)) c (((((()))0)++
sup
j=1
V2Le
Σ
(0))))
4 (0TT00))00+++0000000)))
sup for ggg
We can thus upper bounded URad (g by
N
1
V2L
Σ
1 (B)) (0)) + Σ (((((()))0)++(M0))...)
sup yUR,WieG
2
j=1
2=2
N O ()))((1))) + (ee(((0)))))
=E sup EGG)
V2L j=1
1=2
Repeating this procedure for the other coordinates, can further have
N
URad (gis) S 22LE sup gune
Σ
S /22 ΣB sup goو EgUD
Σ 10 (C)
j=l
which leads to our claim in the lemma.
Proof of Theorem 1. We are now geared up for the complete proof.
Proof. We first introduce the sketch of the proof. We mainly utilize Theorem 2 to attain the claimed results in Theorem 1. Specifically, we set the random sample set = {0,) the function class G as {0 - Cm Cm (m)( 0) (the assumption r, - f((x) [b, indicates that p(x, 0) = min (=-A(w) } 2 b 2 0 and by the definition Equation (7), po(0) is thus al- (,(
ways ((=( 0):; z (.) is L-layer MLP to be specified later). Applying Theorem 2, we can obtain that with probability least 1 1 8.
1 g9(0)
In(4/5) 2N
N (0.) sup Eog(O) 9E9
URad (G|s) + 30m 30.3mm (B/m)"
where the definition of URad and Gis can be found Definition 5. Simply replacing g with {-g: : G}. we can have the inequality of the other direction with probability at least
G
sup Efg( 8E-9
1
Σ
(
2 N
URad (-G1s) 30m (Bmmmm)
In In(4/5)
N
2N
+ sup E.g (
1
Σ
9 (0,)
<
2
URad (-91s) + 3mmm (B/m)m
In(4/5)
N
N
2N
nEG
In(4/5)
sup g9g
Σ (0,) Eog(O)
<
URad (G|s) + 3cmm (Bym)
N
N
2.N
26

---

where we apply the property URad (-9 URad (9ss). We thus, with probability at least - 6 (as result of union bound), can upper bound supyes (Bg(() t Ei (0.) by
Σ 9 (0.) Eng(0) }
max
sup (og(o) gEg
N Σ 9 (0,)
sup
gEg N
2 URad (9|s) + 3cm (B m m)m
In (4/0)
2N
For the next step, we will upper bound URad (G|s) by analyzing the structure of cm (mp((r(0),))) 0)TH
and utilizing the existing bound (see Theorem 3) for Rademacher complexity of MLP xzg.
The main idea of controlling URad (9is) is to obtain the "partially Lipschitz continuity" that (p(xz(0); 0) p( (0), 0) S 11 (0) (0)l for a certain T, > O; with the "partially Lip- schitz continuity" we can apply Lemma 4 and obtain the desired bound. For simplicity, we denote Io(0), Zg. (f) respectively as x, x', and use Xj 's as shorthand for (0)")s. ss We now expand the dif- ference (0(x)(0), 0) p( (0), 0)1 as:
rj fz(xt)
2k fx(('')
min p6mm]
min kE/m]
A3
A
Ti
Tk
fix('')
Tk
fa(('))
T3 - (1)
-max
min jE(m)
min kc(m)
min RE [m]
min jc(m]
Xj
Ax
A
)
we respectively denote the minima index of the two finite-term minimization as and k*, we can then upper bound pl (0), 0) p( (0), 0) by
rk- fK. (x)) rk. fk. (x') Ty. (x.
ry. -5-
max
Ak- Skt (x') Su-(()
Ap-
J5- (())-(5-(4())
15(x) Sz((e')
max
max jeli",k*)
Ak.
Az-
Ag
Teg 1 xel
max )
Xj
We note there is a special property for Xj when j is the minima index: as ||||| = 1, there must be a
certain X, 1/mm, and since rj fx( B, have
T-- 1-- (xx) s
B
b
= >g.
(30)
Ay-
Ay-
1//m
v mmB
With this special property, we obtain
((cs(0),))-p(zza))))) (p(x)(0),0) - p()))(())0))))) s vmB Lg x x'1.
We further have
(cmp(e)(0),00 cmp(x (m)(I:((0))0)) m)
=cm ((())((())))))) 8) (0), Σ m) m- (00
mmB Ly |r - 1'1 (( m Scm
m b (Bvm)" L, x'1,
which establishes the "partially Lipschitz continuity". by Lemma 3. We can then apply Lemma and have
The distance function ((x,,(0),, 0) is bounded
URad (g|)) S 2cm m
(Bym)" Lyn URad (oMLL)
s v2 2cm
m
(Bvm)" Lyn Biolleollee1 + In()).
b
27

---

Combining the pieces above, we finally have
sup NEG
Eog(O)
1 I N
Σ (0.)
In(4/6)
URad (9|s) + 3cm (B (Bym)" mM
N
2N
[222mm Nb L
In(4/0)
Sm m)
B Xolla(1 + 22L + +3/
2N
which is the generalization error bound we claim.
28


Reviews:
[{"review_id": "adVOeO4iWZ", "reviewer": "Reviewer_FLs5", "summary": "This paper presents a novel approach to multiobjective algorithms that allows for the direct modeling of the entire Pareto set. The authors present a novel approach to Pareto set learning (PSL) from a geometric perspective, distinguishing it from existing methods that treat all preferences equally, resulting in a mere partial Pareto front. The contributions of this paper are as follows:\n* Introducing a connection between preferences and their corresponding Pareto solutions, enabling the learning of the complete Pareto front.\n* Proposing a novel geometric perspective for PSL, demonstrating the equivalence of Pareto set learning to Hypervolume maximization.\n* Utilizing a neural network model to effectively approximate the entire Pareto set.\n* The experimental results validate the superiority of the proposed method over baseline approaches.\n", "questions": "See Weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper presents a novel approach to multiobjective algorithms that allows for the direct modeling of the entire Pareto set.\n* The authors establish an equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis of hypervolume for Pareto set learning.\n* The theoretical foundation of the proposed method is well established, especially the derivation of the generalization gap between estimated and true hypervolumes.\n* The proposed approach is evaluated on various benchmark and real-world problems, and compared to multiple state-of-the-art algorithms, and the results are promising. \n", "weaknesses": "* There is limited discussion regarding the design of the neural network structure, and the Machine Learning perspective could benefit from further elaboration.\n* The discussion about the construction of the training data is lacking, which is crucial to understanding the similarities and differences between the training dataset and benchmark problems.\n* The evaluation of the proposed model's PSL ability is solely compared with other PSL methods. It would be beneficial to also compare its optimization performance with other traditional methods, such as decomposition-based and hypervolume-based approaches.\n* The focus of the tested problems is primarily on 2 or 3 objective optimization problems. However, it remains unclear if the proposed method can be generalized to handle more objectives. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. We have taken your concerns, and we hope our following answers can address your concerns more or less.\n\n**W1. There is a limited discussion regarding the design of the neural network structure, and the machine learning perspective could benefit from further elaboration.**\n\nThank you for the suggestion. In response, we conducted additional experiments in the $3rd$ experiment of the **general response** to examine the impact of network structure. Our findings suggest that the depth of the neural model has **minimal** influence on the generated quality, whereas the width **does** affect the results. Specifically, using a width of 64 leads to poor solutions. This finding aligns with the theoretical results discussed in previous research [3].\n\n\n**W2. The discussion about the construction of the training data is lacking, which is crucial to understanding the similarities and differences between the training dataset and benchmark problems.**\n\n As shown in Figure 2, the studied Pareto set learning (PSL) problem involves learning the mapping function $x_\\beta(\\cdot)$ from a preference vector (in polar coordinates) and the corresponding Pareto solution. The considered Pareto neural model is a mapping function denoted as $x_\\beta(\\cdot): {[0, \\pi/2]}^{m-1} \\mapsto R^n$, where $m$ represents the number of objectives and $n$ denotes the dimensionality of the solution space. And therefore, according to Eq. (5), the \"training data\" (preference vectors) are sampled from the uniform distribution, $\\text{Unif}({[0, \\pi/2]}^{m-1})$.  This sampling method provides an unbiased estimation of the true hypervolume.\n\n**W3. The evaluation of the proposed model's PSL ability is solely compared with other PSL methods. It would be beneficial to also compare its optimization performance with other traditional methods, such as decomposition-based and hypervolume-based approaches.**\n\nThank you for your feedback. We conducted \"more\" experiments, comparing the proposed method with traditional MOEAs, including **MOEA/D-Tche** [1], **SMS-EMOA** [2], and **NSGA2** [4]. Please refer to the **general response** and the **attached PDF** for details. In summary, our proposed method aims to generate the entire Pareto front, while traditional MOEAs rely on a finite population to approximate the Pareto set. We also observed that the advantages of the proposed Pareto set learning concept is particularly valuable for problems with many objectives (m ≥ 4) (in 1st experiment in general response), as approximating such a large Pareto manifold becomes increasingly challenging with only finite solutions.\n\n**W4. The ability to handle problems with more objectives.**\n\nIn response to many-objective problems (m ≥ 4) issues, we have provided a comprehensive discussion about that in the 1st experiment in **general response** and the **attached PDF**. In short, our method displays **good scalability** when applied to many-objective problems. As the Pareto set of a 4-objective problem is large (a 3-D continuous manifold), approximating this set with finite solutions by traditional MOEAs poses increasing challenges. However, using the proposed method, it is easy to train a model to approximate the **entire** continuous Pareto set, which provides more alternative solutions for the user. \n\nWe hope our response adequately addresses your concerns. If you have any additional questions or concerns, please feel free to let us know.\n\nReference \n\n[1] MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition. IEEE TEVC, 2007.\n\n[2] SMS-EMOA: Multiobjective selection based on dominated hypervolume. EJOR, 2007.\n\n[3] Any Deep ReLU Network is Shallow. Arxiv, 2023. \n\n[4] A fast and elitist multiobjective genetic algorithm: NSGA-II. TEVC, 2002."}, {"review_id": "Zb3J0lHGnZ", "reviewer": "Reviewer_6icQ", "summary": "The paper presents an algorithm to learn a continous approximation of the Pareto frontier in multi-objective reinforcement learning (MORL). The idea of the algorithm is to train a neural network to produce solutions such that their hypervolume is maximized. \nThe idea iteself is not new, but the proposed method overcome well-known hard limitations of MORL algorithms, most notably the use of the hypervolume to train the neural network. \nTo overcome this limitation, the authors propose an approximation of the hypervolume based on its computation in polar coordinates. \n", "questions": "I understand that the main contribution of the paper are theoretical, but could you run some experiments with more than 3 objectives? The MO-LQR can be easily customized to have N objectives, and it would be interesting to have a short experiment to investigate the computational and time complexity (as well as the other metrics like hypervolume and sparsity) of all algorithms (yours and baselines) on varying the number of objectives (2, 3, 4, 5). ", "limitations": "The authors discuss the limitation of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "To the best of my knowledge, this is the first paper that overcomes the hard computation of the hypervolume as loss to train neural networks to produce continuous Pareto frontier approximation. \nThe hypervolume computation is very expensive for problems with 3 objectives or more, and is non-differentiable. \nThe authors propose an approximation that overcomes these limitations and achieves good results in the experiments. \n\nThe authors also provide proofs in the appendix and acknowledge limitations of their approach, which I highly appreciate. \n", "weaknesses": "The authors missed some fundamental related work that is highly relevant to their method.\n\n- The idea of learning a continuous Pareto frontier was first proposed --even though without neural network-- by Pirotta et al., \"Multi-objective reinforcement learning with continuous pareto frontier approximation\". \n- Proposition 1 is equivalent to say that the Pareto frontier achieves the highest hypervolume. This was already shown by [17], so a reference should be added. \n- The problem formulated in Eq. (3) is a special case of the more general version by Parisi et al., Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation (Eq. (2)). In your case, you are using the hypervolume as indicator to be maximized. \n- It would be nice to stress how hard the computation of the exact hypervolume is. In particular, Friedrich et al., \"Friedrich, T., Horoba, C., & Neumann, F. (2009). Multiplicative approximations and the hypervolume indicator\" showed that it is a #P-hard problem and proposed an approximation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback. We greatly appreciate it. In response, we have carefully considered your concerns and incorporated them into our revisions.\n\n**W1-2. Missing Highly Relevant Work: The idea of learning a continuous Pareto frontier was first proposed --even though without a neural network-- by Pirotta et al. [1]. Proposition 1 is equivalent to say that the Pareto frontier achieves the highest hypervolume in [5]. So a reference should be added.**\n\nWe would like to acknowledge that the concept of learning a continuous Pareto frontier was originally proposed by Pirotta et al. [1], and we recognize the valuable discussion of Proposition 1 by [5]. In the revised version of our paper, we have ensured to appropriately cite their contributions.\n\n**W3. Problem Formulation: The problem formulated in Eq. (3) is a special case of the more general version by Parisi et al. [2] (Eq. (2)). In your case, you are using the hypervolume as an indicator to be maximized.**\n\nThank you for bringing Parisi et al.'s [2] work to our attention. In their work, they primarily focus on three indicators: the accuracy indicator $I_U$, the covering indicator $I_{AU}$, and the mixed indicator $I = \\beta_1 I_{AU} / I_U - \\beta_2$. We will make sure to cite their contributions appropriately. Due to the seminal nature of Parisi et al.'s work [2] in this field, many approaches to Pareto modeling are influenced, to some degree, by their work.\n\nNevertheless, our proposed approach provides significant and **unique contributions** beyond Parisi et al.'s [2] work. The key differences and contributions can be summarized as follows:\n\n- Our method, a **conditioned model**, facilitates mapping preferences to Pareto solutions, enabling the **convenient generation** of user-specific solutions with a single input preference. In contrast, their method does not possess this capability.\n- Our proposed method establishes a crucial connection between Pareto set learning and hypervolume, a key indicator in multi-objective optimization (MOO). In addition, our approach includes an error analysis of the gap with the true hypervolume.\n\n**W4. Computational Complexity and Approximation of the Exact Hypervolume.**\n\nWe agree that determining the hypervolume of a finite set of $n$ solutions with $m$ objectives is indeed a recognized NP-hard problem with respect to $m$, as indicated in reference [3]. We also would like to mention that the current best asymptotic runtime for $n$ solutions and $m$ objectives is $O(n \\log n+n^{m/2})$, as proposed in reference [4]. \n\nOn the other hand, our proposed approach significantly diverges from previous methods [3-5] by estimating the hypervolume through an expectation problem, as depicted in Eq. (5) and Eq. (6). As a result, our primary focus in this paper is on addressing the **statistical** approximation error.\n\nFortunately, the statistical hypervolume approximation error can be effectively bounded by the empirical mean of Eq. (5) when the number of samples is large. In this paper, we offer two types of approximation bounds (Eq. (9) and (10) in the paper).\n\n**Q1. Problems with More Objectives.**\n\nTo address many-objective problems ($m \\geq 4$), we provide a comprehensive discussion in the 1st experiment in the **general response**. Visualization results can be found in the **attached PDF**. In short, the experiments demonstrate the **good scalability** of the proposed method when applied to many-objective problems. Given the challenge of representing a high-dimensional Pareto set with finite populations using traditional MOEAs, our proposed method, which learns the entire Paret set, provides a valuable tool for handling many-objective problems.\n\nIf you have any additional remarks or questions, please feel free to let us know.\n\n\n**Reference**\n\n[1]. Multi–Objective Reinforcement Learning with Continuous Pareto Frontier Approximation. AAAI 2015. \n\n[2]. Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation. JAIR, 2016. \n\n[3]. Multiplicative approximations and the hypervolume indicator. GECCO, 2019. \n\n[4]. S-Metric Calculation by Considering Dominated Hypervolume as Klee's Measure Problem. Evolutionary Computation, 2009. \n\n[5]. The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration. EMO, 2007. "}, {"review_id": "wrdb8j5XCR", "reviewer": "Reviewer_ZpxS", "summary": "The author presents a novel approach to multiobjective algorithms aimed at modeling the Pareto set using neural networks. The proposed approach in the manuscript allows for the direct modeling of the entire Pareto set, and it also establishes an equivalence between learning the complete Pareto set and maximizing the associated hypervolume. In this study, the results of the proposed approach on various benchmark problems and real-world problems are encouraging, which makes it a promising alternative to existing multiobjective algorithms.", "questions": "- The author utilizes a 4-layer fully connected neural network in their demonstration. Is the performance of the network influenced by its structure? Are there any anticipated performance differences when using alternative architectures?\n- In Table 2, what is the analysis regarding the occurrence of 0 values for range and sparsity when employing PSL-LS with ZDT2?\n- What are some examples of classical nonparametric techniques that can enhance the robustness of the method and address the limitations of gradient-based methods?", "limitations": "- One major limitation is the reliance on gradient-based methods, which can lead to locally optimal solutions for non-convex objectives. Consequently, the method may not always find the global optimal solution. To address this issue and enhance the method's statistical guarantees, the authors propose exploring classical nonparametric techniques to boost its robustness. It would be clearer if the author could list some examples of such techniques in the manuscript. \n- Another limitation is the computational latency associated with the PSL approach, which poses a significant challenge in handling large-scale problems. As a result, the method's scalability to very large problems is limited, which restricts its practical applicability. \n- Further, the paper highlights that the finite set obtained from classical methods may not accurately approximate the continuous manifold of the Pareto set, particularly when there are numerous objectives. Thus, the method may fail to precisely represent the Pareto set, potentially leading to suboptimal solutions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors establish a crucial equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which facilitates the convergence analysis of hypervolume for Pareto set learning.\n- The author provides a clear interpretation of Pareto set learning as a hypervolume maximization problem, establishing a theoretical connection between the results of Pareto set learning and the hypervolume. \n- Also, this paper establishes a direct correspondence between specific preferences and the resulting Pareto solution within a polar coordinate system, enhancing the interpretability of the approach. \n- Further, it incorporates essential techniques in hypervolume-based Pareto set learning (PSL) for modeling the entire Pareto set, as discussed in Section 4.3. In this study, the results obtained from applying the proposed approach to various benchmark problems and real-world scenarios are highly encouraging, indicating its potential as a viable alternative to existing multiobjective algorithms. \n- Overall, the main strengths lie in developing a novel approach to multiobjective algorithms, which enables direct modeling of the entire Pareto set using neural networks and demonstrates promising results.", "weaknesses": "-The paper acknowledges that the proposed approach relies on gradient-based methods, which can result in finding solutions that are locally optimal rather than globally optimal when dealing with non-convex objectives. Additionally, the effectiveness of classical nonparametric techniques in practical applications is uncertain.\n-The computational latency of the PSL approach is recognized as a significant obstacle in effectively handling large-scale problems. This limitation prevents the method from scaling well to very large problems, thereby limiting its usefulness in practical scenarios.\n-The paper highlights that the finite set learned from classical methods may not accurately approximate the continuous manifold of the Pareto set, particularly when there are multiple objectives involved. As a result, the method may struggle to accurately represent the Pareto set, which could result in suboptimal solutions. The generalizability of the proposed framework to other problems is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive comments on our work. We hope that our following response can address your concerns more or less.\n\n**Q1. Is the performance of the network influenced by its structure? Are there any anticipated performance differences when using alternative architectures?**\n\nThe performance is **indeed** affected by the network structure. We present **numerical results** for various network structures as the 3rd experiment in the **general response**, revealing that the **depth** of the net has **a minimal impact** on the results, while the **width** does play a significant role. Decreasing the network width to 64 adversely affects performance. This observation aligns with a theoretical study in [6].\n\n**Q2. In Table 2, what is the analysis regarding the occurrence of 0 values for range and sparsity when employing PSL-LS with ZDT2?**\n\nPSL-LS is limited to identifying **a single solution** (Figure 11(e)). Due to the concave nature of the Pareto front ($f_2 = 1 - {(f_1)}^2, 0 \\leq f_1 \\leq 1$) in ZDT2, the LS-based PSL method can only find an arbitrary endpoint ((1,0) or (0,1)) of the Pareto front. \n\n**Q3;L1. What are some examples of classical nonparametric techniques that can enhance the robustness of the method and address the limitations of gradient-based methods?**\n\nThe limitation of finding only a locally optimal solution is inherent in gradient-based multiobjective approaches. In the current stage, achieving global optimality, both theoretically and practically, is still challenging for PSL.\n\nOur rough plan for applying “nonparametric techniques”' is to change the function space of $x_\\beta()$ from neural models to a certain reproducing kernel Hilbert space (RKHS) $H$ with the associated kernel $k(\\cdot, \\cdot): \\Theta \\times \\Theta \\mapsto R$, and turn to a regularized optimization problem $ \\min_{x \\in H} \\frac{c_m}{N} \\sum_{i=1}^N [\\rho(x(\\theta^{(i)}), \\theta^{(i)})^m ] + \\frac{1}{2} \\lambda_r \\\\|x\\\\|_H^2$, where $\\\\|x\\\\|_H$ is the RKHS norm of the function $x()$.\n\nThe usage of the RKHS may cause a loss of representation power while allowing a finer analysis and more robust optimization.\nIn short, due to the representer theorem of RKHS, $x()$ (in contrast to the neural model $x_\\beta()$) is now convex w.r.t. the tunable parameters, and therefore helps the convergence of ``the gradient-based methods''; the inclusion of the regularization term $\\frac12 \\lambda_r \\\\|x\\\\|_{H}^2$ is also supposed to improve the convexity and the robustness of the optimization problem.\n\nWe thank the reviewer for the question and will incorporate the explanation above into the next revision.\n\n**L2. Another limitation is the computational latency associated with the PSL approach, which poses a significant challenge in handling large-scale problems.**\n\nWe acknowledge that searching for the entire Pareto set, an ($m$-1)-dimensional continuous manifold, is indeed more challenging and time-consuming compared to searching for a single or a finite number of solutions. Notably, there have been successful applications of Pareto set learning (PSL) in large-scale problems such as multiobjective neural combinatorial optimization [1] and drug design [2].\n\nTo address **large-scale PSL problems**, a useful technique (proposed in [3]) is to learn a backbone network parameterized by $\\beta^{(sh)}$, which benefits all objectives independently of the preference information. Additionally, separate networks parameterized by $\\beta^{(i)}(\\lambda)$ conditioned on the preference information are employed to learn the trade-off objectives, where $\\lambda$ is an $m$-D preference vector. In such a way, the total number of parameters $\\beta=[\\beta^{(sh)}, \\beta^{(1)}(\\lambda), \\ldots, \\beta^{(m)}(\\lambda)]$ can be maintained in a low level. \n\nCompared to the widely used EPO-based PSL method introduced in [4-5], our proposed method has already shown superior efficiency. Table 2 and Figure 8 provide evidence that our method is 100 times faster and produces higher-quality Pareto solutions. Consequently, when dealing with large-scale problems, the proposed method outperforms EPO-based PSL in terms of efficiency.\n\n**L3. Numerous objectives.** \n\nWe present new experiment results and promising findings on many-objective problems ($m \\geq 4$). For detailed discussions, please refer to the **general response** and the **attached PDF**.\n\nFor many-objective problems, representing the entire Pareto set with a finite number of populations becomes increasingly difficult since the solutions space is very large. The proposed method, aiming to learn the full continuous  Pareto set, provides much more Pareto solutions than traditional MOEAs. We believe that the idea of the proposed method offers an alternative way of solving many-objective problems. \n\nWe welcome any additional comments or suggestions you may have.\n\nReference\n\n[1] Pareto Set Learning for Neural Multi-Objective Combinatorial Optimization. ICLR, 2022.\n\n[2] Multi-Objective GFlowNets. ICML, 2023. \n\n[3]. Multi-Task Learning as Multi-Objective Optimization. NeurIPS, 2018.\n\n[4]. Learning the Pareto Front with Hypernetworks. ICLR, 2021.\n\n[5]. Gradient Descent with Controlled Ascent in Pareto Optimization. ICML, 2020.\n\n[6] Any Deep ReLU Network is Shallow. Arxiv, 2023.\n\n\n"}, {"review_id": "AYS2n7OBFr", "reviewer": "Reviewer_Kyce", "summary": "This paper proposes a geometric perspective for Pareto set learning by establishing an equivalence between learning the complete Pareto set and hypervolume maximization. A theoretical analysis is provided to examine the gap between the estimated hypervolume and the true hypervolume of the Pareto set and empirical studies show its effectiveness.", "questions": "How are the parameters of the experiment set? Is the complexity of the Pareto front taken into account when selecting test problems in the experiments?", "limitations": "It is suggested to analyze the effectiveness of the proposed learning method on problems with irregular Pareto front.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents the equivalence between Pareto set learning and hypervolume maximization\n2. Theoretical analysis of the gap between the estimated hypervolume and the true hypervolume of the Pareto set leads to better understanding in practice.", "weaknesses": "The empirical studies need to involve more problems, e.g., those from the popular benchmarks DTLZ and WFG, which have been shown to represent some difficult properties of real-world problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. We hope that we can address your concerns with the following answers more or less. \n\n**W1. More Problems: The empirical studies need to involve more problems, e.g., those from the popular benchmarks DTLZ and WFG, which have been shown to represent some difficult properties of real-world problems.**\n\nThank you for your suggestion. We have considered \"more problems\", e.g., the real-world problems mentioned in [6], which are four-objective designing problems. Detailed results can be found in the **general response** and the attached **one-page PDF**. These empirical results show that our method demonstrates strong potential as a multi-objective optimization approach.\n \nThe benchmark problems in DTLZ and WFG were primarily intended for evolutionary multiobjective algorithms (EMOAs). Since our method is gradient-based, these problems are still challenging for PSL (and other PSL methods) due to multiple local Pareto fronts. We acknowledge this limitation around line 346 of the main paper. In response to that, we will provide a more comprehensive comparison of the advantages and disadvantages of the proposed method in the revised version.\n\n\n**Q1. Experiment Setting: How are the parameters of the experiment set? Is the complexity of the Pareto front taken into account when selecting test problems in the experiments?**\n\nThe experimental setup is standard. All experiments are conducted using Python, utilizing the PyMoo and PyTorch libraries. Our experiments did not rely on GPUs. The neural model employed for these experiments is a 4-layer ReLU/Sigmoid network, and the optimizer used is SGD, with a clipping norm of 2.0. The batch size was set to 256. \n\nWe also provide new results on the **effect of network structures** as the 3rd experiment in the **general response**. \n\nFor your second question, in our paper, we have considered various \"complexity levels\" of the Pareto front, including convex examples such as ZDT1, concave examples like ZDT2, as well as real-world problems (with 3/4 objectives) with unknown Pareto front shapes, such as Four Bar Truss Design, Rocket Injector Design, and multi-objective linear quadratic regulator problems. \n\n**L1. Problems with Irregular Pareto Front: It is suggested to analyze the effectiveness of the proposed learning method on problems with irregular Pareto front.**\n\nTo answer this question, we categorize irregular Pareto fronts (PFs) into two distinct cases: **disjointed** PFs and **degenerated** PFs. For a degenerated PF, for simplicity, we assume the PF is ($m$-2)-D, where $m$ is the number of objectives. This classification of irregular Pareto fronts is adopted from Hua et al. [5].  \n\nIn terms of disjointed Pareto fronts, the theoretical expectation of estimated hypervolume in our paper matches the true hypervolume of the Pareto front. This fulfills the theoretical analysis for this work. The challenges related to disjointed PFs are discussed in the main paper (lines 165-168). Additionally, a detailed analysis of these challenges can be found in section B.5 and Figure 19 of the supplementary material. \n\nIn the case of a degenerated ($m$-2)-D Pareto front, sampling preferences on an ($m$-2)-D subspace can reconstruct the entire front. Consequently, preference sampling on the original ($m$-1)-D sphere results in duplicated Pareto solutions, reducing the learning efficiency. As far as we know, handling an irregular Pareto front remains a challenging and unresolved topic in multi-objective optimization. The crucial aspect lies in estimating the Pareto front to enable wise sampling of preference vectors. We have identified this as a potential area for future research.\n\nWe hope to address your concerns. If you have any further questions, please let us know.\n\nReference\n\n[1]. Multi–Objective Reinforcement Learning with Continuous Pareto Frontier Approximation. AAAI 2015.\n\n[2]. Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation. JAIR, 2016. \n\n[3]. Pareto Set Learning for Expensive Multi-Objective Optimization. NeurIPS 2022. \n\n[4]. Diversity-Guided Multi-Objective Bayesian Optimization With Batch Evaluations. NeurIPS 2020. \n\n[5]. A Survey of Evolutionary Algorithms for Multi-Objective Optimization Problems with Irregular Pareto Fronts. IEEE/CAA Journal of Automatica Sinica, 2021.\n\n[6] An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 2020."}, {"review_id": "xwdJB842Bk", "reviewer": "Reviewer_Urc8", "summary": "This paper presents a novel perspective of hypervolume maximization in Pareto learning. The proposed method is largely interpretable based on this perspective, and could outperform many peer competitors considering efficiency and solutions. ", "questions": "1. Does the proposed method introduce any type of preferences between objectives?\n2. Does this method incorporate multiple neural models for joint learning? \n3. What is the crucial determinant of the improved efficiency regarding this method?\n4. How does scalability to large-scale problems embody in this work, typically compared with EPO-based approaches? \n", "limitations": "The limitations on solution quality is clearly mentioned.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The hypervolume analysis from the polar coordinate perspective is elaborate and reasonable.\n2. The proposed method achieved a generally better performance than compared evolutionary and Pareto learning algorithms.\n", "weaknesses": "1. The innovations are not emphasized. For example, if the benefit of the proposed method based on geometric analysis include improved efficiency compared with other Pareto learning methods, it is suggested to be emphasized.\n2. The details of the proposed method is inadequate. As the relationship between theoretical reasoning and methodology is obliquitous, the method is vulnerable to ignorance.\n3. Comparisons and discussions may have missed some important Pareto learning techniques like hypernetworks. Contrasts or superiority of the proposed method are deficient.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments on our work. We hope to address your concerns with the following answers.\n\n**W1. The innovations are not emphasized. For example, if the benefit of the proposed method based on geometric analysis includes improved efficiency compared with other Pareto learning methods, it is suggested to be emphasized.**\n\n- The first and main **innovation** of this work is its **error analysis**. While several Pareto learning methods exist (e.g., [1-3]), they lack theoretical analysis, and the properties of the learned Pareto model remain unclear. We establish a connection between PSL and hypervolume maximization, an important **\"geometric\"** quantity in multi-objective optimization (MOO), and provide convergence analysis for PSL with respect to HV maximization. This connection and the error bounds in this paper help us better understand PSL.\n- The second **innovation** of our proposed model is its ability to generate a Pareto solution that precisely aligns with the preference vector, enabling the localization of a Pareto solution from a **\"geometric\"** standpoint (as discussed in Property 2). In other words, our simple yet efficient method can produce an \"exact\" Pareto solution, as defined in EPO [3-4]. It's worth noting that our approach is 100 times faster than the EPO-based PSL method [3-4].\n\nWe sincerely thank your advice, and we will make our motivations more clear in the next version. \n\n**W2. Details of the Proposed Method: The details of the proposed method is inadequate. As the relationship between theoretical reasoning and methodology is obliquitous, the method is vulnerable to ignorance.**\n\nWe thank the reviewer for the suggestion. We will summarize our method in an **algorithm block** in the next revision and highlight the connection between a step in the algorithm and the corresponding quantity in our analysis.\n\n**W3. Comparisons and discussions may have missed some important Pareto learning techniques like hypernetworks. Contrasts or superiority of the proposed method are deficient.**\n\nThe proposed framework is model-agnostic and orthogonal to techniques such as using a hypernetwork. For simplicity and theoretical analysis, the paper employs a fully-connected network. We believe that integrating hypernetworks into our model is straightforward by applying the optimization strategy described in Eq. (7) of the main paper.\n\n**Q1. Does the proposed method introduce any type of preferences between objectives?**\n\nIn our Pareto neural model, \"preferences\" serve as inputs. Proposition 2 establishes the relationship between the preference vector $\\lambda$ (expressed in polar coordinates) and the corresponding Pareto solution. Once the Pareto model is trained, it can produce a new (approximate) Pareto solution based on this new preference vector. \n\n**Q2. Does this method incorporate multiple neural models for joint learning?**\n\nOur current implementation utilizes a single neural network. The extension of the proposed method with joint learning is left as future work.\n\n**Q3-4. What is the crucial determinant of the improved efficiency regarding this method? How does scalability to large-scale problems embody in this work, typically compared with EPO-based approaches?**\n\nIn short, our method enhances efficiency compared to the EPO-based approach by **eliminating** the need to calculate and operate on gradient vectors of **all** ($m$) objectives. Instead, we update the model **solely** based on the gradient of the argmin index in Eq. (6). It is worth noting that calculating a single gradient vector through backward propagation can be time-consuming in the default PyTorch implementation. As a result, the proposed approach becomes even more efficient than the EPO-based approach when applied to large-scale problems. \n\nTo apply the proposed method to large-scale problems, we can utilize a backbone network that provides benefits across all objectives. By employing a small subset of networks, optimization can be performed independently for task-specific objectives, as described in [5]. This strategy allows us to update only the task-specific parameters using PSL. Our next plan is to incorporate this strategy into the proposed approach and apply it to large-scale problems.\n\nWe hope our response adequately addresses your concerns, more or less. We appreciate any further comments or concerns you may have regarding our paper.\n\n\nReference\n\n[1]. Multi–Objective Reinforcement Learning with Continuous Pareto Frontier Approximation. AAAI 2015.\n\n[2]. Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation. JAIR, 2016.\n\n[3]. Learning the Pareto Front with Hypernetworks. ICLR, 2021.\n \n[4]. Gradient Descent with Controlled Ascent in Pareto Optimization. ICML 2020. \n\n[5]. Multi-Task Learning as Multi-Objective Optimization. NeurIPS 2018.\n\n[6]. Learning a Neural Pareto Manifold Extractor with Constraints. UAI, 2022."}, {"review_id": "c4Rx4sVXaa", "reviewer": "Reviewer_jfi6", "summary": "This manuscript introduces a method for multi-objective optimization and mainly contributes to modeling the Pareto set. Compared to previous methods which usually focus on limited and discrete Pareto solutions, the method in this manuscript is capable of modeling the full Pareto set in a continuous way. They also proposed a connection between the complete Pareto set and the related hypervolume, enabling a convergence analysis of the hypervolume as a novel metric for Pareto set learning. Also, this method bridges the Pareto solutions and their representation in a polar coordinate system. The authors also evaluate the proposed method extensively in a set of tasks.", "questions": "1. I wonder how this method generalizes in a more general multi-task learning context - for example, the accuracy of the network and the total norm of it (performance vs regularization).\n2. I wonder how this method compares to Pareto exploration methods like [1], which also explores Pareto set in a continuous way.\n\n[1] Ma, Pingchuan, Tao Du, and Wojciech Matusik. \"Efficient Continuous Pareto Exploration in Multi-Task Learning.\" International Conference on Machine Learning. PMLR, 2020.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation is clear. The visualization is intuitive and helps to understand.\n2. The experiment results look convincing. I do expect some deeper tasks though, besides these simple tasks.", "weaknesses": "1. The text on the figures is too small to read. Maybe considering to add a description in the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. We hope that our following response can address your concerns.\n\n**W1. The text on the figures is too small to read. Maybe considering to add a description in the caption.**\n\nThank you for your guidance. In the revised edition, we have implemented a larger font size ($\\geq 16$) for all labels and tick marks in all figures. \n  \n\n**Q1. Multi-Task Learning: I wonder how this method generalizes in a more general multi-task learning context - for example, the accuracy of the network and the total norm of it (performance vs regularization).**\n\nWe recognize the importance of applying our proposed framework to **large-scale multi-task learning** situations. To achieve this, one potential approach is to utilize task-specific neural networks to focus on learning the trade-off objective. Concurrently, a backbone network can be employed to learn shared parameters that benefit all tasks.\n\nThis effective technique was originally introduced in prior literature [1]. Moreover, it is worth noting that even for complex problems like multi-objective neural combinatorial optimization [2] and multi-objective drug design using GFlowNet [3], the number of task-specific parameters can be managed at a lower level. Future work is to learn the **optimal network structure** that effectively balances the size of the network (as referred to in the context of **normalization**) while simultaneously maximizing accuracy.\n\n\n**Q2. Comparison to Pareto Exploration Method: I wonder how this method compares to Pareto exploration methods like [2], which also explores Pareto set in a continuous way. Ma et al, [4].**\n\nWe appreciate your advice, and we intend to incorporate a discussion on Ma et al.'s method [4] in our revised paper. Their approach employs the Krylov subspace method for local exploration using first-order approximation, with the objective of obtaining approximate Pareto solutions in close proximity to a discovered Pareto solution. The discovered Pareto solutions are pre-computed using the methodology introduced by Lin et al. in PMTL [5]. \n\nDue to the local approximation nature of their method for generating continuous solutions, the accuracy of approximating distant solutions diminishes when they are far from a discovered solution. Consequently, as depicted in Figures (2), (10), and (12) of Ma et al.'s work [4], the ability of their approach to approximate Pareto solutions decreases when the distance between the Pareto solution and the discovered solution increases. \n\nThe proposed method in our work directly learns the mapping from the preference space, which corresponds to the ($m$-1)-dimensional unit sphere (where $m$ represents the number of objectives), to the complete set of Pareto solutions. As a result, the quality of the learned solution remains unaffected by the size of the neighborhood, unlike in Ma et al.'s method [4]. This characteristic ensures that our approach achieves superior overall quality compared to their approach.  \n\n**Reference**\n\n[1]. Multi-Task Learning as Multi-Objective Optimization. NeurIPS 2018. \n\n[2]. Pareto Set Learning for Neural Multi-Objective Combinatorial Optimization. ICLR 2022. \n\n[3]. Multi-Objective GFlowNets. ICML 2023. \n\n[4]. Efficient Continuous Pareto Exploration in Multi-Task Learning.\" ICML. 2020. \n\n[5]. Pareto multi-task learning. NeurIPS 2019.\n"}]

Author rebuttal:
**General response**

We sincerely appreciate the diligent efforts of all the reviewers. In response to their concerns, we have conducted three additional experiments.
- - -
> **1. Many-Objective Problem**

In our first experiment, we evaluate the performance of our proposed method on many-objective problems. We hope that this analysis can more or less address the concerns raised by **W4 for FLs5**, **Q1 for 6icQ**, **L3 for ZpxS**, and **W1 for Kyce**.
| Problem (obj=4)          | Indicator | Proposed PSL-HV2 (8000 sols) | NSGA2 (100 sols) |     PSL-EPO    |
|--------------------------|-----------|------------------------------|------------------|----------------|
| Car side impact design  (CSID)  | HV        | **1.88**                         | 1.66             |     1.08           |
|                          | Time      |     41.46(s)                 |    **26.36(s)**      |       19.63(m)         |
| Conceptual marine design (CMD) | HV        | **1.17**                         | 1.01             |     0.47*           |
|                          | Time      | 41.86(s)                     |     **26.68(s)**     |         19.44(m)       |

(*We have finetuned the parameters and learning rate of EPO using code from [5]. It does not work very well on CMD. )

We find that the proposed method improves the efficiency of learning the Pareto set for a large number of objectives ($m$=4) compared to traditional Multi-Objective Evolutionary Algorithms (MOEAs). In a 4-objective scenario, the Pareto set/front is a continuous 3-D manifold within an $n$/$m$-D space. The Pareto set is significantly larger than the problems considered in the main paper ($m$=2,3). Approximating this extensive Pareto set using finite solutions through traditional MOEAs presents a considerable challenge.

Taking NSGA2 [6] as a representative of conventional MOEAs, it takes approximately 26 seconds for NSGA2 to generate 100 solutions for these two problems. Our proposed method demonstrates remarkable scalability for 4-obj problems.  With a training time of just 40 seconds, we train a model which is able to (approximately) generate the **entire** Pareto set. The hypervolume of the entire Pareto set is greatly improved compared with NSGA2. The trained Pareto model has the theoretical ability to generate an infinite number of Pareto solutions, approximating the true 3-D Pareto manifold. In practice, we present results using 8000 solutions (they can be generated just very fast). For detailed results, please refer to the attached PDF.

We have observed that when applying our method to **many($m \geq 4$)-objective** problems, it **surpasses** the efficiency of EPO-based PSL. EPO-based PSL requires gradient calculations and operations for all objectives, leading to longer running times as reported in [4]. We are sincerely grateful to the reviewers for highlighting the importance of conducting experiments on many-objective problems. Based on our findings, we believe that our method holds great promise as a valuable tool for addressing many-objective problems. 
- - -
> **2. Traditional MOEAs**

As suggested by **W3 for FLs5**, we have compared with more traditional MOEAs on a three-objective Rocket Injector Design problem [1]. Specifically, we consider the**MOEAD-Tche** [2] and **SMS-EMOA** [3]. Both MOEA/D-Tche and SMS-EMOA only generate a finite number (91) of solutions, as shown in the attached file. Figure 1 clearly demonstrates that the continuous Pareto front examined in this paper accurately represents the true Pareto front, showcasing its superior ability in dealing with a 3-obj problem. The running time of these three algorithms is presented in the following table.  As our method has the ability to generate an infinite number of solutions, its running time is **comparable** to that of MOEA-Tche (91 solutions) but significantly **outperforms** SMS-EMOA. 

| Problem (obj=3)        | Indicator | MOEA/D-Tche | SMS-EMOA | Proposed (Training) |
|------------------------|-----------|-------------|----------|---------------------|
| Rocket Injector Design | Time      | 11.77s      | 8.35m    | 33.64s              |
---
> **3. Model Structure** 

The last experiment studies the network structure effect raised by **W1 for FLs5**  and **Q4 for ZpxS**. We found that the **depth** of a ReLU network has **minimal impact**, while the **width does affect** the performance. Reducing the width to 64 generally leads to a decrease in performance. This observation aligns with a theoretical study in [7], and we plan to investigate further and gather additional results.
|                 | Rocket   Injector Design |       |          | Four Bar Truss   Design |       |          |
|-----------------|--------------------------|-------|----------|-------------------------|-------|----------|
| Network         | HV                       | Range | Sparsity | HV                      | Range | Sparsity |
| m-256-n         | 40.7                     | 0.7   | 2.2      | 11.9                    | 1.6   | 1.5      |
| m-256-256-n     | 40.5                     | 0.7   | 2.2      | 11.8                    | 1.5   | 1.8      |
| m-256-256-256-n | 40.6                     | 0.7   | 2.7      | 11.9                    | 1.6   | 1.5      |
| m-64-64-64-n    | 38.1                     | 0.3   | 1.3      | 11.8                    | 1.5   | 1.7      |

---

Reference

[1] An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 2020.

[2] MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition, IEEE Transactions on Evolutionary Computation, 2007.

[3] SMS-EMOA: Multiobjective selection based on dominated hypervolume, European Journal of Operational Research, 2007. 

[4] Learning a Neural Pareto Manifold Extractor with Constraints. UAI, 2022.

[5] https://github.com/dbmptr/EPOSearch.

[6] A fast and elitist multiobjective genetic algorithm: NSGA-II. TEVC, 2002.

[7] Any Deep ReLU Network is Shallow. arXiv, 2023.

Meta-review: This work presents a novel approach to Pareto Set Learning (PSL). It shows the PSL is equivalent to hypervolume maximization, and shows how to efficiently optimize the hypervolume directly. The authors add a theoretical analysis and show their approach improves upon current methods on several benchmarks. Taking everything into account, it is an interesting, novel, and valuable paper that should be included in the following proceedings.
Decision: Accept (poster)