Title: Hypervolume Maximization: A Geometric View of Pareto Set Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: multiobjective optimization;multitask learning;hypervolume maximization;Pareto set learning
Abstract: This paper presents a novel approach to multiobjective algorithms aimed at modeling the Pareto set using neural networks. 
Whereas previous methods mainly focused on identifying a finite number of solutions, our approach allows for the direct modeling of the entire Pareto set. 
Furthermore, we establish an equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis of hypervolume (as a new metric) for Pareto set learning. 
Specifically, our new analysis framework reveals the connection between the learned Pareto solution and its representation in a polar coordinate system. 
We evaluate our proposed approach on various benchmark problems and real-world problems, and the encouraging results make it a potentially viable alternative to existing multiobjective algorithms. Code is available at \url{https://github.com/xzhang2523/hvpsl/tree/master}.

Hypervolume Maximization: A Geometric View of Pareto Set Learning

Abstract

This paper presents a novel approach to multiobjective algorithms aimed at modeling the Pareto set using neural networks. Whereas previous methods mainly focused on identifying a finite number of solutions, our approach allows for the direct modeling of the entire Pareto set. Furthermore, we establish an equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis of hypervolume (as a new metric) for Pareto set learning. Specifically, our new analysis framework reveals the connection between the learned Pareto solution and its representation in a polar coordinate system. We evaluate our proposed approach on various benchmark problems and real-world problems, and the encouraging results make it a potentially viable alternative to existing multiobjective algorithms. Code is available at https://github.com/xzhang2523/hvpsl/tree/master.

1 Introduction

Pareto solutions [1] effectively balance multiple objectives, making optimal tradeoffs among all objectives. For a Pareto solution, improving one objective without adversely affecting others is not possible. The Pareto set (PS) refers to the set of all Pareto solutions, while the Pareto front (PF) denotes the image of the PS in the objective space. Over the years, numerous methods have been developed to generate a single Pareto optimal solution [2, 3, 4], or a finite set of Pareto solutions [5, 6, 7, 8, 9].

Under mild conditions on continuous optimization problems with m objectives, the Pareto set becomes a ( m -1)-D continuous manifold [10, 11]; given that, the finite set learned from classical methods might not accurately approximate the continuous manifold. To overcome this issue, Pareto set learning (PSL) has been recently proposed in [12, 13], with the hope of training a neural network model capable of approximating the entire Pareto set. The significant advantage of learning the complete Pareto set is that a trained model can generate the optimal solution for any preference (conceptually the weight of different objectives, see the definition in Lemma 1) in real time, allowing for comprehensive decisions.

The training approach utilized in prior PSL techniques can be viewed as an extension of decomposition-based multiobjective optimization strategies [5] to an infinite preference scenario. However, several unresolved issues existed in prior PSL methods. Firstly, the interpretation of the sum of aggregation functions under different preferences lacks clarity, simply taking all preferences as equally important and thus leading to a partial Pareto front. Secondly, aside from the EPO-based

b

PSL proposed in [3], there has been limited investigation into the connection between a preference and its corresponding Pareto solution. Despite the effectiveness of the EPO-based PSL approach, its computational latency has been identified as a significant obstacle in its ability to handle largescale problems. Thirdly, theoretical guarantees of PSL are lacking, and therefore the underlying mechanism of PSL remains unknown.

After identifying the challenges regarding existing PSL techniques for solving multiobjective optimization problems, we propose a new approach that addresses all these issues. Our method utilizes a geometric perspective on Pareto set learning, which takes the form of a hypervolume maximization problem. This approach brings several benefits. Firstly, it provides a clear interpretation of Pareto set learning as a hypervolume maximization problem, allowing us to build a theoretical bound between the PSL results and the hypervolume, one of the most important indicators in multiobjective optimization. Secondly, our approach establishes a clear correspondence between a specific preference and the resulting Pareto solution in a polar coordinate system. Lastly, the new perspective implies several important techniques in hypervolume-based PSL to learn an entire Pareto set.

The contributions of this work can be summarized as follows:

Weprovide a novel geometric perspective for Pareto set learning, which recognizes its equivalence to hypervolume maximization. As a result, the Pareto solutions derived from this method are aligned precisely with the polar angle under mild conditions, which is the input of the neural network in use.

We provide the first theoretical analysis for Pareto set learning, specifically examining the gap between the estimated hypervolume and the true hypervolume of the Pareto set. Our analysis enhances a better understanding of Pareto set learning.

Wehave developed techniques to learn a more complete Pareto front, surpassing previous methods that obtain a partial solution set. Our approach achieves promising results on various benchmark problems and real-world problems.

2 Related works

Multiobjective Optimization via Hypervolume Maximization. The hypervolume (hv) indicator measures the quality of multiobjective optimization solution sets and is consistent with Pareto dominance. Maximizing hypervolume is a basic principle in multiobjective optimization algorithm design, and several popular algorithms like SMS-EMOA [14], SIBEA [15, 16], and MO-CMA-ES [17] use it to generate a finite number of optimal solutions. A new method named multiobjective learning using hv maximization was proposed in [18], which leverages hypervolume maximization while training neural networks to obtain a finite set of Pareto solutions. This method differs from our approach as it does not rely on a model to approximate the Pareto set. Instead, it adopts the gradient search method from [19] for hypervolume maximization. A major drawback of this approach is its high computational complexity in obtaining the hypervolume gradient.

Pareto Set Learning and Conditional Models. Pareto set learning involves using a neural network conditioned on user preference to learn the entire Pareto set. This concept was first introduced in [12, 13] and has gained popularity in various areas like drug design [20], multitask image classification [21, 22], and multiobjective neural combinatorial optimization [23]. In addition, some studies have used a similar conditional model to generate a set of diverse solutions, including the works addressing multiobjective reinforcement learning [24, 25, 26], and the works generating policies under different conditional levels [27, 28].

3 Background

A multiobjective optimization (MOO) problem with m objectives can be stated as follows:

where x is the decision variable, X is a compact decision space, and f ( x ) : X → R m is the objective function. The objective domain Y = f ◦ X 1 . Given two candidate solutions x ( a ) , x ( b ) ∈ X , we

say x ( a ) dominates x ( b ) if and only if ∀ i ∈ [ m ] 2 , f i ( x ( a ) ) ≤ f i ( x ( b ) ) , and ∃ j ∈ [ m ] \ i such that f j ( x ( a ) ) < f j ( x ( b ) ) . x ∗ is a Pareto solution if no other solution x ∈ X dominates x ∗ ;its image f ( x ∗ ) is called a Pareto objective . The set of all Pareto solutions is called the Pareto set ( PS ). And the image of PS is called the Pareto front T := f ◦ PS . In a separate note, a solution x ′ is called weakly Pareto optimal if there exists no solution x ∈ X such that f i ( x ) ≺ f i ( x ′ ) 3 . A Pareto solution is a weakly Pareto solution, but a weakly Pareto solution is not necessarily a Pareto solution.

The nadir/ideal point of a multiobjective problem is constructed by the worst/best objective values of the Pareto set. y nadir i = sup y ∈T { y i } , and similarly, y ideal i = inf y ∈T { y i } , ∀ i ∈ [ m ] . The hypervolume indicator is defined as a metric of the optimality of a set of objective vectors A , detailed as follows.

Definition 1 (The hypervolume (HV) indicator [29]) . The hypervolume indicator of a set A is defined as:

where Λ( · ) denotes the Lebesgue measure, and r is a reference vector. We require that r ⪰ y nadir .

Figure 1 depicts the hypervolume indicator of the Pareto front, represented by the yellow region.

4 Pareto Set Learning via Hypervolume Maximization

As shown in Figure 2, Pareto set learning (PSL) aims to learn the whole Pareto set by a model, which differs from population-based MOEAs [5, 6]. One straightforward way is to learn a Pareto neural model x β ( · ) : Θ ↦→ R n that translates a polar coordinate θ ∈ Θ into a Pareto solution x ∈ R n , where Θ = [0 , π 2 ] m -1 . After the training, for any θ ∈ Θ , the Pareto neural model x β ( · ) can directly generate an approximate Pareto solution in realtime ( < 1s ).

In this paper, we present a novel geometry view to formulate PSL based on hypervolume maximization. Firstly, we notice the following equivalence holds:

Proposition 1 (Equivalence between PSL and HV maximization) . When r is a reference point dominating the whole Pareto front T , and A is a non-dominated solution set, we have T = arg max A H r ( A ) .

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2(

x

)

r

Hypervolume

Pareto front

Weakly Pareto solutions

Pareto solutions

Dominated solutions

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2(

x

)

r

Mapping

Solution

f

(

x

(

))

Polar angle

Proposition 1 shows that H r ( A ) achieves the maximal value if the model x β ( · ) learns all Pareto solutions. So, it is natural to convert the Pareto set learning problem with a parameter β as a hypervolume maximization problem:

In Equation (3), the subscript r refers to a fixed reference point satisfying y ⪯ r, ∀ y ∈ T . The objective function measures the quality of the Pareto neural model x β ( · ) and this objective achieves optimality only when the model learns all Pareto solutions.

4.1 Pareto Front Hypervolume as an Expectation

In this section, we are devoted to casting the optimal solution of Equation (3) (i.e., the hypervolume of the Pareto front) into a polar coordinate form. To accomplish this, we leverage an existing technique to transform the hypervolume calculation of a finite set of solutions into an expectation problem, which is formulated as the following lemma,

Lemma 1 (Hypervolume scalarization of a finite set [30, 31, 32]) . Let A = { y (1) , y (2) , . . . , y ( N ) } be set of finite objective vectors and r is a reference point, r ⪰ y ( i ) . The hy-

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2(

x

)

Hypervolume

r

f

(

x

)

(

)

(

x

,

)

Weakly Pareto front

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2(

x

)

r

Hypervolume

Pareto front

(a) HV of (weakly) Pareto front.

(b) HV of finite points.

pervolume of set A with r can be expressed in terms of an expectation,

where we specify θ follows Unif (Θ) , the uniform distribution on Θ . λ ( θ ) ∈ S m -1 + is the preference vector: it is the Cartesian coordinate of θ on the positive unit sphere S m -1 + , with λ 1 ( θ ) = sin θ 1 sin θ 2 ... sin θ m -1 , λ 2 ( θ ) = sin θ 1 sin θ 2 ... cos θ m -1 , . . . , λ m ( θ ) = cos θ 1 . c m = π m/ 2 2 m Γ( m/ 2+1) is a constant that is only dependent on m and Γ( · ) is the Gamma function.

We notice that, it is not 'economic' to estimate the hypervolume H r ( A ) in Lemma 1: numerous samples from Unif (Θ) are needed, while the expensive Monte Carlo estimation only provides the hypervolume for a finite set A . Even when A consists of only one point, the calculation of H r ( A ) still relies on the expectation over Unif (Θ) . However, extending the scope of the input set A in Lemma 1, we discover that the hypervolume of a (weakly) Pareto front containing an infinite number of objective vectors can as well be expressed as an expectation,

with proof provided in Appendix B.2. The distance function denoted as ρ X ( θ ) , as illustrated in Figure 3(a), represents the Euclidean distance between a reference point denoted as r and the Pareto front at the coordinate of θ . The distance function ρ X ( θ ) can be precisely evaluated using the equation provided, with the proof available in Appendix B.4.

where ρ ( x, θ ) = min i ∈ [ m ] { r i -f i ( x ) λ i ( θ ) } represents the projected distance of an objective vector f ( x ) at coordinate θ , as illustrated in Figure 3(a). Let λ be the shorthand of λ ( θ ) , we have the following proposition of the optimal solution of Equation (6).

Proposition 2. Given an objective vector y that satisfies r 1 -y 1 λ 1 = . . . = r m -y m λ m . 1 : If y is weakly Pareto optimal, then y is one of the optimal solutions of ˜ g ( y, λ ) = max y ∈Y min i ∈ [ m ] { r i -y i λ i } . 2 : If y is Pareto optimal, then y is the only optimal solution of ˜ g ( y, λ ) .

̸

Proof. 1 : If there exists a y ′ such that ˜ g ( y ′ , λ ) < ˜ g ( y, λ ) , then y ′ ≺ y , contradicting the weakly Pareto optimality of y . 2 : If there exists a y ′ = y such that ˜ g ( y ′ , λ ) = ˜ g ( y, λ ) , then y ′ ⪯ y , and there exists at least one index j where y ′ j < y j . This contradicts the Pareto optimality of y .

Remark 1 (On the 'exact' Pareto solution) . Proposition 2 establishes a connection between the polar coordinate θ and the resulting Pareto solution. Once the Pareto neural model is well-trained, and assuming the absence of weakly Pareto solutions and the existence of an exact solution [3], the

Pareto neural model can predict the 'exact' Pareto solution aligned with the polar angle θ . Since we only utilize an efficient aggregation function (Equation (6)), this property reveals that there is no need to solve the optimization problem (Eq. (24) in [3]) in order to achieve the 'exact' Pareto solution.

We close the subsection with a note on the case of disjointed Pareto fronts. With a disjointed Pareto front T , the function ρ X ( θ ) can serve to measure the distance between the reference point r and the attainment surface [33]. In this disjointed scenario, the expectation form of the Pareto front hypervolume by ρ X ( θ ) , as specified by Equation (5), still holds (c.f. Appendix B.4).

4.2 Alternative Forms of Hypervolume

In this section, we train the Pareto neural model x β ( · ) by maximizing the alternative forms of hypervolumes. The optimization of the value of H r ( f ◦ x β (Θ)) as defined in Equation (1) is challenging. However, by utilizing Equation (6), we can choose to instead optimize an easy-to-compute form of hypervolume, denoted by H r ( β ) . This surrogate hypervolume function H r ( β ) , called PSL-HV1, is defined as follows:

Here, ρ ( x β ( θ ) , θ ) ≤ ρ X ( θ ) is the projected distance at angle θ as defined by Equation (6). When the Pareto neural model is supposed to learn the whole Pareto set, ρ ( x β ( θ ) , θ ) → ρ X ( θ ) , ∀ θ ∈ Θ and thus H r ( β ) →H r ( T ) .

We also provide a second surrogate hypervolume function, denoted as PSL-HV2 (derivation in Appendix B.5). The Pareto hypervolume can be estimated as the volume difference between the regions dominating r and those that dominate the Pareto front. PSL-HV2 maximizes the following objective,

where ρ β ( θ ) = ρ X ( x β ( θ ) , θ ) m and ρ X ( θ ) is an alternative projected distance function ρ X ( x β ( θ ) , θ ) = max i ∈ [ m ] f i ( x β ( θ )) -y ideal i λ i ( θ ) . Although the Pareto neural model x β ( · ) theoretically has the ability to represent the complete Pareto set [34] with both surrogate hypervolume functions, our empirical results show that the quality of the learned solutions is sensitive to the specific choice: Equation (7) and Equation (8) give different performances on various tasks.

𝜽

𝒙𝜷(𝜽)

𝒇(𝒙𝜷(𝜽))

ഥ

ℋ𝒓(𝜷)

𝒙𝜷(∙)

Angle

𝚯

Neural model

Solution

Objective

Hypervolume

𝝏 ഥ

ℋ𝒓(𝜷)

𝝏𝒇(𝒙𝜷(𝜽))

𝝏𝒇(𝒙𝜷(𝜽))

𝝏𝒙𝜷(𝜽)

𝝏𝒙𝜷(𝜽)

𝝏𝜷

𝝏 ഥ

ℋ𝒓(𝜷)

𝝏𝜷

Chain rule

Algorithm. The goal of HV-based PSL is to maximize the objective function, max β H r ( β ) . Both Equation (7) and Equation (8) can be effectively optimized using gradient descent approaches. The empirical gradient ∇ β ˆ H r ( β ) can be obtained efficiently through backpropagation, as illustrated in Figure 4. Their analytical forms are provided in Appendix B.5. For practical algorithms, the empirical gradient is estimated from a batch of N angles θ (1) , . . . , θ ( N ) through sampling from Unif (Θ) . Then the neural network is updated by β ← β + η ∇ β ˆ H r ( β ) , where η is a learning rate.

4.3 Discussion on Different PSL Approaches

PSL-HV1 and PSL-HV2 have different designs for their projected distance function ρ ( x, θ ) , with PSL-HV2 exhibiting significant growth as λ approaches boundary preferences such as [0,1] or [1,0] (taking the m = 2 case as an example). This property brings both pros and cons compared to PSL-HV1. The steep gradient may present numerical difficulties, but it also permits PSL to adopt larger weights and learn boundary solutions more effectively. We have observed that this feature of PSL-HV2 is particularly useful for learning the complete Pareto set, as opposed to previous PSL methods that only learned a partial Pareto front.

To avoid only learning a partial Pareto set, we propose a simple strategy for PSL-HV1 that involves using a larger reference point than the nadir point. This approach, which is common in hypervolume-based optimization methods [35, 36], gives larger weights to boundary preferences and allows for a more complete Pareto set.

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

x

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1(

x

)

f

2(

x

)

LS

(

)

y

Before our study, two primary approaches for PSL optimization were followed [12, 37]: optimizing the expectation using EPO/LS and optimizing using Tchebycheff functions. However, EPO-based PSL is slow for complex multiobjective problems and solely learns a partial Pareto set, and LS-based PSL only finds the convex portion of a Pareto front. Our work extends the approach in [37] by introducing a hypervolume interpretation for PSL. We establish a clear relationship between preferences and solutions and develop a theoretical analysis framework for PSL. Additionally, we propose methods to recover a more complete Pareto set, thus improving upon the existing approaches.

5 Statistical Guarantees on Hypervolume Convergence

This section first establishes the bounds for the generalization error of the Pareto front when using the exact distance function ρ X ( θ ) in Section 5.1. Subsequently, we discuss the generalization bound of proposed HV-based Pareto set learning in Section 5.2.

5.1 Convergence of the Pareto Front Hypervolume

Proposition 3. 1 : ρ ( x, θ ) is concave when f i 's are convex. 2 : ρ ( x, θ ) is quasi-concave when f i 's are quasi-convex. 3 : ρ β ( θ ) defined in Equation (7) is quasi-concave w.r.t x β ( θ ) when the objective function f ( x ) is quasi-convex.

Proof. The statements 1 and 2 are consequences of the preservation of convexity/quasi-convexity by taking the point-wise maximum of convex/quasi-convex functions. The statement 3 is justified by the fact that a non-decreasing composition of a quasi-concave function preserves quasi-concavity. Additionally, it can be observed that the function f ( x ) = x m for x ≥ 0 (and f ( x ) = x otherwise) is a non-decreasing function.

Remark 2. To better understand Proposition 3, we analyze a modified bi-objective VLMOP2 problem from [38]. In this example, let f 1 ( x ) = 1 . 2(1 -exp( -( x +1) 2 )) and f 2 ( x ) = 1 -exp( -( x -1) 2 ) . Figure 5 illustrates the limitation of the linear scalarization (LS) approach [39]. It shows that the LS-based PSL may yield multiple local optimas when using a non-negative linear combination of quasi-convex functions f 1 and f 2 , resulting local optimal Pareto models. However, it is worth noting that the function -ρ ( x, θ ) considered remains quasi-convex [40].

Assuming that b ≤ r i -y i ≤ B , ∀ i ∈ [ m ] , y ∈ T . Let Z ( θ ) = c m ρ X ( θ ) m , then sup θ Z ( θ ) ≤ c m B m m m/ 2 . Let ̂ H r ( T ) := 1 N ∑ N i =1 Z ( θ ( i ) ) denote the empirical estimation of H r ( T ) with N

samples. Via Hoeffding inequality, similar to [32], we have the following inequality,

5.2 PSL Generalization Bound

In this section we show that the proposed hypervolume metric can fit into the regular neural network studies and similarly enjoy the statistical guarantees on error ∣ ∣ H r ( β ) -H r ( T ) ∣ ∣ for PSL-HV1. Specifically, the term can be interrupted as the difference between the metric of x ˆ β computed and the metric of ground truth x ∗ , indicating the quality of the MLP map x ˆ β . We will shortly show the error above will converge with regular SGD optimizers and large enough sample size N , as expected.

We can first decompose the aforementioned error as follows,

in which we define ̂ H r ( β ) = c m N ∑ N i =1 [ ρ ( x β ( θ ( i ) ) , θ ( i ) ) m ] as the empirical hypervolume estimation associated with parameter β . β ∗ is a set of reference parameters for the MLP map x β . Those errors respectively correspond to generalization error ( ε 1 ), optimization error ( ε 2 ), generalization error ( ε 3 ), and approximation error ( ε 4 ). We can conceptually set the reference parameter β ∗ as a good local maxima (in the local region of ˆ β ) for hypervolume maximization so that the approximation error ε 4 is small due to the universal approximation character of the MLP model family [41]. In our analysis, we assume that regular SGD technique brings ˆ β close enough to the local maxima β ∗ , resulting in a small optimization error ε 2 .

The rest steps to show convergence is to control the generalization error ε 1 and ε 3 . We present the following theorem to give the uniform convergence rate of the generalization error, which applies to both ε 1 and ε 3 .

Theorem 1 (Generalization error) . Let the object functions f i 's all be L f -Lipschitz, and let x β be an L -layer MLP σ L ( W L σ L -1 ( · · · σ 1 ( W 1 θ ) · · · )) , with 1 -Lipschitz positive homogeneous activation σ i 's and ∥ W i ∥ F ≤ B w , ∀ i ∈ [ L ] (under this setting β is the set of W i 's). The design matrix X θ denotes the N collected samples θ i 's. We further set the reference point r so that r i -f i ( x ) ∈ [ b, B ] , ∀ i, for x ∈ T ∪ f ◦ x β ∗ (Θ) ∪ f ◦ x ˆ β (Θ) ⊂ R n . With probability at least 1 -δ ( δ < 1 2 ), for β = β ∗ or ˆ β , we have

Remark 3. The results above imply the generalization errors ε 1 and ε 3 can converge at the rate of 1 / √ N , considering for each row in X θ its L 2 -norm is bounded from above by √ ( m -1) π 2 / 4 .

Obtaining the results require adaptations of classical Rademacher complexity techniques. The difficulty comes from the minimum form of ρ ( x, θ ) = min i ∈ [ m ] { r i -f i ( x ) λ i ( θ ) } , which differs from usual loss functions. We provide the detailed derivation in Appendix B.6. We further remark the scale assumption on r i -f i ( x ) is mild. f ◦ x β ∗ (Θ) and f ◦ x ˆ β (Θ) are supposed to approach T , the Pareto Front, under proper optimization procedures; we are thus able to select a reference point r far away enough from the set T ∪ f ◦ x β ∗ (Θ) ∪ f ◦ x ˆ β (Θ) to make the assumption hold.

6 Experiments

Testing problems. This section demonstrates that our method can generate high-quality continuous Pareto solutions by the Pareto neural model for multiobjective synthetic, design, and control

problems. We evaluated our approach's performance against established methods using well-known benchmark problems such as ZDT1-2 (m=2) [42], VLMOP1-2 (m=2) [38], and real-world design problems like Four Bar Truss Design (RE21, m=2), Hatch Cover Design (RE24, m=2), and Rocket Injector Design (m=3) [43] as well as MO-LQR (m=2,3) [44]. MO-LQRs serve as multiobjective reinforcement learning problems. For simplicity, except MO-LQR, all objectives are normalized to [0,1]. Problem details are described in Appendix A.3.

Neural model architecture and feasibility guarantees. We employed a 4-layer fully connected neural network, similar to [37], to construct our Pareto neural model x β ( · ) . The network is optimized using Stochastic Gradient Descent (SGD) with a batch size of 256 4 . For constrained problems, to conform to the lower bound ( l ) and upper bound ( u ) constraints, a sigmoid activation function mapped the previous layer's output to these boundaries. For unconstrained problems, the output solution was derived from a linear combination of x mid. Further architecture details are available in Appendix A.2.

Baseline methods. Our approaches are compared to existing hypervolume-based multiobjective evolutionary algorithms (MOEAs) SMS-EMOA [14] and various Pareto set learning (PSL) methods. Before our approach was developed, three PSL approaches were developed, namely, EPO-based and Tchebycheff-based. Baseline methods are listed as follows,

One hypervolume-based multiobjective evolutionary algorithm: SMS-MOEA [14], which uses evolutionary algorithms to maximize the hypervolume of a finite set of solutions.

The LS-based PSL [12], optimizing min β E λ ∼ Unif (∆ m -1 ) [ ∑ m i =1 λ i f i ( x β ( θ ))] , where ∆ m -1 is the ( m -1 )-simplex. Since linear scalarization can only find the convex part of the Pareto front [39], theoretically, this method cannot recover the full Pareto front.

The EPO-based PSL [12], optimizing min β E λ ∼ Unif (∆ m -1 ) [ EPO ( f ( x β ( θ )) , λ )] , the expectation of EPO loss [3]. The EPO-based PSL is slow since each solving for each EPO subproblem, which involves an expensive optimization problem.

The Tchebycheff-based PSL [23, 37], optimizing min β E λ ∼ Unif (∆ m -1 ) [max i ∈ [ m ] λ i f i ( x β ( θ ))] .

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.0

0.1

0.2

0.3

0.4

0.5

0.6

f

1

0.65

0.70

0.75

0.80

0.85

0.90

0.95

f

2

(f) SMS-EMOA

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

(f) SMS-EMOA

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

f

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(a) HV1

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(b) HV2

0.3

0.4

0.5

0.6

0.7

0.8

f

1

0.1

0.2

0.3

0.4

0.5

0.6

f

2

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

f

3

(c) Tche

0.10

0.15

0.20

0.25

0.30

0.35

f

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

f

2

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(d) EPO

0.30

0.32

0.34

0.36

f

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

f

2

0.00

0.05

0.10

0.15

0.20

0.25

0.30

f

3

(e) LS

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

0.0

0.2

0.4

0.6

0.8

1.0

f

3

(f) SMS-EMOA

(Metrics.) To evaluate our learned Pareto solutions, we use three indicators. These are: (1) the Hypervolume ( HV ), which measures the covered objective space, where a high value indicates greater

coverage; (2) the Range indicator, which measures the extent of the Pareto front learned by the model, where a larger Range is preferred; and (3) the Sparsity metric, which evaluates the degree of sparsity in the learned Pareto front, where a lower sparsity value indicates a denser Pareto front. See Appendix A.1 for formal metric definitions.

Key findings from experiments are summarized as follows.

1 (Comparison with SMS-EMOA.) PSL and HV-based evolutionary algorithms serve as different purposes. PSL is designed to find an infinite set of solutions using a neural network model, while EMOAs aim to find a finite solution set. This difference becomes evident in three-objective problems, as shown in Figure 8, where 100 evenly distributed solutions by traditional MOEAs fall short of approximating the full Pareto set/front. PSL requires only several seconds of training time and can quickly predict highly approximated Pareto solutions for other unknown preferences. In contrast, SMS-EMOA typically takes a much longer time, especially for objectives larger than three. The advantage of evolution-based SMS-EMOA is its ability to find global optimal solutions, while current PSL models still struggle with handling local optimas.

Table 2 reports the mean results for various PSL approaches under five random seeds. The standard deviation values can be found in Appendix A.4. All experiments were conducted with 1000 iterations for PSL-HV1, PSL-HV2, LS-based PSL, and Tchebycheff-based PSL, while EPO-based PSL is limited to 100 iterations due to time limitation.

2 (Comparison with LS-based PSL.) It is well-recognized that LS-based PSL [12] recovers only the convex part of a Pareto front [39], resulting in finding a small portion of Pareto solutions for concave problems. For example, in the case of ZDT2, LS-based PSL could only find two extreme points. We would like to mention another less-discussed limitation of linear scalarization: the nonuniform distribution of Pareto objectives under uniformly distributed preferences. This issue is evident in problems like ZDT1 and VLMOP2 where Pareto objectives are densely distributed at the front's margins, but fewer objectives are distributed around the center of the front.

3 (Comparison with EPO-based PSL.) As mentioned in Remark 1, both EPO and our proposed method (under mild conditions) can find the exact Pareto objective aligned with the preference vector. However, our proposed approach is approximately 40+ times faster than EPO-based PSL. Unlike the EPO-based PSL method, our approach only requires calculating the gradient of one objective function, avoiding the need to compute gradients for all objectives and solve complex optimization problems as described in their work [3]. This efficiency advantage is particularly evident in

MO-LQR problems, where EPO-based PSL takes significantly more time to execute. Furthermore, we have observed that EPO-based PSL does not perform well on tasks ZDT1 and Rocket Injector Design.

4 (Balancing range and sparsity of a PF.) When simply running PSL, we observed a tendency for the Pareto objectives to concentrate on the central portion of the Pareto front, neglecting solutions that correspond to boundary preferences, such as f = [0 , 1] / [1 , 0] for bi-objective problems. To address this observation, a simple technique is to assign higher weights to the boundary objectives. For PSL-HV1, a larger reference point r is suggested to be used in comparison to the nadir point, and therefore more coordinates will correspond to boundary Pareto objectives (refer to Remark 4 in Appendix B.4). However, setting the reference point to be excessively large can result in a sparse learned Pareto front; we empirically set this offset to be 0.6 (the empirical study is in Figure 10 and 11). In PSL-HV2, the preference λ is placed in the denominator, resulting in greater weighting for the boundary Pareto objectives when the preference value approaches zero.

PSL-HV2 is related to the modified Tchebycheff (mTche) scalarization function [45], but the latter has different sampling techniques and lacks a hypervolume interpretation. When λ approaches [0,1] or [1,0], both mTche and PSL-HV2 faces numerical challenges resulting in a large gradient, which can be effectively addressed the gradient clipping technique [46, 47] with a clipping norm of a certain value (e.g., 4.0 is used). Figure 9 shows that the learning process is unstable without gradient clipping but is stabilized when gradient clipping is applied. In contrast, PSL-HV1 involves minimization rather than maximization in its inner optimization problem and does not suffer from gradient explosion.

7 Conclusion, Limitation, and Future Work

This paper presented a method to learn the set of all Pareto solutions through hypervolume maximization. The method was motivated by a previous lemma, which allows for hypervolume estimation of infinite solutions. We further gave a precise geometric perspective on Pareto set learning; we studied the generalization gap between estimated and true hypervolumes, and discussed the key techniques to train a Pareto neural model that outperformed previous methods. The proposed method will also be incorporated into the Moon library [48].

0

200

400

600

800

1000

Iteration

0

20

40

60

80

100

PSL loss

0

200

400

600

800

1000

Iteration

100

200

300

400

500

PSL loss

Limitation and future works. The acknowledged constraint of our work is the reliance on gradient-based methods, which can result in locally optimal solutions when the objectives f i 's are non-convex. To overcome this issue and provide more solid statistical guarantee, one feasible direction is to explore classical nonparametric techniques that can improve the robustness of the method. Another way is to use some evolutionary methods such as [49, 50, 51, 52] to skip local optimal solutions. Additionally, we plan to apply the more robust Pareto neural models to large-scale multiobjective problems, such as molecular design and deep reinforcement learning.

References

Kaisa Miettinen. Nonlinear multiobjective optimization , volume 12. Springer Science & Business Media, 2012.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr´ eal, Canada , pages 525-536, 2018.

Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6597-6607. PMLR, 2020.

Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep multiobjective learning. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 1306-1311. IEEE, 2021.

Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on evolutionary computation , 11(6):712-731, 2007.

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation , 6(2):182-197, 2002.

Xingchao Liu, Xin Tong, and Qiang Liu. Profiling pareto front with multi-objective stein variational gradient descent. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 14721-14733, 2021.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch´ e-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 12037-12047, 2019.

Liang Zhao and Qingfu Zhang. Hypervolume-guided decomposition for parallel expensive multiobjective optimization. IEEE Transactions on Evolutionary Computation , 2023.

R Timothy Marler and Jasbir S Arora. Survey of multi-objective optimization methods for engineering. Structural and multidisciplinary optimization , 26:369-395, 2004.

William Karush. Minima of functions of several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago , 1939.

Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. ArXiv preprint , abs/2010.06313, 2020.

Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research , 181(3):16531669, 2007.

Eckart Zitzler, Dimo Brockhoff, and Lothar Thiele. The hypervolume indicator revisited: On the design of pareto-compliant indicators via weighted integration. In Evolutionary MultiCriterion Optimization: 4th International Conference, EMO 2007, Matsushima, Japan, March 5-8, 2007. Proceedings 4 , pages 862-876. Springer, 2007.

Michael Emmerich, Nicola Beume, and Boris Naujoks. An emo algorithm using the hypervolume measure as selection criterion. In Evolutionary Multi-Criterion Optimization: Third International Conference, EMO 2005, Guanajuato, Mexico, March 9-11, 2005. Proceedings 3 , pages 62-76. Springer, 2005.

Christian Igel, Nikolaus Hansen, and Stefan Roth. Covariance matrix adaptation for multiobjective optimization. Evolutionary computation , 15(1):1-28, 2007.

Timo M Deist, Monika Grewal, Frank JWM Dankers, Tanja Alderliesten, and Peter AN Bosman. Multi-objective learning using hv maximization. In Evolutionary Multi-Criterion Optimization: 12th International Conference, EMO 2023, Leiden, The Netherlands, March 20-24, 2023, Proceedings , pages 103-117. Springer, 2023.

Michael Emmerich and Andr´ e Deutz. Time complexity and zeros of the hypervolume indicator gradient field. In EVOLVE-a bridge between probability, set oriented numerics, and evolutionary computation III , pages 169-193. Springer, 2014.

Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. ArXiv preprint , abs/2210.12765, 2022.

Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K RoyChowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10955-10964, 2022.

Alan Q Wang, Adrian V Dalca, and Mert R Sabuncu. Computing multiple image reconstructions with a single hypernetwork. ArXiv preprint , abs/2202.11009, 2022.

Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multiobjective reinforcement learning and policy adaptation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch´ e-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 14610-14621, 2019.

Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Now´ e, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 11-20. PMLR, 2019.

Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference-driven multiobjective reinforcement learning algorithm. ArXiv preprint , abs/2208.07914, 2022.

Joey Hong, Aviral Kumar, and Sergey Levine. Confidence-conditioned value functions for offline reinforcement learning. ArXiv preprint , abs/2212.04607, 2022.

Phillip Swazinna, Steffen Udluft, and Thomas Runkler. User-interactive offline reinforcement learning. ArXiv preprint , abs/2205.10629, 2022.

Andreia P Guerreiro, Carlos M Fonseca, and Lu´ ıs Paquete. The hypervolume indicator: Problems and algorithms. ArXiv preprint , abs/2005.00515, 2020.

Ke Shang, Hisao Ishibuchi, Min-Ling Zhang, and Yiping Liu. A new r2 indicator for better hypervolume approximation. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 745-752, 2018.

Jingda Deng and Qingfu Zhang. Approximating hypervolume and hypervolume contributions using polar coordinate. IEEE Transactions on Evolutionary Computation , 23(5):913-918, 2019.

Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multiobjective black box optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 11096-11105. PMLR, 2020.

Carlos M Fonseca and Peter J Fleming. On the performance assessment and comparison of stochastic multiobjective optimizers. In International conference on parallel problem solving from nature , pages 584-593. Springer, 1996.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Hisao Ishibuchi, Ryo Imada, Naoki Masuyama, and Yusuke Nojima. Use of two reference points in hypervolume-based evolutionary multiobjective optimization algorithms. In Parallel Problem Solving from Nature-PPSN XV: 15th International Conference, Coimbra, Portugal, September 8-12, 2018, Proceedings, Part I 15 , pages 384-396. Springer, 2018.

Hisao Ishibuchi, Ryo Imada, Yu Setoguchi, and Yusuke Nojima. How to specify a reference point in hypervolume calculation for fair performance comparison. Evolutionary computation , 26(3):411-440, 2018.

Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. arXiv e-prints , pages arXiv-2210, 2022.

David A Van Veldhuizen and Gary B Lamont. Multiobjective evolutionary algorithm test suites. In Proceedings of the 1999 ACM symposium on Applied computing , pages 351-357, 1999.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Harvey J Greenberg and William P Pierskalla. A review of quasi-convex functions. Operations research , 19(7):1553-1570, 1971.

Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6231-6239, 2017.

Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation , 8(2):173-195, 2000.

Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing , 89:106078, 2020.

Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing , 263:3-14, 2017.

Xiaoliang Ma, Qingfu Zhang, Guangdong Tian, Junshan Yang, and Zexuan Zhu. On tchebycheff decomposition approaches for multiobjective evolutionary optimization. IEEE Transactions on Evolutionary Computation , 22(2):226-244, 2017.

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Xiaoyuan Zhang, Ji Cheng, Liang Zhao, Song Lai, Cheng Gong, Weiduo Liao, Longcan Chen, Xi Lin, Ke Shang, Zhichao Lu, Yifan Chen, Tao Qin, and Qingfu Zhang. Moon: A standardized/flexible framework for multiobjective optimization. 01 2024.

Hongzong Li and Jun Wang. From soft clustering to hard clustering: A collaborative annealing fuzzy c -means algorithm. IEEE Transactions on Fuzzy Systems , 2023.

Hongzong Li and Jun Wang. Capacitated clustering via majorization-minimization and collaborative neurodynamic optimization. IEEE Transactions on Neural Networks and Learning Systems , 2022. in press.

Charu Aggarwal and Karthik Subbian. Evolutionary network analysis: A survey. ACM Computing Surveys (CSUR) , 47(1):1-36, 2014.

Xin Yao and Yong Liu. A new evolutionary system for evolving artificial neural networks. IEEE transactions on neural networks , 8(3):694-713, 1997.

Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 1060710616. PMLR, 2020.

Kaisa Miettinen. Nonlinear multiobjective optimization , volume 12. Springer Science & Business Media, 1999.

Matus Telgarsky. Deep learning theory lecture notes, 2020.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27 , pages 3-17. Springer, 2016.

A Experiment Details

A.1 Metrics

We assess the performance of a Pareto neural model, denoted as x β ( · ) , by examining its output with the input of ̂ Θ . Here, ̂ Θ represents a set of N uniformly discrete angles. The resulting output solution set is denoted as A and can be expressed as A = { y (1) , . . . , y ( N ) } , where A = f ◦ x β ( ̂ Θ) . The three metrics are:

The Hypervolume indicator [29], which measures both the diversity and convergence of A ;

The Range indicator, which measure the angular span of A ;

The Sparsity indicator [53], which measures the average distances between neighborhood points.

(The Hypervolume Indicator.) The hypervolume indicator [29] measuring A is standard, which has been defined in the main paper,

where r is a reference vector, r ⪰ y nadir . For bi-objective problems, the reference point r is set to [3.5, 3.5], whereas for three-objective problems, the reference point is set to [3.5, 3.5, 3.5].

(The Range Indicator.) The range indicator of a set of solutions A : A = { y (1) , . . . , y ( N ) } measures the angular span of A in their polar coordinates. Let ( ρ ( i ) , θ ( i ) ) be the polar coordinate of objective vectors y ( i ) with a reference point r . The relationship between the Cartesian and polar coordinates is,

The Range indicator can be defined as the minimal angle span among all angles.

(The Sparsity Indicator.) The sparsity indicator first introduced in [53]. A small sparsity indicator is preferred, implying that the learned objective set is dense.

where ˜ y ( i ) j is the i-th solution, and the j-th objective values in the sorted list by the non-dominating sorting algorithm [6]. For a better illustration, the unit of the Sparsity indicator is 10 -3 for bi-objective problems and 10 -7 for three objective problems.

A.2 Neural Model Architecture and Feasibility Guarantees

We use a 4-layer fully connected neural network similar to [37] for the Pareto neural model x β ( · ) . We optimize the network using Stochastic Gradient Descent (SGD) optimizer with a batch size of 256. The first three layers are,

For constrained problems, to satisfy the constraint that the solution x β ( λ ) must fall within the lower bound ( l ) and upper bound ( u ) , a sigmoid activation function is used to map the previous layer's output to these boundaries,

For unconstrained problems, the output solution is obtained through a linear combination of x mid,

A.3 Benchmark Multiobjective Problems

Standard Multiobjective Optimization (MOO) problems. ZDT1-2 [42] and VLMOP1-2 [38] are widely recognized as standard multi-objective optimization (MOO) problems and are commonly employed in gradient-based MOO methods. ZDT1 exhibits a convex Pareto front described by ( y 2 = 1 - √ y 1 , 0 ≤ y 1 ≤ 1 ). ZDT2 presents a non-convex Pareto front defined by ( y 2 = 1 -y 2 1 , 0 ≤ y 1 ≤ 1 ), and the LS-based PSL approach can only capture a small number of Pareto solutions.

Real world designing problem. Three real-world design problems with multi-objective optimization are the Four Bar Truss Design (RE21), Hatch Cover Design (RE24), and Rocket Injector Design (RE37). In order to simplify the optimization process, the objectives have been scaled to a range of zero to one.

Multiobjective Linear Quadratic Regulator. The Multiobjective Linear Quadratic Regulator (MO-LQR) problem is first introduced in [44]. MO-LQR is regarded as a specialized form of multi-objective reinforcement learning, where the problem is defined by a set of dynamics presented through the following equations:

In accordance with the settings discussed in the aforementioned work by Parisi et al. [44], A , B , and Σ are adopted as identity matrices. The initial state for the bi-objective problem is set to s 0 = [10 , 10] , whereas for the three-objective problem, it is set to s 0 = [10 , 10 , 10] . The reward function is defined as r i ( s t , a t ) , where i represents the respective objective. The function is formulated as follows:

̸

̸

Here, ξ is the hyperparameter value that has been set to 0.1. The ultimate objective of the MOLQR problem is to optimize the total reward while simultaneously taking into account the discount factor of γ = 0 . 9 . The control matrix K LQR is assumed to be a diagonal matrix, and the diagonal elements of this matrix are treated as decision variables. The objectives are scaled with 0.01 for better illustration purposes.

Table 3 shows the number of decision variables and objectives used in this paper.

A.4 Results on All Problems

Results for all the examined problems are depicted in Figures 12-20, and combined with the results tabulated in Table 2 of the main paper, several conclusions can be made.

Behavior of LS-based PSL. The linear scalarization function can only learn the convex part of a Pareto front [39]. This fact is validated by Figure 13(e), where LS-based PSL can only learn several discrete solutions.

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(a)

r

=

y

nadir

+0

.

1

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2

PSL

True

Pref.

(b)

r

=

y

nadir

+0

.

25

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(c)

r

=

y

nadir

+0

.

4

0.0

0.5

1.0

1.5

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

f

2

PSL

True

Pref.

(d)

r

=

y

nadir

+0

.

55

0.0

0.5

1.0

1.5

2.0

f

1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

f

2

PSL

True

Pref.

(e)

r

=

y

nadir

+0

.

70

However, it is crucial to note that the connection between a solution and its corresponding preference vector, λ ( θ ) , is non-uniform . This observation was rarely discussed in previous literature. Therefore, a uniform sampling of preferences will not result in a uniform sampling of Pareto objectives. This observation is supported by the results depicted in Figures 12(e), 15(e), and 17(e), where the learned Pareto objectives by LS-based PSL are not uniformly distributed. And as a result, the sparsity indicators are rather high, which indicates the learned front is sparse in some regions.

Time Consumption of EPO-based PSL. In comparison to our approach, the Exact Pareto Optimization [3] algorithm, which serves as the foundation for EPO-based PSL [12], exhibits low efficiency due to two factors.

To execute the Exact Pareto Optimization (EPO) algorithm, it is necessary to compute the gradients of all objectives, ∇ f i ( x ) 's. This prerequisite entails performing m backpropagations in pytorch, resulting in higher computational costs. In contrast, our approach relies just one back-propagation operation, rendering it a more efficient option in comparison to EPO.

For each iteration, the Exact Pareto Optimization (EPO) algorithm entails solving a complicated optimization problem based on the specific value of f i 's, utilizing the respective gradients of ∇ f i ( x ) 's. In contrast, our method does not rely on solving optimization problems for each iteration.

Trade-off between Range and Sparsity. We find that the reference point r plays a crucial role in the performance of the final learned Pareto front for PSL-HV1. To demonstrate this effect, we consider different values of r from the set [0 . 1 , 0 . 25 , 0 . 4 , 0 . 55 , 0 . 7] in addition to the nadir point of each problem. The results are shown in the following two problems on the ZDT1 and ZDT2 problems.

Based on our empirical findings, emphasizing boundary Pareto objectives is crucial for recovering a complete Pareto set. From Figure 10 and 11, we observe that treating all coordinates θ equally (small nadir point) typically results in a partial recovery of the Pareto set. When r exceeds a certain threshold (e.g., 0.4), raising its value directs Pareto objectives towards the boundary of the Pareto front, leading to poor uniformity of Pareto objective distribution.

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(a)

r

=

y

nadir

+0

.

1

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2

PSL

True

Pref.

(b)

r

=

y

nadir

+0

.

25

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(c)

r

=

y

nadir

+0

.

4

0.0

0.5

1.0

1.5

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

f

2

PSL

True

Pref.

(d)

r

=

y

nadir

+0

.

55

0.0

0.5

1.0

1.5

2.0

f

1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

f

2

PSL

True

Pref.

(e)

r

=

y

nadir

+0

.

70

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

A.5 Pseudocode

For completeness, we provide the pseudocode of the proposed method. PSL-HV1 is selected as an example. η is a positive learning rate.

Algorithm 1: PSL-HV1.

Input: The multiobjective function f ( · ) and initial Pareto neural model parameter β . for k=1: N iter do Sample N preference angles θ (1) , . . . , θ ( N ) from the positive unit sphere S m -1 + . Calculate the preference λ (1) , . . . , λ ( N ) by Equation (26). Calculate the empirical PSL gradient ∇ β ˆ H r by Equation (28) with N sampled preferences. Update β by: β ← β + η ∇ β ˆ H r . end Output: The trained Pareto neural model x β ( · ) .

B Characters of Hypervolume Maximization

B.1 The Notation Table

To enhance the clarity of the paper, we have included a summary of the main notations in Table 4.

B.2 Hypervolume Calculation in the Polar Coordinate

The idea of calculating the hypervolume is similar to the Lemma 5 proved in Appendix of [32]. The difference is that, we use a distance function ρ X ( θ ) as defined in Equation (6). For completeness, we provide the proof for Equation (5) in this section.

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

f

2

PSL

True

Pref.

(a) HV1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(b) HV2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(c) Tche

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(d) EPO

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

PSL

True

Pref.

(e) LS

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

f

1

1.5

2.0

2.5

3.0

3.5

4.0

4.5

f

2

PSL

True

Pref.

(a) HV1

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

f

1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

f

2

PSL

True

Pref.

(b) HV2

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

f

1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

f

2

PSL

True

Pref.

(c) Tche

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

f

1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

f

2

PSL

True

Pref.

(d) EPO

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

f

1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

f

2

PSL

True

Pref.

(e) LS

2.00

2.25

2.50

2.75

3.00

3.25

3.50

f

1

2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

f

2

2.0

2.5

3.0

3.5

4.0

f

3

(a) HV1

2.0

2.5

3.0

3.5

f

1

2.00

2.25

2.50

2.75

3.00

3.25

3.50

f

2

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

f

3

(b) HV2

2.0

2.2

2.4

2.6

2.8

3.0

3.2

f

1

2.0

2.2

2.4

2.6

2.8

3.0

3.2

f

2

2.0

2.2

2.4

2.6

2.8

3.0

f

3

(c) Tche

2.2

2.4

2.6

2.8

f

1

2.2

2.4

2.6

2.8

3.0

f

2

2.2

2.4

2.6

2.8

3.0

3.2

3.4

f

3

(d) EPO

2.0

2.2

2.4

2.6

2.8

3.0

3.2

f

1

2.0

2.2

2.4

2.6

2.8

3.0

3.2

f

2

2.0

2.2

2.4

2.6

2.8

3.0

f

3

(e) LS

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

f

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(a) HV1

0.0

0.2

0.4

0.6

0.8

1.0

f

1

0.0

0.2

0.4

0.6

0.8

1.0

f

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(b) HV2

0.3

0.4

0.5

0.6

0.7

0.8

f

1

0.1

0.2

0.3

0.4

0.5

0.6

f

2

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

f

3

(c) Tche

0.10

0.15

0.20

0.25

0.30

0.35

f

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

f

2

0.1

0.2

0.3

0.4

0.5

0.6

f

3

(d) EPO

0.30

0.32

0.34

0.36

f

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

f

2

0.00

0.05

0.10

0.15

0.20

0.25

0.30

f

3

(e) LS

𝒇𝟏

𝒇𝟐

𝒓

dv

𝝆

H r ( T ) = ∫ R m I Ω dy 1 . . . dy m ('dv' denoted the infinitesimal sector area. ) = ∫ π 2 0 . . . ∫ π 2 0 ︸ ︷︷ ︸ m -1 dv (dv equals the angle ratio multiplied by ρ m multiplied by the unit for volume. ) = ∫ π 2 0 . . . ∫ π 2 0 ︸ ︷︷ ︸ m -1 c m · ρ X ( θ ) m 2 π · π m -2 dθ 1 . . . dθ m -1 ︸ ︷︷ ︸ dθ (Re-arranging the terms.) = c m 2 π m -1 ∫ π 2 0 . . . ∫ π 2 0 ︸ ︷︷ ︸ m -1 ρ X ( θ ) m dθ = c m 2 π m -1 · ( π 2 ) m -1 · E θ [ ρ X ( θ ) m ] = c m E θ [ ρ X ( θ ) m ] . (18)

We specify θ ∼ Unif (Θ) = Unif ([0 , π 2 ] m -1 ) in Equation (18).

Line 2 holds since it represents the integral of Ω expressed in polar coordinates, wherein the element dv corresponds to the volume associated with a segment obtained by varying dθ .

Line 3 calculates the infinitesimal volume of dv by noticing the fact that the ratio of dv to c m is ρ X ( θ ) m 2 π · π m -2 . Line 4 is a simplification of Line 3. And Line 5 and 6 express the integral in its expectation form.

B.3 MOOTheory

In this section, we give the definitions and give proofs of the propositions used in the main paper.

We first formally define the term of aggregation function.

Definition 2 (Aggregation function (modified from the Definition 2.6.1 in [54])) . A function g ( · , λ ) : R m ↦→ R representing the preferences of the decision maker among the objective is called an aggregation function. g ( · , λ ) : R m ↦→ R is decreasing, indicating that g ( y ( a ) , λ ) < g ( y ( b ) , λ ) , when y ( a ) ≺ y ( b ) .

This definition is slightly different from the original definition in Definition 2.6.1 in [54]), as we modify the condition from 'strictly decreasing' to 'decreasing'. The considered aggregation function ρ ( x, θ ) does not satisfy the 'strictly decreasing' condition in the original definition. When an aggregation function is only decreasing, the optimal objective can be weakly Pareto optimal. For this modified version of aggregation function, we have the following Lemma modified from Theorem 2.6.2 in [54].

Lemma 2. Let y ∗ belongs to the optimal solution set of g ( · , λ ) , then y ∗ is weakly Pareto optimal or Pareto optimal. Further, if y ∗ be the only optimal solution of g ( · , λ ) , then y ∗ is Pareto optimal.

We also provide the upper bound of the distance function ρ X ( θ ) in this section by the following lemma.

Lemma 3 (The bound of in ρ X ( θ ) Equation (6)) .

when b ≤ r i -f i ( x ) ≤ B, ∀ x ∈ X , ∀ i ∈ [ m ] and ∥ λ ( θ ) ∥ = 1 .

Proof. We show that the following inequalities hold,

The transition from line one to line two is due to the fact that the inequality r i -f i ( x ) ≤ B holds for all x ∈ X and for all i ∈ [ m ] . The transition from line two to line three is max ∥ λ ( θ ) ∥ =1 ( min i ∈ [ m ] { B λ i ( θ ) }) is an optimization problem under the constraint ∥ λ ( θ ) ∥ = 1 . The upper bound for this optimization is when λ i = . . . = λ m = m -1 / 2 .

B.4 ρ X ( θ ) as a Max-Min Problem

In this section, we first present the proof of Equation (6) from the main paper. To begin, we define the attainment surface A T [33] to handle disjointed PFs. Intuitively, the attainment surface is continuous, containing the weakly Pareto objectives and Pareto objectives (see Figure 22).

Definition 3 (Attainment surface [33]) . The attainment surface A T induced by the Pareto front (PF) T is defined as the boundary set of the PF .

With this definition, the distance ρ X ( θ ) between the reference point r and the point coordinated at θ can be calculated using the following equation:

where λ i are shorthand's for λ i ( θ ) .

Proof. We first define the point P .

Definition 4. P is defined as the intersection point of the ray from r along angle θ and the attainment surface.

There are two cases to consider:

1 P is a Pareto objective, i.e., P ∈ T . 2 P belongs to the attainment surface but not to the Pareto front, i.e., P ∈ ( A T \ T ) .

Let x ∗ be one of the optimal solutions of Problem max x ∈X ρ ( x, θ ) . Next, we will separately demonstrate the validity of the above-mentioned statements.

1 When P is a Pareto objective, the according to Proposition 2, we have that y = P = f ( x ∗ ) is the only objective of the problem ˜ g ( y, λ ) = max y ∈Y min i ∈ [ m ] { r i -y i λ i } . And therefore, ρ X ( θ ) =

2 When y = P ∈ ( A T \ T ) . Then, according to Proposition 2, y is one of the optimal objective of problem ˜ g ( y, λ ) = max y ∈Y min i ∈ [ m ] { r i -y i λ i } . In such a case, according to the definition of attainment surface (Equation (21)), we have that there exist at least one index j , where j = arg min r j -f j ( x ∗ ) λ j such that ρ X ( θ ) = max x ∈X ρ ( x, θ ) = max x ∈X min i ∈ [ m ] { r i -f i ( x ) λ i } = min i ∈ [ m ] { r i -f i ( x ∗ ) λ i } = r j -f j ( x ∗ ) λ j = r i -P i λ i , i ∈ [ m ] .

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

f

2(

x

)

(

)

r

1

f

1

r

2

f

2

r

Solution

Attainment surface

The previous discussion explained that, in the case of a disjointed Pareto front , the distance between r and the attainment surface corresponding to angle θ is denoted by ρ X ( θ ) . In Figure 22, the black dot represents the solution for this scenario. It is worth noting that the integral of the distance function ρ X ( θ ) yields the hypervolume of a disjointed Pareto front, i.e., H r ( T ) = H r ( A T ) . This property aligns with the purpose of our paper.

In closing this section, we note that the absence of a (weakly) Pareto objective aligned with the preference angle θ .

Remark 4. When P (in Definition 4) is not a (weakly) Pareto objective, The optimal solution y ∗ w.r.t the problem ˜ g ( y, λ ) = max y ∈Y min i ∈ [ m ] { r i -y i λ i } belongs to the boundary of the Pareto front ∂ T . Further y ∗ = arg min y ∈Y max i ∈ [ m ] { y i -P i λ i } .

B.5 Pareto Front Hypervolume Calculation (Type2 and gradients)

In this section, we provide another form to calculate the Pareto front hypervolume. We first define region A as the set of points dominating the Pareto front.

We use the notation Λ( · ) to represent the Lebesgue measure of a set. Geometrically, as illustrated in Figure 23, it can be observed that:

The volume of A can be calculated in a polar coordinate as follows,

where c m is a constant and ρ X ( θ ) represents the distance from the ideal point to the Pareto front at angle θ . This distance function ρ X ( θ ) is obtained by solving the optimization problem assuming that any ray from θ intersects with the Pareto front.

Problem 1.

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

f

1(

x

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

f

2(

x

)

A

Hypervolume

(

)

r

Ideal point

The relationship between preference λ and the polar angle θ is as follows:

Combining Equation (24) and Equation (25) implies that H r ( β ) can be estimated as an expectation problem,

Remark 5. Compared to PSL-HV1, PSL-HV2 has weaker theoretical results due to its reliance on a stronger assumption. This assumption requires the presence of a (weakly) Pareto objective aligned on the ray originating from any coordinate θ . An additional challenge with PSL-HV2 is that the Lipschitz constant cannot be bounded, making generalization theory for PSL-HV2 significantly more difficult (Equation (30) ).

We conclude this section by presenting the analytical expression for ∇ β H r ( β ) . The gradient for PSL-HV1 can be computed using the chain rule, which yields:

where we specify θ to satisfy the law of Unif (Θ) . The gradient of PSL-HV2 can be calculated by,

B.6 Proof of Theorem 1

Definitions and preliminaries . The proof will heavily utilize the existing results on Rademacher complexity of MLPs. We will first provide some useful definitions and facts. We start with the definition of Rademacher complexity as follows:

Definition 5 (Rademacher complexity, Definition 13.1 in [55]) . Given a set of vectors V ⊆ R n , we define the (unnormalized) Rademacher complexity as

where each coordinate ϵ i is an i.i.d. Rademacher random variable, meaning Pr[ ϵ i = +1] = 1 2 = Pr[ ϵ i = -1] . Furthermore, we can accordingly discuss the behavior of a function class G on S = { z i } N i =1 by using the following set:

and its Rademacher complexity is

Utilizing Rademacher complexity, we can conveniently bound the generalization error via the following theorem:

Theorem 2 (Uniform Generalization Error, Theorem 13.1 and Corollary 13.1 in [55]) . Let G be given with g ( z ) ∈ [ a, b ] a.s. ∀ g ∈ G . We collect i.i.d. samples S = { z i } N i =1 from the law of random variable Z . With probability ≥ 1 -δ ,

Specifically, the Rademacher complexity in using MLP is provided by the following theorem:

Theorem 3 (Rademacher complexity of MLP, Theorem 1 in [56]) . Let 1 -Lipschitz positive homogeneous activation σ i be given, and

Then

We can then utilize the following composition character of Rademacher complexity, to help induce the final Rademacher complexity of hypervolume.

Lemma 4 (Rademacher complexity of compositional function class, adapted from Lemma 13.3 in [55]) . Let g : Θ → R n be a vector of n multivariate functions g (1) , g (2) , . . . , g ( n ) , G denote the function class of g , and further G ( j ) be the function class of g ( j ) , ∀ j . We have a 'partially Lipschitz continuous' function ℓ ( g ( θ ) , θ ) so that | ℓ ( g 1 ( θ ) , θ ) -ℓ ( g 2 ( θ ) , θ ) | ≤ L ℓ ∥ g 1 ( θ ) -g 2 ( θ ) ∥ for all g 1 , g 2 ∈ G and a certain L ℓ > 0 ; the associated function class of ℓ is denoted as G ℓ . We then have

Proof. This proof extends Lemma 13.3 in [55] for vector-valued g and 'partially Lipschitz continuous' ℓ . We first similarly have

where ϵ ( j ) 1 's are new i.i.d. Rademacher variables; the last inequality comes from Proposition 6 in [57] (see Equations (5)-(10) in [57] for more details). We can then get rid of the absolute value by

considering swapping f and h ,

We can thus upper bounded URad ( G ℓ | S ) by

Repeating this procedure for the other coordinates, we can further have

which leads to our claim in the lemma.

Proof of Theorem 1 . We are now geared up for the complete proof.

Proof. We first introduce the sketch of the proof. We mainly utilize Theorem 2 to attain the claimed results in Theorem 1. Specifically, we set the random sample set S = { θ i } N i =1 , the function class G as { θ ↦→ c m ρ ( x β ( θ ) , θ ) m } (the assumption r i -f i ( x ) ∈ [ b, B ] indicates that ρ ( x, θ ) = min i ∈ [ m ] { r i -f i ( x ) λ i ( θ ) } ≥ b ≥ 0 and by the definition in Equation (7), ρ β ( θ ) is thus always ρ ( x ( θ ) , θ ) m ; x β ( · ) is an L -layer MLP to be specified later). Applying Theorem 2, we can obtain that with probability at least 1 -δ 2 ,

where the definition of URad and G | S can be found in Definition 5. Simply replacing G with -G := {-g : g ∈ G} , we can have the inequality of the other direction with probability at least 1 -δ 2 :

where we apply the property URad ( -G | S ) = URad ( G | S ) . We thus, with probability at least 1 -δ (as a result of union bound), can upper bound sup g ∈G ∣ ∣ E θ g ( θ ) -1 N ∑ i g ( θ i ) ∣ ∣ by

For the next step, we will upper bound URad ( G | S ) by analyzing the structure of c m ρ ( x β ( θ ) , θ ) m and utilizing the existing bound (see Theorem 3) for Rademacher complexity of MLP x β .

The main idea of controlling URad ( G | S ) is to obtain the 'partially Lipschitz continuity' that | ρ ( x β ( θ ) , θ ) -ρ ( x β ′ ( θ ) , θ ) | ≤ L ρ ∥ x β ( θ ) -x β ′ ( θ ) ∥ for a certain L ρ > 0 ; with the 'partially Lipschitz continuity' we can apply Lemma 4 and obtain the desired bound. For simplicity, we denote x β ( θ ) , x β ′ ( θ ) respectively as x, x ′ , and use λ j 's as shorthand for λ j ( θ ) 's. We now expand the difference | ρ ( x β ( θ ) , θ ) -ρ ( x β ′ ( θ ) , θ ) | as:

If we respectively denote the minima index of the two finite-term minimization as j ∗ and k ∗ , we can then upper bound | ρ ( x β ( θ ) , θ ) -ρ ( x β ′ ( θ ) , θ ) | by

We note there is a special property for λ j when j is the minima index: as ∥ λ ∥ = 1 , there must be a certain λ j ≥ 1 / √ m , and since b ≤ r j -f j ( x ) ≤ B, ∀ j , we have

With this special property, we obtain

We further have

which establishes the 'partially Lipschitz continuity'. The distance function ρ ( x β ( θ ) , θ ) is bounded by Lemma 3. We can then apply Lemma 4 and have

Combining the pieces above, we finally have

which is the generalization error bound we claim.

Peer reviews:
comments: SUMMARY:
This manuscript introduces a method for multi-objective optimization and mainly contributes to modeling the Pareto set. Compared to previous methods which usually focus on limited and discrete Pareto solutions, the method in this manuscript is capable of modeling the full Pareto set in a continuous way. They also proposed a connection between the complete Pareto set and the related hypervolume, enabling a convergence analysis of the hypervolume as a novel metric for Pareto set learning. Also, this method bridges the Pareto solutions and their representation in a polar coordinate system. The authors also evaluate the proposed method extensively in a set of tasks.

STRENGTHS:
1. The presentation is clear. The visualization is intuitive and helps to understand.
2. The experiment results look convincing. I do expect some deeper tasks though, besides these simple tasks.

WEAKNESSES:
1. The text on the figures is too small to read. Maybe considering to add a description in the caption.

QUESTIONS:
1. I wonder how this method generalizes in a more general multi-task learning context - for example, the accuracy of the network and the total norm of it (performance vs regularization).
2. I wonder how this method compares to Pareto exploration methods like [1], which also explores Pareto set in a continuous way.

[1] Ma, Pingchuan, Tao Du, and Wojciech Matusik. "Efficient Continuous Pareto Exploration in Multi-Task Learning." International Conference on Machine Learning. PMLR, 2020.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper presents a novel perspective of hypervolume maximization in Pareto learning. The proposed method is largely interpretable based on this perspective, and could outperform many peer competitors considering efficiency and solutions.

STRENGTHS:
1. The hypervolume analysis from the polar coordinate perspective is elaborate and reasonable.
2. The proposed method achieved a generally better performance than compared evolutionary and Pareto learning algorithms.

WEAKNESSES:
1. The innovations are not emphasized. For example, if the benefit of the proposed method based on geometric analysis include improved efficiency compared with other Pareto learning methods, it is suggested to be emphasized.
2. The details of the proposed method is inadequate. As the relationship between theoretical reasoning and methodology is obliquitous, the method is vulnerable to ignorance.
3. Comparisons and discussions may have missed some important Pareto learning techniques like hypernetworks. Contrasts or superiority of the proposed method are deficient.

QUESTIONS:
1. Does the proposed method introduce any type of preferences between objectives?
2. Does this method incorporate multiple neural models for joint learning? 
3. What is the crucial determinant of the improved efficiency regarding this method?
4. How does scalability to large-scale problems embody in this work, typically compared with EPO-based approaches?

LIMITATIONS:
The limitations on solution quality is clearly mentioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes a geometric perspective for Pareto set learning by establishing an equivalence between learning the complete Pareto set and hypervolume maximization. A theoretical analysis is provided to examine the gap between the estimated hypervolume and the true hypervolume of the Pareto set and empirical studies show its effectiveness.

STRENGTHS:
1. This paper presents the equivalence between Pareto set learning and hypervolume maximization
2. Theoretical analysis of the gap between the estimated hypervolume and the true hypervolume of the Pareto set leads to better understanding in practice.

WEAKNESSES:
The empirical studies need to involve more problems, e.g., those from the popular benchmarks DTLZ and WFG, which have been shown to represent some difficult properties of real-world problems.

QUESTIONS:
How are the parameters of the experiment set? Is the complexity of the Pareto front taken into account when selecting test problems in the experiments?

LIMITATIONS:
It is suggested to analyze the effectiveness of the proposed learning method on problems with irregular Pareto front.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author presents a novel approach to multiobjective algorithms aimed at modeling the Pareto set using neural networks. The proposed approach in the manuscript allows for the direct modeling of the entire Pareto set, and it also establishes an equivalence between learning the complete Pareto set and maximizing the associated hypervolume. In this study, the results of the proposed approach on various benchmark problems and real-world problems are encouraging, which makes it a promising alternative to existing multiobjective algorithms.\n\nSTRENGTHS:\n- The authors establish a crucial equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which facilitates the convergence analysis of hypervolume for Pareto set learning.\n- The author provides a clear interpretation of Pareto set learning as a hypervolume maximization problem, establishing a theoretical connection between the results of Pareto set learning and the hypervolume. \n- Also, this paper establishes a direct correspondence between specific preferences and the resulting Pareto solution within a polar coordinate system, enhancing the interpretability of the approach. \n- Further, it incorporates essential techniques in hypervolume-based Pareto set learning (PSL) for modeling the entire Pareto set, as discussed in Section 4.3. In this study, the results obtained from applying the proposed approach to various benchmark problems and real-world scenarios are highly encouraging, indicating its potential as a viable alternative to existing multiobjective algorithms. \n- Overall, the main strengths lie in developing a novel approach to multiobjective algorithms, which enables direct modeling of the entire Pareto set using neural networks and demonstrates promising results.\n\nWEAKNESSES:\n-The paper acknowledges that the proposed approach relies on gradient-based methods, which can result in finding solutions that are locally optimal rather than globally optimal when dealing with non-convex objectives. Additionally, the effectiveness of classical nonparametric techniques in practical applications is uncertain.\n-The computational latency of the PSL approach is recognized as a significant obstacle in effectively handling large-scale problems. This limitation prevents the method from scaling well to very large problems, thereby limiting its usefulness in practical scenarios.\n-The paper highlights that the finite set learned from classical methods may not accurately approximate the continuous manifold of the Pareto set, particularly when there are multiple objectives involved. As a result, the method may struggle to accurately represent the Pareto set, which could result in suboptimal solutions. The generalizability of the proposed framework to other problems is unclear.\n\nQUESTIONS:\n- The author utilizes a 4-layer fully connected neural network in their demonstration. Is the performance of the network influenced by its structure? Are there any anticipated performance differences when using alternative architectures?\n- In Table 2, what is the analysis regarding the occurrence of 0 values for range and sparsity when employing PSL-LS with ZDT2?\n- What are some examples of classical nonparametric techniques that can enhance the robustness of the method and address the limitations of gradient-based methods?\n\nLIMITATIONS:\n- One major limitation is the reliance on gradient-based methods, which can lead to locally optimal solutions for non-convex objectives. Consequently, the method may not always find the global optimal solution. To address this issue and enhance the method's statistical guarantees, the authors propose exploring classical nonparametric techniques to boost its robustness. It would be clearer if the author could list some examples of such techniques in the manuscript. \n- Another limitation is the computational latency associated with the PSL approach, which poses a significant challenge in handling large-scale problems. As a result, the method's scalability to very large problems is limited, which restricts its practical applicability. \n- Further, the paper highlights that the finite set obtained from classical methods may not accurately approximate the continuous manifold of the Pareto set, particularly when there are numerous objectives. Thus, the method may fail to precisely represent the Pareto set, potentially leading to suboptimal solutions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm to learn a continous approximation of the Pareto frontier in multi-objective reinforcement learning (MORL). The idea of the algorithm is to train a neural network to produce solutions such that their hypervolume is maximized. \nThe idea iteself is not new, but the proposed method overcome well-known hard limitations of MORL algorithms, most notably the use of the hypervolume to train the neural network. \nTo overcome this limitation, the authors propose an approximation of the hypervolume based on its computation in polar coordinates.\n\nSTRENGTHS:\nTo the best of my knowledge, this is the first paper that overcomes the hard computation of the hypervolume as loss to train neural networks to produce continuous Pareto frontier approximation. \nThe hypervolume computation is very expensive for problems with 3 objectives or more, and is non-differentiable. \nThe authors propose an approximation that overcomes these limitations and achieves good results in the experiments. \n\nThe authors also provide proofs in the appendix and acknowledge limitations of their approach, which I highly appreciate.\n\nWEAKNESSES:\nThe authors missed some fundamental related work that is highly relevant to their method.\n\n- The idea of learning a continuous Pareto frontier was first proposed --even though without neural network-- by Pirotta et al., \"Multi-objective reinforcement learning with continuous pareto frontier approximation\". \n- Proposition 1 is equivalent to say that the Pareto frontier achieves the highest hypervolume. This was already shown by [17], so a reference should be added. \n- The problem formulated in Eq. (3) is a special case of the more general version by Parisi et al., Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation (Eq. (2)). In your case, you are using the hypervolume as indicator to be maximized. \n- It would be nice to stress how hard the computation of the exact hypervolume is. In particular, Friedrich et al., \"Friedrich, T., Horoba, C., & Neumann, F. (2009). Multiplicative approximations and the hypervolume indicator\" showed that it is a #P-hard problem and proposed an approximation.\n\nQUESTIONS:\nI understand that the main contribution of the paper are theoretical, but could you run some experiments with more than 3 objectives? The MO-LQR can be easily customized to have N objectives, and it would be interesting to have a short experiment to investigate the computational and time complexity (as well as the other metrics like hypervolume and sparsity) of all algorithms (yours and baselines) on varying the number of objectives (2, 3, 4, 5).\n\nLIMITATIONS:\nThe authors discuss the limitation of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to multiobjective algorithms that allows for the direct modeling of the entire Pareto set. The authors present a novel approach to Pareto set learning (PSL) from a geometric perspective, distinguishing it from existing methods that treat all preferences equally, resulting in a mere partial Pareto front. The contributions of this paper are as follows:\n* Introducing a connection between preferences and their corresponding Pareto solutions, enabling the learning of the complete Pareto front.\n* Proposing a novel geometric perspective for PSL, demonstrating the equivalence of Pareto set learning to Hypervolume maximization.\n* Utilizing a neural network model to effectively approximate the entire Pareto set.\n* The experimental results validate the superiority of the proposed method over baseline approaches.\n\nSTRENGTHS:\n* The paper presents a novel approach to multiobjective algorithms that allows for the direct modeling of the entire Pareto set.\n* The authors establish an equivalence between learning the complete Pareto set and maximizing the associated hypervolume, which enables the convergence analysis of hypervolume for Pareto set learning.\n* The theoretical foundation of the proposed method is well established, especially the derivation of the generalization gap between estimated and true hypervolumes.\n* The proposed approach is evaluated on various benchmark and real-world problems, and compared to multiple state-of-the-art algorithms, and the results are promising.\n\nWEAKNESSES:\n* There is limited discussion regarding the design of the neural network structure, and the Machine Learning perspective could benefit from further elaboration.\n* The discussion about the construction of the training data is lacking, which is crucial to understanding the similarities and differences between the training dataset and benchmark problems.\n* The evaluation of the proposed model's PSL ability is solely compared with other PSL methods. It would be beneficial to also compare its optimization performance with other traditional methods, such as decomposition-based and hypervolume-based approaches.\n* The focus of the tested problems is primarily on 2 or 3 objective optimization problems. However, it remains unclear if the proposed method can be generalized to handle more objectives.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}