SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section we show that the proposed hypervolume metric can fit into the regular neural network studies and similarly enjoy the statistical guarantees on error  $\left|\overline{\mathcal{H}_r}(\beta) - \mathcal{H}_r(\mathcal{T})\right|$  for PSL-HV1. Specifically, the term can be interrupted as the difference between the metric of  $x_{\hat{\beta}}$  computed and the metric of ground truth  $x^*$ , indicating the quality of the MLP map  $x_{\hat{\beta}}$ . We will shortly show the error above will converge with regular SGD optimizers and large enough sample size  $N$ , as expected.
We can first decompose the aforementioned error as follows,
$$
\begin{array}{l} \left| \overline {{\mathcal {H}}} _ {r} (\hat {\beta}) - \mathcal {H} _ {r} (\mathcal {T}) \right| \leq \underbrace {\left| \overline {{\mathcal {H}}} _ {r} (\hat {\beta}) - \widehat {\overline {{\mathcal {H}}}} _ {r} (\hat {\beta}) \right|} _ {\varepsilon_ {1}} + \underbrace {\left| \widehat {\overline {{\mathcal {H}}}} _ {r} (\hat {\beta}) - \widehat {\overline {{\mathcal {H}}}} _ {r} (\beta^ {*}) \right|} _ {\varepsilon_ {2}} + \\ \underbrace {\left| \widehat {\overline {{\mathcal {H}}}} _ {r} \left(\beta^ {*}\right) - \overline {{\mathcal {H}}} _ {r} \left(\beta^ {*}\right) \right|} _ {\varepsilon_ {3}} + \underbrace {\left| \overline {{\mathcal {H}}} _ {r} \left(\beta^ {*}\right) - \mathcal {H} _ {r} (\mathcal {T}) \right|} _ {\varepsilon_ {4}}, \tag {9} \\ \end{array}
$$
in which we define  $\widehat{\mathcal{H}}_r(\beta) = \frac{c_m}{N}\sum_{i=1}^N [\rho(x_\beta(\theta^{(i)}),\theta^{(i)})^m]$  as the empirical hypervolume estimation associated with parameter  $\beta$ .  $\hat{\beta}^*$  is a set of reference parameters for the MLP map  $x_\beta$ . Those errors respectively correspond to generalization error  $(\varepsilon_1)$ , optimization error  $(\varepsilon_2)$ , generalization error  $(\varepsilon_3)$ , and approximation error  $(\varepsilon_4)$ . We can conceptually set the reference parameter  $\beta^*$  as a good local maxima (in the local region of  $\hat{\beta}$ ) for hypervolume maximization so that the approximation error  $\varepsilon_4$  is small due to the universal approximation character of the MLP model family [41]. In our analysis, we assume that regular SGD technique brings  $\hat{\beta}$  close enough to the local maxima  $\beta^*$ , resulting in a small optimization error  $\varepsilon_2$ .
The rest steps to show convergence is to control the generalization error  $\varepsilon_{1}$  and  $\varepsilon_{3}$ . We present the following theorem to give the uniform convergence rate of the generalization error, which applies to both  $\varepsilon_{1}$  and  $\varepsilon_{3}$ .
Theorem 1 (Generalization error). Let the object functions  $f_{i}$ 's all be  $L_{f}$ -Lipschitz, and let  $x_{\beta}$  be an  $L$ -layer  $MLP \sigma_{L}(W_{L}\sigma_{L - 1}(\dots \sigma_{1}(W_{1}\theta)\dots))$ , with 1-Lipschitz positive homogeneous activation  $\sigma_{i}$ 's and  $\| W_{i}\|_{\mathrm{F}}\leq B_{w},\forall i\in [L]$  (under this setting  $\beta$  is the set of  $W_{i}$ 's). The design matrix  $X_{\theta}$  denotes the  $N$  collected samples  $\theta_{i}$ 's. We further set the reference point  $r$  so that  $r_i - f_i(x)\in [b,B],\forall i,$  for  $x\in \mathcal{T}\cup f\circ x_{\beta^*}(\Theta)\cup f\circ x_{\hat{\beta}}(\Theta)\subset \mathbb{R}^n$ . With probability at least  $1 - \delta (\delta < \frac{1}{2})$ , for  $\beta = \beta^{*}$  or  $\hat{\beta}$ , we have
$$
\left| \overline {{\mathcal {H}}} _ {r} (\beta) - \widehat {\overline {{\mathcal {H}}}} _ {r} (\beta) \right| \leq c _ {m} B ^ {m} m ^ {\frac {m}{2}} \left(\frac {2 \sqrt {2} m n}{N b} L _ {f} B _ {w} ^ {L} \| X _ {\theta} \| _ {\mathrm {F}} \left(1 + \sqrt {2 L \ln (2)}\right) + 3 \sqrt {\frac {\ln (4 / \delta)}{2 N}}\right).
$$
Remark 3. The results above imply the generalization errors  $\varepsilon_{1}$  and  $\varepsilon_{3}$  can converge at the rate of  $1 / \sqrt{N}$ , considering for each row in  $X_{\theta}$  its  $L^2$ -norm is bounded from above by  $\sqrt{(m - 1)\pi^2 / 4}$ .
Obtaining the results require adaptations of classical Rademacher complexity techniques. The difficulty comes from the minimum form of  $\rho(x, \theta) = \min_{i \in [m]} \left\{ \frac{r_i - f_i(x)}{\lambda_i(\theta)} \right\}$ , which differs from usual loss functions. We provide the detailed derivation in Appendix B.6. We further remark the scale assumption on  $r_i - f_i(x)$  is mild.  $f \circ x_{\beta^*}(\Theta)$  and  $f \circ x_{\hat{\beta}}(\Theta)$  are supposed to approach  $\mathcal{T}$ , the Pareto Front, under proper optimization procedures; we are thus able to select a reference point  $r$  far away enough from the set  $\mathcal{T} \cup f \circ x_{\beta^*}(\Theta) \cup f \circ x_{\hat{\beta}}(\Theta)$  to make the assumption hold.
Testing problems. This section demonstrates that our method can generate high-quality continuous Pareto solutions by the Pareto neural model for multiobjective synthetic, design, and control
problems. We evaluated our approach's performance against established methods using well-known benchmark problems such as ZDT1-2  $(m = 2)$  [42], VLMOP1-2  $(m = 2)$  [38], and real-world design problems like Four Bar Truss Design (RE21,  $m = 2$ ), Hatch Cover Design (RE24,  $m = 2$ ), and Rocket Injector Design  $(m = 3)$  [43] as well as MO-LQR  $(m = 2,3)$  [44]. MO-LQRs serve as multiobjective reinforcement learning problems. For simplicity, except MO-LQR, all objectives are normalized to [0,1]. Problem details are described in Appendix A.3.
Neural model architecture and feasibility guarantees. We employed a 4-layer fully connected neural network, similar to [37], to construct our Pareto neural model  $x_{\beta}(\cdot)$ . The network is optimized using Stochastic Gradient Descent (SGD) with a batch size of  $256^4$ . For constrained problems, to conform to the lower bound ( $l$ ) and upper bound ( $u$ ) constraints, a sigmoid activation function mapped the previous layer's output to these boundaries. For unconstrained problems, the output solution was derived from a linear combination of  $x_{\mathrm{mid}}$ . Further architecture details are available in Appendix A.2.
Baseline methods. Our approaches are compared to existing hypervolume-based multiobjective evolutionary algorithms (MOEAs) SMS-EMOA [14] and various Pareto set learning (PSL) methods. Before our approach was developed, three PSL approaches were developed, namely, EPO-based and Tchebycheff-based. Baseline methods are listed as follows,
1. One hypervolume-based multiobjective evolutionary algorithm: SMS-MOEA [14], which uses evolutionary algorithms to maximize the hypervolume of a finite set of solutions.
2. The LS-based PSL [12], optimizing  $\min_{\beta} \tilde{\mathbb{E}}_{\lambda \sim \mathrm{Unif}(\Delta^{m-1})} [\sum_{i=1}^{m} \lambda_i f_i(x_\beta(\theta))]$ , where  $\Delta^{m-1}$  is the  $(m-1)$ -simplex. Since linear scalarization can only find the convex part of the Pareto front [39], theoretically, this method cannot recover the full Pareto front.
3. The EPO-based PSL [12], optimizing  $\min_{\beta} \mathbb{E}_{\lambda \sim \mathrm{Unif}(\Delta^{m-1})}[\mathrm{EPO}(f(x_{\beta}(\theta)), \lambda)]$ , the expectation of EPO loss [3]. The EPO-based PSL is slow since each solving for each EPO subproblem, which involves an expensive optimization problem.
4. The Tchebycheff-based PSL [23, 37], optimizing  $\min_{\beta} \mathbb{E}_{\lambda \sim \mathrm{Unif}(\Delta^{m-1})} [\max_{i \in [m]} \lambda_i f_i(x_\beta(\theta))]$ .
(a) HV1
(b) HV2
(c) Tche
(d) EPO
(e) LS
(f) SMS-EMOA
Figure 6: Result comparisons on VLMOP2.
(c) Tche
Figure 7: Result comparisons on the Four Bar Truss Design problem.
(b) HV2
Figure 8: Result comparisons on the Rocket Injector Design problem.
(Metrics.) To evaluate our learned Pareto solutions, we use three indicators. These are: (1) the Hypervolume (HV), which measures the covered objective space, where a high value indicates greater
4Batch size is 128 for EPO-based PSL on LQRs since EPO is extremely slow.
Table 1: The training/prediction time of PSL-HV, and running time of SMS-EMOA in Rocket Injector Design.
SMS-EMOA	MOEA/D	Ours (Train)	Ours (Predict)
Time	6.2m	1.5m	≈ 40s	<1s
coverage; (2) the Range indicator, which measures the extent of the Pareto front learned by the model, where a larger Range is preferred; and (3) the Sparsity metric, which evaluates the degree of sparsity in the learned Pareto front, where a lower sparsity value indicates a denser Pareto front. See Appendix A.1 for formal metric definitions.
Key findings from experiments are summarized as follows.
Table 2: PSL results on all problems. For PSL-HV1,  $r = 0.6 + y^{\mathrm{adir}}$ . For Tche-based PSL, the reference point  $z = 1.1 \cdot y^{\mathrm{ideal}}$ , as claimed in their original paper.
ZDT1	ZDT2	VLMOP1
Method	HV↑	Range↑	Sparsity↓	Time(s)↓	HV	Range	Sparsity	Time(s)	HV	Range	Sparsity	Time(s)
PSL-EPO	10.82	0.94	0.14	90.7	11.21	1.43	0.86	91.1	11.55	1.18	0.11	90.79
PSL-LS	11.89	1.56	1.07	9.17	8.75	0.0	0.0	9.03	12.05	1.56	0.5	9.15
PSL-Tche	11.49	1.37	0.33	21.11	11.19	1.42	0.66	21.65	11.83	1.42	0.22	21.61
PSL-HV1	11.79	1.52	0.72	31.79	11.3	1.46	0.73	36.68	12.07	1.57	0.84	32.24
PSL-HV2	11.77	1.52	0.7	25.71	11.51	1.55	1.0	25.8	12.02	1.54	0.46	25.45
VLMOP2	Four Bar Truss Design	Hatch Cover Design
PSL-EPO	11.37	1.49	0.63	90.95	11.2	1.18	0.21	255.44	11.96	1.16	0.07	649.12
PSL-LS	11.51	1.57	7.22	9.57	11.89	1.56	0.94	41.75	12.14	1.5	0.27	113.52
PSL-Tche	11.47	1.52	0.63	22.14	11.5	1.37	0.31	55.37	12.06	1.37	0.08	125.43
PSL-HV1	11.57	1.57	0.98	32.52	11.9	1.56	0.81	66.6	12.18	1.53	0.7	131.36
PSL-HV2	11.56	1.56	0.66	25.78	11.82	1.53	0.65	60.03	12.15	1.51	0.37	130.4
MO-LQR2	Rocket Injector Design	MO-LQR3
PSL-EPO	3.42	0.3	0.23	546.81	37.23	0.68	0.81	133.03	2.41	0.31	2.33	1149.4
PSL-LS	3.59	0.5	0.84	296.98	35.55	0.19	0.88	21.35	2.58	0.37	1.9	408.59
PSL-Tche	3.52	0.39	0.44	314.75	37.73	0.19	1.34	34.01	2.54	0.38	2.19	424.15
PSL-HV1	3.61	0.73	3.62	310.01	40.84	0.7	3.31	39.09	2.62	0.59	14.7	430.72
PSL-HV2	3.61	0.59	2.19	312.95	40.94	0.7	5.54	36.63	2.6	0.55	8.93	424.51
(Comparison with SMS-EMOA.) PSL and HV-based evolutionary algorithms serve as different purposes. PSL is designed to find an infinite set of solutions using a neural network model, while EMOAs aim to find a finite solution set. This difference becomes evident in three-objective problems, as shown in Figure 8, where 100 evenly distributed solutions by traditional MOEAs fall short of approximating the full Pareto set/front. PSL requires only several seconds of training time and can quickly predict highly approximated Pareto solutions for other unknown preferences. In contrast, SMS-EMOA typically takes a much longer time, especially for objectives larger than three. The advantage of evolution-based SMS-EMOA is its ability to find global optimal solutions, while current PSL models still struggle with handling local optimas.
Table 2 reports the mean results for various PSL approaches under five random seeds. The standard deviation values can be found in Appendix A.4. All experiments were conducted with 1000 iterations for PSL-HV1, PSL-HV2, LS-based PSL, and Tchebycheff-based PSL, while EPO-based PSL is limited to 100 iterations due to time limitation.
(2) (Comparison with LS-based PSL.) It is well-recognized that LS-based PSL [12] recovers only the convex part of a Pareto front [39], resulting in finding a small portion of Pareto solutions for concave problems. For example, in the case of ZDT2, LS-based PSL could only find two extreme points. We would like to mention another less-discussed limitation of linear scalarization: the nonuniform distribution of Pareto objectives under uniformly distributed preferences. This issue is evident in problems like ZDT1 and VLMOP2 where Pareto objectives are densely distributed at the front's margins, but fewer objectives are distributed around the center of the front.
(3) (Comparison with EPO-based PSL.) As mentioned in Remark 1, both EPO and our proposed method (under mild conditions) can find the exact Pareto objective aligned with the preference vector. However, our proposed approach is approximately  $40+$  times faster than EPO-based PSL. Unlike the EPO-based PSL method, our approach only requires calculating the gradient of one objective function, avoiding the need to compute gradients for all objectives and solve complex optimization problems as described in their work [3]. This efficiency advantage is particularly evident in
MO-LQR problems, where EPO-based PSL takes significantly more time to execute. Furthermore, we have observed that EPO-based PSL does not perform well on tasks ZDT1 and Rocket Injector Design.
(4) (Balancing range and sparsity of a PF.) When simply running PSL, we observed a tendency for the Pareto objectives to concentrate on the central portion of the Pareto front, neglecting solutions that correspond to boundary preferences, such as  $f = [0,1] / [1,0]$  for bi-objective problems. To address this observation, a simple technique is to assign higher weights to the boundary objectives. For PSL-HV1, a larger reference point  $r$  is suggested to be used in comparison to the nadir point, and therefore more coordinates will correspond to boundary Pareto objectives (refer to Remark 4 in Appendix B.4). However, setting the reference point to be excessively large can result in a sparse learned Pareto front; we empirically set this offset to be 0.6 (the empirical study is in Figure 10 and 11). In PSL-HV2, the preference  $\lambda$  is placed in the denominator, resulting in greater weighting for the boundary Pareto objectives when the preference value approaches zero.
PSL-HV2 is related to the modified Tchebycheff (mTche) scalarization function [45], but the latter has different sampling techniques and lacks a hypervolume interpretation. When  $\lambda$  approaches [0,1] or [1,0], both mTche and PSL-HV2 faces numerical challenges resulting in a large gradient, which can be effectively addressed the gradient clipping technique [46, 47] with a clipping norm of a certain value (e.g., 4.0 is used). Figure 9 shows that the learning process is unstable without gradient clipping but is stabilized when gradient clipping is applied. In contrast, PSL-HV1 involves minimization rather than maximization in its inner optimization problem and does not suffer from gradient explosion.
7 Conclusion, Limitation, and Future Work
This paper presented a method to learn the set of all Pareto solutions through hypervolume maximization. The method was motivated by a previous lemma, which allows for hypervolume estimation of infinite solutions. We further gave a precise geometric perspective on Pareto set learning; we studied the generalization gap between estimated and true hypervolumes, and discussed the key techniques to train a Pareto neural model that outperformed previous methods. The proposed method will also be incorporated into the Moon library [48].
(a) With gradient clipping.
(b) W.o. gradient clipping.
Figure 9: Learning curves of PSL-HV2 on VLMOP2.
Limitation and future works. The acknowledged constraint of our work is the reliance on gradient-based methods, which can result in locally optimal solutions when the objectives  $f_{i}$ 's are non-convex. To overcome this issue and provide more solid statistical guarantee, one feasible direction is to explore classical nonparametric techniques that can improve the robustness of the method. Another way is to use some evolutionary methods such as [49, 50, 51, 52] to skip local optimal solutions. Additionally, we plan to apply the more robust Pareto neural models to large-scale multi-objective problems, such as molecular design and deep reinforcement learning.
This paper is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China [GRF Project No: CityU 11215622 ] and by Natural Science Foundation of China [Project No: 62276223]. In the last, we sincerely appreciate the helpful insights and feedbacks provided by Liang Zhao;Dake Bu;Yingying Yu (CityU Hong Kong) and Dr. Jingda Deng (XJTU).
[47] Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[48] Xiaoyuan Zhang, Ji Cheng, Liang Zhao, Song Lai, Cheng Gong, Weiduo Liao, Longcan Chen, Xi Lin, Ke Shang, Zhichao Lu, Yifan Chen, Tao Qin, and Qingfu Zhang. Moon: A standardized/flexible framework for multiobjective optimization. 01 2024.
[49] Hongzong Li and Jun Wang. From soft clustering to hard clustering: A collaborative annealing fuzzy  $c$ -means algorithm. IEEE Transactions on Fuzzy Systems, 2023.
[50] Hongzong Li and Jun Wang. Capacitated clustering via majorization-minimization and collaborative neurodynamic optimization. IEEE Transactions on Neural Networks and Learning Systems, 2022. in press.
[51] Charu Aggarwal and Karthik Subbian. Evolutionary network analysis: A survey. ACM Computing Surveys (CSUR), 47(1):1-36, 2014.
[52] Xin Yao and Yong Liu. A new evolutionary system for evolving artificial neural networks. IEEE transactions on neural networks, 8(3):694-713, 1997.
[53] Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10607-10616. PMLR, 2020.
[54] Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
[55] Matus Telgarsky. Deep learning theory lecture notes, 2020.
[56] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297-299. PMLR, 2018.
[57] Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3-17. Springer, 2016.
A.1 Metrics
We assess the performance of a Pareto neural model, denoted as  $x_{\beta}(\cdot)$ , by examining its output with the input of  $\widehat{\Theta}$ . Here,  $\widehat{\Theta}$  represents a set of  $N$  uniformly discrete angles. The resulting output solution set is denoted as  $A$  and can be expressed as  $A = \{y^{(1)}, \ldots, y^{(N)}\}$ , where  $A = f \circ x_{\beta}(\widehat{\Theta})$ . The three metrics are:
1. The Hypervolume indicator [29], which measures both the diversity and convergence of  $A$ ;
2. The Range indicator, which measure the angular span of  $A$ ;
3. The Sparsity indicator [53], which measures the average distances between neighborhood points.
(The Hypervolume Indicator.) The hypervolume indicator [29] measuring  $A$  is standard, which has been defined in the main paper,
$$
\mathcal {H} _ {r} (A) = \Lambda (\{q \mid \exists p \in A: p \preceq q \mathrm {a n d} q \preceq r \}),
$$
where  $r$  is a reference vector,  $r \succeq y^{\mathrm{adir}}$ . For bi-objective problems, the reference point  $r$  is set to [3.5, 3.5], whereas for three-objective problems, the reference point is set to [3.5, 3.5, 3.5].
(The Range Indicator.) The range indicator of a set of solutions  $A: A = \{y^{(1)}, \ldots, y^{(N)}\}$  measures the angular span of  $A$  in their polar coordinates. Let  $(\rho^{(i)}, \theta^{(i)})$  be the polar coordinate of objective vectors  $y^{(i)}$  with a reference point  $r$ . The relationship between the Cartesian and polar coordinates is,
$$
\left\{ \begin{array}{l} y _ {1} = r _ {1} - \rho \sin \theta_ {1} \sin \theta_ {2} \dots \sin \theta_ {m - 1} \\ y _ {2} = r _ {2} - \rho \sin \theta_ {1} \sin \theta_ {2} \dots \cos \theta_ {m - 1} \\ \dots \\ y _ {m} = r _ {m} - \rho \cos \theta_ {1}. \end{array} \right. \tag {10}
$$
The Range indicator can be defined as the minimal angle span among all angles.
$$
\operatorname {R a n g e} (A) = \min  _ {i \in [ m - 1 ]} \max  _ {1 \leq u <   v \leq N} \left\{\left| \theta_ {i} ^ {(u)} - \theta_ {i} ^ {(v)} \right| \right\}. \tag {11}
$$
(The Sparsity Indicator.) The sparsity indicator first introduced in [53]. A small sparsity indicator is preferred, implying that the learned objective set is dense.
$$
\operatorname {S p a r s i t y} (A) = \frac {1}{N - 1} \sum_ {j = 1} ^ {m} \sum_ {i = 1} ^ {N - 1} \left(\tilde {y} _ {j} ^ {(i)} - \tilde {y} _ {j} ^ {(i + 1)}\right) ^ {2} \tag {12}
$$
where  $\tilde{y}_j^{(i)}$  is the i-th solution, and the j-th objective values in the sorted list by the non-dominating sorting algorithm [6]. For a better illustration, the unit of the Sparsity indicator is  $10^{-3}$  for bi-objective problems and  $10^{-7}$  for three objective problems.
A.2 Neural Model Architecture and Feasibility Guarantees
We use a 4-layer fully connected neural network similar to [37] for the Pareto neural model  $x_{\beta}(\cdot)$ . We optimize the network using Stochastic Gradient Descent (SGD) optimizer with a batch size of 256. The first three layers are,
$$
\begin{array}{l} x _ {\beta} (\cdot): \theta \rightarrow \operatorname {L i n e a r} (m - 1, 6 4) \rightarrow \operatorname {R e L U} \\ \rightarrow \operatorname {L i n e a r} (6 4, 6 4) \rightarrow \operatorname {R e L U} \tag {13} \\ \rightarrow \operatorname {L i n e a r} (6 4, 6 4) \rightarrow \operatorname {R e L U} \rightarrow x _ {\mathrm {m i d}}. \\ \end{array}
$$
For constrained problems, to satisfy the constraint that the solution  $x_{\beta}(\lambda)$  must fall within the lower bound  $(l)$  and upper bound  $(u)$ , a sigmoid activation function is used to map the previous layer's output to these boundaries,
$$
\begin{array}{l} x _ {\text {m i d}} \rightarrow \operatorname {L i n e a r} (6 4, n) \rightarrow \text {S i g m o i d} \\ \rightarrow \odot (u - l) + l \rightarrow \operatorname {O u t p u t} x _ {\beta} (\lambda). \tag {14} \\ \end{array}
$$
For unconstrained problems, the output solution is obtained through a linear combination of  $x_{\mathrm{mid}}$
$$
x _ {\text {m i d}} \rightarrow \operatorname {L i n e a r} (6 4, n) \rightarrow \operatorname {O u t p u t} x _ {\beta} (\lambda). \tag {15}
$$
A.3 Benchmark Multiobjective Problems
Standard Multiobjective Optimization (MOO) problems. ZDT1-2 [42] and VLMOP1-2 [38] are widely recognized as standard multi-objective optimization (MOO) problems and are commonly employed in gradient-based MOO methods. ZDT1 exhibits a convex Pareto front described by  $(y_{2} = 1 - \sqrt{y_{1}}, 0 \leq y_{1} \leq 1)$ . ZDT2 presents a non-convex Pareto front defined by  $(y_{2} = 1 - y_{1}^{2}, 0 \leq y_{1} \leq 1)$ , and the LS-based PSL approach can only capture a small number of Pareto solutions.
Real world designing problem. Three real-world design problems with multi-objective optimization are the Four Bar Truss Design (RE21), Hatch Cover Design (RE24), and Rocket Injector Design (RE37). In order to simplify the optimization process, the objectives have been scaled to a range of zero to one.
Multiobjective Linear Quadratic Regulator. The Multiobjective Linear Quadratic Regulator (MO-LQR) problem is first introduced in [44]. MO-LQR is regarded as a specialized form of multi-objective reinforcement learning, where the problem is defined by a set of dynamics presented through the following equations:
$$
\left\{ \begin{array}{l} s _ {t + 1} = A s _ {t} + B a _ {t} \\ a _ {t} \sim \mathcal {N} \left(K _ {\mathrm {L Q R}} \cdot s _ {t}, \Sigma\right). \end{array} \right. \tag {16}
$$
In accordance with the settings discussed in the aforementioned work by Parisi et al. [44],  $A$ ,  $B$ , and  $\Sigma$  are adopted as identity matrices. The initial state for the bi-objective problem is set to  $s_0 = [10, 10]$ , whereas for the three-objective problem, it is set to  $s_0 = [10, 10, 10]$ . The reward function is defined as  $r_i(s_t, a_t)$ , where  $i$  represents the respective objective. The function is formulated as follows: