LORA-MOO: Learning Ordinal Relations and Angles for Expensive Many-Objective Optimization

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:36:36

Abstract: Many-objective optimization (MOO) simultaneously optimizes many conflicting objectives to identify the Pareto front - a set of diverse solutions that represent different optimal balances between conflicting objectives. For expensive MOO problems, due to their costly function evaluations, computationally cheap surrogates have been widely used in MOO to save evaluation budget. However, as the number of objectives increases, the cost of learning and surrogation, as well as the difficulty of maintaining solution diversity, increases rapidly. In this paper, we propose LORA-MOO, a surrogate-assisted MOO algorithm that learns surrogates from spherical coordinates. This includes an ordinal-regression-based surrogate for convergence and $M-1$ regression-based surrogates for diversity. $M$ is the number of objectives. Such a surrogate modeling method makes it possible to use a single ordinal surrogate to do the surrogate-assisted search, and the remaining surrogates are used to select solution for expensive evaluations, which enhances the optimization efficiency. The ordinal regression surrogate is developed to predict ordinal relation values as radial coordinates, estimating how desirable the candidate solutions are in terms of convergence. The solution diversity is maintained via angles between solutions, which is a parameter-free. Experimental results show that LORA-MOO significantly outperforms other surrogate-assisted MOO methods on most MOO benchmark problems and real-world applications.

Main paper: 
 introduction: 
 Many-objective optimization problems (MOOPs) are widely exist in many real-world applications, such as production scheduling [26] , traffic signal control [33] , and water resource engineering [21] .These MOOPs have conflicting objectives to optimize, and thus all objectives cannot reach their optimum simultaneously. As a result the optimum of MOOPs is the Pareto front (PF): A set of non-dominated solutions that represent different optimal balance between conflicting objectives.Multi-/many-objective optimization (MOO) 1 aims to find non-dominated solutions that are close to the PF and also well distributed along the PF, indicating that MOO should consider both convergence and diversity.Various evolutionary optimization algorithms have been proposed to solve MOOPs [10] . These optimization algorithms usually require plenty of solution samplings and evaluations to find converged and diverse non-dominated solutions. However, in many real-world MOOPs, the evaluation of solution performance could be expensive [41] . In these expensive MOOPs, the evaluation budget only allows a limited number of solutions to be evaluated on the expensive objective functions. To address expensive MOOPs, evolutionary optimization algorithms are combined with computationally cheap surrogates to enhance sampling efficiency and save evaluations, which are known as surrogate-assisted evolutionary algorithms (SAEAs).Yet, it is a perennial challenge to use surrogates in a more effective and efficient way for SAEAs, especially when optimization problems have many objectives. For example, conventional SAEAs usually use regression-based surrogates to approximate each objective function separately [5, 34] .For MOOPs, many objectives indicate maintaining many surrogates for surrogate-assisted search and selection, which results in a low efficiency of SAEAs. In addition, it is difficult to maintain solution diversity in high-dimensional objective space. Some SAEAs [24, 43, 5] need to investigate proper parametric strategies to generate reference vectors or divide objective space into subspaces. Recently, a family of classification-based SAEAs [31, 17] attempted to use a single surrogate to learn pairwise dominance relations. However, the training with pairwise relations implies an exponential increase in the size of training dataset. Therefore, a natural question is that whether we can reduce the cost of maintaining many surrogates without increasing the cost of training a single surrogate. Furthermore, whether we can use an non-parametric diversity maintenance strategy to handle the objective space of MOOPs, instead of designing complex reference vectors or points?In this paper, we propose a different way to implement surrogate-assisted evolutionary optimization for expensive MOOPs, named LORA-MOO, where a single surrogate is developed to learn ordinal relations for convergence purpose, and several angular surrogates are generated from spherical coordinates to maintain diversity. Our major contributions are summarized as follows:• We develop a novel ordinal-regression-based model to approximate the ordinal landscape of expensive MOOPs. Our ordinal surrogate is able to handle many objectives simultaneously and assist MOO algorithms to complete the model-based search. Artificial ordinal relations are generated via a clustering method to improve the learning quality of ordinal relations for many objectives. Unlike the pairwise relations learned through classification, the ordinal relations would not increase the size of training dataset, hence high efficiency.• We introduce the idea of spherical coordinates approximation into surrogate-assisted evolutionary optimization and proposed LORA-MOO to solve expensive MOOPs. Different from existing SAEAs which learn approximation models from Cartesian coordinates, we fit several regression-based surrogates to approximate angular coordinates, while our ordinal surrogate can be treated as a radial coordinate. An non-parametric approach is developed to select diverse solutions for expensive evaluations via our angular coordinate surrogates.• Extensive experiments on benchmark and real-world optimization problems are conducted under a range of scales and numbers of objectives. Empirical results show that our LORA-MOO is effective. It is able to obtain a well-distributed solution set that outperforms the state-of-the-arts.2 Related Work 2.1 Multi-/Many-Objective Surrogate-Assisted Evolutionary Algorithms Regression-based SAEAs. Regression-based SAEAs employ regression-based surrogates such as Kriging [36, 39] to approximate either the objective values of solutions or the objective functions of expensive problems [22] . To maintain solution diversity, ParEGO [24] employs a Kriging model to iteratively approximate an aggregate objective function which aggregates all objectives into one via a set of pre-defined scale vectors. In MOEA/D-EGO [43] , plenty of scale vectors are generated uniformly to decompose the target MOOP into many single-objective subproblems. K-RVEA [5] also designs a set of scale vectors as reference vectors to maintain solution diversity. Similarity or density estimation is an alternative option for maintaining diversity. For instance, KTA2 [34] estimates the distribution status of non-dominated solutions by defining a similarity or density indicator.Classification-based SAEAs. In model-based optimization, the optimization is guided by the relation between solutions rather than accurate objective values. Therefore, there is a tendency for recently proposed SAEAs to use classification-based surrogates to learn the relation between solutions directly. CSEA [31] trains a neural network to justify whether candidate solutions can be dominated by given reference points or not. θ-DEA-DP [42] uses two neural networks to predict the Pareto dominance relation and θ-dominance relation between two solutions, respectively. REMO [17] employs a neural network to fit a ternary classifier, which is able to learn the dominance relation between pairs of solutions. Compared with regression-based SAEAs, although classification-based SAEAs take advantage of learning solution relations directly, their drawbacks are also clear: The prediction of solution relations lacks the information of how solutions are distributed in the objective space, making it difficult for classification-based SAEAs to maintain solution diversity. In [31, 17] , a radial projection selection approach is adapted to select diverse reference points. However, its effect on diversity maintenance is limited. In addition, although classification-based SAEAs maintain only one surrogate, the cost of learning pairwise relations from large datasets is inevitably increased.SAEAs based on Other Surrogates. HSMEA [15] uses an ensemble of multiple surrogates in the optimization. In addition, a new category of surrogates, namely ordinal regression surrogate [40] or level-based classification surrogate [28] , is proposed recently to combine regression-based surrogates with classification-based surrogates. However, the shortcoming remains the same as these surrogates lack the information of solution distribution, especially when the number of objectives is large. 
multi-objective bayesian optimization: 
 MOBO. Bayesian Optimization (BO) [35, 18] is also a typical model-based optimization method for expensive optimization, while multi-objective BO (MOBO) methods are designed for expensive MOOPs [7, 8, 27, 1] . Some MOBO generalizes the acquisition functions such as upper confidence bound (UCB) [46] , expected improvement (EI) [14] , Thompson sampling [3] , to solve expensive MOOPs. In addition, entropy search methods have also been employed in MOBO [2, 37] . To maintain solution diversity, the EI of a multi-objective performance indicator, Hypervolume (HV) [45] , was used as the acquisition function in recent MOBO [6, 27] . Based on the Hypervolume improvement (HVI), PSL [27] proposes a learning method to approximate the whole Pareto set for MOBO, and PDBO [1] automatically selects the best acquisition function for objective functions in each iteration. However, the time complexity of computing HV increases exponentially with the number of objectives, which may limit the application of MOBO methods on optimization problems with many objectives.Connection to SAEAs. Both SAEAs and MOBO are model-based optimization methods. A SAEA is also a MOBO if it uses probability models as surrogates, and a MOBO is also a SAEA if it searches candidate solutions with evolutionary search algorithms. Therefore, some model-based optimization methods belong to both SAEAs and MOBO [24, 14, 43] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: When the number of objectives is large, there would be many non-dominated solutions in the archive, however, we have introduced artificial ordinal relations in our surrogate modeling procedure to alleviate this limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
lora-moo: optimization via learning ordinal relations and angles: 
 This section first introduces the LORA-MOO framework, followed by detailed algorithm descriptions.The pseudocode of LORA-MOO is depicted in Alg. 1, it consists of four phases:1. Initialization: An initial dataset of size 11D -1 (As suggested in the literature [24] ) are sampled from the decision space using the Latin hypercube sampling (LHS) [30] (line 1), where D is the dimensionality of decision variables. The sampled solutions are evaluated on objective functions f and then saved in an archive S A (line 2).2. Surrogate modeling: For all solutions x ∈ S A , quantify their ordinal values (line 4) and calculate their angular coordinates (line 9). The set of ordinal values S o is used to train the ordinal surrogate h o (line 5). The angular coordinates are used to fit M -1 angular surrogates h ai separately (line 10). Ordinal training set S o ← Quantify ordinal values for all x i ∈ S A (Alg. 2).Ordinal surrogate h o ← Train Kriging(S A , S o ).Population of candidate solutions P ← Run an optimizer on h o (Alg. 3).x * 1 ← Use the ordinal surrogate to select a solution from P by convergence criterion.Evaluate x * 1 and updateS A = S A ∪ {(x * 1 , f (x * 1 ))}, F E = F E + 1.Angular training set S a ← Calculate angular coordinates for all x i ∈ S A .10:M -1 angular surrogates h ai ← Train Kriging (S A , S a ), i = 1, . . . , M -1.11:x * 2 ← Use angular surrogates to select a solution from P by diversity criterion (Alg. 4).12:Evaluate x * 2 and updateEQUATION))}, F E = F E + 1. 13: end while Output: Non-dominated solutions in archive S A .The ordinal surrogate h o is mainly trained on dominance-based ordinal relations, additional clusteringbased artificial ordinal relations will be introduced for training if the number of objectives M is large. In addition, for an M -objective problem, M -1 angular surrogates h ai are trained on angular coordinates. These surrogates are used in the selection procedure for solution diversity but are idle in the search procedure.In LORA-MOO, the concept of ordinal regression [40] A solution x 1 is said to λ-dominate another solution x 2 (denoted by x 1 ≺ λ x 2 ) if and only if:EQUATIONwhere λ ≥ 0 is the dominance coefficient and g λ is a smooth objective function defined as:EQUATIONEQUATIONwhere f in denotes a normalized objective function,z * = {z * 1 , . . . , z * M }, z nad = {z nad 1 , . . . , z nad M }are ideal point and nadir point for the current non-dominated solutions, respectively.More detailed definitions about the background of MOO are available in Appendix A. All non-λdominated solutions in S A are selected as reference points S RP . There are two reasons to introduce the definition of λ-dominance:• The λ-dominance can smoothen the original PF by excluding dominance resistant solutions (DRSs) [16, 38] . DRSs are solutions that are best or close to best on one or several objectives but extremely poor on at least one of the remaining objectives. Such a solution is apparently not desirable but may be regarded as one of the best solutions since there may not exist any other solutions dominating it in the solution set.• Second, λ-dominance can eliminate some similar non-dominated solutions from the Pareto set, which can be used to adjust the size of Pareto set. When the number of objectives M is large, it is possible that a majority of past evaluated samples are non-dominated to each other. To balance the number of reference points and remaining samples, we introduce the dominance coefficient λ to sightly reduce the ratio of reference points in S A . This alleviates the situation of extreme imbalance of samples in different ordinal levels (see the division of ordinal levels below).Computation of Relation Values. To quantify ordinal relation values, we first calculate extension coefficients ec(x) for each x ∈ S A . ec(x) is defined as the minimal coefficient ec ≥ 1 to make a solution x non-λ-dominated to all solutions x in the extended reference:EQUATIONAlthough extension coefficient ec(x) quantifies the distance between a solution x and reference S RP , it has not been used to train the ordinal regression-based surrogate directly. To generate a stable ordinal regression-based surrogate, solutions inS A are divided into N o = max(n o , |S A |/|S RP |)ordinal levels, where n o is a pre-defined parameter denoting the minimal number of ordinal levels.The solutions in S RP are classified into the non-dominated ordinal level, thus the relation value v 1 = 1.0 is assigned to them. Remaining solutions in S A are sorted by their extension coefficients ec(x)and then divided into N o -1 ordinal levels uniformly. The relation valuev i = 1 -i-1No-1 will be assigned to the solutions x in the i th ordinal level. Lastly, relation values serve as radial coordinates and a Kriging model is employed to approximate them.When the number of objectives M is large, most evaluated solutions in archive S A could be nondominated solutions, indicating that these solutions will be divided into the same non-dominated ordinal level and thus treated as reference points S RP . This is harmful to the ordinal surrogate modeling due to the extreme imbalance between the numbers of training samples in different ordinal levels. To reduce the ratio of S RP , we use a clustering method to generate n_clusters clusters for S RP , where n_clusters is the half of the size of S RP . All solutions x ∈ S RP are mapped to the closest cluster centers. The solutions with the shortest projection on each cluster center will be selected as the new S RP , while the remaining solutions will be moved to the next ordinal level. Such artificial ordinal relations greatly reduce the ratio of S RP in S A . In LORA-MOO, we set a ratio threshold rp_ratio for S RP , once the ratio of S RP is larger than rp_ratio, artificial ordinal relations will be generated for surrogate modeling. Details are available in Appendix C, Alg. 2 and Fig. 5 .Given a solution x ∈ S A with Cartesian coordinates (f 1 (x), . . . , f M (x)), The angular coordinates of solution x are transformed with the following rules:EQUATIONwhere z * is the ideal point. The resulting angular coordinates (ϕ 1 , . . . , ϕ M -1 ) are used to fit M -1 regression-based surrogates separately. In LORA-MOO, we use the Kriging model to approximate angular coordinates. The introduction and usage of Kriging model is given in Appendix B.In this subsection, we describe how to use surrogate h o to search for candidate solutions and how to use surrogates h o and h ai to select optimal ones from candidate solutions for expensive evaluations. Convergence Criterion for environmental selection is the expected improvement (EI) [14] of ordinal values, which is similar to many MOBO methods [24, 43] . Since the output of our ordinal surrogate h o (x) is an 1-D numerical value, the solution with maximal 1-D EI in P is selected as x * 1 .Diversity Criterion to sample x * 2 from P is defined as angles d ang between candidate solutions and reference points S RP . Firstly, the minimal degree between each candidate solution and S RP is measured. Among these minimal degrees md ang , the solution with max(md ang ) is selected as x *(Detailed pseudo-code is available in Appendix C, Alg. 4).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: Not applicable.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 To evaluate the optimization performance of LORA-MOO on expensive MOOPs, we conduct experiments to compare LORA-MOO with other SAEAs on different MOOPs, including a series of scalable multi-/many-objective benchmark optimization problems DTLZ [11] , WFG [19] , and a real-world network architecture search (NAS) problem.Optimization Problem Setup. To ensure a fair comparison, the following optimization problem setup is the same as the setup that has been widely used in the literature [5, 31, 34, 17] . In our experiments, initial datasets of size F E init = 11 D -1 are used to initialize surrogates, while the maximum number of allowed evaluations F E max is 300. The statistical results are obtained from 30 independent runs. For each run, different comparison algorithms share the same initial dataset.Comparison Algorithms. We compare LORA-MOO with 6 state-of-the-art SAEAs, some of them also known as MOBO methods. These comparison algorithms can be classified into three categories:• Regression-based MOO methods: ParEGO [24] , K-RVEA [5] , and KTA2 [34] . ParEGO is a classic regression-based SAEA and also a MOBO, which serves as a baseline. K-RVEA is a typical SAEA which uses reference vector to guide the diversity maintenance. KTA2 is a newly proposed algorithm to use an independent archive to keep solution diversity.• Classification-based MOO methods: CSEA [31] , REMO [17] . CSEA is a classic classification-based SAEA which serves as a baseline. REMO is a newly proposed SAEA which represents the state-of-the-art performance of classification-based SAEAs.• Ordinal-regression-based MOO method: OREA [40] is a new category of SAEA that is different from common regression-based and classification-based SAEAs. We compare with it since it is directly related to our radial surrogate.Note that some classic SAEAs and MOBO methods such as MOEA/D-EGO [43] and CPS-MOEA [44] are not compared in our experiments as they failed to outperform other comparison algorithms on any DTLZ problem [17] . Some HV-based MOBO methods are not compared as they are failed to solve many objectives.Parameter Setup. For the surrogate modeling, the Kriging models used in all comparison algorithms are implemented using DACE [32] , just as [24] suggested. For regression-based Kriging surrogates, the range of hyper-parameter θ ∈ [10 -5 , 100]. And for the neural networks in CSEA and REMO, the parameters are the same as suggested in the literature. In the sampling strategy, the mutation operator used to initialize candidate solutions is polynomial mutation [9] , the mutation probability p m = 1/d and mutation index η m = 20, as recommended in [34, 17] . The size of offspring population is 100.The settings of the PSO optimizer are the range of hyper-parameter in the ordinal-regression-based surrogate are the same as suggested in [40] .For the specific parameters exist in LORA-MOO, such as the dominance coefficient λ and the threshold ratio of reference points to introduce clustering-based ordinal relations rp_ratio. As there is no relevant study in the literature for their setups, we conducted ablation studies to investigate the effect of these parameters on the performance of LORA-MOO. The results are summarized in Section 4.2 and reported in Appendix F. The source code of LORA-MOO2 will be available online.Performance Indicator. To have a comprehensive estimation of optimization performance, we use three different performance indicators in our experiments: The inverted generational distance (IGD) [4] , the inverted generational distance plus (IGD+) [20] , and the Hypervolume (HV) [45] . IGD and IGD+ use a set of truth Pareto front to measure the quality of a set of non-dominated solutions in terms of convergence and diversity. A smaller IGD or IGD+ value indicates better MOO performance.HV use a reference point to calculate the area covered by a set of non-dominated solutions, a large HV value is preferable to MOO. See Appendix D for details and setups about performance indicators.We conduct ablation studies on DTLZ and WFG benchmark problems with D = 10 variables and M ={3, 6, 10} objectives. LHS [30] is used to sample initial dataset. The effects of four parameters are investigated: They are the minimal number of ordinal levels n o , the dominance coefficient λ, the ratio threshold of reference points rp_ratio, and the clustering number for reproduction n c . Three representative results obtained on the WFG5 problem with 3 and 10 objectives are depicted in Fig. 1 .Complete results and statistical analysis of ablation studies are reported in Appendix F.As shown in Fig. 1 (left), when M = 10, a large n o results in poor optimization performance. This is because the ratio of non-dominated solutions in the archive tends to be large when M is large, hence, setting a large n o will lead to a lack of training samples in each dominated ordinal levels, which is detrimental to the performance of surrogate modeling. As such, n o in LORA-MOO is set to 4.The result in Fig. 1 (middle) shows that using λ-dominance to sightly modify the original dominance relations is beneficial to the effectiveness of LORA-MOO. When λ = 0, no λ-dominance would be used and the corresponding LORA-MOO variant has the worst performance among all the variants. In addition, setting a large λ could cause severe damage to the original dominance relations. Therefore, we set λ to 0.2.The effect of introducing artificial ordinal relations via clustering is demonstrated in Fig. 1 (right) .When the ratio threshold of reference points rp_ratio is 1 and M = 10, no artificial ordinal relations are introduced to further divide ordinal levels for plenty of non-dominated solutions in the archive.Consequently, the imbalance of sample numbers in different ordinal levels leads to poor optimization performance. However, dominance relations are preferable to artificial ordinal relations when M = 3 and the size of ordinal levels are well balanced. Hence, we set rp_ratio = 0.5.The optimization performance of LORA-MOO is evaluated on DTLZ and WFG benchmark problems with D = 10 variables and M ={3, 4, 6, 8, 10} objectives. The IGD values obtained on DTLZ Further comparison is conducted on a real-world network architecture search (NAS) problem, the best three algorithms listed in Table 1 are compared: LORA-MOO, KTA2, and KRVEA. The NAS problem tested is the NASbench201 implemented in EvoXBench [29] , it has 6 variables and 5 objectives. Details of this NAS problem is provided in Appendix E. Considering NASbench201 is a real-world application and we do not know its exact PF, we use HV to evaluate optimization performance since HV can be calculated without the exact PF. In practice, log(HV diff ) is employed to amplify the visual difference of the obtained HV values: log(HV diff ) = log(HV max -HV ) where HV max is the maximal HV value on this problem that is provided in EvoXBench. We compare the runtime on benchmark problems for all the comparison algorithms to investigate the relation between their optimization efficiency and the number of objectives M .Fig. 4 illustrates how the runtime of each comparison algorithm varies as the M increases. It can be observed that the runtime of KTA2 increases exactly in the same rate as M increases. In comparison, the runtime of LORA-MOO increases slightly when M increases. This demonstrates that using angular surrogates only at the end of environmental selection process is beneficial to the optimization efficiency of LORA-MOO. In addition, the runtimes of ParEGO, CSEA, REMO, and OREA do not increase significantly with M since they do not maintain specific surrogates to manage the diversity of non-dominated solutions. Consequently, their overall performance reported in Table 1 is not desirable. Overall, LORA-MOO finds a good trade-off between optimization efficiency and optimization results. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Experimental setups are described in detail.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.•The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 In this paper, we propose an efficient MOO method, LORA-MOO, to solve expensive MOOPs.Different from existing surrogate modeling approaches, our LORA-MOO learns surrogate models from ordinal relations and spherical coordinates. Only one ordinal surrogate is used in the modelbased search, which hugely improve the efficiency of optimization. Our empirical studies have demonstrated that our LORA-MOO significantly outperforms other state-of-the-art efficient MOO methods, including SAEAs and MOBO methods. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? 
 
 Appendix: We consider minimization problems and many-objective optimization problems (MOOPs) can be formulated as follows:Definition 2. (Expensive Many-Objective Optimization Problem) Given M expensive objective functions f 1 , . . . , f M and an evaluation budget F E max , obtain the Pareto set for the following many-objective optimization problem:argmin x∈X f (x) = (f 1 (x), . . . , f M (x))where X ⊆ R D is the decision space of the problem.The Pareto set is defined through the following definitions: Pareto set and Pareto front are defined as follows:Definition 3. Pareto dominance:A solution x 1 is said to dominate another solution x 2 (denoted by x 1 ≺ x 2 ) if and only if:∀k ∈ {1, 2, . . . , M } : f k (x 1 ) ≤ f k (x 2 )∧ ∃k ∈ {1, 2, . . . , M } : f k (x 1 ) < f k (x 2 ) Definition 4.A non-dominated solution x in the decision space X is a solution that cannot be dominated by any other solutions in X:x ∈ X : x ≺ x Definition 5. Pareto set: Pareto set S ps is the set of all non-dominated solutions in the decision space X:S ps = {x ∈ X| x ∈ X : x ≺ x } Definition 6. Pareto front:Pareto front S pf is the corresponding unique set of the Pareto set in the objective space:S pf = {f (x)|x ∈ S ps }Kriging model, also known as Gaussian process model [23] or design and analysis of computer experiments (DACE) model [32] , is a stochastic process model used to approximate an unknown objective function. LORA-MOO uses Kriging models to implement angular surrogates and the radial surrogate, to avoid potential confusion and help the understanding of our algorithm, the working mechanism of the Kriging model is described below.A common way to approximate an unknown objective function with n observations is linear regression:EQUATIONwhere x i is the i th sample point observed from the objective function. f k (x i ), β k are a linear or nonlinear function of x i and its coefficient, respectively. N is the number of functions f (x). i is an independent error term, which is normally distributed with mean zero and variance σ 2 .However, a stochastic process model such as Kriging does not assume that the error terms are independent. Hence, an error term i is rewritten as (x i ). Moreover, these error terms are assumed to be related or correlated to each other. The correlation between two error terms (x i ) and (x j ) is inversely proportional to the distance between the corresponding points [23] . The correlation function in the Kriging model is defined as:EQUATIONwhere the distance between two points x i and x j are measured using the special weighted distance formula shown below:EQUATION)where D is the number of decision variables, θ θ θ ∈ R D ≥0 and p ∈ [1, 2] D are parameters of the Kriging model. It can be seen from Eq.( 7) that the correlation is ranged within (0, 1] and is increasing as the distance between two points decreases. Particularly, in Eq.( 8), the parameter θ k can be explained as the importance of the decision variable x k , and the parameter p k can be interpreted as the smoothness of the correlation function in the k th coordinate direction.Due to the effectiveness of correlation modelling, the regression model in Eq.( 6) can be simplified without degrading modelling performance [23] . Clearly, all regression terms are replaced with a constant term, thus the Kriging regression model can be rewritten as follows:EQUATIONwhere µ is the mean of this stochastic process, (x i ) ∼ N (0, σ 2 ).To train the Kriging model and estimate the parameters θ θ θ, p in Eq.( 8), the following likelihood function is maximised:EQUATIONwhere |R| is the determinant of the correlation matrix, each element in the matrix is obtained using Eq.( 7). y is the n-dimensional vector of dependent variables that observed from the objective function.The mean value µ and variance σ 2 in Eq.( 9) and Eq.( 10) can be estimated by:EQUATIONEQUATIONFor a new solution x * , the Kriging model predicts the approximation of ŷ(x * ) and the uncertainty ŝ2 (x * ) as follows:EQUATIONEQUATIONwhere r is a n-dimensional vector of correlations between (x * ) and the error terms at the training data, which can be calculated via Eq. (7) .Further details and a comprehensive description of the Kriging model and Gaussian Process can be found in [39] . In this paper, all regression-based Kriging models have θ θ θ ∈ [10 -5 , 100] D , p = 2 D .This section describes LORA-MOO with more details.In order to learn the ordinal landscape of MOOPs, we need to quantify the ordinal relations between solutions into numerical values. Alg. 2 illustrates the pseudocode of quantifying ordinal relations3 , it describes line 4 in Alg. 1 of the main file. It can be seen that Alg. 2 is mainly working on the quantification of dominance-based ordinal relations. Artificial ordinal relations will not be added unless the ratio of reference points is larger than ratio threshold rp ratio (line 5).An illustration of artificial clustering-based ordinal relations is given in Fig. 5 . By using clustering methods, artificial ordinal relations are generated for training ordinal regression surrogates. Picking one solution from each cluster ensures the diversity of non-dominated solutions in the first ordinal level L 1 . Meanwhile, the selection within each cluster is based on the projection length on cluster center, which is beneficial to the convergence of non-dominated solutions. For x in each cluster, calculate the projection length of x on the corresponding cluster center. 9:L 1 ← Solutions x with the shortest projection on each cluster. Algo. 3 gives the pseudocode of generating candidate solutions, it is the implementation of line 6 in Alg. 1 of the main file. In lines 1-9, a population P 0 is generated. Since reference points S RP are the optimal solutions in S A in terms of convergence, a half initial solutions are generated from S RP (lines 2-8). To obtain a diverse subset of S RP , LORA-MOO divides S RP into n c clusters before sampling solutions (line 2). Once population initialization is completed (line 9), a normal PSO is conducted to produce candidate solutions (lines [11] [12] [13] [14] [15] [16] . Please be noted that, although we are solving expensive MOOPs, only a single ordinal surrogate h o is used in the reproduction process (line 14). This is a great advantage of LORA-MOO since existing regression-based SAEAs involve all M surrogates in the reproduction process. Hence, LORA-MOO is more efficient than these regression-based SAEAs.Alg. 4 x j ← The j th solution in P . /* Assume the first solution in P is selected as x * 1 already. */ 4:d ang ← Calculate the angles between x j and all reference points in S RP .md ang ← The angle between x j and its nearest reference point. In our experiments, we use IGD [4] , IGD+ [20] , and HV [45] to measure the performance of many objective optimization. Both IGD and IGD+ require a subset of Pareto front as reference points. In our experiments, the number of IGD/IGD+ reference points is set to 5000 for 3-, 4-, and 6-objective optimization problems, as widely used in the literature [40] . Considering the large objective space, ) ∈ R M WFG (1,0, . . . , 1.0) ∈ R M NASBench201 (1.0, 1.0, 1.0, 1.0, 1.0)we set the number of IGD/IGD+ reference points to 10000 for 8-and 10-objective optimization problems to achieve a more accurate estimation of optimization performance. The method proposed in [25] is employed to generate well-distributed IGD/IGD+ reference points.In comparison, the calculation of HV values does not require a subset of Pareto front as reference points. For a set of non-dominated solutions, its HV is the volume in the objective space it dominates from the set to a single reference point. Table 2 lists the reference point used for calculating HV values. All HV values are calculated using the reference point and the normalized solutions. A solution x is normalize by the upper bound and lower bound of Pareto front:EQUATION)where ub pf , lb pf are the upper bound and lower bound of Pareto front, respectively.NASbench201 [12] are discrete optimization problems that aim to identify the optimal architecture for neural networks. The search space is defined by a cell with 4 nodes inside, forming a directed acyclic graph as illustrated in Fig. 6 . The decision variables are 6 edges, each edge is associated All these objectives are normalized to [0, 1] in the optimization. The optimization problem can be formulated asEQUATIONwhere decision vector x ∈ {0, 1, 2, 3, 4} 6 .In this section, we report complete results of our ablation studies that are not displayed in the main paper. We conduct four ablation studies to investigate the effect of the following four parameters on the optimization performance of LORA-MOO. 4 .The last four rows of Table 4 shows that λ = 0.2 is the optimal parameter setup for LORA-MOO.The variant of λ = 0.2 is significantly superior to both the variants of λ = 0 and λ = 0.1, and it is equivalent to the variant of λ = 0.3. We note that the variant of λ = 0.3 is also significantly superior to both the variants of λ = 0 and λ = 0.1. However, this variant wins/ties/losses 30/105/9 statistical tests in total, while the variant of λ = 0.2 wins/ties/losses 32/109/3 statistical tests in total. Therefore, setting λ = 0.2 is preferable to setting λ = 0.3.Note that all other LORA-MOO variants outperform the variant of λ = 0, this implies that excluding some samples from the set of non-dominated solutions is beneficial to the performance of ordinal regression. The effectiveness of using our λ-dominance approach in LORA-MOO is demonstrated.F.3 Influence of Ratio Threshold rp ratio .In this subsection, we investigate the influence of ratio threshold rp ratio on the optimization performance. rp ratio is the threshold to determine when to add artificial ordinal relations for the training of ordinal surrogate h o . We set rp ratio = {1, 2/3, 1/2, 1/3} to generate four LORA-MOO variants.For all variants, we set n o , λ to 4, 0.2, respectively, which are consistent with our conclusions in previous ablation studies. Parameter n c is tentatively set to 5. The IGD+ values obtained by four LORA-MOO variants with different rp ratio are reported in Table 5 . It should be noted that, when the number of objectives M = 3, the results of rp ratio = 1 are the same as the results of rp ratio = 2/3, because the ratio of reference points in archive S A is always lower than 2/3. Consequently, when M = 3, setting ratio threshold rp ratio to either 1 or 2/3 makes no difference to the optimization process of LORA-MOO. Similarly, the results of rp ratio = 1/3 on some problems are the same as the results It can be observed that both the variants of n c = 5 and n c = 7 outperform three other variants and are inferior to one variant, showing the optimal performance over other variants in this ablation study.In comparison, the variants of n c = 3 and n c = 10 are significantly superior to two variants but are The solution distribution we obtained on some 3-objective DTLZ problems are plotted.In Section 4.3 of the main file, we display the optimization results of comparison algorithms on DTLZ problems in terms of IGD values. In this section, we provide detailed IGD results on WFG problems and more results on IGD+ and HV values. In addition, the optimization results on DTLZ problems with different scales, such as D = 5 and 20, are reported.Table 7 shows the optimization results on WFG problems in terms of IGD values. The last row summarizes the results of statistical tests, which has reported at the end of Table 1 in the main file.It can be seen that LORA-MOO outperforms all comparison algorithms, followed by KTA2 and 7.56e-2)+ 2.72e+0(6.21e-2) 10 3.08e+0(5.70e-2)+ 3.11e+0(9.16e-2)+ 2.99e+0(9.77e-2)+ 3.09e+0(1.03e-1)+ 3.04e+0(1.12e-1)+ 3.10e+0(9.11e-2)+ 2.93e+0(6.20e-2) WFG27.66e-1(7.11e-2)+ 3.61e-1(3.87e-2)≈ 4.24e-1(6.65e-2)+ 5.48e-1(3.75e-2)+ 5.22e-1(7.67e-2)+ 4.88e-1(6.53e-2)+ 3.72e-1(4.87e-2) 1.05e+0(1.40e-1)+ 5.00e-1(3.97e-2)-5.66e-1(3.80e-2)+ 7.61e-1(1.21e-1)+ 7.48e-1(1.23e-1)+ 7.45e-1(1.45e-1)+ 5.46e-1(3.53e-2) 1.90e+0(3.51e-1)+ 7.77e-1(5.25e-2)-9.00e-1(5.39e-2)+ 1.28e+0(4.02e-1)+ 1.28e+0(3.75e-1)+ 1.49e+0(3.76e-1)+ 8.55e-1(7.00e-2) 2.74e+0(6.68e-1)+ 1.06e+0(5.98e-2)-1.18e+0(1.14e-1)-2.10e+0(6.97e-1)+ 1.90e+0(5.25e-1)+ 2.06e+0(4.58e-1)+ 1.24e+0(1.23e-1) 10 3.73e+0(9.41e-1)+ 1.18e+0(9.32e-2)-1.37e+0(1.03e-1)-2.84e+0(8.61e-1)+ 2.59e+0(9.91e-1)+ 2.95e+0(7.55e-1)+ 1.83e+0(2.27e-1) WFG35.82e-1(3.86e-2)+ 5.39e-1(5.81e-2)+ 3.29e-1(5.99e-2)+ 5.04e-1(6.26e-2)+ 4.60e-1(5.94e-2)+ 3.85e-1(4.76e-2)+ 2.83e-1(5.99e-2) 7.30e-1(6.25e-2)+ 6.66e-1(7.02e-2)+ 5.63e-1(6.47e-2)+ 6.05e-1(7.26e-2)+ 5.64e-1(6.43e-2)+ 5.68e-1(5.92e-2)+ 4.13e-1(5.98e-2) 7.75e-1(9.36e-2)+ 6.76e-1(1.32e-1)≈ 7.94e-1(6.73e-2)+ 7.41e-1(8.33e-2)+ 6.37e-1(9.55e-2)≈ 7.96e-1(6.68e-2)+ 6.51e-1(9.20e-2) 8.38e-1(1.63e-1)≈ 8.27e-1(9.79e-2)≈ 9.45e-1(7.42e-2)+ 7.63e-1(1.06e-1)-6.25e-1(1.18e-1)-8.92e-1(9.90e-2)≈ 8.54e-1(9.98e-2) 10 6.85e-1(1.02e-1)-6.87e-1(8.79e-2)-9.16e-1(8.20e-2)+ 5.91e-1(9.34e-2)-5.19e-1(1.04e-1)-7.28e-1(1.10e-1)-8.23e-1(1.14e-1) WFG46.21e-1(3.68e-2)+ 4.67e-1(2.33e-2)+ 4.21e-1(2.21e-2)+ 4.57e-1(2.88e-2)+ 4.23e-1(2.53e-2)+ 4.34e-1(5.63e-2)+ 3.36e-1(2.95e-2) 1.11e+0(3.45e-2)+ 7.86e-1(2.45e-2)+ 7.78e-1(4.50e-2)+ 9.83e-1(1.22e-1)+ 8.46e-1(8.32e-2)+ 1.07e+0(1.18e-1)+ 6.82e-1(4.97e-2) 2.75e+0(2.36e-1)+ 1.87e+0(8.92e-2)≈ 1.78e+0(7.66e-2)-3.13e+0(3.86e-1)+ 2.69e+0(3.61e-1)+ 2.92e+0(3.04e-1)+ 1.86e+0(1.30e-1) 5.09e+0(9.78e-1)+ 3.47e+0(2.96e-1)-3.26e+0(1.67e-1)-5.81e+0(5.38e-1)+ 4.99e+0(4.67e-1)+ 5.76e+0(4.34e-1)+ 3.62e+0(3.31e-1) 10 7.18e+0(1.21e+0)+ 5.60e+0(6.92e-1)≈ 4.97e+0(1.72e-1)-8.58e+0(8.39e-1)+ 7.78e+0(8.13e-1)+ 8.03e+0(5.03e-1)+ 5.47e+0(4.14e-1) WFG5 4.21e-1(3.05e-2)+ 3.91e-1(4.22e-2)≈ 3.30e-1(9.56e-2)-5.50e-1(3.05e-2)+ 5.30e-1(4.46e-2)+ 4.51e-1(6.51e-2)+ 4.21e-1(1.35e-1) 9.98e-1(8.09e-2)≈ 7.65e-1(2.86e-2)-7.20e-1(6.23e-2)-8.87e-1(3.98e-2)-8.61e-1(4.68e-2)-1.02e+0(4.57e-2)+ 9.81e-1(5.76e-2) 2.82e+0(1.65e-1)+ 1.78e+0(6.23e-2)-1.92e+0(1.03e-1)-2.35e+0(1.86e-1)+ 2.04e+0(1.29e-1)-2.44e+0(1.08e-1)+ 2.11e+0(9.10e-2) 5.25e+0(2.55e-1)+ 3.30e+0(2.61e-1)-3.62e+0(2.64e-1)≈ 4.75e+0(3.77e-1)+ 3.95e+0(2.83e-1)+ 4.57e+0(1.82e-1)+ 3.66e+0(9.43e-2) 10 7.64e+0(3.23e-1)+ 4.67e+0(4.78e-1)-4.76e+0(1.99e-1)-6.88e+0(4.23e-1)+ 6.11e+0(4.62e-1)+ 6.68e+0(3.49e-1)+ 4.98e+0(1.57e-1) WFG6 7.96e-1(5.50e-2)+ 7.05e-1(5.10e-2)+ 6.22e-1(8.49e-2)+ 7.19e-1(4.80e-2)+ 7.09e-1(4.61e-2)+ 5.79e-1(4.68e-2)+ 5.67e-1(1.09e-1) 1.14e+0(3.47e-2)+ 1.02e+0(4.90e-2)+ 9.62e-1(4.46e-2)≈ 1.08e+0(4.82e-2)+ 1.04e+0(4.53e-2)+ 1.17e+0(4.94e-2)+ 9.51e-1(9.85e-2) 2.81e+0(2.60e-1)+ 2.18e+0(7.41e-2)+ 1.96e+0(4.17e-2)-2.56e+0(2.16e-1)+ 2.20e+0(1.61e-1)+ 2.77e+0(1.81e-1)+ 2.04e+0(9.86e-2) 4.70e+0(5.78e-1)+ 3.60e+0(1.17e-1)+ 3.54e+0(1.85e-1)≈ 4.70e+0(5.18e-1)+ 4.13e+0(3.06e-1)+ 5.06e+0(3.20e-1)+ 3.52e+0(1.52e-1) 10 7.66e+0(5.36e-1)+ 5.00e+0(1.33e-1)+ 5.09e+0(1.58e-1)+ 6.73e+0(5.98e-1)+ 5.83e+0(4.69e-1)+ 7.00e+0(4.90e-1)+ 4.76e+0(1.94e-1) WFG76.69e-1(2.70e-2)+ 6.28e-1(2.45e-2)+ 5.73e-1(2.76e-2)+ 5.78e-1(3.23e-2)+ 5.38e-1(3.58e-2)+ 4.43e-1(4.15e-2)+ 3.52e-1(2.22e-2) 1.13e+0(4.94e-2)+ 9.48e-1(2.66e-2)+ 9.04e-1(2.51e-2)+ 9.92e-1(8.75e-2)+ 8.81e-1(3.49e-2)+ 9.72e-1(7.29e-2)+ 7.07e-1(4.29e-2) 3.17e+0(2.89e-1)+ 2.00e+0(5.61e-2)≈ 1.96e+0(5.97e-2)≈ 2.71e+0(3.18e-1)+ 2.18e+0(1.49e-1)+ 2.71e+0(1.91e-1)+ 1.96e+0(1.06e-1) 5.93e+0(3.95e-1)+ 3.64e+0(1.23e-1)-3.37e+0(1.16e-1)-5.19e+0(5.20e-1)+ 4.28e+0(4.59e-1)+ 5.19e+0(3.07e-1)+ 3.82e+0(1.63e-1) 10 8.78e+0(4.70e-1)+ 5.31e+0(3.01e-1)-4.88e+0(1.76e-1)-8.07e+0(5.07e-1)+ 6.77e+0(5.93e-1)+ 7.57e+0(4.12e-1)+ 5.73e+0(3.07e-1) WFG8 8.45e-1(2.87e-2)+ 6.42e-1(2.49e-2)+ 5.09e-1(4.39e-2)-7.49e-1(4.33e-2)+ 7.13e-1(3.87e-2)+ 7.01e-1(4.35e-2)+ 6.02e-1(3.64e-2) 1.33e+0(4.61e-2)+ 1.14e+0(3.89e-2)≈ 1.02e+0(3.96e-2)-1.26e+0(6.23e-2)+ 1.20e+0(5.28e-2)+ 1.36e+0(6.94e-2)+ 1.13e+0(7.12e-2) 3.11e+0(2.82e-1)+ 2.43e+0(7.15e-2)≈ 2.28e+0(5.05e-2)-3.00e+0(1.53e-1)+ 2.80e+0(1.90e-1)+ 3.07e+0(1.74e-1)+ 2.45e+0(9.73e-2) 5.74e+0(3.56e-1)+ 4.01e+0(2.28e-1)-3.92e+0(1.28e-1)-5.56e+0(3.24e-1)+ 5.11e+0(4.10e-1)+ 5.34e+0(2.72e-1)+ 4.22e+0(2.75e-1) 10 8.30e+0(4.83e-1)+ 5.56e+0(5.40e-1)-5.71e+0(3.80e-1)≈ 7.81e+0(4.74e-1)+ 7.32e+0(3.46e-1)+ 7.54e+0(4.88e-1)+ 5.82e+0(2.95e-1) WFG97.14e-1(5.09e-2)+ 6.75e-1(6.73e-2)+ 6.37e-1(8.35e-2)+ 6.74e-1(8.53e-2)+ 6.11e-1(9.76e-2)+ 5.12e-1(7.74e-2)+ 4.34e-1(8.18e-2) 1.24e+0(1.41e-1)+ 1.06e+0(8.72e-2)+ 1.07e+0(9.28e-2)+ 1.16e+0(1.18e-1)+ 1.05e+0(1.61e-1)+ 1.02e+0(7.89e-2)+ 8.43e-1(9.25e-2) 3.14e+0(2.96e-1)+ 2.22e+0(1.94e-1)+ 2.19e+0(1.52e-1)+ 2.83e+0(2.36e-1)+ 2.30e+0(1.82e-1)+ 2.55e+0(1.21e-1)+ 1.97e+0(9.18e-2) 5.78e+0(4.51e-1)+ 3.93e+0(3.00e-1)+ 3.77e+0(2.23e-1)+ 5.43e+0(3.68e-1)+ 4.60e+0(3.92e-1)+ 4.73e+0(3.07e-1)+ 3.61e+0(2.05e-1) 10 8.41e+0(4.80e-1)+ 5.69e+0(6.42e-1)+ 5.26e+0(3.13e-1)≈ 7.77e+0(5.05e-1)+ 6.48e+0(5.60e-1)+ 6.74e+0(4.17e-1)+ 5.16e+0(2.60e-1) 5.11e-2(7.70e-3)+ 1.44e-2(5.17e-3)-5.35e-2(1.14e-2)+ 2.49e-2(6.87e-3)+ 2.01e-2(5.56e-3)≈ 1.89e-2(5.87e-3)≈ 1.87e-2(3.21e-3) 10 1.19e-2(1.01e-3)+ 6.26e-3(9.09e-4)+ 1.19e-2(1.80e-3)+ 7.45e-3(9.85e-4)+ 4.80e-3(1.09e-3)-5.48e-3(9.49e-4)≈ 5.62e-3(1.75e-3) DTLZ6 3 2.42e-1(1.07e-1)+ 3.05e+0(5.23e-1)+ 1.82e+0(4.48e-1)+ 4.85e+0(6.38e-1)+ 4.27e+0(5.48e-1)+ 2.35e-1(4.14e-1)+ 6.74e-2(1.55e-1) 4+/ ≈ /-2.64e-1(1.83e-1)+ 2.44e+0(3.90e-1)+ 1.84e+0(5.17e-1)+ 5.12e+0(4.31e-1)+ 4.07e+0(6.25e-1)+ 1.35e+0(9.45e-1)+ 2.07e-1(2.06e-1) 61.78e-1(1.07e-1)-1.33e+0(2.80e-1)+ 1.49e+0(5.98e-1)+ 3.14e+0(4.44e-1)+ 2.32e+0(5.72e-1)+ 2.04e+0(6.34e-1)+ 9.00e-EQUATION1.07e+0(1.62e-1)≈ 4.31e-1(3.82e-2)-4.39e-1(1.48e-1)-4.80e+0(1.01e+0)+ 2.93e+0(7.01e-1)+ 3.96e+0(1.88e+0)+ 1.46e+0(6.89e-1) 81.28e+0(1.27e-1)-6.29e-1(7.74e-2)-7.72e-1(1.53e-1)-6.03e+0(1.87e+0)+ 3.63e+0(5.55e-1)+ 4.40e+0(2.74e+0)+ 2.25e+0(6.88e-1) 10 1.51e+0(1.37e-1)+ 2.64e+0(4.48e-2)+ 2.66e+0(7.65e-2)+ 2.60e+0(1.15e-1)+ 2.62e+0(6.34e-2)+ 2.55e+0(6.82e-2)+ 2.59e+0(4.96e-2)+ 2.49e+0(7.00e-2) 10 2.88e+0(6.44e-2)+ 2.78e+0(9.91e-2)+ 2.65e+0(1.26e-1)≈ 2.71e+0(1.27e-1)+ 2.71e+0(1.22e-1)+ 2.78e+0(1.04e-1)+ 2.62e+0(7.81e-2) WFG2 3 6.99e-1(9.48e-2)+ 2.58e-1(4.09e-2)≈ 2.39e-1(7.01e-2)≈ 4.68e-1(5.12e-2)+ 4.30e-1(9.29e-2)+ 3.95e-1(7.73e-2)+ 2.47e-1(4.89e-2) 49.74e-1(1.65e-1)+ 3.21e-1(4.70e-2)-3.52e-1(5.16e-2)≈ 6.27e-1(1.42e-1)+ 6.22e-1(1.45e-1)+ 6.23e-1(1.69e-1)+ 3 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 4 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 6 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) DTLZ2 3 3 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 4 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 6 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) DTLZ4 3 4.20e-4(2.03e-3)+ 5 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 6.38e-4(3.44e-3)≈ 1.10e-2(5.92e-2) 10 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 20 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) DTLZ2 5 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 10 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) 20 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0) DTLZ4 .68e-2(1.94e-2) 10 3.91e-3(7.22e-3)+ 0.00e+0(0.00e+0)+ 0.00e+0(0.00e+0)+ 0.00e+0(0.00e+0)+ 0.00e+0(0.00e+0)+ 3.52e-2(2.51e-2)+ 4.91e-2(2.38e-2) 20 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 0.00e+0(0.00e+0)≈ 2.06e-3(7.33e-3) DTLZ7 5 Justification: Claims we made accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: A runtime comparison experiment is reported in the end of our experiment section. We did not provide information about compute workers and memory since our experiments do not have specific requirements on memory or other computation resource.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Multi-objective optimization hasorobjectives, many-objective optimization hasor more objectives. Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.The link of code and data will be released here once the paper is accepted.Symbol '←' indicates the result of a function, Symbol '=' indicates an assignment operation.10-objective WFG problems are plotted in Fig.10.659 


Decision: This paper develops and studies a surrogate-guided multi-objective optimization approach. The key idea is to use ordinal values as the regression labels to train surrogates, and use them to achieve better trade-off between efficiency of optimization and quality of solutions. 

There is a consensus among the reviewers' that the paper is lacking in the following aspects and is not ready for publication.
- Limited novelty as proposed improvements are incremental with respect to prior work.
- Experimental evaluation is limited especially on real-world problems to draw strong conclusions.
- The paper needs to position/contextualize the proposed work better with respect to prior work.

For all the above reasons, I'm recommending to reject this paper and encourage the authors' to improve the paper based on the feedback from reviewers'.

Reviews: review: paper_summary: paper_summary: This paper introduces a surrogate assisted method for multi-objective optimization. The approach learns a surrogate function with the ordinal values as the regression labels. The ordinal values are generated using a iterative algorithm with the most dominated solutions having the highest ordinal values. The ordinal values are used to train a Kriging model used to select a point for observation using the convergence criterion. Another point is selected using the Kriging model trained on spherical coordinates via a diversity criterion. The approach has several parameters which are tuned via experimentation on real and benchmark datasets.
strength_weakness: strength_weakness: - The approach presented in the paper is not sufficiently novel. Ordinal regression for multi-objective optimization has been studied before [1]. The differences with related prior work have not been discussed in detail.
- The proposed algorithm has many tunable parameters, and it is unclear how the parameters affect performance on real world problems when they have only been tuned on benchmark problems.
- The real world experiment on NAS shows improved regret eventually, but converges slower than other existing approaches. It is difficult to judge on the effectiveness of this approach based on a single experiment. Experiments on more real world optimization problems are necessary to make a conclusion.
- The paper is missing several notable MOO approaches from the Bayesian optimization community [2,3,4,5].

[1] Yu, Xunzhao, et al. "Domination-based ordinal regression for expensive multi-objective optimization." 2019 IEEE symposium series on computational intelligence (SSCI). IEEE, 2019.

[2] Tu, Ben, et al. "Joint entropy search for multi-objective bayesian optimization." Advances in Neural Information Processing Systems 35 (2022): 9922-9938.

[3] Zhang, Richard, and Daniel Golovin. "Random hypervolume scalarizations for provable multi-objective black box optimization." International conference on machine learning. PMLR, 2020.

[4] Paria, Biswajit, Kirthevasan Kandasamy, and Barnabás Póczos. "A flexible framework for multi-objective bayesian optimization using random scalarizations." Uncertainty in Artificial Intelligence. PMLR, 2020.

[5] Abdolshah, Majid, et al. "Multi-objective Bayesian optimisation with preferences over objectives." Advances in neural information processing systems 32 (2019).
questions: questions: What parameters were used for the NAS problem and how were they tuned? Do parameters tuned on benchmark problems generalize to real world problems?
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes a novel surrogate-assisted evolutionary algorithm named LORA-MOO, the core contribution is the introduce of ordinal-regression-based model  spherical coordinates approximation to SAEA and LORA-MOO can find a good trade-off between optimization efficiency and optimization results.
strength_weakness: strength_weakness: Motivation and contributions are limited. To the best of my knowledge, the main contrition of LORA-MOO is introducing ordinal-regression-based model for convergence and spherical coordinates for diversity. However, why do it and what is the connections between them? Besides, the manuscript includes a lot of informal expression. The proposed method is complex and effectiveness is limited.
questions: questions: a. Why is Many-objective optimization problems called MOOPs for short? What's the abbreviation of Multi-objective optimization problems？I think the author needs to consult more relevant literature to make the expression more formal.

b. Considering that the author mentioned MOBO, but there was no comparison in the experimental stage, it is interesting to provide a comparison with PSL-MOBO[1], qNEHVI[2], DAPSL[3] and so on. In addition, more SOTA methods should be introduced to demonstrate the superiority of LORA-MOO since almost all the compared methods is out of date.

[1] Lin, Xi, et al. "Pareto set learning for expensive multi-objective optimization." Advances in Neural Information Processing Systems 35 (2022): 19231-19247.

[2] Daulton, S.; Balandat, M.; and Bakshy, E. 2021. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34: 2187–2200.

[3] Lu, Yongfan, Bingdong Li, and Aimin Zhou. "Are You Concerned about Limited Function Evaluations: Data-Augmented Pareto Set Learning for Expensive Multi-Objective Optimization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 13. 2024.

c.  In all the tables, the author doesn't have any highlights, which can't let readers know what LORA-MOO is good at or not. Moreover, most tables in Appendix miss the comparison (=,-,+) in terms of a single question

d. In the experiments section, I only saw some results presented, but I didn't see any in-depth analysis. 

e. Lack of Real-World Application Depth: Although a real-world network architecture search problem is mentioned, the paper does not delve deeply into real-world applications. More case studies or industrial applications should be considered [4].

f. Lack of Theoretical Analysis: The paper focuses heavily on empirical results but lacks a rigorous theoretical analysis to support the empirical findings. Theoretical results, such as convergence proofs or time complexity analysis, would strengthen the paper's contributions.

g．The idea of the introduce of spherical coordinates has been discussed in [5]. What is the main difference between [5] and this paper?
[5] Zhang, Xiaoyuan, et al. "Hypervolume maximization: A geometric view of pareto set learning." Advances in Neural Information Processing Systems 36 (2024).

h. The motivation is not enough. It seems that the main target of LORA-MOO is to enhance efficiency via only training a single model. However, I can’t find obvious advantage in terms of Runtime Comparison. There is not enough motivation to support the introduction of A. There is not enough motivation to support the introduction of spherical coordinates.

i. Why PSO is introduced to conduct offspring generation?

j. Why use S_A as the input of the Kriging model (Algorithm 1)? In my understanding, S_o and S_a are enough for construction of Kriging model.

k. In each iteration, only two solutions (one for convergence and one for diversity) are evaluated. So, why not design a dynamic strategy to determine the number of evaluation solutions considering that the number of promising solutions is varying with the population evoluting.
limitations: limitations: See questions.
paper_summary: paper_summary: This paper proposes a surrogate-assisted evolutionary many-objective optimization algorithm, named LORA-MOO. LORA-MOO is composed of a surrogate for ordinal modeling, which focuses on convergence, and m-1 surrogates for distribution modeling, which focus on diversity. Empirical study demonstrates the effectiveness of the proposed algorithm.
strength_weakness: strength_weakness: Although LORA-MOO obtained better indicator values than the baselines in synthetic problem benchmarks, it is difficult for me to find some fundamental differences between LORA-MOO and previous MOAs. Nor could I see what new insights this paper can provide for solving expensive MOPs. LORA-MOO models the convergence of solutions by a surrogate problem of the domination level, which is a common idea in MOO, adopted by many past methods such as NSGA-II. Such a surrogate is intuitive, but it is unclear to me why it could work better than the existing surrogates such as pairwise relation or function values, and why such a surrogate can be successfully modeled by a Gauss process. LORA-MOO uses m-1 surrogates to predict the spherical coordinate, but it seems identical to predicting function values. LORA-MOO also contains many other components, such as EA, PSO, non-dominated sorting, various clustering methods, and some subset selection mechanisms. These components have long been widely adopted by many MOAs, and there are many alternatives available as well. I agree that these components could usually make an algorithm perform better, but this paper does not adequately demonstrate any necessity for such a combination or any connections between these components. The ablation study appears to be a parameter-tuning experiment, presenting some results under different parameters. However, there is no ablation for the many components in the algorithm, so it is unclear what contributions these components actually make.

Expensive optimization problems are closely connected to real-world applications, and many real-world MOPs are indeed expensive problems. Therefore, I believe this is a very valuable research direction. However, the empirical study in this paper is mainly conducted on synthetic problems. DTLZ and WFG have undoubtedly driven the development of the MOO field, but they have also caused a significant number of researchers to focus narrowly on these synthetic test sets. As a result, there are now many algorithms that perform excellently on synthetic test sets but struggle to adapt to real-world problems. The authors have also conducted tests on NASBench, which is crucial for comprehensively demonstrating the algorithm's capabilities, but the results do not seem to be sufficiently convincing.
questions: questions: 1. Regarding typesetting, I recommend using the Roman font for the operators instead of the italic font. For instance, use $\max$, $\exp$, $\cos$ instead of $max$, $exp$, and $cos$.
2. P2, L45. One of the difficulties of expensive optimization problems is the small data size. Pairwise relations can increase the data size, making it possible to train some DL models. Therefore, this is an advantage of pairwise relations. Additionally, why do the authors believe that the amount of data will increase exponentially? At most, there will only be $O(N^2)$ pairs, so the increase is at most quadratic. Considering that DL models can usually be trained in parallel, I don't think efficiency could be a major drawback. The authors repeatedly claim efficiency advantages (L45, L59), but the experimental results do not demonstrate any efficiency advantage over the baseline.
3. P7, Fig. 1. The caption "Evaluations" is clipped.
4. P7, Line 268. IGD/IGD+ uses a subset sampled from the true PF.
5. Page 9, Fig. 4. Why present the normalized runtime instead of the real time?
6. P17, On the reference point for HV. It is inappropriate to set the reference point for HV to (1,1, ...,1). Firstly, the reference point can only be set near 1 if all solutions have already converged well. Additionally, to better measure diversity, the reference point for HV needs to be larger than the nadir point, so it is generally set to at least $1.1$. For expensive multi-objective optimization problems, the reference point needs to be set to an even larger value because most solutions may not have converged well. For example, in Table 10 and Table 14, the HV values are all 0, indicating that the reference point is set too low.
limitations: limitations: This paper does not summarize its limitations. I suggest the authors reconsider the limitations of this work and fully present them in the paper.
paper_summary: paper_summary: This paper proposed the LORA-MOO framework, a surrogate-assisted MOO algorithm that learns surrogates from spherical coordinates. This includes an ordinal-regression-based surrogate for 10 convergence and M −1 regression-based surrogates for diversity.
strength_weakness: strength_weakness: Major:
1. Line 113-116, a bit too repetitive. 

2. Line 121, an initial dataset size of 11D-1, too specific.

3. Lines 121-134: the algorithms are described too specifically. It is more suitable for EC journals rather than NeurIPS.

4. Selection Criteria, too simple. 

5. What does LORA-MOO means?

6. Some HV-based MOBO methods are not compared as they are failed to solve many objectives. This argument is not accurate, please consider to run "https://github.com/xzhang2523/libmoon/blob/main/libmoon/solver/mobo/run_dirhvego.py", which supports more than ten objectives problems. 


Minor:
There are too many grammar errors in this paper. 
(1) Line 249, they are failed -> they failed. . 

(2) Line 270, HV use -> HV uses . 

(3) Line 175, consider using \max.

(4) line 21, are widely exist -> widely exist. 

(5) line 64, an non-pa .. -> a non-pa..
questions: questions: See weakness.
limitations: limitations: See weakness.

Comments: title: Thanks for your response
comment: Thank you for your response. I increased my rating. However, it is still not clear to me what new insights this paper could provide in solving expensive optimization problems. I suggest the authors submit this manuscript to TEC rather than NeruIPS. Maybe the audiences there are more interested in this paper.
Thanks for your comment.

As we explained in our initial rebuttal, we do have ablation studies for $\textbf{these components}$:

The ablation study reported in Appendix F.2 has demonstrated the contribution of our $\lambda$-dominance (described in Section 3.2.1). 
When $\lambda$= 0, the component of $\lambda$-dominance is actually $\textbf{removed}$ from our algorithm. 
In Table 4, by comparing the results of $\lambda$ = 0 (performance with normal dominance relations) and $\lambda$ > 0 (performance with this component, $\lambda$-dominance relations), we can observe the contribution of this component and why it is necessary to our algorithm (lines 598-600 in our manuscript).

Similarly, our ablation study in Appendix F.3 shows the contribution of our artificial relations (described in Section 3.2.2) since $rp_{ratio}$ = 1 indicates this component is $\textbf{removed}$. We have provided explanations about the contribution of this component in lines 618-627.

Our ablation study in Appendix F.4 shows the contribution of our clustering-based initialization (see Section 3.3.1) since 
$n_c$ = 1 indicates this component is $\textbf{removed}$ and replaced by a random initialization. The performance of $n_c$ = 1 and that of $n_c$ > 1 are compared and explained (lines 641-646).

In summary, we can observe the contributions of our algorithm components directly from our ablation studies.

$\quad$

---

As for theoretical analysis, we admit it is inappropriate to say runtime analysis, but time complexity analysis is provided.

$\quad$

---

Finally, we wonder to know if we have solved your concerns on contributions.
Thank you for your reply.

I said, "There is no ablation study **for these components**". The ablation study in Section 4.2 and Appendix F only discussed the parameters, not the contribution of the components. For, example, you should replace one component with another one to demonstrate this component is necessary.

I read your "runtime analysis". It is only a time complexity analysis, not a runtime analysis. I think the authors do not know what "runtime analysis" means. You can search "runtime analysis" on Google Scholar to get some good papers to learn.

I keep a negative rating for this paper.
Thanks for your comment.

1. $\textbf{About Experiment}$: 

We have conducted a new real-world NAS experiment with different network architecture and 8 objectives (error, parameters, flops, edge gpu latency, edge gpu energy, eyeriss latency, eyeriss energy, and eyeriss arithmetic intensity). The comparison results are consistent with the real-world problem we reported in our manuscript. Our LORA-MOO outperforms the comparison algorithms and have reached a mean HV value of 0.5776 over 30 independent runs. We would add the corresponding figure and descriptions to our manuscript (As we are unable to add figures in our comment now).

---

2. $\textbf{Theoretical Analysis}$: 

We attempt to add the following time complexity analysis:

Notations:
- n: the number of training samples.
- N: the number of test samples.
- m: the number of objectives.
- g: the number of generations for reproducing candidate solutions.
- p: the population size for a generation.

$\quad $

The model used in LORA-MOO is Gaussian Process, the training time complexity is analyzed as follows:
- Time complexity of covariance matrix computation is $O(n^2)$.
- Time complexity of Cholesky decomposition and computation of likelihood: $O(n^3)$.

The prediction time complexity is analyzed as follows:
- Time complexity of computing the covariance between test sample and training samples: $O(n*N)$.
- The time complexity of predicting the mean:  $O(n*N)$.
- The time complexity of predicting the variance: $O(n^2*N)$.

In summary, the overall training complexity is $O(n^3)$, and the overall prediction complexity is $O(n^2*N)$.

$\quad $

Now we analyze the time complexity of model-based optimization algorithms, for each iteration, the number of test samples is $p * g$, so the total number of test samples is approximately $N = n * p * g$.

1. For LORA-MOO, for a $m$-objective optimization problem:
- The time complexity of training an ordinal model and $m-1$ angular models is $O(n^3* m)$.    
- The time complexity of prediction in the ordinal model is $O(n^3 * g * p)$. 
- The time complexity of prediction in $m-1$ angular models: $O(n^3 * p * (m-1))$.
- The overtime time complexity in models for LORA-MOO: 
$O(n^3 * (m + g * p + p * m - p)) \approx O(n^3 * (p * g + p * m)) \approx O(n^3 * p * (m + g))$

2. In comparison, for other optimization algorithms with $m$ surrogate models:
- The time complexity of training $m$ models: $O(n^3 * m)$
- The time complexity of prediction: $O(n^3 * g * p * m)$. 
- Time overtime time complexity in $m$ models: $O(n^3 * (m * (1 + g * p))) \approx O(n^3 * p * m * g))$

3. For other optimization algorithms with only one surrogate model:
- Time overtime time complexity: $O(n^3 * (1 + g * p)) \approx O(n^3 * p * g)$

$\quad $

Therefore, increasing the number of objectives $m$ has limited impact on the time cost of LORA-MOO ($O(n^3 * p * (m + g))$), but for the comparison algorithms with $m$ surrogate models, their time cost would increase rapidly ($O(n^3 * p * m * g)$).

Although LORA-MOO has $m$ surrogate models in total, its time complexity does not significantly larger than the time complexity of optimization algorithms with only one surrogate model ($O(n^3 * p * g)$).
Thanks for your comment.

We would like to clarify the following issues:

$\textbf{About ablation studies and theoretical analysis}$:

- We do have ablation studies in Section 4.2 and Appendix F. More importantly, our ablation studies have already demonstrated the contributions of these algorithm components (Detailed explanations about this are available in our first rebuttal to weaknesses). 
- We have provided a theoretical runtime analysis in the response to another reviewer. 

---

$\textbf{About contributions}$:
- We would like to highlight that our modeling method for convergence is called $\textbf{ordinal-regression model}$, $\textbf{NOT dominance-regression model}$. Although the modeling process includes some dominance related relations, it also includes a lot of artificial ordinal relations which are NOT dominance level related relations (described in Section 3.2.2). 
- The proposed ${\lambda}$-dominance is $\textbf{only a minor contribution (about 10\\%) in our work}$. We have more contributions on our clustering-based artificial ordinal relations, mapping of ordinal relations, our spherical coordinate based diversity maintenance strategy, our global search strategy ... We are confused that why the reviewer ignores our main contributions while concerning severely on the similarity between our minor contribution and existing dominance related works. 
- LORA-MOO is not a combination or recombination of existing methods. Many components in LORA-MOO are not available in any existing studies, such as the clustering-based artificial relations. Again, we hope the reviewer can focus on our main contributions mentioned above.
- There is no motivation for combination since our work is not a combination of existing works.

$\quad$

Thanks for your reading.
Thank you for your response. I know that the strict dominance relation is not suitable for many-objective scenarios. However, there have been many existing relaxed dominance relations such as $\epsilon$-dominance [1], grid-dominance [2], and $k$-optimality [3]. They are designed for many-obj optimization. It is not clear why the proposed $\lambda$-dominance can be more effective than the existing dominance relations.

In short, LORA-MOO seems to be a recombination of some existing methods or similar ideas. The motivation and necessity for such a combination are not clearly presented. There is no theoretical analysis or ablation study for these components. Therefore, I maintain a negative rating for this paper.

[1] Combining convergence and diversity in evolutionary multiobjective optimization, Evolutionary Computation, 2002.

[2] A grid-based fitness strategy for evolutionary many-objective optimization, GECCO 2010.

[3] A fuzzy definition of ‘optimality’ for many criteria optimization problems, TCYB.
Thanks for your comment.

We would like to clarify that:
1. In many-objective optimization, it is impossible to model the dominance level in NSGA-II.
- For a many-objective optimization problem, all solutions in the initial training dataset could be non-dominated, indicating all solutions would locate in the same dominance level. Modeling such a dominance level is useless, it just like to model a classification model with all solutions in the same class.

2. The modeling of ordinal relations in LORA-MOO is able to solve the difficulties mentioned above, it is completely different from the dominance level in NSGA-II and it overcomes the drawback of dominance level in NSGA-II.
- As described in Section 3.2, the ordinal relations LORA-MOO learned from training dataset are a mixture of lambda-dominance relations and a clustering-based artificial relations. Both of two relations are novel concepts we proposed in this work.
- In addition, there is an ordinal relation mapping before using them to train models, which makes our model stable.

3. The novel ordinal regression model is only a part of our contributions. 
- We also proposed a spherical coordinate base modeling method to maintain the diversity of non-dominated solutions.
Thank you for your response. However, your rebuttal does to satisfactorily address my concern about the novelty of the proposed method. Therefore, I maintain my score. The authors seem confused about my comments, so I provide some clarifications.

> We are confused that why the reviewer mention NSGA-II when talking about ''modeling dominance level''

NSGA-II uses a dominance level to directly model convergence. I know LORA-MOO is model-based while NSGA-II is not. LORA-MOO just used a regression-based model to model such a dominance level, so it does not seem to provide new insights about how to model convergence. 

> Question 4: We are confused on this comment. 

In Line 268, you say "IGD/IGD+ use a set of truth Pareto fronts". What does "a set of truth Pareto fronts" mean? It is more accurate to express as "IGD/IGD+ uses a subset sampled from the true PF."
Thank you for your response. My concerns have been partially addressed. Since the lack of additional experiments and theoretical analysis, I have decided to maintain my score.
Thanks for your response. I have no questions for this paper.
Thanks for your comments.

---

About real-world experiment:

We have conducted a new real-world NAS experiment with different network architecture and 8 objectives (error, parameters, flops, edge gpu latency, edge gpu energy, eyeriss latency, eyeriss energy, and eyeriss arithmetic intensity). The comparison results are consistent with the real-world problem we reported in our manuscript. Our LORA-MOO outperforms the comparison algorithms and have reached a mean HV value of 0.5776 over 30 independent runs. We would add the corresponding figure and descriptions to our manuscript.

---
About contributions:

We would like to clarify that:
- Only the first difference between our LORA-MOO and OREA we listed above is an improvement over prior work, and this contribution is a very minor contribution in our work. 
- The remaining three differences (1.Clustering-based artificial relations, 2.Modeling angles in the spherical coordinate system for diversity maintenance, and 3. A novel global search method with novel initialization and reproduction strategies) are our main contributions. These three contributions are completely new ideas we developed to solve many-objective optimization problems, they do not exist in prior studies and they are not improvements of other works. 
- The development of our clustering-based artificial relations defines a novel way to learn ordinal-regression models. It distinguishes our ordinal model from prior work. 
- The development of our angle modeling method provides a novel way to maintain diversity, which is different from existing studies.
- OREA was not designed for many-objective optimization, it can be observed from our experiments that OREA only works well on multi-objective problems, while our LORA-MOO outperforms OREA significantly on many-objective optimization problems.

Although we used ordinal regression model in our work, we hope the reviewer can take a look at our other three main contributions. Just like there are many different classification models in diverse studies, ordinal regression models can also be different and diverse.

Thanks.
Thanks for the response. I am keeping my current score as
- We cannot draw a concrete conclusion based on results from a single real world experiment.
- While there are several improvements proposed in this paper over prior work, the improvements are relatively minor.