Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives
Authors: Shaokun Zhang
Feiran Jia
Chi Wang
Qingyun Wu
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Automatic Machine learning
Hyperparameter tuning
Lexicographic preference

Abstract: Motivated by various practical applications, we propose a novel and general formulation of targeted multi-objective hyperparameter optimization. Our formulation allows a clear specification of an automatable optimization goal using lexicographic preference over multiple objectives. We then propose a randomized directed search method named LexiFlow to solve this problem. We demonstrate the strong empirical performance of the proposed algorithm in multiple hyperparameter optimization tasks.

Published as a conference paper at ICLR 2023
TARGETED HYPERPARAMETER OPTIMIZATION WITH LEXICOGRAPHIC PREFERENCES OVER MULTIPLE OB- JECTIVES
Shaokun Zhang', Feiran Jia', Chi Wang", Qingyun Wu² Pennsylvania State University, State College, PA, USA 2 Microsoft Research, Redmond, Washington, USA (shackun shaack, feiran. jaraaaiiaappuueeee.. gingyun.wu wang. wwwweeicosoottttocmoomm
ABSTRACT
Motivated by various practical applications, we propose novel and general for- mulation of targeted multi-objective hyperparameter optimization. Our formula- tion allows clear specification of an automatable optimization goal using lex- icographic preference over multiple objectives. We then propose a randomized directed search method named LexiFlow to solve this problem. We demonstrate the strong empirical performance of the proposed algorithm in multiple hyperpa- rameter optimization tasks.
INTRODUCTION
Hyperparameter optimization (HPO) of machine learning models, as a core component of AutoML, is process of finding good choice of hyperparameter configuration that optimizes the model "performance". In the context of practical ML systems, there are typically more than one metrics evaluate the model "performance" on which one desires optimize. For instance, latency (He) et al., 2018), faireess (Brookhouse & Freitas 2022), and explainability (Gonzalez et al.) 2021] are important complementary metrics of interest in addition to prediction accuracy in many application scenarios. Typical multi-objective HPO (MO-HPO) approaches (Knowles) 2006), Daulton et al.) 2020) seek to find wide-spread Pareto frontiers for users to choose from. This type of method can only establish partial ordering of the configurations. The final choice on which Pareto frontier to use is typically done manually and is opaque to the optimization algorithm. We call such opti- mization "untargeted". An automated approach is desirable, especially in repetitive tuning scenarios such as continuous integration and delivery (CI/CD) of machine learning models or MLOps in gen- eral (Garg et al., [202]] Mākknnn et al. 2021 Symeonidis al.) (002).. This automation is possible the criteria for selecting the final choice is specified explicitly. In this scenario, untargeted HPO can be inefficient the optimization algorithm may waste resources finding Pareto frontiers that are far from the desired final choice, i.c., the target.
In this work, we consider a targeted HPO scenario: practitioners have a priority order over the objectives, which enables a total ordering of all the configurations. We formalize a general notion of priority order rigorously as a lexicographic preference (Fishburn) 1975) over multiple objectives in an HPO task. It allows users to specify a clear optimization target across multiple objectives before the optimization starts and removes the need for manual post hoc selection. Such priority structure is found in HPO tasks from various application domains. For example, in many bioinformatics applications, besides the primary objective of finding model hyperparameter configurations with low prediction eиror, minimizing feature numbers via feature selection step is found to be helpful in avoiding overfitting and discovering relevant feattures for domain experts and thus is suggested to be used an auxiliary objective in HPO (Bommert et al., 2017 Gonzalez et al. 2021). When both objectives are included, the auxiliary objective is considered less important than the minimization of the prediction error, which naturally forms lexicographic structure.
Despite its appealing practical importance, we find this type of targeted HPO problem remarkably under-explored. In this work, we first provide rigorous problem formulation for the targeted HPO

---

Published as a conference paper at ICLR 2023
LexiFlow
6.0
Trrger
Lath
Val Toss (1at objestive)
3.5
Vil Loos (lat otgectioe))
-55
Val Low (Ji objecativ)
-55
-30
Samang
8.5
4.5
40-
15
0.10
0.10
10.00
0.10
DSSP (2nd objestive)
DSP (2nd objoctioe)
DSPP (2nd objestive
Figure 1: Results in an XGBoost tuning task to find accurate and fair models, in which validation loss is specified as an objective of a higher priority, and DSP (fairness-related objective) of a lower priority. Lexi-Target" optimization starts and 0.05 is the user-specified tolerance onloss degradation. The circles represent proposed configurations from different methods in the objective space (darker color indicates later iteration). Both objectives are smaller the better. SO-HPO easily achieves the target loss value but performs poorly regarding fairness. MO-HPO able to achieve better performance in terms of fairness compared to SO-HPO. However, it also wastes resources on finding points outside the desired loss target as it seeks wide spread of the Pareto front. Compared to MO-HPO, a larger fraction of the configurations in LexiFlow are within the loss target. This allows LexiFlow to try more configurations within the desired loss range and achieves better fairness performance.
1 0.05, in which 1" is the optimal loss value which is unknown before the
task with lexicographic preference over multiple objectives. This formulation provides a general and flexible way for the users to specify customized targets expressed via priority order on the objec- tives and a list of optional goals and tolerances on the objectives. Based on the problem formulation, we propose an algorithm named LexiFlow as general solution. Specifically, LexiFlow conducts the optimization by leveraging pairwise comparisons between hyperparameter configurations in a randomized direct search framework. The pairwise comparisons supported by suite of targeted lexicographic relations, which allow us to navigate toward the more promising region of the search space considering the lexicographic structure in the objective space. By doing so, the algorithm is able to efficiently optimize the objectives with strong any-time performance.
We perform extensive empirical evaluation on four different machine learning model tuning tasks, including a tuning task to find accurate and fast/small neural network models, a tuning task to find accurate and fair Xgboost models, a tuning task on random forest combined with feature selection for gene expression prediction, and a tuning task to mitigate overfitting. Our method has promis- ing performance on all the evaluated tasks. The good empirical performance verified the unique advantages of our proposed algorithm. We demonstrate different performance patterns of meth- ods including our method LexiFlow, a single objective method (SO-HPO), and multiple objective method (MO-HPO) in Figure [
1.1 RELATED WORK
There are a number of works trying to address MO-HPO tasks on machine learning models, includ-
ing evolutionary algorithms (Deb al. 2002 Srinivas & Deb 1994 Zhang & Li 2007 Binder
ct al. 2020).
Bayesian optimization Knowles 2006 Daulton et al. |000; Emmerich & Klinken-
berg 2008 Hernández-LLobtto et al. (016) and multi-fidelity methods (Schmucker et al.) 2020]]
All the aforementioned methods treat all the objectives equally important and seek an ap- proximation of the Pareto front in the objectives space. Some recent work proposes incorporate different types of user preferences into MO-HPO. Paria et al.) (2020) allow users to encode their preferences as a prior on weight vector to scalarize multiple objectives. The prior will induce posterior distribution over the set of Pareto optimal values. Setting a proper prior in practice is non-trivial as the relation between the prior and posterior is difficult, if not impossible, to derive for an average practitioner. Abdolshah et al. (2019) regard preferences tives, using a constrained Bayesian optimization method. An earlier multi-objective optimization method (Zitzler et al. 2008)) incorporates preference information into the multiple objective evolu- tionary frameworks by defining relations between different populations. However, this preference information is defined upon different populations rather than individual configurations.
2021
the stability of objec-
2

---

Published as a conference paper at ICLR 2023
2 PROBLEM FORMULATION
In this section, we first provide a rigorous formulation of the targeted HPO problem with lexico- graphic preferences multiple objectives together with motivating example. We then analyze the challenges in approaching the problem.
2.1 PROBLEM FORMULATION
Throughout the paper, we use [K] to denote the integers from to K. The studied problem needs the following inputs from the user,
d-dimensional hyperparameter search space X. Any hyperparameter configuration x E X is considered feasible configuration.
A list of (K > 1) objective functions regarding x E X, denoted by F(x) = (x), (x)),..., f (x)]. Specifically, each f (vk [K]] is real-valued objective function of x E X, and the superscript in f indicates the priority order (the smaller the index, the high the priority) of the objective. Without loss of generality, we assume minimization problems regarding all the objectives here and throughout this paper.
K K-dimensional tolerance vector on the objectives denoted by T' = each 7* is non-negative number representing an optimality tolerance on the k-th objective. It can be intuitively considered as the amount of performance degradation the user is willing to compromise in order to find choices with better performance the objectives of lower priorities. ,2,,,,.... c a K -ddimensional goal vector, which c² E denotes a goal value on the k-th objective desired by the user.
7* Vk [K],
With the inputs F. T, C' specified above, Vk E [K]. we can define a lexi-target value 2 The lexi- target value z of the k-th objective defined recursively based on the lexi-target values of the more important objectives, i.e., the first (A- objectives. We define X = X and for l [K].
# = max S + + S = inf (x), = e x- - +
(1)
xEX!
where X! P 2 ... X is series of nested sets dubbed lexi-frontiers considering the first 1,2,... K objectives respectively; and f is the inferior limit of lexi-frontiers in X on the k-th objective. Note that the cardinality of [F(x)]xexx can be larger than one. To compare these F(x) vectors, we define lexicographic relations as follows:
F(x) =1 F(x) 4 f² (x) [KE
(2)
F(x) 41 F(x) + 3 [K] (x) < (x)) (VK' < k. (x') -kh (x))
(3)
F(x) 31 F(x) + F(x)) <1 F(x) V F(x') F())
(4)
Our aim is find lexi-optimal configuration, i.e., any element in X (x E X
x, Fx)) 31 F(x)), using minimal cost. We call this problem a Lexi-HPO problem in short.
Remark (Lexi-optimal vs Pareto frontiers). The lexi-optimal solutions X* are always a subset of the Pareto frontiers in the K K-dimensional objective space. The relative location of X. in the Pareto frontiers depends on the lexicographic preferences and the relaxations imposed via tolerance and/or goals. In this way. the users could specify their own 'target' of interest in the Pareto frontiers in straightforward way before the optimization starts.
Remark (On tolerances and goals). Note that the two inputs tolerances T and goals C are both optional. They are introduced to help impose customized needs on the targeted optimum in flexible ways. When there is no tolerance and no particular goal on a particular objective one can just set Tk 0 and c - inf without affecting the nature of the problem.
To facilitate understanding of the Lexi-HPO problem and better motivate the studied scenario, we include concrete example for gene expression prediction from the bio-informatics domain below.
Example: In the HPO task for gene expression prediction from (Bommert et al.) (007)). three minimization objectives considered during the HPO process, including the model's '' prediction
3

---

Published as a conference paper at ICLR 2023
error, the resulting feature numbers after feature selection, and feature selection instability. The three objectives are in descending priority order. Despite the priority order, when trying to find the optimal choice, it is okay to consider the configurations whose loss does not exceed 1" + 0.05, in which 1e denotes the best loss one can achieve by any feasible hyperparameter configuration. In addition, feature number of 500 is considered good enough for the second objective. Say in this example 1- = 0.1 and X XB, XC, XD are whole Pareto frontiers with performance F(XA) [0.2, 100, 0.1], F(xB) = [0.1, 600,,,,000000 0.2), F(xc) = =00.13, 500, 0.2), and F(XD) = [0.1, 0.5].
This example can be formulated as a Lexi-HPO problem, where the objectives are F [loss, feature number, instability), tolerances are T (inf, 500, iinf]. Under the Lexi-HPO problem, we have the following conclusions: (1) xe is the optimal choice among the whole frontiers; (2) Combining the first conclusion with Remark I we can also conclude that xc is also the lexi-optimal choice in the whole space. This example also shows that the tolerances T and goals C. are, in general, intuitive to understand and easy to set by practitioners because they are directly about each single objective value.
[0.05, Oj, and goals are C
2.2 CHALLENGES
We want to first remind the readers that in the context of HPO of ML models, the objectives are mostly black-box, and is also expensive to get function evaluations because it involves training and validating an ML model. The black-box nature of this problem makes methods that require ana- lytic forms of the objectives, e.g., gradient- based methods (Gong et al. (02T)) inapplicable. We also urge methods that are efficient and cost-frugal because of the expensive function evaluations and practical resource limitations. One potential solution for the Lexi-HPO problem is to use MO-HPO approaches to find Pareto frontiers and add a post-processing step to narrow them down to the tar- geted optimal points. This type of method can be inefficient because they strive to find a wide spread of Pareto frontiers while the users are actually only interested in a subset of them in a particular re- gion. Another potential solution is to perform single objective HPO or constrained single objective HPO regarding each objective sequentially following the preference order. More specifically, when optimizing the k-th objective, one can impose constraints of the form (x) S 2 Vi E [k- 1] and estimate z according to the result. However, one tricky problem in this approach is how to allocate limited resources the optimization of different objectives. The resource allocation directly affects how reliable the constraints are. For example, if we allocate insufficient resources to the first objec- tive, we will not reach the optimal value on the first objective which loosens the constraints imposed on the first objective when trying to optimize objectives of lower priorities. If we allocate excessive resources to the first objective, the optimization of other objectives can be hindered due to a lack of resources. We include this approach as a baseline in the experiment section.
3 ALGORITHM
We propose an algorithm named LexiFlow to solve the Lexi-HPO problem. We adopt a randomized direct search framework that could direct the search to the targeted optimum based on lexicographic comparisons over pairs of configurations. During the iterative optimization process, the algorithm automatically adjusts its focus on the different objectives based on their priorities and room for improvement. For example, when there is no room for improvement on the first objective in particular iteration, the algorithm will direct the search toward the region where the second objective could improve without hurting the optimality of the first objective. Since such adjustment is made at every iteration, the algorithm keeps the chance to improve every objective when the optimality has not been reached. By doing so, the algorithm is able adaptively allocate optimization resources over multiple objectíves in flexible order while respecting the priority. This strategy can help the algorithm achieve a good any-time performance.
LexiFlow is presented in Alg. T It takes as inputs the objectives F, goal vector C' (optionally), and a tolerance vector T' (optionally). When C' and T' are not provided, we just set c² to -00, and 7* to 0, Vk E [K]. After initialization step, which sets initial hyperparameter configuration xo and an initial stepsize denit, the algorithm proceeds as follows: At each iteration i, LexiFlow maintains an 'incumbent" point, denoted by x,. The 'incumbent" point is the point in whose neighboring area the algorithm samples randomized directions to get new points to try next. More specifically, a new direction u is sampled uniformly at random from a unit sphere (Line 4 in Alg. [. which leads to
4

---

Published as a conference paper at ICLR 2023
two newly proposed points x; + du (considering both the sampled direction and its opposite direc- tion). After evaluating the newly proposed point(s), i.e., obtaining F(x; + óu) and potentially also F(x; ōu), it decides whether to update the incumbent with one of the newly proposed points, i.e,, x; + Su or X; - ōu. In LexiFlow, this decision is made via a carefully designed Update function. Following the same spirit of existing randomized direct search-based single-objective HPO method (Wu et al. [021]. LexiFlow includes the restart and dynamic stepsize techniques to free the algorithm from local optimum and manual configuration of stepsize (line 9-11 of Alg. []] We also adopt the same setting of the initial stepsize öinst and stepsize lower bound Stower with the aforementioned work. The algorithm terminates if it runs out of resource budget and outputs a lexi-optimal configuration x", which is maintained in a way such that it is lexi-optimal among all the evaluated configurations. Following the general principles of direct search (Kolda et al., 2003), the incumbent point shall be maintained in a way such that it is the best or at least a very promis- ing point among the historically evaluated points (within a local search thread without considering restart). This general principle, together with an iterative random sampling of new points around the incumbent, could gradually lead the search to more promising region (Kolda et al., 2003) Wu et al. 2021]. The algorithm makes decision whether update the incumbent with newly proposed point via the Update procedure attached at the end of Alg. (line 12-18).
Algorithm 1: LexiFlow
Input: Objectives F(-), goals C' (optional) and tolerances I (optional).
Initialization: Initial configuration Xo. Y' r 0, 8 dinit; Obtain F(xo). and x" < Xo. H. < {x ZH <- F(xo)
while = 0, 0, do
Sample u uniformly from unit sphere & if Update(F(x: +8u). Fxx,). Zze) then x;+1 f- x, ou, 1' 4- else if Updato(P(xx - 5u), F(x.), Zx) then X.+1 + X, - ōu, 1' + else Xi+1 t Xx, 5 f- 8 H. + HU (x(+))). }. and update Zn according to Eq. if 2d-1 then s + 0,6 t 6/(6' + I))
10
i < slower then
11
r rrr,,,1 x 1, N(xo, I). 6 + dinit + // Random Restart
12
Procedure Update(F(x'), F(x), Zu):
if F(x) ((Z)) F(x) Or (F(x') ((mn) F(x) and F(x') X F(x)) then
13
14
if F(x') 300) F(x') Or (F(x') ((z)) F(x') and F(x') x1 F(x*)) then
15
x" t 11 Using x' to keep track of the Lexi-optima solution // Accept x'
16
Return True
17
else
18
Return False
// Discard x'
19
Output: lexi-optimal configuration
Update Function. Existing work on randomized direct search (Kolda et al. 2003 Wu et al. 202 I
especially the one about cost-frugal HPO (Wu et al. 2021) indicates the following desiderata for do- ing cost-efficient randomized direct search: (1) Whenever a newly proposed point is better than the incumbent, it should be accepted as the new incumbent. This property time performance of the algorithm and avoids getting stuck into local optimum while maintaining good convergence rate. (2) The algorithm should not accept a point that is strictly worse than the incumbent. This property can help control the evaluation cost in each individual trial according to theoretical analysis of the cost of randomized direct search method (Proposition 4 and in (Wul et al. (02T). Our Update function is designed following these desiderata.
help achieve a good any-
One seemingly straightforward idea for the Update function is to directly make comparisons be- tween the incumbent and the newly proposed points according to the vanilla lexicographic relations defined in Eq. (2) Eq. (3) and Eq. (4). However, one problem with vanilla lexicographic relations is that they cannot accommodate user-specified tolerances and goals on the objectives. To address this challenge, we introduce the targeted lexicographic relations, which are parameterized by a K- dimensional relled target vector Z- [2t, 2² ..... = and denoted by =Z), <()) 3())
F(x') =() F(x) / (x)) = /² (x*(x) & A (x) & 2
Vk E [K]
(5)
F(x') <(2) F(x) + 3K E [K] (x(x <
2 A (x) (2) F - (x') (6)
F(x') 3(2) F(x) + F(X') -12) F(x) V F(x') ((2) F(x)
5

---

Published as a conference paper at ICLR 2023
in which Fk- denotes vector with the first k - dimensions of F(x), i..,, F (x)
[f' fhh (xx] (We define (x) = 0, Vx E X). It is easy to verify that the relation =(z) is reflexive in X and we prove in Lemma [ (in Appendix (A) that it is also transitive in X. With these properties, it can be verified that when 7 is set to the lexi-targets Z. = [z|, ze...., 24" the targeted lexicographic relations exhaustive pairwise comparisons of configurations from the search space X.
be used to find points that are in X,K, i.e., the lexi-frontiers, through
Another caveat in this idea is that the comparisons depend on the knowledge of lexi-targets z vk E [K] which are generally unknown during the optimization process. To address this difficulty, we propose maintain an online approximate of the lexi-targe lexi-targets based on historical observations. Specifically, we maintain all historically evaluated points in H. and introduce the following statistics of H. based on Eq. 1 X (0) H and Ak [K],
" == (c. f + min f (xxx = fx x 15 S 2 x
(8)
Here 2it fir and X based on historical observations in H. With the approximated lexi-targets Zu [2 2 ..... =H1, we decide whether to accept a newly proposed point x' comparing the the current incumbent x according to the Update procedure in Alg.
" can be considered online versions of = f and X respectively, calculated
1 Specifically, we accept x' as the new incumbent in the following two cases: (1) When F(x') <(Zx) F(x) true. This condition allows us to leverage the targeted lexicographic relation <(zx) to direct the search toward the approximated lexi- frontiers based on the approximated lexi-targets Zu. (2) When F F(x') ((xx) F(x) and F(x') Xi F(x). This condition allows to move toward lexi-optimal points (also approximated) when the proposed point is already an approximated lexi-frontier. The update rules on the one hand respect the lexicographic relations considering the existence of lexi-targets and on the other hand follow the established desiderata. In Alg. [ we use x' to keep track of the lexi-optimal solution and update it whenever needed in a similar way in the Update procedure (line 13-15). Note that when there is no random start, x. is simply the current incumbent, i.e., x; at iteration. The extra update steps in line 13-15 are essential considering the potential random restarts.
EXPERIMENTS
We perform evaluations on multiple hyperparameter tuning tasks, which fall into the following two categories of use case scenarios: (1) To directly perform lexicographic optimization in cases where the users do have lexicographic preferences over multiple objectives. (2) To optimize a particular objective that is inaccessible during the optimization, by finding lexi-optimal solutions in terms of a list of proxy objectives. For the first type of use case, we include an efficient NN tuning task, For the
an unfairness mitigation task, and a high-dimensional feature selection task in Section second type of use case, we include a task about overfitting mitigation in Section 4.2. We tune three types of models including Xgboost, Random Forest and Neural Networks different classification tasks. In all the experiments, if not otherwise specified, we show the results of the lexi-optimal solutions calculated based on the historical observations in each method at each particular concerned time point. All reported results are averaged over five runs with different random seeds.
4.1 HPO WITH LEXICOGRAPHIE PREFERENCES OVER MULTIPLE OBJECTIVES
Since there are no existing HPO methods that are directly applicable to the Lexi-HPO problem, we include three general HPO methods which could be extended to the Lexi-HPO problem: (1) A single objective HPO algorithm (SO-HPO), named CFO (Wu et al. 2021). which only optimizes the objec- tive of the highest priority. The method CFO is selected as it is a strong SO-HPO method showing superior performance over other alternatives, e.g., BO-based methods, multi-fidelity-bue -based meth- ods as reported in (Wu et al. 202T). (2) A constrained optimization method (C-HPO) as described and constructed in Section 2.2 This method allocates even resources to different objectives. (3) A popular multiple objective HPO algorithm (MO-HPO) qEHVI (Daulton et al. 2020). We choose qEHVI in this category because it is a state-of oh-the-art MO-HPÒ method with ready-to-use open- source implementation. We use both CFO and its constrained optimization version implementation from the AutoML library FLAML ( (Wang et al. 202I), and use the implementation for qEHVI from the HPO library Optuna (Akiba et al." 2019). In all the experiments, when presenting the anytime
6

---

Published as a conference paper at ICLR 2023
performance, a horizontal dashed line labeled Lexi-Target" to represent the lexi-targel cal- culated based on user imputs and the configurations found by all the methods (unknown before the optimization) according to Eq. (8)
4.1.1 TUNING ACCURATE AND EFFICIENT NEURAL NETWORKS
Deploying neural networks practical applications typically requires a balance between model ac- curacy and efficiency. Lexico-HPO can be a good way to find models achieving the desired balance. For example, as mentioned in (He et al. 2018), for Al applications such as Google Photos, model accuracy is more important than latency, and it is desirable to find smaller models under the condi- tion of not sacrificing accuracy. There of improving model accuracy and reducing latency. In this experiment, we evaluate for the task of tuning accurate and efficient neural networks. More specifically, following a similar search space setting with (Abdolshah et al., 2019 -lernández-Lobato et al., 2016), we tune neural network models on a subsampled Fashion MNIST dataset (Xiao et al.) 2017] with two minimization objectives under lexicographic preference. The 1st objective is the error rate. The 2nd objective is efficiency-related objectives including FLOOPs and parameter numbers in two different experiments respectively. For both experiments, we provide tolerance vector T [T' = 1/v5, x² = 0.0), where represents the number of validation data points since the error metric calculated from validation specific goal vector
natural lexicographic preference over the two objectives
method
data can deviate from the true test error in the scale of 1 1/V5. We do not for this task, as mentioned in (He et al., 2018), latency is typically not a hard constraint.
We perform tuning for 2 hours with each method and show the anytime performance of the methods regarding the two objectives in Figure 2 All four methods eventually achieve good performance in terms of the objective of the highest priority, i.e.. the етrоr rate: all of them are within the calculated lexi-target. LexiFlow, SO-HPO, and C-HPO achieve the target value earlier than MO-HPO because they put more priority on the first objective. significant performance improvement in both two experiments compared to all the baselines. MO- HPO finds models with lower FLOOPs and lower parameter numbers than SO-HPO and C-HPO. But the FLOPs and parameter numbers in MO-HPO are still 2-4 times larger than that in LexiFlow when the error rate is within the tolerance range.
terms of the second objective, LexiFlow shows a
125
Ied JelOkeeis
2el Ofjecise
50430
148
1861 poon 12
(a) Finding accurate and fast NN
(b) Finding accurate and small NN
Figure 2: Results from the neural networks tuning task on Fashion-MNIS" Each of the curves represents averaged result of particular method from 5 runs with different random seeds. The shaded area corresponds to 95% confidence intervals.
4.1.2 TUNING ACCURATE AND FAIR XGBOOST
Due to increasing evidence on fairness-related harms (Angwin et al. 2016 Barocas & Selbst 2016) in MI, there is an emerging need to build ML. models that not only accurate but also fair. One way to achieve this goal is to optimize the predictive performance of model (1st objective) while minimizing the unfair bias (2nd objective) under lexicographic preference model tuning (Brook- house & Freitas 2022). Following the same setting with (Brookhouse & Freitas 2022). we perform the Xgboost tuning task with two minimization objectives, including validation loss (1st objective) according (Brookhouse & Freitas 2022). and DSP (2nd objective). which an unfairnss--relted metric calculated based on statistical disparity (Perrone et al.) 202I). We do the evaluation on datasets including Adult, Compas, and German, which are commonly used datasets for fairness-
7

---

Published as a conference paper at ICLR 2023
related tasks (Perrone al. 2021) We set tolerance of 0.05 the validation loss according to (Brookhouse Freitas 2022). and we do not provide the goal value.
We perform tuning for 1 hour with each method and show the final performance in Table ! The results show that LexiFlow can indeed achieve the best performance considering the lexicographic preference over the objectives compared baselines: LexiFlow has similar performance with baselines considering optimality tolerance (0.05) in terms of the 1st objective and achieves a better performance terms of the 2nd objective across all three datasets. Regarding the performance on the 2nd objective, i.e., the DSP value, 0.05 is a commonly used reference value for good DSP (Perrone) al. 2021). LexiFlow reaches 0.016 in Compas and 0.034 in German, where most of the other baselines got larger than 0.05.
Table 1: Results from tuning Xgboost on tasks that involve fairness considerations, All four methods are within the optimality tolerance range in 1st objective. In terms of the 2nd objective, LexiFlow achieves a significant performance improvement compared to baselines.
Compas
German
Adult
Method
Val Loss
DSP
Val Loss
DSP
Val loss
DSP
SO-HPO
0.344 (+0.004)
0.067
0.431 (+0.007)
0.064
0.368 (+0.003)
0.082
MO-HPO
0.340 (+0.000)
0.064
0.472 (+0.048)
0.078
0.408 (+0.043)
0.198
C-HPO
0.354 (+0.014)
0.029
0.424 (+0.000)
0.100
0.369 (+0.004)
0.088
LexiFlow
0.345 (+0.005)
0.016
0.434 (+0.010)
0.034
0.365 (+0.000)
0.060
4.1.3 FEATURE SELECTION IN BIOINFORMATICS
Ohiettec
Logny suacity
IIPO www.-dicb-iteeeeecmoom CO
I0O TO
Figure 3: The results in tuning Xghoost on biological dataset AP Colon Kidney.
In many applications of ML in bioinformatics, the primary goal is to find model with good predic- tive performance. For high-dimension biological data, due to a large number of features, it is also critical to integrate feature selection into the model fitting process, to make the selected features few and stable (Bommert et al.) 2017), which could help provide insights for domain experts.
In this experiment, we verify LexiFlow in a feature selection task following similar setting with (Bommert et al.] 2017) which implicitly formulates the optimization objectives as lexicographic preference. Specifically, we perform the Xghoost tuning biological dataset AP colon Kidney with three minimizing objectives including error rate, feature number, and the instability of the selected features (with descending priority order). We targets (c
0, c2 500, c² 0] and tolerances T = [r- = 0.01, - = 0.0, T = 0.0) according to (Bommert et al. 2017) We use the filter feature selection method from Scikit-learn (Pedregosa et al. 2011) for feature selection.
We perform tuning for 10 seconds and report the anytime performance regarding the three objec- tives in Figure 3 From the results, we observe that regarding the 1st objective, all four methods are within the optimality tolerance range. In terms of the 2nd objective, only LexiFlow reaches the goal in the end. Moreover, LexiFlow achieves significant performance improvement compared to baselines in the 3rd objective. We also include results from all five random seeds in Appendix B
4.2 LEXIFLOW AS AN OPTIMIZATION TOOL TO MITIGATE OVERFITTING
Although low loss on the final test data is the objective of interest in machine learning, validation loss is commonly used proxy metric for test loss since the final test data is inaccessible in
8

---

Published as a conference paper at ICLR 2023
Table 2: Test-time error rate (%) achieved by different methods on different datasets.
Method
Gisette
Madelon
Ginzl prior
Ginal agnostic
Hicresponse
agnostic
Scetc
Christine
SO-HPO
2,46 = 0.08
4.54 0.3
5.00
2.15
2.41 0.2
2.49
26.4
S0-1IPO w/ FS
2,30 0.2
18.5 1 122
5.01 1.44
4.71 1.0.1
2,14 0.7
2.39 1.0.1
2.52 1.007
26.5 L 0.9
LexiFlow w/ PS 2 0,3
16.3 2.5
4.50 = 0.4
4,45 + 0.2
2.19 0.6
2.36 = 0.1
2.49 = 0.0
26.1 = 1.0
the model building and tuning stage. It is well known that there may exist gap between an ML model's ' performance on validation data and test data. Overfitting is a major factor that causes such performance gap. Evidence from existing work show that optimizing machine learning models with fewer but key features could mitigate overfitring problems (Jović et al., (015; Ying 2019 Abdel-Aal 2005). These findings inspire to explore the possibility of incorporating the objective of minimizing feature numbers as an additional proxy objective in ML model tuning. Specifically, we consider the commonly used validation loss as the primary objective and feature number a
secondary proxy objective during HPO. We tune Random Forest (RF) with training and validation data and evaluate the best model's 's test loss on a reserved test dataset as the final performance metric.
More specifically, we include the following methods in comparison: (1) LexiFlow w/ FS, which LexiFlow with two minimization objectives, including validation loss (Ist), and feature number (2nd). (2) SO-HPO w/ FS, which is the single objective HPO algorithm CFO (Wu et al. 2021) with validation loss as its objective.
(3) SO-HPO w/o FS, which is the single objective HPO al- gorithm CFO with validation loss the objective and without feature selection process. Both LexiFlow w/ FS and SO-HPO w/ FS include a feature selection step (indicated by w/ FS) using the same feature selection method that in Section 4.1.3 in addition to RF model hyperparameter tuning. We include eight classification datasets, which are datasets from two previous feature selec- tion studies (Gonzalez al., 2021 Bommert et al.] (020) satisfying the following two conditions: feature number larger than 100 and available on Openml. We use 0.01 or 0.001 as the tolerance on validation loss according to empirical studies from (Gonzalez al. 202T): for each dataset, if the mean validation loss from a default RF model (from sklearn) is larger than 0.1, 0.01 is used as the tolerance, otherwise 0.001. No tolerance or target is imposed on the 2nd objective.
We show the final test result from different methods in Table 2 LexiFlow w/ FS achieves the best performance on most of the datasets (7/8). We also investigate the number of features selected in each method and find that LexiFlow w/ FS achieves the smallest reserved feature ratio (30. 1)) compared with SO-HPO w/o FS (100%) and SO-HPO w/ FS (66.8%). There are two important takeaways from this experiment, considering the nature of the compared methods and their perfor- mance: (1) The good test performance of SO-HPO w/o FS and LexiFlow w/ FS (over SO-HPO w/o FS) indicates that reducing feature number can indeed help mitigate overfitting and thus improve test performance, which is consistent with findings from existing work mentioned; (2) The dominating performance of LexiFlow w/ FS over SO-HPO w/ FS indicates that leveraging feature number secondary objective in addition to validation loss during model tuning is an effective way to further mitigate overfitting. Our method serves as an ideal optimization tool this important endeavor.
CONCLUSION
In this paper, we propose an HPO algorithm named LexiFlow, which could easily incorporate users' lexicographic preferences across multiple objectives in HPO tasks. LexiFlow simple and effec- tive, showing strong empirical performance over a wide spectrum of tuning tasks and application domains. LexiFlow is of good practical importance especially considering the ubiquitous existence of potentially conflicting objectives in modern machine learning systems, such accuracy, latency, model size, robustness, and ethics-related objectives. We also made an interesting finding on over- fitting mitigation: tuning certain proxy objectives with lexicographic structure could help find models that are less likely to overfit. The implementation of our method is available in the open- source AutoML library [LAMI
Link to the documentation page of LexiFlow in FLMAL:
1o/
FLAML/ R.AMML/ccss// does /Use- TTnee Usee-Use--eee Derrr-De Deee---Urr---eiiedde Fww.eeeddnnttiinnnneeeeeee http:///nennnntneeeeeeeeeeeaaann
code example demonstrating the use of LexiFlow to find accurate and fast neural networks: https:
crmoofr github 10/F AMM. http///ccsxxxaapleeee httpp///ao//////ooooooooooooooor
L.xxeeeiieeiiiie cographi cobographh--bbjctttiee

---

Published as a conference paper at ICLR 2023
REFERENCES
RE Abdel-Aal. Gmdh-based feature ranking and selection for improved classification of medical data. Journal of Biomedical Informatics, 38:456 468464468, 2005.
Majid Abdolshah, Alistair Shilton, Santu Rana, Sunil Gupta, and Svetha Venkatesh, Multi-objective bayesian optimisation with preferences over objectives. Advances in Neural Information Process- ing Systems, 2019.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, 2019.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias there's software used across the country to predict future criminals. and it's biased against blacks. ProPublica, 2016.
Solon Barocas and Andrew Selbst. Big data's disparate impact. Calif. Rev. 104:671, 2016.
Martin Binder, Julia Moosbauer, Janek Thomas, and Bernd Bischl. Multi-objective hyperparameter tuning and feature selection using filter ensembles. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020.
Andrea Bommert, Jörg Rahnenfühher, and Michel Lang. A multicriteria approach to find predictive and sparse models with stable feature selection for high-dimensional data. Computational and mathematical methods in medicine, 2017, 2017.
Andrea Bommert, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang. Benchmark for filter methods for feature selection in high-dimensional classification data. Computational Statistics Data Analysis, 143:106839, 2020.
James Brookhouse and Alex Freitas. Fair feature selection with lexicographic multi-objective genetic algorithm. International Conference on Parallel Problem Solving from Nature, 2022.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel mblti-obeective bayesian optimization. Advances in Neural Information Processing Systems, 2020.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. fast and elitist multiob- jective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6:182-197, 2002.
Michael Emmerich and Jan-willem Klinkenberg. The computation of the expected improvement in dominated hypervolume of pareto front approximations. Rapport technique, Leiden University, 34:7-3, 2008
Peter C. Fishburn. Axioms for lexicographic preferences. The Review of Economic Studies, 42(3): 415-419, 1975.
Satvik Garg, Pradyumn Pundir, Geetanjali Rathee, PK Gupta, Somya Garg, and Saransh Ahlawat. On continuous integration/continoous delivery for automated deployment of machine learning models using mlops. In IEEE Fourth International Conference Artificial Intelligence and Knowledge Engineering, 2021.
Chengyue Gong, Xingchao Liu, and Qiang Liu. Automatic and harmless regularization with con- strained and lexicographic optimization: dynamic barrier approach. Advances in Neural Infor- mation P Processing Systems, 2021.
Jesus Gonzalez, Julio Ortega, Juan Jose Escobar, and Miguel Damas. A lexicographic cooperative co-evolutionary approach for feature selection. Neurocomputing, 463:59-76, 2021.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li. and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision, 2018
10

---

Published as a conference paper at ICLR 2023
Daniel Hernández-Lobatte Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive tropy search for multi-objecctvve bayesian optimization. In International conference on machine learning. 2016.
Alan Jovie, Karla Brkić, and Nikola Bogunović. A review of feature selection methods with appli- cations. In international convention on information and communication technology, electronics and microelectronics, 2015.
Knowles. Parego: a hybrid algorithm with on-line landscape approximation for expensive mul- tiobjective optimization problems. IEEE Transactions Evolutionary Computation, 10:50-66, 2006.
Tamara G Kolda, Robert Michael Lewis, and Virginia Torczon. Optimization by direct search: New perspectives some classical and modern methods. SIAM review, 45:385 482, 2003.
Sasu Mäkinen, Henrik Skogström. Eero Laaksonen, and Tommi Mikkonen. Who needs mlops: What data scientists seek to accomplish and how can mlops help? In 1st Workshop on Al Engineering Software Engineering for AI, pp. 109-112, 2021.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelli- gence, pp. 766-776, 2020.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-lear: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Valerio Perrone, Michele Donini, Muhammad Bilal Zafar, Robin Schmucker, Krishnaram Kentha- padi, and Cédric Archambeau. Fair bayesian optimization. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021.
Robin Schmucker, Michele Donini, Valerio Perrone, Muhammad Bilal Zafar, and Cédric Archam- beau. Multi-objective multi-fidelity hyperparameter optimization with application to fairness. In NeurlPS Workshop on Meta-Learningg 2020.
Robin Schmucker, Michele Donini, Muhammad Bilal Zafar, David Salinas, and Cédric Archam- beau. Multi-objective asynchronous successive halving, arXiv preprint arXiv:2106.1266 2021.
Nidamarthi Srinivas and Kalyanmoy Deb. Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary computation, 2:221-248, 1994.
Georgios Symconidis, Evangelos Nerantzis, Apostolos Kazakis, and George A Papakostas. Mlops- definitions, tools and challenges. arXiv preprint arXhv:2201.00/62. 2022.
Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 2021.
Qingyun Wu, Chi Wang, and Silu Huang. Frugal optimization for cost-related hyperparameters. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
Xue Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1 1168, pp. 022022, 2019.
Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposi- tion. IEEE Transactions on evolutionary computation, 11:712-731, 2007.
Eckart Zitzler, Lothar Thiele, and Johannes Bader. Spam: Set preference algorithm for multiobjec- tive optimization. In International Conference on Parallel Problem Solving from Nature, 2008
11

---

Published as a conference paper at ICLR 2023
A PROOF
Lemma 1, The binary relation Z(z) defined in Eq. 7 is transitive, i.e, V, y. E X, F(x) 3(z)
F(y) A Fly) 3(2) F(w)= = F(x) (() F(w).
Proof of Lemma ! According to Eq. 7
F(x) ((z) F(y) A F(y) Z(z) F(w)= 7
(F(x) (()) F(y) V F(x) (2) F(y)) A (F(y) 3(Z) F(w) V Fly) [z) F(w))
(F(x) (2) F(y) A Fly) -2) F(w)) V (F(x) 3(Z) Fly) A Fly) 3(2) F(w))
(9)
v (F(x)= (2) Fly) A Fly) 3(2) F(w)) V (F(x) 3(2) F(y) F (2) F(w))
Statement I: F(x) (2) F(y) A F(y) -2) F(w) + F(x) -(2) F(w).
Proof of Statement I. From Eq. 5). we have,
F(x) = Fly) A F(y) =(2) F(w) =
(r*(x) = 1 (F S A (²) S ) [K])
A (f*(r)- = f'(w) V
(w() S 2 wk E (K1))
+ Wh [K]
= f(w)
V ((P(()) =
((f(x)=}(()))))) ( S²
S x
(10)
V ((F(() S² A (f) ( (*) (f) f 8r (ww))
V (1*()) S=
xt)
((F*(x) ((+((x)=(((*))) --((w) V (x) pd (w) S , vk E 1x1)
F(x) =(2) F(w)
Statement II: F(x) <(2) Fly) A Fly) (2) F(w) = F(x) <(2) F(w)
Proof of Statement II. According to Eq. (6) we have
F(x) <(2) Fly)+ +
3k1 E [K] (x) < fha A > 2/ A F =(2) (x)
(11)
Fly) R(z) F(w)= 6
3kg E [K] p (w) ^ (
ha F (y) (2) (w)
Let k min(k1, )2), according to Statement and Eq. 11 we have:
(xx) (2) F -1)) ^ (FM'
(2) FK (w)) + F"' =(2) Ft'
(2)
(12)
According to Eq. 12 if = ki k22 we have
gk (x) < (x)) (v) pk (wi) a (2 < < (x) = 8 (ww)
(13)
If = 12 < bj. we have
(ft (x) gk (x) < gth (w)) ^ (2x < (w))
(14)
12

---

Published as a conference paper at ICLR 2023
Otherwise when k' = k1 k2 we have
(M (x)
(y) fh (w) ^ (2 f (v) < fr (w)
(15)
Combing Eq. 12 Eq.
13
Eq.
14 and Eq.
15
we have found existing k = k' such that
(f*(x)= < fh W))
Fk-1
(2) pk-1 (w)).
Statement III: F(x) (2) Fly) Fly) 3(2) F(w)= - F(x) (() F(w).
Proof of Statement III. According to the definitions of (z) and 11z in Eq. 5 and Eq. (6).
F(x) (z) F(y) A (F I() F(w)) =
(yki [K] (fth (x) = (x (ph (x) S t A (y) S x\1)
(16)
(3K2 E [K] f (y) < f (w) A (w) 1 - =()) Fd
Then, find k k2, such that,
(I* = (y) / (w)) V (x* S < (
^ (2* x*((w) A (F+ (x) [z) (w))
(17)
(xx)x) 1h (w)) A (s* < (w))) ^ (Fx [() F*
in which indicates F(x) 212 F(w).
Statement IV: F(x) <(z) Fly) F(y) [() F(w) + F(x) 3(2) F(w).
Proof of Statement IV. According to the definitions of (2) and <(2) in Eq. 5 and Eq. (6)
(((x) R(2) Fly)) A (F (2) F(w))
+ (ThE E [K] : fh (x) < f (y) ^ lhh (y) A F =12) phi p
(18)
(Yke E [K] (y)
= (w)) (f S n A (w)
S 2z,z))
Then, we can find a k ki, such that,
(F*(x) < (y)) = (4(w)) A (s* < (t()) = - (x)) ^ (xx =(2) 1k (w))
(F*(x) < (w)) a ( f(()) ^ (F" (x) (z) F (w)) = F(x) x(z) F(w)
(19)
By substituting the conclusions in statement (I), (II), (III) and (IV) into (i), (ii), (iii,, and (iv) in
Eq. 9 respectively, we get:
F(x) 5(2 Fly) f(y)) 3(2) F(w) + (F(x) 2(z) F(w)) v (F(x) -(2) F(w))
(20)
* F(x) 3(z) F(w)
which concludes the proof.
B SUPPLEMENTARY RESULTS
We provide the results for the feature selection task in bio-informatics (Section under different random seeds in Figure 4
13

---

Published as a conference paper at ICLR 2023
Dijerti
Logn 136022
077
400 WAU-wdlddoe
(a) seed =
Ist Utgeeive
2nd LiveRtNN
Tel Chantove
Lego usality
IIO wwl.-sodd-tttee..oo
(b) seed =
Tst Objecive
2nd Uivective
Tel Chactive
Logs suacity
Lo
-20
I0P www.l-laeeeeeeeo www.l--eettttteee
(c) seed =3
1.0
Le Dljletto
2ed Dilective
Ted Oboectoe
Loggs FOURRry
1.0
066
-20
0.0
30
02
05
56
(d) seed =
1.0
Le Dlipette
3ed Directive
Ted Ohgectoe
Logs 1UUREE
1,0
06
1..9
-20
16
45
30
02
05
56
0.2
3.5
(c) seed =
Figure 4: The detailed results in tuning Xgboost on biological dataset AP.Colon Kidney with dif- ferent random seeds.
14

---

Published as a conference paper at ICLR 2023
C EXPERIMENTATION DETAILS
C.I SEARCH SPACE
The detailed search space in tuning Neural Networks, Random Forest, and XGboost are shown in Table 3 Table and Table respectively.
Table 3: Hyperparameters tuned in Neural Network
hyperparameter
type
range
epoch num
int
[1, 20]
layer
int
[1 3]
hidden units num per layer
int
[4, 128]
dropout value per layer
float
[0.2, 0.5]
learning rate per layer
float
[le-5, le-1]
Table 4: Hyperparameters tuned in Random Forest
hyperparameter type
range
max features float
[min(0.1, dat 1.0]
estimators number int
[4, min(2048, drtasiett)s
max leaves int
[4, train size]
Table 5: Hyperparameters tuned in XGboost
hyperparameter
type
range
estimators number
int
[4, min(32768, train ]
max leaves
int
[4, min(32768, dratiidatssize)
max depth
int
[0, 6, 12]
min child weight
float
[0.001, 128]
learning rate
float
[1/1024, 1.0]
subsample
float
[0.1, 1.0]
colsample by tree
float
[0.01, 1.0]
colsample by level
float
[0.01. 1.0]
reg alpha
float
[1/1024, 1024]
reg lambda float
[1/1024, 1024]
C.2 DATE STATISTICS INFORMATION
All datasets used in experiment are available in OpenML. In Table 6 we show the detailed statistics information of the datasets used Section
Table 6: Date statistics information
Dataset statisties
Gisette
Christine
Sceac
Ginsl_prior
aggossic
Bioresponse
Hiva_agnostic
Madelon
1950
val instance
2098
602
867
1058
650
full feature dimension
5000
1636
299
784
1776
1617
500
C.3 DETAILS OF THE VALIDATION LOSS CALCULATION IN SECTION 4.1.2
In section 4.1.2 in order to keep the same experiment setting with the paper (Brookhouse & Freitas 2022), the accuracy metric named the geometric mean of Sensitivity and Specificity from
15

---

Published as a conference paper at ICLR 2023
[Brookhouse & Freitas 2022 to calculate the 1st objective validation loss in the experiment section. More specifically, the accuracy metric geometric mean of Sensitivity and Specificity GMssnx Spe is calculated by:
TP
TN
Sensitivity
TP + FN Specificity
(21)
TN + FP
Ses
V Sensitivity Specificity
Where TP, TN, FN and FP represent the number of true positive instances, the number of true negative instances, the number of false negative instances and the number of false positive instances,
respectively. The validaiton loss reported in Table 1 is calculated by 1 GMssnx Spe. More infor- mation of this metric could be found in (Brookhouse & Freitas 2022).
16


Reviews:
[{"review_id": "fgmCs0c7125", "reviewer": "Reviewer_Fdr2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In machine learning it is relatively common to have multiple objectives, typically arranged in some loose descending order of importance - for example we typically care most about accuracy, but feature counts, fairness, robustness etc are also important.  Multiobjective optimisation is the obvious way to approach such problems, but can be wasteful as it will explore the entire Pareto front, not just the regions the operator is interested in, so for example it will spend just as much time exploring extremely fair solutions with terrible accuracy - or vice-versa - as it will spend exploring more balanced solutions.\n\nThe paper proposes using lexicographic preferences as a natural way to express the ordering over the objectives, and subsequently proposes an algorithm that seeks solutions satisfying these lexicographic preferences rather than just a batch of Pareto optimal \"solutions\".", "summary_of_the_review": "Overall I think this is a good paper on an important problem.  I have some reservations regarding the clarity of the presentation of the algorithm, but on balance the positives outweigh the negatives.", "strength_and_weaknesses": "Strengths:\n\n- The problem is well motivated and thoroughly described; and the proposed use of lexicographic preferences is well justified.\n- The problem formulation is mostly clear (but see caveats below).\n- In general I was able to follow the development of the algorithm and its description.\n\n\nWeaknesses:\n\n- The algorithm and its description are somewhat hard to parse.  After multiple read-throughs I think I have a reasonable understanding (but not perfect - see questions below), but it would have been easier if the larger \"text chunks\" (e.g. paragraph 2 on page 5) were rewritten using dot-points, tables etc to improve readability.  You may also want to consider motivating the approach taken in the algorithm (what is it based on and why) before presenting it.\n- Have you considered the convergence properties of this algorithm (\"big-O\" behaviour, regret bounds etc).\n\n\nMinor points:\n\n- The presentation was inconsistent in the results section - for example the highlighting in table 1 (bold for all within the error bars of the best result) and table 2 (bold only for the \"best\", ignoring error bars).\n- Why inf (text) in some places, $\\infty$ in others?\n\n\nQuestions:\n\n- Regarding (1), do you intentionally allow for solutions that do not reach the goal for some objectives?  This appears to be the case (z_*^k is max of the goal and \"best we can do plus tolerance\" if I'm reading it correctly) - perhaps some commentary on why such solutions are acceptable would be of assistance here?\n- Was there a reason (technical or otherwise) for using randomised direct search rather than an algorithm specifically designed to deal with expensive optimisation problems (Bayesian optimization or similar)?\n- You mention that the solution found is (potentially) simply one sample of a region of the Pareto front.  What if the algorithms finds multiple such solutions?  Or is this not possible?\n- What is the purpose of lines 9-12 in the algorithm?  In particular, where do the various \"update equations\" here come from?\n- What is the termination condition for the algorithm?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_RgkTbC4pkA", "reviewer": "Reviewer_GSAk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Targeted Hyperparameter Optimization with Lexicographic Preference Over Multiple Objectives\" proposes a novel algorithm for optimizing hyperparameters with multiple objective functions. In their approach, the authors assume a total order over the objective functions to be given and accordingly sort candidate hyperparameter configurations in lexicographic order. To optimize the objectives effectively the authors propose a directed search algorithm and find their method LexiFlow to produce strong results.", "summary_of_the_review": "All in all, I think that this paper makes an interesting contribution and the proposed method shows reasonable performance across tuning tasks and also within tuning tasks compared to state-of-the-art methods.", "strength_and_weaknesses": "Strengths:\n- Interesting method with a lot of potential\n- Strong performance compared to existing state-of-the-art multi-objective HPO methods\n- Robust and strong anytime performance\n\nWeaknesses:\n- Requires the order of objective functions as an input by the user (who might not be able to express such an order)\n- Requires the user to specify goals/thresholds which might be difficult to provide for black-box objectives. Meaning to provide such information the user needs extensive knowledge about the black-box objectives.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ydepr8LSxZl", "reviewer": "Reviewer_NjRw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new formulation of hyperparameter optimization over multi-objectives, LexiHPO, in which the user can specify a lexicographic ordering regarding the priority of each objective. The paper then introduces an algorithm for solving this problem, LexiFlow, which seems to be an extension of recent work on randomized directed search (Wu et al. 2021) to account for the new lexicographic setup. This algorithm is then evaluated on a number of different multi-objective HPO tasks in which the objectives are ordered: finding accurate and fast/small neural networks, finding accurate and fair gradient boosting machines, finding accurate models on a bioinformatics tasks that use a small number of features. Additionally, the paper argues that LexiFlow can be used as a tool to reduce overfitting by transforming a single-objective HPO tasks (e.g. minimizing validation loss) into a dual-objective optimization task in which the second objective is designed to reduce overfitting to the validation set (e.g., by restricting the number of features). Experimental results are provided to demonstrate this idea. ", "summary_of_the_review": "This is a well-written paper with an interesting new perspective on multi-objective hyperparameter optimization. The paper could be improved with more discussion of how their setup relates to the existing work of multi-objective optimization with preferences, and the experimental results in Section 4.2 could be made more convincing by using more datasets and/or statistical hypothesis testing. ", "strength_and_weaknesses": "Strengths:\n- The LexiHPO framework is novel to the best of my knowledge, and may be more intuitive to practitioners than dealing with multi-objective approaches that discover Pareto frontiers. \n- The LexiHPO algorithm is described well and the authors provide intuition about how it was designed. \n- Extensive experimental results are provided that demonstrate the performance of the algorithm, but also the use-cases presented help to justify the idea of introduce a lexicographic ordering into multi-objective optimization. \n\nWeaknesses:\n- Only a few sentences are devoted to existing work regarding multi-objective optimization with preferences. I think the paper would be improved if a little more spaced was devoted to explaining why LexiHPO is somehow a better way to incorporate preferences than Paria et al. 2020, Abdolshah et al. 2019 and Zitzler et al. 2008, and also why none of these methods could be used as experimental baselines.\n- In Section 3 it is stated: \"However one problem with the vanilla lexicographic relations is that they cannot accommodate user-specified tolerances and goals on the objectives.\". I found the flow here a little confusing - wouldn't it be better to introduce equations (5), (6), (7) in Section 2.1 since one can argue they fundamentally relate to the setup itself, rather than the algorithm. Whereas the algorithm is responsible for computing the online statistics in equation (8)? \n- The datasets used in Section 4.2 are very small and the results are not entirely convincing statistically. While I feel that the contributions of the paper stand without this section, I think that in order to convincingly argue that LexiFlow can be used to reduce overfitting it would be necessary to use more datasets and/or perform rigorous statistical hypothesis testing. \n\nMinor corrections:\n- In the example at the bottom of page 3 it is stated the primary objective is the model's prediction error, but later in the example it is referred to as \"loss\".\n- In the same example, it is stated that the goal vector should contain +infinity in the first position, shouldn't this be -infinity?\n- \"We also urge methods that are efficient and cost-frugal\" -- \"urge\" is probably the wrong choice of word in this sentence.\n- In Figure 2 and Figure 3 - is there a reason why the LexiTarget is not shown for all objective?\n- Section 4.1.3 last paragraph - Figure 4 is referenced whereas I think it should be Figure 3?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%