Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper "Targeted Hyperparameter Optimization with Lexicographic Preference Over Multiple Objectives" proposes a novel algorithm for optimizing hyperparameters with multiple objective functions. In their approach, the authors assume a total order over the objective functions to be given and accordingly sort candidate hyperparameter configurations in lexicographic order. To optimize the objectives effectively the authors propose a directed search algorithm and find their method LexiFlow to produce strong results.
strength_and_weaknesses: Strengths:
- Interesting method with a lot of potential
- Strong performance compared to existing state-of-the-art multi-objective HPO methods
- Robust and strong anytime performance

Weaknesses:
- Requires the order of objective functions as an input by the user (who might not be able to express such an order)
- Requires the user to specify goals/thresholds which might be difficult to provide for black-box objectives. Meaning to provide such information the user needs extensive knowledge about the black-box objectives.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well structured and well written. The explanations are easy to follow and the logic flows are reasonable.

The quality of the paper is very good. The method is well motivated, however, still imposing some assumptions which might be non-trivial. Still, for such a first work, such strong assumptions might be reasonable and be resolved in follow-up work.

The methodology is novel to the best of my knowledge.

A good level of detail is given in the paper such that the work should be reproducible.
summary_of_the_review: All in all, I think that this paper makes an interesting contribution and the proposed method shows reasonable performance across tuning tasks and also within tuning tasks compared to state-of-the-art methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In machine learning it is relatively common to have multiple objectives, typically arranged in some loose descending order of importance - for example we typically care most about accuracy, but feature counts, fairness, robustness etc are also important.  Multiobjective optimisation is the obvious way to approach such problems, but can be wasteful as it will explore the entire Pareto front, not just the regions the operator is interested in, so for example it will spend just as much time exploring extremely fair solutions with terrible accuracy - or vice-versa - as it will spend exploring more balanced solutions.

The paper proposes using lexicographic preferences as a natural way to express the ordering over the objectives, and subsequently proposes an algorithm that seeks solutions satisfying these lexicographic preferences rather than just a batch of Pareto optimal "solutions".
strength_and_weaknesses: Strengths:

- The problem is well motivated and thoroughly described; and the proposed use of lexicographic preferences is well justified.
- The problem formulation is mostly clear (but see caveats below).
- In general I was able to follow the development of the algorithm and its description.


Weaknesses:

- The algorithm and its description are somewhat hard to parse.  After multiple read-throughs I think I have a reasonable understanding (but not perfect - see questions below), but it would have been easier if the larger "text chunks" (e.g. paragraph 2 on page 5) were rewritten using dot-points, tables etc to improve readability.  You may also want to consider motivating the approach taken in the algorithm (what is it based on and why) before presenting it.
- Have you considered the convergence properties of this algorithm ("big-O" behaviour, regret bounds etc).


Minor points:

- The presentation was inconsistent in the results section - for example the highlighting in table 1 (bold for all within the error bars of the best result) and table 2 (bold only for the "best", ignoring error bars).
- Why inf (text) in some places, $\infty$ in others?


Questions:

- Regarding (1), do you intentionally allow for solutions that do not reach the goal for some objectives?  This appears to be the case (z_*^k is max of the goal and "best we can do plus tolerance" if I'm reading it correctly) - perhaps some commentary on why such solutions are acceptable would be of assistance here?
- Was there a reason (technical or otherwise) for using randomised direct search rather than an algorithm specifically designed to deal with expensive optimisation problems (Bayesian optimization or similar)?
- You mention that the solution found is (potentially) simply one sample of a region of the Pareto front.  What if the algorithms finds multiple such solutions?  Or is this not possible?
- What is the purpose of lines 9-12 in the algorithm?  In particular, where do the various "update equations" here come from?
- What is the termination condition for the algorithm?
clarity,_quality,_novelty_and_reproducibility: Clarity:

Clarity is mixed.  The introduction and formulation of this problem are mostly clear. However the description of the algorithm and, to a lesser extent, the experiments is rather difficult to parse (see comments above).


Quality/Novelty:

The problem is important and the solution appears novel, and while the paper has some issues I would say that the quality is good overall.

Reproducibility:

Given the details in the paper I am reasonably certain that I could reproduce at least a reasonable representation of the results.
summary_of_the_review: Overall I think this is a good paper on an important problem.  I have some reservations regarding the clarity of the presentation of the algorithm, but on balance the positives outweigh the negatives.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new formulation of hyperparameter optimization over multi-objectives, LexiHPO, in which the user can specify a lexicographic ordering regarding the priority of each objective. The paper then introduces an algorithm for solving this problem, LexiFlow, which seems to be an extension of recent work on randomized directed search (Wu et al. 2021) to account for the new lexicographic setup. This algorithm is then evaluated on a number of different multi-objective HPO tasks in which the objectives are ordered: finding accurate and fast/small neural networks, finding accurate and fair gradient boosting machines, finding accurate models on a bioinformatics tasks that use a small number of features. Additionally, the paper argues that LexiFlow can be used as a tool to reduce overfitting by transforming a single-objective HPO tasks (e.g. minimizing validation loss) into a dual-objective optimization task in which the second objective is designed to reduce overfitting to the validation set (e.g., by restricting the number of features). Experimental results are provided to demonstrate this idea. 
strength_and_weaknesses: Strengths:
- The LexiHPO framework is novel to the best of my knowledge, and may be more intuitive to practitioners than dealing with multi-objective approaches that discover Pareto frontiers. 
- The LexiHPO algorithm is described well and the authors provide intuition about how it was designed. 
- Extensive experimental results are provided that demonstrate the performance of the algorithm, but also the use-cases presented help to justify the idea of introduce a lexicographic ordering into multi-objective optimization. 

Weaknesses:
- Only a few sentences are devoted to existing work regarding multi-objective optimization with preferences. I think the paper would be improved if a little more spaced was devoted to explaining why LexiHPO is somehow a better way to incorporate preferences than Paria et al. 2020, Abdolshah et al. 2019 and Zitzler et al. 2008, and also why none of these methods could be used as experimental baselines.
- In Section 3 it is stated: "However one problem with the vanilla lexicographic relations is that they cannot accommodate user-specified tolerances and goals on the objectives.". I found the flow here a little confusing - wouldn't it be better to introduce equations (5), (6), (7) in Section 2.1 since one can argue they fundamentally relate to the setup itself, rather than the algorithm. Whereas the algorithm is responsible for computing the online statistics in equation (8)? 
- The datasets used in Section 4.2 are very small and the results are not entirely convincing statistically. While I feel that the contributions of the paper stand without this section, I think that in order to convincingly argue that LexiFlow can be used to reduce overfitting it would be necessary to use more datasets and/or perform rigorous statistical hypothesis testing. 

Minor corrections:
- In the example at the bottom of page 3 it is stated the primary objective is the model's prediction error, but later in the example it is referred to as "loss".
- In the same example, it is stated that the goal vector should contain +infinity in the first position, shouldn't this be -infinity?
- "We also urge methods that are efficient and cost-frugal" -- "urge" is probably the wrong choice of word in this sentence.
- In Figure 2 and Figure 3 - is there a reason why the LexiTarget is not shown for all objective?
- Section 4.1.3 last paragraph - Figure 4 is referenced whereas I think it should be Figure 3?

clarity,_quality,_novelty_and_reproducibility: The paper is written to a high standard and is easy to understand. In terms of originality, I think the LexiHPO setup is an interesting new take on multi-objective optimization. The algorithm presented is perhaps less original, seeming to be mostly an adaptation of recent work in randomized direct search to this new setup. In terms of reproducibility, I have checked the anonymous repository associated with the paper and it seems that code is provided to reproduce all results and baselines. 
summary_of_the_review: This is a well-written paper with an interesting new perspective on multi-objective hyperparameter optimization. The paper could be improved with more discussion of how their setup relates to the existing work of multi-objective optimization with preferences, and the experimental results in Section 4.2 could be made more convincing by using more datasets and/or statistical hypothesis testing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all appreciated the quality of the work with its strong empirical results in a clear, well-written manuscript. The proposed multi-objective HPO method will inform future research. They thank the authors for their response and their efforts during the rebuttal phase, which further improves the submission. They enthusiastically recommend the paper for acceptance.
justification_for_why_not_lower_score: * Unanimously supported by the reviewers, and confidently so.
* Strong performance gains in an important and widespread problem (multi-objective hyperparameter optimization).
Decision: Accept: notable-top-5%