Title: UQ-Guided Hyperparameter Optimization for Iterative Learners
Venue: NeurIPS 2024
Accepted: yes
Keywords: Uncertainty quantification; Hyperparameter Optimization; iterative learners
Abstract: Hyperparameter Optimization (HPO) plays a pivotal role in unleashing the potential of iterative machine learning models. This paper addresses a crucial aspect that has largely been overlooked in HPO: the impact of uncertainty in ML model training. The paper introduces the concept of uncertainty-aware HPO and presents a novel approach called the UQ-guided scheme for quantifying uncertainty. This scheme offers a principled and versatile method to empower HPO techniques in handling model uncertainty during their exploration of the candidate space.
By constructing a probabilistic model and implementing probability-driven candidate selection and budget allocation, this approach enhances the quality of the resulting model hyperparameters. It achieves a notable performance improvement of over 50\% in terms of accuracy regret and exploration time.

UQ-Guided Hyperparameter Optimization for Iterative Learners

Abstract

Hyperparameter Optimization (HPO) plays a pivotal role in unleashing the potential of iterative machine learning models. This paper addresses a crucial aspect that has largely been overlooked in HPO: the impact of uncertainty in ML model training. The paper introduces the concept of uncertainty-aware HPO and presents a novel approach called the UQ-guided scheme for quantifying uncertainty. This scheme offers a principled and versatile method to empower HPO techniques in handling model uncertainty during their exploration of the candidate space. By constructing a probabilistic model and implementing probability-driven candidate selection and budget allocation, this approach enhances the quality of the resulting model hyperparameters. It achieves a notable performance improvement of over 50% in terms of accuracy regret and exploration time.

1 Introduction

Hyperparameter optimization (HPO) is essential for unleashing the power of iterative machine learning models [4, 18, 40]. Hyperparameters include traditional parameters like learning rates and more complex ones like neural architectures and data augmentation policies. For iterative learners (multi-fidelity and early stopping methods specifically), practitioners can obtain intermediate validation loss after each iteration and use them for model assessment; the main goal of HPO is to explore a vast candidate space to find candidates that lead to optimal model performance.

There are many designs in the literature to solve the HPO problem. Successive Halving (SH) [19], for example, terminates training of candidate configurations with poor performance early so as to save computing resources for more well-behaved candidates. Bayesian optimization [17, 34], another optimization method, uses a surrogate model to guide the selection of candidate configurations for assessment.

There is however a lack of systematic treatment of an important factor in HPO designs, the uncertainty inherent in the dynamics of the training process of iterative machine learning applications. Because of the uncertainty, a model with a candidate hyperparameter configuration (or candidate in short) performing poorly in an early stage of its training could turn out to be the best model after convergence. Such candidates are however likely to be stopped from proceeding further or be completely discarded by existing HPO methods in the early stages, because their selections of candidates are mostly based on the currently observed performance, for lack of a way to treat the uncertainty properly. In 100 experiments of Successive Halving, for instance, the actually best candidates were discarded in the first 8-22 iterations of training, causing 48% performance regrets in validation loss (details in Section 3.1 Figure 1 and Section 4 Figure 3).

This paper introduces model uncertainty into the design of HPO methods for iterative learners and establishes the concept of uncertainty-aware HPO . At the core of uncertainty-aware HPO is a novel uncertainty quantization (UQ) guided scheme, named UQ-guided scheme , which unifies the selection of candidates and the scheduling of training budget-two most important operations in HPO-into a single UQ-based formal decision process. The UQ-guided scheme builds on a probabilistic uncertainty-based model, designed to approximate the statistical effects of discarding

a set of candidates at the end of a step in HPO. It uses a lightweight method to efficiently quantify model uncertainty on the fly. It offers a principled, efficient way for HPO to treat model training uncertainty.

As a general scheme, the UQ-guided scheme can be integrated into a variety of HPO methods for iterative learners, especially DNNs. This paper demonstrates its usefulness and generality for DNNs by integrating it into four existing HPO methods. Experiments on two widely used HPO benchmarks, NAS-BENCH-201 [9] and LCBench [42], show that the enhanced methods produce models that have 21-55% regret reduction over the models from the original methods at the same exploration cost. And those enhanced methods need only 30-75% time to produce models with accuracy comparable to those by the original HPO methods. The paper further gives a theoretical analysis of the impact of the UQ-guided scheme for HPO.

2 Background and Related Work

Many studies are committed to solving the HPO problem for iterative learners efficiently [26, 20, 29, 39, 41]. Bayesian optimization, early stop-based mechanisms, and multi-fidelity optimizations are some important approaches.

Bayesian Optimization (BO) . BO is a sequential design strategy used to optimize black-box functions [34, 17, 11]. In HPO scenarios, it can be used as a surrogate model to sample high-quality candidates.

Early Stop Mechanisms. Early stop-based approaches can be effective since they evaluate different candidates during training and make adaptive selections accordingly [2, 8, 35]. The early stopping mechanism, which stops the training of poorly-performed candidates early, has been widely employed in the HPO community including Successive Halving (SH) [19] and Hyperband (HB) [25]; BOHB [11] combines both BO and HB methods to take advantage of both the BO surrogate model and the early stopping mechanism.

Multi-fidelity Optimizations. Multi-fidelity evaluation focuses on using low-fidelity results trained with small resources to accelerate the evaluation of candidates [2, 3, 8, 22, 23, 38, 21, 35, 36, 14, 26]. Sub-sampling (SS) [15] is proposed mainly using multi-fidelity methods to collect high-quality data to select good configurations without early stopping.

Model Uncertainty in HPO. Various optimization methods in HPO scenarios focus on specific training metrics to assess candidate performance. However, these methods typically overlook the uncertainty in the candidate selection process. Machine learning models inherently have approximation uncertainties [5, 6, 10, 12, 24, 28, 31]. Some HPO designs sample the candidate space based on distributions on the effect of each hyperparameter dimension on the quality of the candidates, but without considering the uncertainty in the model training process. For example, one of the studies [33] separates candidates into 'good' or 'bad' groups in order to build the distributions. The separation is based on the same deterministic metrics as other HPO methods use, giving no consideration of the uncertainty in model trainings. The only work we find that considers uncertainty in training metrics [32] selects configurations for further training based on its assessment of the upperbound of those configurations. In each round, the configurations it chooses are those that, considering the best possible performance at the last iteration, show a smaller validation loss than the validation loss current best configuration shows. The selection treats model uncertainty preliminarily and does not use it to guide the allocation of training budget. We compare other related works in Appendix E.

3 Uncertainty Quantification (UQ)-Guided Hyperparameter Optimization

This section gives an exploration of model uncertainty, introduces UQ-guided scheme for incorporating UQ into the design of HPO, discusses examples of ways to use the UQ-guided scheme to enhance existing HPO methods, and theoretically analyzes its effects.

3.1 Uncertainty in Iterative Machine Learning

Uncertainty in iterative machine learning originates mainly from two factors: inherent noise in the data and the variability of model predictions due to restricted knowledge [16, 1]. Since data uncertainty is constant, it is the variability in model predictions, referred to as model uncertainty, that primarily influences decisions on HPO. To estimate the model uncertainty, we can incur small perturbations to the model, evaluate these model variants, and calculate the variance of the results as the approximation for the model uncertainty [16].

Figure 1 shows how the model uncertainty affects the quality of the returned candidate. In a given SH run, half of the candidates are eliminated at each checkpoint marked by a vertical red dashed line. The solid blue line represents the best validation loss up to the current point, while the orange dashed line signifies the true quality (in terms of validation loss after convergence) of the candidates SH retains at that specific juncture. From the figure, we see that in every round, SH discards the actually best candidates, causing a continuous increase of the regret. The reason is that the discarding decision of SH is solely based on the current validation loss, but model uncertainty, particularly pronounced in the early stages, obfuscating the true model potential.

3.2 Quantify Uncertainty and the Impact

We explain how we quantify model uncertainty, and how, based on it, at any point of time, estimate the performance and uncertainty of a candidate model when its training converges.

High efficiency is essential here as the UQ happens during the HPO process. We employ a lightweight approach to conduct the UQ efficiently on the fly, as explained next.

Let γ 1 , γ 2 , · · · , γ K ∈ Γ be K candidates drawn from the hyperparameter space Γ . Consider a supervised learning setting, where a machine learning model M is trained on some training set D Train = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , · · · , ( x n train , y n train ) } . Let M t γ denote the model with hyperparameter γ trained on D Train after t epochs, and M ∗ γ the converged model. M t γ ( x ) gives the prediction on a given input x ∈ R d .

0

1

2

3

4

5

0

20

40

60

80

100

Validation Loss

Iteration

Current Best

Final Best

Ground Truth

Last halving

3

rd

halving

2

nd

halving

1

st

halving

Regret

We use ℓ ( · , · ) to denote the metric that evaluates the performance of a candidate model. Given a hyperparameter configuration γ c , the validation loss of a model instance, ℓ ( y , M ∗ γ c ( X )) , can be affected by training data and other factors, and hence has some uncertainty. Let N ( µ c ∗ , σ 2 c ∗ ) represent the distribution of ℓ ( y , M ∗ γ c ( X )) , where, σ 2 c ∗ embodies the uncertainty.

Our objective here is a way that can, at any point in the HPO process, estimate σ ∗ 2 c and µ ∗ c for a candidate model configured with γ c . Our solution leverages the validation history in the HPO process, ℓ ( y , M 1 γ c ( X )) , · · · , ℓ ( y , M t γ c ( X )) , to construct an estimated loss curve, explained as follows.

Decomposition of momentum and the underlying structure of the metric. This part uses breakdowns to characterize the loss curve and introduces the objective function we want to minimize to estimate the curve parameters. The first component, referred to as momentum, models the decaying trend of the loss curve. The second component is the bias term for each candidate's loss curve; it models a latent effect underlying the hyperparameter space by allowing correlation among the candidates. The details are as follows.

Typically a candidate in HPO can be represented as a vector. We use a set U r ⊂ R r to represent the candidates { γ i , i ∈ [ n ] } ; r > 0 is the vector length. For modeling the loss curve, we set k t ∈ R L as a vector whose elements are functions of training epochs t = 1 , 2 , · · · . (In our experiments, we set k t = [ t -1 / 2 , t -1 ] , for the general decreasing trend of loss curves as training epochs increase.) We model metric ℓ u t of candidate u ∈ U r at time t as the summation of the impact from three sources:

where η u and Z are the parameter vectors to determine. The three components in Equation 3.1 are (1) the momentum part ( k ⊤ t η u ) that denotes the contribution from the trends in the loss curve of the training specific candidate model variant, (2) the contribution from the underlying model structure ( u ⊤ Z ), and (3) the noises by other elements ϵ u t , which is asssumed to follow a Gaussian distribution N (0 , σ 2 t ) independent of each other and of Z .

For a candidate u ∈ U r , let α u = u ⊤ Z . The loss curve parameters { α u , η u } can be determined by optimizing a weighted least squares objective

where w jt = 1 F t σ 2 t . F t is the number of models trained with time t .

Solving for the Momentum Mean and Variance. This part analyzes the mean and variance of the loss curve we constructed and makes inferences to the converged loss based on the loss curve. The quantified uncertainty of the estimated converged loss is then used for model selection and budget resource allocation. The specifics are as follows.

For a candidate i at iteration T , concatenating the validation losses across training epochs (indexed by t ) will lead to a validation loss vector v of dimension D = ∑ T t =1 F t . For each d ∈ { 1 , · · · , D } , the d th element in v is an observation of loss at time t d that follows N ( µ i t d , σ 2 t d ) with t d mapping d to its corresponding epoch t .

For a given candidate u (for better readability, we omit the superscript u in the notations in the following discussion), the weighted least squares problem can be formulated as solving the equation W 1 2 v = W 1 2 A β for β with W ∈ R D × D being a diagonal matrix of weights W dd = 1 F t d σ 2 t d .

A = [ 1 K ] , K ∈ R D × ( L +1) with K [ d, :] = k ⊤ t d , and β ⊤ = [ α η ⊤ ] . The empirical estimate of σ 2 i is computed as the variance of the loss in the recent several epochs of i -those instances can be regarded as results of small perturbations to the model at epoch i .

Solving the weighted least square objective, we have the estimator as ˆ β = ( A ⊤ WA ) -1 ( A ⊤ Wv ) . The covariance of the estimator is given by Σ ˆ β = ( A ⊤ WA ) -1 .

Since the estimated curve is given by ˆ v ( t ) = [1 k ⊤ t ] ˆ β , the variance of this estimation is given by

With the formulas for ˆ v ( t ) and ˆ σ 2 ( t ) , we can then approximate the distribution of ℓ ( y , M ∗ γ i ( X )) as N (ˆ v ( N ) , ˆ σ 2 ( N )) for a large N value ( N = 200 in our experiments).

Algorithm. Based on the analysis, we devise an iterative algorithm to compute the estimated loss and variance. Without the loss of generality, we make the following assumption.

Assumption 1. There exist positive constants u such that for any r , max u ∈U r ∥ u ∥ ≤ u, and the set of candidates U r ⊂ R r has r linearly independent elements b 1 , · · · , b r .

The algorithm goes as follows. At the beginning of HPO, it sets Z with a random vector. At the end of each epoch, it conducts the following two operations. First, it solves the weighted least squares objective 3.2 for each of the r candidates (mentioned in Assumption 1) by following the formulas described earlier in this section, with the current Z value being used. Second, for p = 1 , 2 , · · · r , it observes the metrics X b p ( t ) = ℓ b p t -k ⊤ t ˆ η b p and refines the ordinary least square estimate for Z as follows:

At the end of an HPO round (e.g., at the halving time in SH), it performs the first step for every candidate model to compute the estimated distributions of their ℓ ( y , M ∗ γ i ( X )) , so that the HPO can use the estimates to select promising candidates to continue in the next round.

We next show how to use the approximated loss and uncertainty to compare two candidates:

Definition 1 (UQ-guided comparison of candidates) . UQ-guided comparison of candidates compares two candidates based on the probability that the validation loss of the converged model γ c 1 is lower than that of γ c 2 , represented as follows based on the approximation from the current validation losses and uncertainty of the two candidates:

K

Round Start

Budget

Ends

k

Round 1

Round 2

…

Round 3

Round r

Probabilistic

Model

Discarding

Mechanism

P

Confidence Curve

k

Configs

Uncertainty Quantification

f

(

(

µi

,

σ

i

2

)

#$%

&

,

k

)

k

Determine the number of

k

models to keep

k

↓

k

↑

µ1

'(()

(

µ2

µK

l

:

probability

distribution of loss

after convergence

l

i

~

N

(

µi

,

σ

i

2

)

l

1

l

2

lK

The main idea of Definition 1 is to use the current validation loss history and quantified uncertainty to approximate the converged validation loss, so that we compare two candidates - more precisely, we compute the probability that one candidate is better than the other - based on the probability distribution of their converged validation loss. For example, if the approximated loss and uncertainty of two candidates γ j and γ k , at epoch t , are ( µ j , σ j ) and ( µ k , σ k ) , using converged validation loss as the metric, we have Pr( ℓ ( y , M t γ j ( X )) > ℓ ( y , M t γ k ( X )))= Φ ( µ k -µ j √ σ 2 j + σ 2 k ) , where Φ denotes the cumulative distribution function (CDF) of the standard normal distribution.

We next present UQ-guided scheme , a principled way to use UQ to guide HPO.

3.3 UQ-Guided Scheme

Figure 2 illustrates how UQ-guided scheme works in HPO. For the purpose of clarity, we base our explanation of the scheme on HPO that uses early stop mechanisms, but will show in Section 3.5 that the scheme is general, applicable to other HPO methods for iterative learners as well.

The original HPO method, Successive Halving [19], evaluates and eliminates candidates over multiple rounds. At the end of each round, it drops those candidates regarded as unpromising. With our UQ-guided scheme, at the end of each round, the scheme derives a confidence curve from the current probabilistic model, and uses a discarding mechanism to drop candidates that are unlikely to perform well after convergence. In contrast to the original HPO that drops a fixed amount (or fraction) of candidates in each round, the UQ-guided scheme carefully calculates the number of candidates to drop in a round based on the probabilistic model such that the expected quality of the HPO outcomes can be maximized, as explained later.

The UQ-guided scheme respects the HPO budget-that is, the total amount of time usable by the HPO for identifying the best candidate. By default, it works around the given budget constraint: the budget for each round ( R ) equals the total budget divided by the number of rounds. We next discuss each step.

3.3.1 Confidence Curve Derived from Uncertainty Quantification

The concept of confidence curve is central in UQ-guided HPO. Define [ n ] = { 1 , 2 , · · · , n } .

Choose discard

Definition 2 (Confidence curve) . At epoch t , we evaluate each candidate's performance and sort them based on validation loss. A confidence curve C is a trajectory of a series of probabilities, { P k | k ∈ [ n ] } , that depicts the probability that the optimal configuration (with the lowest loss after convergence) is among the first k configurations. For k ∈ [ n ] , P k can be expressed as

The confidence curve is derived based on joint probability distribution in the following way. Suppose there are n candidates. At the end of a certain round, the probabilistic model returns n pairs of ( µ 1 , σ 1 ) , ( µ 2 , σ 2 ) , · · · , ( µ n , σ n ) as estimations for ℓ ( y , M ∗ γ 1 ( X )) , ℓ ( y , M ∗ γ 2 ( X )) , · · · , ℓ ( y , M ∗ γ n ( X )) . For simplicity, assume that µ 1 < µ 2 < · · · < µ n , and σ 1 = σ 2 = · · · = σ .

Let Φ and ϕ be the CDF and PDF of the standard normal distribution. For k = m , we can calculate

The details of obtaining Equation 3.4 are in Appendix A.1.

3.3.2 Discarding Mechanism

The next step is to decide, at the end of each round, the appropriate value of k , which determines how many ( n -k ) lowest-ranked candidates will be discarded in this round. Our scheme decides k based on the confidence curve: choosing the smallest k that satisfies P k ≥ τ , where τ is a parameter determined by our scheme adaptively as follows.

Choosing τ . At the end of round i , we have the confidence curve C i ( P i 1 , P i 2 , · · · , P i n ) that is the trajectory of a series of probabilities. We quantify how τ influences the probability for the HPO to select the best candidate.

Let τ i be the value of τ for round i , k i be min { k : C i ( P k ) ≥ τ i } . As the scheme discards the worst n -k i candidates and further trains the best k i candidates in round i +1 , we can derive the confidence curve of round i +1 as C i +1 ( P i +1 1 , P i +1 2 , · · · , P i +1 k i ) based on those selected k i candidates. Since we want to quantify the effect of τ on the probability that the scheme returns the best candidate (that is, to suppose round i +1 is our final round), P i +1 1 is the target we desire to maximize. Define ξ i to be the current condition ( µ i , σ i ) n i =1 . Let f ( · , · ) be a mapping such that f ( ξ i , τ i ) = P i +1 1 . We want to use a selector function Ψ : D → [0 , 1] where D = ([0 , ∞ ) × [0 , ∞ )) n × [0 , 1] . Ψ takes ξ i as input and returns an optimal τ i :

The effect of τ i on f manifests through its influence on the number of candidates k i retained in the subsequent round, and can be ultimately broken down into the influence of (1) exploration , meaning keeping more candidates in the next round can reduce this round's discarding error, and (2) exploitation , meaning keeping fewer candidates in the next round can allow each candidate to receive more training time (recall that the training time budget is fixed for each round) and hence will increase the reliability of the validation at the end of the next round.

Exploration. If k i drops by 1 to k ′ i , according to the definition of the confidence curve , the probability that the final optimal configuration is among the remaining candidates we keep drops by ∆ c ↓ = P k i -P k ′ i :

Exploitation. At the same time, a drop in k i leads to an increase in the individual training budget b . Let ζ be the coefficient that relates the increase in the number of training epochs to its corresponding effect on confidence. Using an approach similar to that employed in formulating the confidence curve ,

where t represents the current total number of epochs and ∆ t σ represents the reduction in the uncertainty σ that would result from training each candidate for one additional epoch. The specifics for Equations 3.6 and 3.7 can be found in Appendix A.2. Given that b increases by R k ′ i -R k i , the overall influence of exploitation on the probability of selecting an optimal candidate is ∆ c ↑ = R k i ( k i -1) ζ .

Let ζ ( k i , ξ i ) be the confidence increase, given condition ξ i , when each of the k i candidates gets a unit extra training budget. C i ( P k ) , k ∈ [ n ] are the confidence curves . Balancing exploration and exploitation leads to a sweet point where ∆ c ↓ = ∆ c ↑ . That gives the way to derive the appropriate value for τ , which just needs to make the following hold:

3.4 Theoretical Analysis

We consider how the method performs in terms of identifying the best candidate. For convenience, we let ℓ i,t be the approximation of the converged loss for the model with hyperparameter γ i at time t . For each i , assume ν i = lim τ →∞ ℓ i,τ exists. The goal is to identify arg min i ν i . Without loss of generality, assume that ν 1 < ν 2 ≤ · · · ≤ ν n . The assumption that lim τ →∞ ℓ i,τ exists implies that as τ grows, the overall gap between ℓ i,τ and ν i decreases. Let σ t = f ( t ) be the model uncertainty at epoch t . We then introduce a random variable that characterizes the approximation error of ℓ i,t relative to ν i , modeling it as a distribution that incorporates t as a parameter:

By applying Chebyshev inequality, we have

Let A denote the event ℓ i,t > ℓ 1 ,t , then by Equation 3.9

Equation 3.10 tells us that ℓ i,t > ℓ 1 ,t has a high probability with respect to t if f ( t ) ∈ O ( t -1 / 4 ) (see Lemma in Appendix C). That is, comparing the intermediate values at a certain time t is likely to establish an order similar to the order of the final values of ν i and ν 1 .

The following theorem is stated using the abovementioned quantities with proofs in Appendix C.1.

Theorem 1. Let n be the number of total candidates, and ν i = lim τ →∞ ℓ i,τ . For a given c > 0 , there exists a T > 0 s.t. n ∏ i =2 (1 -( 4 f ( T ) 2 ( ν i -ν 1 ) 2 ) 2 ) > 1 -c . If the round budget R > T · n , then the best candidate is returned with probability P > (1 -⌊ B ⌋ c )(1 -c ) , where B is the total budget.

R

In comparison, the bound in the UQ-oblivious approach is as follows:

Theorem 2. Let δ > 0 , ν i = lim τ →∞ ℓ i,τ and assume ν 1 ≤ ν 2 ≤ · · · ≤ ν n . Let γ -1 ( ϵ, δ ) = min { t ∈ N : f ( t ) ϵ ≤ δ 1 4 } , and

If the UQ-oblivious early stopping method is run with any budget B ob > z ob then the best candidate is returned with probability P ob > 1 -nδ .

Example 3. Consider f ( t ) = 1 t . According to Theorem 2, if B ob > 2 ⌈ log 2 ( n ) ⌉ ( n + ∑ i =1 ,...,n γ -1 ( ν i -ν 1 2 , δ )) , the UQ-oblivious method can return the best candidate with probability over 1 -nδ . But if B UQ ≃ γ -1 ( ν 2 -ν 1 2 , δ ) · n 1 , the UQ method can return the best candidate with probability over 1 -nδ . As shown in Appendix C.2, Theorems 1 and 2 together show that the UQ approach guarantees the same probability of identifying the optimal candidate as the UQ-oblivious counterpart with a smaller budget lowerbound B (see Corollary 6).

3.5 UQ-Guided HPO Family

The UQ-guided scheme is a general approach to enhancing HPO with uncertainty awareness. We next explain how it is integrated into several existing HPO methods to transform them into UQ-guided ones, yielding a UQ-guided HPO family. In the following, we use the suffix 'plus (+)" to indicate the UQ-guided HPO methods.

Successive Halving plus (SH+) is derived from the early stop-based HPO design Successive Halving (SH) [19]. Algorithms 1 and 2 in Appendix A.3 show the pseudo-code. Given total budget B and round budget R and an initial K , SH+ first trains K candidates each with the initial b = ⌊ R K ⌋ units of budget, and ranks them by the evaluation performance. Then SH+ updates K based on Section 3.3.2 and keeps the top K configurations according to the UQ-guided scheme ( OracleModel in Algorithms 1 and 2), and continues the process until the budget runs out.

Hyperband plus (HB+) originates from the popular HPO design Hyperband (HB). HB is an HPO method trying to better balance exploration and exploitation than SH does [25] by adding an outer loop for grid search of the value of K . HB+ simply extends HB by using SH+ rather than SH as its inner loop, changing the target of the grid search to the initial value of K .

Bayesian Optimization and Hyperband plus (BOHB+) is developed from BOHB [11]. BOHB is similar to HB except that it replaces the random sampling from the uniform distribution with BO-based sampling. BOHB+ makes the corresponding changes from HB+ by adopting BO-based sampling for its outer loop.

Sub-sampling plus (SS+) is derived from the Sub-sampling (SS) algorithm [15]. It showcases the applicability of the UQ-guided scheme to non-early stop-based methods. Similar to other methods, in each round, SS also chooses candidates for further training based on its assessment of the potential of those candidates. But unlike the other methods, SS does not discard any candidates, but keeps all in play throughout the entire HPO process. In each round, the candidates it chooses are those that show smaller validation loss than the most trained candidate shows. If there is none, it trains only the most trained candidate in that round. SS+ integrates the UQ-guided scheme into the candidate selection process of SS. When SS+ compares a candidate ( c i ) against the most trained candidate ( c m ), rather than checking their validation losses, it uses the UQ-guided scheme to compute the probability for the convergence loss of c i to be smaller than that of c m and checks whether the probability is over a threshold τ (0.9 in our experiments), that is, Pr( ℓ ( y, M ∗ γ cm ( x )) ≥ ℓ ( y, M ∗ γ c i ( x ))) ≥ τ .

4 Experiments

We conduct a series of experiments on the four UQ-guided HPO methods to validate the efficacy of the UQ-guided scheme for HPO.

4.1 Experimental Setup

Methodology. To check the benefits of the UQ-guided scheme for HPO, we apply the proposed UQ-guided HPO family to different HPO benchmarks, including NAS-BENCH-201 and LCBench, each for 30 repetitions, to measure the performance for different hyperparameter optimization tasks, and compared those with their original UQ-oblivious versions.

Platform. Our experiments are conducted on a platform equipped with an Intel i9-9900k CPU and an NVIDIA GEFORCE RTX 2080 TI GPU. The CPU has 8 cores, each of which can support 2 threads. The GPU has 4,352 cores of Turing architecture with a computing capability of 7.5. The GPU can achieve a maximum memory bandwidth of 616 GB/s, 0.4 tera floating-point operations per second (TFLOPS) on double-precision, and 13 TFLOPS on single-precision.

Workloads. We evaluate the UQ-guided methods on two real-world benchmarks. Nas-Bench-201 [9] (CC-BY 4.0) encompasses three heavyweight neural architecture search tasks (NAS) on CIFAR-10, CIFAR-100, and ImageNet-16-12 (CC-BY 4.0) datasets. In addition, we investigate the performance of optimizing traditional ML pipelines, hyperparameters, and neural architecture in LCBench [42]. For example, we optimized 7 parameters for the Fashion-MNIST dataset [7], where the resource type is determined by the number of iterations. Additional information regarding these benchmarks can be found in Appendix F. In this context, one unit of budget equates to a single training epoch, and by default, the total HPO budget ( B ) allocated for each method is 4 hours.

4.2 Experimental Results

Figure 3 illustrates the results of NAS-BENCH-201 trained on ImageNet-16-120. It shows the results of four UQ-guided methods compared to their original ones. For each comparison, we show three

Top 1 Rank

Top 1 Rank

Top 1 Rank

Top 1 Rank

1

Different Trials

5

10

SH

SH+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

2

4

6

8

10

Top 1 Rank

SH

SH+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

0

5

10

15

Regret

SH

SH+

1

Different Trials

5

10

HB

HB+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

2

4

6

8

10

Top 1 Rank

HB

HB+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

0

5

10

15

Regret

HB

HB+

1

Different Trials

5

10

BOHB

BOHB+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

2

4

6

8

10

Top 1 Rank

BOHB

BOHB+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

0

5

10

15

Regret

BOHB

BOHB+

1

Different Trials

5

10

SS

SS+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

2

4

6

8

10

Top 1 Rank

SS

SS+

0

0.2

0.4

0.6

0.8

1

Fraction of Budgets

0

5

10

15

Regret

SS

SS+

metrics, namely top-1 rank on different trials, top-1 rank on different fractions of budgets, and regret on different fractions of budgets. The fraction of budgets denotes the portion of the budget that we allocate for that particular experiment compared to the standard full budget. Top-1 rank refers to the real ranking of the candidate ultimately chosen by the method. Regret (%) refers to the accuracy difference between the returned candidate and the real best candidate. In Figure 3, the average results of 30 repetitions are reported. For the right two columns, we also report the uncertainty bands, defined as the interval between the 30th and 70th percentiles. The benefits of the UQ-guided scheme are obvious, both for individual trials and across different fractions of budgets. It brings a 21-55% regret reduction. Similar results are observed on other benchmarks (LCBench results shown in Appendix G).

Table 1 provides the fraction of the total exploration time needed for the UQ-guided methods to achieve comparable model accuracy as the original methods do. The UQ-guided methods need much less time than their counterparts to obtain a similar performance. For instance, SH+ achieves the same average regret of 5% on NAS with only half of the budgets required by SH. These results indicate that the UQ technique can conduct HPO efficiently and effectively.

Our paper's experiments concentrate on DNN because efficient HPO is crucial for the time-consuming nature of DNN training. We also use the ridge regression in Section H as a demonstration to show the

potential of the methodology for other iterative learners, and leave a systematic study of which to the future.

5 Conclusion

This paper points out the importance of systematic treatment to the uncertainty in model trainings for HPO. It introduces a novel scheme named UQ-guided scheme , which offers a general way to enhance HPO methods for DNNs with uncertainty awareness. Experiments demonstrate that the UQ-guided scheme can be easily integrated into various HPO methods. The enhanced methods achieve 21-55% reduction of regret over their original versions, and require only 30-75% time to identify a candidate with a matching performance as the original methods do. The paper in addition provides a theoretical analysis of the effects of the UQ-guided scheme for HPO.

The key characteristic of the UQ method is the necessity to rank multiple learners during the HPO process. Gradient-based HPO methods [30], for instance, may not benefit from our UQ-guided scheme because of their sequential properties. One limitation of this paper is that it is mostly suitable for iterative learners, and needs adaptations for other learners: To go beyond, it could be, for instance, applied to the model selection work in previous studies [31] that use training dataset size as the budget dimension. In this case, the learner does not need to be iterative; the selection is based on the validation loss history trained with incremental dataset sizes. The UQ component can still guide the configuration selection and budget allocation in the HPO process.

Overall, this study concludes that UQ is important for HPO to consider, simple on-the-fly UQ goes a long way for HPO, and the UQ-guided scheme can serve as a general effective scheme for enhancing HPO designs.

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion , 76:243-297, 2021.

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823 , 2017.

Hadrien Bertrand, Roberto Ardon, Matthieu Perrot, and Isabelle Bloch. Hyperparameter optimization of deep neural networks: Combining hyperband with bayesian model selection. In Conférence sur l'Apprentissage Automatique , 2017.

Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 13(2):e1484, 2023.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning , pages 1613-1622. PMLR, 2015.

Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. Uncertainty estimation by fisher information-based evidential deep learning. In International conference on machine learning . PMLR, 2023.

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine , 29(6):141-142, 2012.

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence , 2015.

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 , 2020.

Vincent Dumont, Casey Garner, Anuradha Trivedi, Chelsea Jones, Vidya Ganapati, Juliane Mueller, Talita Perciano, Mariam Kiran, and Marc Day. Hyppo: A surrogate-based multi-level parallelism tool for hyperparameter optimization. In 2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) , pages 81-93. IEEE, 2021.

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1437-1446. PMLR, 10-15 Jul 2018.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR.

Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren. An open source automl benchmark. arXiv preprint arXiv:1907.00909 , 2019.

Yi-Qi Hu, Yang Yu, Wei-Wei Tu, Qiang Yang, Yuqiang Chen, and Wenyuan Dai. Multi-fidelity automatic hyper-parameter tuning via transfer series expansion. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3846-3853, 2019.

Yimin Huang, Yujun Li, Hanrong Ye, Zhenguo Li, and Zhihua Zhang. Improving model training with multi-fidelity hyperparameter evaluation. In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems , volume 4, pages 485-502, 2022.

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning , 110:457-506, 2021.

Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization , pages 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.

Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges . Springer Nature, 2019.

Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 240-248, Cadiz, Spain, 09-11 May 2016. PMLR.

Jie Jiang, Jiawei Jiang, Bin Cui, and Ce Zhang. Tencentboost: A gradient boosting tree system with parameter server. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE) , pages 281-284. IEEE, 2017.

Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi-fidelity bayesian optimisation with continuous approximations. In International Conference on Machine Learning , pages 1799-1808. PMLR, 2017.

Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics , pages 528-536. PMLR, 2017.

Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In International Conference on Learning Representations , 2016.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. , 18(1):6765-6816, jan 2017.

Yang Li, Yu Shen, Jiawei Jiang, Jinyang Gao, Ce Zhang, and Bin Cui. Mfes-hb: Efficient hyperband with multi-fidelity quality measurements. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8491-8500, 2021.

M. Lichman. Uci machine learning repository, 2013. Accessed: [Insert access date here].

Siyan Liu, Pei Zhang, Dan Lu, and Guannan Zhang. PI3NN: Out-of-distribution-aware prediction intervals from three neural networks. In International Conference on Learning Representations , 2022.

Jingwei Ma, Jiahui Wen, Mingyang Zhong, Weitong Chen, and Xue Li. Mmm: multi-source multi-net micro-video recommendation with clustered hidden item representation learning. Data Science and Engineering , 4:240-253, 2019.

Paul Micaelli and Amos J Storkey. Gradient-based hyperparameter optimization over long horizons. Advances in Neural Information Processing Systems , 34:10798-10809, 2021.

Felix Mohr and Jan N van Rijn. Towards model selection using learning curve cross-validation. In 8th ICML Workshop on automated machine learning (AutoML) , 2021.

Felix Mohr and Jan N van Rijn. Fast and informative model selection using learning curve cross-validation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Alejandro Morales-Hernández, Inneke Van Nieuwenhuyse, and Gonzalo Nápoles. Multi-objective hyperparameter optimization with performance uncertainty. Communications in Computer and Information Science , 1684:37-46, 2022.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012.

Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems , 26, 2013.

Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. Multi-fidelity bayesian optimization with max-value entropy search and its parallelization. In International Conference on Machine Learning , pages 9334-9345. PMLR, 2020.

Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter , 15(2):49-60, 2014.

Jiazhuo Wang, Jason Xu, and Xuejun Wang. Combination of hyperband and bayesian optimization for hyperparameter optimization in deep learning. arXiv preprint arXiv:1801.01596 , 2018.

Shiwen Wu, Yuanxing Zhang, Chengliang Gao, Kaigui Bian, and Bin Cui. Garg: anonymous recommendation of point-of-interest in mobile networks by graph convolution network. Data Science and Engineering , 5:433-447, 2020.

Quanming Yao, Mengshuo Wang, Yuqiang Chen, Wenyuan Dai, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu. Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv:1810.13306 , 2018.

Wentao Zhang, Jiawei Jiang, Yingxia Shao, and Bin Cui. Snapshot boosting: a fast ensemble framework for deep neural networks. Science China Information Sciences , 63:1-12, 2020.

Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(9):3079-3090, 2021.

A Method Details

A.1 Formulation of Confidence Curve

The following contents detail how to compute the confidence curve C ( P 1 , P 2 , · · · , P n ) based on current validation loss and quantified uncertainty of the n candidates.

Let Y be the random variable denoting the negation of the lowest converged validation loss:

Since ℓ ( y , M ∗ γ i ( X )) ∼ N ( µ i , σ 2 i ) , the cumulative distribution function (CDF) of Y , F Y ( y ) , is

Accordingly, the probability density function (PDF) of Y , f Y ( y ) , is

Now we can construct the confidence curve by calculating each P k ( k ∈ [ n ] ). For k = m , P k can be expressed as

Let ℓ ( y , M ∗ γ 1 ( X )) , · · · , ℓ ( y , M ∗ γ n ( X )) be mutually independent, thus

A.2 Computing ζ

Equation 3.6 can be calculated by directly subtracting P k i by P k i -1 :

ζ is the coefficient that relates the increase in the number of training epochs to its corresponding effect on confidence. Consider a working round that starts with k candidates. We have approximations at the end of the round for the converged loss ℓ ( y, M ∗ γ i ( x )) ∼ N ( µ i , σ 2 ) for i ∈ [ k ] . Here, t denotes the epochs. Letting each candidate train one extra unit of resource results in lower uncertainty, thus increasing the f score. First compute f ( ξ , τ ) at epoch t :

If each configuration is trained with one extra unit of resource (a total of t + 1 epochs), model uncertainty would be reduced. We use the same µ i to approximate the converged validation loss

for t + 1 epochs, and use σ -∆ t σ as an approximation for model uncertainty. Here ∆ t σ is the decrease in model uncertainty that is determined through offline profiling with details in Section D. This approximation leads us to f ( ξ , τ ) at epoch t +1 as

Subtracting Equation A.3 by Equation A.2 gives us the result in Equation 3.7:

A.3 Pseudo-code for UQ-Guided Hyperparameter Optimization (SH+)

Algorithm 2 OracleModel for determining K candidates into the next round.

A.4 Pseudo-code for UQ-Guided Hyperparameter Optimization (HB+)

A.5 Pseudo-code for UQ-Guided Hyperparameter Optimization (BOHB+)

A.6 Pseudo-code for UQ-Guided Hyperparameter Optimization (SS+)

Algorithm 5

B More Theoretical Analysis

We have been concerned with identifying the best candidate, while in practice, it is often sufficient to consider a situation where the difference between the result of candidate i ϵ ( ν i ϵ ) and the result of the best candidate ( ν 1 ) is less than or equal to a small value ϵ . We obtain the following theorem with proofs in Appendix C.3.

Theorem 4. For a budget B > R and a set of n candidates, let ˆ i be the output of the UQ-guided approach. Then

In comparison, ˆ i D , the output of the UQ-oblivious counterpart, satisfies

Example 5. Consider f ( t ) = 1 t . Substitution of f ( t ) in Theorem 4 can clearly show a smaller upperbound of the UQ-guided approach than that of the UQ-oblivious counterpart (see Appendix C.3 for details).

The theorems provide some insights into the theoretical benefits of the UQ-guided scheme. But it is worth noting that neither this bound comparison nor the budget bound comparison in Example 3 is sufficient to prove that the UQ-guided approach definitely would outperform the UQ-oblivious approach, a reason for the empirical comparisons in Section 4.

C Proofs

In this section, we provide proofs for the theorems presented in Section 3.4. At the end of the proof, we let f ( t ) = 1 t and obtained the results in Example 3 and Example 5.

The Lemma stated next will prove to be useful.

Lemma 1. For i > 1 , if min { t 1 , t i } > t , then we have a high probability that ℓ i,t i > ℓ 1 ,t 1 with respect to t if f ( t ) ∈ O ( t -1 / 4 ) .

Proof. In Section 3.4, we come to the conclusion that

This shows that the event ℓ i,t > ℓ 1 ,t happens with high probability with respect to t .

Now consider a more general setting, where each ℓ i has its own t i :

Comparing ℓ i,t i and ℓ 1 ,t 1 for a particular i ∈ [ n ] gives us the following:

Since t 1 > t and t i > t ,

C.1 Proof of Theorem 1

Proof. Let S i be the set of candidates the UQ scheme evaluates at the beginning of the i -th round. We assume that the n infinitely long loss sequences [ ℓ i,t ] with limits { ν i } n i =1 .

We compute the probability that the algorithm includes the best candidate in the last round, namely, 1 ∈ S ⌊ B R ⌋ , and the probability that the UQ scheme returns the best candidate in S ⌊ B R ⌋ .

Let r k be the round budget for each candidate in S k . R k = ∑ k j =0 r k . The probability that the best candidate is among the final kept candidate set is

Since the probability that ℓ 1 ,t is the smallest among all ℓ k,t ( k ∈ [ n ] ) is greater than

we have

Therefore, the probability that the scheme returns the best candidate is no less than

That is to say, for a given confidence threshold c , there exists a T such that as long as min { t 1 , t 2 , ..., t n } > T , then Pr( ∧ i =2 ,...,n ( ℓ i,t i > ℓ 1 ,t 1 )) > 1 -c . Recall that in the UQ scheme design, the goal is to select an optimal hyperparameter configuration from n candidates, and each round is allocated for R resources. This means that if choose R ≥ T · n , then the best candidate is returned from the algorithm with probability P > (1 -⌊ B R ⌋ c )(1 -c ) .

C.2 Proof of Theorem 2

The representation of z ob on the right-hand-side of the inequality is very intuitive: For each i , to confirm that the final loss of the i -th candidate is greater than the best candidate's with a probability of at least 1 -δ , it is necessary to train both candidates for at least the number of steps indicated by the i -th term in the sum. Repeating this reasoning for all i justifies the sum over all candidates.

Proof. First we show that, given budget z = z ob , round budget for round k satisfies

The last inequality is derived because ⌊| S k | / 2 ⌋ ≥ | S k | / 2 -1 .

Let τ i := γ -1 ( ν i -ν 1 2 , δ ) . We then show that, for a time t :

The second line follows by the definition of γ -1 ( ϵ, δ ) . Since r k ≥ τ ⌊| S k | / 2 ⌋ +1 , we can compute the following probability

where the first line follows by the definition of the early stopping algorithm (Successive Halving), the second by τ i being non-increasing. Namely, for all i > ⌊| S k | / 2 ⌋ +1 , we have τ i ≤ τ ⌈| S k | / 2 ⌉ +1 and consequently, Pr( ℓ i,R k > ℓ 1 ,R k ) ≥ Pr( ℓ ⌊| S k | / 2 ⌋ ,R k > ℓ 1 ,R k ) ≥ 1 -δ .

Consequently, the probability that the UQ-oblivious approach returns the optimal candidate is

We show in the next Corollary that, for c = n · δ 2 , the probability P obtained in Theorem 1 is no less than 1 -nδ .

Corollary 6. For the threshold c in Theorem 1 and δ in Theorem 2, let c = n · δ 2 and β -1 ( ϵ 2 , ϵ 3 , · · · , ϵ n , c ) = min { T : n ∏ i =2 (1 -( f ( T ) ϵ i ) 4 ) ≥ 1 -c } . Then by Theorem 1 the UQ approach returns the best candidate with probability over 1 -nδ if B ≃ γ -1 ( ν 2 -ν 1 2 , δ ) · n .

Proof. Let T > 4 √ 2 · γ -1 ( ν 2 -ν 1 2 , δ ) , we have

This shows us that T > β -1 ( ν 2 -ν 1 2 , ..., ν n -ν 1 2 , c ) . Consequently, according to Theorem 1, for B = R > T · n , the UQ approach returns the best candidate with probability

The UQ-oblivious approach returns the optimal candidate with probability over 1 -nδ if the budget B ob > z ob . But the UQ approach achieves the guarantee with budget B > 4 √ 2 · γ -1 ( ν 2 -ν 1 2 , δ ) · n , which can be empirically substantially smaller than the budget required in Theorem 2.

C.3 Proof of Theorem 4

Proof. Let R k = ∑ k j =0 r k , namely, the total number of epochs allocated for each candidate in S k . We can guarantee that, for the UQ-oblivious approach, the output candidate ˆ i D satisfies

by inspecting how the approach eliminates candidates and plugging in an upper bound for E (2 | ν i -ℓ i,R k | ) for all k in the last inequality. We can calculate the bound for the UQ-guided method in a

similar way:

The smaller upperbound of the UQ-guided approach than that of the UQ-oblivious counterpart in Theorem 4.

A simple calculation reveals that

by diving the first term by the second:

The last inequality holds because both B R and ⌈ log 2 ( n ) ⌉ are the number of rounds and are considered the same.

D Computational Details

Approximating ∆ σ for Candidates. In our probabilistic model, ℓ ( y , M ∗ γ c ( X )) is approximated by a Gaussian distribution parameterized by µ c and σ c . To compute the uncertainty reduction that would result from training each candidate for one additional epoch, we consider the uncertainty as a time series in the form of ( σ c ( t )) T t =1 . We examine the uncertainty trendings in NAS-BENCH201 [9]. Figure 4 shows a certain pattern of uncertainty behavior as t increases, both individually and aggregately, for different candidates.

We then model ( σ c ( t )) T t =1 according to different phases in the following way:

For t ∈ (0 , 6) , σ ( t ) increases. We use linear regression to fit the σ ( t ) , namely, σ ( t ) = a 1 t + b 1 .

For t ∈ (6 , 50) , σ ( t ) drops quickly. We use the exponential model to fit σ ( t ) , namely, σ ( t ) = a 1 e -b 1 x .

For t ∈ (50 , 180) , σ ( t ) increases steadily. We use another linear regression to fit σ ( t ) .

0

0.01

0.02

0.03

0.04

0.05

1

11

21

31

41

51

61

71

81

91

101

111

121

131

141

151

161

171

181

191

x

σ

0

0.01

0.02

0.03

0.04

0.05

1

11

21

31

41

51

61

71

81

91

101

111

121

131

141

151

161

171

181

191

x

Average

σ

(a) Uncertainty scope for different candidates.

(b) Average uncertainty for different candidates.

Confidence

Candidates

Threshold

τ

k1

k3

round 1

r

2

r

3

r

4

Threshold

τ

Confidence

Candidates

r

i

for round i

r

'

∆𝑐

(a) Confidence curves in different rounds.

(b) Analysis of the choice of τ .

For t ∈ (180 , 200) , σ ( t ) reverses the trend and decreases again. We use linear regression.

For each launch, we sample a few candidates and train each one fully till convergence. We then use the abovementioned way to model the uncertainty behavior for the whole dataset. This makes the approximation for ∆ t σ = σ ( t ) -σ ( t +1) effective and efficient.

An alternative way to approximate ∆ t σ is to use

Building Probabilistic Model. At any given time t , the approximation of converged validation loss follows the Gaussian distribution: ℓ ( y , M ∗ γ c ( X )) ∼ N ( µ c , σ 2 c ) . In our experiments, δ = 10 , µ c is the current accuracy ℓ ( y , M t γ c ( X )) , and σ c is the unbiased estimation of the standard deviation of ℓ ( y , M i γ c ( X )) t i = t -10 .

E Additional Related Work

Freeze-thaw Bayesian optimization. This paper proposes a new method to improve Bayesian optimization (BO) for HPO. It avoids the limits of the expected improvement (EI) criterion in naive BO which always favors picking new candidates rather than running old ones for more iterations. Instead of always sampling new candidates, it can also choose old candidates for further evaluation based on the modified EI in each round. This method, however, is limited to BO method. In contrast, our UQ scheme is applicable to the vast number of early stopped and multi-fidelity-based HPO methods.

HyperJump improves HB in that, during HPO, it skips certain rounds for certain candidates if the risk of skipping is within a threshold. For the NAS-BENCH-201 trained on ImageNet-16-120, HJ reduces the running time compared to the original HB method (only needs a 5.3% fraction of budget to achieve close to optimal results achieved by the original HB with a standard full budget), but it does not improve the HPO performance (i.e., Top-1 Rank and Regret). In contrast, our method only needs less than a 3% fraction of the budget to achieve close to optimal results and when using around 5% fraction of budget, our method reduces the regret by 33%.

F Benchmark and Dataset Information

Table 2 consolidates information on the datasets, hyperparameters, fidelity, and dataset sizes for Nas-Bench-201 and LCBench. The datasets for LCBench are drawn from various sources [37, 13].

G More Results on Experiments

We include more results on NAS-Bench-201 and LCBench. For example, Figure 10 and 11 show the results on LCBench, where we proved the consistently better performance of the UQ-guided approaches than the UQ-oblivious methods on Fashion-MNIST. Figure 10 shows the results of the validation loss while Figure 11 demonstrates the results of regret. UQ-guided approaches obtained an average of over 50% improvement over the UQ-oblivious counterparts.

88

88.5

89

89.5

90

1e-2

1e-1

1e0

Fraction of Budgets

SH

SH+

88

88.5

89

89.5

90

1e-2

1e-1

1e0

Fraction of Budget

HB

HB+

88

88.5

89

89.5

90

1e-2

1e-1

1e0

Fraction of Budget

SS

SS +

80

82

84

86

88

90

1e-2

1e-1

1e0

Fraction of Budget

BOHB

BOHB+

0

0.5

1

1.5

2

1e-2

1e-1

1e0

Fraction of Budget

HB

HB+

0

0.5

1

1.5

2

1e-2

1e-1

1e0

Fraction of Budget

SH

SH+

0

2

4

6

8

1e-2

1e-1

1e0

Fraction of Budget

BOHB

BOHB+

0

0.5

1

1.5

2

1e-2

1e-1

1e0

Fraction of Budget

SS

SS+

64

65

66

67

68

69

70

1e-2

1e-1

1e0

Fraction of Budgets

SH

SH+

64

65

66

67

68

69

70

1e-2

1e-1

1e0

Fraction of Budget

HB

HB+

60

62

64

66

68

70

1e-2

1e-1

1e0

Fraction of Budget

BOHB

BOHB+

60

62

64

66

68

70

1e-2

1e-1

1e0

Fraction of Budget

SS

SS +

0

1

2

3

4

5

6

1e-2

1e-1

1e0

Fraction of Budget

SH

SH+

0

1

2

3

4

5

6

1e-2

1e-1

1e0

Fraction of Budget

HB

HB+

0

1

2

3

4

5

6

7

8

1e-2

1e-1

1e0

Fraction of Budget

BOHB

BOHB+

0

1

2

3

4

5

6

7

8

1e-2

1e-1

1e0

Fraction of Budget

SS

SS+

12

13

14

15

16

0.3

0.6

0.9

Fraction of Budget

SS

SS+

12

13

14

15

16

0.3

0.6

0.9

Fraction of Budget

HB

HB+

12

13

14

15

16

0.3

0.6

0.9

Fraction of Budgets

SH

SH+

12

14

16

18

0.3

0.6

0.9

Fraction of Budget

BOHB

BOHB+

0

2

4

0.3

0.6

0.9

Fraction of Budget

SS

SS+

0

2

4

0.3

0.6

0.9

Fraction of Budget

HB

HB+

0

2

4

0.3

0.6

0.9

Fraction of Budget

BOHB

BOHB+

0

2

4

0.3

0.6

0.9

Fraction of Budget

SH

SH+

H Experiments on Other Iterative Learners

We focus on DNNs in our experiments because (1) DNNs are among the most influential models today and (2) DNN training takes a long time so selecting the optimal hyperparameters is a critical concern, making the problem more pressing.

Even though we focus on DNN methods, our approach can be applied to other iterative learners. We consider a ridge regression problem trained with stochastic gradient descent on this objective function with step size . 01 / √ 2 + T . The l 2 penalty hyperparameter λ ∈ [10 -6 , 10 0 ] was chosen uniformly at random on a log scale per trial. We use the Million Song Dataset year prediction task [27] with the same experiment settings as in the original SH paper. We show the results of ridge regression on "SH" and "SH+". Figure 12 shows the Top-1 Rank results and the regret of the test error for different fractions of budgets. The average results of 30 repetitions are reported. The benefits are obvious: SH+ obtained an average of over 40% improvement over the SH.

0.03

0.2

0.4

0.6

0.8

1

Fraction of Budgets

4

7

Top 1 Rank

SH

SH+

0.03

0.2

0.4

0.6

0.8

1

Fraction of Budgets

0

0.01

0.02

0.03

0.04

0.05

Regret

SH

SH+

Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 1, Section 3.4, and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We state them at the end of Section 5. The key characteristic of the UQ method is the necessity to rank multiple learners during the HPO process. Gradient-based HPO methods [30], for instance, may not benefit from our UQ-guided scheme because of their sequential properties. One limitation of this paper is that it is mostly suitable for iterative learners, and needs adaptations for other learners: To go beyond, it could be, for instance, applied to the model selection work in previous studies [31] that use training dataset size as the budget dimension. In this case, the learner does not need to be iterative; the selection

is based on the validation loss history trained with incremental dataset sizes. The UQ component can still guide the configuration selection and budget allocation in the HPO process.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: See Section 3.4 and appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include them in Section 4 and the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include them in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted 30 repetitions for the experiments. For the right two columns in Figure 3, besides the average results of the repetitions, we also report the interval between the 30th and 70th percentiles.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: See Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: As stated in Section 5, our work is a general algorithm aiming at optimizing HPO and belongs to a foundational research; its societal impacts are neutral.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the creators for the existing assets and included the name of the license. See Section 4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors presented a novel scheme for hyperparameter optimization (HPO) in machine learning (ML) based on quantization uncertainty. The paper reflects a clear consequent way of discussion from introduction and literature overview to conclusions and very detailed appendixes. The authors demonstrated a strong mathematical background of these results with a full understanding uncertainly in machine learning model optimization process. Work presents theoretical analysis, examples and experiments on different related algorithms. Presented uncertainly quantization guided scheme can be widely applied for HPO in ML models optimization.\n\nSTRENGTHS:\n•\tUniversalizing provided improving technique (originality and quality). \n•\tUsing different conventional HPO algorithms for demonstration novel approach (clarity and quality).\n•\tStrong mathematical background and proofs (quality, originality and significance).\n•\tTheorical analysis presented guided scheme (quality and significance).\n•\tClear experiments methodology and results (clarity).\n\nWEAKNESSES:\nPerhaps authors should provide more detailed discussion about UQ-scheme limitations and adaptation for non-iterative learners and add some links about such learners\n\nMistypes and etc.\nAppendix C, page 21, lines 2 and 3 in the first paragraph. Broken links.\nAppendix D, page 24, line four in the first paragraph. Broken equation link.\n\nQUESTIONS:\nHas the UQ-scheme some limitations depending on the number of initial candidates for SH algorithm, for example? Could this be the case where UQ-scheme performance will be almost the same as in the original HPO algorithm, where regret between SH will equal SH+ with large or small vise-versa amount of the initial candidates? Or accuracy derivation for confidence curve is absolutely stable for any quantity of candidates?\n\nLIMITATIONS:\nLimitations this work mainly addressed to HPO in machine learning applications. Authors build adequate descriptions and analysis of their technique only within ML applications (abstract introduction and conclusion). Moreover, the authors noted that their HPO tuning scheme can be mostly suitable for iterative learners.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents UQ-guided scheme, an approach for hyper-parameter optimisation (HPO) that quantifies uncertainty of each candidate configuration. This uncertainty quantification mechanism is applied to several state-of-the-art iterative HPO algorithms to prevent that promising configurations that have low performance in the early stages of the iterative process are not incorrectly discarded. The evaluation shows that UQ allows to find similarly performing configurations at a fraction of the exploration budget.\n\nSTRENGTHS:\n**Originality:** \n\nThe authors claim the method is new and that they only found one work that accounts for uncertainty (reference 30), however there are more works that account for uncertainty when performing hyper-parameter optimization. For example, the following:\n\n[a] Mendes, Pedro, et al. \"HyperJump: accelerating HyperBand via risk modelling.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 8. 2023.\n\nThis other work may also be of relevance for the authors:\n\n[b] Swersky, Kevin, Jasper Snoek, and Ryan Prescott Adams. \"Freeze-thaw Bayesian optimization.\" arXiv preprint arXiv:1406.3896 (2014).\n\nI would have liked to see a comparison of the proposed method against reference [30], given that it was the only work the authors found that accounted for uncertainty, and also against [a].\n\n&NewLine;  \n**Quality:** \n\nThe claims seem well supported and the theoretical methodology sound. The methods used are appropriate. Weaknesses could be discussed in more detail and comparison agains related work could be improved/extended.\n\n&NewLine;  \n**Clarity:** \n\nOverall, the paper is well written. There are a couple of sentences here and there that could be improved, but it doesn't substantially affect understandability. \n\nI believe the main manuscript should have pseudo-code illustrating the extra steps that should be added to any iterative learner that a user would like to change to account for model uncertainty. There is one example in the supplemental material but it seems to be specific to SH. What is generalisable from that example and what is not?\n\nThe manuscript is quite heavy on math and it seems like some parts could profit from an intuition-guided explanation. For example, in section 3.2, paragraphs \"Decomposition of momentum and the underlying structure of the metric\" and \"Solving for the Momentum Mean and Variance\" could have a more high-level description of why it is important to analyse those decomposed parts, what type on information is expected to be extracted from each, and how that information can contribute to improving the selection of the (near-)optimal configuration. Some math details can be relegated to the supplemental material, should space be a concern.\n\nFinally, which parameters does the UQ approach have and that a user must set? How do those parameters affect the improvements attainable by UQ?\n\n&NewLine;  \n**Significance:**\n\nImproving the efficiency of hyper-parameter optimization is a relevant problem and efficient methods that are shown to substantially improve over existing methods are relevant. However, I felt the current version of the manuscript misses comparisons against recent related work.\n\nWEAKNESSES:\nSee above for strengths and weaknesses.\n\nQUESTIONS:\n**a)** \nWhy do you claim that \"data uncertainty is constant\"? (This is in the beginning of section 3.1)\n\n**b)** \nSection 3.3. -- exploration vs exploitation: is k_1 - 1 == k'_i ?\n\n**c)** \nSection 3.4, Example 3 is not clear to me. In both cases you mention in the example, the UQ method \"can return the best candidate with probability over 1 − n\\delta\". Do you mean that the probability of UQ returning the best candidate is larger than \"1 − n\\delta\" (which is the probability of the UQ-oblivious method returning the best candidate)? Also, there is a 'But' in the middle of the example, but I don't understand the 'But' and I don't see where the expression  for B_{UQ} comes from.\n\n**d)** \nWhich parameters does the UQ approach have and that a user must set? How do those parameters affect the improvements attainable by UQ?\n\n&NewLine;  \n**Comments:**\n\n- Section C.1 of the appendix has some missing pointers to assumptions.\n\n- The evaluation plots would all be much easier to read if the y-axis was labelled. \n\n- You could also highlight the moment when the UQ-version of each baseline achieves comparable performance to the UQ-oblivious version. This would make it easier to visually identify the gains achievable via UQ in terms of budget savings.\n\nLIMITATIONS:\nThere could be more discussion on the limitations of the proposed method. For example, when is it applicable, how do the different assumptions impact its applicability, how could future work improve the current version of the proposal to address these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper \"UQ-Guided Hyperparameter Optimization for Iterative Learners\" the authors present an uncertainty quantification method for optimizing the hyperparameters of iterative learners in a multi-fidelity setting. It is argued that the best possible candidate is often mistakenly discarded at some point and uncertainty quantification can help avoid such mistakes. Furthermore, it can help to identify the best-performing candidate with certainty earlier than running the entire promotion scheme of, e.g., successive halving.\n\nSTRENGTHS:\n- The paper presents a novel and interesting approach to HPO considering uncertainties throughout the HPO process to incorporate them and making decisions more cautiously.\n- The approach is theoretically grounded and the authors also provide a theoretical framework for their method.\n- In experiments, the method also performs favorably compared to methods that do not leverage the information about uncertainty.\n\nWEAKNESSES:\n- The authors claim that their method is working for any kind of iterative learners. However, this claim is not supported in the experimental section as only deep learning methods are considered. Therefore, it is questionable whether the same quality of results could be observed for other iterative learners, e.g., logistic regression.\n- Moreover, to make their approach work, the authors need a learner that is able to quantify its epistemic uncertainty, i.e., model uncertainty. However, I do not see how you would make this work for iterative learners in general where you cannot make use of simple tricks as in deep learning models to obtain ensembles through MCMC dropout or something similar.\n- It could be made clearer that the authors specifically target multi-fidelity/early stopping methods with their method.\n- The writing of the paper could be improved. For example, in the introduction the authors write about \"steps\" but it is not clear what is meant by steps, in particular, what are \"steps of exploration\". In general, there are some linguistic issues throughout the paper which the authors could easily get rid off using tools like Grammarly.\n- In Section 3.2 the authors formalize their setting by attributing hyperparameters to a model, however, some hyperparameters belong to the learning algorithm, e.g., weight decay or learning rate, but not to the model.\n- It is somewhat strange that the plus variants (including UQ) always start better than the basic method. In particular, it seems awkward for zero budget that the proposed method can better identify the top 1 rank than the base method. Also for the regret it is unclear how UQ can help identifying a more suitable candidate if nothing is known about this candidate.\n\nQUESTIONS:\n- How can the epistemic uncertainty be quantified for other learners?\n- Why does the performance of X+ always improve already over X if there is no budget used?\n- How is the uncertainty or let us say better certainty about the top 1 rank leveraged in the HPO tool? It would be quite straightforward to stop evaluating other candidates if there is enough evidence already that the top 1 candidate is correctly identified.\n- If in every iteration of successive halving the amount of budget spent for this iteration is kept at a certain level, how is then the budget distributed if more candidates are to be evaluated in that iteration or could it also happen that more budget is assigned to every remaining candidate than originally planned according to e.g. successive halving?\n\nLIMITATIONS:\nLimitations are not sufficiently addressed. Although the authors mention in the questionaire that limitations would be addressed there is just a single sentence that it is mostly suitable for iterative learners and needs adaptions for other learners. This is definitely not a sufficient limitations discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an incorporate uncertainty quantification for hyperparamter tuning (learning rate, neural architecture). It is assumed the model performance metrics of interest, such as validation error, are Gaussian, and we can use the training output of first $N$ epochs to estimate the mean and variance of the performance metrics distribution. Using the distribution, this paper proposes a UQ-guided scheme by using confidence curves to find the highest performant combinations of hyperparameters.\n\nSTRENGTHS:\nIt is shown by theory and empirical study that the proposed UQ method can enhance the existing hyperparameter tuning method like successive halving, which suffers from model uncertainty and incorrectly eliminating highly performant hyperparameter values\n\nWEAKNESSES:\nEstimating the mean and variance of the performance metrics curve may need to take some epochs.\n\nQUESTIONS:\n1. On page 4, it is stated that $\\hat v$ and $\\hat\\sigma^2$ are estimated with N=200 epochs. My understanding is that the first N=200 epochs are only for estimating $\\hat v$ and $\\hat\\sigma^2$ and the UQ-guided HPO are not used in these 200 epochs. However, I suspect we don’t have the luxury of running more than 200 epochs in some applications with large data. \n\n2. The $Z$ in eq (3.1) seems to model a latent effect underlying the hyper parameter space. $Z$ is initialized randomly and then updated using the equation on p.4. No further analysis about $Z$ is provided in later sections, for example, whether we can use $Z$ to measure the contribution of certain combinations of hyperparameters. If estimating $Z$ explicitly doesn’t yield any useful products, perhaps there is no need to estimate Z and we just need to estimate $\\alpha_u$?\n\n3. Theorems:\n- How to interpret the assumptions of Theorem 1?\n- Does Theorem 2 apply to “any” UQ oblivious method like SH, HB? If so, it may be good to comment on/verify whether these existing methods satisfy the assumptions of Thm 2.\n\n4. Can more details about $z_ob$ in Theorem 2 be given, e.g. how to arrive at the inequality? It is unclear how to interpret this quantity.\nThe proof of Theorem 1 refers to two assumptions but it looks the pdf was not compiled correctly so the reference hyperlinks are missing\n\nLIMITATIONS:\nThe work is regarding a general algorithm aiming at optimizing HPO and belongs to a foundational research; its societal impacts are neutral.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}