Title: Utilitarian Algorithm Configuration
Venue: NeurIPS 2023
Accepted: yes
Keywords: algorithm configuration

algorithm selection

data-driven algorithm design

utility of runtime
Abstract: We present the first nontrivial procedure for configuring heuristic algorithms to maximize the utility provided to their end users while also offering theoretical guarantees about performance. Existing procedures seek configurations that minimize expected runtime. However, very recent theoretical work argues that expected runtime minimization fails to capture algorithm designers' preferences. Here we show that the utilitarian objective also confers significant algorithmic benefits. Intuitively, this is because mean runtime is dominated by extremely long runs even when they are incredibly rare; indeed, even when an algorithm never gives rise to such long runs, configuration procedures that provably minimize mean runtime must perform a huge number of experiments to demonstrate this fact. In contrast, utility is bounded and monotonically decreasing in runtime, allowing for meaningful empirical bounds on a configuration's performance. This paper builds on this idea to describe effective and theoretically sound configuration procedures. We prove upper bounds on the runtime of these procedures that are similar to theoretical lower bounds, while also demonstrating their performance empirically.

Utilitarian Algorithm Configuration

Abstract

We present the first nontrivial procedure for configuring heuristic algorithms to maximize the utility provided to their end users while also offering theoretical guarantees about performance. Existing procedures seek configurations that minimize expected runtime. However, very recent theoretical work argues that expected runtime minimization fails to capture algorithm designers' preferences. Here we show that the utilitarian objective also confers significant algorithmic benefits. Intuitively, this is because mean runtime is dominated by extremely long runs even when they are incredibly rare; indeed, even when an algorithm never gives rise to such long runs, configuration procedures that provably minimize mean runtime must perform a huge number of experiments to demonstrate this fact. In contrast, utility is bounded and monotonically decreasing in runtime, allowing for meaningful empirical bounds on a configuration's performance. This paper builds on this idea to describe effective and theoretically sound configuration procedures. We prove upper bounds on the runtime of these procedures that are similar to theoretical lower bounds, while also demonstrating their performance empirically.

1 Introduction

Heuristic algorithms are surprisingly effective at solving hard computational problems. However, no single set of heuristics works well on all problems; the design choices must be tuned to a distribution of problem instances in the same way that the parameters of a machine learning model are tuned to a dataset. This black-box optimization problem is called algorithm configuration . Roughly speaking, the literature on algorithm configuration is divided into two camps. The older and larger camp designs heuristic procedures that aim to find good configurations as quickly as possible (Birattari et al., 2002; Hutter et al., 2009, 2011; Ansótegui et al., 2009; López-Ibáñez et al., 2016). A second, more recent line of work aims to provably identify approximately optimal configurations and to offer guarantees about the runtime required to do so (Kleinberg et al., 2017; Weisz et al., 2018; Kleinberg et al., 2019; Weisz et al., 2019, 2020). Virtually all of the work in both camps has focused on minimizing some version of capped average runtime, although one exception of which we are aware is Tornede et al. (2020), which explores objectives like higher-order polynomials of runtime that might better capture a user's level of risk-aversion.

Very recently, a theoretical argument has been made that minimizing expected runtime is inconsistent with a plausible set of axioms about algorithm designers' preferences (Graham et al., 2023). Runtime is an economic good and, like other economic goods, the value an individual assigns to it is not

necessarily proportional to the value of the good itself-nor should it be. We can characterize a set of bounded and monotonically decreasing utility functions that are implied by certain axioms. These are essentially the von Neumann-Morgenstern axioms (Von Neumann and Morgenstern, 1944) from classical decision theory, plus two novel runtime-specific axioms that assert a (weak) preference for faster runs over slower ones, and a (strict) preference for algorithm runs that complete over ones that time out. Preferences that follow these axioms are described by a utility function that is monotonically decreasing in runtime from 1 to 0. This utility function can incorporate factors like the benefit an end user attains from completing a run, the cost they pay for cloud computing resources, and any uncertainty they may have about when an answer will stop being useful to them. This utilitarian perspective opens a new and exciting direction for algorithm configuration.

While it is reassuring to know that optimizing utility instead of runtime is the right thing to do, it also offers significant algorithmic benefits. Because utilities are bounded and monotonically decreasing, exceedingly long runs contribute negligibly to estimates of an algorithm's expected utility. This means that there will always be some captime that allows us to accurately estimate true expected utility from capped samples. We can use tools from the multi-armed bandit literature for best arm identification, and the algorithms we describe will draw on the elimination and racing algorithms described in Mannor and Tsitsiklis (2004); Even-Dar et al. (2006) and elsewhere. In contrast, runtimebased configuration procedures like Structured Procrastination (Kleinberg et al., 2017), Structured Procrastination with Confidence (Kleinberg et al., 2019), LeapsAndBounds (Weisz et al., 2018), CapsAndRuns (Weisz et al., 2019), and ImpatientCapsAndRuns (Weisz et al., 2020) need to do many runs at large captimes in order to make theoretical guarantees. Furthermore, in order to be able to make any provable guarantees at all, they have to introduce an additional parameter specifying how much of the runtime CDF can be ignored; even if an algorithm has always finished very quickly on every instance seen so far, it could always take so long on the next instance that its expected runtime is arbitrarily large. In this setting, no optimality guarantees can be made without broadening the definition of optimality.

Other theoretically-motivated methods such as Gupta and Roughgarden (2017); Balcan et al. (2017, 2021) offer performance guarantees based on different measures of complexity and guarantee notions of PAC optimality akin to those we present here. These papers do not focus on runtime, instead studying traditional sample complexity, with each sample (i.e., algorithm run) contributing the same 'sampling cost.' In our algorithm configuration setting, the cost of a sample is the amount of (capped) runtime spent acquiring it, with longer captimes potentially leading to more expensive (but also more informative) samples. Because of this difference, these two lines of work propose quite different types of learning algorithms. However, Balcan et al. (2021) do assume the existence of a bounded utility function which measures algorithm performance, and others (Hoos and Stützle, 2004) have argued that utility functions of this form are better objectives to optimize than runtime when choosing algorithms. Some methods have been specifically designed to exploit the parallel nature of the algorithm configuration problem; AC-Band (Brandt et al., 2023) is a bandit-based procedure inspired by the Hyperband algorithm (Li et al., 2017) that runs multiple configurations simultaneously while ruling out poor-performing ones along the way and offering theoretical guarantees with respect to the set of configurations considered.

Inspired by the mantra of procrastination that has found success in previous work, and by the benefits that come with the use of utility functions, this paper presents a procedure we dub Utilitarian Procrastination (UP), so-named because it performs as many low-captime runs as possible before proceeding to higher-captime runs. We offer theoretical guarantees about its performance, showing that it will return a good configuration and proving that its worst-case upper bound is similar to the theoretical lower bound that any procedure must require. We also present experimental results showing how this procedure performs in practice and how it compares to a more naive baseline.

2 Setup

We assume there is a monotonically decreasing runtime utility function u : R ≥ 0 → [0 , 1] with u (0) = 1 and lim t →∞ u ( t ) = 0 . The existence of u follows from simple axioms (Graham et al., 2023). The value u ( t ) describes the (expected) well-being of an individual who uses an algorithm to solve a problem instance (e.g., an integer program). During the configuration process, the goal will be to choose an algorithm that maximizes u ( t ) , in expectation over t .

Given a set of algorithms indexed by i = 1 , ..., n , our goal is to find an approximately optimal algorithm using capped runtime samples. We assume we have access to a stream of input instances indexed by j = 1 , 2 , ... drawn from some distribution D J . We will use t ij to denote the true uncapped runtime of algorithm i on input j , and t ij ( κ ) = min( t ij , κ ) to be the κ -capped runtime of i on j . When we do a run, we observe t ij ( κ ) , not t ij . Of course these coincide for any run that completes. The instance distribution D J , along with any randomness of the algorithm or execution environment will together induce a runtime distribution for each algorithm i . We will use D i to denote this runtime distribution, and F i to denote its CDF. For each algorithm i , the true uncapped expected utility is

The capped expected utility is

And given any capped runtime samples t i 1 ( κ ) , ..., t im ( κ ) , the capped empirical average utility is

Definition 1 ( /epsilon1 -optimal) . An algorithm i ∗ is called /epsilon1 -optimal if U i ∗ ≥ max i U i -/epsilon1 .

Our goal will be to find /epsilon1 -optimal algorithms with probability at least 1 -δ , and to do so as quickly as possible. Our first lemma deterministically bounds an algorithm's uncapped expected utility in terms of its capped expected utility and CDF.

Proof. This follows from the law of total expectation and the fact that u is non-increasing, and so u ( κ ) ≥ u ( t ) for all t ≥ κ .

Lemma 1. For any i and κ we deterministically have U i ( κ ) -u ( κ ) ( 1 -F i ( κ ) ) ≤ U i ≤ U i ( κ ) .

Intuitively, Lemma 1 is true because the capped expected utility counts runs that cap as having just completed at the captime, when really they would have taken longer if given the chance. This makes the capped expected utility look more favourable than the uncapped expected utility. Our second lemma shows that if we do runs at captime κ , then we can accurately estimate the algorithm's runtime CDF at κ .

Lemma 2. For any i , m , κ and δ , let ̂ F im ( κ ) be the fraction of the m runs that A completes within captime κ . Then ̂ F im ( κ ) -√ ln(1 / δ ) 2 m ≤ F i ( κ ) ≤ ̂ F im ( κ ) + √ ln(1 / δ ) 2 m with probability at least 1 -2 δ .

Proof. For each j , let X j = 1 if t ij < κ and X j = 0 otherwise. The proof then follows from a straightforward application of Hoeffding's inequality.

Lemma 2 simply says that if we take enough samples at captime κ , the fraction of those runs that complete will be close to the true likelihood of a run completing before κ . Our third lemma shows that expected capped utility can be estimated accurately using capped runtime samples.

Proof. Since u ( t ij ( κ ) ) ∈ [ u ( κ ) , 1] for all j , the proof follows immediately from Hoeffding's inequality.

Lemma 3 simply says that if we take enough samples, then the empirical capped average utility will be close to the true expected capped utility. Together, these lemmas imply empirical confidence bounds on the true expected utility. Define the upper and lower confidence bounds

The next lemma shows that these are good confidence bounds if both the captime κ and the number of samples m are sufficiently large. With probability at least 1 -δ , each algorithm's true expected utility will fall within these confidence bounds, and the width of each confidence range will not be too large.

Lemma 4. If we do m runs of each algorithm at captime κ , then with probability at least 1 -δ we will have

and

for all i simultaneously.

See Appendix A for a full proof. The idea is that the definition of the bounds together with Lemmas 2 and 3 mean the confidence bounds hold and are accurate.

We can see in Lemma 4 that the error in our estimate of an algorithm's expected utility comes from two sources: sampling and capping. The term 2 √ ln(4 n/ δ ) 2 m represents the error due to sampling, while the term u ( κ ) ( 1 -F i ( κ ) ) represents the error due to capping. To make good guarantees, configuration procedures will need to ensure that both of these terms are sufficiently small.

3 Configuration Procedures

We first describe two hypothetical procedures that give us lower bounds on the number of samples and the captime that any configuration procedure will need to use. The bound on the number of samples (Section 3.1) is a classic result. In Section 3.2, we use a novel 'prover-skeptic' argument to show the lower bound on captime. We then describe in Section 3.3 a simple usable procedure that returns an /epsilon1 -optimal algorithm with probability at least 1 -δ , but which suffers from two major drawbacks. First, it requires that we specify an accuracy parameter /epsilon1 and a captime κ as input ahead of time. Making poor choices for these parameters can have a large impact on total configuration time (see Section 4 for an illustration). Indeed, many choices of /epsilon1 , and κ are mutually incompatible, giving rise to meaningless bounds. To avoid this, we hope to design procedures that are anytime : they continue to improve the accuracy guarantees they can make as they spend more time running. By taking gradually more and more samples, and by gradually taking them at higher and higher captimes, an anytime procedure continues to shrink the /epsilon1 that it can guarantee, rather than trying to target a fixed /epsilon1 it is given as input. Second, the naive procedure is not input-adaptive in any way. It does the same number of samples at the same captime for every algorithm. Both of these drawbacks are fixed by our UP procedure in Section 3.4. UP is an anytime procedure, meaning it requires neither an /epsilon1 nor a κ as input, but instead gradually reduces the /epsilon1 it can guarantee by increasing both the number of samples and the captime.

3.1 Hypothetical Runtime Oracle Procedure

The unique characteristic of our setting is the fact that we observe capped rather than true runtime samples, and that the cost we pay for each sample is equal to the time we spend collecting it. If this were not the case, then our problem would already be solved. If we had some oracle that we could simply query for the true runtime of an algorithm run on a given instance, then all we would need to do is collect sufficiently many samples from each algorithm. In this case, the optimal procedure simply takes an increasing number of samples of each algorithm and rules out each suboptimal one was soon as it can. The Runtime Oracle Procedure (Algorithm 1) is an instantiation of the Successive Elimination algorithm of Even-Dar et al. (2006) (their Algorithm 3). With probability at least 1 -δ it will eventually eliminate all algorithms except the optimal one.

Theorem 1. With probability at least 1 -δ the Runtime Oracle procedure will eventually return the optimal algorithm, and it will return an /epsilon1 -optimal algorithm if it is run until m is large enough that 2 √ ln(4 nm 2 / δ ) 2 m ≤ /epsilon1 . At that point it will have taken m uncapped samples of each /epsilon1 -optimal algorithm,

Algorithm 1 Runtime Oracle Procedure

/negationslash

and m i ≤ m uncapped samples of each /epsilon1 -suboptimal algorithm i , where 2 √ ln(4 nm 2 i / δ ) 2 m i ≤ ∆ i , and where ∆ i is the difference between the expected utility of an optimal algorithm and of algorithm i .

See Even-Dar et al. (2006), Theorem 8 and Remark 9 for proofs and details. The Runtime Oracle procedure has two desirable qualities that we would like to preserve in the configuration procedures we design. Firstly, we do not need to supply the parameter /epsilon1 ahead of time. Instead, the procedure maintains an internal /epsilon1 that it can guarantee, continually shrinking this toward 0. Secondly, the number of samples needed to eliminate /epsilon1 -suboptimal algorithms grows with the square of the suboptimality gap ∆ i = max i ′ U i ′ -U i rather than the square of /epsilon1 . Theorem 1 gives an input-dependent guarantee for the Runtime Oracle procedure that we would like to preserve in the usable procedures we design. If some algorithm i is very suboptimal, with ∆ i /greatermuch /epsilon1 , then we will be able to eliminate it with many fewer samples than an algorithm that is almost /epsilon1 -optimal.

The Runtime Oracle procedure gives us a baseline against which to compare. Its sample complexity guarantee is essentially optimal in the following data-dependent sense.

Theorem 2 (Theorem 5 of Mannor and Tsitsiklis (2004)) . There exists some set of input distributions for which every oracle procedure that returns an /epsilon1 -optimal algorithm with probability at least 1 -δ must take Ω ( n -| N ( /epsilon1 ) | /epsilon1 2 log 1 δ + ∑ i ∈ N ( /epsilon1 ) 1 ∆ 2 i log 1 δ ) samples, where N ( /epsilon1 ) is the set of /epsilon1 -suboptimal algorithms.

3.2 Hypothetical Captime Verification Procedure

We now consider a lower bound on the captime we will be required to use, even in a world where we do not need to take samples. We imagine there are a prover and a skeptic . The prover has access to the runtime CDF of each algorithm i . The skeptic will get to see each runtime CDF only up to some κ i that the prover will choose. The prover then recommends an algorithm i ∗ and the skeptic should be convinced that i ∗ is /epsilon1 -optimal, based only on the truncated CDFs. As the κ i 's tend to infinity, the skeptic will see more and more of the true CDFs and therefore eventually be convinced of the prover's claim (assuming it is true). The goal is to convince the skeptic that i ∗ is /epsilon1 -optimal using κ i 's that are as small as possible. The skeptic will be able to compute the following deterministic bounds

Algorithm 2 Captime Verification Procedure

on an algorithm's expected utility

The skeptic knows that LB i ≤ U i ≤ UB i , but also that the runtime distributions could be such as to make U i fall anywhere in this range. So whatever algorithm i ∗ the prover returns to the skeptic, the skeptic will be convinced that i ∗ is /epsilon1 -optimal if, and only if, they observe LB i ∗ ≥ UB i -/epsilon1 for all i = i ∗ . The following lemma justifies this condition.

/negationslash

/negationslash

Lemma 5. If LB i ∗ ≥ UB i -/epsilon1 for all i = i ∗ , then i ∗ is /epsilon1 -optimal. If there exists an i = i ∗ with LB i ∗ < UB i -/epsilon1 , then regardless of the values of the CDFs F i up to the captimes κ i , there are some input distributions for which i ∗ is not /epsilon1 -optimal.

/negationslash

The proof works by constructing counter-example CDFs, but takes the CDFs as given up κ i , so does not rely on specifying any information available to the skeptic. No matter what the CDF of each i looks like up to κ i , there will be some inputs with LB i ∗ < UB i -/epsilon1 for which i ∗ is not /epsilon1 -optimal.

Algorithm 2 describes the prover-skeptic interaction. The next lemma shows that using these captimes is essentially optimal. It gives us a baseline minimum captime against which to compare.

Lemma 6. The skeptic will be convinced that both i opt and i ∗ = arg max i LB i are /epsilon1 -optimal if the captimes κ i are large enough that u ( κ i ) ( 1 -F i ( κ i ) ) ≤ ∆ i + /epsilon1 2 for all algorithms i .

Lemma 7. There are some inputs for which any verification procedure must use captimes κ i large enough that u ( κ i ) ( 1 -F i ( κ i ) ) ≤ ∆ i + /epsilon1 for all algorithms i , or the skeptic will not be convinced that the returned algorithm is /epsilon1 -optimal.

The proof simply constructs a counterexample for which doing runs at a smaller captime will lead any configuration procedure to make a wrong conclusion.

3.3 Naive Procedure

We now turn to the setting we are actually interested in, where runs cost time and long ones must eventually be terminated. The simplest thing we could do is just pick a captime (or perhaps we somehow 'know' the right captime) and do the necessary number of runs. With a fixed captime, the scenario is not much different from the runtime oracle scenario above. The main difference is that the 'resolution' at which we can understand an algorithm's expected utility will be limited by the size of the captime we use to take our samples. We will never know what the runtime CDF looks like beyond the captime, no matter how many samples we take. Given any κ , the confidence bounds described in

Algorithm 3 Naive Procedure

Section 2 tell us we could simply choose the smallest m satisfying 2 √ ln(2 n/ δ ) 2 m + u ( κ ) ≤ /epsilon1 , do m runs of each algorithm at captime κ , then return the one with largest empirical average utility. We can think of 2 √ ln(2 n/ δ ) 2 m as the error due to sampling, and u ( κ ) as the error due to capping. This is the Naive procedure.

Theorem 3. With probability at least 1 -δ the Naive procedure returns an /epsilon1 -optimal algorithm. It takes enough κ -capped runtime samples of each algorithm to ensure that 2 √ ln(2 n/ δ ) 2 m + u ( κ ) ≤ /epsilon1 .

The proof of this and of subsequent theorems are deferred to Appendix A. Each essentially follows from the fact that the upper and lower confidence bounds are specified to satisfy Hoeffding's inequality and the union bound. Comparing Theorem 3 to Theorem 1 we can see the reason for the increased number of samples required when we observe capped instead of uncapped runs: if u ( κ ) is relatively large, it will take a much larger m to shrink the term 2 √ ln(2 n/ δ ) 2 m enough to satisfy the inequality in Theorem 3.

The Naive procedure has some undesirable qualities. Choosing the right κ may not be easy and in Section 4 we will show what a difference this choice can make for the total runtime of the Naive procedure. We are also required to specify an /epsilon1 beforehand; it may hard to know the 'right' /epsilon1 , and the best m and κ for one /epsilon1 may be quite different from the best for a slightly different /epsilon1 . What's more, this procedure ignores information about an algorithm's runtime distribution that could be learned by observing runs along the way; it essentially assumes that every algorithm always times out at the given captime. As a result, it takes more samples of /epsilon1 -suboptimal algorithms than is necessary. The UP procedure corrects these defects.

3.4 Utilitarian Procrastination

Our Utilitarian Procrastination (UP) procedure starts by doing runs at the smallest captime possible, trying to rule out whatever configurations we can, and only starts doing runs at a larger captime when necessary.

In Theorem 4 we can clearly see the analog of both Theorem 1 and Theorem 3. UP is input dependent in two senses. The condition for ruling out a suboptimal i depends on ∆ i and also on the runtime CDFs F i and F i opt . Just as in the pure sample complexity case of Theorem 1, if i is very suboptimal and ∆ i is large, it will be easier to satisfy the condition in Theorem 4 and thus rule i out. Additionally, the condition will be easier to satisfy if the inputs are such that either i or i opt is able to complete a lot of runs at their respective captimes, so that their CDFs are close to 1. Theorem 4 also implies the following theorem and corollary, which together constitute our main theoretical result.

Algorithm 4 Utilitarian Procrastination

Theorem 5. For any m , let /epsilon1 = 3 √ ln(11 nm 4 / δ ) 2 m . Then with probability at least 1 -δ , UP returns an /epsilon1 -optimal algorithm once it takes m samples. At that point, if κ ∗ i is the largest captime algorithm i has been run with then κ ∗ i ≤ 2 inf { κ : u ( κ ) ( 1 -F i ( κ ) ) < /epsilon1 3 √ 2 } .

Corollary 1. With probability at least 1 -δ , UP returns an /epsilon1 -optimal algorithm after taking a number of samples that at most a logarithmic factor more than is optimally required by any procedure, and at a captime that is a constant larger than inf { κ : u ( κ ) ( 1 -F i ( κ ) ) < /epsilon1 3 √ 2 } .

The proof follows from Theorem 4 and from UP's specific choice of captime doubling condition. We note that there is room for improvement, but that this captime bound is comparable to the worst-case lower bound captime needed by any configuration procedure as presented in Section 3.2. The smallest captime that satisfies u ( κ ) ( 1 -F i ( κ ) ) < /epsilon1 3 √ 2 needs to be larger than the smallest captime that satisfies u ( κ ) ( 1 -F i ( κ ) ) < ∆ i + /epsilon1 , which is the best we can hope to do.

Proof. The number of samples follows immediately from the definition of /epsilon1 in Theorem 5. Because the captimes are always doubled, the total time spent by UP on any single instance is at most a constant times larger than the time spent running that instance the final time it was run. Since the captime at that point was at most κ ∗ i , the total time spent running any instance is at most a constant times inf { κ : u ( κ ) ( 1 -F i ( κ ) ) < /epsilon1 3 √ 2 } .

4 Experiments

We now illustrate the runtime costs of utilitarian algorithm configuration and the impacts of the adaptive improvements offered by UP over Naive. We would have liked to go further, and in particular to offer comparisons against baselines other than Naive. However, we saw no straightforward way of doing so: there is simply no previous work on offering algorithm configuration in the utilitymaximizing setting. Our paper focuses on algorithm configuration with theoretical guarantees. There

do exist many methods with guarantees in the runtime minimizing setting; we could have run existing procedures like Structured Procrastination, LeapsAndBounds, or CapsAndRuns and then evaluated them in terms of a utilitarian objective. However, such an apples-to-oranges comparison would likely have yielded very poor performance and regardless would have dispensed with the guarantees that motivate these methods. Second, we could have compared to heuristic methods like SMAC, ParamILS and GGA. Again, however, they optimize a different objective function. It is possible to imagine modifying one of these algorithms to optimize utility, but doing so would require fundamental algorithmic changes (e.g., to their so-called adaptive capping heuristics) and nontrivial software engineering effort. Even if we could have made such changes in a non-controversial way, comparing heuristic methods to methods offering guarantees would again be an apples-to-oranges comparison. Perhaps for these reasons, we note that no paper of which we are aware has yet compared any heuristic algorithm configuration method to any offering theoretical guarantees. We do intend to pursue such an investigation ourselves in future work, but anticipate that a careful study of this question will require an entire research paper.

Experimental Setup. We leverage three datasets from Weisz et al. (2020). The first is a set of runtimes for the minisat SAT solver on data generated by the CNFuzzdd instance generator. The others are sets of runtimes for the CPLEX integer program solver on the combinatorial auction winner determination instances (regions) and on woodpecker conservation problems (rcw); see Appendix D of Weisz et al. (2020) for details. We only used the first seed for the CPLEX datasets. 1

Choosing the Right Captime Matters. In Fig. 1 we plot the total configuration time of the Naive procedure with different input captimes κ on three datasets from Weisz et al. (2020) using a logLaplace utility function from Graham et al. (2023): u ( t ) = u LL ( t ; 60 , 1) = 1 -1 2 t 60 if t ≤ 60 and u ( t ) = 1 2 60 t otherwise. We used /epsilon1 values of 0 . 1 , 0 . 15 and 0 . 2 and set δ = 0 . 1 . The first observation we make is that choosing a bad captime can have a large effect on total configuration time, especially for smaller /epsilon1 values. The second observation is that, for any fixed κ , different values of /epsilon1 can also have huge effects on total runtime of the configuration procedure. Both of these points help to emphasize the need for a procedure like UP that starts out with a small κ , only increasing it as needed, and refines the /epsilon1 it can guarantee 'on the fly', based on the runs it has observed.

Anytime Speedups Matter. In Fig. 2, we plot the total configuration time of UP and Naive as a function of /epsilon1 . We used the same log-Laplace utility function as above and set δ = 0 . 1 (within reasonable ranges, the value of δ has relatively little effect on total runtime). UP can drastically outperform Naive, especially for smaller values of /epsilon1 .

Unsurprisingly, being anytime in κ helps most when we do not know how to provide a good κ as input beforehand. If we somehow guess or know the right captime to use, then it can be hard to beat Naive. But if we use the wrong captime, UP can be much faster than Naive. In Fig. 3 we use a uniform utility function from Graham et al. (2023), with u ( t ) = u unif ( t ; 60) = 1 -t 60 for t < 60 and 0 otherwise. In the top row, we have used a captime of κ = 60 for Naive, which is appropriate for this utility function since it is linear up to that point and then 0 thereafter. In the bottom row, we have set the captime poorly at κ = 600 , meaning that we are not terminating some runs even though

they are so long they give us 0 utility. We can see that if we get the captime right, then Naive will perform quite well. But if we get it wrong, Naive can drastically underperform compared to UP.

5 Conclusion

We have presented Utilitarian Procrastination (UP), a utility-based approach to automated algorithm configuration. This is the first procedure that we know of to incorporate utility functions into the algorithm configuration framework. UP is anytime, meaning it requires minimal input from the user and continues to refine its guarantees as it is run. In simple experiments, we show that by freeing the user from having to provide either an accuracy parameter /epsilon1 or a captime κ UP can help avoid the excessively long runtimes that come when these inputs are chosen poorly. Of course we would have liked to have shown that UP never uses a captime larger than κ ∗ i ≤ 2 inf { κ : u ( κ ) ( 1 -F i ( κ ) ) < ∆ i + /epsilon1 2 } , matching the hypothetical procedure. The bound we do show is a result of the condition UP checks when deciding whether to double the the captime κ i . UP uses a simple condition that balances the error due to sampling with the error due to capping. Other, more intelligent choices may be possible, but improving this will have to wait for future work.

References

Ansótegui, C., Sellmann, M., and Tierney, K. (2009). A gender-based genetic algorithm for the automatic configuration of algorithms. In International Conference on Principles and Practice of Constraint Programming , pages 142-157. Springer.

Balcan, M.-F., DeBlasio, D., Dick, T., Kingsford, C., Sandholm, T., and Vitercik, E. (2021). How much data is sufficient to learn high-performing algorithms? generalization guarantees for datadriven algorithm design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 919-932.

Balcan, M.-F., Nagarajan, V., Vitercik, E., and White, C. (2017). Learning-theoretic foundations of algorithm configuration for combinatorial partitioning problems. In Conference on Learning Theory , pages 213-274. PMLR.

Birattari, M., Stützle, T., Paquete, L., and Varrentrapp, K. (2002). A racing algorithm for configuring metaheuristics. In Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation , page 11-18. Morgan Kaufmann Publishers Inc.

Brandt, J., Schede, E., Haddenhorst, B., Bengs, V., Hüllermeier, E., and Tierney, K. (2023). Ac-band: A combinatorial bandit-based approach to algorithm configuration. Proceedings of the AAAI Conference on Artificial Intelligence , 37(10):12355-12363.

Even-Dar, E., Mannor, S., Mansour, Y., and Mahadevan, S. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6).

Graham, D. R., Leyton-Brown, K., and Roughgarden, T. (2023). Formalizing preferences over runtime distributions. In International Conference on Machine Learning , pages 11659-11682. PMLR.

Gupta, R. and Roughgarden, T. (2017). A pac approach to application-specific algorithm selection. SIAM Journal on Computing , 46(3):992-1017.

Hoos, H. H. and Stützle, T. (2004). Stochastic local search: Foundations and applications . Elsevier.

Hutter, F., H. Hoos, H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In International Conference on Learning and Intelligent Optimization , pages 507-523. Springer.

Hutter, F., Hoos, H. H., Leyton-Brown, K., and Stützle, T. (2009). Paramils: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research , 36:267-306.

Kleinberg, R., Leyton-Brown, K., and Lucier, B. (2017). Efficiency through procrastination: Approximately optimal algorithm configuration with runtime guarantees. In IJCAI , volume 3, page 1.

Kleinberg, R., Leyton-Brown, K., Lucier, B., and Graham, D. (2019). Procrastinating with confidence: Near-optimal, anytime, adaptive algorithm configuration. arXiv preprint arXiv:1902.05454 .

Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research , 18(1):6765-6816.

López-Ibáñez, M., Dubois-Lacoste, J., Cáceres, L. P., Birattari, M., and Stützle, T. (2016). The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives , 3:43-58.

Mannor, S. and Tsitsiklis, J. N. (2004). The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research , 5(Jun):623-648.

Tornede, A., Wever, M., Werner, S., Mohr, F., and Hüllermeier, E. (2020). Run2survive: a decisiontheoretic approach to algorithm selection based on survival analysis. In Asian Conference on Machine Learning , pages 737-752. PMLR.

Von Neumann, J. and Morgenstern, O. (1944). Theory of games and economic behavior . Princeton university press.

Weisz, G., György, A., Lin, W.-I., Graham, D., Leyton-Brown, K., Szepesvari, C., and Lucier, B. (2020). Impatientcapsandruns: Approximately optimal algorithm configuration from an infinite pool. Advances in Neural Information Processing Systems , 33:17478-17488.

Weisz, G., Gyorgy, A., and Szepesvári, C. (2018). Leapsandbounds: A method for approximately optimal algorithm configuration. In International Conference on Machine Learning , pages 52575265. PMLR.

Weisz, G., Gyorgy, A., and Szepesvári, C. (2019). Capsandruns: An improved method for approximately optimal algorithm configuration. In International Conference on Machine Learning , pages 6707-6715. PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the algorithm configuration problem with a monotone (invertible) utility function. We are given a set of $n$ algorithms, and instances are drawn from a distribution, $\\mathcal{D}$. We can observe samples of running times of $i\\in [n]$ on $j\\sim\\mathcal{D}$ capped by $\\kappa>0$, and we aim to find an $\\epsilon$-optimal algorithm from $[n]$. The characteristic of the setting studied in this paper is that the cost paid for each sample depends on the time we spent collecting it and that only capped runtime can be observed. \n\nThe authors first discuss lower bounds on the number of samples and captime required for identifying an $\\epsilon$-optimal algorithm (Sections 3.1 and 3.2). The result on captime was constructively proved via a prover--skeptic procedure. The authors then give a naive configuration method (Naive), which requires specifying $\\epsilon$. This requirement is impractical since a slight change in $\\epsilon$ may largely affect appropriate $m$ and $\\kappa$ values (this is also confirmed empirically in Section 4). Section 3.4 addresses this problem and presents the Utilitarian Procrastination method (UP), an anytime algorithm that starts from the smallest captime and increases it if necessary. Experiments in Section 4 demonstrate the advantage of UP over Naive.\n\nSTRENGTHS:\n1. The paper studies an interesting direction of algorithm configuration based on the utility-theoretic approach of (Graham et al. 2022).\n2. The setup and algorithms are well described. \n3. The technical motivation of UP is described clearly. The results are fairly strong, considering the lower bounds.\n\nWEAKNESSES:\n1. The technical novelty appears to be somewhat marginal, given that the idea of procrastination is already developed.\n2. The utilitarian setting, albeit interesting, looks somewhat artificial. It is not easy to imagine realistic utility functions that naturally appear in practice. \n3. The lower bound in Section 3.2 seems to be considering too pessimistic inputs, which in turn makes it easier to obtain nearly tight upper bounds in the subsequent sections (although similar tightness results are often appreciated).\n\nQUESTIONS:\n1. Although algorithm configuration with bounded utility functions is new, I have not felt it is technically very different from the previous setting (Kleinberg et al. 2017, 2019), particularly because monotone and invertible utility functions could simplify technical treatment. I would appreciate it if the authors could describe the main technical novelty in the utilitarian setting. \n2. Are there natural examples of utility functions used in practice?\n3. While $u$ is assumed to be invertible in Section 2, the function used in Section 4 (i.e., $u(t) = 10/t$ for $t>10$ and $u(t) = 0$ o.w.) is not on $\\\\{t: u(t)=0 \\\\}$. Is it possible to relax the assumption to deal with such non-invertible functions? (As far as I see, the invertibility on $\\\\{t: u(t)>0\\\\}$ seems sufficient).\n\nLIMITATIONS:\nWhile not explicitly discussed, the range to which the results apply appears to be clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
The paper presents the first nontrivial procedure for configuring heuristic algorithms to  maximize the utility provided to their end users while also offering theoretical guarantees about performance. The paper shows how to avoid extremely long runs with low probability by putting limits on the runtime, assuming a bounded utility that is monotonically decreasing in runtime. The paper compares several ideas
(1) the idea behind the runtime Oracle Procedure that eliminates algorithms progressively;
(2) the idea of using a time limit whose value doubles until the optimal algorithm is identified (this idea is also used in restart procedures)
(3) estimations of the mean time and the fraction of the distribution that has been explored

The results shows that the proposed procedure returns an optimal algorithm with a certain probability with a number of samples. It also characterizes the upper bound on the threshold. The results are with a logarithmic factor of optimality for the samples and provide a constant bound for the time limit.

STRENGTHS:
- novel algorithms with performance guarantees
- well-written and intuitive paper
- automatic tuning of the time limit

WEAKNESSES:
- experimental results could be stronger and use a lot more problems

QUESTIONS:
no question

LIMITATIONS:
yes, the authors mention that the bounds are not tight yet.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTheoretical guarantees for algorithm configuration (AC) when using a utilitarian objective function are proposed. This work differs from previous work in several ways, the first being that it focuses on the \"utilitarian\" objective function, which differs from the objective functions used in essentially all previous work. Second, the algorithm proposed provides anytime theoretical guarantees, meaning key parameters from previous approaches need not be input. A short experimental evaluation is performed on some well-known datasets for examining theoretical AC approaches to further drive home the results from the theorems.\n\nSTRENGTHS:\n1. The paper is well-organized and clearly written.\n\n2. The introduction of an anytime approach for AC with theoretical guarantees is a rather significant result. One of the key drawbacks of previous work is that the decision maker must choose values like epsilon and kappa that have little to no meaning to a non-expert. Integrating these values into the approach itself is a very welcome advancement that will have an impact on future work in this area.\n\n3. I am not sure whether the use of the utilitarian objective function is a strength or weakness, it has both aspects to it, so I include it here. On the one hand, the paper bases all of its work on the utilitarian objective function, which has not caught on with the empirical side of AC yet. Perhaps it will. At any rate, while it offers a way forward in terms of the proofs that was potentially unavailable before, it also somewhat strands this work away from the rest of the AC literature, making comparisons difficult and it somewhat hard to assess exactly where this work stands. I think overall the utilitarian objective function is a net positive, hence I place this comment under strengths.\n\nWEAKNESSES:\n1. I found the proofs both in the paper and in the supplementary material (although I admit I have not fully checked the supplementary material) somewhat... let's call it almost arrogant? I am sorry to use that word, but for less mathematically inclined readers, I think they will come away from this paper feeling rather stupid. I know I did.\n\n2. Regarding lemma 1, and potentially putting myself in the category of stupid reviewers, I do not understand how U_i <= U_i(kappa). I just do not understand how the uncapped expected utility could possibly be less than the capped utility. Clearly, they can in some cases be equal. But is this a typo? This is an example where an actual proof would lead to better understanding.\n\n3. I am including the dataset used as a weakness, because a better dataset is sorely needed for comparing theoretical approaches. I do note that the dataset used is better than nothing, and this point does not influence my score, as the authors practically have no choice other than to expend significant computational resources.\n\n4. I would have appreciated a more detailed explanation of utilitarian AC. The reader really cannot be expected to know about or have read the previous paper in detail to evaluate this one. In the future, this would serve as a refresher for those who have read the utilitarian paper and need to be reminded.\n\n5. The authors provide overall a good justification for not comparing to previous techniques, I do wonder, though, what the performance would be like if one modeled a utility function after the functions usually used -- would this not be possible?\n\n6. The authors left out the most recent theoretical approach, AC-Band, in the related work, but this is not important for my evaluation.\n\nQUESTIONS:\n1. Relating to weakness #5, would it not be possible to craft a utility function close to, say, PAR10 or mean runtime for comparison purposes?\n\n2. In AC, mean runtime and PAR10 have become standard functions because (1) they are easy to compute and (2) they relatively closely reflect what algorithm users actually want. Long runtimes are indeed reduced by PAR10, even if the 10 is very arbitrary. If I do not have many timeouts, is the utility function still \"better\"? Are there some clear use cases that can be described?\n\nLIMITATIONS:\nThe limitations are adequately discussed within the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a setting and an algorithm for algorithm configuration based on utility of the algorithm's running time. Theoretical guarantees are provided and upper and lower bounds are compared against. The algorithm is evaluated on case studies.\n\nSTRENGTHS:\nUtility-based decision making approach is applied to algorithm configuration. The proposed approach is theoretically analyzed and empirically evaluated.\n\nWEAKNESSES:\n## Literature\n\nOn time (or resource in general) constrained utility based decision making, start with this seminal work by Stuart Russell and his PhD student Eric Wefald: https://mitpress.mit.edu/9780262513821/do-the-right-thing/\n\nFor a   more updated recent account on bounded rationality (that is, on making utility based decisions under computational resource constraints) look at this work (UAI 2012): https://arxiv.org/abs/1207.5879\n\nEric Horvitz is the 'father' of resource-bounded decision making.  Citations on bounded rationality are here: http://erichorvitz.com/computational_rationality_readings.htm. A good collection of articles coedited by Eric Horvitz and Shlomo Zilberstein is in this special issue of AIJ: https://www.sciencedirect.com/journal/artificial-intelligence/vol/126/issue/1\n\n## Problem with the submission under discussion\n\nAn algorithm is selected based on its expected net utility. The net utility consists of two terms, the intrinsic utility and the deliberation cost. Relevant work on algorithm configuration assumes that all algorithms have the same intrinsic utility (solve the problem 'exactly' or with solutions of indistinguishable quality), and their total utility depends on deliberation cost --- which is in many settings just the running time. \n\nIntuitively, the intrinsic utility is what you get and the deliberation cost is what you pay. The theory of utility to which the authors refer applies to the intrinsic utility.  The computation/time cost is an additive negative term in the total utility, which conforms to different laws, see the citations above for detailed discussion. \n\nThe authors confuse the intrinsic utility and the time cost, and apply non-linear bounding transformation to the running time instead of to a parameter of the solution quality. They then come up with an algorithm that allows to select algorithms more efficiently than the baseline based on the time cost. However, due to inherent inconsistency in applying bounding transformation to the time cost, the authors' approach results in a selection/configuration scheme that may prefer an algorithm that does not always terminate to an algorithm that terminates, even within reasonable time limits. This is just an extreme example.\n\nAs another intuition, assume there are two luxury car models, one costs USD1 million, the other USD2 million. The authors argue that USD1 million is such a huge sum that USD1 million or USD2 million do not differ much, and if you have USD1000000 but want the USD2000000 car, you **can just add some pocket change** to buy it. However, this will most probably not work. To get a USD2000000 car you still need twice amount of money you would pay for USD1000000 car, the amount of money you can buy **2 USD1000000 cars**.  However, the value you get from a USD2000000 car, whether for vanity or practicality, is not twice the value of USD1000000 car, it is likely to be the same or just a little bit higher (unless there is a club of USD2000000 car owners you want badly to be accepted to). \nHere, the car price is the cost (deliberation cost, time cost in algorithm selection) and the value is the intrinsic utility (how good the algorithm solution is). \n\nThe problem in the submission is in improper use of utility-based decision making to select computations. The theory is detailed in the works cited above.\n\nQUESTIONS:\nIs this the net or the intrinsic utility?\n\nLIMITATIONS:\nThe limitations are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}