SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Devon R. Graham
drgraham@cs.cs.ubc.ca
Kevin Leyton-Brown
kevinlb@cs.ubc.ca
Columbia University & a16z crypto
We present the first nontrivial procedure for configuring heuristic algorithms to maximize the utility provided to their end users while also offering theoretical guarantees about performance. Existing procedures seek configurations that minimize expected runtime. However, very recent theoretical work argues that expected runtime minimization fails to capture algorithm designers' preferences. Here we show that the utilitarian objective also confers significant algorithmic benefits. Intuitively, this is because mean runtime is dominated by extremely long runs even when they are incredibly rare; indeed, even when an algorithm never gives rise to such long runs, configuration procedures that provably minimize mean runtime must perform a huge number of experiments to demonstrate this fact. In contrast, utility is bounded and monotonically decreasing in runtime, allowing for meaningful empirical bounds on a configuration's performance. This paper builds on this idea to describe effective and theoretically sound configuration procedures. We prove upper bounds on the runtime of these procedures that are similar to theoretical lower bounds, while also demonstrating their performance empirically.
Heuristic algorithms are surprisingly effective at solving hard computational problems. However, no single set of heuristics works well on all problems; the design choices must be tuned to a distribution of problem instances in the same way that the parameters of a machine learning model are tuned to a dataset. This black-box optimization problem is called algorithm configuration. Roughly speaking, the literature on algorithm configuration is divided into two camps. The older and larger camp designs heuristic procedures that aim to find good configurations as quickly as possible (Birattari et al., 2002; Hutter et al., 2009, 2011, Ansótegui et al., 2009, López-Íbanez et al., 2016). A second, more recent line of work aims to provably identify approximately optimal configurations and to offer guarantees about the runtime required to do so (Kleinberg et al., 2017; Weisz et al., 2018; Kleinberg et al., 2019; Weisz et al., 2019, 2020). Virtually all of the work in both camps has focused on minimizing some version of capped average runtime, although one exception of which we are aware is Tornede et al. (2020), which explores objectives like higher-order polynomials of runtime that might better capture a user's level of risk-aversion.
Very recently, a theoretical argument has been made that minimizing expected runtime is inconsistent with a plausible set of axioms about algorithm designers' preferences (Graham et al. 2023). Runtime is an economic good and, like other economic goods, the value an individual assigns to it is not
necessarily proportional to the value of the good itself—nor should it be. We can characterize a set of bounded and monotonically decreasing utility functions that are implied by certain axioms. These are essentially the von Neumann-Morgenstern axioms (Von Neumann and Morgenstern 1944) from classical decision theory, plus two novel runtime-specific axioms that assert a (weak) preference for faster runs over slower ones, and a (strict) preference for algorithm runs that complete over ones that time out. Preferences that follow these axioms are described by a utility function that is monotonically decreasing in runtime from 1 to 0. This utility function can incorporate factors like the benefit an end user attains from completing a run, the cost they pay for cloud computing resources, and any uncertainty they may have about when an answer will stop being useful to them. This utilitarian perspective opens a new and exciting direction for algorithm configuration.
While it is reassuring to know that optimizing utility instead of runtime is the right thing to do, it also offers significant algorithmic benefits. Because utilities are bounded and monotonically decreasing, exceedingly long runs contribute negligibly to estimates of an algorithm's expected utility. This means that there will always be some captime that allows us to accurately estimate true expected utility from capped samples. We can use tools from the multi-armed bandit literature for best arm identification, and the algorithms we describe will draw on the elimination and racing algorithms described in [Mannor and Tsitsiklis (2004); Even-Dar et al. (2006) and elsewhere. In contrast, runtime-based configuration procedures like Structured Proccrastination (Kleinberg et al. 2017), Structured Proccrastination with Confidence (Kleinberg et al. 2019), LeapsAndBounds (Weisz et al. 2018), CapsAndRuns (Weisz et al. 2019), and ImpatientCapsAndRuns (Weisz et al. 2020) need to do many runs at large captimes in order to make theoretical guarantees. Furthermore, in order to be able to make any provable guarantees at all, they have to introduce an additional parameter specifying how much of the runtime CDF can be ignored; even if an algorithm has always finished very quickly on every instance seen so far, it could always take so long on the next instance that its expected runtime is arbitrarily large. In this setting, no optimality guarantees can be made without broadening the definition of optimality.
Other theoretically-motivated methods such as Gupta and Roughgarden (2017); Balcan et al. (2017) offer performance guarantees based on different measures of complexity and guarantee notions of PAC optimality akin to those we present here. These papers do not focus on runtime, instead studying traditional sample complexity, with each sample (i.e., algorithm run) contributing the same "sampling cost." In our algorithm configuration setting, the cost of a sample is the amount of (capped) runtime spent acquiring it, with longer captimes potentially leading to more expensive (but also more informative) samples. Because of this difference, these two lines of work propose quite different types of learning algorithms. However, Balcan et al. (2021) do assume the existence of a bounded utility function which measures algorithm performance, and others (Hoos and Stützle, 2004) have argued that utility functions of this form are better objectives to optimize than runtime when choosing algorithms. Some methods have been specifically designed to exploit the parallel nature of the algorithm configuration problem; AC-Band (Brandt et al. 2023) is a bandit-based procedure inspired by the Hyperband algorithm (Li et al., 2017) that runs multiple configurations simultaneously while ruling out poor-performing ones along the way and offering theoretical guarantees with respect to the set of configurations considered.
Inspired by the mantra of procrastination that has found success in previous work, and by the benefits that come with the use of utility functions, this paper presents a procedure we dub Utilitarian Procrastination (UP), so-named because it performs as many low-captime runs as possible before proceeding to higher-captime runs. We offer theoretical guarantees about its performance, showing that it will return a good configuration and proving that its worst-case upper bound is similar to the theoretical lower bound that any procedure must require. We also present experimental results showing how this procedure performs in practice and how it compares to a more naive baseline.
We assume there is a monotonically decreasing runtime utility function  $u: \mathbb{R}_{\geq 0} \to [0,1]$  with  $u(0) = 1$  and  $\lim_{t \to \infty} u(t) = 0$ . The existence of  $u$  follows from simple axioms (Graham et al. 2023). The value  $u(t)$  describes the (expected) well-being of an individual who uses an algorithm to solve a problem instance (e.g., an integer program). During the configuration process, the goal will be to choose an algorithm that maximizes  $u(t)$ , in expectation over  $t$ .
Given a set of algorithms indexed by  $i = 1, \dots, n$ , our goal is to find an approximately optimal algorithm using capped runtime samples. We assume we have access to a stream of input instances indexed by  $j = 1, 2, \ldots$  drawn from some distribution  $\mathcal{D}_j$ . We will use  $t_{ij}$  to denote the true uncapped runtime of algorithm  $i$  on input  $j$ , and  $t_{ij}(\kappa) = \min(t_{ij}, \kappa)$  to be the  $\kappa$ -capped runtime of  $i$  on  $j$ . When we do a run, we observe  $t_{ij}(\kappa)$ , not  $t_{ij}$ . Of course these coincide for any run that completes. The instance distribution  $\mathcal{D}_j$ , along with any randomness of the algorithm or execution environment will together induce a runtime distribution for each algorithm  $i$ . We will use  $\mathcal{D}_i$  to denote this runtime distribution, and  $F_i$  to denote its CDF. For each algorithm  $i$ , the true uncapped expected utility is
$$
U _ {i} = \underset {t \sim D _ {i}} {\mathbb {E}} \left[ u (t) \right].
$$
The capped expected utility is
$$
U _ {i} (\kappa) = \underset {t \sim D _ {i}} {\mathbb {E}} \left[ u \left(\min  (t, \kappa)\right) \right].
$$
And given any capped runtime samples  $t_{i1}(\kappa),\dots,t_{im}(\kappa)$  , the capped empirical average utility is
$$
\widehat {U} _ {i m} (\kappa) = \frac {1}{m} \sum_ {j = 1} ^ {m} u \big (t _ {i j} (\kappa) \big).
$$
Definition 1 (ε-optimal). An algorithm  $i^*$  is called ε-optimal if  $U_{i^*} \geq \max_i U_i - \epsilon$ .
Our goal will be to find  $\epsilon$ -optimal algorithms with probability at least  $1 - \delta$ , and to do so as quickly as possible. Our first lemma deterministically bounds an algorithm's uncapped expected utility in terms of its capped expected utility and CDF.
Lemma 1. For any  $i$  and  $\kappa$  we deterministically have  $U_{i}(\kappa) - u(\kappa)\big(1 - F_{i}(\kappa)\big)\leq U_{i}\leq U_{i}(\kappa)$ .
Proof. This follows from the law of total expectation and the fact that  $u$  is non-increasing, and so  $u(\kappa) \geq u(t)$  for all  $t \geq \kappa$ .
Intuitively, Lemma  $\square$  is true because the capped expected utility counts runs that cap as having just completed at the captime, when really they would have taken longer if given the chance. This makes the capped expected utility look more favourable than the uncapped expected utility. Our second lemma shows that if we do runs at captime  $\kappa$ , then we can accurately estimate the algorithm's runtime CDF at  $\kappa$ .
Lemma 2. For any  $i$ ,  $m$ ,  $\kappa$  and  $\delta$ , let  $\widehat{F}_{im}(\kappa)$  be the fraction of the  $m$  runs that  $A$  completes within captime  $\kappa$ . Then  $\widehat{F}_{im}(\kappa) - \sqrt{\frac{\ln(1 / \delta)}{2m}} \leq F_i(\kappa) \leq \widehat{F}_{im}(\kappa) + \sqrt{\frac{\ln(1 / \delta)}{2m}}$  with probability at least  $1 - 2\delta$ .
Proof. For each  $j$ , let  $X_{j} = 1$  if  $t_{ij} < \kappa$  and  $X_{j} = 0$  otherwise. The proof then follows from a straightforward application of Hoeffding's inequality.
Lemma simply says that if we take enough samples at captime  $\kappa$ , the fraction of those runs that complete will be close to the true likelihood of a run completing before  $\kappa$ . Our third lemma shows that expected capped utility can be estimated accurately using capped runtime samples.
Lemma 3. For any  $i, m, \kappa$  and  $\delta$  we have  $\widehat{U}_{im}(\kappa) - (1 - u(\kappa))\sqrt{\frac{\ln(1 / \delta)}{2m}} \leq U_i(\kappa) \leq \widehat{U}_{im}(\kappa) + (1 - u(\kappa))\sqrt{\frac{\ln(1 / \delta)}{2m}}$  with probability at least  $1 - 2\delta$ .
Proof. Since  $u\big(t_{ij}(\kappa)\big) \in [u(\kappa), 1]$  for all  $j$ , the proof follows immediately from Hoeffding's inequality.
Lemma 3 simply says that if we take enough samples, then the empirical capped average utility will be close to the true expected capped utility. Together, these lemmas imply empirical confidence bounds on the true expected utility. Define the upper and lower confidence bounds
$$
\begin{array}{l} U C B _ {i m} (\kappa) = \widehat {U} _ {i m} (\kappa) + \left(1 - u (\kappa)\right) \sqrt {\frac {\ln (4 n / \delta)}{2 m}} \\ L C B _ {i m} (\kappa) = \widehat {U} _ {i m} (\kappa) - \sqrt {\frac {\ln (4 n / \delta)}{2 m}} - u (\kappa) (1 - \widehat {F} _ {i m} (\kappa)). \\ \end{array}
$$
The next lemma shows that these are good confidence bounds if both the captime  $\kappa$  and the number of samples  $m$  are sufficiently large. With probability at least  $1 - \delta$ , each algorithm's true expected utility will fall within these confidence bounds, and the width of each confidence range will not be too large.
Lemma 4. If we do  $m$  runs of each algorithm at captime  $\kappa$ , then with probability at least  $1 - \delta$  we will have
$$
L C B _ {i m} \leq U _ {i} \leq U C B _ {i m}
$$
$$
U C B _ {i m} - L C B _ {i m} \leq 2 \sqrt {\frac {\ln (4 n / \delta)}{2 m}} + u (\kappa) \big (1 - F _ {i} (\kappa) \big)
$$
for all  $i$  simultaneously.
See Appendix A for a full proof. The idea is that the definition of the bounds together with Lemmas 2 and 3 mean the confidence bounds hold and are accurate.
We can see in Lemma 4 that the error in our estimate of an algorithm's expected utility comes from two sources: sampling and capping. The term  $2\sqrt{\frac{\ln(4n / \delta)}{2m}}$  represents the error due to sampling, while the term  $u(\kappa)\left(1 - F_i(\kappa)\right)$  represents the error due to capping. To make good guarantees, configuration procedures will need to ensure that both of these terms are sufficiently small.
3 Configuration Procedures
We first describe two hypothetical procedures that give us lower bounds on the number of samples and the captime that any configuration procedure will need to use. The bound on the number of samples (Section 3.1) is a classic result. In Section 3.2, we use a novel "prover-skeptic" argument to show the lower bound on captime. We then describe in Section 3.3 a simple usable procedure that returns an  $\epsilon$ -optimal algorithm with probability at least  $1 - \delta$ , but which suffers from two major drawbacks. First, it requires that we specify an accuracy parameter  $\epsilon$  and a captime  $\kappa$  as input ahead of time. Making poor choices for these parameters can have a large impact on total configuration time (see Section 4 for an illustration). Indeed, many choices of  $\epsilon$ , and  $\kappa$  are mutually incompatible, giving rise to meaningless bounds. To avoid this, we hope to design procedures that are anytime: they continue to improve the accuracy guarantees they can make as they spend more time running. By taking gradually more and more samples, and by gradually taking them at higher and higher captimes, an anytime procedure continues to shrink the  $\epsilon$  that it can guarantee, rather than trying to target a fixed  $\epsilon$  it is given as input. Second, the naive procedure is not input-adaptive in any way. It does the same number of samples at the same captime for every algorithm. Both of these drawbacks are fixed by our UP procedure in Section 3.4 UP is an anytime procedure, meaning it requires neither an  $\epsilon$  nor a  $\kappa$  as input, but instead gradually reduces the  $\epsilon$  it can guarantee by increasing both the number of samples and the captime.
3.1 Hypothetical Runtime Oracle Procedure
The unique characteristic of our setting is the fact that we observe capped rather than true runtime samples, and that the cost we pay for each sample is equal to the time we spend collecting it. If this were not the case, then our problem would already be solved. If we had some oracle that we could simply query for the true runtime of an algorithm run on a given instance, then all we would need to do is collect sufficiently many samples from each algorithm. In this case, the optimal procedure simply takes an increasing number of samples of each algorithm and rules out each suboptimal one was soon as it can. The Runtime Oracle Procedure (Algorithm 1) is an instantiation of the Successive Elimination algorithm of Even-Dar et al. (2006) (their Algorithm 3). With probability at least  $1 - \delta$  it will eventually eliminate all algorithms except the optimal one.
Theorem 1. With probability at least  $1 - \delta$  the Runtime Oracle procedure will eventually return the optimal algorithm, and it will return an  $\epsilon$ -optimal algorithm if it is run until  $m$  is large enough that  $2\sqrt{\frac{\ln(4nm^2 / \delta)}{2m}} \leq \epsilon$ . At that point it will have taken  $m$  uncapped samples of each  $\epsilon$ -optimal algorithm,
Algorithm 1 Runtime Oracle Procedure
Inputs: Algorithms  $i = 1,\dots,n$  ; stream of instance  $j = 1,2,\ldots$  ; utility function  $u$  ; parameter  $\delta$  runtime oracle  $\mathcal{R}$ $I\gets \{1,\dots,n\}$  candidate algorithms   
for  $m = 1,2,3,\ldots$  do   
for  $i\in I$  do   
 $t_{im}\gets$  obtain i's runtime from  $\mathcal{R}$  on m-th instance   
 $\widehat{U}_{im}\leftarrow \frac{1}{m}\sum_{j = 1}^{m}u(t_{ij})$    
end for   
 $i^{*}\gets$  arg max  $_{i\in I}\widehat{U}_{im}$ $\alpha_{m}\gets \sqrt{\frac{\ln(4nm^{2} / \delta)}{2m}}$    
for  $i\neq i^{*}$  do if  $\widehat{U}_{im} <   \widehat{U}_{i^{*}m} - 2\alpha_{m}$  then  $\triangleright i$  is suboptimal remove i   
end if   
end for   
if  $|I| = 1$  or execution is interrupted then   
return  $i^{*}$    
end if
and  $m_{i} \leq m$  uncapped samples of each  $\epsilon$ -suboptimal algorithm  $i$ , where  $2\sqrt{\frac{\ln(4nm_i^2 / \delta)}{2m_i}} \leq \Delta_i$ , and where  $\Delta_i$  is the difference between the expected utility of an optimal algorithm and of algorithm  $i$ .
See Even-Dar et al. (2006), Theorem 8 and Remark 9 for proofs and details. The Runtime Oracle procedure has two desirable qualities that we would like to preserve in the configuration procedures we design. Firstly, we do not need to supply the parameter  $\epsilon$  ahead of time. Instead, the procedure maintains an internal  $\epsilon$  that it can guarantee, continually shrinking this toward 0. Secondly, the number of samples needed to eliminate  $\epsilon$ -suboptimal algorithms grows with the square of the suboptimality gap  $\Delta_{i} = \max_{i'} U_{i'} - U_{i}$  rather than the square of  $\epsilon$ . Theorem  $\boxed{1}$  gives an input-dependent guarantee for the Runtime Oracle procedure that we would like to preserve in the usable procedures we design. If some algorithm  $i$  is very suboptimal, with  $\Delta_{i} \gg \epsilon$ , then we will be able to eliminate it with many fewer samples than an algorithm that is almost  $\epsilon$ -optimal.
The Runtime Oracle procedure gives us a baseline against which to compare. Its sample complexity guarantee is essentially optimal in the following data-dependent sense.
Theorem 2 (Theorem 5 of Mannor and Tsitsiklis (2004)). There exists some set of input distributions for which every oracle procedure that returns an  $\epsilon$ -optimal algorithm with probability at least  $1 - \delta$  must take  $\Omega\left(\frac{n - |N(\epsilon)|}{\epsilon^2} \log \frac{1}{\delta} + \sum_{i \in N(\epsilon)} \frac{1}{\Delta_i^2} \log \frac{1}{\delta}\right)$  samples, where  $N(\epsilon)$  is the set of  $\epsilon$ -suboptimal algorithms.
3.2 Hypothetical Captime Verification Procedure
We now consider a lower bound on the captime we will be required to use, even in a world where we do not need to take samples. We imagine there are a prover and a skeptic. The prover has access to the runtime CDF of each algorithm  $i$ . The skeptic will get to see each runtime CDF only up to some  $\kappa_{i}$  that the prover will choose. The prover then recommends an algorithm  $i^{*}$  and the skeptic should be convinced that  $i^{*}$  is  $\epsilon$ -optimal, based only on the truncated CDFs. As the  $\kappa_{i}$ 's tend to infinity, the skeptic will see more and more of the true CDFs and therefore eventually be convinced of the prover's claim (assuming it is true). The goal is to convince the skeptic that  $i^{*}$  is  $\epsilon$ -optimal using  $\kappa_{i}$ 's that are as small as possible. The skeptic will be able to compute the following deterministic bounds
Algorithm 2 Captime Verification Procedure
Inputs: Algorithms  $i = 1,\dots,n$  stream of instances  $j = 1,2,\ldots$  ; utility function  $u$  ; parameter  $\epsilon$  runtime CDFs  $F_{1},\ldots ,F_{n}$    
for  $i = 1,\dots,n$  do U<  $\begin{array}{r}\int_0^\infty u(t)dF_i(t) \end{array}$  expected utility   
end for   
 $i^{opt}\gets \arg \max_iU_i$  optimal algorithm   
for  $i = 1,\dots,n$  do  $\Delta_{i}\leftarrow U_{i^{opt}} - U_{i}$ $\kappa_{i}\gets \inf \left\{\kappa :u(\kappa)(1 - F_{i}(\kappa))\leq \Delta_{i} + \frac{\epsilon}{2}\right\}$    
end for   
 $i^{*} = \arg \max_{i}LB_{i}$    
if  $LB_{i^*}\geq UB_i - \epsilon$  for  $i = 1,\dots,n$  then  $\triangleright$  skeptic's check return  $i^{*}$    
else return "failed"   
end if
on an algorithm's expected utility
$$
U B _ {i} = \int_ {0} ^ {\kappa_ {i}} u (t) d F _ {i} (t) + u (\kappa_ {i}) \left(1 - F _ {i} (\kappa_ {i})\right)
$$
$$
L B _ {i} = \int_ {0} ^ {\kappa_ {i}} u (t) d F _ {i} (t).
$$
The skeptic knows that  $LB_{i} \leq U_{i} \leq UB_{i}$ , but also that the runtime distributions could be such as to make  $U_{i}$  fall anywhere in this range. So whatever algorithm  $i^{*}$  the prover returns to the skeptic, the skeptic will be convinced that  $i^{*}$  is  $\epsilon$ -optimal if, and only if, they observe  $LB_{i^{*}} \geq UB_{i} - \epsilon$  for all  $i \neq i^{*}$ . The following lemma justifies this condition.
Lemma 5. If  $LB_{i^{*}} \geq UB_{i} - \epsilon$  for all  $i \neq i^{*}$ , then  $i^{*}$  is  $\epsilon$ -optimal. If there exists an  $i \neq i^{*}$  with  $LB_{i^{*}} < UB_{i} - \epsilon$ , then regardless of the values of the CDFs  $F_{i}$  up to the captimes  $\kappa_{i}$ , there are some input distributions for which  $i^{*}$  is not  $\epsilon$ -optimal.
The proof works by constructing counter-example CDFs, but takes the CDFs as given up  $\kappa_{i}$ , so does not rely on specifying any information available to the skeptic. No matter what the CDF of each  $i$  looks like up to  $\kappa_{i}$ , there will be some inputs with  $LB_{i^{*}} < UB_{i} - \epsilon$  for which  $i^{*}$  is not  $\epsilon$ -optimal.
Lemma 6. The skeptic will be convinced that both  $i^{opt}$  and  $i^* = \arg \max_i LB_i$  are  $\epsilon$ -optimal if the captimes  $\kappa_i$  are large enough that  $u(\kappa_i)\big(1 - F_i(\kappa_i)\big) \leq \Delta_i + \frac{\epsilon}{2}$  for all algorithms  $i$ .
Algorithm describes the prover-skeptic interaction. The next lemma shows that using these captimes is essentially optimal. It gives us a baseline minimum captime against which to compare.
Lemma 7. There are some inputs for which any verification procedure must use captimes  $\kappa_{i}$  large enough that  $u(\kappa_i)\big(1 - F_i(\kappa_i)\big)\leq \Delta_i + \epsilon$  for all algorithms  $i$ , or the skeptic will not be convinced that the returned algorithm is  $\epsilon$ -optimal.
The proof simply constructs a counterexample for which doing runs at a smaller captime will lead any configuration procedure to make a wrong conclusion.
3.3 Naive Procedure
We now turn to the setting we are actually interested in, where runs cost time and long ones must eventually be terminated. The simplest thing we could do is just pick a captime (or perhaps we somehow "know" the right captime) and do the necessary number of runs. With a fixed captime, the scenario is not much different from the runtime oracle scenario above. The main difference is that the "resolution" at which we can understand an algorithm's expected utility will be limited by the size of the captime we use to take our samples. We will never know what the runtime CDF looks like beyond the captime, no matter how many samples we take. Given any  $\kappa$ , the confidence bounds described in
Algorithm 3 Naive Procedure
Inputs: Algorithms  $i = 1, \dots, n$ ; stream of instances  $j = 1, 2, \dots$ ; utility function  $u$ ; parameters  $\epsilon, \delta$ ; captime  $\kappa$  satisfying  $u(\kappa) < \epsilon$ .
$$
m \leftarrow \left\lceil \frac {2 \ln (2 n / \delta)}{(\epsilon - u (\kappa)) ^ {2}} \right\rceil
$$
for  $i = 1,\dots,n$  do
$t_{i1}(\kappa),\dots,t_{im}(\kappa)\gets$  run  $i$  on  $m$  instances using captime  $\kappa$