SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\widehat{U}_{im}(\kappa)\gets \frac{1}{m}\sum_{j = 1}^{m}u\big(t_{ij}(\kappa)\big)$
$\triangleright$  empirical average utility
$$
i ^ {*} \leftarrow \arg \max _ {i} \widehat {U} _ {i m} (\kappa)
$$
return  $i^{*}$
Section 2 tell us we could simply choose the smallest  $m$  satisfying  $2\sqrt{\frac{\ln(2n / \delta)}{2m}} + u(\kappa) \leq \epsilon$ , do  $m$  runs of each algorithm at captime  $\kappa$ , then return the one with largest empirical average utility. We can think of  $2\sqrt{\frac{\ln(2n / \delta)}{2m}}$  as the error due to sampling, and  $u(\kappa)$  as the error due to capping. This is the Naive procedure.
Theorem 3. With probability at least  $1 - \delta$  the Naive procedure returns an  $\epsilon$ -optimal algorithm. It takes enough  $\kappa$ -capped runtime samples of each algorithm to ensure that  $2\sqrt{\frac{\ln(2n / \delta)}{2m}} + u(\kappa) \leq \epsilon$ .
The proof of this and of subsequent theorems are deferred to Appendix A. Each essentially follows from the fact that the upper and lower confidence bounds are specified to satisfy Hoeffding's inequality and the union bound. Comparing Theorem 3 to Theorem 1 we can see the reason for the increased number of samples required when we observe capped instead of uncapped runs: if  $u(\kappa)$  is relatively large, it will take a much larger  $m$  to shrink the term  $2\sqrt{\frac{\ln(2n / \delta)}{2m}}$  enough to satisfy the inequality in Theorem 3
The Naive procedure has some undesirable qualities. Choosing the right  $\kappa$  may not be easy and in Section 4 we will show what a difference this choice can make for the total runtime of the Naive procedure. We are also required to specify an  $\epsilon$  beforehand; it may hard to know the "right"  $\epsilon$ , and the best  $m$  and  $\kappa$  for one  $\epsilon$  may be quite different from the best for a slightly different  $\epsilon$ . What's more, this procedure ignores information about an algorithm's runtime distribution that could be learned by observing runs along the way; it essentially assumes that every algorithm always times out at the given captime. As a result, it takes more samples of  $\epsilon$ -suboptimal algorithms than is necessary. The UP procedure corrects these defects.
3.4 Utilitarian Procrastination
Our Utilitarian Procrastination (UP) procedure starts by doing runs at the smallest captime possible, trying to rule out whatever configurations we can, and only starts doing runs at a larger captime when necessary.
Theorem 4. With probability at least  $1 - \delta$  UP eventually returns the optimal algorithm and it returns an  $\epsilon$ -optimal algorithm if it is run until  $m$  is large enough that  $2\sqrt{\frac{\ln(11nm^2(\log\kappa_{i^{opt}} + 1)^2 / \delta)}{2m}} + u(\kappa_{i^{opt}})\big(1 - F_{i^{opt}}(\kappa_{i^{opt}})\big) \leq \epsilon$ . For any suboptimal  $i$ , if  $m, \kappa_i, \kappa_{i^{opt}}$  are ever large enough that  $2\sqrt{\frac{\ln(11nm^2(\log\kappa_i + 1)^2 / \delta)}{2m}} + 2\sqrt{\frac{\ln(11nm^2(\log\kappa_{i^{opt}} + 1)^2 / \delta)}{2m}} + u(\kappa_i)\big(1 - F_i(\kappa_i)\big) + u(\kappa_{i^{opt}})\big(1 - F_{i^{opt}}(\kappa_{i^{opt}})\big) \leq \Delta_i$  then  $i$  will be eliminated.
In Theorem 4 we can clearly see the analog of both Theorem 1 and Theorem 3. UP is input dependent in two senses. The condition for ruling out a suboptimal  $i$  depends on  $\Delta_i$  and also on the runtime CDFs  $F_i$  and  $F_{i^{opt}}$ . Just as in the pure sample complexity case of Theorem 1, if  $i$  is very suboptimal and  $\Delta_i$  is large, it will be easier to satisfy the condition in Theorem 4 and thus rule  $i$  out. Additionally, the condition will be easier to satisfy if the inputs are such that either  $i$  or  $i^{opt}$  is able to complete a lot of runs at their respective captimes, so that their CDFs are close to 1. Theorem 4 also implies the following theorem and corollary, which together constitute our main theoretical result.
Algorithm 4 Utilitarian Procrastination
Inputs: Algorithms  $i = 1,\dots,n$  stream of instances  $j = 1,2,\ldots$  ; utility function  $u$  ; parameter  $\delta$ $I\gets \{1,\dots,n\}$  candidate algorithms   
 $\kappa_{i}\gets 1$  for all  $i\in I$    
for  $m = 1,2,3,\ldots$  do   
for  $i\in I$  do   
 $t_{i1}(\kappa_i),\dots,t_{im}(\kappa_i)\gets$  run  $i$  on  $m$  instances using captime  $\kappa_{i}$ $\widehat{F}_{im}(\kappa_i)\gets \frac{|\{j:t_{ij}(\kappa_i) <   \kappa_i\}|}{m}$  fraction of runs that completed   
 $\widehat{U}_{im}(\kappa_i)\gets \frac{1}{m}\sum_{j = 1}^{m}u(t_{ij}(\kappa_i))$  empirical average utility   
 $\alpha_{im}\gets \sqrt{\frac{\ln(11nm^2(\log\kappa_i + 1)^2 / \delta)}{2m}}$    
UCBim  $\leftarrow \widehat{U}_{im}(\kappa_{i}) + (1 - u(\kappa_{i}))\alpha_{im}$    
LCBim  $\leftarrow \widehat{U}_{im}(\kappa_{i}) - \alpha_{im} - u(\kappa_{i})(1 - \widehat{F}_{im}(\kappa_{i}))$    
end for   
 $i^{*}\gets$  arg max  $_{i\in I}$  LCBim   
for  $i\in I$  do   
if UCBim  $<  LCB_{i^{*}m}$  then  $\triangleright i$  is suboptimal   
 $I\gets I\setminus \{i\}$  remove i end if   
if  $2\alpha_{im}\leq u(\kappa_i)(1 - \widehat{F}_{im}(\kappa_i))$  then captime doubling condition   
 $\kappa_{i}\gets 2\kappa_{i}$    
end if   
end for   
if  $|I| = 1$  or execution is interrupted then return  $i^{*}$    
end if
Theorem 5. For any  $m$ , let  $\epsilon = 3\sqrt{\frac{\ln(11nm^4 / \delta)}{2m}}$ . Then with probability at least  $1 - \delta$ , UP returns an  $\epsilon$ -optimal algorithm once it takes  $m$  samples. At that point, if  $\kappa_i^*$  is the largest captime algorithm i has been run with then  $\kappa_i^* \leq 2\inf \left\{\kappa : u(\kappa)(1 - F_i(\kappa)) < \frac{\epsilon}{3\sqrt{2}}\right\}$ .
The proof follows from Theorem 4 and from UP's specific choice of captime doubling condition. We note that there is room for improvement, but that this captime bound is comparable to the worst-case lower bound captime needed by any configuration procedure as presented in Section 3.2. The smallest captime that satisfies  $u(\kappa)\big(1 - F_i(\kappa)\big) < \frac{\epsilon}{3\sqrt{2}}$  needs to be larger than the smallest captime that satisfies  $u(\kappa)\big(1 - F_i(\kappa)\big) < \Delta_i + \epsilon$ , which is the best we can hope to do.
Corollary 1. With probability at least  $1 - \delta$ , UP returns an  $\epsilon$ -optimal algorithm after taking a number of samples that at most a logarithmic factor more than is optimally required by any procedure, and at a captime that is a constant larger than  $\inf \left\{\kappa : u(\kappa) \left(1 - F_i(\kappa)\right) < \frac{\epsilon}{3\sqrt{2}}\right\}$ .
Proof. The number of samples follows immediately from the definition of  $\epsilon$  in Theorem[5]. Because the captimes are always doubled, the total time spent by UP on any single instance is at most a constant times larger than the time spent running that instance the final time it was run. Since the captime at that point was at most  $\kappa_i^*$ , the total time spent running any instance is at most a constant times inf  $\{\kappa : u(\kappa)(1 - F_i(\kappa)) < \frac{\epsilon}{3\sqrt{2}}\}$ .
We now illustrate the runtime costs of utilitarian algorithm configuration and the impacts of the adaptive improvements offered by UP over Naive. We would have liked to go further, and in particular to offer comparisons against baselines other than Naive. However, we saw no straightforward way of doing so: there is simply no previous work on offering algorithm configuration in the utility-maximizing setting. Our paper focuses on algorithm configuration with theoretical guarantees. There
Figure 1: Runtime of the Naive procedure for different values of input captime  $\kappa$  on three datasets using a log-Laplace utility function. Choosing a bad captime can have a large effect on total configuration time, especially for smaller  $\epsilon$  values.
do exist many methods with guarantees in the runtime minimizing setting; we could have run existing procedures like Structured Procrastination, LeapsAndBounds, or CapsAndRuns and then evaluated them in terms of a utilitarian objective. However, such an apples-to-oranges comparison would likely have yielded very poor performance and regardless would have dispensed with the guarantees that motivate these methods. Second, we could have compared to heuristic methods like SMAC, ParamILS and GGA. Again, however, they optimize a different objective function. It is possible to imagine modifying one of these algorithms to optimize utility, but doing so would require fundamental algorithmic changes (e.g., to their so-called adaptive capping heuristics) and nontrivial software engineering effort. Even if we could have made such changes in a non-controversial way, comparing heuristic methods to methods offering guarantees would again be an apples-to-oranges comparison. Perhaps for these reasons, we note that no paper of which we are aware has yet compared any heuristic algorithm configuration method to any offering theoretical guarantees. We do intend to pursue such an investigation ourselves in future work, but anticipate that a careful study of this question will require an entire research paper.
Experimental Setup. We leverage three datasets from Weisz et al. (2020). The first is a set of runtimes for the minisat SAT solver on data generated by the CNFuzzdd instance generator. The others are sets of runtimes for the CPLEX integer program solver on the combinatorial auction winner determination instances (regions) and on woodpecker conservation problems (rcw); see Appendix D of Weisz et al. (2020) for details. We only used the first seed for the CPLEX datasets.
Choosing the Right Captime Matters. In Fig.1 we plot the total configuration time of the Naive procedure with different input captimes  $\kappa$  on three datasets from Weisz et al. (2020) using a log-Laplace utility function from Graham et al. (2023):  $u(t) = u_{LL}(t;60,1) = 1 - \frac{1}{2}\frac{t}{60}$  if  $t\leq 60$  and  $u(t) = \frac{1}{2}\frac{60}{t}$  otherwise. We used  $\epsilon$  values of 0.1, 0.15 and 0.2 and set  $\delta = 0.1$ . The first observation we make is that choosing a bad captime can have a large effect on total configuration time, especially for smaller  $\epsilon$  values. The second observation is that, for any fixed  $\kappa$ , different values of  $\epsilon$  can also have huge effects on total runtime of the configuration procedure. Both of these points help to emphasize the need for a procedure like UP that starts out with a small  $\kappa$ , only increasing it as needed, and refines the  $\epsilon$  it can guarantee "on the fly", based on the runs it has observed.
Anytime Speedups Matter. In Fig. 2 we plot the total configuration time of UP and Naive as a function of  $\epsilon$ . We used the same log-Laplace utility function as above and set  $\delta = 0.1$  (within reasonable ranges, the value of  $\delta$  has relatively little effect on total runtime). UP can drastically outperform Naive, especially for smaller values of  $\epsilon$ .
Unsurprisingly, being anytime in  $\kappa$  helps most when we do not know how to provide a good  $\kappa$  as input beforehand. If we somehow guess or know the right captime to use, then it can be hard to beat Naive. But if we use the wrong captime, UP can be much faster than Naive. In Fig. 3 we use a uniform utility function from Graham et al. (2023), with  $u(t) = u_{\text{unif}}(t;60) = 1 - \frac{t}{60}$  for  $t < 60$  and 0 otherwise. In the top row, we have used a captime of  $\kappa = 60$  for Naive, which is appropriate for this utility function since it is linear up to that point and then 0 thereafter. In the bottom row, we have set the captime poorly at  $\kappa = 600$ , meaning that we are not terminating some runs even though
$^{1}$ Code to reproduce all plots can be found at https://github.com/drgrhm/utilitarian-ac
Figure 2: Runtime of the different configuration procedures for different values of  $\epsilon$  on three different datasets using a log-Laplace utility function. UP easily outperforms Naive.
Figure 3: Runtime of the different configuration procedures for different values of  $\epsilon$  on three different datasets using a uniform utility function. The top row shows a scenario where the Naive procedure has been given an appropriate captime, the bottom row shows a scenario where it has not.
they are so long they give us 0 utility. We can see that if we get the captime right, then Naive will perform quite well. But if we get it wrong, Naive can drastically underperform compared to UP.
We have presented Utilitarian Procrastination (UP), a utility-based approach to automated algorithm configuration. This is the first procedure that we know of to incorporate utility functions into the algorithm configuration framework. UP is anytime, meaning it requires minimal input from the user and continues to refine its guarantees as it is run. In simple experiments, we show that by freeing the user from having to provide either an accuracy parameter  $\epsilon$  or a captime  $\kappa$  UP can help avoid the excessively long runtimes that come when these inputs are chosen poorly. Of course we would have liked to have shown that UP never uses a captime larger than  $\kappa_{i}^{*} \leq 2\inf \left\{\kappa : u(\kappa)(1 - F_{i}(\kappa)) < \Delta_{i} + \frac{\epsilon}{2}\right\}$ , matching the hypothetical procedure. The bound we do show is a result of the condition UP checks when deciding whether to double the the captime  $\kappa_{i}$ . UP uses a simple condition that balances the error due to sampling with the error due to capping. Other, more intelligent choices may be possible, but improving this will have to wait for future work.
Graham and Leyton-Brown were funded by an NSERC Discovery Grant, a DND/NSERC Discovery Grant Supplement, a CIFAR Canada AI Research Chair (Alberta Machine Intelligence Institute), a Compute Canada RAC Allocation, and DARPA award FA8750-19-2-0222, CFDA #12.910 (Air Force Research Laboratory). Roughgarden's research at Columbia University is supported in part by NSF awards CCF-2006737 and CNS-2212745.
Ansótegui, C., Sellmann, M., and Tierney, K. (2009). A gender-based genetic algorithm for the automatic configuration of algorithms. In International Conference on Principles and Practice of Constraint Programming, pages 142-157. Springer.
Balcan, M.-F., DeBlasio, D., Dick, T., Kingsford, C., Sandholm, T., and Vitercik, E. (2021). How much data is sufficient to learn high-performing algorithms? generalization guarantees for data-driven algorithm design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 919-932.
Balcan, M.-F., Nagarajan, V., Vitercik, E., and White, C. (2017). Learning-theoretic foundations of algorithm configuration for combinatorial partitioning problems. In Conference on Learning Theory, pages 213-274. PMLR.
Birattari, M., Stützle, T., Paquete, L., and Varrentrapp, K. (2002). A racing algorithm for configuring metaheuristics. In Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, page 11-18. Morgan Kaufmann Publishers Inc.
Brandt, J., Schede, E., Haddenhorst, B., Bengs, V., Hüllermeier, E., and Tierney, K. (2023). Ac-band: A combinatorial bandit-based approach to algorithm configuration. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):12355-12363.
Even-Dar, E., Mannor, S., Mansour, Y., and Mahadevan, S. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6).
Graham, D. R., Leyton-Brown, K., and Roughgarden, T. (2023). Formalizing preferences over runtime distributions. In International Conference on Machine Learning, pages 11659-11682. PMLR.
Gupta, R. and Roughgarden, T. (2017). A pac approach to application-specific algorithm selection. SIAM Journal on Computing, 46(3):992-1017.
Hoos, H. H. and Stutzle, T. (2004). Stochastic local search: Foundations and applications. Elsevier.
Hutter, F., H. Hoos, H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In International Conference on Learning and Intelligent Optimization, pages 507-523. Springer.
Hutter, F., Hoos, H. H., Leyton-Brown, K., and Stützle, T. (2009). Paramils: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36:267-306.
Kleinberg, R., Leyton-Brown, K., and Lucier, B. (2017). Efficiency through procrastination: Approximately optimal algorithm configuration with runtime guarantees. In *IJCAI*, volume 3, page 1.
Kleinberg, R., Leyton-Brown, K., Lucier, B., and Graham, D. (2019). Procrastinating with confidence: Near-optimal, anytime, adaptive algorithm configuration. arXiv preprint arXiv:1902.05454.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research, 18(1):6765-6816.
López-Ibanez, M., Dubois-Lacoste, J., Cáceres, L. P., Birattari, M., and Stützle, T. (2016). The irace package: Iterated racing for automatic algorithm configuration. *Operations Research Perspectives*, 3:43-58.
Mannor, S. and Tsitsiklis, J. N. (2004). The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623-648.
Tornede, A., Wever, M., Werner, S., Mohr, F., and Hüllermeier, E. (2020). Run2survive: a decision-theoretic approach to algorithm selection based on survival analysis. In *Asian Conference on Machine Learning*, pages 737-752. PMLR.
Von Neumann, J. and Morgenstern, O. (1944). Theory of games and economic behavior. Princeton university press.
Weisz, G., György, A., Lin, W.-I., Graham, D., Leyton-Brown, K., Szeptesvari, C., and Lucier, B. (2020). Impatientcapsandruns: Approximately optimal algorithm configuration from an infinite pool. Advances in Neural Information Processing Systems, 33:17478-17488.
Weisz, G., Gyorgy, A., and Szepesvári, C. (2018). Leapsandbounds: A method for approximately optimal algorithm configuration. In International Conference on Machine Learning, pages 5257-5265. PMLR.
Weisz, G., Gyorgy, A., and Szepesvári, C. (2019). Capsandruns: An improved method for approximately optimal algorithm configuration. In International Conference on Machine Learning, pages 6707-6715. PMLR.