Title: Generalization Bounds for Model-based Algorithm Configuration
Venue: NeurIPS 2025
Accepted: yes
Keywords: generalization bound

data-driven algorithm design

theory of algorithm configuration

algorithmic stability
Abstract: Algorithm configuration, which involves selecting algorithm parameters based on sampled problem instances, is a crucial step in applying modern algorithms such as SAT solvers. Although prior work has attempted to understand the theoretical foundations of algorithm configuration, we still lack a comprehensive understanding of why practical algorithm configurators exhibit strong generalization performances in real-world scenarios. In this paper, through the lens of machine learning theory, we provide an algorithm-dependent generalization bound for the widely used model-based algorithm configurators under mild assumptions. Our approach is based on the algorithmic stability framework for generalization bounds. To the best of our knowledge, this is the first generalization bound that applies to a model closely approximating practical model-based algorithm configurators.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the first generalization bound for model-based algorithm configurators (AC), a widely used class of methods for automatic parameter tuning of complex algorithms (e.g., SAT or MIP solvers). The authors focus on the practical setting where the AC process is guided by surrogate models (here: random forests) and search heuristics based on acquisition functions like expected improvement. Using the framework of algorithmic stability, the paper derives a generalization bound that accounts for continuous and multi-dimensional parameter spaces, and that is dependent on properties of the configurator itself (rather than the parameterized algorithm). The bound incorporates terms reflecting the surrogate model complexity and the sampling procedure’s convergence, offering insights into how algorithm configurations found on a finite set of instances can generalize well to unseen problem instances.\n\nSTRENGTHS AND WEAKNESSES:\nThe major strength of the paper is that it addresses a longstanding gap between empirical success and theoretical understanding of algorithm configuration, offering the first generalization guarantees aligned with methods that are used in practice, such as SMAC. The use of algorithm-dependent stability for generalization in algorithm configuration, combined with an analysis of random-forest-based surrogate models, which are typically used for their speed, robustness in generalization performance, and native applicability to mixed input spaces (i.e., parameter spaces comprising categorical, Boolean, integer, and float parameters). The theoretical framework is sound and uses established learning theory tools, e.g., Rademacher complexity, Pinsker’s inequality, and MCMC convergence. The problem setup and assumptions are precisely formulated, with clear definitions of the algorithm configuration process, surrogate model, and generalization objective.\n\nHowever, the paper also comes with a couple of weaknesses. First of all, the use of random forests as surrogate models introduces significant limitations not accounted for in the theory. Most notably, RFs cannot extrapolate, which constrains the EI acquisition function and may limit the configurator’s ability to explore promising but unseen regions of the parameter space. While mean predictions might be relatively stable and are bound in the paper to not deviate too much, the ordering of configurations can, in fact, also be affected by smaller changes in the predictions, e.g., when values are relatively close to each other. In practice, even small prediction changes can flip orderings and substantially alter the sampled configurations, which is not reflected in the analysis. \nFurthermore, the mean might be too prone to be distorted by “outliers”, as in algorithm configuration, we often deal with performance landscapes that exhibit heavy-tailed distributions. I am wondering to what extent this impedes the stability of the learning algorithms.\nAnother issue is with the acquisition function maximization and whether it is sufficiently stable via MCMC. In practice, EI is known to be sensitive, especially with non-smooth or multimodal landscapes, which are indeed induced by random forests. This raises concerns about the robustness of the practical applicability of the bound. Lastly, while the theoretical contributions are substantial, empirical analysis (as described in the appendix) is limited in demonstrating whether the bound meaningfully correlates with observed generalization for diverse algorithm configuration scenarios. This includes considerations about the homogeneity of the problem instances, i.e., whether assumptions on the best configuration being consistent for all problem instances (or even entire rankings of configurations in the case of strong homogeneity) need to be fulfilled to allow for generalization bounds. Note that there exists the classical algorithm configuration and the per-instance algorithm configuration setup, which differ in their assumptions about the homogeneity of the problem instances. While in classical algorithm configuration, one would assume homogeneity for all problem instances, in the case of per-instance algorithm configuration, the assumption is only about homogeneous subsets of problem instances. In other words, for different such subsets, different algorithm configurations work best. It is quite surprising to me that the only assumption on the problem instances in the paper is that the problem instances are i.i.d.\n\nQUESTIONS:\n* How confident are you that the MCMC abstraction sufficiently captures the true search dynamics for EI maximization, as EI maximization can be relatively tough in practice?\n* Since RFs can only poorly predict values beyond observed samples (i.e. extrapolation), the configurator may miss regions with better configurations. Could this limitation be reflected in the theoretical bound (e.g., via a bias term)?\n* The stability analysis focuses on predicted values, but practical configuration decisions depend on the relative ranking of configurations. Could not changes in configuration orderings, even under small perturbations, cause significant differences in behavior? \n* What happens when problem instances are non-i.i.d.? The generalization bound assumes i.i.d. sampling from a distribution D, but in real applications (e.g., solver competitions), instances may be clustered or adversarial. Also, algorithm configuration might be performed on a system different from where the found configuration is ultimately deployed. How sensitive is the theory to this assumption?\n* Can we deduct requirements for an intensification scheme from your theory to reduce the number of trials conducted during algorithm configuration?\n\nLIMITATIONS:\nThe authors discuss their assumptions and mention some practical caveats (e.g., surrogate model choice, algorithmic assumptions), but they do not fully address limitations such as the inability of RFs to extrapolate or the fragility of configuration rankings under slight perturbations. Including a deeper discussion of these issues would strengthen the paper and increase its transparency.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the theoretical guarantee on the generalization performance in algorithm configuration. Specifically, this paper utilizes the theoretical connection between the stability and the generalization performance of algorithms. It considers an algorithm configurator based on random forests surrogate model. Then it derives a generalization guarantee of such model-based algorithm configurator by proving its stability. The key technical tool in establish the theoretical results is algorithmic stability (which should be credited to Bousquet & Elisseeff 2002, not Ben-David’s textbook).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n* The application of algorithm stability in this context seems interesting.\n\nWeakness: \n\nWeakness: \n\n* The paper is not clearly written. There are missing details and errors, which make the paper confusing. Ignoring the minor things, the main confusion is the blurred boundary between two notions: the **true performance** of some $(\\theta, I)$ (which is measured by function $u(\\cdot)$ and the **surrogate performance** of $(\\theta, I)$ (namely, the performance measured under the surrogate model).  Another confusion is the distinction between two different sources of randomness: the first is the randomness in the true performance (or in the evaluated algorithm): the function $u(\\theta, I, \\xi) $ depends on a random seed $\\xi$, and the second is the randness in the surrogate random forest model. These two sources are different but the paper might have used the second to replace the first without justification.  -- I am not too sure about this though.\n\n* Possibly due its lack of clarity,  I think that the main result, Theorem 5, is irrelevant to the proposed algorithm.  Specifically, Theorem 5 upper bounds the gap between the expected **true performance** of  configuration $\\tilde{\\theta}$ and empirical **true performance** of $\\tilde{\\theta}$.  However, there can be a gap between the empirical **true performance** of $\\tilde{\\theta}$ and the empirical **surrogate performance** of $\\tilde{\\theta}$. The proposed algorithm, under the guidance of the surrogate model, finds some $\\tilde{\\theta}$ whose **surrogate performance**  is good, but the theorem has nothing to do with the surrogate performance. In other words, the theorem does not justify that $\\tilde{\\theta}$ found by the proposed algorithm has good **true performance**.\n\nTo be more concrete, let me explain my understanding of the proposed algorithm.The algorithm runs in two phases. \n\nIn Phase 1, it first sample a number of $\\theta$'s, and for each $\\theta$, sample some tasks $I$, and for each pair $(\\theta, I)$, evaluate its performance $u(\\theta, I)$, which is inherently random (due toi the first source of randomness). It then first a best $\\theta$, which I denote by $\\theta_0$  in this list of $\\theta$'s that has the best average performance over the sampled tasks. \n\nIn Phase 2, a surrogate random forest model is trained, which tries to predict the expected performance of $u(\\theta, I)$. I will denote the learned predictor by $\\hat{u} (\\theta, I)$.  Note that $\\hat{u}(\\theta, I)$ is also random due to random forest predictor, i.e., this is the second source of randomness. Guided by $\\hat{u}$, $\\theta_0$ is iteratively updated to $\\tilde{\\theta}$, where $\\tilde{\\theta}$ improves upon $\\theta_0$ in some sense,  under **the learned surrogate performance metric** $\\hat{u}$.   \n\nTheorem 5 is about true performance $u$, bounding the gap between the expected $u$ and the emprical average of $u$ for $\\tilde{\\theta}$. But there is still a gap between the emprical average of $u$ and the emprical average of $\\hat{u}$ for $\\tilde{\\theta}$. In other words, although $\\tilde{\\theta}$ may be good measured under $\\hat{u}$, it may not be good under $u$, namely, the surrogate model may not generalize over changing $\\theta$.\n\nMy above complaints may result from some misunderstanding of the paper, due to its lack of clarify. If this is the case, I hope the authors in your rebuttal please clearly summarize the algorithm flow and make clear distinctions of these notions with clear notations.\n\nQUESTIONS:\n1. The improvement equation after line 207: the LHS depends on $\\hat{u}$, but in the RHS, $\\hat{u}$ is expected over. \n2. What is $\\sigma_{\\rm min}$ in Theorem 5? I did not see it defined in the paper.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies stability-based generalization guarantees for model-based algorithm configuration. The basic setup is that we have a parameterized family of algorithms (parameterized by $\\theta \\in [0,1]^n$), a distribution $D$ over input instances to those algorithms, and a utility function $u$ that evaluates the performance (e.g., running time) of a parameter vector on a single input instance. The goal is to leverage a collection of sampled problem instances $I_1, \\ldots, I_m$ drawn from $D$ in order to find an algorithm configuration that has high expected utility for new samples drawn from $D$. Prior work has studied this problem through the lens of uniform convergence, showing that when the algorithm family is not too complicated, a relatively small sample of instances is sufficient to guarantee uniform convergence of every parameter's empirical and expected utility, ensuring that the empirically optimal configuration is also approximately optimal for the underlying distribution. This work instead applies the machinery of stability based convergence guarantees to algorithm configuration to argue that due to the stability of the configuration procedure itself, its output configuration has empirical and expected utilities that are close in expectation.\n\nThe authors study a model-based configuration procedure that at each iteration approximates the mapping from parameters to expected utility with a random forest and uses a Metropolis-Hastings local search sampling method to update the current parameters. The stability argument has a few key components. Fix a pair of datasets that differ on a single instance. First, the authors study the Rademacher complexity of the dual of decision trees, arguing that uniformly for all parameters the surrogate models trained on each dataset make similar utility predictions. They use this result to argue that a single round of the local search has similar output distributions on the two datasets leveraging the fact that the limiting distribution of the Metropolis-Hastings distribution are similar (due to the utility estimates being similar) and a convergence analysis of the sampling. Finally, they argue that the final iterate distribution is also close by decomposing the difference into a product of the differences of each step, which were just shown to be small.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. This paper fills a gap by applying stability-based convergence guarantees to the problem of algorithm configuration.\n1. The generalization guarantee holds for arbitrary algorithm families by relying only on properties of the configuration procedure, which I think covers realistic cases that are not addressed by prior work.\n1. The main ideas of the analysis are relatively easy to follow.\n\n# Weaknesses\n1. The authors correctly point out an important limitation of prior work on data-driven algorithm design: the analysis tends to hold for a single algorithm family which usually only has a small number of parameters (often one). However, it is worth mentioning that these works typically come with a much stronger guarantee than provided in this submission. As the authors write, the prior work on problem-specific results provide uniform convergence guarantees (and often also efficient empirical risk minimizers), which ensure that those configuration procedures output essentially optimal configurations for the underlying distribution. The guarantee from this submission is much weaker: it provides a bound on the difference between the expected and empirical utilities of the single configuration the method outputs. Essentially, this only rules out surprises where the output configuration performs much worse on new instances than it did on the training instances. But, it cannot rule out the existence of significantly better configurations. In particular, even if the configurator happens to output an empirically optimal configuration, there is no guarantee that it is close to optimal on the underlying distribution (whether this is true or not depends primarily on the algorithm class being optimized over, not the configurator). It is useful to know that the parameters output from realistic configurators will generalize their utility from the sample to the distribution, but this is a significant reduction in scope from the prior work that should be clearly acknowledged.\n1. The authors make a number of simplifications compared to modern configurators (greedy local search is replaced by Metropolis-Hastings sampling, there is no intensification, the configurator outputs final iterate instead of the best observed). The changes do seem relatively minor, but the appendix only compares them empirically for one algorithm family and two problem domains. Given that the analysis depends heavily on these simplifications, it would be good to verify that they do not come at too great a cost to utility.\n1. I would have appreciated more detail about the specific configurator studied in the main body. A few examples:\n    - a. Lines 186-189: Is the surrogate model trained to predict the expected utility of a parameter vector $\\theta$ over the draw of a random instance from the data distribution in addition to $\\xi$? This seems like the intent but it's not explicitly written.\n    - b. Lines 198: Even if it is standard, it might be helpful to state how the decision trees are trained on each bag. (From the proof of Lemma 21, it seems that all that is required is that each tree has at most one leaf per data point and those leaves correspond to hyperrectangles, but it would still be good to clarify).\n    - c. In line 224, my understanding of this statement is that modern algorithm configurators retry the same parameters multiple times to improve the accuracy of the estimates, but the proposed method does not. Is the proposed configurator likely to try each configuration exactly once? Does that impact the configurator's ability to rule out high-variance configurations?\n1. I think the paper would be strengthened by trying to move more of the core analysis into the main body (for example, the VC bounds on the dual class of decision trees with few leaves seem like they could be explained in the main body).\n\nQUESTIONS:\n1. Please see questions listed in the strengths and weaknesses.\n1. Given that Theorem 5 scales with the number of algorithm parameters, I wonder how the result compares to the following simpler argument: suppose that each of the $n$ parameters is represented as a $b$-bit number so that there are $2^{bn}$ possible configurations. We can establish uniform convergence over those configurations by applying Hoeffding's inequality and the union bound to argue that with probability at least $1-\\delta$, every configuration's empirical utility is within $O\\left(\\sqrt{\\frac{bn + \\ln(1/\\delta)}{m}}\\right)$ of its expected utility. Now applying an arbitrary algorithm configurator to selecting values for the parameters automatically enjoys convergence guarantees, and, moreover, if the configuration is nearly optimal empirically then they are nearly optimal on the underlying distribution (among the $b$-bit values for each parameter). This bound has a worse dependence than the proposed method on the number of parameters ($\\sqrt{n}$ instead of $n^{1/4}$), but it makes no assumption at all on the configurator, does not degrade with the number of iterations of the configurator, provides uniform convergence, and yet still applies to a wide class of multi-parameter algorithm families. Restricting to $b$-bit numerical parameters might miss out on extremely precise algorithm configurations, but it seems like model-based configurators are likely to miss these configurations as well because the surrogate model will be unlikely to capture high utility parameters that are like needles in a haystack. Aside from the better dependence on the number of parameters, are there other advantages of the proposed method?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany general purpose algorithms such as algorithms for SAT solving, linear programming, etc, have several configurable parameters. Algorithm Configuration is the process of tuning such parameters to fit a given application. This process, when performed manually may be very time consuming, and therefore several approaches for automated algorithm configuration have been proposed along the years. One popular approach is model-based algorithm configuration, where random forests are used as surrogate models. In this work, the authors provide a generalization bound for this particular algorithm configuration method. More specifically, they show essentially that the error of this method converges with a rate of m^{-1/2}, where m is the number of samples, plus an additive factor of ((n+d)/Q)^{1/4} where n is the number of parameters of the model, d is the number of instance features and Q the number of random forests. In summary, the higher the number of samples and the higher the number of random forests, the smallest the error.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is (in most parts) well written and the results are interesting. In particular, this seems to be the first generalization bound for the model based AC with random forests that works both with continuous and with discrete parameters. According to the authors, previous generalization bounds only work with discrete parameters. \t\n\nWeaknesses:\n\nThere are some minor issues with the presentation that if fixed would make the paper easier to read. \n\nThe expression \"generalization bound\" is being used since the abstract, but no intuitive discussion of what this means is given until about the statement of theorem 1. It would be easier for the reader if you can state right away by what you mean with \"generalization bound\", preferentially in the abstract. \n\nIt seems that in line 96, the expression should be  ((n+d)/Q)^{1/4}  and not {-1/4}. \n\nFigure 1 needs improvement. The color choice is not ideal. Using black text with lightly colored background would make the figure much more readable. The text is also overlapping with boxes and arrows (eg. Black-box optimization). Finally the figure is not informative, in the sense that it does not really help understanding the content of the paper. So, my suggestion is simply to remove it and use the space to improve on the clarity of the text.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSequential model-based optimization (SMBO) techniques employ a surrogate model (e.g., a random forest) to fit observed performance data and then iteratively propose promising parameter vectors; these methods have demonstrably excelled in practice. However, no theoretical generalization guarantees exist to explain why configurations optimized on finite training instances maintain effectiveness on unseen instances from the same distribution. This paper establishes the first generalization guarantees for model-based algorithm configuration (AC) with continuous parameters. Addressing a critical gap in theory-practice alignment, it analyzes widely used configurators that employ random forest surrogates and Bayesian optimization.\n\nThe core contributions of this paper are as follows:\n1) This paper proposes the first problem-independent generalization bound for AC with multiple continuous parameters, overcoming limitations of prior discrete or algorithm-specific analyses. It leverages algorithmic stability to link random forest surrogate properties to generalization error.\n2) By directly analyzing industry-standard configurators with minimal performance-preserving adaptations, this work derives a non-vacuous bound where dominant terms become negligible under practical settings.\n3) This paper demonstrates the tightness of key theoretical terms through data-dependent bounds.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper is the first to deliver a model-based, algorithm-dependent generalization bound that explicitly matches how SMBO systems such as SMAC operate—i.e., random-forest surrogates plus local search for EI maximisation. Previous bounds were either problem-specific or relied on uniform convergence. Here the authors close that gap and provide a guarantee that aligns with the configurators practitioners actually deploy. \n2) The bound aligns with real-world algorithm configurators and provides a robust theoretical foundation for algorithm tuning in practice.\n3) This paper demonstrates the negligibility of dominant theoretical terms (e.g., Rademacher complexity) via data-dependent bounds and verifies minimal performance impact from analytical adaptations.\n\nWeaknesses:\nThe model-based approach may struggle with large parameter spaces or high-dimensional problems, leading to increased computational complexity.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}