Title: Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data-driven algorithm design

generalization guarantee

PAC-learnable bound

provable hyperparameter tuning
Abstract: Modern machine learning algorithms, especially deep learning-based techniques, typically involve careful hyperparameter tuning to achieve the best performance. Despite the surge of intense interest in practical techniques like Bayesian optimization and random search-based approaches to automating this laborious and compute-intensive task, the fundamental learning-theoretic complexity of tuning hyperparameters for deep neural networks is poorly understood. Inspired by this glaring gap, we initiate the formal study of hyperparameter tuning complexity in deep learning through a recently introduced data-driven setting. We assume that we have a series of learning tasks, and we have to tune hyperparameters to do well on average over the distribution of tasks. A major difficulty is that the utility function as a function of the hyperparameter is very volatile, and furthermore, it is given implicitly by an optimization problem over the model parameters. To tackle this challenge, we introduce a new technique to characterize the discontinuities and oscillations of the utility function on any fixed problem instance as we vary the hyperparameter; our analysis relies on subtle concepts, including tools from algebraic geometry, differential geometry, and constrained optimization. We use this to show that the learning-theoretic complexity of the corresponding family of utility functions is bounded. We instantiate our results and provide sample complexity bounds for concrete applications—tuning a hyperparameter that interpolates neural activation functions and setting the kernel parameter in graph neural networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to address a special case for hyperparameter setting wherein one wishes to tune hyperparameters for training in a manner suitable for retaining the same set of hyperparameters throughout a variety of learning tasks/objectives applied to a comparable data corpus. Their method involves tuning (one hyperparameter at a time?) by way of aggregating the tuning into a dual utility function in which they show they can bound the number of discontinuities in a manner that bounds oscillations and a resulting pseudo-randomness of such dual objective function. This helps them to reduce sample complexity. They leave extension of the methods towards a setting with multiple hyperparameters to future work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nQuestions surrounding how we could treat functions containing discontinuities as polynomial class are very interesting and of potential relevance to all kind of domains.\n\nWeaknesses:\nIn the context of an AI research conference the idea that a hyperparameter training objective targeting a single variable at a time is possibly kind of laughable, but then I have seen interviews with Turing award winners where they describe spending the majority of their research career trying to iterate on how they train models on MNIST data set, so perhaps it is not that far fetched.\n\nQUESTIONS:\nI found the claim in Related Work section represents this as \"the first work to focus on tuning hyperparameters for deep networks using a data driven lens\" kind of suspect. Can you please clarify what is meant by \"a data driven lens\" in this context and how it significantly differs from prior training conventions?\n\nLIMITATIONS:\nThe limitations to practical applications leveraging multiple hyperparameters are fairly apparent. I am not sure if the claimed contributions, even taken at face value, would be sufficient to compensate for that.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the challenge of hyperparameter tuning in deep learning by analyzing its sample complexity in a multi-task setting, where tasks are drawn from an unknown distribution. The authors develop a theoretical framework to study the learning complexity of tuning hyperparameters, focusing on utility functions that depend on both hyperparameters and model parameters. By analyzing the structure of the dual utility function—which measures performance under optimal model parameters—they derive bounds on its discontinuities and oscillations using tools from differential geometry, algebraic geometry, and constrained optimization. \n\nTheir key results include sample complexity guarantees for tuning hyperparameters when the underlying parameter-dependent dual function is piecewise constant or piecewise polynomial. They apply these insights to practical settings, such as interpolating between activation functions in neural networks and tuning graph kernel parameters in graph neural networks. Additionally, they relax the assumption of exact optimization, showing their guarantees hold even with approximate empirical risk minimization. The work provides a rigorous foundation for understanding hyperparameter selection in multi-task learning scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Theoretical results are technically solid and fine-grained, proofs are well-structured. Although I didn't check every detail of the proofs, the results seem to present no apparent error.\n2. Their results captures good characteristics of the parameter-dependent utility function and its dual functions, and then establishes concrete bounds for learnability of the utility function class. New techniques are developed to support these results. The formalization of this general framework in analysis of hyperparameter tuning is a significant contribution.\n3. This paper is well-written and easy to under stand, presenting complex theoretical concepts in a clear and accessible manner.\n\nWeaknesses:\n\n1. The theoretical result is developed for one-dimensional hyperparameter, whilst real-world tuning often involves many hyperparameters of very different nature (such as learning rates, dropout rates, batch sizes...). Extending the analysis to hyperparameter spaces with larger dimensions remains an open challenge.\n2. I am concerning the applicability of the result in practice. Since the theory is build over the draw of problem instances $S=\\left\\{x_1, \\ldots, x_m\\right\\} \\sim \\mathcal{D}^m$, as most real-world problems occurs to one dataset, without access to multiple independent problem instances for hyperparameter tuning.\n\nQUESTIONS:\n1. The piecewise-polynomial assumption for the dual function $f_x(\\alpha, w)$ is central to the theory. Is there any possible empirical studies (even on simple synthetic tasks) that the theoretical results holds for some toy models?\n\n2. (related to weaknesses 2) Could the analysis be extended to multiple hyperparameters? What are the major challenges in such multi-dimensional scenarios.\n\n3. The theoretical guatantee is build on the exact minimizer of the optimization problem (u^*_{x}(\\alpha) = \\sup_{w}f_{x}(w,\\alpha)). However, in practice, optimized with solvers that converge to approximate solutions. I am wondering whether the structural properties (e.g., discontinuities, oscillations of $u_x^*$) are preserved under approximation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel theoretical framework to analyze the sample complexity of tuning a single, continuous hyperparameter for deep neural networks in a data-driven, multi-task setting. The central challenge addressed is that the network's performance for a given hyperparameter, $\\alpha$, is defined implicitly through an optimization over the high-dimensional weight parameter, $w$. This makes the utility function, $u_x^*(\\alpha) = \\sup_w f_x(\\alpha, w)$, difficult to analyze directly.\n\nThe authors' main contribution is a new technique that leverages tools from differential and algebraic geometry to bound the learning-theoretic complexity of the utility function class. They show that even if $u_x^*(\\alpha)$ is not a well-behaved function (e.g., not piecewise polynomial), its structural properties can be bounded. Specifically, by bounding the number of discontinuities and local extrema of $u_x^*(\\alpha)$, they can bound its pseudo-dimension. This is achieved by analyzing the geometry of the underlying parameter-dependent dual function, $f_x(\\alpha, w)$. The analysis is presented for two cases: when $f_x(\\alpha, w)$ is piecewise constant and, more generally, when it is piecewise polynomial. These theoretical results are then instantiated to provide sample complexity guarantees for tuning an activation function interpolation parameter and a graph kernel parameter in GNNs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. Addresses the critical and under-theorized problem of hyperparameter tuning in deep learning.\n2. The technical approach is highly novel, applying advanced geometric tools in a creative way to a problem in learning theory. \n3. The work appears technically rigorous and provides a foundational result for a difficult problem.\n\nWeakness:\n\n1. The analysis is restricted to a single hyperparameter, and the path to extending it to multiple hyperparameters is not obvious. \n2. The theory relies on regularity assumptions that may be difficult to verify for large, practical deep learning models. \n3. The paper is very technical and dense, which may limit the reach of its core ideas to a broad audience.\n\nQUESTIONS:\n1.  he restriction to a single hyperparameter ($\\alpha \\in \\mathbb{R}$) is the main limitation. The analysis relies on properties of 1D curves and partitioning the real line. Could you elaborate on the fundamental obstacles to extending this to $\\alpha \\in \\mathbb{R}^n$? What would be the high-dimensional analog of \"bounded oscillations\" or the \"monotonic curves\" that are central to your proof? A more detailed discussion on a potential (even if speculative) path forward would greatly strengthen the paper's perceived long-term impact.\n\n2.  The sample complexity bounds (e.g., in Theorem 6.1) have dependencies on network size ($W, L$) and polynomial degrees. Could you comment on whether you believe these bounds are tight or if they are primarily intended to show polynomial learnability? Furthermore, regarding Assumption 1 (ELICQ/ND), could you provide some intuition on why one might expect these to hold (or fail) for a standard architecture like a ResNet with common loss functions? For instance, do symmetries in the network architecture pose a problem for the LICQ condition?\n\n3.  Theorem 5.1 accounts for an approximate-ERM oracle via an additive error term $\\xi$. In practice, the optimization error from GD-based methods might depend on the hyperparameter, i.e., $\\xi = \\xi(\\alpha)$, as $\\alpha$ can affect the difficulty of the optimization landscape. Does your analysis assume $\\xi$ is a constant independent of $\\alpha$? If so, how would the results change if $\\xi$ is a function of $\\alpha$, and what assumptions might be needed on the function $\\xi(\\alpha)$ to retain the guarantees?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Hyperparameter tuning across tasks is still dominated by heuristic search – theory lags behind practice.\n- The paper develops a learning-theoretic framework that links the number of discontinuities and local extrema of the task-wise dual utility to the pseudo-dimension of the whole class.  \n- The paper analyzes two structural cases: (i) piecewise-constant and (ii) piecewise-polynomial parameter-dependent dual functions $f_x(\\omega,w)$, yielding explicit sample-complexity bounds.  \n- Applications illustrate how the theory certifies learnability when tuning an activation-function interpolation parameter and a GNN kernel bandwidth, although only single-hyperparameter settings are treated.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- *Originality* - Introduces a novel bridge between algebraic/differential-geometric regularity of deep nets and classical capacity measures.\n- *Quality* - Proofs are mathematically detailed and cover both piecewise-constant and considerably harder piecewise-polynomial regimes.  \n- *Significance* - Provides the first non-trivial sample-complexity guarantees for continuous hyperparameter tuning in deep networks.\n\n**Weaknesses**\n\n- *Clarity* - Core definitions (e.g. monotonic curves, ELICQ/ND conditions) are deferred to long appendices; the main text gives few intuitive examples, making it hard to grasp practical meaning.  \n- *Significance* - Results hold for a single scalar hyperparameter and require piecewise-polynomial structure – assumptions whose prevalence in modern architectures is unverified.  \n- *Quality* - No experiments demonstrate whether the theoretical bounds are tight or how they translate to wall-clock savings vs. Bayesian optimization; even toy simulations would strengthen claims.\n\nQUESTIONS:\n1. Can the authors supply concrete evidence that common activations or graph kernels yield piecewise-polynomial $f_x$ in practice?  \n2. How do the bounds scale when extending to two continuous hyperparameters?\n3. Compared to Bayesian optimization, do your bounds imply a smaller or larger sample requirement under comparable smoothness assumptions? A discussion would clarify practical relevance.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-19"}