Title: Learning Theory for Kernel Bilevel Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: bilevel optimization

kernel methods

learning theory

empirical process theory

U-statistic

generalization error bounds
Abstract: Bilevel optimization has emerged as a technique for addressing a wide range of machine learning problems that involve an outer objective implicitly determined by the minimizer of an inner problem. While prior works have primarily focused on the parametric setting, a learning-theoretic foundation for bilevel optimization in the nonparametric case remains relatively unexplored. In this paper, we take a first step toward bridging this gap by studying Kernel Bilevel Optimization (KBO), where the inner objective is optimized over a reproducing kernel Hilbert space. This setting enables rich function approximation while providing a foundation for rigorous theoretical analysis. In this context, we derive novel finite-sample generalization bounds for KBO, leveraging tools from empirical process theory. These bounds further allow us to assess the statistical accuracy of gradient-based methods applied to the empirical discretization of KBO. We numerically illustrate our theoretical findings on a synthetic instrumental variable regression task.


Peer reviews:
comments: SUMMARY:
The paper considers bilevel optimization in the rkhs. The algorithm  involves using grading method like in the parametric case.

STRENGTHS AND WEAKNESSES:
The introduction is well written, showing the importance and challenge of the problem.  However, the analysis is not quite different from the parametric case. One just need to carefully treat the concept of gradient in function space. The empirical process technique used is also just the same as the parametric case.  The result showing that both the function value and its gradient has some uniform convergence property, which is expected. Convergence of the gradient algorithm is also standard. Thus I'm not too enthusiastic about the work. The problem itself is important, and this is probably a reasonable start.

QUESTIONS:
1. what is the essential challeng of doing kernel version?
2. is cobvergence rate optimal?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the formalism of kernel bilevel optimization, a way to frame a specific instance of a recently proposed functional bilevel optimization setting in terms of reproducing kernel Hilbert spaces (RKHSs) - thereby making it nonparametric. The inner optimization problem is given by a classical ridge regression / Tikhonov regularized objective with a generic loss function, which is often found in the kernel literature and allows for the application of a representer theorem.\n\nA gradient-based plugin estimator is investigated. The theoretical core contributions are worst-case inequalities for the distances between the empirical and analytical versions of the objective as well as its gradient in expectation. The regularity assumptions are typical and relatively mild. The bounds exhibit an additive dependence on a split sample with 1/sqrt in both sample sizes each. They also exhibit a generic multiplicative constant which is briefly explained in the main text and in a more detailed form throughout the appendix. For example, a typical diameter-type dependence on the compact parameter domain (outer objective) as well as a regularization parameter dependence (inner objective) are the two most obvious properties reflected in the constant.\n\nThe authors propose a gradient scheme and give generalization bounds in expectation, showing a similar sample dependence as before as well as an additional 1/sqrt dependence on the iteration number under additional positive definiteness assumptions. The structure and assumptions of this bound seem to be classical within the context of first-order optimization theory. Finally, a very basic synthetic experiment for the specific instance of kernel instrumental variable regression framework as proposed by Sing et al. (2019) is performed, supposedly confirming the theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\nI see the main contribution of the paper in the abstract framing of nonparametric bilevel optimization with generic loss functions and the first steps towards analyzing KBO theoretically, which this paper manages to do well. \n\nThe derived error bounds are of relatively simple type (analytical regularized loss vs. empirical regularized loss, no ground truth assumptions and corresponding convergence theory for $\\lambda \\to 0$, only expectation bounds, no high probability bounds under noise assumptions, no fast rates, no lower bounds), but sufficiently outline core properties of the proposed estimator under relatively weak assumptions as a first step towards a theory of KBO. I do think that the main paper could benefit from a bit more insight that the appendix provides, in particular with regard to the structure of the multiplicative constants in the bounds. This could make connections to related theory more clear. This will also open up additional avenues for further investigation, which I believe are plentiful.\n\nNevertheless, I believe the contribution should be sufficient for the paper to be accepted, as this can be addressed during the rebuttal phase by adding suitable comments and using the additional space.\n\n**Strengths**\n\nTo my understanding, this is the first time that kernel-based bilevel optimization is discussed in this form of abstraction.\n\nGenerally good presentation and accessibility; newcomers to the field should be able to read and understand the basics of this work without significant issues.\n\nAppropriate degree of mathematical rigor, formalism and attention to detail in technical writing.\n\nThe generality of the proposed formalism will allow more targeted investigations based on known theory and techniques from adjacent fields and allow more detailed analyses.\n\n**Weaknesses**\n\nThe paper lists two applications as motivation in order to investigate general KBO, in particular distribution shift and instrumental variable regression—the latter is specifically investigated in the experiment. However, the authors do not compare their theoretical results and their proposed gradient scheme to existing results for those applications (to which they are inherently related). It is left unclear when (and how) their results extend these applications and/or give new insights.\n\nI also believe that the main paper hides a lot of theoretical insight in the generic multiplicative constants, which are given in full detail the appendix. In particular, connections to regularization theory and vector-valued regression (inner objective) and general optimization (outer objective) will become more obvious and make the results comparable to related work in those fields; see “Questions” section.\n\nQUESTIONS:\nI suggest the authors address the following two points:\n\n1) Using the additional space in order to expand the remarks and discussions around the constants in the main text and add some additional insight about how they depend on parameters related to the problem and the gradient scheme. In particular, I believe the interplay between regularization theory in the inner objective and optimization theory in the outer objective should be interesting to the reader. Furthermore, expanding the discussion of the constants might allow to highlight connections to available results in a more direct way. Note that Corollary 4.2 could specifically benefit from additional remarks.\n\n2) Discussing available results for the applications they list as motivation in order to consider KBO. In particular, how do these results relate to the new findings in the new, more general KBO setting? In particular, for kernel instrumental variable regression as given by [1], the two stage estimator allows the direct application of high-probability lower and optimal upper bounds for kernel ridge regression with vector-valued kernels [2], leading to minimax optimality for the instrumental variable procedure [3]. It seems reasonable to believe that similar arguments can be made for the inner objective of KBO under suitable assumptions about loss and noise, potentially leading to a fundamental lower bound. It might also be insightful to briefly mention relevant classical upper and lower bounds for general gradient schemes in the context of the outer objective in the main text. \n\nA minor technical point:\n\nMeasurability of the kernel needs to formally be required in order to ensure that expectations of functions and embedded variables in the RKHS are well-defined (e.g. [4, Chapter 4]). I think this could be added to the formal assumptions, in particular since the rest of the paper is written quite rigorously. \n\nThere is another very general question which is outside of the scope of the paper. I wondered the following:\n\nTwo-stage kernel instrumental variable regression as given by [1] uses a vector-valued ridge estimate in *both* stages.\nThe work [3] proposes to use a general spectral algorithm [5] for the *first* stage. \nIt is known that general spectral algorithms include iterative schemes such as gradient descent. If, instead of in the first stage, one uses a general spectral algorithm in the *second stage*, this might lead to a very similar (or even the equivalent) estimator that the present paper proposes in the special instance of instrumental variable regression and could lead to optimal rates by repeating the arguments by [3]. Incorporating loss calibration results [4] could allow to transfer these rates to more general losses.\nMaybe the authors also have some better intuition or opinions about this.\n\n[1] Singh et. al. Kernel instrumental variable regression. NeurIPS 2019.\n\n[2] Li et al. Towards Optimal Sobolev Norm Rates for the Vector-Valued Regularized Least-Squares Algorithm. JMLR 2024.\n\n[3] Meunier et al. Nonparametric Instrumental Regression via Kernel Methods is Minimax Optimal. arXiv:2411.19653 2024.\n\n[4] Steinwart & Christmann. Support Vector Machines. 2008.\n\n[5] Meunier et al. Optimal rates for vector-valued spectral regularization learning algorithms. NeurIPS 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies generalization properties of Kernel Bilevel Optimization (KBO). This task comes with many additional challenges compared to existing work (e.g. the parameter space growing with sample size for kernel methods), which the authors solve with several technical contributions. \n\nThe first contribution is showing that differentiation and plug-in emprical estimation of gradient derived via implicit differentiation commute under specified assumptions. Secondly, based on the commutative property the authors derive generalization bounds in Theorem 4.1 using the maximal inequalities for degenerate U-processes of order 2. These two results form the core of the paper and the intuition behind the proofs is well described. There is an additional result for gradient descent solver of KBO. Numerical experiments match the the theoretical bounds asymptotically when the sample size of the inner and outer loops are the same (m=n), although the scaling in terms of $m\\neq n$ is not explored numerically.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper in my opinion makes significant contribution and is written with great detail (although I checked the appendices only very quickly).\n\nFor weaknesses, I have mostly minor suggestions hoping to help with clarity for other readers and a remark, that I would like to see the experimental results when $m\\neq n$, perhaps as a figure in the appendices.\n\nSuggestions for clarity:\nThis is a long paper and I imagine that it was difficult to fit it into the page requirement.\n* If I understand correctly Sec 2.3. is about the difficulty to performing implicit differentiation (which is needed for KBO) for a kernel method. If this is the main point of Sec 2.3. and Prop 2.1, I think the clarity of the text would be improved if this was briefly stated at the beginning to motivate the section. Generally, I would like to better understand the purpose of Prop 2.1 (I guess it is later explained as motivating the plug-in estimate in 3.2), and also to have explained the intuition how it gets around the intractibility of the block equation above. \n* L156: Can you specify which of the operator is abstract and intractable? My guess it is the $\\nabla_h$, but it would improve clarity to be precise.\n* Literature review: I find the literature review very short, which might be also due to the page limit restrictions. Nevertheless, I want to ask the authors to extend this and perhaps include a short section in the appendices that gives:\n    * A short paragraph of few sentences on computational aspects of solving KBO and existing approaches. \n    * A few sentences about the existing generalization results for Bilevel optimization. There is a paragraph in L207 saying why the existing results don’t hold for kernel bilevel optimization, but it would be useful to have some information about the scaling of the existing bounds for the non-kernel case.\n\nL224: “and show that it recovers the previously introduced estimator $\\nabla \\hat{F}(\\omega)$”, this is technically not accurate as only $\\hat{F}(\\omega)$ was introduced previously and its gradient is firstly written right below.\n\nQUESTIONS:\n* In the first block equation in 2.2, is the distribution of $x,y$ in the expectation different, i.e., that one of them is on train and the other on the test data? I think it would be useful to explicitly highlight in the equation.\n* In the experiments you show an experiment when $m = n$ which does not demonstrate the theoretical scaling of $1/sqrt(n) + 1/sqrt(m)$. Could you vary $m$ and $n$ to show the scaling in both dimensions? This might reveal that the upper bounds are not tight in this regime, but it would be still very important to know.\n* L213: Claim that [6, 61] do not hold for parameter dimension growing with $n$, thus not for kernel methods. Is it possible to make some comparison for the asymptotic scaling in m or n?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper develops the first finite-sample generalization bounds for nonparametric bilevel optimization with inner optimization on reproducing kernel Hilbert spaces. Specifically, it gives an explicit gradient expression using implicit differentiation and then derives uniform generalization bounds for both the bilevel value function and its gradient using empirical process and U-process theory.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The work is technically solid, leveraging advanced empirical process methods to handle infinite-dimensional inner spaces for both function and gradient approximations.
- The paper is overall clearly written. The main ideas and proof strategies are well organized, with clear motivation, literature review, and fully discussed theoretical assumptions.
- Bilevel methods underpin many ML tasks (hyperparameter tuning, meta-learning, causal inference), yet lack nonparametric generalization theory. This fills a fundamental gap.
- It introduces a novel perspective to investigate the theoretical properties of bilevel optimization with kernels, which may largely inspire future work to loosen the inner problem's constraints.

**Weaknesses**
- The assumptions (e.g., smoothness and convexity) are relatively strong and may not hold in more complex or practical settings. It would be valuable to validate these assumptions empirically for real-world kernels, though this is admittedly a challenging task.
- Analysis focuses on basic gradient methods.

QUESTIONS:
- Could the authors comment on the possibility of relaxing Assumptions (C) and (D)?
- Can your framework extend to stochastic bilevel algorithms (e.g., [3,24])?

LIMITATIONS:
Yes. The limitations and scope are clearly discussed in Section 6.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02