Title: Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generalization bound

algorithm-dependent bound

data-dependent kernel

neural tangent kernel
Abstract: Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity bounds for kernel methods—specifically those based on the RKHS norm and kernel trace—through a data-dependent kernel called the loss path kernel (LPK). Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics, leading to tighter and more informative generalization guarantees. Moreover, the bound highlights how the norm of the training loss gradients along the optimization trajectory influences the final generalization performance. The key technical ingredients in our proof combine stability analysis of gradient flow with uniform convergence via Rademacher complexity. Our bound recovers existing kernel regression bounds for overparameterized neural networks and shows the feature learning capability of neural networks compared to kernel methods. Numerical experiments on real-world datasets validate that our bounds correlate well with the true generalization gap.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper moves beyond the static NTK framework by using the Loss Path Kernel to capture training dynamics. It derives a novel generalization bound using Rademacher complexity, whose leading term is determined by the term $\\int_0^T \\| \\nabla_w l(w_t,z) \\|^2 dt$. This approach yields a tighter, data-dependent bound about $O(\\frac{1}{n})$ that directly links optimization dynamics to generalization, surpassing existing results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper has enough technical contribution and beautiful results. It derives a tighter bound on the generalization error, and the theory is in line with many known phenomena, like why memorizing random labels hurts generalization. The authors also verified the bound on CIFAR-10 with ResNet, and the experiments are consistent with the theoretical results, which is a strong point.\n\nWeaknesses: My main concern is about the exact order of the upper bound. The main term, which is based on the integral of gradients $\\int_0^T \\| \\nabla_w l(w_t,z) \\|^2 dt$, can only be calculated after training is finished. This means the bound is a post-hoc analysis tool, and cannot be used to predict generalization or guide the training process beforehand. Also, computing this term is very expensive as it requires the full gradient history, making it hard to use in practice.\n\nQUESTIONS:\n1、Do we have a way to bound the term $\\int_0^T \\| \\nabla_w l(w_t,z) \\|^2 dt$\n\n2、Do Gradient Descent and Gradient Flow have a difference in the order of the resulting bound?\n\nLIMITATIONS:\nThe proposed bound $\\Gamma$ cannot be determined theoretically a priori. While the authors claim it can be used for monitoring and prediction in practice, they do not adequately discuss its high computational cost. This makes their claim impractical for large-scale applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes new generalization error bounds for the gradient flow algorithm. The method is an extension of the previous work of Chen et al. on loss path kernel. The authors integrate uniform stability and uniform convergence analysis with loss path kernel, leading to tighter generalization bounds. The results demonstrate that the generalization gap is highly dependent on the norm of the training loss gradients along the optimization trajectory. Further extension to stochastic gradient flow and several applications on the NTK regime and kernel ridge regression are discussed, and the results are empirically evaluated to show high correlation with the true generalization gap.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* The paper addresses an important problem in learning theory. The authors improve the results of a previous work by Chen et al, tightening the bound and easing the computation in practice. Empirical results have shown the superiority of these results compared to some other bounds in the literature.\n\nWeakness:\n* To my understanding, using the method of uniform stability or uniform convergence requires additional assumptions like Lipschitzness and smoothness, while the previous one of Chen et al does not need such assumptions. While they are prevalent ones also seen in many generalization analysis papers, they are actually quite strong and make the bound computationally intractable, since the Lipschitz or smoothness parameters are hard to estimate in practice.\n* It is not directly clear how to compare the tightness between the bounds of this paper and the previous one by Chen et al. While the $\\Gamma$ part of the bound is improved as clearly seen, these new bounds also introduce some extra terms of stability constants. There perhaps also miss some comparison between these bounds and those ones derived solely based on stability or uniform convergence approaches. The authors claimed a tighter bound, but this is not directly clear to me.\n* For the same reason, the empirical comparison may not be sufficient since only the $\\Gamma$ part of the bound is computed and plotted. It is not clear if the bound is still tight when added with other stability or uniform convergence terms.\n\nQUESTIONS:\n* Can the authors elaborate more on the impact of the big O terms on the tightness and computational tractability of these bounds? How are these bounds empirically compared to the previous one by Chen et al?\n* Can these bounds provide some new insights for the design of learning algorithms? From eq. (3), what I can tell is that minimizing the norm of gradients may help, encouraging techniques like gradient clipping. The other part in eq. (3) looks like a fixed constant, as we always get something like $L_S(w_0) \\approx 0.5$ and $L_S(w_T) \\approx 0$ in case of binary classification. Are there any other insights we can tell from these results?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel generalization bound for parametric models trained with gradient flow. It builds on a recent result showing that the loss at the end of training of any parametric model is a kernel machine for a certain data-dependent kernel called Loss Path Kernel (LPK). Therefore the class of models realizable by gradient flow could be enumerated by corresponding LPKs for all possible training datasets.\n\nIf there was a single kernel, one could use a classical bound for Rademacher complexity which scales as a square root of a kernel Gram matrix trace over the dataset size $n$. One could generalize this bound for a class of kernel machines realizable by a set of kernels, but such a bound does not generally decrease with $n$.\n\nThe main novelty of the present paper is a concentration result on LPKs: that is, as $n$ grows to infinity, all LPKs concentrate around their average over training datasets. This leads to a generalization gap bound whose main term is the classical Rademacher complexity bound mentioned above evaluated on the LPK on the training dataset.\n\nThe main term of the bound closely tracks the actual generalization gap times a constant. This constant is relatively small $(\\approx 3)$ for a binary CIFAR classification using a two-layer MLP, but much larger $(\\approx 1e5)$ when ResNet18 is considered.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. I believe, the concentration of LPKs is a natural, novel and generally useful result, closely related to algorithmic stability.\n2. The paper features a very nicely written background section which makes it easy to read and comprehend.\n3. The main term of the bound tracks the time evolution of the generalization gap really well, even in realistic scenarios like ResNet18 trained on CIFAR10, up to a multiplicative constant. Maybe, it could be useful for predicting the optimal early stopping time?\n\n**Weaknesses**\n\n1. The regularity assumption on the loss as a function of model weights (Assumption 4.2) looks too strong for me. While it is quite natural for some models, like Random Feature Model, it is too restrictive for neural networks with at least two trainable layers. Indeed, consider a two-layer MLP with a sigmoid activation function, no bias terms, and a bounded (say, also sigmoid) loss. Suppose the input layer weights are zeros, while the output layer weights are very large. Then even a very small change of input weights results in a finite change of the loss. This shows that the Lipschitz property is not satisfied even for this toy model. Having a non-bounded activation function, non-bounded loss, and more layers cannot make the situation better.\n2. If the above is correct, the bound cannot be applied to any of the scenarios considered in Section 7, Numerical Experiments.\n\n**Minor comments**\n\n1. The work [1] constructs a generalization bound for neural nets close to being linear; this is also a special case of algorithmic stability which heavily restricts the hypothesis class (i.e. only weights close to a linear network could be found), hence could be mentioned in the Related Work section.\n\n[1] https://openreview.net/pdf?id=tRpWaK3pWh\n\nQUESTIONS:\n1. I haven't looked at the proof, but do you have any intuitive explanation why the epsilon term in your Theorem 5.2 is a minimum between two terms, not a maximum? Because naively when you construct an upper-bound, you have to take the largest between competing terms, not the smallest.\n2. Have you tried to compute your bound up to a number? How close is it to gamma on your experiments? Does it also track the actual generalization gap well?\n\nLIMITATIONS:\nI think the point I discuss above in Weaknesses has to be mentioned as a limitation.\nI will increase my score if the authors either (a) prove me wrong concerning this point, or (b) correct/remove the Numerical Experiments section and include the aforementioned point as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel theoretical framework to derive algorithm-dependent generalization bounds for gradient flow (GF), grounded in Rademacher complexity and a data-dependent kernel—the loss path kernel (LPK). The authors leverage the dynamics of gradient flow to build a function class characterized by the LPK and provide uniform convergence bounds with high probability. Key contributions include tighter generalization bounds than prior works, extensions to stochastic gradient flow (SGF), and experimental evidence that the derived bounds correlate with the true generalization gap. The results also recover known bounds in NTK and kernel ridge regression regimes, while demonstrating the advantages of feature learning in neural networks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Theoretical Novelty and Depth: The paper offers a nontrivial integration of algorithmic stability and uniform convergence through a novel kernel—the LPK—capturing the full training trajectory. This provides a compelling alternative to static kernels like NTK.\n\n2. Unification and Generalization: The results unify classical kernel generalization bounds with modern deep learning training via gradient flow, while being applicable across convex, strongly convex, and non-convex regimes.\n\n3. Empirical Validation: Numerical results on CIFAR-10 with ResNet and two-layer NNs convincingly support the theoretical claims. The derived bounds track the generalization gap and explain overfitting dynamics and label noise effects.\n\nWeaknesses:\n\n1. Limited Practical Scope: Although insightful, the theoretical analysis is limited to full-batch gradient flow with infinitesimal step size, which does not align with practical stochastic optimization scenarios in deep learning.\n\n2. LPK Computation Complexity: Although discussed as “data-dependent,” computing LPK in practice may be computationally demanding, especially when integrating over long training trajectories.\n\n3. When training time T is large, the asymptotic generalization bound aligns with standard bounds via Radamacher complexity analysis and poorer than that via covering number analysis.\n\nQUESTIONS:\n1. How robust is the LPK-based generalization bound to architectural variations? The experiments mainly focus on ResNets and two-layer NNs. It would be valuable to understand if the same correlation between the bound and generalization gap holds for transformers or larger language models.\n\n2. Can the authors extend the results to discrete-time gradient descent or SGD with fixed learning rate? The current analysis assumes gradient flow with infinitesimal steps. A more practical extension would improve relevance.\n\n3. What is the computational complexity of evaluating the LPK term? \n\n4. How sensitive is the bound to initialization? Since LPK integrates gradient information across the trajectory, the choice of initialization could significantly impact the generalization bound. Clarifying this would help with practical deployment.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}