Title: Generalization Bounds for Rank-sparse Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning Theory

Generalization Bounds

Schatten Norms

Low-rank
Abstract: It has been recently observed in much of the literature that neural networks exhibit a bottleneck rank property: for larger depths, the activation and weights of neural networks trained with gradient-based methods tend to be of approximately low rank. In fact, the rank of the activations of each layer converges to a fixed value referred to as the ``bottleneck rank", which is the minimum rank required to represent the training data. This perspective is in line with the observation that regularizing linear networks (without activations) with weight decay is equivalent to minimizing the Schatten $p$ quasi norm of the neural network. In this paper we investigate the implications of this phenomenon for generalization. More specifically, we prove generalization bounds for neural networks which exploit the approximate low rank structure of the weight matrices if present. The final results rely on the Schatten $p$ quasi norms of the weight matrices: for small p, the bounds exhibit a sample complexity $ \widetilde{O}(WrL^2)$ where $W$ and $L$ are the width and depth of the neural network respectively and where $r$ is the rank of the weight matrices. As $p$ increases, the bound behaves more like a norm-based bound instead. The proof techniques involve a careful interpolation between the parametric and norm based regimes. We also demonstrate in experiments that this bound outperforms both classic parameter counting and norm based bounds in the typical overparametrized regime.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves a novel generalization bound for fully-connected and convolutional neural networks. This bound features a free parameter $p_l$ for each layer $l$ which could be chosen a-posteriori (that is, the bound is uniform wrt these parameters). By choosing these parameters, one could interpolate between a norm based bound similar to [1] to a classical parameter-counting bound, individually for each layer. To be specific, the bound is based on $p$-Schatten norms of layer weights, which interpolate between a Frobenius norm ($p=2$) and a usual matrix rank ($p=0$). Being able to interpolate gives flexibility that allows to cover the case of approximately low-rank layer weights conveniently.\n\nThe paper provides a thorough overview on existing norm-based and parameter-counting generalization bounds, as well as on a low-rank bias of neural nets. There are also experiments conducted for simple multi-layer fully-connected and convolutional networks trained on MNIST and CIFAR dataset.\n\n[1] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The approach taken in the paper generalizes two large classes of generalization bounds (norm-based and parameter counting) in a natural and neat way.\n2. The literature overview is rather thorough.\n3. The paper spends a lot of time comparing its main result and approach to relevant works, thus showing clearly where this work stands among other works.\n4. The main theorems are stated in a clean and easy-to-absorb way.\n\n**Weaknesses**\n\n1. Rank sparsity was specifically enforced in experiments for fully-connected networks.  To me, this is not really fair because this looks like forging a setup that favors the bound of the paper but not the bounds the paper compares to.\n2. The architecture used for experiments on convolutional nets looks strange: 3 conv layers followed by 5 fully-connected layers. What was the reason to use this structure? This looks neither like a standard setup to compare with other works, nor like a simple well-performing ConvNet.\n3. The comparison ignores logarithmic factors, and, probably, constants; are the exact numbers provided in Appendix D informative then? Maybe, an asymptotic for large depth or width could be more informative.\n4. The paper does not disclose any exact values of the bounds they prove. This makes me suspect these bounds to be vacuous in more-or-less realistic setups when evaluated up to a number. At the same time, non-vacuous bounds do exist (see e.g. [2,3]), but the paper never mentions them.\n5. One relevant citation seems to be missing: [1] studies low rank biases for linear networks induced by $l_2$-regularization.\n\n**Minor issues**\n\n1. Not clear from this point what is L_1 in (1) (it is introduced later). If it is a Lipschitz constant, why \"we assume that the nonlinearities are 1-Lipschitz when discussing related works\"?\n2. Line 111: there is no $p$ in the above formula.\n3. Line 110: is the factor $L^{p/(2+p)}$ forgotten here?\n4. Not clear from this point what is $p_l$ in the formula on line 110 (before the main result is formally introduced).\n5.  Line 78: \"expreimntally\"\n6. Line 285: the argument $x_i$ seems to be at the wrong place.\n\n[1] Jacot, A., Ged, F., Gabriel, F., Simsek, B., and Hongler, C. (2021). Deep linear networks dynamics: Low-rank biases induced by initialization scale and l2 regularization. https://arxiv.org/abs/2106.15933v1\n\n[2] Zhou, Wenda, et al. \"Non-vacuous generalization bounds at the imagenet scale: a PAC-bayesian compression approach.\" \n\n[3] Biggs, Felix, and Benjamin Guedj. \"Non-vacuous generalisation bounds for shallow neural networks.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n1. Am I right that Line 110 does not coincide with (2) when all $p_l=2$? In other words, does your Theorem 3.2 recover the main result of [Neyshabur et al. 2018] when all $p_l=2$? If no, why?\n2. Lines 90 — 94: which \"deep results\" specifically are you referring to?\n\nLIMITATIONS:\nIf the obtained bounds are numerically vacuous, I would mention it as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides generalization bounds via covering numbers for neural networks with low-rank (or approximately low-rank) weight matrices.\n\nSTRENGTHS AND WEAKNESSES:\nGeneralization bounds based on low-rank weight matrices have the potential to be much tighter than existing bounds. It has been observed empirically and theoretically that depth induces an inductive bias towards low-rank weight matrices, this result is a great first step towards tightening generalization bounds in this setting.\n\nHowever, the paper is currently poorly presented. Technical notation is used in the introduction long before it is defined in section 3. I was unable to verify technical details because of the very, very long proof without a sketch. A readable sketch that gives more insight into the techniques would be helpful.\n\nAdditionally, the final bound has poor dependency on the input space dimension. At least this work is *upfront* about the fact that dependency on input dimension is a major limitation. \n\nOther minor comments/suggestions to improve clarity:\n- Line 26: \"DNNs *on* natural data\"\n- Line 38: the parenthetical (for simplicity...) is better as a footnote\n- Eq 1,2,3 and Lines 104, 110, 114: *Lots* of undefined notation here, including $p_\\ell, L_1$, etc. Perhaps simplify the presentation by assuming some terms are constants here-- $L_1, B, \\rho$, etc. \n- Line 56: make the norm *small*\n- Line 75: Should also cite Parkinson et al. 2023 (arXiv:2305.15598)\n- Line 83: Typo: $A \\in \\mathbb R^{C \\times d}$ not $A \\in \\mathbb R^{w \\times d}$\n- Line 91: These results are more about the effect of adding a $L_2$ regularization term to the loss than about optimization itself.\n- Line 115: While later you specify that it's best to choose $M_\\ell$ to be zero in which case talking about the low rank structure in the term makes sense, it's confusing that $M_\\ell$ is dropped here. If $M_\\ell = 0$ is the best choice, why include $M_\\ell$ in line 110 at all?\n- Line 237: The first sentence seems to have a word missing?\n- Line 250-251: This doesn't make sense as written. I think there's a typo in equation it's comparing to?\n- Line 285: is this supposed to be the notation from Line 266?\n\nQUESTIONS:\nCan you provide a proof sketch of a simplified form of Theorems E.2? I am not comfortable accepting this paper without a readable proof sketch to fascilitate verification of the validity of the proof, and I think a good proof sketch will make this paper much more useful to the community. Aim for as much simplicity in the sketch as possible; e.g. assume that $L_1 = B = \\rho_i = 1, L=2$, etc., and use $\\lesssim$ liberally instead of explicitly showing constants, etc.. If the sketch is clear and verifiable, I will increase my score to a 5.\n\nYou mention that you believe that the dependency on input dimension is likely removable. Do you believe that the dependency on the output dimension and/or hidden width is also removable?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a generalization bound that utilizes the low-rankness of the weights in deep learning. Choice of the matrix Schatten $p$-norm to characterize network layers result in a particular effect in the generalization bound: as $p\\to0$, the remaining dominant term in the bound makes the bound act like a parameter-counting bound, while as $p\\to2$, the bound acts like a product-of-norms bound, while still incorporating the approximate low-rankness of the matrices. The authors extend their results to CNNs as well.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is clearly situated and motivated within the context of previous literature.\n- The authors present generalization bounds that exploit the (approximate) low-rankness of the weight matrices. Although their bounds are not necessarily strictly tighter than previous bounds (e.g. with full-rank matrices), that they can be tighter utilizing the approximate low-rankness of the matrices is important.\n- That their bounds behave like a parameter-counting bound vs. product-of-norms bound based on the $p$ value is independently interesting\n- The authors expand their results to CNNs.\n\n## Weaknesses\n- The paper presents a strong narrative and addresses a meaningful problem, but it has significant issues with clarity of the presentation, particularly in the introduction. The reader is presented with technical claims before being properly introduced to the notation and setup; the introduction repeatedly emphasizes the low-rankness of representations in deep learning, only to later clarify that their results are based on the low-rankness of *weights*; the abstract refers to Schatten quasi-norm, but only later does it become apparent that the results apply across the full range $0 \\leq p \\leq 2$. None of the issues are individually disqualifying but their cumulative effect significantly undermines the readability of the paper. See the section below for more details.\n\nQUESTIONS:\n- L8: Schatten norms are introduced as quasi-norms in the abstract, and described as such in L84. Then in the later section results with $0\\leq p \\leq 2$ are presented. This conflict persists elsewhere in the paper as well, please fix.\n- L42: Using a more specific term instead of \"architectural parameters\" would be helpful\n- L56: Cut-off sentence?\n- L60: Difficult to understand summary\n- L75: The abstract and introduction are unclear regarding whether the focus of the paper will be low-rankness of weights vs. representations. A stark example is L94 discussing representation rank vs. L95 declaring that the paper will focus on weight rank.\n- L78: \"experimentally\" typo\n- L83: Dimensions of $A$ wrong?\n- L89: \"with more tolerance for small non-zero singular values\". This is unclear. What does tolerance mean in this case?\n- L93: \"rank of the activation layer\"? Rank of the post-activation representations?\n- L94: The term “bottleneck rank” is used multiple times by now without a clear or formal definition, two of these in the abstract. Its relevance to the paper’s main results is also not well motivated. Unless it's central to the core theorems - which currently does not appear to be the case - I suggest removing all references to it. An alternative is to present this within a cogent discussion as in L365 (and doing so *once*).\n- L107: \"over the classes\"?\n- L115: Omitting the reference matrix before the clarification in L223 is confusing\n- L123: I appreciate that the authors are not burying the lead by introducing baseline vs. their own proposed bounds in Section 1, but without a proper introduction to the notation this gets real confusing real fast\n- L139: \"spacial\" typo\n- L145: I cannot understand the main point of this paragraph\n- L175: I wonder if this detailed a discussion between previous and current work should be left for after the main results.\n- L208: \"common joint distribution\"\n- L208: $\\mathcal{D}$ undefined\n- L226: \"extreme\"?\n- L228: I don't think this paragraph clearly introduces the significance of the linear network results\n- L276: \"indices\"?\n- L277: max. input norm is already defined\n- L281: Is the replaced term realistically/often smaller than the maximum input norm?\n- L317: Unclear, use the standard term \"matricization\" perhaps?\n- L323: \"spacial\"\n\nLIMITATIONS:\nThe authors are transparent about the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies why neural networks often form approximately low-rank representations in their layers during training, a property sometimes called the bottleneck rank. The authors analyze this phenomenon and develop new theoretical generalization bounds that explicitly account for the rank-sparse structure of neural network weights. Using the framework of Schatten p quasi norms, they derive sample complexity results that interpolate between traditional norm-based bounds and parameter-counting bounds, capturing the benefit of low-rank weight matrices. The paper extends these results to linear networks, fully-connected deep neural networks, and convolutional neural networks (CNNs), providing a new perspective on how low-rank properties induced by depth can improve generalization. Overall, the work offers novel theoretical insights on the role of implicit low-rank regularization in deep learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is rigorously grounded in mathematical theory, presenting novel generalization bounds for neural networks by leveraging the low-rank (“bottleneck rank”) structure of their weight matrices. \n\n1. Extending low-rank sample complexity arguments beyond linear networks to FNNs and deep CNNs, while explicitly considering the Schatten p quasi-norm, is innovative.\n2. Overall the paper is clearly written, especially in its statement of the main results and their significance. Morever, the related works are concrete including key literatures.\n3. The analysis is thorough, covering linear networks, deep neural networks, and CNNs, with detailed treatment of covering numbers and complexity measures. Clear notations, definitions, and assumptions are well documented.\n\nWeaknesses:\n\n1. It presents main theoretical results in section 3, while there is no any problem set up, no empirical results and detailed experimental settings in the main text, though some results presented in the appendix. However, they are not well referred to the theorems presented in the main text. \n2. The proofs, while thorough, are highly technical and may be difficult to follow for a broader ML audience not specialized in generalization theory.\n3. There is no experimental verification of the tightness of these bounds beyond theoretical discussion (though they mention an appendix with MNIST/CIFAR10 references, it is not sufficiently emphasized and lacks of comparisons with recent approaches).\n4. Some transitions between theoretical claims and related works are abrupt, making it hard to track their originality relative to prior results without repeatedly cross-referencing. \n5. There is limited demonstration of the practical impact for network design or training, which might weaken its appeal to practitioners.\n6. The assumptions of low-rank weight matrices may not hold in all architectures, especially in smaller models or architectures with strong regularization.\n\nQUESTIONS:\n1. The paper introduces a parameter 𝑝 in the Schatten 𝑝-quasi norm, but does not give any practical guidance for choosing or interpreting this parameter. Could the authors discuss how to select 𝑝 or its influence on the bound? \n2. Could the authors clarify the approximate magnitude of the bound, or at least discuss whether they are likely to be meaningful for realistic network sizes?\n2. The paper’s interpolation between norm-based and rank-based bounds is interesting. Could the authors clarify more explicitly how their framework compares to, or might be combined with, margin-based generalization analyses (e.g., in kernel methods or neural tangent kernel literature)?\n2. The theoretical guarantees appear to rest on the existence of low-rank (or approximate low-rank) weight matrices. Can the authors comment on how realistic this assumption is across a wider range of architectures, for example transformers or large language models?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper obtains generalization bounds for neural networks with low complexity weight matrices - both in terms of low rank as well as low Schatten $p$-quasi norm for $0<p<1$. The bound obtained by the authors scales as $O \\left ( \\sqrt{\\frac{L^2 r W}{n}} \\right) $ and does not scale exponentially with the depth or width of the network. The authors also evaluate their bounds on trained deep networks and show that their results compare favorably to other generalization bounds.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper is very well written and comprehensively discusses a lot of related papers on generalization bounds. Their appendix sections serve as a good review of the literature on generalization bounds for deep networks. The application of covering number based bounds allows the authors to avoid exponential dependence on the depth. The authors derive their bound for fully connected networks as well as convolutional networks. They also study how their techniques interact with others such as loss function augmentation to obtain data dependent bounds. The results seem quite favorable, especially when looking at the comparisons on trained networks.\n\nWeaknesses: While this is a strong paper with a lot of technical results, it mostly applies techniques developed by prior work in the context of deep network generalization bounds. Rademacher complexity bounds through metric entropy, parametric interpolation for bounding the covering numbers of approximately low rank matrices, and complexity bounds for convolutional networks that depend on the weights of the filters rather than the number of patches are all known results. Putting them together is a valuable contribution, but that also means that the mathematical novelty in this paper is limited. In case this is mistaken, the authors should do a more precise job of outlining their novel mathematical contributions.\n\nQUESTIONS:\n1. Are your bounds still larger than the test error? It seems like the are in Fig D.6 In which case, are they still technically vacuous?\n\n2. Is there a route to obtaining non-vacuous generalization bounds through the low-rank bias of deep networks? Possibly applying your techniques with PAC-Bayesian bounds.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}