Title: Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural Networks

Generalization

Implicit Bias

Theory

Matrix Factorization
Abstract: Conventional wisdom attributes the mysterious generalization abilities of overparameterized neural networks to gradient descent (and its variants). The recent volume hypothesis challenges this view: it posits that these generalization abilities persist even when gradient descent is replaced by Guess & Check (G&C), i.e., by randomly drawing weight settings until one that fits the training data is found. The validity of the volume hypothesis for wide and deep neural networks remains an open question. In this paper, we theoretically investigate this question for matrix factorization (with linear and non-linear activation): a canonical testbed in neural network theory. We first prove that generalization under G&C deteriorates with increasing width, establishing what is, to our knowledge, the first canonical case where G&C is provably inferior to gradient descent. Conversely, we prove that generalization under G&C improves with increasing depth, revealing a stark contrast between wide and deep networks, which we further validate empirically. These findings suggest that even in simple settings, there may not be a simple answer to the question of whether neural networks need gradient descent to generalize well.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization properties of over-parameterized neural networks by considering scenario of low-rank matrix sensing. The central contribution of this paper derives generalization bounds for a guess-and-check optimization algorithm. In particular, when the weights of a fully connected FFN is drawn from an iid Gaussian distribution:\n\n- In the limit as the width of a fully-connected FFN $\\to \\infty$, the out-of-sample loss of guess and check is as bad as random guessing.\n\n- Under additional assumptions such as *restricted isometry property*, in the limit as the depth of the network $\\to \\infty$, the out-of-sample loss approaches 1.\n\nThese two results demonstrate that whether guess-and-check can match gradient descent in generalization performance is instance dependent.\n\nSTRENGTHS AND WEAKNESSES:\nI do not have too many comments on this work because the main takeaway of this paper is clear and the overall technical setup is not particularly complicated. It is actually refreshing to read this paper given the recent trend of content creeps in ML theory papers.\n\nIn terms of the main message of the paper, I think the testbed should be called low-rank matrix sensing rather than matrix factorization, since the later does not make much sense when the activation in eq (4) is non-linear.\n\nThe paper as whole is not difficult to understand and the author did a good job at identifying a concise theoretical framework that captures the intended messages. Personally, I feel that the first result on infinite width (Theorem 1) is not too surprising, and the second result on infinite depth (Theorem 2) does require some fairly strong assumptions. But both of these results answered very relevant questions and therefore they should be shared with the wider theory community.\n\nFor the technical contents, I briefly read over the proofs in Appendix A and B, though I did not check the details line-by-line. The math seems to be sound overall. The theorem statements in the main body are clear and easy to understand, but their counterparts in the Appendix (e.g. Theorem 6) are somewhat bloated.\n\nThings that could be improved:\n1. Despite the conciseness of the overall message, I found the paper to be a bit more difficult to read than it should be. In particular, Definition 3 should be split into 2. And in many instances, equations should be on their own lines and equation block alignments are often not good.\n\n2. Definition 1 should be moved to Section 4.3. And the author should add a bit more discussing the RIP property because it may be unfamiliar to a lot of readers.\n\n3. I feel the experiments in Section 5 is unnecessary.\n\n4. The proof sketches for both Theorem 1 and 2 are kind of rushed and do not seem to capture all of the main technical steps of the actual proofs.\n\n5. The assumptions for Theorem 2 is really, really strong. Imposing the ground truth matrix to be rank 1 really simplifies the problem by a lot. I recommend the authors to extend this result to more general settings.\n\n6. The proof of Theorem 1 seems to heavily rely on the well-known equivalence between infinite width neural networks and Gaussian processes. This does somewhat diminish the technical novelties of the paper.\n\nOverall, I think the results of this paper is sufficiently interesting for publication. But it is not particularly sophisticated in the technical contribution and some might argue that the guess-and-check algorithm does not warrant research in the first place. Also, there is very little from this paper that could lead to wider implications for both future theoretical and empirical studies. So I think it is fair to recommend a **borderline accept**.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the volume hypothesis -- the claim that \"most\" near-interpolating solutions have small generalization error -- in the context of low-rank matrix factorization. The authors specifically consider a model parameterized by a (deep) feed-forward neural network with i.i.d random initialization. The first main result, Theorem 1, is that as the width of the neural network increases, the probability that a randomly sampled near-interpolating solution has small generalization error tends to zero. On the other hand, the second main result, Theorem 2, is that as the depth increases, the probability that a near-interpolator has small generalization error tends to 1.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n- The problem studied here is well motivated, and matrix factorization is a tractable and interesting testbed to understand how randomly sampled near-interpolators generalize. I thus think the results are of interest to the community.\n- To the best of my knowledge, the results presented here are novel.\n- The submission is well-written and easy to understand. I skimmed the appendices, and the proofs appear to be correct.\n- Theorems 1 and 2 are supported empirically.\n\n## Weaknesses\n- As the authors indeed admit, the theoretical results rely on a number of assumptions, such as the choice of activation function. In my opinion, the most restrictive assumption is that Theorem 2 requires the ground truth target to be rank 1. The proof of Theorem 2 relies on showing that in the large depth limit, the product $W_d \\cdots W_1$ is close to a rank 1 matrix. As such, if the ground truth were actually rank 2, it does not seem like the same proof technique would still work.\n- Implications beyond matrix factorization towards more standard neural network settings (such as training a fully connected network on a simple dataset) are not clear.\n- (minor) In Figure 2, it is not obvious that as depth increases, the generalization error converges to zero. The authors state that running gradient descent at larger depths is computationally prohibitive. But it seems like it should be straightforward to run guess and check at larger depths?\n\nQUESTIONS:\n- The initialization in Theorem 1 seems to be an \"NTK\" initialization (hidden layer variances are 1/width), and the proof relies on showing that the pre-activations are close to Gaussian. It is not particularly surprising that in the linear/kernel regime, interpolating solutions do not generalize -- the intuition from linear regression being that the parameters in the subspace of the datapoints and the orthogonal subspace are independent. Is this understanding correct? And can you comment on what would happen if a different initialization were used that permitted feature learning (say muP/mean-field where the hidden layer variances are 1/width^2)?\n- Theorems 1 and 2 focus on what happens if either width or depth is taken to infinity, while the other is fixed. What if both are taken to infinity at the same time (say at a fixed ratio)? What would the generalization error look like?\n- I would also appreciate if the authors could comment on my above concerns re the rank 1 assumption in Theorem 2.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the question if the use of gradient descent as an optimizer generalizes systematically better than a random sampling strategy, Guess and Check (G&C), at the exemplary problem of (low rank) matrix recovery. More specifically, it uses linear measurements of a (ground truth) low-rank matrix as training data, and measures generalization via the L2 norm of the projection of predicted and ground truth matrix on the subspace orthogonal to the measurements. The work proves that, in the limit of infinite width, G&C does not generalize well, despite the existence of a result proving that – in a specific case – gradient descent does generalize: The probability of G&C generalizing well becomes as bad as randomly drawing from the prior distribution independent of the training data. In contrast, in the limit of infinite depth d, the probability of G&C generalizing converges to 1 at a rate of 1/d. The authors claim that the former result makes this paper the first case where G&C has proven to be inferior to gradient descent. Overall, the paper concludes that the is no simple answer to the question if networks need gradient descent to generalize well.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles a very interesting question and is very well-written. All technical results are described (in terms of their meaning, interpretation and intuition) before stating them formally, allowing readers to see the reasons for certain formal definitions. Similarly, the main ideas of the proofs have been stated in the papers, while all details are deferred to the appendix. \n\nWhile the contribution and overall study are nice, I have some doubts about the conclusions. The following part is based on my (possibly wrong) understanding of the results and I’d highly appreciate a comment/correction in the rebuttal: The results seem to heavily depend on the desire to find low-rank matrices. The setup is not a supervised learning scenario. The training data could be identical for many different distributions of ground truth matrices, i.e., the same y_i could originate from a sparse ground truth matrix or a ground truth matrix with a small $\\ell_p$ norm. As for identical training data the resulting matrices produced by gradient descent and G&C will not change, the generalization seems to depend on the (hidden) desired ground truth. If G&C and gradient descent (GD) yield probability distributions p( . | L_{train}<epsilon) that are sufficiently different, doesn’t that immediately mean there are settings where G&C generalizes better than GD and settings where GD generalizes better than GD (because the unknown ground truth distribution could coincide with either of the two without really changing the training data)? In this sense generalization does not seem to be a property to associate with an algorithm (alone). \n\nIf we, for instance, consider the least-squares problem $||Ax-y||^2$. Running gradient descent converges to a solution such that $x-x_0 \\in \\text{kern}(A)^\\perp$, i.e., it commonly ends at a solution close to the minimal norm solution (for typical initializations). If $y = Ax_{true}$ and $x_{true}$ is sparse, then the solution of gradient descent will be quite different from $x_{true}$. But would we really blame this on gradient descent not generalizing well? Don’t we need to talk about the property of generalization for the combination of architecture, optimization algorithm and data distribution? If so, this would be something to emphasize in the paper.\n\nQUESTIONS:\nPlease clarify my main question raised under \"strength and weaknesses\" above. I might very well just be mistaken on this, as I am not an expert on this topic. If not, please explain how you plan to change the interpretation of the result.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the generalization behavior of the Guess & Check (G&C) algorithm in the context of matrix factorization problems, both with and without activation functions. The G&C algorithm operates by sampling parameters from a prior distribution without any gradient-based training, and selecting those that yield sufficiently low training loss. The paper presents two main theoretical findings. First, under certain assumptions, as the network width tends to infinity, the generalization performance of the G&C algorithm degrades to the level of random parameter selection. Second, under specific rank-1 assumption, increasing the network depth can lead to improved generalization performance of the G&C algorithm. The author uses this to illustrate whether gradient-based algorithms are necessary for the generalization of neural networks. The conclusion is that there is no simple answer and specific analysis is required for specific situations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The volume hypothesis that the author is trying to solve seems to be a very interesting problem. Generalization, as the most fundamental ability of neural networks, is guaranteed by the optimization algorithm or by the architecture itself, which is a question worth studying.\n\n2. The authors provide a reasonably comprehensive review of relevant literature and appropriately cite several classical and foundational results in the field.\n\n3. The conclusions that the author wants to convey are relatively clear, and the overall writing logic of the paper is relatively clear.\n\n4. I checked some theoretical details, and the author's theoretical results are relatively solid. The derivations build upon well-established statistical principles, and the arguments are generally rigorous.\n\n**Weakness**\n\n1. My primary concern lies in the theoretical contributions. The core results, particularly Theorems 1 and 2, appear to be largely derived from well-established prior work on the limiting behavior of neural networks as width or depth tends to infinity. For instance, it is a well-known result that infinitely wide neural networks converge to Gaussian processes. The theoretical derivations in the current paper seem to build directly upon such existing limit results to obtain implications for matrix sensing problems. In this context, the derivation of the limiting distribution itself is arguably more fundamental, and the additional insights provided in this work seem rather limited in terms of novelty. I think it would be a greater theoretical contribution if the authors could provide a generalization analysis of the G&C algorithm under finite width/depth (this setting in Chiang et al.).\n\n2. Another major concern is the limited scope of the theoretical results. All analysis in the paper is conducted under the matrix sensing setting, where the number of observed samples $n < m \\times m'$. Theorem 1 essentially states that in the infinite-width limit, under undersampling, the training and test data become statistically independent—i.e., no generalization is possible. However, this conclusion heavily relies on the special structure of matrix sensing, particularly the inner-product structure of the measurement operator. I am skeptical whether this result would hold in more general settings, such as learning a target function using a fully connected neural network (which allows unlimited number of samples). In such cases, the training and test are unlikely to be completely independent.\n\n3. The authors cite Proposition 1 to argue that gradient descent (GD) generalizes well in the infinite-width limit. However, I believe this claim requires more careful justification and proof. Specifically, the cited result from [92] shows that for any fixed width  $k$, gradient descent can recover the target matrix with high probability as the initialization scale tends to zero. This setting corresponds to the nonlinear training dynamics regime, and crucially, it does **not** require $k \\to \\infty$. The authors seem to use it to illustrate the generalization guarantee of GD when the width tends to infinity. I think the conditions of these two theorems are completely different, so the two cannot be compared to show the generalization of GD is guaranteed, and the generalization of G \\& C is weaker.\n\n4. Although the authors cite a fairly comprehensive literature, there are some important and related works that are not mentioned. For example, [1, 2] discovered the frequency principle (or spectral bias) of the general neural network structure itself. Because of the smoothness of the activation function, the network itself will learn from low frequency to high frequency, and thus has good generalization for the actual data distribution (often low frequency). In addition, more specifically, for matrix factorization models, [3] systematically studied the implicit regularization effect of data connectivity on the low rank and low nuclear norm of the matrix decomposition problem, especially highlighting the impact of the intrinsic invariant manifold induced by the model structure on global dynamics. In addition, [4] studied how the implicit preference for depth promotes generalization.\n\n[1] Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 28(5):1746–1767, 2020.\n\n[2] On the spectral bias of deep neural networks. International Conference on Machine Learning, 2019.\n\n[3] Connectivity shapes implicit regularization in matrix factorization models for matrix completion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, volume 37, pages 45914–45955, 2024.\n\n[4] The implicit bias of depth: How incremental learning drives generalization, in: International Conference on Learning Representations, 2019.\n\nQUESTIONS:\n1. On the one hand, the authors aim to generalize matrix factorization results to multi-layer neural networks with nonlinear activation functions, which goes beyond the classical matrix factorization setting.  However, why the analysis must be restricted specifically to matrix sensing settings? Could the authors clarify whether this limitation is inherent to their theoretical framework, or if it is possible to extend the results to more general supervised learning settings?\n\n2. According to the restatement of Proposition 1, is it implied that the generalization loss tends to zero as the width $k \\to \\infty$? If so, I believe this conclusion requires a more careful justification or even a formal proof. It is not immediately clear how this follows from Theorem 3.3.\n\n3. In Figure 1, what is the precise relationship between the plotted curves and the training error threshold $\\epsilon_{\\text{train}}$? How would the results change if a different value of $\\epsilon_{\\text{train}}$ were used? A sensitivity analysis would help clarify this.\n\n4. Why is the prior baseline curve omitted in Figure 2? It was included in Figure 1, so it would be helpful to explain the rationale for excluding it here.\n\n4. The experimental section lacks sufficient persuasive power. According to the authors’ theoretical claims:\n   - As width increases, the difference between the generalization error of the G&C algorithm and that of random guessing should follow a straight line with slope $-1/2$ in a log-log plot.\n   - As depth increases, the generalization error of the G&C algorithm should follow a straight line with slope $-1$ in a log-log plot.\n\n   A more quantitative and explicit visualization of these trends in Figures 1 and 2—e.g., by plotting the log-log slope or fitting lines—would significantly enhance the clarity and credibility of the empirical findings.\n\n\nOther Minor Comments:\n\n1. Line 26: Reference [102] appears to be cited twice; please check for redundancy.\n\n2. The notation for sets is somewhat confusing. In Line 142, parentheses are used to denote a set, while in Line 148, the same notation appears to indicate concatenation. It would be clearer and more conventional to use curly braces for sets to avoid ambiguity.\n\nLIMITATIONS:\nIn addition to the theoretical assumptions and settings discussed by the authors, the limited empirical validation also constitutes a potential limitation of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies whether gradient descent is necessary for neural networks to generalise well, using matrix factorisation as a simplified and analytically tractable setting. The authors compare the generalisation performance of gradient descent and a non-optimised alternative known as Guess & Check (G&C). They analyse how generalisation under G&C behaves as network width and depth vary. Their theoretical results suggest that generalisation under G&C deteriorates with increasing width but improves with increasing depth, in contrast to gradient descent, which performs consistently well. These theoretical findings are complemented by empirical experiments within the matrix factorisation framework. The study aims to shed light on the conditions under which gradient descent may — or may not — be necessary for generalisation.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nThe paper is solid and presents rigorous theoretical analyses under well-defined assumptions. The authors provide formal results characterising the generalisation performance of G&C as a function of network width and depth. The empirical results are consistent with the theoretical predictions (see question regarding the dimensionality). Naturally, as acknowledged by the authors, the results are derived in a very specific setting (matrix factorisation with specific activations and priors), and their relevance to more realistic networks remains to be established in this context. Since matrix factorisation is a traditional playground in theoretical analysis, its use is well motivated. That said, I would highlight more carefully the additional restrictions imposed by Theorem 2 compared to Theorem 1, and their connection to the most standard settings in matrix factorisation.\n\n\n**Clarity**\n\nThe paper is generally well written and well structured. The motivation is clear, and the presentation of results is good.\n\n\n**Significance**\n\nThe work addresses a timely and important question raised by recent publications: whether gradient-based optimisation is necessary for generalisation. While the matrix factorisation setting is simplified, it serves as a common theoretical proxy. The observed contrast between width and depth is interesting, though its underlying cause still needs to be better understood.\n\n\n**Originality**\n\nThe paper offers new theoretical contributions regarding the limitations and capabilities of G&C, particularly the contrasting behaviour under increasing width and depth. This perspective contributes to the ongoing discussion around the volume hypothesis and the role of implicit bias in generalisation.\n\nQUESTIONS:\n**1)** In Figure 1c, a double descent behaviour [Belkin et al., 2018] appears to emerge around width $\\approx 12$. Would the authors say that, in this setting, this behaviour stems from the algorithmic component of the learning process (i.e., the choice of optimiser), rather than from data or model architecture alone?\n\n**2)** Do the authors have any intuition about the contrasting behaviour under increasing width and increasing depth? I wonder whether this contrast could be related—perhaps even indirectly—to the classical universal approximation theorem for networks of arbitrary width [Cybenko, 1989].\n\n**3)** The numerical experiments are performed in very low dimensions. For instance, in Figure 1, we have $m=m' = 5$ and $n=15$. Could there be finite-size effects influencing the observed trends? This is particularly relevant since Theorem 1 considers the limit $k \\to \\infty$.\n\n**4)** The paper seems to support the perspective that architecture, data, and algorithmic choices cannot be studied in isolation when analysing generalisation in overparameterised systems [Zdeborová, 2020, Nature Physics]. Do the authors agree with this interpretation, and if so, could they comment on how their findings might inform this broader view?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}