Title: The $\varphi$ Curve: The Shape of Generalization through the Lens of Norm-based Capacity Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: generalization

norm-based capacity

deterministic equivalence
Abstract: Understanding how the test risk scales with model complexity is a central question in machine learning. Classical theory is challenged by the learning curves observed for large over-parametrized deep networks. Capacity measures based on parameter count typically fail to account for these empirical observations. To tackle this challenge, we consider norm-based capacity measures and develop our study for random features based estimators, widely used as simplified theoretical models for more complex networks. In this context, we provide a precise characterization of how the estimator’s norm concentrates and how it governs the associated test error. Our results show that the predicted learning curve admits a phase transition from under- to over-parameterization, but no double descent behavior. This confirms that more classical U-shaped behavior is recovered considering appropriate capacity measures based on models norms rather than size. From a technical point of view, we leverage deterministic equivalence as the key tool and further develop new deterministic quantities which are of independent interest.


Peer reviews:
comments: SUMMARY:
The paper explains the double descent phenomenon for RFM from a theoretical perspective. Specifically, a simple neural network model was considered and the relationship between test error and network norm was calcualted.

STRENGTHS AND WEAKNESSES:
Strengths:

The theory presented in this article is novel and undoubtedly has helped us further understand the double descent  phenomenon for RFM. 

Weakness:

$The\ writing\ is\ not\ good\ enough,$ the author has placed too much basic settings in the appendix, which is not good, if only read the  main text, it will be hard to understand the paper.

$The setting is too simple.$ From the author's perspective, this is essentially a variant of a linear problem. The author needs to discuss how this differs from the classical linear problems?

$Need\ more\ References$, such as:《On the Double Descent of Random Features Models Trained with SGD》,nips2022.

QUESTIONS:
(1): The network and data used in the paper is simple, even close to a linear model, because the author only analyzed the linear function in the output layer of RFM, which can directly help derive the optimal formula (1) and simplify the calculation. And the label $y$  is also selected by linear with noise. However, commonly used neural networks and data are highly nonlinear, does your conclusion help us understand the  phenomenon of double decline in general situation?

(2): Is RFM being used in some field nowadays? Some examples?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-06

comments: SUMMARY:
This paper studies the relationship between norm-based capacity and test risk by applying the deterministic equivalent technique from random matrix theory. Using random feature models, the authors precisely characterize this relationship both theoretically and through empirical validation.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper is clearly written and is methodically presented. The author's claim are sufficiently supported by formal mathematical proofs and are validated through empirical testing via synthetic and real-life datasets.
2. The proposed theory reveal that, under specific assumptions about the data, the test curve in the overparameterized regime consistently below the underparameterized test curve. This finding is novel in my opinion as it differs from the standard double descent phenomenon, where a " $\varphi$ " shaped test curve is typically observed.

Weaknesses
NA

QUESTIONS:
1. In line 246, can you further explain as to why the bias follows a U-shaped curve, as opposed to the monotonic decrease shown in the classical bias-variance trade-off explanation? Specifically, what is the mechanism that causes a model's attempt to fit training data more closely to result in higher bias? The references shown in the sentence do not seem to discuss this in detail.
2. A natural question to ask is that, under what conditions does the test error curve in the overparameterized regime transition from being consistently below its underparameterized counterpart to above it ( $\varphi$ -shaped curve) and vice versa? It would be interesting to touch on this point.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo tackle the challenge of characterizing the double descent learning curve based on the number of model parameters, this paper propose using the $l_2$ norm of the estimator's weights as a more suitable measure of capacity. When the learning curve is plotted against this norm-based capacity, the double descent disappears and is replaced by a more traditional U-shaped curve, aligning with classical statistical intuition. To prove this, the paper uses a mathematical tool from random matrix theory called deterministic equivalence to derive precise formulas for the test risk and the estimator's norm. This analysis is performed for linear regression and Random Feature Models (RFMs), which serve as theoretical stand-ins for two-layer neural networks. The theoretical results are shown to be in excellent agreement with experiments on both synthetic and real-world data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tStrong Theoretical Foundation: The paper uses the deterministic equivalence in random matrix theory to provide precise, non-asymptotic characterizations of the relationship between estimator norm and test risk. A key strength is the development of new, more general deterministic quantities, which is a technical contribution of independent interest. The study is thorough, covering various model settings, including regularized $(\\lambda>0)$ and min-norm interpolating $(\\lambda=0)$ solutions.\n2.\tClean Result: The key idea that a more appropriate capacity measure (norm) resolves the double descent puzzle is compelling and intuitive. It provides a clear explanation for the success of over-parameterized models without discarding classical statistical result.\n3.\tConvincing Validation: The paper's theoretical predictions are meticulously validated with experiments. The plots consistently show a near-perfect match between the derived theoretical curves and the empirical data points, lending strong support to the analysis.\n\nWeaknesses\n\n1.\tLimited Scope of Models: The primary weakness, which the authors acknowledge, is that the theoretical analysis is confined to linear models and Random Feature Models (RFMs). The result also only holds for the second layer of RFMs, which is far from modern deep neural networks.\n2.\tPracticality of the Findings: The paper suggests that the model norm can be controlled via the regularization parameter, $\\lambda$. However, it also points out that the behavior is different in the under- and over-parameterized regimes, meaning one must know which state the model is in to properly control the norm, which is a practical challenge.\n3.\tDependence on Assumptions: The theoretical results rely on specific assumptions about the data, such as power-law eigenvalue decay (Assumption 2). The authors show that on real-world datasets like FashionMNIST, where these assumptions may not hold, the observed learning curve can deviate from the main theoretical prediction.\n4.\tNovelty Concern: The paper's contribution, while valuable, can be viewed as an incremental extension of the very recent work in reference [14]. The paper heavily leverages the framework, assumptions, and theorems from [14] for its analysis. The core methodological novelty is extending the tools from [14] to derive the deterministic equivalent for the estimator's norm—a different quantity than the test risk studied in [14]. This makes the contribution an insightful application and extension of an existing framework rather than the introduction of a fundamentally new one.\n\nQUESTIONS:\n1.\tCan you provide some intuition on how to generalize these results to deep, multi-layer neural network where all parameters are trained?\n2.\tHow can the analyses be generalized to other norms? Are other measures like the \"path norm\" more suitable for complex models?\n3.\tHow do these insights into norm-based capacity translate to tasks beyond regression, such as classification?\n\nLIMITATIONS:\n1.\tThe theoretical analysis is restricted to linear models and Random Feature Models (RFMs), and does not extend to end-to-end trained deep neural networks.\n2.\tThe study only considers models with closed-form solutions and overlooks the implicit regularization effects introduced by optimization methods such as Stochastic Gradient Descent (SGD). In practice, the choice of optimizer can significantly influence the model norm, which is not captured in the current analysis.\n3.\tThe theoretical guarantees rely on strong assumptions about data structure, such as power-law eigenvalue decay. The paper also shows that when these assumptions are not satisfied—e.g., on real-world datasets—the empirical results can deviate from theoretical predictions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor the problem of understanding how the test risk scales with model complexity in machine learning, this paper studies the shape of the generalization curve through a norm-based capacity measure. This paper uses deterministic equivalence as a key theoretical tool to characterize the relationship between risk and norm. In addition, this paper verifies that capacity control based on the $\\ell_2$ norm can recover the classic U-shaped curve in RFM, which challenges the double descent behavior. Validation on synthetic data and MNIST shows that the theoretical predictions are consistent with the empirical results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper argues that when using a norm-based capacity metric, the learning curve of an over-parameterized deep learning model changes from a \"double-descent\" shape to a U-shaped curve that is more in line with classical statistical intuition. The authors verify this view through theoretical analysis and experiments using random feature models (RFMs).\n\n2. Another key finding of this paper is that in the over-parameterized region, the test risk increases monotonically with the norm capacity. This seems to confirm the conclusion of classical learning theory to some extent: small-capacity models have better generalization performance. This finding is very interesting and seems to suggest that the current mainstream view that \"classical learning theory is not applicable to deep learning\" is a misunderstanding, that is, people's understanding of capacity is not in place.\n\n3. This paper clarifies the essential connection between norm capacity and generalization, reveals the monotonic relationship between the regularization parameter $lambda$ and norm capacity, and provides a new perspective for model tuning.\n\nWeaknesses:\n\nI really appreciate this work, especially some of the insights it brings to the table through experimental phenomena, which, although not new in some ways, have been controversial in the learning theory community. The consistency between the theoretical analysis and experimental verification in this paper has answered some controversies very well. My only concern is the universality of the conclusions of this paper, because the analysis and verification involved are limited to RFM and linear models, lacking empirical evidence for deep models (more than 2 layers), so supplementary experimental verification in deep cases is necessary. In addition, it is necessary to discuss how to apply the control of norm capacity to actual model selection or training strategies (such as early stopping, weight decay).\n\nQUESTIONS:\nPlease refer to the Weaknesses for details.\n\nLIMITATIONS:\nThis work does not seem to have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}