SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mirroring the analyses in Sec. 2, we now investigate the effects of  $P^{PC}$  and  $P^{ex}$  in Fig. 5. In the left plot, we once more replicate [BHMM19]'s original experiment, and observe the same apparent trend that double descent emerges as we increase the number of raw parameters  $P^{\phi} = P^{PC} + P^{ex}$  (where the min-norm solution needs to be applied once  $P^{\phi} = n$ ). We then proceed to analyze the effects of varying  $P^{PC}$  and  $P^{ex}$  separately (while holding the other fixed). As before, we observe in the center plot that varying  $P^{PC}$  (determining the actual number of parameters being fit in the regression) for different levels of excess features indeed gives rise to the traditional U-shaped generalization curve. Conversely, in the right plot, we observe that increasing  $P^{ex}$  for a fixed number of  $P^{PC}$  results in an L-shaped generalization curve – indeed providing evidence that the effect of increasing the number of raw parameters past  $n$  in the original experiment can be more accurately explained as a gradual improvement in the quality of a basis to which a fixed capacity model is being fit.
As before, note that if we connect the solid lines in the center and right plots we recover exactly the double descent curve shown in the left plot of Fig. 5. Alternatively, as we demonstrate in Fig. 6(a), fixing  $P^{PC}$  at other values and then starting to increase the total number of parameters through  $P^{ex}$  allows us to move or remove the first peak arbitrarily. In Fig. 6(b), we demonstrate that one could even create multiple peaks by switching between parameter-increasing mechanisms more than once. This highlights that, while the transition from parameter increase through  $P^{PC}$  to  $P^{ex}$  naturally occurs at  $P^{\phi} = n$  due to the use of the min-norm solution, the second descent is not actually caused by the interpolation threshold  $P^{\phi} = n$  itself – but rather is due to the implicit change in model at exactly this point. Indeed, comparing the generalization curves in Fig. 6 with their train-error trajectories which we plot in Appendix E.2, it becomes clear that such a second descent can also occur in models that have not yet and will never achieve interpolation of the training data.
4 Part 2: Rethinking parameter counting through a classical statistics lens
Thus far, we have highlighted that "not all model parameters are created equal" - i.e. the intuitive notion that not all ways of increasing the number of raw parameters in an ML method have the same
This experiment is inspired by [CMBK21], who show multiple descent in regression by controlling the data-generating process (DGP), altering the order of revealing new (un)informative features. Even more striking than through changing the DGP, we show that simply reordering the mechanisms by which the same raw parameters are added to the model allows us to arbitrarily increase and decrease test loss.
effect. However, as we saw in the linear regression example, it is not always trivial to deconstruct the underlying mechanisms driving performance. Therefore, instead of having to reason about implicit complexity axes on a case-by-case basis for different models, hyperparameters, or inductive biases, we would rather be able to quantify the effect of these factors objectively. In what follows, we highlight that all previously considered methods can be interpreted as smoothers (in the classical statistics sense [HT90]). By making this connection in Sec. 4.1, we can exploit the properties of this class of models providing us with measures of their effective number of parameters. After adapting this concept to our setting (Sec. 4.2), we are finally able to re-calibrate the complexity axis of the original double descent experiments, finding that they do indeed fold back into more traditional U-shapes (Sec. 4.3).
4.1 Connections to smoothers
Smoothers are a class of supervised learning methods that summarize the relationship between outcomes  $Y \in \mathcal{Y} \subset \mathbb{R}^k$  and inputs  $X \in \mathcal{X} \subset \mathbb{R}^d$  by "smoothing" over values of  $Y$  observed in training. More formally, let  $k = 1$  w.l.o.g., and denote by  $\mathcal{D}^{\mathrm{train}} = \{(y_i, x_i)\}_{i=1}^n$  the training realizations of  $(X, Y) \in \mathcal{X} \times \mathcal{Y}$  and by  $\mathbf{y}_{\mathrm{train}} = (y_1, \ldots, y_n)^T$  the  $n \times 1$  vector of training outcomes with respective training indices  $\mathcal{T}_{\mathrm{train}} = \{1, \ldots, n\}$ . Then, for any admissible input  $x_0 \in \mathcal{X}$ , a smoother issues predictions
$$
\hat {f} \left(x _ {0}\right) = \hat {\mathbf {s}} \left(x _ {0}\right) \mathbf {y} _ {\text {t r a i n}} = \sum_ {i \in \mathcal {I} _ {\text {t r a i n}}} \hat {s} ^ {i} \left(x _ {0}\right) y _ {i} \tag {1}
$$
where  $\hat{\mathbf{s}}(x_0) = (\hat{s}^1(x_0), \dots, \hat{s}^n(x_0))$  is a  $1 \times n$  vector containing smoother weights for input  $x_0$ . A smoother is linear if  $\hat{\mathbf{s}}(\cdot)$  does not depend on  $\mathbf{y}_{\mathrm{train}}$ . The most well-known examples of smoothers rely on weighted (moving-) averages, which includes k-nearest neighbor (kNN) methods and kernel smoothers as special cases. (Local) linear regression and basis-expanded linear regressions, including splines, are other popular examples of linear smoothers (see e.g. [HT90, Ch. 2-3]).
In Appendix C, we show that all methods studied in Part 1 can be interpreted as smoothers, and derive  $\hat{\mathbf{s}} (\cdot)$  for each method. To provide some intuition, note that linear regression is a simple textbook example of a linear smoother [HT90], where  $\hat{\mathbf{s}} (\cdot)$  is constructed from the so-called projection (or hat) matrix [Eub84]. Further, trees - which issue predictions by averaging training outcomes within leaves - are sometimes interpreted as adaptive nearest neighbor methods [HTF09, Ch. 15.4.3] with learned (i.e. non-linear) weights, and as a corollary, boosted trees and sums of either admit similar interpretations.
4.2 A generalized measure of the effective number of parameters used by a smoother
The effective number of parameters  $p_e$  of a smoother was introduced to provide a measure of model complexity which can account for a broad class of models as well as different levels of model regularization (see e.g. [HT90, Ch. 3.5], [HTF09, Ch. 7.6]). This generalized the approach of simply counting raw parameters - which is not always possible or appropriate - to measure model complexity. This concept is calibrated towards linear regression so that, as we might desire, effective and raw parameter numbers are equal in the case of ordinary linear regression with  $p < n$ . In this section, we adapt the variance based effective parameter definition discussed in [HT90, Ch. 3.5]. Because for fixed  $\hat{\mathbf{s}}(\cdot)$  and outcomes generated with homoskedastic variance  $\sigma^2$  we have  $Var(\hat{f}(x_0)) = ||\hat{\mathbf{s}}(x_0)||^2\sigma^2$ , this definition uses that  $\frac{1}{n}\sum_{i\in\mathcal{I}_{\mathrm{train}}}Var(\hat{f}(x_i)) = \frac{\sigma^2}{n}\sum_{i\in\mathcal{I}_{\mathrm{train}}}||\hat{\mathbf{s}}(x_i)||^2$  for linear regression: one can define  $p_e = \sum_{i\in\mathcal{I}_{\mathrm{train}}}||\hat{\mathbf{s}}(x_i)||^2$  and thus have  $p_e = p$  for ordinary linear regression with  $n < p$ . As we discuss further in Appendix D, other variations of such effective parameter count definitions can also be found in the literature. However, this choice is particularly appropriate for our purposes as it has the unique characteristic that it can easily be adapted to arbitrary input points - a key distinction we will motivate next.
Historically, the smoothing literature has primarily focused on prediction in the classical fixed design setup where expected in-sample prediction error on the training inputs  $x_{i}$ , with only newly sampled targets  $y_{i}^{\prime}$ , was considered the main quantity of interest [HT90, RT19]. The modern ML literature, on the other hand, largely focuses its evaluations on out-of-sample prediction error in which we are interested in model performance, or generalization, on both unseen targets and unseen inputs (see e.g. [GBC16, Ch. 5.2]; [Mur22, Ch. 4.1]). To make effective parameters fit for modern purposes, it is therefore necessary to adapt  $p_{e}$  to measure the level of smoothing applied conditional on a given input, thus distinguishing between training and testing inputs. As  $||\hat{\mathbf{s}}(x_0)||^2$  can be computed for any input  $x_0$ , this is straightforward and can be done by replacing  $\mathcal{I}_{\mathrm{train}}$  in the definition of  $p_{e}$  by any other set of inputs indexed by  $\mathcal{I}_0$ . Note that the scale of  $p_{e}$  would then depend on  $|\mathcal{I}_0|$  due to the summation,
Figure 7: The effective number of parameters does not increase past the transition threshold. Plotting  $p_{\mathrm{s}}^{\mathrm{train}}$  (orange) and  $p_{\mathrm{s}}^{\mathrm{test}}$  (green) for the tree (left), boosting (centre) and RFF-linear regression (right) experiments considered in Part 1, using the original composite parameter axes of [BHMM19].
while it should actually depend on the number of training examples  $n$  (previously implicitly captured through  $|\mathcal{I}_{\mathrm{train}}|$ ) across which the smoothing occurs – thus we also need to recalibrate our definition by  $n / |\mathcal{I}_0|$ . As presented in Definition 1, we can then measure the generalized effective number of parameters  $p_{\hat{\mathbf{s}}}^{0}$  used by a smoother when issuing predictions for any set of inputs  $\mathcal{I}_0$ .
Definition 1 (Generalized Effective Number of Parameters). For a set of inputs  $\{x_j^0\}_{j\in \mathcal{I}_0}$ , define the Generalized Effective Number of Parameters  $p_{\hat{\mathbf{s}}}^{0}$  used by a smoother with weights  $\hat{\mathbf{s}} (\cdot)$  as
$$
p _ {\hat {\mathbf {s}}} ^ {0} \equiv p (\mathcal {I} _ {0}, \hat {\mathbf {s}} (\cdot)) = \frac {n}{| \mathcal {I} _ {0} |} \sum_ {j \in \mathcal {I} _ {0}} \left\| \hat {\mathbf {s}} \left(x _ {j} ^ {0}\right) \right\| ^ {2} \tag {2}
$$
Note that for the training inputs we recover the original quantity exactly ( $p_{e} = p_{\hat{\mathbf{s}}}^{\mathrm{train}}$ ). Further, for moving-average smoothers with  $\sum_{i=1}^{n} \hat{s}^{i}(x_{0}) = 1$  and  $0 \leq \hat{s}^{i}(x_{0}) \leq 1$  for all  $i$ , it holds that  $1 \leq p_{\hat{\mathbf{s}}}^{0} \leq n$ . Finally, for kNN estimators we have  $p_{\hat{\mathbf{s}}}^{0} = \frac{n}{k}$ , so that for smoothers satisfying  $\sum_{i=1}^{n} \hat{s}^{i}(x_{0}) = 1$  and  $0 \leq \hat{s}^{i}(x_{0}) \leq 1$  for all  $i$ , the quantity  $\tilde{k}_{\hat{\mathbf{s}}}^{0} = \frac{n}{p_{\hat{\mathbf{s}}}^{0}}$  also admits an interesting interpretation as measuring the effective number of nearest neighbors.
4.3 Back to U: Measuring effective parameters folds apparent double descent curves
Using Definition 1, we can now measure the effective number of parameters to re-examine the examples of double descent described in Sections 2 & 3. We plot the results of the 0-vs-all sub-problem in this section as the 10 one-vs-all models in the full experiment are each individually endowed with effective parameter counts  $p_{\hat{\mathbf{s}}}^{0}$ . Fig. 7 presents results plotting this quantity measured on the training inputs ( $p_{\hat{\mathbf{s}}}^{\mathrm{train}}$ ) and testing inputs ( $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$ ) against the original, composite parameter axes of [BHMM19], Fig. 8 considers the behavior of  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  for the two distinct mechanisms of parameter increase separately, and in Fig. 9 we plot test error against  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$ , finally replacing raw with effective parameter axes.
By examining the behavior of  $p_{\hat{\mathbf{s}}}^{0}$  in Fig. 7 and Fig. 8, several interesting insights emerge. First, we observe that these results complement the intuition developed in Part 1: In all cases, measures of the effective number of parameters never increase after the threshold where the mechanism for increasing the raw parameter count changes. Second, the distinction of measuring effective number of parameters used on unseen inputs, i.e. using  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  instead of  $p_{\hat{\mathbf{s}}}^{\mathrm{train}}$ , indeed better tracks the double descent phenomenon which itself emerges in generalization error (i.e. is also estimated on unseen inputs). This is best illustrated in Fig. 7 in the linear regression example where  $p_{\hat{\mathbf{s}}}^{\mathrm{train}} = n$  is constant once  $P^{\phi} \geq n$  - as is expected: simple matrix algebra reveals that  $\hat{\mathbf{s}}(x_i) = \mathbf{e}_i$  (with  $\mathbf{e}_i$  the  $i^{th}$  indicator vector) for any training example  $x_i$  once  $P^{\phi} \geq n$ . Intuitively, no smoothing across training labels occurs on the training inputs after the interpolation threshold as each input simply predicts its own label (note that this would also be the case for regression trees if each training example fell into its own leaf; this does not occur in these experiments as leaves are already pure before this point). As we
Figure 8: Understanding the behavior of the test-time effective number of parameters across the two parameter axes of each ML method. Plotting  $p_{\mathrm{s}}^{\mathrm{test}}$  by  $P^{leaf}$  for fixed  $P^{ens}$  in trees (left), by  $P^{boost}$  for fixed  $P^{ens}$  in boosting (centre) and by  $P^{PC}$  for fixed  $P^{ex}$  in RFF-linear regression (right) highlights that increases along the first parameter axes increase  $p_{\mathrm{s}}^{\mathrm{test}}$ , while increases along the second axes decrease  $p_{\mathrm{s}}^{\mathrm{test}}$ . Larger blue and red points  $(\bullet, \bullet)$  are points from [BHMM19]'s original parameter sequence.
Figure 9: Back to U. Plotting test error against the effective number of parameters as measured by  $p_{\mathrm{s}}^{\mathrm{test}}$  (larger blue and red points:  $\bullet$ ,  $\bullet$ ) for trees (left), boosting (centre) and RFF-linear regression (right) eliminates the double descent shape. In fact, points from the first dimension of [BHMM19]'s composite axis  $(\bullet)$  continue to produce the familiar U-shape, while points from the second dimension of the composite axis  $(\bullet)$  – which originally created the apparent second descent – fold back into classical U-shapes. Dotted lines  $(\dots)$  show the effect of perturbing the first complexity parameter at different fixed values of the second.
make more precise in Appendix C.2, this observation leads us to an interesting impossibility result: the in-sample prediction error of interpolating models therefore cannot experience a second descent in the (raw) overparameterized regime. This provides a new reason for the historical absence of double descent shapes (in one of the primary settings in which the U-shaped tradeoff was originally motivated [HT90]), complementing the discussion on the lack of previous observations in [BHMM19]. Third, while  $p_{\mathrm{s}}^{\mathrm{train}}$  is therefore not useful to understand double descent in generalization error, we note that for unseen inputs different levels of smoothing across the training labels can occur even in the interpolation regime – and this, as quantified by  $p_{\mathrm{s}}^{\mathrm{test}}$ , does result in an informative complexity proxy.
This becomes obvious in Fig. 9, where we plot the test errors of [BHMM19]'s experimental setup against  $p_{\mathrm{s}}^{\mathrm{test}}$  (larger blue and red points:  $\bullet$ ,  $\bullet$ ). For each such point after the interpolation threshold  $(\bullet)$ , we also provide dotted contour lines  $(\dots)$  representing the effect of reducing the parameters along the first complexity axis  $(P^{leaf}$ ,  $P^{boost}$  and  $P^{PC}$ ), resulting in models that can no longer perfectly interpolate (e.g. for trees, we fix their number  $P^{ens}$  and reduce their depth through  $P^{leaf}$ ). Finally, we discover that once we control for the effective parameters of each model in this way, we consistently observe that each example indeed folds back into shapes that are best characterized as standard convex (U-shaped) curves – providing a resolution to the apparent tension between traditional statistical intuition and double descent! We also note an interesting phenomenon: increasing  $P^{ens}$  and  $P^{ex}$  appears to not only shift the complexity-generalization contours downwards (decreasing error for each value of  $P^{leaf}$ ,  $P^{boost}$  and  $P^{PC}$ ), but also shifts them to the left (decreasing the effective complexity implied by each value of  $P^{leaf}$ ,  $P^{boost}$  and  $P^{PC}$ , as also highlighted in Fig. 8) and flattens their ascent (making error less sensitive to interpolation). That is, models in (or close to) the interpolation regime discovered by [BHMM19] exhibit a very interesting behavior where setting  $P^{ex}$  and  $P^{ens}$  larger actually decreases the test error through a reduction in the effective number of parameters used.
5 Consolidation with Related Work
In this section, we consolidate our findings with the most relevant recent work on understanding double descent; we provide an additional broader review of the literature in Appendix A. A large portion of this literature has focused on modeling double descent in the raw number of parameters in linear regression, where these parameter counts are varied by employing random feature models [BHMM19, BHX20] or by varying the ratio of  $n$  to input feature dimension  $d$  [ASS20, BLLT20, DLM20, HMRT22]. This literature, typically studying the behavior and conditioning of feature covariance matrices, has produced precise theoretical analyses of double descent for particular models as the raw number of parameters is increased (e.g. [BLLT20, HMRT22]). Our insights in Sections 3 and 4 are complementary to this line of work, providing a new perspective in terms of (a) decomposing raw parameters into two separate complexity axes which jointly produce the double descent shape only due to the choice of (min-norm) solution and (b) the underlying effective number of parameters used by a model. Additionally, using theoretical insights from this line of work can make more precise the role of  $P^{ex}$  in improving the quality of a learned basis: as we show in Appendix B.3, increasing  $P^{ex}$  in our experiments indeed leads to the PC feature matrices being better conditioned.
Further, unrelated to the smoothing lens we use in Part 2 of this paper, different notions of effective parameters or complexity have appeared in other studies of double descent in specific
ML methods: [BM21] use Rademacher complexity (RC) in their study of random forests but find that RC cannot explain their generalization because forests do not have lower RC than individual trees (unlike the behavior of our  $p_{\mathrm{s}}^{\mathrm{test}}$  in this case). [MBW20] use [Mac91]'s Bayesian interpretation of effective parameters based on the Hessian of the training loss when studying neural networks. While this proxy was originally motivated in a linear regression setting, it cannot explain double descent in linear regression because, as discussed further in Appendix D, it does not decrease therein once  $p > n$ . Finally, [DLM20] compute effective ridge penalties implied by min-norm linear regression, and [DSYW20] consider minimum description length principles to measure complexity in ridge regression. Relative to this literature, our work differs not only in terms of the ML methods we can consider (in particular, we are uniquely able to explain the tree- and boosting experiments through our lens), but also in the insights we provide e.g. we distinctively propose to distinguish between effective parameters used on train- versus test-examples, which we showed to be crucial to explaining the double descent phenomenon in Sec. 4.3.
Conclusion: A Resolution to the ostensible tension between non-deep double descent and statistical intuition. We demonstrated that existing experimental evidence for double descent in trees, boosting and linear regression does not contradict the traditional notion of a U-shaped complexity-generalization curve: to the contrary, we showed that in all three cases, there are actually two independent underlying complexity axes that each exhibit a standard convex shape, and that the observed double descent phenomenon is a direct consequence of transitioning between these two distinct mechanisms of increasing the total number of model parameters. Furthermore, we highlighted that when we plot a measure of the effective, test-time, parameter count (instead of raw parameters) on their x-axes, the apparent double descent curves indeed fold back into more traditional U-shapes.
What about deep double descent? In this work, we intentionally limited ourselves to the study of non-deep double descent. Whether the approach pursued in this work could provide an alternative path to understanding double descent in the case of deep learning – arguably its most prominent setting – is thus a very natural next question. It may indeed be instructive to investigate whether there also exist multiple implicitly entangled complexity axes in neural networks, and whether this may help to explain double descent in that setting. In particular, one promising approach to bridging this gap could be to combine our insights of Sec. 3 with the known connections between random feature models and two-layer neural networks [SGT18], and stochastic gradient descent and min-norm solutions  $\mathrm{[GWB^{+}17]}$ . We consider this a fruitful and non-trivial direction for future research.
What are the practical implications of these findings? On the one hand, with regards to the specific ML methods under investigation, our empirical results in Sections 2 and 3 imply interesting trade-offs between the need for hyperparameter tuning and raw model size. All methods appear to have one hyperparameter axis to which error can be highly sensitive  $P^{leaf}$ ,  $P^{boost}$  and  $P^{PC}$  while along the other axis, "bigger is better" (or at least, not worse). In fact, it appears that the higher  $P^{ens}$  or  $P^{ex}$ , the less sensitive the model becomes to changes along the first axis. This may constitute anecdotal evidence that the respective first axis can be best understood as a train-time bias-reduction axis - it controls how well the training data can be fit (increasing parameters along this axis reduces underfitting - only when set to its maximum can interpolation be achieved). The second axis, conversely, appears to predominantly achieve variance-reduction at test-time: it decreases  $||\hat{\mathbf{s}}(x_0)||$ , reducing the impact of noise by smoothing over more training examples when issuing predictions for unseen inputs.
On the other hand, our results in Sec. 4 suggest interesting new avenues for model selection more generally, by highlighting potential routes of redemption for classical criteria trading off in-sample performance and parameter counts (e.g. [Aka74, Mal73]) – which have been largely abandoned in ML in favor of selection strategies evaluating held-out prediction error [Ras18]. While criteria based on raw parameter counts may be outdated in the modern ML regime, selection criteria based on effective parameter counts  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  used on a test-set could provide an interesting new alternative, with the advantage of not requiring access to labels on held-out data, unlike error-based methods. In Appendix E.5, we provide anecdotal evidence that when choosing between models with different hyperparameter settings that all achieve zero training error, considering each model's  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  could be used to identify a good choice in terms of generalization performance: we illustrate this for the case of gradient boosting where we use  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  to choose additional hyperparameters. Investigating such approaches to model selection more extensively could be another promising avenue for future work.