SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alicia Curth*
University of Cambridge amc253@cam.ac.uk
Alan Jeffares*
University of Cambridge aj659@cam.ac.uk
University of Cambridge  
mv472@cam.ac.uk
Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a  $U$ -shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count  $p$  grows past sample size  $n$  – a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at the evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple, distinct complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold  $p = n$ . We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves do indeed fold back into more traditional convex shapes – providing a resolution to the ostensible tension between double descent and traditional statistical intuition.
Historically, throughout the statistical learning literature, the relationship between model complexity and prediction error has been well-understood as a careful balancing act between underfitting, associated with models of high bias, and overfitting, associated with high model variability. This implied tradeoff, with optimal performance achieved between extremes, gives rise to a U-shaped curve, illustrated in the left panel of Fig. 1. It has been a fundamental tenet of learning from data, omnipresent in introductions to statistical learning [HT90, Vap95, HTF09], and is also practically reflected in numerous classical model selection criteria that explicitly trade off training error with model complexity [Mal73, Aka74, Sch78]. Importantly, much of the intuition relating to this U-shaped curve was originally developed in the context of the earlier statistics literature (see e.g. the historical note in [Nea19]), which focussed on conceptually simple learning methods such as linear regression, splines or nearest neighbor methods [WW75, HT90, GBD92] and their expected in-sample prediction error, which fixes inputs and resamples noisy outcomes [Mal73, HT90, RT19].
Figure 1: A 3D generalization plot with two complexity axes unfolding into double descent. A generalization plot with two complexity axes, each exhibiting a convex curve (left). By increasing raw parameters along different axes sequentially, a double descent effect appears to emerge along their composite axis (right).
The modern machine learning (ML) literature, conversely, focuses on far more flexible methods with relatively huge parameter counts and considers their generalization to unseen inputs [GBC16, Mur22]. A similar U-shaped curve was long accepted to also govern the complexity-generalization relationship of such methods [GBD92, Vap95, HTF09]– until highly overparametrized models, e.g. neural networks, were recently found to achieve near-zero training error and excellent test set performance [NTS14, BLLT20, Bel21]. In this light, the seminal paper of Belkin et al. (2019) [BHMM19] sparked a new line of research by arguing for a need to extend on the apparent limitations of classic understanding to account for a double descent in prediction performance as the total number of model parameters (and thus – presumably – model complexity) grows. This is illustrated in the right panel of Fig. 1. Intuitively, it is argued that while the traditional U-curve is appropriate for the regime in which the number of total model parameters  $p$  is smaller than the number of instances  $n$ , it no longer holds in the modern, zero train-error, interpolation regime where  $p > n$  – here, test error experiences a second descent. Further, it was demonstrated that this modern double descent view of model complexity applies not only in deep learning where it was first observed [BO96, NMB+18, SGd+18, ASS20], but also ubiquitously appears across many non-deep learning methods such as trees, boosting and even linear regression [BHMM19].
Contributions. In this work, we investigate whether the double descent behavior observed in recent empirical studies of such non-deep ML methods truly disagrees with the traditional notion of a U-shaped tradeoff between model complexity and prediction error. In two parts, we argue that once careful consideration is given to what is being plotted on the axes of these double descent plots, the originally counter-intuitive peaking behavior can be comprehensively explained under existing paradigms:
- Part 1: Revisiting existing experimental evidence. We show that in the experimental evidence for non-deep double descent – using trees, boosting, and linear regressions – there is implicitly more than one complexity axis along which the parameter count grows. Conceptually, as illustrated in Fig. 1, we demonstrate that this empirical evidence for double descent can thus be comprehensively explained as a consequence of an implicit unfolding of a 3D plot with two orthogonal complexity axes (that both individually display a classical convex curve) into a single 2D-curve. We also highlight that the location of the second descent is thus not inherently tied to the interpolation threshold. While this is straightforward to show for the tree- and boosting examples (Sec. 2), deconstructing the underlying axes in the linear regression example is non-trivial (and involves understanding the connections between min-norm solutions and unsupervised dimensionality reduction). Our analysis in this case (Sec. 3) could thus be of independent interest as a simple new interpretation of double descent in linear regression.
- Part 2: Rethinking parameter counting through a classical statistics lens. We then note that all methods considered in Part 1 can be interpreted as smoothers (Sec. 4), which are usually compared in terms of a measure of the effective (instead of raw) number of parameters they use when issuing predictions [HT90]. As existing measures were derived with in-sample prediction in mind, we propose a generalized effective parameter measure  $p_{\mathrm{s}}^{0}$  that allows to consider arbitrary sets of inputs  $\mathcal{I}_0$ . Using  $p_{\mathrm{s}}^{0}$  to measure complexity, we then indeed discover that the apparent double descent curves fold back into more traditional U-shapes – because  $p_{\mathrm{s}}^{0}$  is not actually increasing in the interpolation regime. Further, we find that, in the interpolation regime, trained models tend to use a different number of effective parameters when issuing predictions on unseen test inputs than on previously observed training inputs. We also note that, while such interpolating models can generalize well to unseen inputs, overparametrization cannot improve their performance in terms of the in-sample prediction error originally of interest in statistics – providing a new reason for the historical absence of double descent curves. Finally, we discuss practical implications for e.g. model comparison.
Figure 2: Decomposing double descent for trees. Reproducing [BHMM19]'s tree experiment (left). Test error by  $P^{leaf}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{leaf}$  (right).
Part 1: Revisiting the evidence for double descent in non-deep ML models
Experimental setup. We center our study around the non-neural experiments in [BHMM19] as it is the seminal paper on double descent and provides the broadest account of non-deep learning methods that exhibit double descent. Through multiple empirical studies, [BHMM19] demonstrate that double descent arises in trees, boosting and linear regression. Below, we re-analyze these experiments and highlight that in each study, as we transition from the classical U-shaped regime into the subsequent second descent regime, something else implicitly changes in the model or training definition, fundamentally changing the class of models under consideration exactly at the transition threshold between the observed regimes – which is precisely the cause of the second descent phenomenon. In Sec. 2, we begin by investigating the tree and boosting experiments, where this is straightforward to show. In Sec. 3, we then investigate the linear regression example, where decomposing the underlying mechanisms is non-trivial. Throughout, we closely follow [BHMM19]'s experimental setup: they use standard benchmark datasets and train all ML methods by minimizing the squared loss<sup>1</sup>. Similarly to their work, we focus on results using MNIST (with  $n_{train} = 10000$ ) in the main text. We present additional results, including other datasets, and further discussion of the experimental setup in Appendix E.
2 Warm-up: Observations of double descent in trees and boosting
2.1 Understanding double descent in trees
In the left panel of Fig. 2, we replicate the experiment in [BHMM19]'s Fig. 4, demonstrating double descent in trees. In their experiment, the number of model parameters is initially controlled through the maximum allowed number of terminal leaf nodes  $P^{leaf}$ . However,  $P^{leaf}$  for a single tree cannot be increased past  $n$  (which is when every leaf contains only one instance), and often  $\max(P^{leaf}) < n$  whenever larger leaves are already pure. Therefore, when  $P^{leaf}$  reaches its maximum, in order to further increase the raw number of parameters, it is necessary to change how further parameters are added to the model. [BHMM19] thus transition to showing how test error evolves as one averages over an increasing number  $P^{ens}$  of different trees grown to full depth, where each tree will generally be distinct due to the randomness in features considered for each split. As one switches between plotting increasing  $P^{leaf}$  and  $P^{ens}$  on the x-axis, one is thus conceptually no longer increasing the number of parameters within the same model class: in fact, when  $P^{ens} > 1$  one is no longer actually considering a tree, but instead an ensemble of trees (i.e. a random forest [Bre01] without bootstrapping).
In Fig. 2, we illustrate this empirically: in the center plot we show that, on the one hand, for fixed  $P^{ens}$ , error exhibits a classical convex U- (or L-)shape in tree-depth  $P^{leaf}$ . On the other hand, for fixed  $P^{leaf}$  in the right plot, error also exhibits an L-shape in the number of trees  $P^{ens}$ , i.e. a convex shape without any ascent - which is in line with the known empirical observation that adding trees to a random forest generally does not hurt [HTF09, Ch. 15.3.4]. Thus, only by transitioning from increasing  $P^{leaf}$  (with  $P^{ens} = 1$ ) to increasing  $P^{ens}$  (with  $P^{leaf} = n$ ) - i.e. by connecting the two solid curves across the middle and the right plot - do we obtain the double
Figure 3: Shifting the peak. Transitioning from  $P^{leaf}$  to  $P^{ens}$  at different values of  $P^{leaf}$ .
<sup>1</sup>In multi-class settings, [BHMM19] use a one-vs-rest strategy and report squared loss summed across classes. Their use of the squared loss is supported by recent work on squared loss for classification [HB21, MNS+21], and practically implies the use of standard regression implementations of the considered ML methods.
Figure 4: Decomposing double descent for gradient boosting. Reproducing [BHMM19]'s boosting experiment (left). Test error by  $P^{boost}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{boost}$  (right).
descent curve in the left plot of Fig. 2. In Fig. 3, we show that we could therefore arbitrarily move or even remove the first peak by changing when we switch from increasing parameters through  $P^{leaf}$  to  $P^{ens}$ . Finally, we note that the interpolation threshold  $p = n$  plays a special role only on the  $P^{leaf}$  axis where it determines maximal depth, while parameters on the  $P^{ens}$  axis can be increased indefinitely – further suggesting that parameter counts alone are not always meaningful<sup>2</sup>.
2.2 Understanding double descent in gradient boosting
Another experiment is considered in Appendix S5 of [BHMM19] seeking to provide evidence for the emergence of double descent in gradient boosting. Recall that in gradient boosting, new base-learners (trees) are trained sequentially, accounting for current residuals by performing multiple boosting rounds which improve upon predictions of previous trees. In their experiments, [BHMM19] use trees with 10 leaves as base learners and a high learning rate of  $\gamma = 0.85$  to encourage quick interpolation. The raw number of parameters is controlled by first increasing the number of boosting rounds  $P^{boost}$  until the squared training error reaches approximately zero, after which  $P^{boost}$  is fixed and ensembling of  $P^{ens}$  independent models is used to further increase the raw parameter count.
In Fig. 4, we first replicate the original experiment and then again provide experiments varying each of  $P^{boost}$  and  $P^{ens}$  separately. Our findings parallel those above for trees: for a fixed number of ensemble members  $P^{ens}$ , test error has a U- or L-shape in the number of boosting rounds  $P^{boost}$  and an L-shape in  $P^{ens}$  for fixed boosting rounds  $P^{boost}$ . As a consequence, a double descent shape occurs only when and where we switch from one method of increasing complexity to another.
3 Deep dive: Understanding double descent in linear regression
We are now ready to consider Fig. 2 of [BHMM19], which provides experiments demonstrating double descent in the case of linear regression. Recall that linear regression with  $\mathbf{y} \in \mathbb{R}^n$  and  $\mathbf{X} \in \mathbb{R}^{n \times d}$  estimates the coefficients in a model  $\mathbf{y} = \mathbf{X}\beta$ , thus the number of raw model parameters equals the number of input dimensions (i.e. the dimension  $d$  of the regression coefficient vector  $\beta$ ) by design. Therefore, in order to flexibly control the number of model parameters, [BHMM19] apply basis expansions using random Fourier features (RFF). Specifically, given input  $\mathbf{x} \in \mathbb{R}^d$ , the number of raw model parameters  $P^\phi$  is controlled by randomly generating features  $\phi_p(\mathbf{x}) = \mathrm{Re}(\exp^{\sqrt{-1}\mathbf{v}_p^T\mathbf{x}})$  for all  $p \leq P^\phi$ , where each  $\mathbf{v}_p \stackrel{\mathrm{id}}{\sim} \mathcal{N}(\mathbf{0}, \frac{1}{5^2} \cdot \mathbf{I}_d)$ . For any given number of features  $P^\phi$ , these are stacked to give a  $n \times P^\phi$  dimensional random design matrix  $\Phi$ , which is then used to solve the regression problem  $\mathbf{y} = \Phi\beta$  by least squares. For  $P^\phi \leq n$ , this has a unique solution  $(\hat{\beta} = (\Phi^T\Phi)^{-1}\Phi^T\mathbf{y})$  while for  $P^\phi > n$  the problem becomes underdetermined (i.e. there are infinite solutions) which is why [BHMM19] rely on a specific choice: the min-norm solution  $(\hat{\beta} = \Phi^T(\Phi\Phi^T)^{-1}\mathbf{y})$ .
Unlike the experiments discussed in Sec. 2, there appears to be only one obvious mechanism for increasing raw parameters in this case study. Instead, as we show in Sec. 3.1, the change in the used solution at  $P^{\phi} = n$  turns out to be the crucial factor here: we find that the min-norm solution leads to implicit unsupervised dimensionality reduction, resulting in two distinct mechanisms for increasing the total parameter count in linear regression. Then, we again demonstrate empirically in Sec. 3.2 that each individual mechanism is indeed associated with a standard generalization curve, such that the combined generalization curve exhibits double descent only because they are applied in succession.
${}^{2}$  A related point is raised in [BM21],who notice that a double descent phenomenon in random forests appears in [BHMM19] only because the total number of leaves is placed on the x-axis, while the true complexity of a forest is actually better characterized by the average number of leaves in its trees according to results in learning theory. However,this complexity measure clearly does not change once  ${P}^{\text{leaf }}$  is fixed in the original experiment.
Figure 5: Decomposing double descent for RFF Regression. Double descent reproduced from [BHMM19] (left) can be decomposed into the standard U-curve of ordinary linear regression with  $P^{PC}$  features (center) and decreasing error achieved by a fixed capacity model with basis improving in  $P^{ex}$  (right).
3.1 Understanding the connections between min-norm solutions and dimensionality reduction
In this section, we show that, while the min-norm solution finds coefficients  $\hat{\beta}$  of raw dimension  $P^{\phi}$ , only  $n$  of its dimensions are well-determined - i.e. the true parameter count is not actually increasing in  $P^{\phi}$  once  $P^{\phi} > n$ . Conceptually, this is because min-norm solutions project  $\mathbf{y}$  onto the row-space of  $\Phi$ , which is  $n$ -dimensional as  $\text{rank}(\Phi) = \min(P^{\phi}, n)$  (when  $\Phi$  has full rank). To make the consequence of this more explicit, we can show that the min-norm solution (which has  $P^{\phi}$  raw parameters) can always be represented by using a  $n$ -dimensional coefficient vector applied to a  $n$ -dimensional basis of  $\Phi$ . In fact, as we formalize in Proposition 1, this becomes most salient when noting that applying the min-norm solution to  $\Phi$  is exactly equivalent to a learning algorithm that (i) first constructs a  $n$ -dimensional basis  $\mathbf{B}_{SVD}$  from the  $n$  right singular vectors of the input matrix computed using the singular value decomposition (SVD) in an unsupervised pre-processing step and (ii) then applies standard (fully determined) least squares using the discovered  $n$ -dimensional basis<sup>3</sup>.
Proposition 1. [Min-norm least squares as dimensionality reduction.] For a full rank matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  with  $n < d$  and a vector of targets  $\mathbf{y} \in \mathbb{R}^n$ , the min-norm least squares solution  $\hat{\beta}^{MN} = \{\min_{\boldsymbol{\beta}} ||\boldsymbol{\beta}||_2^2 : \mathbf{X}\boldsymbol{\beta} = \mathbf{y}\}$  and the least squares solution  $\hat{\beta}^{SVD} = \{\beta : \mathbf{B}\boldsymbol{\beta} = \mathbf{y}\}$  using the matrix of basis vectors  $\mathbf{B} \in \mathbb{R}^{n \times n}$ , constructed using the first  $n$  right singular vectors of  $X$ , are equivalent; i.e.  $\mathbf{x}^T\hat{\beta}^{MN} = \mathbf{b}^T\hat{\beta}^{SVD}$  for all  $\mathbf{x} \in \mathbb{R}^d$  and corresponding basis representation  $\mathbf{b} \equiv \mathbf{b}(\mathbf{x})$ . Proof. Please refer to Appendix B.2.
Then what is really causing the second descent if not an increasing number of fitted dimensions? While the addition of feature dimensions does correspond to an increase in fitted model parameters while  $P^{\phi} < n$ , the performance gains in the  $P^{\phi} > n$  regime are better explained as a linear model of fixed size  $n$  being fit to an increasingly rich basis constructed in an unsupervised step. To disentangle the two mechanisms further, we note that the procedure described above, when applied to a centered design matrix<sup>4</sup> is a special case of principal component (PC) regression [Jol82], where we select all empirical principal components to form a complete basis of  $\Phi$ . The more general approach would instead consist of selecting the top  $P^{PC}$  principal components and fitting a linear model to that basis. Varying  $P^{PC}$ , the number of used principal components, is thus actually the first mechanism by which the raw parameter count can be altered; this controls the number of parameters being fit in the supervised step. The second, less obvious, mechanism is then the number of excess features  $P^{ex} = P^{\phi} - P^{PC}$ ; this is the number of raw dimensions that only contribute to the creation of a richer basis, which is learned in an unsupervised manner<sup>5</sup>.
Based on this, we can now provide a new explanation for the emergence of double descent in this case study: due to the use of the min-norm solution, the two uncovered mechanisms are implicitly entangled
This application of the fundamental connection between min-norm solutions and the singular value decomposition (see e.g. [GVL13, Ch. 5.7]) reinforces previous works which have noted related links between the min-norm solution and dimensionality reduction [RD98, KL16].
4Centering reduces the rank of the input matrix by 1 and thus requires appending an intercept to the PC design matrix. Without centering, the procedure is using the so-called uncentered PCs [CJ09] instead. We use the centered version with intercept in our experiments, but obtained identical results when using uncentered PCs.
The implicit inductive bias encoded in this step essentially consists of constructing and choosing the top-  $P^{PC}$  features that capture the directions of maximum variation in the data. Within this inductive bias, the role of  $P^{ex}$  appears to be that - as more excess features are added - the variation captured by each of the top- $P^{PC}$  PCs is likely to increase. Using the directions of maximum variation is certainly not guaranteed to be optimal [Jol82], but it tends to be an effective inductive bias in practice as noted by Tukey [Tuk77] who suggested that high variance components are likely to be more important for prediction unless nature is "downright mean".
(a) Shifting the peak. Transitioning from  $P^{PC}$  to  $P^{ex}$  at different values of  $P^{PC}$ .
(b) Multiple descent. Generalization curves with arbitrarily many peaks and locations (including peaks at  $P^{\phi} > n$ ) can be created by switching between increasing parameters through  $P^{PC}$  and  $P^{ex}$  multiple times.
Figure 6: Disentangling double descent from the interpolation threshold. The location of the peak(s) in RFF regression generalization error is not inherently linked to the point where  $P^{\phi} = n$ . Instead, changes in the mechanism for parameter increase determine the appearance of peaks.
through  $P^{\phi} = P^{PC} + P^{ex}$ ,  $P^{PC} = \min(n, P^{\phi})$  and  $P^{ex} = \max(0, P^{\phi} - n)$ . Thus, we have indeed arrived back at a setup that parallels the previous two experiments: when  $P^{\phi} \leq n$ ,  $P^{PC}$  increases monotonically while  $P^{ex} = 0$  is constant, while when  $P^{\phi} > n$  we have constant  $P^{PC} = n$  but  $P^{ex}$  increases monotonically. Below, we can now test empirically whether studying the two mechanisms separately indeed leads us back to standard convex curves as before. In particular, we also show that – while a transition between the mechanisms increasing  $P^{PC}$  and  $P^{ex}$  naturally happens at  $P^{\phi} = n$  in the original experiments – it is possible to transition elsewhere across the implied complexity axes, creating other thresholds, and to thus disentangle the double descent phenomenon from  $n$ .
3.2 Empirical resolutions to double descent in RFF regression