Title: A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Double Descent

Statistical Machine Learning

Interpolation Regime

Effective Parameters
Abstract: Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a _U-shaped curve_ reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count $p$ grows past sample size $n$  -- a  phenomenon dubbed  _double descent_. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include _linear regression, trees, and boosting_. In this work, we take a closer look at the evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of  double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to _what is being plotted_ on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple, distinct complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and _only_) when and where the transition between these underlying axes occurs, and that its location is thus _not_ inherently tied to the interpolation threshold $p=n$. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as _smoothers_ and propose a generalized measure for the _effective_ number of parameters they use _on unseen examples_, using which we find that their apparent double descent curves do indeed fold back into more traditional convex shapes -- providing a resolution to the ostensible tension between double descent and traditional statistical intuition.

A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

Abstract

Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent . While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting . In this work, we take a closer look at the evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple, distinct complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only ) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p = n . We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples , using which we find that their apparent double descent curves do indeed fold back into more traditional convex shapes - providing a resolution to the ostensible tension between double descent and traditional statistical intuition.

1 Introduction

Historically, throughout the statistical learning literature, the relationship between model complexity and prediction error has been well-understood as a careful balancing act between underfitting , associated with models of high bias, and overfitting , associated with high model variability. This implied tradeoff, with optimal performance achieved between extremes, gives rise to a U-shaped curve, illustrated in the left panel of Fig. 1. It has been a fundamental tenet of learning from data, omnipresent in introductions to statistical learning [HT90, Vap95, HTF09], and is also practically reflected in numerous classical model selection criteria that explicitly trade off training error with model complexity [Mal73, Aka74, Sch78]. Importantly, much of the intuition relating to this Ushaped curve was originally developed in the context of the earlier statistics literature (see e.g. the historical note in [Nea19]), which focussed on conceptually simple learning methods such as linear regression, splines or nearest neighbor methods [WW75, HT90, GBD92] and their expected in-sample prediction error, which fixes inputs and resamples noisy outcomes [Mal73, HT90, RT19].

U-curve

L-curve

Error

Complexity dim 1

Complexity

dim 2

Double descent

'unfolding'

Complexity

dim 2

The modern machine learning (ML) literature, conversely, focuses on far more flexible methods with relatively huge parameter counts and considers their generalization to unseen inputs [GBC16, Mur22]. Asimilar U-shaped curve was long accepted to also govern the complexity-generalization relationship of such methods [GBD92, Vap95, HTF09]- until highly overparametrized models, e.g. neural networks, were recently found to achieve near-zero training error and excellent test set performance [NTS14, BLLT20, Bel21]. In this light, the seminal paper of Belkin et al. (2019) [BHMM19] sparked a new line of research by arguing for a need to extend on the apparent limitations of classic understanding to account for a double descent in prediction performance as the total number of model parameters (and thus - presumably - model complexity) grows. This is illustrated in the right panel of Fig. 1. Intuitively, it is argued that while the traditional U-curve is appropriate for the regime in which the number of total model parameters p is smaller than the number of instances n , it no longer holds in the modern, zero train-error, interpolation regime where p>n - here, test error experiences a second descent. Further, it was demonstrated that this modern double descent view of model complexity applies not only in deep learning where it was first observed [BO96, NMB + 18, SGd + 18, ASS20], but also ubiquitously appears across many non-deep learning methods such as trees, boosting and even linear regression [BHMM19].

Contributions. In this work, we investigate whether the double descent behavior observed in recent empirical studies of such non-deep MLmethods truly disagrees with the traditional notion of a Ushaped tradeoff between model complexity and prediction error. In two parts, we argue that once careful consideration is given to what is being plotted on the axes of these double descent plots, the originally counter-intuitive peaking behavior can be comprehensively explained under existing paradigms:

Part 1: Revisiting existing experimental evidence. We show that in the experimental evidence for non-deep double descent - using trees, boosting, and linear regressions - there is implicitly more than one complexity axis along which the parameter count grows. Conceptually, as illustrated in Fig. 1, we demonstrate that this empirical evidence for double descent can thus be comprehensively explained as a consequence of an implicit unfolding of a 3D plot with two orthogonal complexity axes (that both individually display a classical convex curve) into a single 2D-curve. We also highlight that the location of the second descent is thus not inherently tied to the interpolation threshold. While this is straightforward to show for the tree- and boosting examples (Sec. 2), deconstructing the underlying axes in the linear regression example is non-trivial (and involves understanding the connections between minnorm solutions and unsupervised dimensionality reduction ). Our analysis in this case (Sec. 3) could thus be of independent interest as a simple new interpretation of double descent in linear regression.

Part 2: Rethinking parameter counting through a classical statistics lens. We then note that all methods considered in Part 1 can be interpreted as smoothers (Sec. 4), which are usually compared in terms of a measure of the effective (instead of raw) number of parameters they use when issuing predictions [HT90]. As existing measures were derived with in-sample prediction in mind, we propose a generalized effective parameter measure p 0 ˆ s that allows to consider arbitrary sets of inputs I 0 . Using p 0 ˆ s to measure complexity, we then indeed discover that the apparent double descent curves fold back into more traditional U-shapes - because p 0 ˆ s is not actually increasing in the interpolation regime. Further, we find that, in the interpolation regime, trained models tend to use a different number of effective parameters when issuing predictions on unseen test inputs than on previously observed training inputs. We also note that, while such interpolating models can generalize well to unseen inputs, overparametrization cannot improve their performance in terms of the in-sample prediction error originally of interest in statistics - providing a new reason for the historical absence of double descent curves. Finally, we discuss practical implications for e.g. model comparison.

200

×

1

×

1

×

50

0.1

0.4

0.6

0.8

Squared test error

Double Descent in Trees

0

100

200

300

400

500

P

leaf

Error by

P

leaf

,

P

ens

fixed

P

ens

= 1

P

ens

= 5

P

ens

= 10

P

ens

= 50

0

20

40

60

80

100

P

ens

Error by

P

ens

,

P

leaf

fixed

P

leaf

= 500

P

leaf

= 100

P

leaf

= 50

P

leaf

= 20

Part 1: Revisiting the evidence for double descent in non-deep ML models

Experimental setup. We center our study around the non-neural experiments in [BHMM19] as it is the seminal paper on double descent and provides the broadest account of non-deep learning methods that exhibit double descent. Through multiple empirical studies, [BHMM19] demonstrate that double descent arises in trees, boosting and linear regression. Below, we re-analyze these experiments and highlight that in each study, as we transition from the classical U-shaped regime into the subsequent second descent regime, something else implicitly changes in the model or training definition, fundamentally changing the class of models under consideration exactly at the transition threshold between the observed regimes - which is precisely the cause of the second descent phenomenon. In Sec. 2, we begin by investigating the tree and boosting experiments, where this is straightforward to show. In Sec. 3, we then investigate the linear regression example, where decomposing the underlying mechanisms is non-trivial. Throughout, we closely follow [BHMM19]'s experimental setup: they use standard benchmark datasets and train all ML methods by minimizing the squared loss 1 . Similarly to their work, we focus on results using MNIST (with n train = 10000 ) in the main text. We present additional results, including other datasets, and further discussion of the experimental setup in Appendix E.

2 Warm-up: Observations of double descent in trees and boosting

2.1 Understanding double descent in trees

In the left panel of Fig. 2, we replicate the experiment in [BHMM19]'s Fig. 4, demonstrating double descent in trees. In their experiment, the number of model parameters is initially controlled through the maximum allowed number of terminal leaf nodes P leaf . However, P leaf for a single tree cannot be increased past n (which is when every leaf contains only one instance), and often max( P leaf ) <n whenever larger leaves are already pure. Therefore, when P leaf reaches its maximum, in order to further increase the raw number of parameters, it is necessary to change how further parameters are added to the model. [BHMM19] thus transition to showing how test error evolves as one averages over an increasing number P ens of different trees grown to full depth, where each tree will generally be distinct due to the randomness in features considered for each split. As one switches between plotting increasing P leaf and P ens on the x-axis, one is thus conceptually no longer increasing the number of parameters within the same model class : in fact, when P ens > 1 one is no longer actually considering a tree, but instead an ensemble of trees (i.e. a random forest [Bre01] without bootstrapping).

In Fig. 2, we illustrate this empirically: in the center plot we show that, on the one hand, for fixed P ens , error exhibits a classical convex U- (or L-)shape in tree-depth P leaf . On the other hand, for fixed P leaf in the right plot, error also exhibits an L-shape in the number of trees P ens , i.e. a convex shape without any ascent - which is in line with the known empirical observation that adding trees to a random forest generally does not hurt [HTF09, Ch. 15.3.4]. Thus, only by transitioning from increasing P leaf (with P ens = 1 ) to increasing P ens (with P leaf = n ) - i.e. by connecting the two solid curves across the middle and the right plot - do we obtain the double

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Squared error

P

leaf

= 500

P

leaf

= 200

P

leaf

= 100

P

leaf

= 50

10

×

1

100

×

1

×

1

×

10

0.1

0.2

0.4

0.5

Squared test error

Double Descent in Boosting

0

50

100

150

200

P

boost

Error by

P

boost

,

P

ens

fixed

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 20

5

10

15

20

P

ens

Error by

P

ens

,

P

boost

fixed

P

boost

= 200

P

boost

= 50

P

boost

= 25

P

boost

= 10

descent curve in the left plot of Fig. 2. In Fig. 3, we show that we could therefore arbitrarily move or even remove the first peak by changing when we switch from increasing parameters through P leaf to P ens . Finally, we note that the interpolation threshold p = n plays a special role only on the P leaf axis where it determines maximal depth, while parameters on the P ens axis can be increased indefinitely - further suggesting that parameter counts alone are not always meaningful 2 .

2.2 Understanding double descent in gradient boosting

Another experiment is considered in Appendix S5 of [BHMM19] seeking to provide evidence for the emergence of double descent in gradient boosting. Recall that in gradient boosting, new baselearners (trees) are trained sequentially , accounting for current residuals by performing multiple boosting rounds which improve upon predictions of previous trees. In their experiments, [BHMM19] use trees with 10 leaves as base learners and a high learning rate of γ =0 . 85 to encourage quick interpolation. The raw number of parameters is controlled by first increasing the number of boosting rounds P boost until the squared training error reaches approximately zero, after which P boost is fixed and ensembling of P ens independent models is used to further increase the raw parameter count.

In Fig. 4, we first replicate the original experiment and then again provide experiments varying each of P boost and P ens separately. Our findings parallel those above for trees: for a fixed number of ensemble members P ens , test error has a U- or L-shape in the number of boosting rounds P boost and an L-shape in P ens for fixed boosting rounds P boost . As a consequence, a double descent shape occurs only when and where we switch from one method of increasing complexity to another.

3 Deep dive: Understanding double descent in linear regression

We are now ready to consider Fig. 2 of [BHMM19], which provides experiments demonstrating double descent in the case of linear regression. Recall that linear regression with y ∈ R n and X ∈ R n × d estimates the coefficients in a model y = X β , thus the number of raw model parameters equals the number of input dimensions (i.e. the dimension d of the regression coefficient vector β ) by design. Therefore, in order to flexibly control the number of model parameters, [BHMM19] apply basis expansions using random Fourier features (RFF). Specifically, given input x ∈ R d , the number of raw model parameters P ϕ is controlled by randomly generating features ϕ p ( x ) = Re (exp √ -1 v T p x ) for all p ≤ P ϕ , where each v p iid ∼ N ( 0 , 1 5 2 · I d ) . For any given number of features P ϕ , these are stacked to give a n × P ϕ dimensional random design matrix Φ , which is then used to solve the regression problem y = Φ β by least squares. For P ϕ ≤ n , this has a unique solution ( ˆ β = ( Φ T Φ ) -1 Φ T y ) while for P ϕ > n the problem becomes underdetermined (i.e. there are infinite solutions) which is why [BHMM19] rely on a specific choice: the min-norm solution ( ˆ β = Φ T ( ΦΦ T ) -1 y ).

Unlike the experiments discussed in Sec. 2, there appears to be only one obvious mechanism for increasing raw parameters in this case study. Instead, as we show in Sec. 3.1, the change in the used solution at P ϕ = n turns out to be the crucial factor here: we find that the min-norm solution leads to implicit unsupervised dimensionality reduction , resulting in two distinct mechanisms for increasing the total parameter count in linear regression. Then, we again demonstrate empirically in Sec. 3.2 that each individual mechanism is indeed associated with a standard generalization curve, such that the combined generalization curve exhibits double descent only because they are applied in succession.

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

PC

ex

0.1

3.3

81.5

1998.2

Squared test error

Double Descent in Regression

0

2000

4000

6000

8000

10000

P

PC

Error by

P

PC

,

P

ex

fixed

P

ex

= 0

P

ex

= 2500

P

ex

= 10000

P

ex

= 50000

0

10000

20000

30000

40000

50000

P

ex

Error by

P

ex

,

P

PC

fixed

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

P

+

P

3.1 Understanding the connections between min-norm solutions and dimensionality reduction

In this section, we show that, while the min-norm solution finds coefficients ˆ β of raw dimension P ϕ , only n of its dimensions are well-determined - i.e. the true parameter count is not actually increasing in P ϕ once P ϕ > n . Conceptually, this is because min-norm solutions project y onto the row-space of Φ , which is n -dimensional as rank ( Φ ) = min( P ϕ , n ) (when Φ has full rank). To make the consequence of this more explicit, we can show that the min-norm solution (which has P ϕ raw parameters) can always be represented by using a n -dimensional coefficient vector applied to a n -dimensional basis of Φ . In fact, as we formalize in Proposition 1, this becomes most salient when noting that applying the min-norm solution to Φ is exactly equivalent to a learning algorithm that (i) first constructs a n -dimensional basis B SV D from the n right singular vectors of the input matrix computed using the singular value decomposition (SVD) in an unsupervised pre-processing step and (ii) then applies standard (fully determined) least squares using the discovered n -dimensional basis 3 .

Proposition 1. [Min-norm least squares as dimensionality reduction.] For a full rank matrix X ∈ R n × d with n < d and a vector of targets y ∈ R n , the min-norm least squares solution ˆ β MN = { min β || β || 2 2 : X β = y } and the least squares solution ˆ β SVD = { β : B β = y } using the matrix of basis vectors B ∈ R n × n , constructed using the first n right singular vectors of X, are equivalent; i.e. x T ˆ β MN = b T ˆ β SVD for all x ∈ R d and corresponding basis representation b ≡ b ( x ) . Proof. Please refer to Appendix B.2.

Then what is really causing the second descent if not an increasing number of fitted dimensions? While the addition of feature dimensions does correspond to an increase in fitted model parameters while P ϕ < n , the performance gains in the P ϕ > n regime are better explained as a linear model of fixed size n being fit to an increasingly rich basis constructed in an unsupervised step . To disentangle the two mechanisms further, we note that the procedure described above, when applied to a centered design matrix 4 is a special case of principal component (PC) regression [Jol82], where we select all empirical principal components to form a complete basis of Φ . The more general approach would instead consist of selecting the top P PC principal components and fitting a linear model to that basis. Varying P PC , the number of used principal components, is thus actually the first mechanism by which the raw parameter count can be altered; this controls the number of parameters being fit in the supervised step. The second, less obvious, mechanism is then the number of excess features P ex = P ϕ -P PC ; this is the number of raw dimensions that only contribute to the creation of a richer basis, which is learned in an unsupervised manner 5 .

Based on this, we can now provide a new explanation for the emergence of double descent in this case study: due to the use of the min-norm solution, the two uncovered mechanisms are implicitly entangled

0

20000

40000

60000

PC

ex

0.1

3.3

81.5

1998.2

Squared error

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

0

20000

40000

60000

φ

PC

ex

0

.

1

0

.

3

0

.

5

Squared Error

P

ex

= 0 fixed,

P

PC

↑

P

PC

= 5000 fixed,

P

ex

↑

P

ex

= 2500 fixed,

P

PC

↑

P

PC

= 7500 fixed,

P

ex

↑

P

ex

= 5000 fixed,

P

PC

↑

P

PC

= 10000 fixed,

P

ex

↑

P

+

P

(a) Shifting the peak. Transitioning from P PC to P ex at different values of P PC .

P

=

P

+

P

through P ϕ = P PC + P ex , P PC =min( n, P ϕ ) and P ex =max(0 , P ϕ -n ) . Thus, we have indeed arrived back at a setup that parallels the previous two experiments: when P ϕ ≤ n , P PC increases monotonically while P ex =0 is constant, while when P ϕ > n we have constant P PC = n but P ex increases monotonically. Below, we can now test empirically whether studying the two mechanisms separately indeed leads us back to standard convex curves as before. In particular, we also show that while a transition between the mechanisms increasing P PC and P ex naturally happens at P ϕ = n in the original experiments - it is possible to transition elsewhere across the implied complexity axes, creating other thresholds, and to thus disentangle the double descent phenomenon from n .

3.2 Empirical resolutions to double descent in RFF regression

Mirroring the analyses in Sec. 2, we now investigate the effects of P PC and P ex in Fig. 5. In the left plot, we once more replicate [BHMM19]'s original experiment, and observe the same apparent trend that double descent emerges as we increase the number of raw parameters P ϕ = P PC + P ex (where the min-norm solution needs to be applied once P ϕ = n ). We then proceed to analyze the effects of varying P PC and P ex separately (while holding the other fixed). As before, we observe in the center plot that varying P PC (determining the actual number of parameters being fit in the regression) for different levels of excess features indeed gives rise to the traditional U-shaped generalization curve. Conversely, in the right plot, we observe that increasing P ex for a fixed number of P PC results in an L-shaped generalization curve - indeed providing evidence that the effect of increasing the number of raw parameters past n in the original experiment can be more accurately explained as a gradual improvement in the quality of a basis to which a fixed capacity model is being fit.

As before, note that if we connect the solid lines in the center and right plots we recover exactly the double descent curve shown in the left plot of Fig. 5. Alternatively, as we demonstrate in Fig. 6(a), fixing P PC at other values and then starting to increase the total number of parameters through P ex allows us to move or remove the first peak arbitrarily . In Fig. 6(b), we demonstrate that one could even create multiple peaks 6 by switching between parameter-increasing mechanisms more than once . This highlights that, while the transition from parameter increase through P PC to P ex naturally occurs at P ϕ = n due to the use of the min-norm solution, the second descent is not actually caused by the interpolation threshold P ϕ = n itself - but rather is due to the implicit change in model at exactly this point. Indeed, comparing the generalization curves in Fig. 6 with their train-error trajectories which we plot in Appendix E.2, it becomes clear that such a second descent can also occur in models that have not yet and will never achieve interpolation of the training data.

4 Part 2: Rethinking parameter counting through a classical statistics lens

Thus far, we have highlighted that 'not all model parameters are created equal' - i.e. the intuitive notion that not all ways of increasing the number of raw parameters in an ML method have the same

effect. However, as we saw in the linear regression example, it is not always trivial to deconstruct the underlying mechanisms driving performance. Therefore, instead of having to reason about implicit complexity axes on a case-by-case basis for different models, hyperparameters, or inductive biases, we would rather be able to quantify the effect of these factors objectively. In what follows, we highlight that all previously considered methods can be interpreted as smoothers (in the classical statistics sense [HT90]). By making this connection in Sec. 4.1, we can exploit the properties of this class of models providing us with measures of their effective number of parameters. After adapting this concept to our setting (Sec. 4.2), we are finally able to re-calibrate the complexity axis of the original double descent experiments, finding that they do indeed fold back into more traditional U-shapes (Sec. 4.3).

4.1 Connections to smoothers

Smoothers are a class of supervised learning methods that summarize the relationship between outcomes Y ∈Y⊂ R k and inputs X ∈X ⊂ R d by 'smoothing' over values of Y observed in training. More formally, let k =1 w.l.o.g., and denote by D train = { ( y i , x i ) } n i =1 the training realizations of ( X,Y ) ∈X×Y and by y train =( y 1 ,. . ., y n ) T the n × 1 vector of training outcomes with respective training indices I train = { 1 , . . . , n } . Then, for any admissible input x 0 ∈X , a smoother issues predictions

where ˆ s ( x 0 ) = (ˆ s 1 ( x 0 ) , . . . , ˆ s n ( x 0 )) is a 1 × n vector containing smoother weights for input x 0 . A smoother is linear if ˆ s ( · ) does not depend on y train. The most well-known examples of smoothers rely on weighted (moving-) averages, which includes k-nearest neighbor (kNN) methods and kernel smoothers as special cases. (Local) linear regression and basis-expanded linear regressions, including splines, are other popular examples of linear smoothers (see e.g. [HT90, Ch. 2-3]).

In Appendix C, we show that all methods studied in Part 1 can be interpreted as smoothers, and derive ˆ s ( · ) for each method. To provide some intuition, note that linear regression is a simple textbook example of a linear smoother [HT90], where ˆ s ( · ) is constructed from the so-called projection (or hat) matrix [Eub84]. Further, trees - which issue predictions by averaging training outcomes within leaves - are sometimes interpreted as adaptive nearest neighbor methods [HTF09, Ch. 15.4.3] with learned (i.e. non-linear) weights, and as a corollary, boosted trees and sums of either admit similar interpretations.

4.2 A generalized measure of the effective number of parameters used by a smoother

The effective number of parameters p e of a smoother was introduced to provide a measure of model complexity which can account for a broad class of models as well as different levels of model regularization (see e.g. [HT90, Ch. 3.5], [HTF09, Ch. 7.6]). This generalized the approach of simply counting raw parameters - which is not always possible or appropriate - to measure model complexity. This concept is calibrated towards linear regression so that, as we might desire, effective and raw parameter numbers are equal in the case of ordinary linear regression with p < n . In this section, we adapt the variance based effective parameter definition discussed in [HT90, Ch. 3.5]. Because for fixed ˆ s ( · ) and outcomes generated with homoskedastic variance σ 2 we have V ar ( ˆ f ( x 0 )) = || ˆ s ( x 0 ) || 2 σ 2 , this definition uses that 1 n ∑ i ∈I train V ar ( ˆ f ( x i )) = σ 2 n ∑ i ∈I train || ˆ s ( x i ) || 2 = σ 2 n p for linear regression: one can define p e = ∑ i ∈I train || ˆ s ( x i ) || 2 and thus have p e = p for ordinary linear regression with n < p . As we discuss further in Appendix D, other variations of such effective parameter count definitions can also be found in the literature. However, this choice is particularly appropriate for our purposes as it has the unique characteristic that it can easily be adapted to arbitrary input points - a key distinction we will motivate next.

Historically, the smoothing literature has primarily focused on prediction in the classical fixed design setup where expected in-sample prediction error on the training inputs x i , with only newly sampled targets y ′ i , was considered the main quantity of interest [HT90, RT19]. The modern ML literature, on the other hand, largely focuses its evaluations on out-of-sample prediction error in which we are interested in model performance, or generalization , on both unseen targets and unseen inputs (see e.g. [GBC16, Ch. 5.2]; [Mur22, Ch. 4.1]). To make effective parameters fit for modern purposes, it is therefore necessary to adapt p e to measure the level of smoothing applied conditional on a given input , thus distinguishing between training and testing inputs. As || ˆ s ( x 0 ) || 2 can be computed for any input x 0 , this is straightforward and can be done by replacing I train in the definition of p e by any other set of inputs indexed by I 0 . Note that the scale of p e would then depend on |I 0 | due to the summation,

200

×

1

×

1

×

50

P

leaf

P

ens

1

10

10

2

p

0

ˆ

s

(log scale)

Trees

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

10

1

10

2

10

3

Boosting

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

PC

ex

10

2

10

4

10

6

Linear Regression

I

train

I

test

×

×

P

+

P

while it should actually depend on the number of training examples n (previously implicitly captured through |I train | ) across which the smoothing occurs - thus we also need to recalibrate our definition by n/ |I 0 | . As presented in Definition 1, we can then measure the generalized effective number of parameters p 0 ˆ s used by a smoother when issuing predictions for any set of inputs I 0 .

Definition 1 (Generalized Effective Number of Parameters) . For a set of inputs { x 0 j } j ∈I 0 , define the Generalized Effective Number of Parameters p 0 ˆ s used by a smoother with weights ˆ s ( · ) as

Note that for the training inputs we recover the original quantity exactly ( p e = p train ˆ s ). Further, for moving-average smoothers with ∑ n i =1 ˆ s i ( x 0 ) = 1 and 0 ≤ ˆ s i ( x 0 ) ≤ 1 for all i , it holds that 1 ≤ p 0 ˆ s ≤ n . Finally, for kNN estimators we have p 0 ˆ s = n k , so that for smoothers satisfying ∑ n i =1 ˆ s i ( x 0 ) = 1 and 0 ≤ ˆ s i ( x 0 ) ≤ 1 for all i , the quantity ˜ k 0 ˆ s = n p 0 ˆ s also admits an interesting interpretation as measuring the effective number of nearest neighbors .

4.3 Back to U: Measuring effective parameters folds apparent double descent curves

Using Definition 1, we can now measure the effective number of parameters to re-examine the examples of double descent described in Sections 2 & 3. We plot the results of the 0-vs-all sub-problem in this section as the 10 one-vs-all models in the full experiment are each individually endowed with effective parameter counts p 0 ˆ s . Fig. 7 presents results plotting this quantity measured on the training inputs ( p train ˆ s ) and testing inputs ( p test ˆ s ) against the original, composite parameter axes of [BHMM19], Fig. 8 considers the behavior of p test ˆ s for the two distinct mechanisms of parameter increase separately, and in Fig. 9 we plot test error against p test ˆ s , finally replacing raw with effective parameter axes.

By examining the behavior of p 0 ˆ s in Fig. 7 and Fig. 8, several interesting insights emerge. First, we observe that these results complement the intuition developed in Part 1: In all cases, measures of the effective number of parameters never increase after the threshold where the mechanism for increasing the raw parameter count changes. Second, the distinction of measuring effective number of parameters used on unseen inputs , i.e. using p test ˆ s instead of p train ˆ s , indeed better tracks the double descent phenomenon which itself emerges in generalization error (i.e. is also estimated on unseen inputs). This is best illustrated in Fig. 7 in the linear regression example where p train ˆ s = n is constant once P ϕ ≥ n - as is expected: simple matrix algebra reveals that ˆ s ( x i ) = e i (with e i the i th indicator vector) for any training example x i once P ϕ ≥ n . Intuitively, no smoothing across training labels occurs on the training inputs after the interpolation threshold as each input simply predicts its own label (note that this would also be the case for regression trees if each training example fell into its own leaf; this does not occur in these experiments as leaves are already pure before this point). As we

0

100

200

300

400

500

P

leaf

1

10

10

2

p

test

ˆ

s

(log scale)

Trees

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

0

50

100

150

200

P

boost

10

1

10

2

10

3

Boosting

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

0

2000

4000

6000

8000

10000

P

PC

10

2

10

4

10

6

Linear Regression

P

ex

= 0

P

ex

= 2500

P

ex

= 5000

P

ex

= 10000

P

ex

= 20000

P

ex

= 50000

1

10

1

10

2

p

test

ˆ

s

(log scale)

0

.

01

0

.

03

0

.

05

0

.

07

Squared test error

Trees

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

P

ens

= 50

P

ens

= 100

10

1

10

2

10

3

p

test

ˆ

s

(log scale)

Boosting

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

10

2

10

3

10

4

p

test

ˆ

s

(log scale)

Linear Regression

P

ex

= 0

P

ex

= 2500

P

ex

= 5000

P

ex

= 10000

P

ex

= 20000

P

ex

= 50000

make more precise in Appendix C.2, this observation leads us to an interesting impossibility result: the in-sample prediction error of interpolating models therefore cannot experience a second descent in the (raw) overparameterized regime. This provides a new reason for the historical absence of double descent shapes (in one of the primary settings in which the U-shaped tradeoff was originally motivated [HT90]), complementing the discussion on the lack of previous observations in [BHMM19]. Third, while p train ˆ s is therefore not useful to understand double descent in generalization error, we note that for unseen inputs different levels of smoothing across the training labels can occur even in the interpolation regime - and this, as quantified by p test ˆ s , does result in an informative complexity proxy.

This becomes obvious in Fig. 9, where we plot the test errors of [BHMM19]'s experimental setup against p test ˆ s (larger blue and red points: , ). For each such point after the interpolation threshold ( ), we also provide dotted contour lines ( ) representing the effect of reducing the parameters along the first complexity axis ( P leaf , P boost and P PC ), resulting in models that can no longer perfectly interpolate (e.g. for trees, we fix their number P ens and reduce their depth through P leaf ). Finally, we discover that once we control for the effective parameters of each model in this way, we consistently observe that each example indeed folds back into shapes that are best characterized as standard convex (U-shaped) curves - providing a resolution to the apparent tension between traditional statistical intuition and double descent! We also note an interesting phenomenon: increasing P ens and P ex appears to not only shift the complexity-generalization contours downwards (decreasing error for each value of P leaf , P boost and P PC ), but also shifts them to the left (decreasing the effective complexity implied by each value of P leaf , P boost and P PC , as also highlighted in Fig. 8) and flattens their ascent (making error less sensitive to interpolation). That is, models in (or close to) the interpolation regime discovered by [BHMM19] exhibit a very interesting behavior where setting P ex and P ens larger actually decreases the test error through a reduction in the effective number of parameters used.

5 Consolidation with Related Work

In this section, we consolidate our findings with the most relevant recent work on understanding double descent; we provide an additional broader review of the literature in Appendix A. A large portion of this literature has focused on modeling double descent in the raw number of parameters in linear regression, where these parameter counts are varied by employing random feature models [BHMM19, BHX20] or by varying the ratio of n to input feature dimension d [ASS20, BLLT20, DLM20, HMRT22]. This literature, typically studying the behavior and conditioning of feature covariance matrices, has produced precise theoretical analyses of double descent for particular models as the raw number of parameters is increased (e.g. [BLLT20, HMRT22]). Our insights in Sections 3 and 4 are complementary to this line of work, providing a new perspective in terms of (a) decomposing raw parameters into two separate complexity axes which jointly produce the double descent shape only due to the choice of (min-norm) solution and (b) the underlying effective number of parameters used by a model. Additionally, using theoretical insights from this line of work can make more precise the role of P ex in improving the quality of a learned basis: as we show in Appendix B.3, increasing P ex in our experiments indeed leads to the PC feature matrices being better conditioned.

Further, unrelated to the smoothing lens we use in Part 2 of this paper, different notions of effective parameters or complexity have appeared in other studies of double descent in specific

ML methods: [BM21] use Rademacher complexity (RC) in their study of random forests but find that RC cannot explain their generalization because forests do not have lower RC than individual trees (unlike the behavior of our p test ˆ s in this case). [MBW20] use [Mac91]'s Bayesian interpretation of effective parameters based on the Hessian of the training loss when studying neural networks. While this proxy was originally motivated in a linear regression setting, it cannot explain double descent in linear regression because, as discussed further in Appendix D, it does not decrease therein once p > n . Finally, [DLM20] compute effective ridge penalties implied by min-norm linear regression, and [DSYW20] consider minimum description length principles to measure complexity in ridge regression. Relative to this literature, our work differs not only in terms of the ML methods we can consider (in particular, we are uniquely able to explain the tree- and boosting experiments through our lens), but also in the insights we provide e.g. we distinctively propose to distinguish between effective parameters used on train- versus testexamples, which we showed to be crucial to explaining the double descent phenomenon in Sec. 4.3.

6 Conclusion and Discussion

Conclusion: A Resolution to the ostensible tension between non-deep double descent and statistical intuition. We demonstrated that existing experimental evidence for double descent in trees, boosting and linear regression does not contradict the traditional notion of a U-shaped complexitygeneralization curve: to the contrary, we showed that in all three cases, there are actually two independent underlying complexity axes that each exhibit a standard convex shape, and that the observed double descent phenomenon is a direct consequence of transitioning between these two distinct mechanisms of increasing the total number of model parameters. Furthermore, we highlighted that when we plot a measure of the effective, test-time, parameter count (instead of raw parameters) on their x-axes, the apparent double descent curves indeed fold back into more traditional U-shapes.

What about deep double descent? In this work, we intentionally limited ourselves to the study of non-deep double descent. Whether the approach pursued in this work could provide an alternative path to understanding double descent in the case of deep learning - arguably its most prominent setting - is thus a very natural next question. It may indeed be instructive to investigate whether there also exist multiple implicitly entangled complexity axes in neural networks, and whether this may help to explain double descent in that setting. In particular, one promising approach to bridging this gap could be to combine our insights of Sec. 3 with the known connections between random feature models and two-layer neural networks [SGT18], and stochastic gradient descent and minnorm solutions [GWB + 17]. We consider this a fruitful and non-trivial direction for future research.

What are the practical implications of these findings? On the one hand, with regards to the specific MLmethods under investigation, our empirical results in Sections 2 and 3 imply interesting trade-offs between the need for hyperparameter tuning and raw model size. All methods appear to have one hyperparameter axis to which error can be highly sensitive P leaf , P boost and P PC - while along the other axis, 'bigger is better' (or at least, not worse). In fact, it appears that the higher P ens or P ex , the less sensitive the model becomes to changes along the first axis. This may constitute anecdotal evidence that the respective first axis can be best understood as a train-time bias-reduction axis - it controls how well the training data can be fit (increasing parameters along this axis reduces underfitting - only when set to its maximum can interpolation be achieved). The second axis, conversely, appears to predominantly achieve variance-reduction at test-time : it decreases || ˆ s ( x 0 ) || , reducing the impact of noise by smoothing over more training examples when issuing predictions for unseen inputs.

On the other hand, our results in Sec. 4 suggest interesting new avenues for model selection more generally, by highlighting potential routes of redemption for classical criteria trading off in-sample performance and parameter counts (e.g. [Aka74, Mal73]) - which have been largely abandoned in ML in favor of selection strategies evaluating held-out prediction error [Ras18]. While criteria based on raw parameter counts may be outdated in the modern ML regime, selection criteria based on effective parameter counts p test ˆ s used on a test-set could provide an interesting new alternative, with the advantage of not requiring access to labels on held-out data, unlike error-based methods. In Appendix E.5, we provide anecdotal evidence that when choosing between models with different hyperparameter settings that all achieve zero training error , considering each model's p test ˆ s could be used to identify a good choice in terms of generalization performance: we illustrate this for the case of gradient boosting where we use p test ˆ s to choose additional hyperparameters. Investigating such approaches to model selection more extensively could be another promising avenue for future work.

References

[ACHL19] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

[Aka74] Hirotugu Akaike. A new look at the statistical model identification. IEEE transactions on automatic control , 19(6):716-723, 1974.

[AP20] Ben Adlam and Jeffrey Pennington. Understanding double descent requires a finegrained bias-variance decomposition. Advances in neural information processing systems , 33:11022-11032, 2020.

[ASS20] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks , 132:428-446, 2020.

[Bel21] Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021.

[BFSO84] Leo Breiman, Jerome Friedman, Charles J. Stone, and R.A. Olshen. Classification and Regression Trees . Chapman and Hall/CRC, 1984.

[BH89] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks , 2(1):53-58, 1989.

[BHMM19] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

[BHX20] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

[BLLT20] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):3006330070, 2020.

[BM21] Sebastian Buschjäger and Katharina Morik. There is no double-descent in random forests. arXiv preprint arXiv:2111.04409 , 2021.

[BMM18] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning , pages 541-549. PMLR, 2018.

[BO96] Siegfried Bös and Manfred Opper. Dynamics of training. Advances in Neural Information Processing Systems , 9, 1996.

[Bre01] Leo Breiman. Random forests. Machine learning , 45:5-32, 2001.

[CJ09] Jorge Cadima and Ian Jolliffe. On relationships between uncentred and column-centred principal component analysis. Pakistan Journal of Statistics , 25(4), 2009.

[CL21] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. The Journal of Machine Learning Research , 22(1):5721-5750, 2021.

[CMBK21] Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems , 34:8898-8912, 2021.

[DB22] Yehuda Dar and Richard G Baraniuk. Double double descent: on generalization errors in transfer learning between linear regression tasks. SIAM Journal on Mathematics of Data Science , 4(4):1447-1472, 2022.

[DFO20] Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning . Cambridge University Press, 2020.

[DLM20] Michal Derezinski, Feynman T Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Advances in neural information processing systems , 33:5152-5164, 2020.

[DMB21] Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the biasvariance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355 , 2021.

[DMLB20] Yehuda Dar, Paul Mayer, Lorenzo Luzi, and Richard Baraniuk. Subspace fitting meets regression: The effects of supervision and orthonormality constraints on double descent of generalization errors. In International Conference on Machine Learning , pages 2366-2375. PMLR, 2020.

[dRBK20] Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning , pages 2280-2290. PMLR, 2020.

[DSYW20] Raaz Dwivedi, Chandan Singh, Bin Yu, and Martin J Wainwright. Revisiting complexity and the bias-variance tradeoff. arXiv preprint arXiv:2006.10189 , 2020.

[Dui95] Robert PW Duin. Small sample size generalization. In Proceedings of the Scandinavian Conference on Image Analysis , volume 2, pages 957-964. Citeseer, 1995.

[Eub84] RL Eubank. The hat matrix for smoothing splines. Statistics & probability letters , 2(1):9-14, 1984.

[FM23] Ronald A Fisher and Winifred A Mackenzie. Studies in crop variation. ii. the manurial response of different potato varieties. The Journal of Agricultural Science , 13(3):311320, 1923.

[GBC16] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

[GBD92] Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation , 4(1):1-58, 1992.

[GVL13] Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013.

[GWB + 17] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

[HB21] Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. International Conference on Learning Representations (ICLR) , 2021.

[HMRT22] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

[Hot57] Harold Hotelling. The relations of the newer multivariate statistical methods to factor analysis. British Journal of Statistical Psychology , 10(2):69-79, 1957.

[HT90] Trevor Hastie and Robert Tibshirani. Generalized additive models. Monographs on statistics and applied probability. Chapman & Hall , 43:335, 1990.

[HTF09] Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

[HXZQ22] Zheng He, Zeke Xie, Quanzhi Zhu, and Zengchang Qin. Sparse double descent: Where network pruning aggravates overfitting. In International Conference on Machine Learning , pages 8635-8659. PMLR, 2022.

[Jol82] Ian T Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C: Applied Statistics , 31(3):300-303, 1982.

[Kal96] Dan Kalman. A singularly valuable decomposition: the svd of a matrix. The college mathematics journal , 27(1):2-23, 1996.

[Ken57] Maurice G Kendall. A course in multivariate analysis: London. Charles Griffin & Co , 1957.

[KH + 09] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

[KL16] Jesse H Krijthe and Marco Loog. The peaking phenomenon in semi-supervised learning. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop, S+ SSPR 2016, Mérida, Mexico, November 29-December 2, 2016, Proceedings , pages 299-309. Springer, 2016.

[KSR + 21] Ilja Kuzborskij, Csaba Szepesvári, Omar Rivasplata, Amal Rannen-Triki, and Razvan Pascanu. On the role of optimization in double descent: A least squares study. Advances in Neural Information Processing Systems , 34:29567-29577, 2021.

[LBBH98] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

[LD21] Licong Lin and Edgar Dobriban. What causes the test error? going beyond bias-variance via anova. The Journal of Machine Learning Research , 22(1):6925-7006, 2021.

[LDB21] Lorenzo Luzi, Yehuda Dar, and Richard Baraniuk. Double descent and other interpolation phenomena in gans. arXiv preprint arXiv:2106.04003 , 2021.

[LVM + 20] Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences , 117(20):10625-10626, 2020.

[Mac91] David MacKay. Bayesian model comparison and backprop nets. Advances in neural information processing systems , 4, 1991.

[Mal73] Colin L Mallows. Some comments on cp. Technometrics , 42(1):87-94, 1973.

[MBB18] Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning , pages 3325-3334. PMLR, 2018.

[MBW20] Wesley J Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited. arXiv preprint arXiv:2003.02139 , 2020.

[MNS + 21] Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? The Journal of Machine Learning Research , 22(1):1010410172, 2021.

[Moo91] John Moody. The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. Advances in neural information processing systems , 4, 1991.

[Mur22] Kevin P Murphy. Probabilistic machine learning: an introduction . MIT press, 2022.

[Nea19] Brady Neal. On the bias-variance tradeoff: Textbooks need an update. arXiv preprint arXiv:1912.08286 , 2019.

[NKB + 21] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

[NMB + 18] Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:1810.08591 , 2018.

[NTS14] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

[NWC + 11] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

[Pea01] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science , 2(11):559-572, 1901.

[PKB19] Tomaso Poggio, Gil Kur, and Andrzej Banburski. Double descent in the condition number. arXiv preprint arXiv:1912.06190 , 2019.

[PVG + 11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

[Ras18] Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808 , 2018.

[RD98] Sarunas Raudys and Robert PW Duin. Expected classification error of the fisher linear classifier with pseudo-inverse covariance matrix. Pattern recognition letters , 19(56):385-392, 1998.

[RT19] Saharon Rosset and Ryan J Tibshirani. From fixed-x to random-x regression: Biasvariance decompositions, covariance penalties, and prediction error estimation. Journal of the American Statistical Association , 2019.

[Sch78] Gideon Schwarz. Estimating the dimension of a model. The annals of statistics , pages 461-464, 1978.

[SD96] Marina Skurichina and Robert PW Duin. Stabilizing classifiers for very small sample sizes. In Proceedings of 13th International Conference on Pattern Recognition , volume 2, pages 891-896. IEEE, 1996.

[SGd + 18] Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665 , 2018.

[SGT18] Yitong Sun, Anna Gilbert, and Ambuj Tewari. On the approximation properties of random relu features. arXiv preprint arXiv:1810.04374 , 2018.

[SKL92] Jon M Sutter, John H Kalivas, and Patrick M Lang. Which principal components to utilize for principal component regression. Journal of chemometrics , 6(4):217-225, 1992.

[THV22] Ningyuan Teresa, David W Hogg, and Soledad Villar. Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal on Mathematics of Data Science , 4(1):126-152, 2022.

[Tuk77] John W Tukey. Exploratory data analysis , volume 2. Reading, MA, 1977.

[Vap95] Vladimir Vapnik. The nature of statistical learning theory . Springer science & business media, 1995.

[VCR89] F Vallet, J-G Cailton, and Ph Refregier. Linear and nonlinear extension of the pseudoinverse solution for learning boolean functions. Europhysics Letters , 9(4):315, 1989.

[VL22] Tom Viering and Marco Loog. The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

[WRR03] Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. A practical approach to microarray data analysis , pages 91-109, 2003.

[WW75] Grace Wahba and Svante Wold. A completely automatic french curve: fitting spline functions by cross validation. Communications in Statistics-Theory and Methods , 4(1):117, 1975.

Appendix

This Appendix is structured as follows: In Appendix A, we present an additional, broader, literature review. In Appendix B, we present additional background on the linear regression case study, including the proof of Proposition 1 (Appendix B.2) and an empirical study of the effect of excess features on the conditioning of PC feature matrices (Appendix B.3). In Appendix C, we present additional background on smoothers, including an analysis of the impossibility of double descent in prediction error in fixed design settings (Appendix C.2) and derivation of the smoothing matrices associated with linear regression, trees and boosting (Appendix C.3). In Appendix D, we present additional background and other definitions of effective parameter measures. Finally, in Appendix E, we further discuss the experimental setup and close by presenting additional results.

A Additional literature review

Double descent as a phenomenon. Although double descent only gained popular attention since [BHMM19]'s influential paper on the topic, the phenomenon itself had previously been observed: [LVM + 20] provide a historical note highlighting that [VCR89] may have been the first to demonstrate double descent in min-norm linear regression; [BHMM19] themselves note that double descent-like phase transitions in neural networks had previously been observed in [BO96, ASS20, NMB + 18, SGd + 18]. As discussed above, in addition to linear regressions and neural networks, [BHMM19] also present double descent curves for trees and boosting. Since then, a rich literature on other deep double descent phenomena has emerged, including double descent in the number of training epochs[NKB + 21], sparsity [HXZQ22], data generation [LDB21] and transfer learning [DB22]. We also note that the occurrence of double descent has been studied for subspace regression methods themselves [DMLB20, THV22], but has not been linked to double descent in min-norm linear regressions as we do here. For a recent review of double descent more broadly see [VL22].

Theoretical understanding of double descent. In addition to the studies of double descent in linear regression discussed in the main text[ASS20, BLLT20, DLM20, BHX20, KSR + 21, HMRT22], we note that further theoretical studies of double descent in neural networks exist (see e.g. [DMB21] for a comprehensive review). These mainly focus on exact expressions of bias and variance terms by taking into account all sources of randomness in model training and data sampling [NMB + 18, AP20, dRBK20, LD21]. Finally, different to our peak-moving experiments in Sec. 3.2 and Appendix E.2 which show that the location of the second descent in all original non-deep experiments of [BHMM19] is determined by a change in the underlying method or training procedure, [CMBK21] show that one can 'design your own generalization curve' in linear regression by controlling the data-generating process through the order by which newly (un)informative features are revealed, meaning that one could construct a learning curve with essentially arbitrary number of descents in this way. Similarly, we show in Fig. 6(b) in the main text and Fig. 13 in Appendix E.2 that multiple peaks can be achieved by switching between mechanisms for adding parameters multiple times. Note that [CMBK21] consider only the effect of data-generating mechanisms and arrive at the conclusion that the generalization curves observed in practice must arise due to interactions between typical data-generating processes and inductive biases of algorithms and 'highlight that the nature of these interactions is far from understood' [CMBK21, p. 3] - with our paper, we contributed to the understanding of these interactions by studying the role of changes in inductive biases in the case of trees, boosting and linear regression.

Interpolation regime. Entangled in the double descent literature is the interpolation regime or the benign overfitting effect (e.g. [BMM18, MBB18, BLLT20, CL21]). That is, the observation that models with near-zero training error can achieve excellent test set performance. This insight was largely motivated by large deep learning architectures where this is commonplace [MBB18]. While this line of research does provide an important update to historic intuitions which linked perfect train set performance with poor generalization, it is entirely compatible with the ideas presented in this work. In particular, in Figure 20 we demonstrate that within the interpolation regime the number of effective parameters used on test examples and the predictive performance on those test examples can vary greatly. Therefore, benign overfitting may indeed result in excellent performance without contradicting the traditional notion of a U-shaped complexity curve. Additionally, note that while the double descent phenomenon is often presented as a consequence of benign overfitting in the interpolation regime, we show in Appendix E.2 that the location of the second descent is not

inherently linked to the interpolation regime at all in the non-deep experiments of [BHMM19] - but rather a consequence of the point of change between multiple mechanisms of increasing parameter count.

Other related works. Finally, we briefly discuss some other works that might provide further context on topics touched upon in this text. [BH89] study linear neural networks through the lens of principal component analysis. Recent theoretical work suggests that, when optimized using stochastic gradient descent, these networks tend to converge to low norm or rank solutions (e.g. [GWB + 17, ACHL19]). Aspects of double descent, which were referred to as peaking in older literature, have been associated with the eigenvalues of the sample covariance matrix in a limited setting involving linear classifiers as far back as [RD98]. Much of this work focused on increasing the number of samples for a fixed model (e.g. [Dui95, SD96]). Prior to being rediscovered in the context of neural networks and larger models, more recent work investigated the peaking phenomenon in modern machine learning settings such as semi-supervised learning [KL16].

B More on linear regression

In this section, we first present a brief refresher on SVD, PCA and PCR to provide some further background to the discussion in the main text and the following subsections. We then present a proof of Proposition 1 in Appendix B.2 and finally empirically analyze the effects of excess features on the conditioning of the PCR feature matrix in Appendix B.3.

B.1 Background on SVD, PCA and PCR

In this section we provide a brief refresher on some of the mathematical tools that are fundamental to the insights presented in the main text. We begin with the singular value decomposition (SVD) which acts as a generalization of the standard eigendecomposition for a non-square matrix X ∈ R n × d (see e.g. [Kal96, DFO20]). The SVD consists of the factorization X = UΣV T where the columns of U ∈ R n × n and rows of V T ∈ R d × d consist of the left and right singular vectors of X respectively and both form orthonormal bases of X . Then, Σ ∈ R n × d contains the so-called singular values σ 1 ≥ . . . ≥ σ min( n,d ) along the diagonal with zeros everywhere else. Analogous to eigenvalues and eigenvectors, the SVD satisfies XV i = σ i U i for all i ∈ 1 , . . . , min( n, d ) . We note that for any σ i = 0 , the corresponding V i lies in the nullspace of X (while U i lies in the nullspace of X T ). The SVD results in the geometric intuition of factorizing the transformation X into three parts consisting of a basis change in R d , followed by a scaling by the singular values that adds or removes dimensions, and then a further basis change in R n . We note that any columns of U and V corresponding to either null singular values (i.e. σ i = 0 ) or the max( n, d ) -min( n, d ) non-square dimensions of X are redundant in the SVD. Therefore we can equivalently express the SVD in its so-called compact form with these vectors and their corresponding rows/columns in Σ removed.

The SVD is intimately linked to dimensionality reduction through principal component analysis (PCA) [Pea01, FM23, WRR03]. Recall that PCA is a technique that finds a basis of a centered matrix X by greedily selecting each successive basis dimension to maximize the variance within the data. Typically, this is presented as selecting each basis vector b i = arg max || b || =1 || ˆ X i b || 2 where ˆ X i denotes the original (centered) data matrix X with the first i -1 directions of variation removed (i.e. ˆ X i = X -∑ i -1 k =1 b k b T k X ). By expanding || ˆ X i b || 2 = b T ˆ X T i ˆ Xb T we can notice that the basis vectors obtained are also the eigenvectors of X T X which is proportional to the sample covariance matrix. The dimensionality reduction then occurs by selecting a subset of these new coordinates, ordered by highest variance, to represent a low-rank approximation of X . The relationship to the SVD is illuminated by noticing that, for a centered X , we can use the singular value decomposition such that

This reveals that V , the orthonormal basis containing right singular vectors of X and obtained through the SVD, produces exactly the eigenvectors of X T X required for PCA. Then the PCA operation PCA k ( X ) : R n × d → R n × k (where k ≤ d is the user-defined number of principal components

retained) may be obtained as PCA k ( X ) = XV k = U k Σ k where we overload notation such that the subscript of each matrix denotes keeping only the vectors corresponding to those first k principal components (ordered according to the magnitude of their singular values).

A natural idea that emerges from the description of PCA is to apply it in the supervised setting. Specifically, we might wish to first perform the unsupervised dimensionality reduction step of PCA followed by least squares linear regression on the transformed data. This is exactly the approach taken in principal component regression (PCR) [Ken57, Hot57]. Mathematically this can be expressed as solving ˆ β = min β ( y -XV k β ) 2 for coefficients β ∈ R k and targets y ∈ R n . Then the least squares solution takes the form ˆ β = ( V T k X T XV k ) -1 V T k X T y . The assumption of PCR is that the high variance components are most important for modeling the targets y . While this is not necessarily always the case 7 [Jol82], PCR has historically been an important method in the statistician's toolbox (e.g. [HTF09]). In the main text we make use of the connection between PCR and min-norm least squares on underdetermined problems (i.e. a fat data matrix). In particular, we show that applying min-norm least squares in this setting is equivalent to applying PCR with all principal components retained. We provide a proof for this proposition in Appendix B.2. Of course, related mathematical connections between min-norm solutions and the singular value decomposition have been made in previous work (see e.g. [GVL13, Ch. 5.7]). The regularization effect of PCR also has a known connection to ridge regression where both methods penalize the coefficients of the principal components [HTF09, Ch. 3.5.1]. The former discards components corresponding to the smallest singular values while the latter shrinks coefficients in proportion to the size of the singular values 8 .

B.2 Proof of Proposition 1

Proposition 1. [Min-norm least squares as dimensionality reduction.] For a full rank matrix X ∈ R n × d with n < d and a vector of targets y ∈ R n , the min-norm least squares solution ˆ β MN = { min β || β || 2 2 : X β = y } and the least squares solution ˆ β SVD = { β : B β = y } using the matrix of basis vectors B ∈ R n × n , constructed using the first n right singular vectors of X, are equivalent; i.e. x T ˆ β MN = b T ˆ β SVD for all x ∈ R d and corresponding basis representation b ≡ b ( x ) .

Proof. We will use the compact singular value decomposition X = UΣV T (i.e. with the d -n redundant column vectors dropped from the full Σ and V ). Specifically, the columns of U ∈ R n × n and V T ∈ R n × d form orthonormal bases and Σ ∈ R n × n is a diagonal matrix with the singular values along the diagonal. The transformation of X to B is then given by XV = UΣ = B . It is sufficient to show that for some test vector x ∈ R d in the original data space and its projection to the reduced basis space b = V T x ∈ R n , then the predictions ˆ y of both models are equal such that x T ˆ β MN = b T ˆ β SVD .

0

2000

4000

6000

8000

10000

-

7

.

5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

7

.

5

log(

σ

k

)

Singular values

σ

k

P

φ

= 10000

P

φ

= 12500

P

φ

= 15000

P

φ

= 20000

P

φ

= 30000

P

φ

= 60000

0

2000

4000

6000

8000

10000

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

log(

κ

(

B

k

))

Condition number

κ

(

B

k

)

P

φ

= 10000

P

φ

= 12500

P

φ

= 15000

P

φ

= 20000

P

φ

= 30000

P

φ

= 60000

k

k

B.3 Understanding the impact of P ex : Examining its impact on singular values of the PCR feature matrix

In this section, we empirically investigate how increasing P ex for a fixed number of P PC results in a gradual improvement in the quality of a basis of fixed dimension, revisiting the experimental setting of Sec. 3.2 in the main text.

In particular, we consider how excess features affect the conditioning of the feature matrix B k = PCA k ( X ) = XV k = U k Σ k , used in principal components regression, for different values of k ≡ P PC . As noted in [PKB19], the condition number κ ( A ) of a matrix A measures how much errors in y impact solutions when solving a system of equations A β = y , and is given by κ ( A ) = σ max ( A ) σ min ( A ) , the ratio of maximal to minimal singular value of A . Because in principal components regression B k is constructed from the top k ∈ { 1 , . . . , n } principal components, it is easy to see that σ max ( B k ) = σ 1 and σ min ( B k ) = σ k - the largest and k-th largest singular value of X respectively (as defined in Appendix B.1) - so that in principal components regression we generally have κ ( B k ) = σ 1 σ k .

To understand how the raw number of features P ϕ in the experimental setup of Sec. 3.2 in the main text impacts basis quality, it is thus instructive to study the behavior of the singular values of the random Fourier feature matrix Φ as P ϕ grows. In Fig. 10, we therefore plot σ k and κ ( B k ) for different values of P ϕ and observe that indeed, as P ϕ (and thereby P ex = P ϕ -k ) increases, σ k substantially increases in magnitude especially for large k ≡ P PC . This provides empirical evidence that the conditioning of feature matrices for principal components regression at large values of k indeed substantially improves as we add excess raw features (i.e. κ ( B k ) decreases substantially particularly at large k - as can be seen in the right panel of Fig. 10).

C More on smoothers: Bias-variance, on the impossibility of in-sample double descent and derivation of smoothing matrices

In this section, we give some further background on the well-known relationship between bias and variance in smoothers, then discuss the impossibility of double descent in prediction error in fixed design settings, and finally derive the smoothing matrices implied by the ML methods used in the main text.

C.1 Background: Bias and variance in smoothers

In this section, we discuss the well-known bias-variance relationship in smoothers (which indeed determines the U-shape of their original complexity-error curves), following the discussion in [HT90, Sec. 3.3]. Under the standard assumption that targets are generated with true expectation f ∗ ( x 0 ) and homoskedastic variance σ 2 , the bias and variance of a smoother ˆ s ( · ) evaluated at test point x 0 (for

fixed input points { x i } i ∈I train and fixed smoother ˆ s ( · ) ), can be written as:

Note that some of the historical intuition behind the U-shaped bias-variance tradeoff with model complexity on the x-axis appears to lead back to exactly the comparison between these two terms evaluated for training inputs in smoothers -[Nea19] trace explicit discussion of bias-variance tradeoff back to at least [HT90] (who indeed discuss it in the context of smoothers), preceding [GBD92]'s influential machine learning paper on the topic. A bias-variance tradeoff is easily illustrated by considering e.g. a kNN smoother 9 : a 1NN estimator has no bias when predicting outcomes for a training input but variance σ 2 , while a k ( > 1) NN estimator generally incurs some bias but has variance σ 2 /k - we thus expect a U-shaped curve in squared prediction error as we vary k . Similarly, note that for linear regression on a p < n -dimensional subset of the input, the bias of the fit generally decreases in p , while the variance increases in p as 1 n ∑ n i =1 V ar (ˆ s ( x i )) = pσ 2 n [HTF09] - leading to another well-known example of a U-shaped generalization curve (in p < n ).

When the expected mean-squared error (MSE) is the loss function of interest, a bias-variance tradeoff can also be motivated more explicitly by noting that the MSE can be decomposed as

C.2 On the impossibility of a second prediction error descent in the interpolation regime in fixed design settings

[BHMM19] provide a discussion of possible cultural and practical reasons for the historical absence of double descent shapes in the statistics literature. They note that observing double descent in the number of parameters requires parametric families in which complexity can arbitrarily be controlled and that therefore the classical statistics literature - which either focuses either on linear regression in fixed p < n settings or considers highly flexible nonparametric classes but then applies heavy regularization - could not have observed it because of this cultural focus on incompatible model classes (in which [BHMM19] argue double descent cannot emerge due to a lack of interpolation). Through our investigation, we find another fundamental reason for the historical absence of double descent shapes in statistics: even when using a sufficiently flexible parametric class (e.g. unregularized RFF linear regression), a second descent in the interpolation regime cannot emerge in fixed design settings, the primary setting considered in the early statistics literature - where much of the intuition regarding the bias-variance tradeoff was initially established.

After recalling some definitions in Appendix C.2.1, we demonstrate this impossibility of a second descent in fixed design settings for general interpolating predictors in Appendix C.2.2 below, and make some additional observations specific to smoothers in Appendix C.2.3.

C.2.1 Preliminaries.

We begin by recalling the definition of the fixed design setting of statistical learning.

Definition 2 (Fixed design setting) . In the fixed design setting, the inputs x 1 , . . . , x n are deterministic, while their associated targets are random (e.g. y i = f ( x i ) + ϵ i where ϵ i is a random noise term).

The fixed design setting was a core consideration of much of the statistical literature [RT19] and was suitable for several traditional applications in e.g. agriculture. While many modern applications require a random design setting in which the inputs are also random, several modern applications are still suitably modeled within this framework (e.g. image denoising). Next, we will define in-sample prediction error in line with [HTF09].

Definition 3 (In-sample prediction error) . Given a set of fixed design points x 1 , . . . , x n , training targets y 1 , . . . , y n and some model f , suppose we observe a new set of random responses at each of

the training points denoted y 0 i . Then the expected in-sample prediction error for some loss function L is given by

Remark: Note that in the fixed design setting, the expected in-sample prediction error is indeed the natural quantity of interest at test time.

Finally, we define an interpolating predictor below.

Definition 4 (Interpolating predictor.) . A predictor ˆ f ( · ) interpolates the training data if for y train i ∈ R , ∀ i ∈ I train

C.2.2 Main result: General impossibility of a second descent in test error for the interpolation regime in fixed design settings

We may now proceed to the main observation of this section. That is, in the fixed design setting a second descent in test error observed in the interpolation regime cannot occur. Intuitively, this is because any improvement in fixed-design test-time performance by a more complex model would require sacrificing the perfect performance on the train set. As a consequence, while - as we observe in the main text - different interpolating predictors can have wildly different (out-of-sample) generalization performance , we show below that all models in the interpolation regime have identical expected in-sample prediction error ERRin. That is, increasing the raw number of parameters p past n , e.g. in min-norm linear regression, does not and can not lead to a second descent in ERRin. We express this more formally below.

Proposition 2 (Impossibility of a second error descent in the interpolation regime for fixed design settings.) . Suppose we observe inputs x 1 , . . . , x n in a fixed design prediction problem, with associated random labels y train 1 , . . . , y train n in the training set. For all interpolating models f , test-time predictions will be identical and, therefore, a second descent in ERRin in this regime is impossible.

Proof. By definition of fixed design problems, our test set consists of examples in which the features are duplicates from the training set and the targets are new realizations from some target distribution. Then the test dataset consists of the pairs ( x 1 , y test 1 ) , . . . , ( x n , y test n ) . For any interpolating model f we can calculate the test loss as L test = ∑ n i =1 L ( f ( x i ) , y test i ) . But by Definition 4 (and because interpolating models achieve zero training error), we know that f ( x i ) = y train i . Thus L test = ∑ n i =1 L ( y train i , y test i ) , and since this is not a function of f , this quantity is independent of the number of parameters (or any other property) of f . Therefore, any change in L test (or expectations thereof) in the interpolation regime is impossible - including the second descent necessary for creating a double descent shape.

While most, if not all, discussion of the double descent phenomenon in the ML literature is not within the fixed design setting, we believe that the impossibility of double descent in this setting provides a new useful partial explanation for its historical absence from the statistical discourse.

C.2.3 Additional observations specific to smoothers

For the special case of interpolating linear smoothers, we make an additional interesting observation with regard to their smoothing weights and implied effective number of parameters. To do so, first recall that a linear smoother is defined as follows:

Definition 5 (Linear smoother) . A smoother with weights ˆ s ( x 0 ) ≡ ˆ s ( x 0 ; X train , y train ) is linear if ˆ s ( x 0 ) does not depend on y train ; that is ˆ s ( x 0 ; X train , y 1 ) = ˆ s ( x 0 ; X train , y 2 ) for ∀ y 1 , y 2 ∈ R n

We are now ready to show in Prop. 3 that all interpolating linear smoothers must have the same smoother weights and effective parameters for training inputs.

Proposition 3 (Interpolating linear smoother.) . Any interpolating linear smoother must have identical smoother weights ˆ s ( x i ) = e i for all i ∈ I train. This immediately implies also that all interpolating linear smoothers use the same effective number of parameters p e = n in fixed design settings.

Proof. In linear smoothers, ˆ s ( x i ) can not depend on y train by definition. To have ˆ s ( x i ) y ′ train = y ′ i for all i ∈ I train and any possible training outcome realization ∀ y ′ train ∈ R n , we must have ˆ s ( x i ) = e i for i ∈ I train .

From this, as || e i || 2 = 1 , it also follows that p train ˆ s = p e = ∑ n i =1 || e i || 2 = n . In fixed design settings, training and testing inputs are identical so p test ˆ s = n .

̸

Note that this specific property does not hold for more general (non-linear) interpolating smoothers: whenever y train contains duplicates, a non-linear interpolating smoother can have ˆ s ( x i ) = e i for some i ∈ I train . When all y i observed at train-time are unique, then it is easy to see that even non-linear interpolating smoothers must have ˆ s ( x i ) = e i for all i ∈ I train .

C.3 Derivation of smoothing weights of linear regression, trees and boosting

C.3.1 Smoothing weights for RFF linear regression

Derivation of the smoothing weights for the RFF linear regression example is trivial, as linear regression is a textbook example of a smoother [HT90]. To see this, let ϕ ( x 0 ) = ( ϕ 1 ( x 0 ) , . . . , ϕ P ( x 0 )) denote the P ϕ × 1 row-vector of RFF-projections of arbitrary admissible x 0 ∈ R d and let Φ train = [ ϕ ( x 1 ) T , . . . , ϕ ( x n ) T ] T denote the n × P ϕ design matrix of the training set and note that the least squares prediction obtained through the min-norm solution can be written as

i.e.

Note also that by design, as alluded to in Appendix C.2.3, when P ϕ ≥ n we have Φ train Φ T train ( Φ train Φ T train ) -1 = I n , implying that that ˆ s LR ( x i ) = e i for any x i observed at train time.

C.3.2 Smoothing weights for trees and forests

Similarly, we can exploit that trees are sometimes considered adaptive nearest neighbor methods [HTF09, Ch. 15.4.3], i.e. nearest neighbor estimators with an adaptively constructed (outcomeoriented) kernel (or distance measure). To see this, note that regression trees, that is trees trained to minimize the squared loss and/or classification trees that make predictions through averaging (not voting), can all be understood as issuing predictions that simply average across training instances within a terminal leaf. To make this more formal, consider that any tree with P leaf leaves can be represented by P leaf contiguous axis-aligned hypercubes l p , which store the mean outcome of all training examples that fall into this leaf: If we denote by l ( x 0 ) the leaf that x 0 falls into and n l ( x 0 ) = ∑ n i =1 1 { l ( x 0 ) = l ( x i ) } the number of training examples in this leaf, then we have that

giving

Further, we note that sums ˆ f Σ ( x 0 ) = ∑ M m =1 ˆ f m ( x 0 ) of smoothers ˆ f m ( x 0 ) = ˆ s m ( x 0 ) y train , m = 1 , . . . , M are also smoothers with ˆ s Σ ( x 0 ) = ∑ M m =1 ˆ s m ( x 0 ) . This immediately implies that the tree ensembles considered in Sec. 2.1 are also smoothers, and, as we show in Appendix C.3.3 below, so are boosted trees (and ensembles thereof) as considered in Sec. 2.2.

Remark: We note that, although the smooth ˆ s tree ( x 0 ) can be written in linear form without explicit reference to y train, trees (and their sums) are not truly linear smoothers because the smoother ˆ s tree ( · ) does depend in its construction on y train, reflecting their epithet as adaptive smoothers [HTF09, Ch.

15.4.3]. This mainly implies that when calculating bias and variance, we can no longer consider ˆ s ( x 0 ) as fixed and Eqs. (3) and (4) no longer hold exactly. However, this is not an issue for our purposes as we are not interested in actually estimating bias and variance in this work, rather we are interested in smoothers because - as discussed in Sec. 4 and Appendix D - they allow us to compute the most useful measure of effective parameters. Conceptually, this allows us to assess the effective amount of smoothing used, which can also be interpreted relative to e.g. a standard kNN estimator. Because for kNN estimators we have p 0 ˆ s = n k and for trees and forests it holds that ∑ n i =1 ˆ s i ( x 0 ) = 1 , the quantity ˜ k 0 ˆ s = n p 0 ˆ s admits an interesting interpretation as measuring the effective number of nearest neighbors examples have in a tree or forest (and this interpretation does not require fixed ˆ s ).

C.3.3 Smoothing weights for boosting

Deriving the smoothing weights for gradient boosting is more involved. To do so, we first re-state the gradient boosting algorithm (as presented in [HTF09, Ch. 10.10]) and then recursively construct weights from it.

The gradient boosted trees algorithm. We consider gradient boosting with learning rate η and the squared loss as L ( · , · ) :

Initialize f 0 ( x ) = 0

For p ∈ { 1 , . . . , P boost }

(a) For i ∈ { 1 , . . . , n } compute

(b) Fit a regression tree to { ( x i , g i,p ) } n i =1 , giving leaves l jp for j = 1 , . . . , J p

(c) Compute optimal predictions for each leaf j ∈ { 1 , . . . , J p } :

(d) Denote by ˜ f p ( x ) = ∑ J p j =1 1 { x ∈ l jp } γ jp the predictions of the tree built in this fashion

(e) Set f p ( x ) = f p -1 ( x ) + η ˜ f p ( x )

Output f ( x ) = f P boost ( x )

Recursive construction of smoothing weights. Equation (12) highlights that we can recursively construct smoothing weights similarly as for trees. In fact, note that in the first boosting round, ˜ f 1 ( x ) is a standard tree with smoothing weights ˆ s tree, ˜ f 1 ( · ) computed as in Eq. (10), so that f 1 ( x ) = η ˜ f 1 ( x ) has smoothing weights ˆ s boost, 1 ( · ) = η ˆ s tree, ˜ f 1 ( · ) . Now we can recursively consider the smoothing weights associated with newly added residual trees ˜ f p ( x ) by realizing that its predictions can be written as a function of standard tree weights ˆ s tree, ˜ f p ( · ) and a correction depending on ˆ s boost, 1 ( · ) alone :

train

Standard tree prediction: ˆ s tree, ˜ fp ( x ) y train

Residual correction:

ˆ

s

corr,fp

(

x

)

y

where, using the 1 × J p indicator vector e l p ( x ) = ( 1 { x ∈ l 1 } , . . . , 1 { x ∈ l J p } ) , we have

with ˆ R p the J p × n leaf-residual correction matrix with j -th row given by

Thus, we have constructed the smoothing matrix for gradient boosting recursively as:

D More on effective parameters

In what follows, we discuss the different effective parameter definitions considered in the smoothing literature, and follow closely the presentation in [HT90, Ch.3.4 & 3.5] and [HTF09, Ch. 7.5]. Recall that, as discussed in Sec. 4 of the main text, these definitions are all naturally calibrated towards linear regression so that, as we might desire, effective and raw parameter numbers are equal in the case of ordinary linear regression with p < n .

To understand their motivation, we introduce some further notation. Assume that outcomes are generated as y = f ∗ ( x ) + ϵ with unknown expectation f ∗ ( x ) and ϵ a zero-mean error term with variance σ 2 . Let f ∗ = [ f ∗ ( x 1 ) , . . . , f ∗ ( x n )] T denote the vector of true expectations. Further, let ˆ S = [ˆ s ( x 1 ) T , . . . , ˆ s ( x n ) T ] T denote the smoother matrix for all input points, so that we can write the vector of in-sample predictions as ˆ f = ˆ Sy train. For linear smoothers (and/or fixed ˆ S ), bias and variance at any input point are given by Eqs. (3) and (4), so that we have 1 n ∑ i ∈I train bias 2 ( ˆ f ( x i )) = 1 n b T ˆ S b ˆ S for bias vector b ˆ S = ( f ∗ -ˆ Sf ∗ ) and 1 n ∑ i ∈I train var ( ˆ f ( x i )) = σ 2 n tr ( ˆ S ˆ S T ) .

Covariance-based effective parameter definition. In this setup, one way of defining the effective number of parameters that is very commonly used is to consider the covariance between predictions ˆ f ( x i ) and the observed training label y i [HTF09]:

As expected, tr ( ˆ S ) = p in ordinary linear regression ( p < n ).

Error-based effective parameter definition. Another way considers the residual sum of squares (RSS) in the training sample RSS ( ˆ f ) = ∑ n i =1 ( y i -ˆ f ( x i )) 2 = ( y train -ˆ Sy train ) T ( y train -ˆ Sy train ) and notes that it has expected value:

and uses because in the linear regression case the degrees of freedom for error are n -p [HT90] and tr (2 ˆ S -ˆ S ˆ S T ) = p .

Variance-based effective parameter definition. Finally, the variance-based definition p e discussed in the main text is motivated from ∑ n i =1 var ( ˆ f ( x i )) = σ 2 ∑ n i =1 || ˆ s ( x i ) || 2 = σ 2 tr ( ˆ S ˆ S T ) giving

as ∑ n i =1 var ( ˆ f ( x i ) = pσ 2 in linear regression.

Note that, in ordinary linear regression with p < n and full rank design matrix, ˆ S is idempotent so that tr ( ˆ S ˆ S T ) = tr ( ˆ S ) = tr (2 ˆ S -ˆ S ˆ S T ) = rank ( ˆ S ) = p .

Why did we choose p var e ? We chose to adapt p var e because it is the definition that can most obviously be adapted to out-of-sample settings: both the error-based definition and the covariance-based definition focus on the effect a training outcome has in predicting itself (i.e. overfitting on the own label), while for out-of-sample prediction, this is not of interest. Instead, we therefore rely on the variance-based definition because || ˆ s ( x 0 ) || 2 can not only be computed for every input x 0 but also

admits a meaningful interpretation as measuring the amount of smoothing over the different training examples: for an averaging smoother with ∑ n i =1 ˆ s i ( x 0 ) = 1 , the simplest and smoothest prediction is a sample average with ˆ s i ( x 0 ) = 1 n - which minimizes || ˆ s ( x 0 ) || 2 = ∑ n i =1 1 n 2 = 1 n . Conversely, the least smooth weights assign all weight to a single training observation j , i.e. ˆ s ( x 0 ) = e j so that || ˆ s ( x 0 ) || 2 = 1 is maximized.

Effective parameters for methods that cannot be written in linear form. The effective parameter definitions considered above all have in common that they require a way of writing predictions ˆ y as an explicit function of y train - which is possible for the three ML methods considered in this paper, but not generally the case. Instead, for loss-based methods like neural networks, [HTF09, Ch. 7.7] note that one can make a quadratic approximation to the error function R ( w ) and get

where θ p are the eigenvalues of the Hessian ∂ 2 R ( w ) /∂ w ∂ w T and α is the weight-decay penalty used. This proxy effective parameters, motivated in a Bayesian context [Mac91], is used in [MBW20]'s study of effective parameters used in overparameterized neural networks. We note that, in addition to not being applicable to our tree-based experiments (which do not possess a differentiable loss function), this effective parameter proxy is also usually motivated in the context where in-sample prediction error is of interest [Moo91] - and might therefore also need updating to reflect the modern focus on generalization to unseen inputs. This is definitely the case in the context of linear regression: to make Eq. (22) applicable to the unregularized/ridgeless regression setup we consider, one needs to let α tend to 0 . In this case, the Hessian is X ′ X and we will have that p α e → ∑ p j =1 1 { θ j > 0 } = rank ( X ′ X ) = rank ( X ) = min ( n, p ) - which will be constant at value n once p > n , just like p train ˆ s .

Other related concepts. [DLM20] theoretically study implicit regularization of min-norm solutions by computing effective ridge penalties λ n implied by min-norm solutions by solving n = tr (Σ µ (Σ µ + λ n I n ) -1 ) with Σ µ = E [ XX T ] . Further, building on the principle of minimum description length (MDL), [DSYW20] define a complexity measure in the context of linear regression via an optimality criterion over the encodings induced by a good Ridge estimator class.

E Additional results

In this section, we first provide further discussion of the experimental setup in Appendix E.1 and then present additional results: In Appendix E.2 we show that the location of the second descent can be arbitrarily controlled for all methods under consideration, in Appendix E.3 we plot binary test error, in Appendix E.4 we plot training error, in Appendix E.5 we provide anecdotal evidence that p test ˆ s could be used for model selection and in Appendix E.6 we finally present results using further datasets.

E.1 Experimental setup

Our experimental setup largely replicates that of [BHMM19]. We describe the datasets, computation, and each experiment in detail in this section. Code is provided at https://github.com/ alanjeffares/not-double-descent .

Datasets - We perform our experiments in the main text on the MNIST image recognition dataset [LBBH98] with inputs of dimension 784. In this section, we also repeat these experiments on the SVHN digit recognition task [NWC + 11] as well as the CIFAR-10 object classification tasks [KH + 09], both containing inputs of dimension 1024 as color images have been converted to grayscale. All inputs are normalized to lie in the range [0 , 1] . All three tasks are 10 class classification problems. Furthermore, we randomly sample a subset of 10,000 training examples and use the full test set in our evaluation. In SVHN we randomly sample a balanced test set of 10,000 examples. We note that the less common choice of using squared error for this classification problem in [BHMM19] is supported by the same authors' work in [HB21, MNS + 21] on the utility of the squared loss for classification.

Compute - All experiments are performed on an Azure FX4mds. This machine runs on an Intel Xeon Gold 6246R (Cascade Lake) processor with 84 GiB of memory. The majority of the computational

cost was due to matrix inversions of a large data matrix in the case of linear regression experiments and fitting a large number of deep trees in the tree-based experiments. Each of the three experiments had a total runtime in the order of hours.

Trees (Sec. 2.1) - We use regression trees and random forests as implemented in scikit-learn [PVG + 11] for our tree experiments, which in turn implement an optimized version of the CART algorithm [BFSO84]. As in [BHMM19], we disable bootstrapping throughout and consider random feature subsets of size √ d at each split. Like [BHMM19] rely on 10 one-vs-all models due to the multi-class nature of the datasets.

Gradient Boosting (Sec. 2.2) - Gradient boosting is implemented as described in [HTF09, Ch. 10.10] and restated in Appendix C.3.3, using trees with P leaf = 10 and a random feature subset of size √ d considered at each split, and learning rate γ = . 85 as used in [BHMM19]. We rely on the scikit-learn [PVG + 11] implementation GradientBoostingRegressor to make use of the squared loss function, and like [BHMM19] rely on 10 one-vs-all models to capture the multi-class nature of the datasets. We create ensembles of P ens different boosted models by initialing them with different seeds.

Linear regression (Sec. 3) - As described in the main text we begin with a random Fourier Features (RFF) basis expansion. Specifically, given input x ∈ R d , the number of raw model parameters P ϕ is controlled by randomly generating features ϕ p ( x ) = Re (exp √ -1 v T p x ) for all p ≤ P ϕ , where each v p iid ∼ N ( 0 , 1 5 2 · I d ) . For any given number of features P ϕ , these are stacked to give a n × P ϕ dimensional random design matrix Φ , which is then used to solve the regression problem y = Φ β by least squares. Following the methodology of [BHMM19] we perform one-vs-all regression for each class resulting in 10 regression problems. For principal component regression we standardize the design matrix, apply PCA, add a bias term, and finally perform ordinary least squares regression.

E.2 'Peak-moving' experiments

In this section, we show that the first peak in generalization error can be arbitrarily (re)moved by changing when , i.e. at which value of P leaf , P boost or P PC , we transition between the two mechanisms for increasing the raw number of parameters in each of the three methods. We do so for two reasons: First, as can be seen in Fig. 11, it allows us to highlight that the switch between the two parameter-increasing mechanisms is indeed precisely the cause of the second descent in generalization error - in all three methods considered. Second, this also implies that the second descent itself is not inherently caused by interpolation : to the contrary, comparing the generalization curves in Fig. 11 with their train-error trajectories in Fig. 12, it becomes clear that such a second descent can also occur in models that have not yet and will not reach zero training error. Thus, the interpolation threshold does not itself cause the second descent - rather, it often coincides with a switch between parameter increasing mechanisms because parameters on the P leaf , P boost and P PC axis can inherently not be further increased once interpolation is achieved.

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Squared test error

Trees

P

leaf

= 500

P

leaf

= 200

P

leaf

= 100

P

leaf

= 50

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Boosting

P

boost

= 200

P

boost

= 100

P

boost

= 50

P

boost

= 25

Composite Axis

0.1

3.3

81.5

1998.2

Regression

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

Finally, we note that one could also create a generalization curve with arbitrarily many peaks and arbitrary peak locations (including peaks at raw parameter counts p > n ) by switching between parameter-increasing mechanisms more than once . In Fig. 13, we show this for the linear regression example, where we switch between increasing parameters through P PC and P ex multiple times. This experiment is inspired by [CMBK21], who show one can 'design your own generalization curve'

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Squared train error

Trees

P

leaf

= 500

P

leaf

= 200

P

leaf

= 100

P

leaf

= 50

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Boosting

P

boost

= 200

P

boost

= 100

P

boost

= 50

P

boost

= 25

Composite Axis

0

.

1

0

.

3

0

.

5

0

.

7

Regression

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

in linear regression by controlling the data-generating process through the order by which newly (un)informative features are revealed. In our case, rather than inducing the change by changing the data, we manipulate the location of the peak by simply changing the sequence in which parameter counts are increased.

0

20000

40000

60000

φ

PC

ex

0

.

0

0

.

2

0

.

4

0

.

6

Squared Error

Test error

0

20000

40000

60000

φ

PC

ex

Train error

P

ex

= 0 fixed,

P

PC

increasing in [0

,

5000]

P

PC

= 5000 fixed,

P

ex

increasing in [0

,

2500]

P

ex

= 2500 fixed,

P

PC

increasing in [5000

,

7500]

P

PC

= 7500 fixed,

P

ex

increasing in [2500

,

5000]

P

ex

= 5000 fixed,

P

PC

increasing in [7500

,

10000]

P

PC

= 10000 fixed,

P

ex

increasing in [5000

,

60000]

P

=

P

+

P

P

=

P

+

P

E.3 Plots of binary test error

In Figs. 14 to 16, we revisit Figs. 2, 4 and 5 of the main text by plotting 0-1 (or binary) loss achieved on the test-set - measuring whether the class with the largest predicted probability indeed is the true class - instead of summed squared loss across classes as in the main text. We observe qualitatively identical behavior in all experiments also for this binary loss.

200

×

1

×

1

×

50

leaf

ens

0.2

0.4

0.6

Binary test error

Double Descent in Trees

0

100

200

300

400

500

P

leaf

Error by

P

leaf

,

P

ens

fixed

P

ens

= 1

P

ens

= 5

P

ens

= 10

P

ens

= 50

0

20

40

60

80

100

P

ens

Error by

P

ens

,

P

leaf

fixed

P

leaf

= 500

P

leaf

= 100

P

leaf

= 50

P

leaf

= 20

P

×

P

10

×

1

100

×

1

×

1

×

10

boost

ens

0.1

0.2

0.3

Binary test error

Double Descent in Boosting

0

50

100

150

200

P

boost

Error by

P

boost

,

P

ens

fixed

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 20

5

10

15

20

P

ens

Error by

P

ens

,

P

boost

fixed

P

boost

= 200

P

boost

= 50

P

boost

= 25

P

boost

= 10

P

×

P

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

P

PC

+

P

ex

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Binary test error

Double Descent in Regression

0

2000

4000

6000

8000

10000

P

PC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Error by

P

PC

,

P

ex

fixed

P

ex

= 0

P

ex

= 2500

P

ex

= 10000

P

ex

= 50000

0

10000

20000

30000

40000

50000

P

ex

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Error by

P

ex

,

P

PC

fixed

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

E.4 Plots of training loss

In Figs. 17 to 19 we provide plots of summed squared training losses associated with the plots of summed squared test loss in presented Figs. 2, 4 and 5 in the main text.

200

×

1

×

1

×

50

leaf

ens

0.1

0.3

0.5

0.7

Squared train error

Error on composite axis

0

100

200

300

400

500

P

leaf

Error by

P

leaf

,

P

ens

fixed

P

ens

= 1

P

ens

= 5

P

ens

= 10

P

ens

= 50

0

20

40

60

80

100

P

ens

Error by

P

ens

,

P

leaf

fixed

P

leaf

= 500

P

leaf

= 100

P

leaf

= 50

P

leaf

= 20

P

×

P

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

0.1

0.2

0.3

Squared train error

Error on composite axis

0

50

100

150

200

P

boost

Error by

P

boost

,

P

ens

fixed

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 20

5

10

15

20

P

ens

Error by

P

ens

,

P

boost

fixed

P

boost

= 200

P

boost

= 50

P

boost

= 25

P

boost

= 10

×

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

P

PC

+

P

ex

0.1

0.3

0.5

0.7

Squared train error

Error on composite axis

0

2000

4000

6000

8000

10000

P

PC

Error by

P

PC

,

P

ex

fixed

P

ex

= 0

P

ex

= 2500

P

ex

= 10000

P

ex

= 50000

0

10000

20000

30000

40000

50000

P

ex

Error by

P

ex

,

P

PC

fixed

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

E.5 Investigating applications of p test ˆ s to model selection

In this section, we test whether a measure like p test ˆ s , measuring the effective parameters used on the test inputs, could be used for model selection purposes, possibly providing some route for redemption for parameter count based selection criteria like [Aka74, Mal73, Sch78] that explicitly trade off training error and parameters used (and importantly, do not require access to held-out labels). We are motivated by empirical observations presented in Sec. 4. In particular, we note from Fig. 7 that across all three methods, there are models in the interpolation regime that perform very well in terms of generalization - and they happen to be those with the smallest p test ˆ s . This observation indicates that one straightforward model selection strategy when choosing between different hyperparameter settings that all achieve zero training error could be to compare models by their p test ˆ s .

Here, we test whether it is possible to exploit this in more general settings when there are multiple other hyperparameter axes to choose among. In particular, we consider tuning two other hyperparameters in gradient boosting, tree depth P leaf ∈ { 10 , 50 , 100 , 200 , 500 , max } and learning rate η ∈ { 0 . 02 , 0 . 05 , . 1 , . 2 , . 3 , . 5 , . 85 } . We do so in single boosted models (i.e. P ens = 1 ), all trained until squared training error reaches approximately zero, i.e. we terminate after P boost ∈ [1 , 500] rounds, where P boost is chosen as the first round (if any) where squared training error l train drops below ϵ = 10 -4 (conversely, whenever l train > ϵ for all P boost ≤ 500 , we do not consider the associated ( P leaf , η ) configurations an interpolating model).

10

10

2

10

3

p

test

ˆ

s

(log scale)

0

.

0050

0

.

0075

0

.

0100

0

.

0125

0

.

0150

0

.

0175

0

.

0200

Squared test error

P

leaf

P

leaf

= 500

P

leaf

= 200

P

leaf

= 100

P

leaf

= 50

P

leaf

= 10

Best with

l

train

> /epsilon1

(oracle)

In Fig. 20 we observe that the intuition indeed carries over to this more general scenario and that interpolating models with lower p test ˆ s are indeed associated with lower test error. In particular, choosing the model with lowest p test ˆ s - which only requires access to test-time inputs x but no label information y - would indeed lead to the best test-performance among the interpolating models (with l train < ϵ ). Further, using a red cross, we also plot the performance of the best non-interpolating model (with l train > ϵ ) among the considered ( P leaf , η ) configurations; this is selected among the models with less boosting rounds P boost by an oracle with access to test-time performance. We observe that this

non-interpolating oracle has essentially the same performance as the best interpolating model chosen by p test ˆ s .

E.6 Results using other datasets

Below, we replicate our main experimental results using other datasets considered in [BHMM19], SVHN and CIFAR-10, exhibiting the same trends as the experiments on MNIST presented in the main paper.

E.6.1 SVHN

200

×

1

×

1

×

50

P

leaf

P

ens

0.6

0.9

1.3

1.5

Squared test error

Double Descent in Trees

0

100

200

300

400

500

P

leaf

Error by

P

leaf

,

P

ens

fixed

P

ens

= 1

P

ens

= 5

P

ens

= 10

P

ens

= 50

0

20

40

60

80

100

P

ens

Error by

P

ens

,

P

leaf

fixed

P

leaf

= 500

P

leaf

= 100

P

leaf

= 50

P

leaf

= 20

×

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

0.6

0.9

1.3

1.5

Squared test error

Double Descent in Boosting

0

50

100

150

200

P

boost

Error by

P

boost

,

P

ens

fixed

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 20

5

10

15

20

P

ens

Error by

P

ens

,

P

boost

fixed

P

boost

= 200

P

boost

= 50

P

boost

= 25

P

boost

= 10

×

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

P

PC

+

P

ex

0.1

3.3

81.5

1998.2

Squared test error

Double Descent in Regression

0

2000

4000

6000

8000

10000

P

PC

Error by

P

PC

,

P

ex

fixed

P

ex

= 0

P

ex

= 2500

P

ex

= 10000

P

ex

= 50000

0

10000

20000

30000

40000

50000

P

ex

Error by

P

ex

,

P

PC

fixed

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

200

×

1

×

1

×

50

P

leaf

P

ens

1

10

10

2

p

0

ˆ

s

(log scale)

Trees

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

10

1

10

2

10

3

Boosting

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

PC

ex

10

2

10

4

10

6

Linear Regression

I

train

I

test

×

×

P

+

P

1

10

1

10

2

p

test

ˆ

s

(log scale)

0

.

07

0

.

10

0

.

13

Squared test error

Trees

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

P

ens

= 50

P

ens

= 100

10

1

10

2

10

3

p

test

ˆ

s

(log scale)

Boosting

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

10

2

10

3

10

4

p

test

ˆ

s

(log scale)

0.1

0.4

0.7

1

Linear Regression

P

ex

= 0

P

ex

= 2500

P

ex

= 5000

P

ex

= 10000

P

ex

= 20000

P

ex

= 50000

E.6.2 CIFAR-10

200

×

1

×

1

×

50

leaf

ens

0.9

1.1

1.3

1.5

Squared test error

Double Descent in Trees

0

100

200

300

400

500

P

leaf

Error by

P

leaf

,

P

ens

fixed

P

ens

= 1

P

ens

= 5

P

ens

= 10

P

ens

= 50

0

20

40

60

80

100

P

ens

Error by

P

ens

,

P

leaf

fixed

P

leaf

= 500

P

leaf

= 100

P

leaf

= 50

P

leaf

= 20

P

×

P

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

0.8

0.9

1.1

1.2

Squared test error

Double Descent in Boosting

0

50

100

150

200

P

boost

Error by

P

boost

,

P

ens

fixed

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 20

5

10

15

20

P

ens

Error by

P

ens

,

P

boost

fixed

P

boost

= 200

P

boost

= 50

P

boost

= 25

P

boost

= 10

×

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

P

PC

+

P

ex

0.1

3.3

81.5

1998.2

Squared test error

Double Descent in Regression

0

2000

4000

6000

8000

10000

P

PC

Error by

P

PC

,

P

ex

fixed

P

ex

= 0

P

ex

= 2500

P

ex

= 10000

P

ex

= 50000

0

10000

20000

30000

40000

50000

P

ex

Error by

P

ex

,

P

PC

fixed

P

PC

= 10000

P

PC

= 7500

P

PC

= 5000

P

PC

= 2500

200

×

1

×

1

×

50

P

leaf

P

ens

1

10

10

2

p

0

ˆ

s

(log scale)

Trees

10

×

1

100

×

1

×

1

×

10

P

boost

P

ens

10

1

10

2

10

3

Boosting

0 + 0

10,000 + 0

10,000 + 25,000

10,000 + 50,000

P

PC

+

P

ex

10

2

10

4

10

6

Linear Regression

I

train

I

test

×

×

1

10

1

10

2

p

test

ˆ

s

(log scale)

0

.

07

0

.

10

0

.

13

0

.

16

Squared test error

Trees

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

P

ens

= 50

P

ens

= 100

10

1

10

2

10

3

p

test

ˆ

s

(log scale)

Boosting

P

ens

= 1

P

ens

= 2

P

ens

= 5

P

ens

= 10

P

ens

= 20

10

2

10

3

10

4

p

test

ˆ

s

(log scale)

0.07

0.16

0.3

0.45

Linear Regression

P

ex

= 0

P

ex

= 2500

P

ex

= 5000

P

ex

= 10000

P

ex

= 20000

P

ex

= 50000

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to explain the well-known phenomenon of double-descent in the scope of several traditional ML methods (random forests, boosting and linear regression on random Fourier features). The main message of the paper (in the mentioned scope) is that what appears to be a double-descent on the first sight (using plots that count raw number of model parameters), might actually be a consequence of an intristincly two-dimensional parameter development composed of two traditionally behaved, complementary components. In all the studied cases, authors disentangle the parameter count into two parts - one accounting for a complexity of \"a single model within the considered class\", while the second accounting for something, that one could call \"extent of statistical power in terms of some proxy of mixture of experts, or ensembles\". The authors show that once the parameters are counted in this two-dimensional manner, the double-descent curve folds into two curves, of which one usually exhibits traditional U-shape and the other one exhibits an L-shape. For linear regression, authors consider a non-trivial disentanglement - into the number of basis vectors that are directly used for regression and the number of exceeding features. \n\nIn an independent part, authors consider models that fall into the category of smoothers (kNN, trees and forests, LR) and show that if one counts parameters in a more sensible way (in the paper via a specific, smoothers-tailored effective number of parameters measure), one can not even register a double-descent behavior anymore. The authors show that the test performance can be partially explained via the effective number of parameters (though it seems the number of ensembles/extra features should be taken into account too).\n\nSTRENGTHS:\nS1: The idea of decomposing the parameter count into two independent dimensions which unfold (and therefore partially explain) the well-known double-descent is (I believe) novel, surprising, important (at least within the scope of the studied models), interesting, neat and probably impactful. The analysis done for the models considered is strong, empirical evidence convincing and theoretical analysis for RFF-LR makes sense. The experiments in this section are well-chosen and well-done and to the point. \n\nS2: As a part of the paper's exposition, another very important observation is made (around lines 192-195) - the double descent, being a consequence of different mechanisms (explained in the paper), is not strictly tight to the point where the model capacity enters the interpolation regime. We can obtain similar double-descent curves with breaking points far below the interpolation threshold.\n\nS3: The quality of writing and the overall clarity and structure of the paper are well-above the average.\n\nWEAKNESSES:\nThe weaknesses are roughly sorted by severity, starting with the most severe. \n\nW1: The whole part 2 of the paper - explaining the double descent with alternative parameter counting - is not novel, as acknowledged by the authors. The idea of the double-descent appearing to happen only if raw parameters are displayed, but disappearing, or being more explainable, if effective number of parameters measure is used instead, is already done in several works cited by the paper. The authors defend the originality and novelty of this part and of the used effective parameter count as a tool for explaining double descent, within the scope of smoothers, by claiming that (i) unlike some of the previous work, it covers model classes that use trees and (ii) the proposed distinction between train and test time parameter count is essential to explaining the double-descent and (iii) the proposed parameter count does not require test-time labels, unlike the previous methods. However: (i) while this is true, the disadvantage is that this method is only usable for smoothers, and therefore it is very unclear how this can be used for, for instance, neural networks (unlike the measures used in the cited literature), which limits not only the scope of this part, but also, more importantly, the potential impact of this part. On the other hand, (ii) seems not to be well-reasoned. While it is true that for this particular measure and for smoothers and in-sample double-descent this seems to be the case, other complexity measures, such as the one considered in [1], but also those that [1] compares itself against, seem perfectly capable of explaining double-descent (for most of the models excluding trees) without this train-test distinction. Finally, (iii) seems not to even be true, as the method considered in [1] and the ones cited by [1] do not require the held-out / test data at all. \n\nW2: Related to W1, the authors do not explain at all why the effective parameter count drops so significantly if we increase the number of ensembles/exceeding features (i.e. the secondary parameter count dimension). Perhaps it is not in the scope of the paper, but I would find it a significant strengthening of the paper, if this was included. In fact, the authors don't even explain (except in the in-sample RFF-LR case with the first dimension of parameters already being equal $n$), why the effective parameter count in the test time doesn't increase. I would, moreover, want to see an alternative to Figure 7, but with $P^{PCA}$ being fixed and $P^{Ex}$ being the variable for different base levels of $P^P{PCA}$, because with this it is not even clear what the relation between the $P^{Ex}$ and the effective parameter count with fixed (and potentially small) $P^{PCA}$ is. \n\nW3: Though the authors provide a heuristic explanation on why increasing $P^{Ex}$ should improve the model's performance (by providing better basis features and better-conditioned feature matrices), this does not really explain the improvement in model's performance. The connection between the quality of basis features in terms of the properties of the basis feature matrix and the test performance is not clear. Information-theoretically, all the information necessary to predict $y$ should already be encoded in $X$, thus representing $X$ with increasing number of random features, just because some matrix becomes better conditioned, does not seem to causally explain the improvement in test performance. \n\nW4: Related to W3, the authors do not at all discuss the apparent non-symmetry between the first and the second dimension. How come the second dimension of parameter count enjoys L-curve, while the first dimension still displays U-curve? Can you provide an explanation that would extend beyond the particularities of Appendix B and wrap all the considered models in a simple (or at least general-enough) way? \n\nW5 (minor): I don't agree with line 290: \"First, we observe that these results perfectly match the intuition developed in Part 1\" - In part 1, there is no intuition built on why the measures of effective number of parameters should never increase beyond the first dimension of parameter count. \n\n[1] Maddox, Wesley J., Gregory Benton, and Andrew Gordon Wilson. \"Rethinking parameter counting in deep models: Effective dimensionality revisited.\" arXiv preprint arXiv:2003.02139 (2020).\n\nQUESTIONS:\nQ1: Why is the difference between test and train effective parameter count in Figure 6 for RFF-LR at the interpolation threshold so huge? \n\nQ2: Do you have an intuition on how to extend the results of part 1 to neural networks? What would be the candidates for the two dimensions? What about other models? Is there a way to make this more model-agnostic? \n\nQ3: Can you explain any of the questions raised in the \"weaknesses\" section? \n\nBesides, I have a few comments: \n\nC1: The statement in lines 159-160 seems to be incorrect. I don't see how min-norm solutions can project $y$ into a row space. \n\nSummary: \n\nDespite several significant concerns I have raised in the \"weaknesses\" section, the paper brings some truly novel and important ideas (most of the part 1). Therefore, I recommend accepting the paper. However, I would be more happy, if the authors addressed the issues raised in the \"weaknesses\" section, either in the rebuttal, or, more conveniently for them, by withdrawing the paper and submitting the improved version to another venue, producing a very strong submission. \n\nNote: After seeing the author's response and having most of my concerns resolved, I have decided to raise my score from the original 6 to new 7.\n\nLIMITATIONS:\nI didn't find any explicit discussion of limitations, despite being marked present by the authors in the paper checklist. Perhaps a few sentences here-and-there can be interpreted as a discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically looks at the double descent phenomena, and it is shown that the number of parameters on the x-axis does not represent the complexity well for double descent curves, and in essence, causes the double descent. If a more appropriate complexity measure is adopted, in this paper, the effective number of parameters, the phenomena disappears, and we observe a typical U-shape or L-shape, as expected from learning theory. \n\nNote: I did not have time to study the appendix in detail.\n\nSTRENGTHS:\nThis paper addresses and resolves some paradoxes in the community surrounding the double descent phenenoma. As such, I believe it will have a high impact and is relevant for many researchers. Furthermore, the introduced complexity measure can be highly relevant for model selection tasks. \n\nThe analysis is straightforward and complete. Especially nice is that it revisits the original experiments of Belkin exactly and finds the root-causes of the behavior. \n\nThe figures are clear and intuitive, especially the color-coding with blue and red curves is very helpful to our understanding.\n\nThe analysis of linear regression seems to agree with some very early analysis by \nS. Raudys and R. Duin, “Expected classification error of the\nFisher linear classifier with pseudo-inverse covariance matrix,”\nPattern Recognit. Lett., vol. 19, no. 5-6, pp. 385–392, 1998\nwho noted that dimensionality reduction indeed plays the suggested role. A more recent work that covers this is:\nJ. H. Krijthe and M. Loog, “The peaking phenomenon in semisupervised learning,” in S+SSPR. Springer, 2016, pp. 299–309\nwhich can be slightly easier to read. However, the presentation in the current paper is much more clear and also covers the boosting and random forest experiments, while these papers only cover linear regression.\n\nMore connections and background on the double descent phenomena can be found in:\nViering, T., & Loog, M. (2022). The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence.\nFor example, double descent can either be visible in learning curves or feature curves, a connection that is often overlooked. \n\nNote that, historically, the double descent behavior was studied inbetween 1990 and 2019 (as also apparent from above), but that this was in a subcommunity that was not popular. \n\nAs an aside, we have attempted a similar analysis as the one given in the paper via VC bounds and other similar measures for regression, but they generally are not fine grained enough for the analysis required. I am very happy the authors did manage to unmask the paradox via the proposed complexity measure.\n\nWEAKNESSES:\nnone\n\nQUESTIONS:\nAre there any connections with the proposed metric of Generalized Effective Number of parameters and generalization? E.g. are there any upperbounds on the generalization that can be given in terms of the proposed measure? That would be an added strength\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new perspective for understanding the double descent behavior in non deep learning models: Once the number of raw parameters goes pass the interpolation threshold, the effective parameter number actually stops increasing and if one keeps increasing the number of parameters, the underlying model class will be modified. The authors analyze three examples from existing literature: Decision tree, gradient boosting and linear regression. In particular, the authors perform a novel analysis on overparameterized linear regression shows that the increment of Fourier feature dimensions would first increase the number of PCA regression dimensions, which follows the classic U curve structure, and then increase the number of excess feature dimension, which results in a L shape of test error. However, studying the effective dimension size requires model-specific derivation, therefore the author proposes a unified framework for studying the effective parameter size in different models using techniques from smoothers. With, the proposed measurement as the x-axis, the double descent shape on y-axis disappears, which aligns with the experiment results for individual models.\n\nSTRENGTHS:\n- The paper is well written and well motivated.\n\n- The analysis and proposed measurement of effective number of parameter is technically sound.\n\n- The relationship with existing literature is thoroughly discussed.\n\nWEAKNESSES:\n- If I understand correctly, the proposed measurement needs to be computed using test samples, which is not as straightforward as other metrics that only requires training sample, e.g. the measurement proposed in [MBW20].\n\nQUESTIONS:\n- The authors show that for decision tree, gradient boosting, and linear regression, there exists an axis in model complexity which follows \"bigger is better\". Does this axis exist for all types of models?\n\n- How does the proposed measurement fit into unsupervised learning models where there does not exist `y` in the task.\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
This paper studies the double descent phenomenon where it has been suggested that optimal generalization performance is achieved as model complexity gets extremely large and training data is perfectly overfit. However, this work suggests that true model complexity is not described simply by the number of parameters. This claim is backed by replications of experiments from a foundational double descent paper, where the main result is empirical evidence and a theoretical explanation on how linear regression in high dimension reduces to a lower dimension problem on principle component features. Lastly, the authors propose a way to measure the number of effective parameters for each replicated experiment.

STRENGTHS:
- The paper is well-written and the motivation for the work is well-defined
- Proposition 1 is an interesting and novel observation and its proof is rigorous from what I can tell
- The area of overparameterized machine learning is of very high relevance today

WEAKNESSES:
An obvious weakness of the paper is that it does not use modern neural network models. I recognize that work on linear models is necessary as a foundation and the authors do discuss this limitation in the discussion, however it is troubling that there is no proper justification/commentary on the connection between the work and modern architectures.

QUESTIONS:
I'm curious how well your linear regression explanation extends to other statistical models like in BLLT20 and HMRT22.

LIMITATIONS:
The authors adequately address the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

comments: SUMMARY:
This paper studies the double descent phenomenon focusing on the complexity axis used to display the phenomenon. For trees and boosting, they show that double descent is a result of peculiar axes used by one previous work and that an alternative formulation of complexity removes the phenomenon. They then propose a generalized measure for the effective number of parameters for both trees, boosting, and lienar models which recovers a more classical U-shape.

STRENGTHS:
- The paper is clearly written
- The reparameterization of complexity for trees and boosting is clear and needed + for linear models is well-motivated
- The introduced generalized effective number of parameters based on smoothers is interesting and seems to work well

WEAKNESSES:
- The evidence for double descent in trees is very limited and weak (one main figure and one appendix figure in BHMM19); the authors may want to dedicate more time to the rest of the paper rather than refuting this phenomenon
- The experimental settings shown are very limited, would be nice to see less synthetic settings, particularly in Fig 7 to see how well the proposed complexity measure holds

QUESTIONS:
- A deeper comparison with related work would be helpful, particularly MBW20, DLM20, and DSYW20.
- The authors should connect their analysis in Section 3 based on principal components  with the more popular analyses using ridge regression rather than limiting PCs (e.g. that in HMRT22). The empirical observations are extremely similar and the connection is fairly close, as ridge regression can (often) be seen as a smooth eversion of PCA regression.

LIMITATIONS:
- See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the seeming appearance of double descent in non-deep models, and finds various ways in which it disappears when thinking more carefully about how the parameters are scaled. Trees and boosted models see double descent because parameters start increasing along a different and more effective dimension (ensemble size) once p=n. Linear regression sees double descent because the min-norm optimization solution used in prior work functions differently before and after p=n. And all of these methods no longer see double descent when the \"effective number of parameters\" is used as the x-axis instead of the raw number of parameters.\n\nSTRENGTHS:\nI greatly enjoyed reading the paper. The findings on linear regression are very interesting and contribute towards the understanding of the important phenomenon of double descent. It is well-presented, with concise and easy-to-understand figures and clear writing. It is likely to encourage follow-on work in the field.\n\nWEAKNESSES:\nThe portions on trees/boosting and linear regression sit a bit unevenly together. In my view, the trees/boosting section doesn't actually tell us much about double descent since we're talking about independently trained ensembles, which is a different issue altogether. Meanwhile, the linear regression bit is very interesting.\n\nThe connections between min-norm solutions, SVD, and PCA (and the Moore-Pensrose pseudoinverse) are reasonably well understood (see e.g. these class notes http://www.sci.utah.edu/~gerig/CS6640-F2012/Materials/pseudoinverse-cis61009sl10.pdf), so I'd question the novelty of the theory a bit. However, the application of the theory to understand the problem was excellent.\n\nThe section on effective number of parameters didn't spend enough time analyzing the relationship between raw parameter count and effective parameter count (and any implications thereof), which seems worth dwelling a bit from the purpose of understanding double descent (more so than showing the lack of double descent using that metric as the x-axis).\n\nI think the paper would be improved with more connections to related work and related ideas, e.g. regularization and particularly ridge regression (due to its connections with PCA), work on multi-layer linear networks and PCA (e.g. Baldi and Hornik 1988, \"Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima), connections between neural network optimization and min-norm solutions (which you do give a brief mention to, but seems quite relevant)\n\nQUESTIONS:\nI like the paper and applaud your work, but as a take-it-or-leave-it suggestion I would consider whether a reframing around your core contributions (insights about overparameterized linear regression) vs. being a rebuttal to another paper (forces simple and less interesting critiques about ensembles to take up space) would help. There's interesting stuff in the appendix (e.g. condition number analyses of bases) that could certainly fit in the main paper if you end up going that route.\n\nOne major question on my mind after reading this paper is what it \"means\" to find a richer basis among the input features and why that helps. Do you have any insights on this? Are there applications to preconditioning? How closely tied is it to the choice of RFF features, vs. alternate setups where e.g. there are p>n features by default?\n\nPer the 'weakness' above, can you add the relationship/analysis of raw parameter count and effective parameter count?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-21"}