SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Covariance-based effective parameter definition. In this setup, one way of defining the effective number of parameters that is very commonly used is to consider the covariance between predictions  $\hat{f}(x_i)$  and the observed training label  $y_i$  [HTF09]:
$$
p _ {e} ^ {c o v} = \frac {\sum_ {i = 1} ^ {n} c o v \left(y _ {i} , \hat {f} \left(x _ {i}\right)\right)}{\sigma^ {2}} = t r (\hat {\mathbf {S}}) \tag {18}
$$
As expected,  $tr(\hat{\mathbf{S}}) = p$  in ordinary linear regression ( $p < n$ ).
Error-based effective parameter definition. Another way considers the residual sum of squares (RSS) in the training sample  $RSS(\hat{\mathbf{f}}) = \sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2 = (\mathbf{y}_{train} - \hat{\mathbf{S}}\mathbf{y}_{train})^T(\mathbf{y}_{train} - \hat{\mathbf{S}}\mathbf{y}_{train})$  and notes that it has expected value:
$$
\mathbb {E} \left[ R S S (\hat {\mathbf {f}}) \right] = (n - t r \left(2 \hat {\mathbf {S}} - \hat {\mathbf {S}} \hat {\mathbf {S}} ^ {T}\right)) \sigma^ {2} + \mathbf {b} _ {\hat {\mathbf {S}}} ^ {T} \mathbf {b} _ {\hat {\mathbf {S}}} \tag {19}
$$
$$
n - p _ {e} ^ {\text {e r r}} = n - t r \left(2 \hat {\mathbf {S}} - \hat {\mathbf {S}} \hat {\mathbf {S}} ^ {T}\right) \tag {20}
$$
because in the linear regression case the degrees of freedom for error are  $n - p$  [HT90] and  $tr(2\hat{\mathbf{S}} - \hat{\mathbf{S}}\hat{\mathbf{S}}^T) = p$ .
Variance-based effective parameter definition. Finally, the variance-based definition  $p_e$  discussed in the main text is motivated from  $\sum_{i=1}^{n} \text{var}(\hat{f}(x_i)) = \sigma^2 \sum_{i=1}^{n} ||\hat{\mathbf{s}}(x_i)||^2 = \sigma^2 \text{tr}(\hat{\mathbf{S}}\hat{\mathbf{S}}^T)$  giving
$$
p _ {e} \equiv p _ {e} ^ {\text {v a r}} = t r (\hat {\mathbf {S}} \hat {\mathbf {S}} ^ {T}) \tag {21}
$$
as  $\sum_{i=1}^{n} var(\hat{f}(x_i) = p\sigma^2$  in linear regression.
Note that, in ordinary linear regression with  $p < n$  and full rank design matrix,  $\hat{\mathbf{S}}$  is idempotent so that  $tr(\hat{\mathbf{S}}\hat{\mathbf{S}}^T) = tr(\hat{\mathbf{S}}) = tr(2\hat{\mathbf{S}} - \hat{\mathbf{S}}\hat{\mathbf{S}}^T) = rank(\hat{\mathbf{S}}) = p$ .
Why did we choose  $p_e^{var}$ ? We chose to adapt  $p_e^{var}$  because it is the definition that can most obviously be adapted to out-of-sample settings: both the error-based definition and the covariance-based definition focus on the effect a training outcome has in predicting itself (i.e. overfitting on the own label), while for out-of-sample prediction, this is not of interest. Instead, we therefore rely on the variance-based definition because  $||\hat{\mathbf{s}}(x_0)||^2$  can not only be computed for every input  $x_0$  but also
admits a meaningful interpretation as measuring the amount of smoothing over the different training examples: for an averaging smoother with  $\sum_{i=1}^{n} \hat{\mathbf{s}}^i(x_0) = 1$ , the simplest and smoothest prediction is a sample average with  $\hat{\mathbf{s}}^i(x_0) = \frac{1}{n}$  - which minimizes  $||\hat{\mathbf{s}}(x_0)||^2 = \sum_{i=1}^{n} \frac{1}{n^2} = \frac{1}{n}$ . Conversely, the least smooth weights assign all weight to a single training observation  $j$ , i.e.  $\hat{\mathbf{s}}(x_0) = \mathbf{e}_j$  so that  $||\hat{\mathbf{s}}(x_0)||^2 = 1$  is maximized.
Effective parameters for methods that cannot be written in linear form. The effective parameter definitions considered above all have in common that they require a way of writing predictions  $\hat{y}$  as an explicit function of  $\mathbf{y}_{train} -$  which is possible for the three ML methods considered in this paper, but not generally the case. Instead, for loss-based methods like neural networks, [HTF09, Ch. 7.7] note that one can make a quadratic approximation to the error function  $R(\mathbf{w})$  and get
$$
p _ {e} ^ {\alpha} = \sum_ {j = 1} ^ {p} \frac {\theta_ {j}}{\theta_ {j} + \alpha} \tag {22}
$$
where  $\theta_{p}$  are the eigenvalues of the Hessian  $\partial^2 R(\mathbf{w}) / \partial \mathbf{w} \partial \mathbf{w}^T$  and  $\alpha$  is the weight-decay penalty used. This proxy effective parameters, motivated in a Bayesian context [Mac91], is used in [MBW20]'s study of effective parameters used in overparameterized neural networks. We note that, in addition to not being applicable to our tree-based experiments (which do not possess a differentiable loss function), this effective parameter proxy is also usually motivated in the context where in-sample prediction error is of interest [Moo91] - and might therefore also need updating to reflect the modern focus on generalization to unseen inputs. This is definitely the case in the context of linear regression: to make Eq. (22) applicable to the unregularized/ridgeless regression setup we consider, one needs to let  $\alpha$  tend to 0. In this case, the Hessian is  $\mathbf{X}'\mathbf{X}$  and we will have that  $p_e^\alpha \rightarrow \sum_{j=1}^{p} \mathbf{1} \{\theta_j > 0\} = \text{rank}(\mathbf{X}'\mathbf{X}) = \text{rank}(\mathbf{X}) = \min(n, p)$  - which will be constant at value  $n$  once  $p > n$ , just like  $p_{\hat{\mathbf{s}}}^{\text{train}}$ .
Other related concepts. [DLM20] theoretically study implicit regularization of min-norm solutions by computing effective ridge penalties  $\lambda_{n}$  implied by min-norm solutions by solving  $n = tr(\Sigma_{\mu}(\Sigma_{\mu} + \lambda_n I_n)^{-1})$  with  $\Sigma_{\mu} = E[XX^T]$ . Further, building on the principle of minimum description length (MDL), [DSYW20] define a complexity measure in the context of linear regression via an optimality criterion over the encodings induced by a good Ridge estimator class.
In this section, we first provide further discussion of the experimental setup in Appendix E.1 and then present additional results: In Appendix E.2 we show that the location of the second descent can be arbitrarily controlled for all methods under consideration, in Appendix E.3 we plot binary test error, in Appendix E.4 we plot training error, in Appendix E.5 we provide anecdotal evidence that  $p_{s}^{\mathrm{test}}$  could be used for model selection and in Appendix E.6 we finally present results using further datasets.
E.1 Experimental setup
Our experimental setup largely replicates that of [BHMM19]. We describe the datasets, computation, and each experiment in detail in this section. Code is provided at https://github.com/alanjeffares/not-double-descent.
Datasets - We perform our experiments in the main text on the MNIST image recognition dataset [LBBH98] with inputs of dimension 784. In this section, we also repeat these experiments on the SVHN digit recognition task  $\mathrm{[NWC^{+}11]}$  as well as the CIFAR-10 object classification tasks  $\mathrm{[KH^{+}09]}$ , both containing inputs of dimension 1024 as color images have been converted to grayscale. All inputs are normalized to lie in the range [0, 1]. All three tasks are 10 class classification problems. Furthermore, we randomly sample a subset of 10,000 training examples and use the full test set in our evaluation. In SVHN we randomly sample a balanced test set of 10,000 examples. We note that the less common choice of using squared error for this classification problem in [BHMM19] is supported by the same authors' work in [HB21, MNS $^{+}$ 21] on the utility of the squared loss for classification.
Compute - All experiments are performed on an Azure FX4mds. This machine runs on an Intel Xeon Gold 6246R (Cascade Lake) processor with 84 GiB of memory. The majority of the computational
cost was due to matrix inversions of a large data matrix in the case of linear regression experiments and fitting a large number of deep trees in the tree-based experiments. Each of the three experiments had a total runtime in the order of hours.
Trees (Sec. 2.1) - We use regression trees and random forests as implemented in scikit-learn  $\mathrm{[PVG^{+}11]}$  for our tree experiments, which in turn implement an optimized version of the CART algorithm [BFSO84]. As in [BHMM19], we disable bootstrapping throughout and consider random feature subsets of size  $\sqrt{d}$  at each split. Like [BHMM19] rely on 10 one-vs-all models due to the multi-class nature of the datasets.
Gradient Boosting (Sec. 2.2) - Gradient boosting is implemented as described in [HTF09, Ch. 10.10] and restated in Appendix C.3.3, using trees with  $P^{leaf} = 10$  and a random feature subset of size  $\sqrt{d}$  considered at each split, and learning rate  $\gamma = .85$  as used in [BHMM19]. We rely on the scikit-learn [PVG+11] implementation GradientBoostingRegressor to make use of the squared loss function, and like [BHMM19] rely on 10 one-vs-all models to capture the multi-class nature of the datasets. We create ensembles of  $P^{ens}$  different boosted models by initialing them with different seeds.
Linear regression (Sec. 3) - As described in the main text we begin with a random Fourier Features (RFF) basis expansion. Specifically, given input  $\mathbf{x} \in \mathbb{R}^d$ , the number of raw model parameters  $P^\phi$  is controlled by randomly generating features  $\phi_p(\mathbf{x}) = \mathrm{Re}(\exp^{\sqrt{-1}\mathbf{v}_p^T\mathbf{x}})$  for all  $p \leq P^\phi$ , where each  $\mathbf{v}_p \stackrel{\mathrm{iid}}{\sim} \mathcal{N}(\mathbf{0}, \frac{1}{5^2} \cdot \mathbf{I}_d)$ . For any given number of features  $P^\phi$ , these are stacked to give a  $n \times P^\phi$  dimensional random design matrix  $\Phi$ , which is then used to solve the regression problem  $\mathbf{y} = \boldsymbol{\Phi}\boldsymbol{\beta}$  by least squares. Following the methodology of [BHMM19] we perform one-vs-all regression for each class resulting in 10 regression problems. For principal component regression we standardize the design matrix, apply PCA, add a bias term, and finally perform ordinary least squares regression.
E.2 “Peak-moving” experiments
In this section, we show that the first peak in generalization error can be arbitrarily (re)moved by changing when, i.e. at which value of  $P^{leaf}$ ,  $P^{boost}$  or  $P^{PC}$ , we transition between the two mechanisms for increasing the raw number of parameters in each of the three methods. We do so for two reasons: First, as can be seen in Fig. 11, it allows us to highlight that the switch between the two parameter-increasing mechanisms is indeed precisely the cause of the second descent in generalization error – in all three methods considered. Second, this also implies that the second descent itself is not inherently caused by interpolation: to the contrary, comparing the generalization curves in Fig. 11 with their train-error trajectories in Fig. 12, it becomes clear that such a second descent can also occur in models that have not yet and will not reach zero training error. Thus, the interpolation threshold does not itself cause the second descent – rather, it often coincides with a switch between parameter increasing mechanisms because parameters on the  $P^{leaf}$ ,  $P^{boost}$  and  $P^{PC}$  axis can inherently not be further increased once interpolation is achieved.
Figure 11: Shifting the location of the double descent peak in generalization error by changing when we switch between the two parameter-increasing mechanisms. Transitioning from  $P^{leaf}$  to  $P^{ens}$  at different values of  $P^{leaf}$  in the tree experiments (left), transitioning from  $P^{boost}$  to  $P^{ens}$  at different values of  $P^{boost}$  in the boosting experiments (middle) and transitioning from  $P^{PC}$  to  $P^{ex}$  at different values of  $P^{PC}$  in the regression experiments (right).
Finally, we note that one could also create a generalization curve with arbitrarily many peaks and arbitrary peak locations (including peaks at raw parameter counts  $p > n$ ) by switching between parameter-increasing mechanisms more than once. In Fig. 13, we show this for the linear regression example, where we switch between increasing parameters through  $P^{PC}$  and  $P^{ex}$  multiple times. This experiment is inspired by [CMBK21], who show one can "design your own generalization curve"
Figure 12: Evolution of the training error for different transitions between the two parameter-increasing mechanisms. Transitioning from  $P^{leaf}$  to  $P^{ens}$  at different values of  $P^{leaf}$  in the tree experiments (left), transitioning from  $P^{boost}$  to  $P^{ens}$  at different values of  $P^{boost}$  in the boosting experiments (middle) and transitioning from  $P^{PC}$  to  $P^{ex}$  at different values of  $P^{PC}$  in the regression experiments (right).
in linear regression by controlling the data-generating process through the order by which newly (un)informative features are revealed. In our case, rather than inducing the change by changing the data, we manipulate the location of the peak by simply changing the sequence in which parameter counts are increased.
Figure 13: "Design your own generalization curve". We show using the regression example that a generalization curve with arbitrarily many peaks and arbitrary peak locations (including peaks at  $P^{\phi} > n$ ) can be created simply by switching between increasing parameters through  $P^{PC}$  and  $P^{ex}$  multiple times (left). Train error remains monotonically decreasing (right).
E.3 Plots of binary test error
In Figs. 14 to 16, we revisit Figs. 2, 4 and 5 of the main text by plotting 0-1 (or binary) loss achieved on the test-set – measuring whether the class with the largest predicted probability indeed is the true class – instead of summed squared loss across classes as in the main text. We observe qualitatively identical behavior in all experiments also for this binary loss.
Figure 14: Decomposing double descent in binary test error for trees. Reproducing the tree experiments of [BHMM19] (left). Test error by  $P^{leaf}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{leaf}$  (right).
Figure 15: Decomposing double descent in binary test error for boosting. Reproducing the boosting experiments of [BHMM19] (left). Test error by  $P^{boost}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{boost}$  (right).
Figure 16: Decomposing double descent in binary test error for RFF regression. Reproducing the regression experiments of [BHMM19] (left). Test error by  $P^{PC}$  for fixed  $P^{ex}$  (center). Test error by  $P^{ex}$  for fixed  $P^{PC}$  (right).
E.4 Plots of training loss
In Figs. 17 to 19 we provide plots of summed squared training losses associated with the plots of summed squared test loss in presented Figs. 2, 4 and 5 in the main text.
Figure 17: Training error in the tree double descent experiments. Training error against [BHMM19]'s composite axis (left). Train error by  $P^{leaf}$  for fixed  $P^{ens}$  (center). Train error by  $P^{ens}$  for fixed  $P^{leaf}$  (right).
Figure 18: Training error in the boosting double descent experiments. Training error against [BHMM19]'s composite axis (left). Training error by  $P^{boost}$  for fixed  $P^{ens}$  (center). Training error by  $P^{ens}$  for fixed  $P^{boost}$  (right).
Figure 19: Training error in the RFF regression double descent experiments. Training error against [BHMM19]'s composite axis (left). Training error by  $P^{PC}$  for fixed  $P^{ex}$  (center). Training error by  $P^{ex}$  for fixed  $P^{PC}$  (right).
E.5 Investigating applications of  $p_{\mathrm{s}}^{test}$  to model selection
In this section, we test whether a measure like  $p_{\hat{\mathbf{s}}}^{\text{test}}$ , measuring the effective parameters used on the test inputs, could be used for model selection purposes, possibly providing some route for redemption for parameter count based selection criteria like [Aka74, Mal73, Sch78] that explicitly trade off training error and parameters used (and importantly, do not require access to held-out labels). We are motivated by empirical observations presented in Sec. 4. In particular, we note from Fig. 7 that across all three methods, there are models in the interpolation regime that perform very well in terms of generalization – and they happen to be those with the smallest  $p_{\hat{\mathbf{s}}}^{\text{test}}$ . This observation indicates that one straightforward model selection strategy when choosing between different hyperparameter settings that all achieve zero training error could be to compare models by their  $p_{\hat{\mathbf{s}}}^{\text{test}}$ .
Here, we test whether it is possible to exploit this in more general settings when there are multiple other hyperparameter axes to choose among. In particular, we consider tuning two other hyperparameters in gradient boosting, tree depth  $P^{leaf} \in \{10, 50, 100, 200, 500, \max\}$  and learning rate  $\eta \in \{0.02, 0.05, .1, .2, .3, .5, .85\}$ . We do so in single boosted models (i.e.  $P^{ens} = 1$ ), all trained until squared training error reaches approximately zero, i.e. we terminate after  $P^{boost} \in [1, 500]$  rounds, where  $P^{boost}$  is chosen as the first round (if any) where squared training error  $l_{train}$  drops below  $\epsilon = 10^{-4}$  (conversely, whenever  $l_{train} > \epsilon$  for all  $P^{boost} \leq 500$ , we do not consider the associated ( $P^{leaf}, \eta$ ) configurations an interpolating model).
Figure 20: Interpolating models with lower  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  are associated with lower test error. Test error vs  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$  for interpolating gradient-boosting models with of different  $(P^{leaf}, \eta)$ -combinations.
In Fig. 20 we observe that the intuition indeed carries over to this more general scenario and that interpolating models with lower  $p_{\mathrm{s}}^{\mathrm{test}}$  are indeed associated with lower test error. In particular, choosing the model with lowest  $p_{\mathrm{s}}^{\mathrm{test}}$  – which only requires access to test-time inputs  $x$  but no label information  $y$  – would indeed lead to the best test-performance among the interpolating models (with  $l_{train} < \epsilon$ ). Further, using a red cross, we also plot the performance of the best non-interpolating model (with  $l_{train} > \epsilon$ ) among the considered ( $P^{leaf}, \eta$ ) configurations; this is selected among the models with less boosting rounds  $P^{boost}$  by an oracle with access to test-time performance. We observe that this
non-interpolating oracle has essentially the same performance as the best interpolating model chosen by  $p_{\hat{\mathbf{s}}}^{\mathrm{test}}$ .
E.6 Results using other datasets
Below, we replicate our main experimental results using other datasets considered in [BHMM19], SVHN and CIFAR-10, exhibiting the same trends as the experiments on MNIST presented in the main paper.
E.6.1 SVHN
Figure 21: Decomposing double descent for trees on the SVHN dataset. Reproducing the tree experiment of [BHMM19] (left). Test error by  $P^{leaf}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{leaf}$  (right).
Figure 22: Decomposing double descent for gradient boosting on the SVHN dataset. Reproducing the boosting experiments of [BHMM19] (left). Test error by  $P^{boost}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{boost}$  (right).
Figure 23: Decomposing double descent for RFF Regression on the SVHN dataset. Reproducing the RFF regression experiments of [BHMM19] (left). Test error by  $P^{PC}$  for fixed  $P^{ex}$  (center). Test error by  $P^{ex}$  for fixed  $P^{PC}$  (right).
Figure 24: The effective number of parameters does not increase past the transition threshold on the SVHN dataset. Plotting  $p_{\mathrm{s}}^{\mathrm{train}}$  (orange) and  $p_{\mathrm{s}}^{\mathrm{test}}$  (green) for the tree (left), boosting (center) and RFF-linear regression (right) experiments, using the original composite parameter axes of [BHMM19].
Figure 25: Back to U (on the SVHN dataset). Plotting test error against the effective number of parameters as measured by  $p_{\mathrm{s}}^{\mathrm{test}}$ .
E.6.2 CIFAR-10
Figure 26: Decomposing double descent for trees on the CIFAR-10 dataset. Reproducing the tree experiment of [BHMM19] (left). Test error by  $P^{leaf}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{leaf}$  (right).
Figure 27: Decomposing double descent for gradient boosting on the CIFAR-10 dataset. Reproducing the boosting experiments of [BHMM19] (left). Test error by  $P^{boost}$  for fixed  $P^{ens}$  (center). Test error by  $P^{ens}$  for fixed  $P^{boost}$  (right).
Figure 28: Decomposing double descent for RFF Regression on the CIFAR-10 dataset. Reproducing the RFF regression experiments of [BHMM19] (left). Test error by  $P^{PC}$  for fixed  $P^{ex}$  (center). Test error by  $P^{ex}$  for fixed  $P^{PC}$  (right).
Figure 29: The effective number of parameters does not increase past the transition threshold on the CIFAR-10 dataset. Plotting  $p_{\mathrm{s}}^{\mathrm{train}}$  (orange) and  $p_{\mathrm{s}}^{\mathrm{test}}$  (green) for the tree (left), boosting (center) and RFF-linear regression (right) experiments, using the original composite parameter axes of [BHMM19].
Figure 30: Back to U (on the CIFAR-10 dataset). Plotting test error against the effective number of parameters as measured by  $p_{\mathrm{s}}^{\mathrm{test}}$ .