SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To understand how the raw number of features  $P^{\phi}$  in the experimental setup of Sec. 3.2 in the main text impacts basis quality, it is thus instructive to study the behavior of the singular values of the random Fourier feature matrix  $\Phi$  as  $P^{\phi}$  grows. In Fig. 10, we therefore plot  $\sigma_{k}$  and  $\kappa(\mathbf{B}_{k})$  for different values of  $P^{\phi}$  and observe that indeed, as  $P^{\phi}$  (and thereby  $P^{ex} = P^{\phi} - k$ ) increases,  $\sigma_{k}$  substantially increases in magnitude especially for large  $k \equiv P^{PC}$ . This provides empirical evidence that the conditioning of feature matrices for principal components regression at large values of  $k$  indeed substantially improves as we add excess raw features (i.e.  $\kappa(\mathbf{B}_{k})$  decreases substantially – particularly at large  $k$  – as can be seen in the right panel of Fig. 10).
C More on smoothers: Bias-variance, on the impossibility of in-sample double descent and derivation of smoothing matrices
In this section, we give some further background on the well-known relationship between bias and variance in smoothers, then discuss the impossibility of double descent in prediction error in fixed design settings, and finally derive the smoothing matrices implied by the ML methods used in the main text.
C.1 Background: Bias and variance in smoothers
In this section, we discuss the well-known bias-variance relationship in smoothers (which indeed determines the U-shape of their original complexity-error curves), following the discussion in [HT90, Sec. 3.3]. Under the standard assumption that targets are generated with true expectation  $f^{*}(x_{0})$  and homoskedastic variance  $\sigma^2$ , the bias and variance of a smoother  $\hat{\mathbf{s}} (\cdot)$  evaluated at test point  $x_0$  (for
fixed input points  $\{x_{i}\}_{i\in \mathcal{I}_{\mathrm{train}}}$  and fixed smoother  $\hat{\mathbf{s}} (\cdot)$ , can be written as:
$$
\operatorname {B i a s} (\hat {f}, x _ {0}) = f ^ {*} (x _ {0}) - \sum_ {i \in \mathcal {I} _ {\text {t r a i n}}} \hat {s} ^ {i} (x _ {0}) f ^ {*} (x _ {i}) \tag {3}
$$
$$
\operatorname {V a r} (\hat {f}, x _ {0}) = \operatorname {V a r} (\hat {s} (x _ {0}) \mathbf {y} _ {\text {t r a i n}}) = | | \hat {s} (x _ {0}) | | ^ {2} \sigma^ {2} \tag {4}
$$
Note that some of the historical intuition behind the U-shaped bias-variance tradeoff with model complexity on the x-axis appears to lead back to exactly the comparison between these two terms evaluated for training inputs in smoothers -[Nea19] trace explicit discussion of bias-variance tradeoff back to at least [HT90] (who indeed discuss it in the context of smoothers), preceding [GBD92]'s influential machine learning paper on the topic. A bias-variance tradeoff is easily illustrated by considering e.g. a kNN smoother: a 1NN estimator has no bias when predicting outcomes for a training input but variance  $\sigma^2$ , while a  $k(>1)$  NN estimator generally incurs some bias but has variance  $\sigma^2 / k$  - we thus expect a U-shaped curve in squared prediction error as we vary  $k$ . Similarly, note that for linear regression on a  $p < n$ -dimensional subset of the input, the bias of the fit generally decreases in  $p$ , while the variance increases in  $p$  as  $\frac{1}{n} \sum_{i=1}^{n} Var(\hat{s}(x_i)) = \frac{p \sigma^2}{n}$  [HTF09] - leading to another well-known example of a U-shaped generalization curve (in  $p < n$ ).
When the expected mean-squared error (MSE) is the loss function of interest, a bias-variance tradeoff can also be motivated more explicitly by noting that the MSE can be decomposed as
$$
\mathbb {E} \left[ \left(Y - \hat {f} \left(x _ {0}\right)\right) ^ {2} \mid X = x _ {0} \right] = \sigma^ {2} + \operatorname {B i a s} ^ {2} (\hat {f}, x _ {0}) + \operatorname {V a r} (\hat {f}, x _ {0}) \tag {5}
$$
C.2 On the impossibility of a second prediction error descent in the interpolation regime in fixed design settings
[BHMM19] provide a discussion of possible cultural and practical reasons for the historical absence of double descent shapes in the statistics literature. They note that observing double descent in the number of parameters requires parametric families in which complexity can arbitrarily be controlled and that therefore the classical statistics literature – which either focuses either on linear regression in fixed  $p < n$  settings or considers highly flexible nonparametric classes but then applies heavy regularization – could not have observed it because of this cultural focus on incompatible model classes (in which [BHMM19] argue double descent cannot emerge due to a lack of interpolation). Through our investigation, we find another fundamental reason for the historical absence of double descent shapes in statistics: even when using a sufficiently flexible parametric class (e.g. unregularized RFF linear regression), a second descent in the interpolation regime cannot emerge in fixed design settings, the primary setting considered in the early statistics literature – where much of the intuition regarding the bias-variance tradeoff was initially established.
After recalling some definitions in Appendix C.2.1, we demonstrate this impossibility of a second descent in fixed design settings for general interpolating predictors in Appendix C.2.2 below, and make some additional observations specific to smoothers in Appendix C.2.3.
C.2.1 Preliminaries.
We begin by recalling the definition of the fixed design setting of statistical learning.
Definition 2 (Fixed design setting). In the fixed design setting, the inputs  $x_{1}, \ldots, x_{n}$  are deterministic, while their associated targets are random (e.g.  $y_{i} = f(x_{i}) + \epsilon_{i}$  where  $\epsilon_{i}$  is a random noise term).
The fixed design setting was a core consideration of much of the statistical literature [RT19] and was suitable for several traditional applications in e.g. agriculture. While many modern applications require a random design setting in which the inputs are also random, several modern applications are still suitably modeled within this framework (e.g. image denoising). Next, we will define in-sample prediction error in line with [HTF09].
Definition 3 (In-sample prediction error). Given a set of fixed design points  $x_{1},\ldots ,x_{n}$ , training targets  $y_{1},\ldots ,y_{n}$  and some model  $f$ , suppose we observe a new set of random responses at each of
Recall that, possibly counterintuitively, a kNN smoother is most complex when  $k = 1$ , where at every training input the prediction is simply the original training label, and least complex when  $k = n$ , where it just outputs a sample average
the training points denoted  $y_{i}^{0}$ . Then the expected in-sample prediction error for some loss function  $\mathcal{L}$  is given by
$$
E R R _ {i n} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {y ^ {0}} \left[ \mathcal {L} \left(y _ {i} ^ {0}, f \left(x _ {i}\right)\right) \mid \left\{\left(x _ {1}, y _ {1}\right), \dots , \left(x _ {n}, y _ {n}\right) \right\} \right]
$$
Remark: Note that in the fixed design setting, the expected in-sample prediction error is indeed the natural quantity of interest at test time.
Finally, we define an interpolating predictor below.
Definition 4 (Interpolating predictor). A predictor  $\hat{f}(\cdot)$  interpolates the training data if for  $y_i^{train} \in \mathbb{R}, \forall i \in \mathcal{I}_{train}$
$$
\hat {f} (x _ {i}) = y _ {i} ^ {\text {t r a i n}} \tag {6}
$$
C.2.2 Main result: General impossibility of a second descent in test error for the interpolation regime in fixed design settings
We may now proceed to the main observation of this section. That is, in the fixed design setting a second descent in test error observed in the interpolation regime cannot occur. Intuitively, this is because any improvement in fixed-design test-time performance by a more complex model would require sacrificing the perfect performance on the train set. As a consequence, while - as we observe in the main text - different interpolating predictors can have wildly different (out-of-sample) generalization performance, we show below that all models in the interpolation regime have identical expected in-sample prediction error  $\mathrm{ERR}_{\mathrm{in}}$ . That is, increasing the raw number of parameters  $p$  past  $n$ , e.g. in min-norm linear regression, does not and can not lead to a second descent in  $\mathrm{ERR}_{\mathrm{in}}$ . We express this more formally below.
Proposition 2 (Impossibility of a second error descent in the interpolation regime for fixed design settings.). Suppose we observe inputs  $x_{1}, \ldots, x_{n}$  in a fixed design prediction problem, with associated random labels  $y_{1}^{\text{train}}$ ,  $\ldots$ ,  $y_{n}^{\text{train}}$  in the training set. For all interpolating models  $f$ , test-time predictions will be identical and, therefore, a second descent in  $\text{ERR}_{in}$  in this regime is impossible.
Proof. By definition of fixed design problems, our test set consists of examples in which the features are duplicates from the training set and the targets are new realizations from some target distribution. Then the test dataset consists of the pairs  $(x_{1},y_{1}^{\mathrm{test}}),\ldots ,(x_{n},y_{n}^{\mathrm{test}})$ . For any interpolating model  $f$  we can calculate the test loss as  $\mathcal{L}^{\mathrm{test}} = \sum_{i = 1}^{n}\mathcal{L}(f(x_i),y_i^{\mathrm{test}})$ . But by Definition 4 (and because interpolating models achieve zero training error), we know that  $f(x_{i}) = y_{i}^{\mathrm{train}}$ . Thus  $\mathcal{L}^{\mathrm{test}} = \sum_{i = 1}^{n}\mathcal{L}(y_{i}^{\mathrm{train}},y_{i}^{\mathrm{test}})$ , and since this is not a function of  $f$ , this quantity is independent of the number of parameters (or any other property) of  $f$ . Therefore, any change in  $\mathcal{L}^{\mathrm{test}}$  (or expectations thereof) in the interpolation regime is impossible - including the second descent necessary for creating a double descent shape.
While most, if not all, discussion of the double descent phenomenon in the ML literature is not within the fixed design setting, we believe that the impossibility of double descent in this setting provides a new useful partial explanation for its historical absence from the statistical discourse.
C.2.3 Additional observations specific to smoothers
For the special case of interpolating linear smoothers, we make an additional interesting observation with regard to their smoothing weights and implied effective number of parameters. To do so, first recall that a linear smoother is defined as follows:
Definition 5 (Linear smoother). A smoother with weights  $\hat{\mathbf{s}}(x_0) \equiv \hat{\mathbf{s}}(x_0; \mathbf{X}_{train}, \mathbf{y}_{train})$  is linear if  $\hat{\mathbf{s}}(x_0)$  does not depend on  $\mathbf{y}_{train}$ ; that is  $\hat{\mathbf{s}}(x_0; \mathbf{X}_{train}, \mathbf{y}^1) = \hat{\mathbf{s}}(x_0; \mathbf{X}_{train}, \mathbf{y}^2)$  for  $\forall \mathbf{y}^1, \mathbf{y}^2 \in \mathbb{R}^n$ .
We are now ready to show in Prop. 3 that all interpolating linear smoothers must have the same smoother weights and effective parameters for training inputs.
Proposition 3 (Interpolating linear smoother.). Any interpolating linear smoother must have identical smoother weights  $\hat{\mathbf{s}}(x_i) = \mathbf{e}_i$  for all  $i \in \mathcal{I}_{\text{train}}$ . This immediately implies also that all interpolating linear smoothers use the same effective number of parameters  $p_e = n$  in fixed design settings.
Proof. In linear smoothers,  $\hat{\mathbf{s}}(x_i)$  can not depend on  $\mathbf{y}_{\mathrm{train}}$  by definition. To have  $\hat{\mathbf{s}}(x_i)\mathbf{y}_{\mathrm{train}}' = y_i'$  for all  $i \in \mathcal{I}_{\mathrm{train}}$  and any possible training outcome realization  $\forall \mathbf{y}_{\mathrm{train}}' \in \mathbb{R}^n$ , we must have  $\hat{\mathbf{s}}(x_i) = \mathbf{e}_i$  for  $i \in \mathcal{I}_{\mathrm{train}}$ .
From this, as  $||\mathbf{e}_i||^2 = 1$ , it also follows that  $p_{\hat{\mathbf{s}}}^{\mathrm{train}} = p_e = \sum_{i=1}^{n} ||\mathbf{e}_i||^2 = n$ . In fixed design settings, training and testing inputs are identical so  $p_{\hat{\mathbf{s}}}^{\mathrm{test}} = n$ .
Note that this specific property does not hold for more general (non-linear) interpolating smoothers: whenever  $\mathbf{y}_{train}$  contains duplicates, a non-linear interpolating smoother can have  $\hat{\mathbf{s}}(x_i) \neq \mathbf{e}_i$  for some  $i \in \mathcal{I}_{\mathrm{train}}$ . When all  $y_i$  observed at train-time are unique, then it is easy to see that even non-linear interpolating smoothers must have  $\hat{\mathbf{s}}(x_i) = \mathbf{e}_i$  for all  $i \in \mathcal{I}_{\mathrm{train}}$ .
C.3 Derivation of smoothing weights of linear regression, trees and boosting
C.3.1 Smoothing weights for RFF linear regression
Derivation of the smoothing weights for the RFF linear regression example is trivial, as linear regression is a textbook example of a smoother [HT90]. To see this, let  $\phi(x_0) = (\phi_1(x_0), \ldots, \phi_P(x_0))$  denote the  $P^{\phi} \times 1$  row-vector of RFF-projections of arbitrary admissible  $x_0 \in \mathbb{R}^d$  and let  $\Phi_{\mathrm{train}} = [\phi(x_1)^T, \ldots, \phi(x_n)^T]^T$  denote the  $n \times P^{\phi}$  design matrix of the training set and note that the least squares prediction obtained through the min-norm solution can be written as
$$
\hat {f} ^ {L R} \left(x _ {0}\right) = \phi \left(x _ {0}\right) \beta_ {L R} = \hat {s} ^ {L R} \left(x _ {0}\right) \mathbf {y} _ {\text {t r a i n}} = \left\{ \begin{array}{l l} \phi \left(x _ {0}\right) \left(\boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} \boldsymbol {\Phi} _ {\text {t r a i n}}\right) ^ {- 1} \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} \mathbf {y} _ {\text {t r a i n}} & \text {i f} P ^ {\phi} <   n \\ \phi \left(x _ {0}\right) \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} \left(\boldsymbol {\Phi} _ {\text {t r a i n}} \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T}\right) ^ {- 1} \mathbf {y} _ {\text {t r a i n}} & \text {i f} P ^ {\phi} \geq n \end{array} \right. \tag {7}
$$
$$
\hat {\mathbf {s}} ^ {L R} \left(x _ {0}\right) = \left\{ \begin{array}{l l} \phi \left(x _ {0}\right) \left(\boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} \boldsymbol {\Phi} _ {\text {t r a i n}}\right) ^ {- 1} \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} & \text {i f} P ^ {\phi} <   n \\ \phi \left(x _ {0}\right) \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T} \left(\boldsymbol {\Phi} _ {\text {t r a i n}} \boldsymbol {\Phi} _ {\text {t r a i n}} ^ {T}\right) ^ {- 1} & \text {i f} P ^ {\phi} \geq n \end{array} \right. \tag {8}
$$
Note also that by design, as alluded to in Appendix C.2.3, when  $P^{\phi} \geq n$  we have  $\Phi_{\mathrm{train}} \Phi_{\mathrm{train}}^{T} (\Phi_{\mathrm{train}} \Phi_{\mathrm{train}}^{T})^{-1} = \mathbf{I}_{n}$ , implying that that  $\hat{\mathbf{s}}^{LR}(x_i) = \mathbf{e}_i$  for any  $x_i$  observed at train time.
C.3.2 Smoothing weights for trees and forests
Similarly, we can exploit that trees are sometimes considered adaptive nearest neighbor methods [HTF09, Ch. 15.4.3], i.e. nearest neighbor estimators with an adaptively constructed (outcome-oriented) kernel (or distance measure). To see this, note that regression trees, that is trees trained to minimize the squared loss and/or classification trees that make predictions through averaging (not voting), can all be understood as issuing predictions that simply average across training instances within a terminal leaf. To make this more formal, consider that any tree with  $P^{leaf}$  leaves can be represented by  $P^{leaf}$  contiguous axis-aligned hypercubes  $l_{p}$ , which store the mean outcome of all training examples that fall into this leaf: If we denote by  $l(x_0)$  the leaf that  $x_0$  falls into and  $n_{l(x_0)} = \sum_{i=1}^{n} \mathbf{1}\{l(x_0) = l(x_i)\}$  the number of training examples in this leaf, then we have that
$$
\hat {f} \left(x _ {0}\right) = \frac {1}{n _ {l \left(x _ {0}\right)}} \sum_ {i = 1} ^ {n} \mathbf {1} \left\{l \left(x _ {0}\right) = l \left(x _ {i}\right) \right\} y _ {i} \tag {9}
$$
$$
\hat {\mathbf {s}} ^ {t r e e} \left(x _ {0}\right) = \left(\frac {\mathbf {1} \left\{l \left(x _ {0}\right) = l \left(x _ {1}\right) \right\}}{n _ {l \left(x _ {0}\right)}}, \dots , \frac {\mathbf {1} \left\{l \left(x _ {0}\right) = l \left(x _ {1}\right) \right\}}{n _ {l \left(x _ {0}\right)}}\right) \tag {10}
$$
Further, we note that sums  $\hat{f}_{\Sigma}(x_0) = \sum_{m=1}^{M} \hat{f}^m(x_0)$  of smoothers  $\hat{f}^m(x_0) = \hat{s}^m(x_0)\mathbf{y}_{\mathrm{train}}$ ,  $m = 1, \ldots, M$  are also smoothers with  $\hat{s}^{\Sigma}(x_0) = \sum_{m=1}^{M} \hat{s}^m(x_0)$ . This immediately implies that the tree ensembles considered in Sec. 2.1 are also smoothers, and, as we show in Appendix C.3.3 below, so are boosted trees (and ensembles thereof) as considered in Sec. 2.2.
Remark: We note that, although the smooth  $\hat{s}^{tree}(x_0)$  can be written in linear form without explicit reference to  $\mathbf{y}_{\mathrm{train}}$ , trees (and their sums) are not truly linear smoothers because the smoother  $\hat{s}^{tree}(\cdot)$  does depend in its construction on  $\mathbf{y}_{\mathrm{train}}$ , reflecting their epithet as adaptive smoothers [HTF09, Ch.
15.4.3]. This mainly implies that when calculating bias and variance, we can no longer consider  $\hat{\mathbf{s}}(x_0)$  as fixed and Eqs. (3) and (4) no longer hold exactly. However, this is not an issue for our purposes as we are not interested in actually estimating bias and variance in this work, rather we are interested in smoothers because - as discussed in Sec. 4 and Appendix D - they allow us to compute the most useful measure of effective parameters. Conceptually, this allows us to assess the effective amount of smoothing used, which can also be interpreted relative to e.g. a standard kNN estimator. Because for kNN estimators we have  $p_{\hat{\mathbf{s}}}^0 = \frac{n}{k}$  and for trees and forests it holds that  $\sum_{i=1}^{n} \hat{s}^i(x_0) = 1$ , the quantity  $\tilde{k}_{\hat{\mathbf{s}}}^0 = \frac{n}{p_{\hat{\mathbf{s}}}^0}$  admits an interesting interpretation as measuring the effective number of nearest neighbors examples have in a tree or forest (and this interpretation does not require fixed  $\hat{\mathbf{s}}$ ).
C.3.3 Smoothing weights for boosting
Deriving the smoothing weights for gradient boosting is more involved. To do so, we first re-state the gradient boosting algorithm (as presented in [HTF09, Ch. 10.10]) and then recursively construct weights from it.
The gradient boosted trees algorithm. We consider gradient boosting with learning rate  $\eta$  and the squared loss as  $L(\cdot, \cdot)$ :
1. Initialize  $f_0(x) = 0$
2. For  $p\in \{1,\ldots ,P^{boost}\}$
(a) For  $i\in \{1,\ldots ,n\}$  compute
$$
g _ {i, p} = - \left[ \frac {\partial L \left(y _ {i} , f \left(x _ {i}\right)\right)}{\partial f \left(x _ {i}\right)} \right] _ {f = f _ {p - 1}} \tag {11}
$$
(b) Fit a regression tree to  $\{(x_{i},g_{i,p})\}_{i = 1}^{n}$ , giving leaves  $l_{jp}$  for  $j = 1,\dots ,J_{p}$
(c) Compute optimal predictions for each leaf  $j \in \{1, \dots, J_p\}$ :
$$
\gamma_ {j p} = \arg \min  _ {\gamma \in \mathbb {R}} \sum_ {x _ {i} \in l _ {j p}} L \left(y _ {i}, f _ {p - 1} \left(x _ {i}\right) + \gamma\right) = \frac {1}{n _ {l _ {j p}}} \sum_ {x _ {i} \in l _ {j p}} \left(y _ {i} - f _ {p - 1} \left(x _ {i}\right)\right) \tag {12}
$$
(d) Denote by  $\tilde{f}_p(x) = \sum_{j=1}^{J_p} \mathbf{1}\{x \in l_{jp}\} \gamma_{jp}$  the predictions of the tree built in this fashion.
(e) Set  $f_{p}(x) = f_{p - 1}(x) + \eta \tilde{f}_{p}(x)$
3. Output  $f(x) = f_{P^{boost}}(x)$
Recursive construction of smoothing weights. Equation (12) highlights that we can recursively construct smoothing weights similarly as for trees. In fact, note that in the first boosting round,  $\tilde{f}_1(x)$  is a standard tree with smoothing weights  $\hat{\mathbf{s}}^{tree,\tilde{f}_1}(\cdot)$  computed as in Eq. (10), so that  $f_{1}(x) = \eta \tilde{f}_{1}(x)$  has smoothing weights  $\hat{\mathbf{s}}^{boost,1}(\cdot) = \eta \hat{\mathbf{s}}^{tree,\tilde{f}_1}(\cdot)$ . Now we can recursively consider the smoothing weights associated with newly added residual trees  $\tilde{f}_p(x)$  by realizing that its predictions can be written as a function of standard tree weights  $\hat{\mathbf{s}}^{tree,\tilde{f}_p}(\cdot)$  and a correction depending on  $\hat{\mathbf{s}}^{boost,1}(\cdot)$  alone:
$$
\begin{array}{l} \tilde {f} _ {p} (x) = \sum_ {j = 1} ^ {J _ {p}} \mathbf {1} \{x \in l _ {j p} \} \gamma_ {j p} = \sum_ {j = 1} ^ {J _ {p}} \mathbf {1} \{x \in l _ {j p} \} \left(\frac {1}{n _ {l _ {j p}}} \sum_ {x _ {i} \in l _ {j p}} \left(y _ {i} - f _ {p - 1} \left(x _ {i}\right)\right)\right) \\ = \underbrace {\sum_ {j = 1} ^ {J _ {p}} \mathbf {1} \{x \in l _ {j p} \} \left(\frac {1}{n l _ {j p}} \sum_ {x _ {i} \in l _ {j p}} y _ {i}\right)} _ {\text {S t a n d a r d t r e e p r e d i c t i o n :} \hat {\mathbf {s}} ^ {t r e e, \bar {f} _ {p}} (x) \mathbf {y} _ {t r a i n}} - \underbrace {\sum_ {j = 1} ^ {J _ {p}} \mathbf {1} \{x \in l _ {j p} \} \left(\frac {1}{n l _ {j p}} \sum_ {x _ {i} \in l _ {j p}} \hat {\mathbf {s}} ^ {b o o s t , p - 1} \left(x _ {i}\right) \mathbf {y} _ {t r a i n}\right)} _ {\text {R e s i d u a l c o r r e c t i o n :} \hat {\mathbf {s}} ^ {c o r r, f _ {p}} (x) \mathbf {y} _ {t r a i n}} (13) \\ = \left(\hat {\mathbf {s}} ^ {t r e e, \tilde {f} _ {p}} (x) - \hat {\mathbf {s}} ^ {c o r r, f _ {p - 1}} (x)\right) \mathbf {y} _ {t r a i n} (14) \\ \end{array}
$$
where, using the  $1 \times J_{p}$  indicator vector  $\mathbf{e}_{l^{P}(x)} = (\mathbf{1}\{x \in l_{1}\}, \ldots, \mathbf{1}\{x \in l_{J_{p}}\})$ , we have
$$
\hat {\mathbf {s}} ^ {c o r r, f _ {p}} (x) = \mathbf {e} _ {l ^ {p} (x)} \hat {\mathbf {R}} ^ {p} \tag {15}
$$
with  $\hat{\mathbf{R}}^p$  the  $J_{p} \times n$  leaf-residual correction matrix with  $j$ -th row given by
$$
\hat {\mathbf {R}} ^ {p} = \frac {1}{n _ {l _ {j p}}} \sum_ {x _ {i} \in l _ {j p}} \hat {\mathbf {s}} ^ {\text {b o o s t ,} p - 1} \left(x _ {i}\right) \tag {16}
$$
Thus, we have constructed the smoothing matrix for gradient boosting recursively as:
$$
\hat {\mathbf {s}} ^ {b o o s t, p} (\cdot) = \hat {\mathbf {s}} ^ {b o o s t, p - 1} (\cdot) + \eta \left(\hat {\mathbf {s}} ^ {t r e e, \tilde {f} _ {p}} (\cdot) - \hat {\mathbf {s}} ^ {c o r r, f _ {p}} (\cdot)\right) \tag {17}
$$
D More on effective parameters
In what follows, we discuss the different effective parameter definitions considered in the smoothing literature, and follow closely the presentation in [HT90, Ch.3.4 & 3.5] and [HTF09, Ch. 7.5]. Recall that, as discussed in Sec. 4 of the main text, these definitions are all naturally calibrated towards linear regression so that, as we might desire, effective and raw parameter numbers are equal in the case of ordinary linear regression with  $p < n$ .
To understand their motivation, we introduce some further notation. Assume that outcomes are generated as  $y = f^{*}(x) + \epsilon$  with unknown expectation  $f^{*}(x)$  and  $\epsilon$  a zero-mean error term with variance  $\sigma^2$ . Let  $\mathbf{f}^{*} = [f^{*}(x_{1}),\dots ,f^{*}(x_{n})]^{T}$  denote the vector of true expectations. Further, let  $\hat{\mathbf{S}} = [\hat{\mathbf{s}} (x_1)^T,\dots ,\hat{\mathbf{s}} (x_n)^T ]^T$  denote the smoother matrix for all input points, so that we can write the vector of in-sample predictions as  $\hat{\mathbf{f}} = \hat{\mathbf{S}}\mathbf{y}_{\mathrm{train}}$ . For linear smoothers (and/or fixed  $\hat{\mathbf{S}}$ ), bias and variance at any input point are given by Eqs. (3) and (4), so that we have  $\frac{1}{n}\sum_{i\in \mathcal{I}_{train}}bias^2 (\hat{f} (x_i)) = \frac{1}{n}\mathbf{b}_\hat{\mathbf{S}}^T\mathbf{b}_{\hat{\mathbf{S}}}$  for bias vector  $\mathbf{b}_{\hat{\mathbf{S}}} = (\mathbf{f}^{*} - \hat{\mathbf{S}}\mathbf{f}^{*})$  and  $\frac{1}{n}\sum_{i\in \mathcal{I}_{train}}var(\hat{f} (x_i)) = \frac{\sigma^2}{n} tr(\hat{\mathbf{S}}\hat{\mathbf{S}}^T)$ .