SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moritz Haas $^{1*}$  David Holzmüller $^{2*}$  Ulrike von Luxburg $^{1}$  Ingo Steinwart $^{2}$
<sup>1</sup>University of Tübingen and Tübingen AI Center, Germany
$^{2}$ Institute for Stochastics and Applications, University of Stuttgart, Germany
{mo.haas,ulrike.luxburg}@uni-tuebingen.de
{david.holzmueller,ingo.steinwart}@mathematik.uni-stuttgart.de
The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that rate-optimal benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.
While neural networks have shown great practical success, our theoretical understanding of their generalization properties is still limited. A promising line of work considers the phenomenon of benign overfitting, where researchers try to understand when and how models that interpolate noisy training data can generalize (Zhang et al., 2021, Belkin et al., 2018, 2019). In the high-dimensional regime, where the dimension grows with the number of sample points, consistency of minimum-norm interpolants has been established for linear models and kernel regression (Hastie et al., 2022, Bartlett et al., 2020, Liang and Rakhlin, 2020, Bartlett et al., 2021). In fixed dimension, minimum-norm interpolation with standard kernels is inconsistent (Rakhlin and Zhai, 2019, Buchholz, 2022).
In this paper, we shed a differentiated light on benign overfitting with kernels and neural networks. We argue that the dimension-dependent perspective does not capture the full picture of benign overfitting. In particular, we show that harmless interpolation with kernel methods and neural networks is possible, even in small fixed dimension, with adequately designed kernels and activation functions. The key is to properly design estimators of the form 'signal+spike'. While minimum-norm criteria have widely been considered a useful inductive bias, we demonstrate that designing unusual norms can resolve the shortcomings of standard norms. For wide neural networks, harmless interpolation can be
realized by adding tiny fluctuations to the activation function. Such networks do not require explicit regularization and can simply be trained to overfit (Figure 1).
Figure 1: Spiky-smooth overfitting in 2 dimensions. a. We plot the predicted function for ridgeless kernel regression with the Laplace kernel (blue) versus our spiky-smooth kernel (4) with Laplace components (orange) on  $\mathbb{S}^1$ . The dashed black line shows the true regression function, black 'x' denote noisy training points. Further details can be found in Section 6.2. b. The predicted function of a trained 2-layer neural network with ReLU activation (blue) versus ReLU plus shifted high-frequency sin-function (8) (orange). Using the weights learned with the spiky-smooth activation function in a ReLU network (green) disentangles the spike component from the signal component. c. Training error (solid lines) and test error (dashed lines) over the course of training for b. evaluated on  $10^{4}$  test points. The dotted black line shows the optimal test error. The spiky-smooth activation function does not require regularization and can simply be trained to overfit.
On a technical level, we additionally prove that overfitting in kernel regression can only be consistent if the estimators have large derivatives. Using neural tangent kernels or neural network Gaussian process kernels, we can translate our results from kernel regression to the world of neural networks (Neal, 1996, Jacot et al., 2018). In particular, our results enable the design of activation functions that induce benign overfitting in fixed dimension: the spikes in kernels can be translated into infinitesimal fluctuations that can be added to an activation function to achieve harmless interpolation with neural networks. Such small high frequency oscillations can fit noisy observations without affecting the smooth component too much. Training finite neural networks with gradient descent shows that spiky-smooth activation functions can indeed achieve good generalization even when interpolating small, low-dimensional data sets (Figure 1 b,c).
Thanks to new technical contributions, our inconsistency results significantly extend existing ones. We use a novel noise concentration argument (Lemma D.6) to generalize existing inconsistency results on minimum-norm interpolants to the much more realistic regime of overfitting estimators with comparable Sobolev norm scaling, which includes training via gradient flow and gradient descent with "late stopping" as well as low levels of ridge regularization. Moreover, a novel connection to eigenvalue concentration results for kernel matrices (Proposition 5) allows us to relax the smoothness assumption and to treat heteroscedastic noise in Theorem 6. Lastly, our Lemma E.1 translates inconsistency results from bounded open subsets of  $\mathbb{R}^d$  to the sphere  $\mathbb{S}^d$ , which leads to results for the neural tangent kernel and neural network Gaussian processes.
2 Setup and prerequisites
General approach. We consider a general regression problem on  $\mathbb{R}^d$  with an arbitrary, fixed dimension  $d$  and analyze kernel-based approaches to solve this problem: kernel ridge regression, kernel gradient flow and gradient descent, minimum-norm interpolation, and more generally, overfitting norm-bounded estimators. We then translate our results to neural networks via the neural network Gaussian process and the neural tangent kernel. Let us now introduce the formal framework.
Notation. We denote scalars by lowercase letters  $x$ , vectors by bold lowercase letters  $\pmb{x}$  and matrices by bold uppercase letters  $\pmb{X}$ . We denote the eigenvalues of  $\mathbf{A}$  as  $\lambda_1(\mathbf{A}) \geq \dots \geq \lambda_n(\mathbf{A})$  and the Moore-Penrose pseudo-inverse by  $A^+$ . We say that a probability distribution  $P$  has lower and upper bounded density if its density  $p$  satisfies  $0 < c < p(\pmb{x}) < C$  for suitable constants  $c, C$  and all  $\pmb{x}$  on a given domain.
Regression setup. We consider a data set  $D = ((\pmb{x}_1, y_1), \dots, (\pmb{x}_n, y_n)) \in (\mathbb{R}^d \times \mathbb{R})^n$  with i.i.d. samples  $(\pmb{x}_i, y_i) \sim P$ , written as  $D \sim P^n$ , where  $P$  is a probability distribution on  $\mathbb{R}^d \times \mathbb{R}$ . We define  $\pmb{X} := (\pmb{x}_1, \dots, \pmb{x}_n)$  and  $\pmb{y} := (y_1, \dots, y_n)^\top \in \mathbb{R}^n$ . Random variables  $(\pmb{x}, y) \sim P$  denote
test points independent of  $D$ , and  $P_{X}$  denotes the probability distribution of  $\pmb{x}$ . The (least squares) empirical risk  $R_{D}$  and population risk  $R_{P}$  of a function  $f:\mathbb{R}^d\to \mathbb{R}$  are defined as
$$
R _ {D} (f) := \frac {1}{n} \sum_ {i = 1} ^ {n} (y _ {i} - f (x _ {i})) ^ {2}, \qquad R _ {P} (f) := \mathbb {E} _ {\boldsymbol {x}, y} [ (y - f (\boldsymbol {x})) ^ {2} ].
$$
We assume  $\operatorname{Var}(y|\pmb{x}) < \infty$  for all  $\pmb{x}$ . Then,  $R_P$  is minimized by the target function  $f_P^*(\pmb{x}) = \mathbb{E}[y|\pmb{x}]$ , and the excess risk of a function  $f$  is given by
$$
R _ {P} (f) - R _ {P} (f _ {P} ^ {*}) = \mathbb {E} _ {\boldsymbol {x}} (f _ {P} ^ {*} (\boldsymbol {x}) - f (\boldsymbol {x})) ^ {2}.
$$
We call a data-dependent estimator  $f_{D}$  consistent for  $P$  if its excess risk converges to 0 in probability, that is, for all  $\varepsilon > 0$ ,  $\lim_{n\to \infty}P^n \left( D \in (\mathbb{R}^d \times \mathbb{R})^n \mid R_P(f_D) - R_P(f_P^*) \geq \varepsilon \right) = 0$ . We call  $f_{D}$  consistent in expectation for  $P$  if  $\lim_{n\to \infty} \mathbb{E}_D R_P(f_D) - R_P(f_P^*) = 0$ . We call  $f_{D}$  universally consistent if it is consistent for all Borel probability measures  $P$  on  $\mathbb{R}^d \times \mathbb{R}$ .
Solutions by kernel regression. Recall that a kernel  $k$  induces a reproducing kernel Hilbert space  $\mathcal{H}_k$ , abbreviated RKHS (more details in Appendix B). For  $f \in \mathcal{H}_k$ , we consider the objective
$$
\mathcal {L} _ {\rho} (f) := \frac {1}{n} \sum_ {i = 1} ^ {n} (y _ {i} - f (\pmb {x} _ {i})) ^ {2} + \rho \| f \| _ {\mathcal {H} _ {k}} ^ {2}
$$
with regularization parameter  $\rho \geq 0$ . Denote by  $f_{t,\rho}$  the solution to this problem that is obtained by optimizing on  $\mathcal{L}_{\rho}$  in  $\mathcal{H}_k$  with gradient flow until time  $t \in [0, \infty]$ , using fixed a regularization constant  $\rho > 0$ , and initializing at  $f = 0 \in \mathcal{H}_k$ . We show in Appendix C.1 that it is given as
$$
f _ {t, \rho} (\boldsymbol {x}) := k (\boldsymbol {x}, \boldsymbol {X}) \left(\boldsymbol {I} _ {n} - e ^ {- \frac {2}{n} t (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n})}\right) \left(k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {y}, \tag {1}
$$
where  $k(\pmb{x},\pmb{X})$  denotes the row vector  $(k(\pmb{x},\pmb{x}_i))_{i\in [n]}$  and  $k(\pmb{X},\pmb{X}) = (k(\pmb{x}_i,\pmb{x}_j))_{i,j\in [n]}$  the kernel matrix.  $f_{t,\rho}$  elegantly subsumes several popular kernel regression estimators as special cases: (i) classical kernel ridge regression for  $t\to \infty$ , (ii) gradient flow on the unregularized objective for  $\rho \searrow 0$ , and (iii) kernel "ridgeless" regression  $f_{\infty,0}(\pmb{x}) = k(\pmb{x},\pmb{X})k(\pmb{X},\pmb{X})^{+}\pmb{y}$  in the joint limit of  $\rho \rightarrow 0$  and  $t\rightarrow \infty$ . If  $k(\pmb{X},\pmb{X})$  is invertible,  $f_{\infty,0}$  is the interpolating function  $f\in \mathcal{H}_k$  with the smallest  $\mathcal{H}_k$ -norm.
From kernels to neural networks: the neural tangent kernel (NTK) and the neural network Gaussian process (NNGP). Denote the output of a NN with parameters  $\theta$  on input  $x$  by  $f_{\theta}(x)$ . It is known that for suitable random initializations  $\theta_0$ , in the infinite-width limit the random initial function  $f_{\theta_0}$  converges in distribution to a Gaussian Process with the so-called Neural Network Gaussian Process (NNGP) kernel (Neal, 1996, Lee et al., 2018, Matthews et al., 2018). In Bayesian inference, the posterior mean function is then of the form  $f_{\infty,\rho}$ . With minor modifications (Arora et al., 2019, Zhang et al., 2020), training infinitely wide NNs with gradient flow corresponds to learning the function  $f_{t,0}$  with the neural tangent kernel (NTK) (Jacot et al., 2018, Lee et al., 2019). If only the last layer is trained, the NNGP kernel should be used instead (Daniely et al., 2016). For ReLU activation functions, the RKHS of the infinite-width NNGP and NTK on the sphere  $\mathbb{S}^d$  is typically a Sobolev space (Bietti and Bach, 2021, Chen and Xu, 2021), see Appendix B.4. Using other parametrizations induces feature learning infinite-width limits for neural networks (Yang and Hu, 2021); an analysis of such neural network algorithms is left for future work.
We here provide a short summary of related work. A more detailed account is provided in Appendix A.
Kernel regression. With appropriate regularization, kernel ridge regularization with typical universal kernels like the Gauss, Matérn, and Laplace kernels is universally consistent (Steinwart and Christmann, 2008, Chapter 9). Optimal rates in Sobolev RKHS can also be achieved using cross-validation of the regularization  $\rho$  (Steinwart et al., 2009) or early stopping rules (Yao et al., 2007, Raskutti et al., 2014, Wei et al., 2017). The above kernels as well as NTKs and NNGPs of standard fully-connected neural networks are rotationally invariant. In the high-dimensional regime, the class of functions that is learnable with rotation-invariant kernels is quite limited (Donhauser et al., 2021, Ghorbani et al., 2021, Liang et al., 2020).
Inconsistency results. Besides Rakhlin and Zhai (2019) and Buchholz (2022), Beaglehole et al. (2023) derive inconsistency results for ridgeless kernel regression given assumptions on the spectral tail in the Fourier basis, and contemporaneously propose a special case of our spiky-smooth kernel sequence to mimic kernel ridge regression without providing any quantitative statements. Li et al. (2023) show that polynomial convergence is impossible for common kernels including ReLU NTKs. Mallinar et al. (2022) conjecture inconsistency for interpolation with ReLU NTKs based on their semirigorous result, which essentially assumes that the eigenfunctions can be replaced by structureless Gaussian random variables. Lai et al. (2023) show an inconsistency-type result for overfitting two-layer ReLU NNs with  $d = 1$ , but for fixed inputs  $X$ . They also note that an earlier inconsistency result by Hu et al. (2021) relies on an unproven result. Mücke and Steinwart (2019) show that global minima of NNs can overfit both benignly and harmfully, but their result does not apply to gradient descent training. Overfitting with typical linear models around the interpolation peak is inconsistent (Ghosh and Belkin, 2022, Holzmuller, 2021).
Classification. For binary classification, benign overfitting is a more generic phenomenon than for regression (Muthukumar et al., 2021, Shamir, 2022), and consistency has been shown under linear separability assumptions (Montanari et al., 2019, Chatterji and Long, 2021, Frei et al., 2022), through complexity bounds for reference classes (Cao and Gu, 2019, Chen et al., 2021) or as long as the total variation distance of the class conditionals is sufficiently large and  $f^{*}(\boldsymbol{x}) = \mathbb{E}[y|\boldsymbol{x}]$  lies in the RKHS with bounded norm (Liang and Recht, 2023). Chapter 8 of Steinwart and Christmann (2008) discusses how the overlap of the two classes may influence learning rates under positive regularization.
4 Inconsistency of overfitting with common kernel estimators
We consider a regression problem on  $\mathbb{R}^d$  in arbitrary, fixed dimension  $d$  that is solved by kernel regression. In this section, we derive several new results, stating that overfitting estimators with moderate Sobolev norm are inconsistent, in a variety of settings. In the next section, we establish the other direction: overfitting estimators can be consistent when we adapt the norm that is minimized.
4.1 Beyond minimum-norm interpolants: general overfitting estimators with bounded norm
Existing generalization bounds often consider the perfect minimum norm interpolant. This is a rather theoretical construction; estimators obtained by training with gradient descent algorithms merely overfit and, in the best case, approximate interpolants with small norm. In this section, we extend existing bounds to arbitrary overfitting estimators whose norm does not grow faster than the minimum norm that would be required to interpolate the training data. Before we can state the theorem, we need to establish some technical assumptions.
Assumptions on the data generating process. The following assumptions (as in Buchholz (2022)) allow for quite general domains and distributions. They are standard in nonparametric statistics.
(D1) Let  $P_{X}$  be a distribution on a bounded open Lipschitz domain  $\Omega \subseteq \mathbb{R}^d$  with lower and upper bounded Lebesgue density. Consider data sets  $D = \{(x_1, y_1), \ldots, (x_n, y_n)\}$ , where  $x_i \sim P_X$  i.i.d. and  $y_i = f^*(x_i) + \varepsilon_i$ , where  $\varepsilon_i$  is i.i.d. Gaussian noise with positive variance  $\sigma^2 > 0$  and  $f^* \in C_c^\infty(\Omega) \backslash \{0\}$  denotes a smooth function with compact support.
Assumptions on the kernel. Our assumption on the kernel is that its RKHS is equivalent to a Sobolev space. For integers  $s \in \mathbb{N}$ , the norm of a Sobolev space  $H^{s}(\Omega)$  can be defined as
$$
\| f \| _ {H ^ {s} (\Omega)} ^ {2} := \sum_ {0 \leq | \alpha | \leq s} \| D ^ {\alpha} f \| _ {L _ {2} (\Omega)} ^ {2},
$$
where  $D^{\alpha}$  denotes partial derivatives in multi-index notation for  $\alpha$ . It measures the magnitude of derivatives up to some order  $s$ . For general  $s > 0$ ,  $H^{s}(\Omega)$  is (equivalent to) an RKHS if and only if  $s > d / 2$ . For example, Laplace and Matérn kernels (Kanagawa et al., 2018, Example 2.6) have Sobolev RKHSs. The RKHS of the Gaussian kernel  $\mathcal{H}^{\mathrm{Gauss}}$  is contained in every Sobolev space,  $\mathcal{H}^{\mathrm{Gauss}} \subsetneq H^{s}$  for all  $s \geq 0$  (Steinwart and Christmann, 2008, Corollary 4.36). Due to its smoothness, the Gaussian kernel is potentially even more prone to harmful overfitting than Sobolev kernels (Mallinar et al., 2022). We make the following assumption on the kernel:
(K) Let  $k$  be a positive definite kernel function whose RKHS  $\mathcal{H}_k$  is equivalent to the Sobolev space  $H^s$  for some  $s \in \left(\frac{d}{2}, \frac{3d}{4}\right]$ .
Now we are ready to state the main result of this section:
Theorem 1 (Overfitting estimators with small norms are inconsistent). Let assumptions  $(D1)$  and  $(K)$  hold. Let  $c_{\mathrm{fit}} \in (0,1]$  and  $C_{\mathrm{norm}} > 0$ . Then, there exist  $c > 0$  and  $n_0 \in \mathbb{N}$  such that the following holds for all  $n \geq n_0$  with probability  $1 - O(1/n)$  over the draw of the data set  $D$  with  $n$  samples: Every function  $f \in \mathcal{H}_k$  that satisfies the following two conditions
$(O)$ $\frac{1}{n}\sum_{i = 1}^{n}(f(x_i) - y_i)^2\leq (1 - c_{\mathrm{fit}})\cdot \sigma^2$  (training error of  $f$  is below Bayes risk)
(N)  $\| f\|_{\mathcal{H}_k}\leq C_{\mathrm{norm}}\| f_{\infty ,0}\|_{\mathcal{H}_k}$  (norm comparable to minimum-norm interpolant (1)),
has an excess risk that satisfies
$$
R _ {P} (f) - R _ {P} \left(f ^ {*}\right) \geq c \sigma^ {2} > 0. \tag {2}
$$
In words: In fixed dimension  $d$ , every differentiable function  $f$  that overfits the training data and is not much "spikier" than the minimum RKHS-norm interpolant is inconsistent!
Proof idea. Our proof follows a similar approach as Rakhlin and Zhai (2019), Buchholz (2022), and also holds for kernels with adaptive bandwidths. For small bandwidths,  $\| f_{\infty,0} \|_{L_2(P_X)} \ll \| f^* \|_{L_2(P_X)}$  because  $f_{\infty,0}$  decays to 0 between the training points, which shows that purely 'spiky' estimators are inconsistent. In this case, the lower bound is independent of  $\sigma^2$ . For all other bandwidths, interpolating  $\Theta(n)$  many noisy labels  $y_i$  incurs  $\Theta(1)$  error in an area of volume  $\Omega(1/n)$  around  $\Theta(n)$  data points with high probability, which accumulates to a total error  $\Omega(1)$ . Our observation is that the same logic holds when overfitting by a constant fraction. Formally, we show that  $f^*$  and  $f$  must then be separated by a constant on a constant fraction of training points, with high probability, by using the fact that a constant fraction of the total noise cannot concentrate on less than  $\Theta(n)$  noise variables, with high probability (Lemma D.6). The full proof can be found in Appendix D.
Assumption (O) is necessary in Theorem 1, because optimally regularized kernel ridge regression fulfills all other assumptions of Theorem 1 while achieving consistency with minimax optimal convergence rates (see Section 3). The necessity of Assumption (N) is demonstrated by Section 5.
The following proposition establishes that Theorem 1 covers the entire overfitting regime of the popular (regularized) gradient flow estimators  $f_{t,\rho}$  for all times  $t\in [0,\infty ]$  and any regularization  $\rho \geq 0$ . The proof in Appendix C.2 also covers gradient descent.
Proposition 2 (Popular estimators fulfill the norm bound  $(\mathbf{N}))$ . For arbitrary  $t \in [0, \infty]$  and  $\rho \in [0, \infty)$ ,  $f_{t,\rho}$  as defined in (1) fulfills Assumption (N) with  $C_{\mathrm{norm}} = 1$ .
Remark 3 (Dimension dependency). Some works argue that for specific sequences of kernels  $(k_d)_{d\in \mathbb{N}}$ , the constant  $c$  in Theorem 1 decreases with increasing dimension  $d$  (Liang et al., 2020, Liang and Rakhlin, 2020, Mallinar et al., 2022). In Theorem 1, if the equivalence constants in Assumption (K) are uniformly bounded in  $d$ , the behavior in  $d$  might still depend on the definition of the Sobolev norms. Overall, similar to Rakhlin and Zhai (2019) and Buchholz (2022), our proof techniques do not allow to easily obtain a dependence on  $d$ .
4.2 Inconsistency of overfitting with neural kernels
We would now like to apply the above results to neural kernels, which would allow us to translate our inconsistency results from the kernel domain to neural networks. However, to achieve this, we need to take one more technical hurdle: the equivalence results for NTKs and NNGPs only hold for probability distributions on the sphere  $\mathbb{S}^d$  (detailed summary in Appendix B.4). Lemma E.1 provides the missing technical link: It establishes a smooth correspondence between the respective kernels, Sobolev spaces, and probability distributions. The inconsistency of overfitting with (deep) ReLU NTKs and NNGP kernels then immediately follows from adapting Theorem 1 via Lemma E.1.
Theorem 4 (Overfitting with neural network kernels in fixed dimension is inconsistent). Let  $c \in (0,1)$ , and let  $P$  be a probability distribution with lower and upper bounded Lebesgue density on an arbitrary spherical cap  $T \coloneqq \{ \pmb{x} \in \mathbb{S}^d \mid x_{d+1} < v \} \subseteq \mathbb{S}^d$ ,  $v \in (-1,1)$ . Let  $k$  either be
(i) the fully-connected ReLU NTK with 0-initialized biases of any fixed depth  $L \geq 2$ , and  $d \geq 2$ , or  
(ii) the fully-connected ReLU NNGP kernel without biases of any fixed depth  $L \geq 3$ , and  $d \geq 6$ .
Then, if  $f_{t,\rho}$  fulfills Assumption (O) with probability at least  $c$  over the draw of the data set  $D$ ,  $f_{t,\rho}$  is inconsistent for  $P$ .
Theorem 4 also holds for more general estimators as in Theorem 1, cf. the proof in Appendix E.
Mallinar et al. (2022) already observed empirically that overfitting common network architectures yields suboptimal generalization performance on large data sets in fixed dimension. Theorem 4 now provides a rigorous proof for this phenomenon since sufficiently wide trained neural networks and the corresponding NTKs have a similar generalization behavior (e.g. (Arora et al., 2019, Theorem 3.2)).
4.3 Relaxing smoothness and noise assumptions via spectral concentration bounds
In this section, we consider a different approach to derive lower bounds for the generalization error of overfitting kernel regression: through concentration results for the eigenvalues of kernel matrices. On a high level, we obtain similar results as in the last section. The novelty of this section is on the technical side, and we suggest that non-technical readers skip this section in their first reading.