SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We define the convolution kernel of a given kernel  $k$  as  $k_{*}(\pmb{x},\pmb{x}^{\prime}) \coloneqq \int k(\pmb{x},\pmb{x}^{\prime \prime})k(\pmb{x}^{\prime \prime},\pmb{x}^{\prime})\mathrm{d}P_{X}(\pmb{x}^{\prime \prime})$ , which is possible whenever  $k(\pmb{x},\cdot) \in L_2(P_X)$  for all  $\pmb{x}$ . The latter condition is satisfied for bounded kernels. Our starting point is the following new lower bound:
Proposition 5 (Spectral lower bound). Assume that the kernel matrix  $k(\mathbf{X}, \mathbf{X})$  is almost surely positive definite, and that  $\operatorname{Var}(y|\mathbf{x}) \geq \sigma^2$  for  $P_X$ -almost all  $\mathbf{x}$ . Then, the expected excess risk satisfies
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {t, \rho}\right) - R _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {X}} \frac {\lambda_ {i} \left(k _ {*} (\boldsymbol {X} , \boldsymbol {X}) / n\right) \left(1 - e ^ {- 2 t \left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho\right)\right) ^ {2}}{\left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho\right) ^ {2}}. \tag {3}
$$
Using concentration inequalities for kernel matrices and the relation between the integral operators of  $k$  and  $k_{*}$ , it can be seen that for  $t = \infty$  and  $\rho = 0$ , every term in the sum in Eq. (3) should converge to 1 as  $n \to \infty$ . However, since the number of terms in the sum increases with  $n$  and the convergence may not be uniform, this is not sufficient to show inconsistency in expectation. Instead, relative concentration bounds that are even stronger than the ones by Valdivia (2018) would be required to show inconsistency in expectation. However, by combining multiple weaker bounds and further arguments on kernel equivalences, we can still show inconsistency in expectation for a class of dot-product kernels on the sphere, including certain NTK and NNGP kernels (Appendix B.4):
Theorem 6 (Inconsistency for Sobolev dot-product kernels on the sphere). Let  $k$  be a dot-product kernel on  $\mathbb{S}^d$ , i.e., a kernel of the form  $k(\pmb{x},\pmb{x}^{\prime}) = \kappa (\langle \pmb{x},\pmb{x}^{\prime}\rangle)$ , such that its RKHS  $\mathcal{H}_k$  is equivalent to a Sobolev space  $H^{s}(\mathbb{S}^{d})$ ,  $s > d / 2$ . Moreover, let  $P$  be a distribution on  $\mathbb{S}^d\times \mathbb{R}$  such that  $P_{X}$  has a lower and upper bounded density w.r.t. the uniform distribution  $\mathcal{U}(\mathbb{S}^d)$ , and such that  $\mathrm{Var}(y|\pmb {x})\geq \sigma^2 >0$  for  $P_{X}$ -almost all  $\pmb {x}\in \mathbb{S}^d$ . Then, for every  $C > 0$ , there exists  $c > 0$  independent of  $\sigma^2$  such that for all  $n\geq 1$ ,  $t\in (C^{-1}n^{2s / d},\infty ]$ , and  $\rho \in [0,Cn^{-2s / d})$ , the expected excess risk satisfies
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {t, \rho}\right) - R _ {P} ^ {*} \geq c \sigma^ {2} > 0.
$$
The assumptions of Theorem 6 and Theorem 4 differ in several ways. Theorem 6 applies to arbitrarily high smoothness  $s$  and therefore to ReLU NTKs and NNGPs in arbitrary dimension  $d$ . Moreover, it applies to distributions on the whole sphere and allows more general noise distributions. On the flip side, it only shows inconsistency in expectation, which we believe could be extended to inconsistency for Gaussian noise. Moreover, it only applies to functions of the form  $f_{t,\rho}$  but provides an explicit bound on  $t$  and  $\rho$  to get inconsistency. For  $t = \infty$ , the bound  $\rho = O(n^{-2s / d})$  appears to be tight, as larger  $\rho$  yield consistency for comparable Sobolev kernels on  $\mathbb{R}^d$  (Steinwart et al., 2009, Corollary 3).
We only prove Theorem 6 for dot-product kernels on the sphere since we can show for these kernels that  $\mathcal{H}_{k_*}$  is equivalent to a Sobolev space (Lemma F.13), while this is not true for open domains  $\Omega$  (Schaback, 2018). However, an improved understanding of  $\mathcal{H}_{k_*}$  for such  $\Omega$  could potentially allow us to extend our proof to the non-spherical case.
The spectral lower bounds in Theorem F.2 show that our approach can directly benefit from developing better kernel matrix concentration inequalities. Conversely, the investigation of consistent kernel interpolation might provide information about where such concentration inequalities do not hold.
5 Consistency via spiky-smooth estimators - even in fixed dimension
In Section 4, we have seen that when common kernel estimators overfit, they are inconsistent for many kernels and a wide variety of distributions. We now design consistent interpolating kernel estimators. The key is to violate Assumption (N) for every fixed Sobolev RKHS norm  $\| \cdot \|_{\mathcal{H}_k}$  and introduce an inductive bias towards learning spiky-smooth functions.
5.1 Almost universal consistency of spiky-smooth ridgeless kernel regression
In high dimensional regimes (where the dimension  $d$  is supposed to grow with the number of data points), benign overfitting of linear and kernel regression has been understood by an additive decomposition of the minimum-norm interpolant into a smooth regularized component that is responsible for good generalization, and a spiky component that interpolates the noisy data points while not harming generalization (Bartlett et al., 2021). This inspires us to enforce such a decomposition in arbitrary fixed dimension by adding a sharp kernel spike  $\rho \check{k}_{\gamma_n}$  to a common kernel  $\tilde{k}$ . In this way, we can still generate any Sobolev RKHS (see Appendix G.2).
Definition 7 (Spiky-smooth kernel). Let  $\tilde{k}$  denote any universal kernel function on  $\mathbb{R}^d$ . We call it the smooth component. Consider a second, translation invariant kernel  $\tilde{k}_{\gamma}$  of the form  $k_{\gamma}(\boldsymbol{x},\boldsymbol{y}) = q(\frac{\boldsymbol{x} - \boldsymbol{y}}{\gamma})$ , for some function  $q:\mathbb{R}^d\to \mathbb{R}$ . We call it the spiky component. Then we define the  $\rho$ -regularized spiky-smooth kernel with spike bandwidth  $\gamma$  as
$$
k _ {\rho , \gamma} (\boldsymbol {x}, \boldsymbol {y}) = \tilde {k} (\boldsymbol {x}, \boldsymbol {y}) + \rho \cdot \check {k} _ {\gamma} (\boldsymbol {x}, \boldsymbol {y}), \quad \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}. \tag {4}
$$
Figure 2: The spiky-smooth kernel with Laplace components (orange) consists of a Laplace kernel (blue) plus a Laplace kernel of height  $\rho$  and small bandwidth  $\gamma$ .
We now show that the minimum-norm interpolant of the spiky-smooth kernel sequence with properly chosen  $\rho_{n},\gamma_{n}\rightarrow 0$  is consistent for a large class of distributions, on a space with fixed (possibly small) dimension  $d$ . We establish our result under the following assumption (as in Mücke and Steinwart (2019)), which is weaker than our previous Assumption (D1).
(D2) There exists a constant  $\beta_{X} > 0$  and a continuous function  $\phi :[0,\infty)\to [0,1]$  with  $\phi (0) = 0$  such that the data generating probability distribution satisfies  $P_{X}(B_{t}(\pmb {x}))\leq \phi (t) = O(t^{\beta_{X}})$  for all  $\pmb {x}\in \Omega$  and all  $t\geq 0$  (here  $B_{t}(\pmb {x})$  denotes the Euclidean ball of radius  $t$  around  $\pmb{x}$ ).
Theorem 8 (Consistency of spiky-smooth ridgeless kernel regression). Assume that the training set  $D$  consists of  $n$  i.i.d. pairs  $(\pmb{x}, y) \sim P$  such that the marginal  $P_X$  fulfills (D2) and  $\mathbb{E}y^2 < \infty$ . Let the kernel components satisfy:
-  $\tilde{k}$  is a universal kernel, and  $\rho_{n} \to 0$  and  $n \rho_{n}^{4} \to \infty$ .
-  $\check{k}_{\gamma_n}$  denotes the Laplace kernel with a sequence of positive bandwidths  $(\gamma_n)$  fulfilling  $\gamma_n \leq n^{-\frac{2 + \alpha}{d}} \left( \left( \frac{9}{4} + \frac{\alpha}{2} \right) \ln n \right)^{-1}$ , where  $\alpha > 0$  arbitrary.
Then the minimum-norm interpolant of the  $\rho_{n}$ -regularized spiky-smooth kernel sequence  $k_{n} \coloneqq k_{\rho_{n},\gamma_{n}}$  is consistent for  $P$ .
Remark 9 (Benign overfitting with optimal convergence rates). Suppose that we have a Sobolev target function  $f^{*} \in H^{s^{*}}(\Omega) \backslash \{0\}$ , that the noise satisfies a moment condition and that  $P_{X}$  has an upper- and lower-bounded density on a Lipschitz domain or the sphere. Then, we show in Theorem G.5 that, by using smooth components  $\tilde{k}$  whose RKHS is equivalent to a Sobolev space  $H^{s}$ ,  $s > \max(s^{*}, d/2)$ , and choosing the spike components  $\tilde{k}_{\gamma_{n}}$  as in Theorem 8, the minimum-norm interpolant of  $k_{n} \coloneqq k_{\rho_{n}, \gamma_{n}}$  achieves the convergence rate  $n^{-\frac{s^{*}}{(s^{*} + d/2)}} \log^{2}(n)$  when choosing the quasi-regularization  $\rho_{n}$  properly. Moreover, for  $s^{*} > d/2$ , this rate is known to be optimal up to the factor  $\log^{2}(n)$  (Remark G.6). Since optimal rates can be achieved both with optimal regularization and with interpolation, our results show that in Sobolev RKHSs, overfitting is neither intrinsically helpful nor harmful for generalization with the right choice of kernel function.
Proof idea. With sharp spikes  $\gamma \rightarrow 0$ , it holds that  $\check{k}_{\gamma}(\boldsymbol{X},\boldsymbol{X}) \approx \boldsymbol{I}_n$ , with high probability. Hence, ridgeless kernel regression with the spiky-smooth kernel interpolates the training set while approximating kernel ridge regression with the smooth component  $\tilde{k}$  and regularization  $\rho$ .
The theorem even holds under much weaker assumptions on the decay behavior of the spike component  $\check{k}_{\gamma_n}$ , including Gaussian and Matérn kernels. The full version of the theorem and its proof can be found in Appendix G. It also applies to kernels and distributions on the sphere  $\mathbb{S}^d$ .
Remark 10 (Interplay between smoothness and dimensionality). Irrespective of the dimension  $d$ , we achieve benign overfitting with estimators in RKHS of arbitrary degrees of smoothness. With increasing  $d$ , for the Laplace kernel the spike bandwidth is allowed to be chosen as  $\gamma_{n} = \Omega (n^{-(2 + \alpha) / d})$ ,  $\alpha > 0$ , for covariate distributions with upper bounded Lebesgue density (see Remark G.2). Hence the magnitude of derivatives of the spikes is allowed to scale less aggressively with increasing dimension.
5.2 From spiky-smooth kernels to spiky-smooth activation functions
So far, our discussion revolved around the properties of kernels and whether they lead to estimators that are consistent. We now turn our attention to the neural network side. The big question is whether it is possible to specifically design activation functions that enable benign overfitting in fixed, possibly small dimension. We will see that the answer is yes: similarly to adding sharp spikes to a kernel, we add tiny fluctuations to the activation function. Concretely, we exploit (Simon et al., 2022, Theorem 3.1). It states that any dot-product kernel on the sphere that is a dot-product kernel in every dimension  $d$  can be written as an NNGP kernel or an NTK of two-layer fully-connected networks with a specifically chosen activation function. Further details can be found in Appendix H.
Theorem 11 (Connecting kernels and activation functions (Simon et al., 2022)). Let  $\kappa : [-1, 1] \to \mathbb{R}$  be a function such that  $k_d : \mathbb{S}^d \times \mathbb{S}^d \to \mathbb{R}, k_d(\boldsymbol{x}, \boldsymbol{x}') = \kappa(\langle \boldsymbol{x}, \boldsymbol{x}' \rangle)$  is a kernel for every  $d \geq 1$ . Then, there exist  $b_i \geq 0$  with  $\sum_{i=0}^{\infty} b_i < \infty$  such that  $\kappa(t) = \sum_{i=0}^{\infty} b_i t^i$ , and for any choice of signs  $(s_i)_{i \in \mathbb{N}_0} \subseteq \{-1, +1\}$ , the kernel  $k_d$  can be realized as the NNGP kernel or NTK of a two-layer fully-connected network without biases and with activation function
$$
\phi_ {N N G P} ^ {k _ {d}} (x) = \sum_ {i = 0} ^ {\infty} s _ {i} \left(b _ {i}\right) ^ {1 / 2} h _ {i} (x), \quad \phi_ {N T K} ^ {k _ {d}} (x) = \sum_ {i = 0} ^ {\infty} s _ {i} \left(\frac {b _ {i}}{i + 1}\right) ^ {1 / 2} h _ {i} (x). \tag {5}
$$
Here,  $h_i$  denotes the  $i$ -th Probabilist's Hermite polynomial normalized such that  $\| h_i \|_{L_2(\mathcal{N}(0,1))} = 1$ .
The following proposition justifies the approach of adding spikes  $\rho^{1/2}\phi^{k_{\gamma}}$  to an activation function to enable harmless interpolation with wide neural networks. Here we state the result for the case of the NTK; an analogous result holds for induced NNGP activation functions.
Proposition 12 (Additive decomposition of spiky-smooth activation functions). Fix  $\tilde{\gamma}, \rho > 0$  arbitrary. Let  $k = \tilde{k} + \rho \check{k}_{\gamma}$  denote the spiky-smooth kernel where  $\tilde{k}$  and  $\check{k}_{\gamma}$  are Gaussian kernels of bandwidth  $\tilde{\gamma}$  and  $\gamma$ , respectively. Assume that we choose signs  $\{s_i\}_{i \in \mathbb{N}}$  and then the activation functions  $\phi_{NTK}^k$ ,  $\phi_{NTK}^{\tilde{k}}$  and  $\phi_{NTK}^{\check{k}_{\gamma}}$  as in Theorem 11. Then, for  $\gamma > 0$  small enough, it holds that
$$
\| \phi_ {N T K} ^ {k} - (\tilde {\phi} _ {N T K} ^ {k} + \sqrt {\rho} \cdot \tilde {\phi} _ {N T K} ^ {k _ {\gamma}}) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \leq 2 ^ {1 / 2} \rho \gamma^ {3 / 2} \exp \left(- \frac {1}{\gamma}\right) + \frac {4 \pi (1 + \tilde {\gamma}) \gamma}{\tilde {\gamma}}.
$$
Proof idea. When the spikes are sharp enough ( $\gamma$  small enough), the smooth and the spiky component of the activation function are approximately orthogonal in  $L_{2}(\mathcal{N}(0,1))$  (Figure 3c), so that the spiky-smooth activation function can be approximately additively decomposed into the smooth activation component  $\phi^{\tilde{k}}$  and the spike component  $\phi^{\tilde{k}}$  responsible for interpolation.
To motivate why the added spike functions  $\rho^{1/2}\phi^{\tilde{k}_{\gamma}}$  should have small amplitudes, observe that Gaussian activation components  $\phi^{\tilde{k}_{\gamma}}$  satisfy
$$
\left\| \phi_ {N N G P} ^ {\tilde {k} _ {\gamma}} \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} = 1, \quad \left\| \phi_ {N T K} ^ {\tilde {k} _ {\gamma}} \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} = \frac {\gamma}{2} \left(1 - \exp \left(- \frac {2}{\gamma}\right)\right). \tag {6}
$$
Hence, the average amplitude of NNGP spike activation components  $\rho^{1/2}\phi^{k_{\gamma}}$  does not depend on  $\gamma$ , while the average amplitude of NTK spike components decays to 0 with  $\gamma \rightarrow 0$ . Since consistency requires the quasi-regularization  $\rho \rightarrow 0$ , the spiky component of induced NTK as well as NNGP activation functions should vanish for large data sets  $n \rightarrow \infty$  to achieve consistency.
Figure 3: a., b. Gaussian NTK activation components  $\phi_{NTK}^{k_{\gamma}}$  defined via (5) induced by the Gaussian kernel with varying bandwidth  $\gamma \in [0.2, 0.1, 0.05]$  (the darker, the smaller  $\gamma$ ) for a. bi-alternating signs  $s_i = +1$  iff  $\lfloor i / 2 \rfloor$  even, and b. randomly iid chosen signs  $s_i \sim \mathcal{U}(\{-1, +1\})$ . c. Coefficients of the Hermite series of a Gaussian NTK activation component with varying bandwidth  $\gamma$ . Observe peaks at  $2 / \gamma$ . For reliable approximations of activation functions use a truncation  $\geq 4 / \gamma$ . The sum of squares of the coefficients follows Eq. (6). Figure I.8 visualizes NNGP activation components.
Now we explore how appropriate spiky-smooth activation functions might look like and whether they indeed enable harmless interpolation for trained networks of finite width on finite data sets. Further experimental results are reported in Appendix I.
6.1 What do common activation functions lack in order to achieve harmless interpolation?
To understand which properties we have to introduce into activation functions to enable harmless interpolation, we plot NTK spike components  $\phi^{\tilde{k}_{\gamma}}$  induced by the Gaussian kernel (Figure 3a,b) as well as their Hermite series coefficients (Figure 3c). Remarkably, the spike components  $\phi^{\tilde{k}_{\gamma}}$  approximately correspond to a shifted, high-frequency sin-curve, when choosing the signs  $s_i$  in (5) to alternate every second  $i$ , that is  $s_i = +1$  iff  $\lfloor i / 2\rfloor$  even (Figure 3a). Proposition H.1 shows that the NNGP activation functions correspond to the fluctuation function
$$
\omega_ {\mathrm {N N G P}} (x; \gamma) := \sqrt {2} \cdot \sin (\sqrt {2 / \gamma} \cdot x + \pi / 4) = \sin (\sqrt {2 / \gamma} \cdot x) + \cos (\sqrt {2 / \gamma} \cdot x), \tag {7}
$$
where the last equation follows from the trigonometric addition theorem. For small bandwidths  $\gamma$ , the NTK activation functions are increasingly well approximated (Appendix I.6) by
$$
\omega_ {\mathrm {N T K}} (x; \gamma) := \sqrt {\gamma} \cdot \sin (\sqrt {2 / \gamma} \cdot x + \pi / 4) = \sqrt {\gamma / 2} (\sin (\sqrt {2 / \gamma} \cdot x) + \cos (\sqrt {2 / \gamma} \cdot x)). \tag {8}
$$
With decreasing bandwidth  $\gamma \rightarrow 0$  the frequency increases, while the amplitude decreases for the NTK and remains constant for the NNGP (see Eq. (6)). Plotting equivalent spike components  $\phi^{\tilde{k} -\gamma}$  with different choices of the signs  $s_i$  (Figure 3b and Appendix I.5) suggests that harmless interpolation requires activation functions that contain small high-frequency oscillations or that explode at large  $|x|$ , which only affects few neurons. The Hermite series expansion of suitable activation functions should contain non-negligible weight spread across high-order coefficients (Figure 3c). While Simon et al. (2022) already truncate the Hermite series of induced activation functions at order 5, Figure 3c shows that an accurate approximation of spiky-smooth activation functions requires the truncation index to be larger than  $2 / \gamma$ . Only a careful implementation allows us to capture the high-order fluctuations in the Hermite series of the spiky activation functions. Our implementation can be found at https://github.com/moritzhaas/mind-the-spikes.
6.2 Training neural networks to achieve harmless interpolation in low dimension
In Figure 1, we plot the results of (a) ridgeless kernel regression and (b) trained 2-layer neural networks with standard choices of kernels and activation functions (blue) as well as our spiky-smooth alternatives (orange). We trained on 15 points sampled i.i.d. from  $x = (x_{1},x_{2})\sim \mathcal{U}(\mathbb{S}^{1})$  and  $y = x_{1} + \varepsilon$  with  $\varepsilon \sim \mathcal{N}(0,0.25)$ . The figure shows that both the Laplace kernel and standard ReLU networks interpolate the training data too smoothly in low dimension, and do not generalize well. However, our spiky-smooth kernel and neural networks with spiky-smooth activation functions achieve close to optimal generalization while interpolating the training data with sharp spikes.
We achieve this by using the adjusted activation function with high-frequency oscillations  $x \mapsto \mathrm{ReLU}(x) + \omega_{\mathrm{NTK}}(x; \frac{1}{5000})$  as defined in Eq. (8). With this choice, we avoid activation functions with exploding behavior, which would induce exploding gradients. Other choices of amplitude and frequency in Eq. (8) perform worse. To bring our neural networks close to the kernel regime, we use the neural tangent parameterization (Jacot et al., 2018) and make the networks very wide (20000 hidden neurons). To ensure that the initial function is identically zero, we use the antisymmetric initialization trick (Zhang et al., 2020). Over the course of training (Figure 1c), the standard ReLU network exhibits harmful overfitting, whereas the NN with a spiky-smooth activation function quickly interpolates the training set with nearly optimal generalization. Training details and hyperparameter choices can be found in Appendix I.1. Although the high-frequency oscillations perturb the gradients, the NN with spiky smooth activation has a stable training trajectory using gradient descent with a large learning rate of 0.4 or stochastic gradient descent with a learning rate of 0.04. Since our activation function is the sum of two terms, we can additively decompose the network into its ReLU-component and its  $\omega_{\mathrm{NTK}}$ -component. Figure 1b and Appendix I.2 demonstrate that our interpretation of the  $\omega_{\mathrm{NTK}}$ -component as 'spiky' is accurate: The oscillations in the hidden neurons induced by  $\omega_{\mathrm{NTK}}$  interfere constructively to interpolate the noise in the training points and regress to 0 between training points. This entails immediate access to the signal component of the trained neural network in form of its ReLU-component.
Conceptually, our work shows that inconsistency of overfitting is quite a generic phenomenon for regression in fixed dimension. However, particular spiky-smooth estimators enable benign overfitting, even in fixed dimension. We translate the spikes that lead to benign overfitting in kernel regression into infinitesimal fluctuations that can be added to activation functions to consistently interpolate with wide neural networks. Our experiments verify that neural networks with spiky-smooth activation functions can exhibit benign overfitting even on small, low-dimensional data sets.
Technically, our inconsistency results cover many distributions, Sobolev spaces of arbitrary order, and arbitrary RKHS-norm-bounded overfitting estimators. Lemma E.1 serves as a generic tool to extend generalization bounds to the sphere  $\mathbb{S}^d$ , allowing us to cover (deep) ReLU NTKs and ReLU NNGPs.
Future work. While our experiments serve as a promising proof of concept, it remains unclear how to design activation functions that enable harmless interpolation of more complex neural network architectures and data sets. As another interesting insight, our consistent kernel sequence shows that although kernels may have equivalent RKHS (see Appendix G.2), their generalization error can differ arbitrarily much; the constants of the equivalence matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach (2021) is too simplified. More promisingly, analyses that extend our analysis in the infinite-width limit to a joint scaling of width and depth could help us to understand the influence of depth (Fort et al., 2020, Li et al., 2021, Seleznova and Kutyniok, 2022). Finite-sample analyses of moderate-width neural networks with feature learning parametrizations (Yang and Hu, 2021) and other initializations could enable to understand how to induce a spiky-smooth inductive bias in feature learning neural architectures.
Funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016 and EXC 2064/1 - Project 390727645, as well as the DFG Priority Program 2298/1, project STE 1074/5-1. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Moritz Haas and David Holzmuller. We want to thank Tizian Wenzel and Václav Voráček for interesting discussions. We also thank Nadine Groß, Jens Wirth, and Daniel Winkle for helpful comments on Sobolev spaces, and Nilotpal Sinha for pointing us to Laurent series.
Robert A. Adams and John J.F. Fournier. Sobolev Spaces. Elsevier Science, 2003.