SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {s i g n a l} (\boldsymbol {x}) = \tilde {k} (\boldsymbol {x}, \boldsymbol {X}) \cdot (\tilde {\boldsymbol {K}} + \rho_ {n} \breve {\boldsymbol {K}} _ {\gamma_ {n}}) ^ {- 1} \boldsymbol {y}, \qquad f _ {s p i k e s} (\boldsymbol {x}) = \rho_ {n} \breve {k} _ {\gamma_ {n}} (\boldsymbol {x}, \boldsymbol {X}) \cdot (\tilde {\boldsymbol {K}} + \rho_ {n} \breve {\boldsymbol {K}} _ {\gamma_ {n}}) ^ {- 1} \boldsymbol {y}.
$$
We plot the results in Figure I.3. Observe that the spikes  $f_{spikes}$  regress to 0 more reliably than in the neural network.
Although spiky-smooth estimators can be consistent, any method that interpolates noise cannot be adversarially robust. The signal component  $f_{signal}$  may be a simple correction towards robust estimators. Figure I.4 suggests that the signal components of spiky-smooth estimators behave more robustly than ReLU networks or minimum-norm interpolants of Laplace kernels in terms of finite-sample variance.
Figure I.3: a. The signal component  $f_{signal}$  (blue) and the full minimum-norm interpolant  $\hat{f}_0^k$  (orange) of the spiky-smooth kernel from Figure 1. b. The spike component  $f_{spikes}$  of the spiky-smooth kernel from Figure 1 against the label noise in the training set, derived by subtracting the signal from the training points.
I.3 Repeating the finite-sample experiments
We repeat the experiment from Figure 1 100 times, both randomizing with respect to the training set and with respect to neural network initialization.
For the kernels (Figure I.4a), observe that all minimum-norm kernel interpolants are biased towards 0. While the Laplace kernel and the signal component (I.2) of the spiky-smooth kernel have similar averages, the spiky-smooth kernel has a slightly larger bias. However, both the spiky-smooth kernel as well as its signal component produces lower variance estimates than the Laplace kernel.
Considering the trained neural networks (Figure I.4b), the ReLU networks are approximately unbiased, but have large variance. The neural networks with spiky-smooth activation function as well as the extracted signal network (I.1) are similar on average: They are slightly biased towards 0, but have much smaller variance than the ReLU networks.
Figure I.4: We repeat the experiment from Figure 1 100 times and report the mean values (lines). Confidence bands denote the interval between the empirical  $2.5\%$  - and  $97.5\%$  -quantiles from the 100 independent runs.
The training curves (Figure I.4c) offer similar conclusions as Figure 1: While the ReLU networks harmfully overfit over the course of training, the neural networks with spiky-smooth activation function quickly overfit to 0 training error with monotonically decreasing test error, which on average is almost optimal, already with only 15 training points. The spiky-smooth networks have smaller confidence bands, indicating increased robustness compared to the ReLU networks. If the ReLU networks would be early-stopped with perfect timing, they would generalize similarly well as the networks with spiky-smooth activation function.
I.4 Spiky-smooth activation functions
In Figures I.5 and I.6 we plot the 2-layer NTK activation functions induced by spiky-smooth kernels with Gaussian components, where  $\tilde{k}$  has bandwidth 1, and in the first figure  $\rho = 1$  while in the second figure  $\rho = 0.1$ .
Figure I.5: The 2-layer NTK activation functions for Gaussian-Gaussian spiky-smooth kernels with varying  $\gamma$  (columns) with  $k_{max} = 1000$ ,  $\tilde{k}$  has bandwidth 1,  $\rho = 1$ . Top: all  $s_i = +1$ , middle:  $+, +, -, -, +, +, \ldots$ , bottom: Random  $+1$  and  $-1$ . Although the activation function induced by the spiky-smooth kernel is not exactly the sum of the activation functions induced by its components, this approximation is accurate because the spike components approximately live in a subspace of higher frequency in the Hermite basis orthogonal to the low-frequency subspace of the smooth component.
Figure I.6: Same as above but  $\rho = 0.1$ . The high-frequency fluctuations are much smaller compared to Figure I.5.
In Figure I.7 we plot the corresponding 2-layer NNGP activation functions with  $\rho = 1$ . In contrast to the NTK activation functions the amplitudes of the fluctuations only depends on  $\rho$  and not on  $\gamma$ . Our intuition is the following: Since the first layer weights are not learned in case of the NNGP, the first layer cannot learn constructive interference, so that the oscillations in the activation function need to be larger.
The additive approximation  $\phi^k\approx \phi^{\bar{k}} + \rho^{1 / 2}\phi^{\bar{k}\gamma}$  remains accurate in all considered cases (Appendix I.6).
Figure I.7: Same as above but NNGP and  $\rho = 1$ . As expected from the isolated spike plots: Spikes essentially add fluctuations that increase in frequency and stay constant in amplitude for  $\gamma \rightarrow 0$ ,  $\rho$  regulates the amplitude.
I.5 Isolated spike activation functions
Figure I.8 is the equivalent of Figure 3 for the NNGP.
By plotting the NTK activation components corresponding to Gaussian spikes  $\phi^{\tilde{k}_{\gamma}}$  with varying choices of the signs  $s_i$  in Figure I.9, we observe the following properties:
1. All  $s_i = +1$  leads to exponentially exploding activation functions, cf. Eq. (H.2).
2. If the signs  $s_i$  alternate every second  $i$ , i.e.  $s_i = +1$  iff  $\left\lfloor \frac{i}{2} \right\rfloor$  even,  $\phi^{\hat{k}_{\gamma}}$  is approximately a single shifted sin-curve with increasing frequency and decreasing/constant amplitude for NTK/NNGP activation functions, cf. Eq. (6).
3. If  $s_i$  is chosen uniformly at random, with high probability,  $\phi^{\bar{k}_{\gamma}}$  both oscillates at a high frequency around 0 and explodes for  $|x| \to \infty$ .
Figure I.8: Same as Figure 3 but for the NNGP. In contrast to the NTK, the amplitudes of the oscillations in a. do not shrink with  $\gamma \rightarrow 0$ . Otherwise the behaviour is analogous. For example, the Hermite coefficients peak at  $2 / \gamma$ . The squared coefficients sum to 1 (Eq. (6)).
ee eae
Figure I.9: The spike activation components of 2-layer NTK for Gaussian spikes with varying  $\gamma$  (columns),  $k_{max} = 1000$ , top: all  $s_i = +1$ , middle: signs alternate every second index, bottom: 3 draws from uniformly random signs. With  $\gamma \rightarrow 0$ , the amplitude shrinks, while the frequency increases.
Figure I.10 visualizes NNGP activation functions induced by Gaussian spikes with varying bandwidth  $\gamma$ . Observe similar behaviour as for the NTK but amplitudes invariant to  $\gamma$  as predicted by Eq. (6). For smaller  $\gamma$  the explosion of  $(\mathrm{all}+)$  activation functions starts at larger  $x$ , but appears sharper as can be seen in the analytic expression (H.2).
Figure I.11 resembles Figure I.9 but plotted on a larger range to visualize the exploding behaviour for  $|x| \to \infty$ .
Figure I.10: Spike activation components as in Figure I.9, but for the NNGP with  $x$  between  $[-4, 4]$ .
Figure I.11: Spike NTK activation components as in Figure I.9 but with  $x$  between  $[-4, 4]$ . The all+ NTK activation explodes exponentially. While random sign activations explode as well,  $+ + - -$  activations remain stable sin-fluctuations with slowly decaying amplitude for  $|x| \to \infty$ .
I.6 Additive decomposition and sin-fit
Here we quantify the error of the sin-approximation (8) of Gaussian NTK activation components. The additive decomposition  $\phi^k\approx \phi^{\tilde{k}} + \rho^{1 / 2}\phi^{\tilde{k}_\gamma}$  quickly becomes accurate in the limit  $\gamma \rightarrow 0$  (Figures I.12 and I.13), the sin-approximation seems to converge pointwise at rate  $\Theta (|x|\gamma)$ , where a good approximation can be expected when  $|x|\ll 1 / \gamma$ . The error at large  $|x|$  arises because the spike component decays for  $|x|\to \infty$ . For  $O(1)$  inputs, we conjecture that this inaccuracy does not dramatically affect the test error of neural networks when  $\gamma$  is chosen to be small.
Figure I.12: The isolated NTK spike activation function (orange), the difference between spiky-smooth and smooth activation function (blue) and a fitted sin-curve (8) (green). All curves roughly align, in particular for  $\gamma \rightarrow 0$ .
Figure I.13: The error of the additive decomposition  $\phi^k\approx \phi^k +\rho^{1 / 2}\phi^{\bar{k}\gamma}$  (blue) and the sin-fit (8) (orange) for the NTK. While the additive decomposition makes errors of order  $10^{-3}$ ,  $10^{-4}$ ,  $10^{-9}$ ,  $10^{-15}$  (from left to right) in the domain  $[-4,4]$ , the sin-fit is increasingly inaccurate for  $|x|\to \infty$ , and increasingly accurate for  $\gamma \rightarrow 0$ .
Now we evaluate the numerical approximation quality of the sin-fits (7) and (8) to the isolated spike activation components  $\phi^{\tilde{k}_{\gamma}}$ . As expected by Proposition H.1, the NNGP oscillating activation function  $\phi^{\tilde{k}_{\gamma}}$  of a Gaussian spike component corresponds to Eq. (7) up to numerical errors. Both for the NNGP and for the NTK, the approximations become increasingly accurate with smaller bandwidths  $\gamma \rightarrow 0$  (Figure I.14). Again the approximation quality suffers for  $|x| \to \infty$ , since  $\phi_{NTK}^{\tilde{k}_{\gamma}}$  slowly decay to 0 for  $|x| \to \infty$ .
Figure I.14: Absolute numerical error between the oscillating activation function  $\phi^{\tilde{k}_{\gamma}}$  of a Gaussian spike component and a. its analytical expression Eq. (7) for the NNGP and b. the approximation Eq. (8) for the NTK with varying bandwidth  $\gamma$ .
I.7 Spiky-smooth kernel hyper-parameter selection
To understand the empirical performance of spiky-smooth kernels on finite data sets, we generate i.i.d. data where  $\pmb{x} \sim \mathcal{U}(\mathbb{S}^d)$  and
$$
y = \boldsymbol {x} _ {1} + \boldsymbol {x} _ {2} ^ {2} + \sin (\boldsymbol {x} _ {3}) + \prod_ {i = 1} ^ {d + 1} \boldsymbol {x} _ {i} + \varepsilon ,
$$
with  $\varepsilon \sim \mathcal{N}(0, \sigma^2)$  independent of  $x$  and evaluate the least squares excess risk of the minimum-norm interpolant. Figure I.15 shows that
- the smaller the spike bandwidth  $\gamma$ , the better. At some point, the improvement saturates,
-  $\rho$  should be carefully tuned, it has large impact. As with  $\gamma \rightarrow 0$  ridgeless regression with the spiky-smooth kernel approximates ridge regression with  $\tilde{k}$  and regularization  $\rho$ , simply choose the optimal regularization  $\rho^{opt}$  of ridge regression.
- The spiky-smooth kernel with Gaussian components exhibits catastrophic overfitting, when  $\gamma$  is too large (cf. Mallinar et al. (2022)), the Laplace kernel is more robust with respect to  $\gamma$ .
- With sufficiently thin spikes and properly tuned  $\rho$ , spiky-smooth kernels with Gaussian components outperform the Laplace counterparts.
We repeat the experiment in Figure I.16 with a slightly more complex generating function and come to the same conclusions.
Figure I.15: Least squares excess risk for spiky-smooth kernel ridgeless regression with Laplace components (left) and Gaussian components (right), with  $n = 1000$ ,  $d = 2$ , estimated on 10000 independent test points,  $\sigma^2 = 0.5$ ,  $\tilde{\gamma} = 1$ . The smaller the spike bandwidth  $\gamma$ , the better. Properly tuning  $\rho$  is important.
Figure I.16: Same as Figure I.15 but with the more complex generating function  $y = |x_1| + x_2^2 + \sin(2\pi x_3) + \prod_{i=1}^{d+1} x_i + \varepsilon$ . The errors are larger compared to Figure I.15 and the optimal values of  $\rho$  are smaller, but the conceptual conclusions remain the same.