SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The following proposition states that the sum of kernels with equivalent RKHS yields an RKHS that is equivalent to the RKHS of the summands. For example, the spiky-smooth kernel with Laplace components possesses an RKHS equivalent to the RKHS of the Laplace kernel.
Proposition G.9. Let  $\mathcal{H}_1$  and  $\mathcal{H}_2$  denote the RKHS of  $k_{1}$  and  $k_{2}$  respectively. If  $\mathcal{H}_1 = \mathcal{H}_2$  then the RKHS  $\mathcal{H}$  of  $k = k_{1} + k_{2}$  fulfills  $\mathcal{H} = \mathcal{H}_1$ . Moreover, if  $C \geq 1$  is a constant with  $\frac{1}{C}\|f\|_{\mathcal{H}_2} \leq \|f\|_{\mathcal{H}_1} \leq C\|f\|_{\mathcal{H}_2}$ , then we have  $\frac{1}{\sqrt{2}C}\|f\|_{\mathcal{H}_1} \leq \|f\|_{\mathcal{H}} \leq \|f\|_{\mathcal{H}_1}$ .
Proof. The RKHS of  $k = k_{1} + k_{2}$  is given by  $\mathcal{H} = \mathcal{H}_1 + \mathcal{H}_2$  with norm
$$
\left\| f \right\| _ {\mathcal {H}} ^ {2} = \min  \left\{\left\| f _ {1} \right\| _ {\mathcal {H} _ {1}} ^ {2} + \left\| f _ {2} \right\| _ {\mathcal {H} _ {2}} ^ {2}: f = f _ {1} + f _ {2}, f _ {1} \in \mathcal {H} _ {1}, f _ {2} \in \mathcal {H} _ {2} \right\}.
$$
To see this we consider the map  $\Phi : X \to \mathcal{H}_1 \times \mathcal{H}_2$  defined by  $\Phi(\boldsymbol{x}) \coloneqq (\Phi_1(\boldsymbol{x}, \cdot), \Phi_2(\boldsymbol{x}, \cdot))$  for all  $\boldsymbol{x} \in X$ , where  $X$  is the set, the spaces  $\mathcal{H}_i$  live on and  $\Phi_i(\boldsymbol{x}) \coloneqq k_i(\boldsymbol{x}, \cdot)$ . The reproducing property of  $k_1$  and  $k_2$  immediately ensures that  $\Phi$  is a feature map of  $k_1 + k_2$  and Theorem E.3 then shows
$$
\begin{array}{l} \mathcal {H} = \left\{\langle w, \Phi (\cdot) \rangle_ {\mathcal {H} _ {1} \times \mathcal {H} _ {2}}: w \in \mathcal {H} _ {1} \times \mathcal {H} _ {2} \right\} \\ = \left\{\langle w _ {1}, \Phi_ {1} (\cdot) \rangle_ {\mathcal {H} _ {1}} + \langle w _ {2}, \Phi_ {2} (\cdot) \rangle_ {\mathcal {H} _ {2}}: w _ {1} \in \mathcal {H} _ {1}, w _ {2} \in \mathcal {H} _ {2} \right\} = \mathcal {H} _ {1} + \mathcal {H} _ {2} \\ \end{array}
$$
as well as the formula for the norm on  $\mathcal{H}$ . Now let  $f\in \mathcal{H}$ . Considering the decomposition  $f = f_{1} + 0$  then gives  $\| f\|_{\mathcal{H}}\leq \| f\|_{\mathcal{H}_1}$ . Moreover, for  $f = f_{1} + f_{2}$  with  $f_{i}\in \mathcal{H}_{i}$  we have
$$
\| f \| _ {\mathcal {H} _ {1}} \leq \| f _ {1} \| _ {\mathcal {H} _ {1}} + \| f _ {2} \| _ {\mathcal {H} _ {1}} \leq \| f _ {1} \| _ {\mathcal {H} _ {1}} + C \| f _ {2} \| _ {\mathcal {H} _ {2}} \leq \sqrt {2} C \big (\| f _ {1} \| _ {\mathcal {H} _ {1}} ^ {2} + \| f _ {2} \| _ {\mathcal {H} _ {1}} ^ {2} \big) ^ {1 / 2}.
$$
Taking the infimum over all decomposition then yields the estimate  $\| f\|_{\mathcal{H}_1}\leq \sqrt{2} C\| f\|_{\mathcal{H}}$
H Spiky-smooth activation functions induced by Gaussian components
Here we explore the properties of the NNGP and NTK activation functions induced by spiky-smooth kernels with Gaussian components.
To offer some more background, it is well-known that NNGPs and NTKs on the sphere  $\mathbb{S}^d$  are dot-product kernels, i.e., kernels of the form  $k_{d}(\pmb{x},\pmb{x}^{\prime}) = \kappa (\langle \pmb{x},\pmb{x}^{\prime}\rangle)$ , where the function  $\kappa$  has a series representation  $\kappa (t) = \sum_{i = 0}^{\infty}b_{i}t^{i}$  with  $b_{i}\geq 0$  and  $\sum_{i = 0}^{\infty}b_{i} < \infty$ . The function  $\kappa$  is independent of the dimension  $d$  of the sphere. Conversely, all such kernels can be realized as NNGPs or NTKs (Simon et al., 2022, Theorem 3.1).
As dot-product kernel  $k(\pmb{x}, \pmb{y}) = \kappa(\langle \pmb{x}, \pmb{y} \rangle)$  on the sphere, the Gaussian kernel has the simple analytic expression,
$$
\kappa_ {\gamma} ^ {G a u s s} (z) = \exp \left(\frac {2 (z - 1)}{\gamma}\right),
$$
with Taylor expansion
$$
\kappa_ {\gamma} ^ {G a u s s} (z) = \sum_ {i = 0} ^ {\infty} \underbrace {\frac {2 ^ {i}}{\gamma^ {i} i !} \exp (- 2 / \gamma)} _ {b _ {i} ^ {G a u s s}} z ^ {i}.
$$
For spiky-smooth kernels  $k = \tilde{k} + \rho \tilde{k}_{\gamma}$  with Gaussian components  $\tilde{k}$  and  $\tilde{k}_{\gamma}$  of width  $\tilde{\gamma}$  and  $\gamma$  respectively, we get Taylor series coefficients
$$
b _ {i} = \frac {\exp (- 2 / \tilde {\gamma})}{i !} \left(\frac {2}{\tilde {\gamma}}\right) ^ {i} + \rho \frac {\exp (- 2 / \gamma)}{i !} \left(\frac {2}{\gamma}\right) ^ {i}. \tag {H.1}
$$
Now Theorem 11 states that as soon as  $\kappa$  induces a dot-product kernel for every input dimension  $d$ , then the dot-product kernels can be written as the NNGP kernel of a 2-layer fully-connected network without biases and with the induced activation function
$$
\phi_ {N N G P} ^ {\kappa} (x) = \sum_ {i = 0} ^ {\infty} s _ {i} b _ {i} ^ {1 / 2} h _ {i} (x),
$$
or as the NTK of a 2-layer fully-connected network without biases and with the induced activation function
$$
\phi_ {N T K} ^ {\kappa} (x) = \sum_ {i = 0} ^ {\infty} s _ {i} \left(\frac {b _ {i}}{i + 1}\right) ^ {1 / 2} h _ {i} (x),
$$
where  $h_i$  denotes the  $i$ -th Probabilist's Hermite polynomial normalized such that  $\| h_i \|_{L_2(\mathcal{N}(0,1))} = 1$  and  $s_i \in \{-1, +1\}$  are arbitrarily chosen for all  $i \in \mathbb{N}_0$ .
Now we can study the induced activation functions if we know the kernel's Taylor coefficients  $(b_{i})_{i\in \mathbb{N}_{0}}$ . If infinitely many  $b_{i} > 0$ , then infinitely many activation functions induce the same dot-product kernel, with different choices of the signs  $s_i$ . For alternating signs  $s_i = (-1)^i$ , the symmetry property  $h_i(-x) = (-1)^i h_i(x)$  of the Hermite polynomials implies
$$
\phi_ {N N G P, + -} (x) = \phi_ {N N G P, +} (- x), \qquad \phi_ {N T K, + -} (x) = \phi_ {N T K, +} (- x).
$$
To form an orthonormal basis of  $L_{2}(\mathcal{N}(0,1))$  the unnormalized Probabilist's Hermite polynomials  $He_{i}$  have to be normalized by  $h_i(x) = \frac{1}{\sqrt{i!}} He_i(x)$ . We can use the identity  $\exp (xt - \frac{t^2}{2}) = \sum_{i = 0}^{\infty}He_{i}(x)\frac{t^{i}}{i!}$  with  $t = \sqrt{2 / \gamma}$  to analytically express the NNGP activation of the Gaussian kernel with all  $s_i = +1$  as the exponential function
$$
\phi_ {N N G P, +} ^ {G a u s s} (x) = \exp (- 1 / \gamma) \sum_ {i = 0} ^ {\infty} \frac {1}{i !} \left(\frac {2}{\gamma}\right) ^ {\frac {i}{2}} h _ {i} (x) = \exp \left(\left(\frac {2}{\gamma}\right) ^ {\frac {1}{2}} \cdot x - \frac {2}{\gamma}\right). \tag {H.2}
$$
Remarkably, the Gaussian kernel can not only be induced by an exponential activation function, but also by a single shifted sine activation function. This is shown in the following proposition.
Proposition H.1 (Trigonometric Gaussian NNGP activation functions). For any  $\gamma > 0$  and the bi-alternating choice of signs  $\{(-1)^{\lfloor i/2 \rfloor}\}_{i=0,1,2,\ldots}$ , the Gaussian kernel of bandwidth  $\gamma$  can be realized as the NNGP kernel of a two-layer fully-connected network without biases and with activation function
$$
\phi_ {N N G P, + + - -} ^ {G a u s s} (x) = \sin ((2 / \gamma) ^ {1 / 2} x) + \cos ((2 / \gamma) ^ {1 / 2} x).
$$
Proof. We write  $c = 2 / \gamma$ . We need to show that
$$
\sin (c ^ {1 / 2} x) + \cos (c ^ {1 / 2} x) = e ^ {- c / 2} \sum_ {i = 0} ^ {\infty} (- 1) ^ {\lfloor i / 2 \rfloor} \frac {c ^ {i / 2}}{i !} H e _ {i} (x).
$$
We will use the fact that
$$
e ^ {2 x z - z ^ {2}} = \sum_ {i = 0} ^ {\infty} \frac {z ^ {i}}{i !} H _ {i} (x),
$$
with the choices  $z_{1} = i\sqrt{c / 2}$  and  $z_{2} = -i\sqrt{c / 2}$ . Now, using  $e^{iax + b} = e^b (\cos (ax) + i\sin (ax))$ , observe that
$$
\begin{array}{l} \sin (\sqrt {c} x) = \sin (\sqrt {2 c} x / \sqrt {2}) = \frac {1}{2 i e ^ {c / 2}} \left(e ^ {i x \sqrt {c} + c / 2} - e ^ {i x \sqrt {c} + c / 2}\right) \\ = \frac {1}{2 i e ^ {c / 2}} \left(\sum_ {i = 0} ^ {\infty} \frac {(i \sqrt {c / 2}) ^ {i}}{i !} H _ {i} (x / \sqrt {2}) (1 - (- 1) ^ {i})\right) \\ = e ^ {- c / 2} \sum_ {i = 0} ^ {\infty} (- 1) ^ {i} \frac {(\sqrt {c / 2}) ^ {2 i + 1}}{(2 i + 1) !} H _ {2 i + 1} (x / \sqrt {2}). \\ \end{array}
$$
An analogous calculation yields
$$
\cos (c ^ {1 / 2} x) = e ^ {- c / 2} \sum_ {i = 0} ^ {\infty} (- 1) ^ {i} \frac {(\sqrt {c / 2}) ^ {2 i}}{(2 i) !} H _ {2 i} (x / \sqrt {2}).
$$
Finally, using  $H_{i}(x / \sqrt{2}) = 2^{i / 2}He_{i}(x)$ , we get
$$
\begin{array}{l} \sin (c ^ {1 / 2} x) + \cos (c ^ {1 / 2} x) = e ^ {- c / 2} \sum_ {i = 0} ^ {\infty} (- 1) ^ {\lfloor i / 2 \rfloor} \frac {(c / 2) ^ {i / 2}}{i !} H _ {i} (x / \sqrt {2}) \\ = e ^ {- c / 2} \sum_ {i = 0} ^ {\infty} (- 1) ^ {\lfloor i / 2 \rfloor} \frac {c ^ {i / 2}}{i !} H e _ {i} (x). \\ \end{array}
$$
For  $\phi_{NNGP}(x) = \sum_{i=0}^{\infty} s_i \sqrt{b_i} h_i(x)$ , we get  $\| \phi \|_{L_2(\mathcal{N}(0,1))}^2 = \sum_{i=0}^{\infty} b_i$  invariant to the choice  $\{s_i\}_{i \in \mathbb{N}}$ . For Gaussian NNGP activation components with bandwidth  $\gamma > 0$  this yields
$$
\left\| \phi_ {N N G P} ^ {G a u s s} \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} = \exp (- 2 / \gamma) \sum_ {i = 0} ^ {\infty} \frac {1}{i !} \left(\frac {2}{\gamma}\right) ^ {i} = 1, \tag {H.3}
$$
because  $\{h_i\}_{i\in \mathbb{N}_0}$  is an ONB of  $L_{2}(\mathcal{N}(0,1))$ . Analogously, for Gaussian NTK activation components, we get
$$
\begin{array}{l} \| \phi_ {N T K} ^ {G a u s s} \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} = \exp (- 2 / \gamma) \sum_ {i = 0} ^ {\infty} \frac {1}{(i + 1) !} \left(\frac {2}{\gamma}\right) ^ {i} \\ = \exp (- 2 / \gamma) \frac {\gamma}{2} \sum_ {i = 1} ^ {\infty} \frac {1}{i !} \left(\frac {2}{\gamma}\right) ^ {i} = \frac {\gamma}{2} \left(1 - \exp \left(- \frac {2}{\gamma}\right)\right). \qquad (\mathrm {H . 4}) \\ \end{array}
$$
This implies that the average amplitude of NNGP activation functions does not depend on  $\gamma$ , while the average amplitude of NTK activation functions decays with  $\gamma \rightarrow 0$ .
By the fact  $h_n'(x) = \sqrt{n} h_{n-1}(x)$ , we know that any activation function  $\phi(x) = \sum_{i=0}^{\infty} s_i a_i h_i(x)$  has the derivative  $\phi'(x) = \sum_{i=0}^{\infty} s_i a_{i+1} \sqrt{i+1} \cdot h_i(x)$  as long as  $\sum_{i=0}^{\infty} |a_{i+1} \sqrt{i+1}| < \infty$ .
The following proposition formalizes the additive approximation  $\phi^k\approx \phi^{\bar{k}} + \rho^{1 / 2}\phi^{\bar{k}\gamma}$ , and quantifies the necessary scaling of  $\gamma$  for any demanded precision of the approximation.
Proposition H.2. Fix  $\tilde{\gamma}, \rho > 0$  arbitrary. Let  $k = \tilde{k} + \rho \tilde{k}_{\gamma}$  denote the spiky-smooth kernel where  $\tilde{k}$  and  $\check{k}_{\gamma}$  are Gaussian kernels of bandwidth  $\tilde{\gamma}$  and  $\gamma$ , respectively. Assume that we choose the activation functions  $\phi_{NTK}^{k}$ ,  $\phi_{NTK}^{\tilde{k}}$  and  $\phi_{NTK}^{\check{k}_{\gamma}}$  as in Theorem 11 with same signs  $\{s_i\}_{i \in \mathbb{N}}$ . Then, for  $\gamma > 0$  small enough, it holds that
$$
\| \phi_ {N T K} ^ {k} - (\tilde {\phi_ {N T K} ^ {k}} + \sqrt {\rho} \cdot \tilde {\phi_ {N T K} ^ {k _ {\gamma}}}) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \leq 2 ^ {1 / 2} \rho \gamma^ {3 / 2} \exp \left(- \frac {1}{\gamma}\right) + \frac {4 \pi \gamma (1 + \tilde {\gamma})}{\tilde {\gamma}},
$$
$$
\| \phi_ {N N G P} ^ {k} - (\tilde {\phi} _ {N N G P} ^ {k} + \sqrt {\rho} \cdot \phi_ {N N G P} ^ {\check {k} _ {\gamma}}) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \leq 2 ^ {3 / 2} \rho \gamma^ {1 / 2} \exp \left(- \frac {1}{\gamma}\right) + \frac {8 \pi \gamma (1 + \tilde {\gamma})}{\tilde {\gamma} ^ {2}}.
$$
Proof. Let  $b_{i,\gamma} = \frac{2^i}{\gamma^i i!} \exp(-2/\gamma)$  denote the Taylor coefficients of the Gaussian kernel. All considered infinite series converge absolutely.
$$
\begin{array}{l} \left\| \tilde {\phi_ {N N G P} ^ {\gamma , \gamma , \rho}} - \left(\tilde {\phi_ {N N G P} ^ {\gamma}} + \sqrt {\rho} \cdot \phi_ {N N G P} ^ {\gamma}\right) \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \\ = \| \sum_ {i = 0} ^ {\infty} s _ {i} \sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} h _ {i} (x) - \sum_ {i = 0} ^ {\infty} s _ {i} (\sqrt {b _ {i , \tilde {\gamma}}} + \sqrt {\rho b _ {i , \gamma}}) h _ {i} (x) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \\ = \sum_ {i = 0} ^ {\infty} \left(\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - \left(\sqrt {b _ {i , \tilde {\gamma}}} + \sqrt {\rho b _ {i , \gamma}}\right)\right) ^ {2} \\ \leq 2 \underbrace {\sum_ {i = 0} ^ {I} (\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - b _ {i , \tilde {\gamma}} ^ {1 / 2}) ^ {2}} _ {(I)} + 2 \underbrace {\rho \sum_ {i = 0} ^ {I} b _ {i , \gamma}} _ {(I I)} + 2 \underbrace {\sum_ {i = I + 1} ^ {\infty} (\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - \rho^ {1 / 2} b _ {i , \gamma} ^ {1 / 2}) ^ {2}} _ {(I I I)} + 2 \underbrace {\sum_ {i = I + 1} ^ {\infty} b _ {i , \tilde {\gamma}}} _ {(I V)}, \\ \end{array}
$$
for any  $I\in \mathbb{N}$  . To bound  $(I)$  observe
$$
\sum_ {i = 0} ^ {I} \left(\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - b _ {i, \tilde {\gamma}} ^ {1 / 2}\right) ^ {2} = \sum_ {i = 0} ^ {I} \left(\rho b _ {i, \gamma} + 2 b _ {i, \tilde {\gamma}} \left(1 - \sqrt {1 + \frac {\rho b _ {i , \gamma}}{b _ {i , \tilde {\gamma}}}}\right)\right) \leq \rho \sum_ {i = 0} ^ {I} b _ {i, \gamma}.
$$
An analogous calculation for (III) yields
$$
\sum_ {i = I + 1} ^ {\infty} \left(\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - \rho^ {1 / 2} b _ {i, \gamma} ^ {1 / 2}\right) ^ {2} \leq \sum_ {i = I + 1} ^ {\infty} b _ {i, \tilde {\gamma}}.
$$
So overall we get the bound
$$
\left\| \phi_ {N N G P} ^ {\tilde {\gamma}, \gamma , \rho} - \left(\phi_ {N N G P} ^ {\tilde {\gamma}} + \sqrt {\rho} \cdot \phi_ {N N G P} ^ {\gamma}\right) \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \leq 4 \rho \sum_ {i = 0} ^ {I} b _ {i, \gamma} + 4 \sum_ {i = I + 1} ^ {\infty} b _ {i, \tilde {\gamma}}. \tag {H.5}
$$
Now, defining  $c\coloneqq 2 / \gamma$
$$
\sum_ {i = 0} ^ {I} b _ {i, \gamma} = \exp (- c) \sum_ {i = 0} ^ {I} \frac {1}{i !} e ^ {i} = \frac {\Gamma (I + 1 , c)}{I !},
$$
where  $\Gamma(k + 1, c)$  denotes the upper incomplete Gamma function. Choosing  $I = \left\lfloor \frac{c}{2\pi} \right\rfloor$ , (Pinelis, 2020, Theorem 1.1) yields, for  $c \geq 121$ ,
$$
\begin{array}{l} \frac {\Gamma (I + 1 , c)}{I !} \leq \exp (- c) \frac {(c + (I + 1) ! ^ {1 / I}) ^ {I + 1}}{(I + 1) ! \cdot (I + 1) ! ^ {1 / I}} \leq \frac {\exp (- c) (c + I) ^ {I + 1}}{(I + 1) ! ^ {(I + 1) / I}} \\ \leq \frac {\exp (- c) (c + I) ^ {I + 1}}{(2 \pi (I + 1)) ^ {1 / 2} \left(\frac {I + 1}{e}\right) ^ {(I + 1) ^ {2} / I}} \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {1}{\sqrt {2 \pi (I + 1)}} \exp \left(- c + (I + 1) (\ln (c + I) - \ln (I + 1) + 1)\right) \\ \leq \frac {1}{\sqrt {c}} \exp \Big (- c + (\frac {c}{2 \pi} + 1) \big (\ln (\frac {2 \pi + 1}{2 \pi} c) - \ln (\frac {c}{2 \pi}) + 1 \big) \Big) \\ \leq \frac {1}{\sqrt {c}} \exp \left(- c + \left(\frac {c}{2 \pi} + 1\right) \left(\ln (2 \pi + 1) + 1\right)\right) \leq \frac {\exp \left(- \frac {c}{2}\right)}{\sqrt {c}}, \tag {H.6} \\ \end{array}
$$
where we used  $(I + 1)!^{1 / I} \leq I$  for  $I \geq 3$  in the first line, Stirling's approximation in the second line, and  $\left(\frac{c}{2\pi} + 1\right)\left(\ln (2\pi + 1) + 1\right) \leq c / 2$  for  $c \geq 121$  in the last line.
It is obvious that
$$
\sum_ {i = I + 1} ^ {\infty} b _ {i, \tilde {\gamma}} \rightarrow 0, \quad \text {f o r} I = \left\lfloor \frac {c}{2 \pi} \right\rfloor\rightarrow \infty .
$$
To quantify the rate of convergence, we use the bound  $\Gamma(I + 1, c_0) \geq e^{-c_0} I!(1 + c_0 / (I + 1))^I$ , which follows from applying Jensen's inequality to  $\Gamma(I + 1, c_0) = e^{-c_0} I! \mathbb{E}(1 + c_0 / G)^I$ , where  $G \sim \Gamma(I + 1, 1)$  and  $\mathbb{E}G = I + 1$ . Defining  $c_0 = 2 / \tilde{\gamma}$ , it holds that
$$
\sum_ {i = I + 1} ^ {\infty} b _ {i, \tilde {\gamma}} \leq 1 - \frac {\Gamma (I + 1 , c _ {0})}{I !} \leq 1 - e ^ {- c _ {0}} \left(1 + \frac {c _ {0}}{I + 1}\right) ^ {I} \leq 1 - e ^ {- c _ {0}} \left(1 + \frac {c _ {0}}{I + 1}\right) ^ {I + 1}.
$$
Taking the first two terms of the Laurent series expansion of  $n \mapsto \left(1 + \frac{c_0}{n}\right)^n$  about  $n = \infty$  yields  $\left(1 + \frac{c_0}{I + 1}\right)^{I + 1} > e^{c_0}\left(1 - \frac{c_0^2}{2(I + 1)}\right)$  for  $I$  large enough (where we demand  $\gamma \in o(\tilde{\gamma}^2)$ ), thus
$$
\begin{array}{l} \sum_ {i = I + 1} ^ {\infty} b _ {i, \tilde {\gamma}} \leq 1 - e ^ {- c _ {0}} \left(1 + \frac {c _ {0}}{I + 1}\right) ^ {I + 1} \cdot \left(1 + \frac {c _ {0}}{I + 1}\right) ^ {- 1} \\ \leq \frac {c _ {0} / (I + 1) + c _ {0} ^ {2} / (2 (I + 1))}{1 + c _ {0} / (I + 1)} \leq \frac {c _ {0}}{I + 1} + \frac {c _ {0} ^ {2}}{2 (I + 1)} \leq \frac {4 \pi}{\tilde {\gamma} c} + \frac {4 \pi}{\tilde {\gamma} ^ {2} c}. \qquad (\mathrm {H .} 7) \\ \end{array}
$$
Plugging (H.6) and (H.7) into (H.5) yields, for  $\gamma \leq 1 / 61$
$$
\| \phi_ {N N G P} ^ {\tilde {\gamma}, \gamma , \rho} - (\phi_ {N N G P} ^ {\tilde {\gamma}} + \sqrt {\rho} \cdot \phi_ {N N G P} ^ {\gamma}) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \leq 2 ^ {3 / 2} \rho \gamma^ {1 / 2} \exp \left(- \frac {1}{\gamma}\right) + \frac {8 \pi \gamma (1 + \tilde {\gamma})}{\tilde {\gamma} ^ {2}}.
$$
For the NTK we get
$$
\begin{array}{l} \left\| \phi_ {N T K} ^ {\tilde {\gamma}, \gamma , \rho} - \left(\phi_ {N T K} ^ {\tilde {\gamma}} + \sqrt {\rho} \cdot \phi_ {N T K} ^ {\gamma}\right) \right\| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \\ = \| \sum_ {i = 0} ^ {\infty} s _ {i} \sqrt {\frac {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}}{i + 1}} h _ {i} (x) - \sum_ {i = 0} ^ {\infty} s _ {i} \left(\sqrt {\frac {b _ {i , \tilde {\gamma}}}{i + 1}} + \sqrt {\frac {\rho b _ {i , \gamma}}{i + 1}}\right) h _ {i} (x) \| _ {L _ {2} (\mathcal {N} (0, 1))} ^ {2} \\ = \sum_ {i = 0} ^ {\infty} \frac {1}{i + 1} \left(\sqrt {b _ {i , \tilde {\gamma}} + \rho b _ {i , \gamma}} - (\sqrt {b _ {i , \tilde {\gamma}}} + \sqrt {\rho b _ {i , \gamma}})\right) ^ {2}. \\ \end{array}
$$
We can proceed exactly as for the NNGP, but choose  $I = \left\lfloor \frac{c}{2\pi} \right\rfloor - 1$  to get
$$
\sum_ {i = 0} ^ {I} \frac {b _ {i , \gamma}}{i + 1} = \exp (- c) \sum_ {i = 0} ^ {I} \frac {c ^ {i}}{(i + 1) !} = \frac {\exp (- c)}{c} \left(\sum_ {i = 0} ^ {I + 1} \frac {c ^ {i}}{i !} - 1\right) \leq \frac {\exp (- c / 2)}{c ^ {3 / 2}} - \frac {\exp (- c)}{c},
$$
and replace (H.7) with
$$
\sum_ {i = I + 1} ^ {\infty} \frac {b _ {i , \tilde {\gamma}}}{i + 1} = \frac {\exp (- c _ {0})}{c _ {0}} \sum_ {i = I + 2} ^ {\infty} \frac {c _ {0} ^ {i}}{i !} \leq \frac {1}{I + 2} + \frac {c _ {0}}{2 (I + 2)} \leq \frac {\pi \gamma (1 + \tilde {\gamma})}{\tilde {\gamma}}.
$$
I Additional experimental results
The code to reproduce all our experiments is provided in the supplementary material and under
https://github.com/moritzhaas/mind-the-spikes
Our implementations rely on PyTorch (Paszke et al., 2019) for neural networks and mphpath (Johansson et al., 2023) for high-precision calculations.
I.1 Experimental details of Figure 1
For the kernel experiment (Figure 1a), we used the Laplace kernel with bandwidth 0.4 and the spiky-smooth kernel (4) with Laplace components with  $\rho = 1, \tilde{\gamma} = 1, \gamma = 0.01$ .
For the neural network experiment (Figure 1b,c) we initialize 2-layer networks with NTK parametrization (Jacot et al., 2018) and He initialization (He et al., 2015). Using the antisymmetric initialization trick from Zhang et al. (2020) doubles the network width from 10000 to 20000 and helps to prevent errors induced by the random initialization function. It might also be helpful to increase the initialization variance (Chizat et al., 2019). We train the network with stochastic gradient descent of batch size 1 over the 15 training samples with learning rate 0.04 for 2500 epochs. Training with gradient descent and learning rate 0.4 produces similar results. We use the spiky-smooth activation function given by  $x \mapsto ReLU(x) + 0.01 \cdot (\sin(100x) + \cos(100x))$ , which corresponds to  $x \mapsto ReLU(x) + \omega_{NTK}(x, \frac{1}{5000})$ , including both even and uneven Hermite coefficients.
I.2 Disentangling signal from noise in neural networks with spiky-smooth activation functions
Since our spiky-smooth activation function has the additive form  $\sigma_{spsm}(x) = ReLU(x) + \omega_{NTK}(x; \frac{1}{5000})$ , we can dissect the learned neural network
$$
f _ {s p s m} (\boldsymbol {x}) = \boldsymbol {W} _ {2} \cdot \sigma_ {s p s m} \left(\boldsymbol {W} _ {1} \cdot \boldsymbol {x} + \boldsymbol {b} _ {1}\right) + b _ {2} = f _ {R e L U} (\boldsymbol {x}) + f _ {s p i k e s} (\boldsymbol {x}) \tag {I.1}
$$
into its ReLU-component
$$
f _ {R e L U} (\boldsymbol {x}) = \boldsymbol {W} _ {2} \cdot R e L U \left(\boldsymbol {W} _ {1} \cdot \boldsymbol {x} + \boldsymbol {b} _ {1}\right) + b _ {2},
$$
and its spike component
$$
f _ {s p i k e s} (\boldsymbol {x}) = \boldsymbol {W} _ {2} \cdot \omega_ {N T K} \left(\boldsymbol {W} _ {1} \cdot \boldsymbol {x} + \boldsymbol {b} _ {1}; \frac {1}{5 0 0 0}\right).
$$
If the analogy to the spiky-smooth kernel holds and  $f_{spikes}$  fits the noise in the labels while having a small  $L_2$ -norm, then  $f_{ReLU}$  would have learned the signal in the data. Indeed Figure I.1 demonstrates that this simple decomposition is useful to disentangle the learned signal from the spike component in our setting. The figure also suggests that the oscillations in the activations of the hidden layer constructively interfere to interpolate the training points, while the differing frequencies and phases approximately destructively interfere on most of the remaining covariate support. Figure I.2 shows some of the functions generated by the hidden layer neurons of the spike component  $f_{spikes}$ . Both the phases and frequencies vary. Destructive interference in sums of many oscillations occurs, for example, under a uniform phase distribution.
An exciting direction of future work will be to understand when and why the neural networks with spiky-smooth activation functions learn the target function well, and when the decomposition into  $ReLU$ - and spike component succeeds to disentangle the noise from the signal. Particular challenges will be to design architectures and learning algorithms that provably work on complex data sets and to determine their statistical convergence rates. A different line of work could evaluate whether there exist useful spike components for deep and narrow networks beyond the pure infinite-width limit. Maybe for deep architectures is suffices to apply spiky-smooth activation functions only between the penultimate and the last layer.
Figure I.1: a. The  $ReLU$ -component  $f_{ReLU}$  (blue) and the full spiky-smooth network  $f_{spsm}$  (orange) of the learned neural network from Figure 1. b. The spike component  $f_{spikes}$  of the learned neural network from Figure 1 against the label noise in the training set, derived by subtracting the signal from the training points. Observe that the  $ReLU$ -component has learned the signal, while the spike component has fitted the noise in the data while regressing to 0 between data points.
Figure I.2: Here we plot the functions learned by 12 random hidden layer neurons of the spike component network  $f_{spikes}$  corresponding to Figure 1.
Of course an analogous additive decomposition exists for the minimum-norm interpolant  $\hat{f}_0^k$  of the spiky-smooth kernel,
$$
\hat {f} _ {0} ^ {k} (\boldsymbol {x}) = \left(\tilde {k} + \rho_ {n} \check {k} _ {\gamma_ {n}}\right) (\boldsymbol {x}, \boldsymbol {X}) \cdot \left(\tilde {\boldsymbol {K}} + \rho_ {n} \check {\boldsymbol {K}} _ {\gamma_ {n}}\right) ^ {- 1} \boldsymbol {y} = f _ {\text {s i g n a l}} (\boldsymbol {x}) + f _ {\text {s p i k e s}} (\boldsymbol {x}), \tag {I.2}
$$