# Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks

David Holzmüller, Max Schölpple

<figure><img src="image_1.png" /></figure>

► To cite this version:

David Holzmüller, Max Schölpple. Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks. 2025. hal-05136950

HAL Id: hal-05136950

https://hal.science/hal-05136950v1

Preprint submitted on 30 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# BEYOND RELU: HOW ACTIVATIONS AFFECT NEURAL KERNELS AND RANDOM WIDE NETWORKS

A PREPRINT

David Holzmüller*

INRIA

Ecole Normale Supérieure

PSL Research University

Max David Schölpple*

University of Stuttgart

Faculty of Mathematics and Physics

Institute for Stochastics and Applications

June 27, 2025

## ABSTRACT

While the theory of deep learning has made some progress in recent years, much of it is limited to the ReLU activation function. In particular, while the neural tangent kernel (NTK) and neural network Gaussian process kernel (NNGP) have given theoreticians tractable limiting cases of fully connected neural networks, their properties for most activation functions except for powers of the ReLU function are poorly understood. Our main contribution is to provide a more general characterization of the RKHS of these kernels for typical activation functions whose only non-smoothness is at zero, such as SELU, ELU, or LeakyReLU. Our analysis also covers a broad set of special cases such as missing biases, two-layer networks, or polynomial activations. Our results show that a broad class of not infinitely smooth activations generate equivalent RKHSs at different network depths, while polynomial activations generate non-equivalent RKHSs. Finally, we derive results for the smoothness of NNGP sample paths, characterizing the smoothness of infinitely wide neural networks at initialization.

## 1 Introduction

Despite great efforts, our theoretical understanding of when and why deep learning works remains limited, and much of it is distributed across fragmented case studies that do not integrate into a bigger theory. For example, the influence of activation functions on the training dynamics and generalization behavior of deep learning methods is unclear.

One of the most prominent approaches towards deep learning theory is the study of infinite-width limits of neural networks (Neal, 1996; Jacot et al., 2018; Yang and Hu, 2021). Different parametrizations of neural networks lead to different behavior at infinite width (Yang and Hu, 2021). Here, we will focus on the neural tangent parametrization (NTP), whose infinite-width limit leads to kernel behavior (Jacot et al., 2018). While the kernel regime cannot model all aspects of neural network behavior (Yehudai and Shamir, 2019; Ortiz-Jiménez et al., 2021; Vyas et al., 2022; Wenger et al., 2023), it is one of the models most amenable to theoretical analysis. In particular, at initialization, the function represented by certain infinite-width neural networks is distributed like a Gaussian process with the so-called neural network Gaussian process (NNGP) kernel (Neal, 1996; Daniely et al., 2016; Lee et al., 2018; Matthews et al., 2018). The training then follows the dynamics of kernel gradient descent with the so-called neural tangent kernel (NTK) (Jacot et al., 2018; Lee et al., 2019).

To understand the behavior of neural networks in the kernel regime, it is essential to understand the properties of the associated kernels themselves. In particular, the structure of the reproducing kernel Hilbert spaces (RKHSs) corresponding to these kernels is central to further theoretical analysis. Here, we limit ourselves to fully connected neural networks, which are sufficient to study the influence of activation functions, and are still practically relevant by themselves (Holzmüller et al., 2024; Gorishniy et al., 2025; Ye et al., 2024; Erickson et al., 2025). While these

<footnote>*Equal contribution.</footnote>
kernels have been analyzed for (powers of) the ReLU activation and infinitely smooth activation functions (Chen and Xu, 2021; Bietti and Bach, 2021; Vakili et al., 2023), not much is known about their structure for other activation functions.

**Contribution** Our main contribution is Theorem 9, illustrated in Remark 10, where we prove a general result analyzing the structure of the RKHS corresponding to the NTK and NNGP kernels on the sphere $\mathbb{S}^d$ of fully connected neural networks. Our proof, of which an overview is given in Appendix A, builds upon the analysis of Bietti and Bach (2021), but generalizes existing results in multiple ways:

* We study a significantly larger class of activations. In particular, we provide exact characterizations for typical activation functions that are infinitely smooth everywhere except at zero. We also provide new results for polynomial activation functions, and we more precisely analyze the conditions under which results for infinitely smooth activation functions hold.

* We study a more general class of fully connected neural networks, covering the cases of normally distributed bias, zero-initialized bias, and no bias. We also study all special cases arising for two-layer networks.

* We analyze all cases for both NNGP and NTK (except discontinuous activation functions, for which the NTK is not well-defined).

As an additional contribution, in Theorem 13, we show how the results on the RKHS translate to the smoothness of functions sampled from the Gaussian process with the NNGP kernel, and hence the smoothness of infinitely wide neural networks at initialization.

After discussing preliminaries (Section 2), we introduce our main results in Section 3 and Section 4. Section 5 discusses some implications of our results on the broader infinite-width theory, such as the (non-)benefit of depth. Related work is discussed in Section 6, before Section 7 concludes with a discussion of possible extensions.

## 2 Preliminaries

**Neural kernels** For $d \ge 1$, we consider fully connected NNs in neural tangent parametrization (NTP) operating on inputs $\mathbf{x} \in \mathbb{R}^{d_0}$, $d_0 := d+1$. Typically, neural networks use componentwise-normalized input vectors $\mathbf{x} \in \mathbb{R}^{d_0}$ such that $\|\mathbf{x}\|_2 \approx \sqrt{d_0}$. Following related theoretical literature, we instead consider inputs $\mathbf{x} \in \mathbb{S}^{d_0-1}$ and compensate for this by rescaling the first layer by $\sqrt{d_0}$, which leads to equivalent learning dynamics with gradient descent. We consider the following network architecture:

**Definition 1.** Let $L \ge 2$ be the number of layers, let $\sigma_w > 0$, and let $\sigma_b, \sigma_i \ge 0$. The network is the function $f_\theta: \mathbb{R}^{d_0} \to \mathbb{R}^{d_L}$ given by $f_\theta(\mathbf{x}^{(0)}) := z^{(L)}$, where

$$ \begin{aligned} z^{(1)} &:= \sigma_w \mathbf{W}^{(1)} \mathbf{x}^{(0)} + \sigma_b \mathbf{b}^{(1)} \in \mathbb{R}^{d_1} \\ z^{(l)} &:= \frac{\sigma_w}{\sqrt{d_{l-1}}} \mathbf{W}^{(l)} \mathbf{x}^{(l-1)} + \sigma_b \mathbf{b}^{(l)} \in \mathbb{R}^{d_l}, \quad \mathbf{x}^{(l-1)} := \varphi(z^{(l-1)}) \in \mathbb{R}^{d_l}. \quad (l \ge 2) \end{aligned} $$

We assume that all parameters are initialized independently as $\mathbf{W}_{jk}^{(l)} \sim \mathcal{N}(0, 1)$ and $\mathbf{b}_j^{(l)} \sim \mathcal{N}(0, \sigma_i^2)$. $\llap{\blacksquare}$

Definition 1 subsumes the three cases of

* bias-free networks by setting $\sigma_b = 0$,

* zero-initialized biases by setting $\sigma_b \ne 0$, $\sigma_i = 0$, and

* randomly initialized biases by setting $\sigma_b, \sigma_i \ne 0$.

For $\sigma_i^2 = 1$, the recursive formula for NTK and NNGP kernel are derived in Lee et al. (2019). We derive formulas for the different possible values of $\sigma_i, \sigma_b, \sigma_w$ in Lemma D.2.

To analyze NTK and NNGP kernels, we restrict them to the sphere $\mathbb{S}^d$, where they are dot-product kernels due to the rotation-invariance induced by the Gaussian initialization of the weight matrices $\mathbf{W}^{(l)}$.

**Dot-product kernels on the sphere** Let $\kappa: [-1, 1] \to \mathbb{R}$ such that $k := k_{\kappa,d} : \mathbb{S}^d \times \mathbb{S}^d \to \mathbb{R}$, $(\mathbf{x}, \mathbf{x}') \mapsto \kappa(\langle \mathbf{x}, \mathbf{x}' \rangle)$ is a (positive semidefinite) kernel. Then, $k$ is called a *dot-product kernel*. We can leverage the theory of dot-product kernels on spheres to study the RKHS $\mathcal{H}_k$ associated with $k$ (see e.g. Bietti and Bach, 2021; Hubbert et al., 2023; Haas et al., 2023): Let $\{Y_{l,1}, \dots, Y_{l,N_{l,d}}\}$ be an orthonormal basis of the space of spherical harmonics of degree $l$ in $L_2(\mathbb{S}^d)$. The associated integral operator $T_k : L_2(\mathbb{S}^d) \to L_2(\mathbb{S}^d)$ given by

$$ (T_k f)(\mathbf{x}) = \int_{\mathbb{S}^d} k(\mathbf{x}, \mathbf{x}') f(\mathbf{x}') \, d\mathbf{x}' $$
can be diagonalized as

$$T_k f = \sum_{l=0}^{\infty} \sum_{i=1}^{N_{l,d}} \mu_l Y_{l,i} \langle Y_{l,i}, f \rangle_{L_2(\mathbb{S}^d)}.$$

The RKHS $\mathcal{H}_k$ associated with $k$ can be written as

$$\mathcal{H}_k = \left\{ \sum_{l=0}^{\infty} \sqrt{\mu_l} \sum_{i=1}^{N_{l,d}} a_{l,i} Y_{l,i} \left| \sum_{l=0}^{\infty} \sum_{i=1}^{N_{l,d}} a_{l,i}^2 < \infty \right. \right\}$$

and its inner product as

$$\left\langle \sum_{l=0}^{\infty} \sqrt{\mu_l} \sum_{i=1}^{N_{l,d}} a_{l,i} Y_{l,i}, \sum_{l=0}^{\infty} \sqrt{\mu_l} \sum_{i=1}^{N_{l,d}} b_{l,i} Y_{l,i} \right\rangle_{\mathcal{H}_k} = \sum_{l=0}^{\infty} \sum_{i=1}^{N_{l,d}} a_{l,i} b_{l,i}.$$

Note that the eigenfunctions $Y_{l,i}$ are the same for all dot-product kernels but the eigenvalues $\mu_l = \mu_l(k) = \mu_l(\kappa, d)$ can be different. Hence, the RKHS is characterized by 1) the asymptotic behavior of the eigenvalues and 2) the set of indices for which the eigenvalues are nonzero.

We introduce some adapted asymptotic notation:

**Definition 2 (Asymptotic notation).** For functions $\kappa$, we use standard asymptotic notation like $\kappa(t) = O(t^{1/2})$ for $t \searrow 0$. For sequences $(a_n)_{n \in I}$, $(b_n)_{n \in I} \subseteq \mathbb{R}_{\ge 0}$ indexed by $I \subseteq \mathbb{N}_0$, we use an index “$\forall n$” for asymptotic notation to denote that it should hold for all $n$ and not only for almost all $n$, and to indicate the variable $n$ that the constant is independent of. Specifically,

$$a_n = O_{\forall n}(b_n) :\Leftrightarrow \exists C > 0 : \forall n \in I : a_n \le C b_n$$

$$a_n = \Omega_{\forall n}(b_n) :\Leftrightarrow b_n = O_{\forall n}(a_n)$$

$$a_n = \Theta_{\forall n}(b_n) :\Leftrightarrow a_n = O_{\forall n}(b_n) \text{ and } b_n = O_{\forall n}(a_n)$$

$$a_n = o_{\forall n}(b_n) :\Leftrightarrow a_n = O_{\forall n}(b_n) \text{ and } \forall C > 0 : \exists n_0 \in \mathbb{N}_0 : \forall n \ge n_0 : a_n \le C b_n$$

$$a_n = \omega_{\forall n}(b_n) :\Leftrightarrow b_n = o_{\forall n}(a_n).$$

Using this notation, when we write $\mu_l = \Theta_{\forall l}((l+1)^{-d-2s})$, it implies that all $\mu_k$ are positive. We write $l+1$ and not $l$ to make the right-hand side well-defined for all $l \in \mathbb{N}_0$. The constant $C$ implied in this notation can depend on $d$ and $s$. $\blacksquare$

We call two Hilbert spaces $\mathcal{H}_1, \mathcal{H}_2$ equivalent if they are equal as sets and their norms are equivalent. Polynomial eigenvalue decays lead to RKHSs equivalent to Sobolev spaces:

**Lemma 3 (Sobolev dot-product kernels on the sphere).** Let $d \in \mathbb{N}$ and $s > d/2$. The RKHS $\mathcal{H}_k$ corresponding to a kernel $k$ on $\mathbb{S}^d$ is equivalent to the Sobolev space $H^s(\mathbb{S}^d)$ if and only if $\mu_l = \Theta_{\forall l}((l+1)^{-2s})$.

*Proof.* Depending on the definition of $H^s(\mathbb{S}^d)$, this follows either directly or from classical theory, see Hubbert et al. (2023) and Lemma B.1 in Haas et al. (2023). $\blacksquare$

# 3 The structure of neural kernels

Before we introduce our main theorem, we need to impose some assumptions on the involved activation functions.

**Definition 4 (Functions with polynomially bounded derivatives).** Let $I \subseteq \mathbb{R}$ be an interval. Then, we define the set $S^{(\infty)}(I)$ to contain all functions $\varphi_I \in C^{\infty}(I)$ with polynomially bounded derivatives. In other words: For all $m \in \mathbb{N}_0$, there exist $a_m, b_m, q_m > 0$ such that for all $x \in I$, $|\varphi_I^{(m)}(x)| \le a_m|x|^{q_m} + b_m$. $\blacksquare$

Polynomial boundedness for $m \in \{0, 1\}$ is required to get the NTK formulas from (Yang, 2020, Theorem 7.2, Box 1); for higher derivatives we only need $\varphi_I^{(m)} \in L_2(\mathcal{N}(0, \sigma^2))$ for all $\sigma^2 > 0$.

**Assumption 5 (Activation function).** We assume that $\varphi: \mathbb{R} \to \mathbb{R}$ is of the form$^2$

$$\varphi(x) = \begin{cases} \varphi_{+}(x) & , x > 0 \\ \varphi_{-}(x) & , x < 0 \\ \frac{1}{2}(\lim_{x \searrow 0} \varphi_{+}(x) + \lim_{x \searrow 0} \varphi_{-}(x)) & , x = 0 \end{cases}$$

with $\varphi_{-} \in S^{(\infty)}((-\infty, 0))$ and $\varphi_{+} \in S^{(\infty)}((0, \infty))$ and that $\varphi$ is not the zero function. $\blacksquare$

<footnote>$^2$The definition at zero ensures that $\varphi$ is continuous when $\varphi_+$ and $\varphi_-$ allow, and that its even/odd parts are zero everywhere whenever they are zero almost everywhere.</footnote>
<table><caption>Table 1: Activation functions and their smoothness (Definition 7). Here, $\mathcal{P}_m$ means that the function is a polynomial of degree $m$. $\mathcal{P}_{-\infty}$ means that the function is zero. For a more extensive overview, we refer to Dubey et al. (2022).</caption><thead><tr><th rowspan="2">Activation</th><th colspan="2">Formula</th><th colspan="3">Smoothness s</th></tr><tr><th>$x < 0$</th><th>$x \ge 0$</th><th>$\varphi$</th><th>$\varphi_{\text{even}}$</th><th>$\varphi_{\text{odd}}$</th></tr></thead><tbody><tr><td>ReLU</td><td>0</td><td>$x$</td><td>1</td><td>1</td><td>$\mathcal{P}_1$</td></tr><tr><td>LeakyReLU ($\varepsilon \ne -1$)</td><td>$-\varepsilon x$</td><td>$x$</td><td>1</td><td>1</td><td>$\mathcal{P}_1$</td></tr><tr><td>SELU</td><td>$\lambda\alpha(e^x - 1)$</td><td>$\lambda x$</td><td>1</td><td>1</td><td>2</td></tr><tr><td>ELU ($\alpha \ne 1$)</td><td>$\alpha(e^x - 1)$</td><td>$x$</td><td>1</td><td>1</td><td>2</td></tr><tr><td>ELU ($\alpha = 1$), CELU</td><td>$\alpha(e^{x/\alpha} - 1)$</td><td>$x$</td><td>2</td><td>3</td><td>2</td></tr><tr><td>RePU, power $m$ even</td><td>0</td><td>$x^m$</td><td>$m$</td><td>$\mathcal{P}_m$</td><td>$m$</td></tr><tr><td>RePU, power $m$ odd</td><td>0</td><td>$x^m$</td><td>$m$</td><td>$m$</td><td>$\mathcal{P}_m$</td></tr><tr><td>Heaviside</td><td>$\frac{1}{2}1_{\{0\}}(x) + 1_{\mathbb{R}_{>0}}(x)$</td><td></td><td>0</td><td>$\mathcal{P}_0$</td><td>0</td></tr><tr><td>tanh</td><td>$(e^x - e^{-x})/(e^x + e^{-x})$</td><td></td><td>$\infty$</td><td>$\mathcal{P}_{-\infty}$</td><td>$\infty$</td></tr><tr><td>Sigmoid</td><td>$1/(1 + e^{-x})$</td><td></td><td>$\infty$</td><td>$\mathcal{P}_0$</td><td>$\infty$</td></tr><tr><td>GeLU</td><td>$\frac{1}{2}x(1 + \operatorname{erf}(x/\sqrt{2}))$</td><td></td><td>$\infty$</td><td>$\infty$</td><td>$\mathcal{P}_1$</td></tr><tr><td>SiLU</td><td>$x/(1 + e^{-x})$</td><td></td><td>$\infty$</td><td>$\infty$</td><td>$\infty$</td></tr><tr><td>RBF</td><td>$\exp(-x^2)$</td><td></td><td>$\infty$</td><td>$\infty$</td><td>$\mathcal{P}_{-\infty}$</td></tr></tbody></table>

**Proposition 6** (All common activation function satisfy Assumption 5).

(a) If $f$ is constructed by addition, multiplication, and composition of polynomials, sigmoid, tanh, softplus, sin, cos, RBF and $\Phi$, then $f \in S^{(\infty)}(\mathbb{R})$. Moreover, the functions $g(x) = \exp(ax)$ are in $S^{(\infty)}((-\infty, 0])$ for every $a \ge 0$.

(b) Assumption 5 is satisfied for all activation functions from Table 1. It is also satisfied for all $\phi \in S^{(\infty)}$.

We prove Proposition 6 in Appendix E.

**Definition 7** (Smoothness of an activation function). For an activation function $\varphi$ as in Assumption 5, we define its smoothness as

$$ \text{smoothness}(\varphi) := \inf\{m \in \mathbb{N}_0 \mid \lim_{t \searrow 0} \varphi^{(m)}(t) \ne \lim_{t \searrow 0} \varphi^{(m)}(t)\} \in \mathbb{N}_0 \cup \{\infty\}. \quad \llap{\vphantom{\lim_{t \searrow 0} \varphi^{(m)}(t)}} $$

**Example 8** (Smoothness of the ReLU activation). The ReLU activation function $\varphi(x) = \max\{0, x\}$ fulfills Assumption 5 and decomposes as $\varphi_+(x) = x, x > 0, \varphi_-(x) = 0, x < 0$. Hence we have

$$ \lim_{t \searrow 0} \varphi(t) = 0 = \lim_{t \searrow 0} \varphi(t) \quad \text{but} \quad \lim_{t \searrow 0} \varphi'(t) = 1 \ne 0 = \lim_{t \searrow 0} \varphi'(t), $$

$$ \text{implying smoothness}(\text{ReLU}) = 1. \quad \llap{\vphantom{\lim_{t \searrow 0} \varphi^{(m)}(t)}} $$

We now introduce our main result, which contains quite some special cases. The “regular” cases are discussed afterward in Remark 10. These special cases in the bias-free case involve the common even/odd decomposition

$$ \varphi_{\text{even}}(x) := \frac{\varphi(x) + \varphi(-x)}{2}, \quad \varphi_{\text{odd}}(x) := \frac{\varphi(x) - \varphi(-x)}{2}, $$

fulfilling $\varphi = \varphi_{\text{even}} + \varphi_{\text{odd}}$. We additionally define

$$ \mathcal{F}_{\text{even/odd}} := \{f : \mathbb{S}^d \to \mathbb{R} \mid f \text{ is even/odd}\}, $$

where the context determines the parity even/odd. For example, $\text{ReLU}_{\text{even}}(x) = \frac{1}{2}|x|$ and $\text{ReLU}_{\text{even}}(x) = \frac{1}{2}x$. The function $f_\varphi$ of a two-layer network with activation $\varphi$ can be written as $f_\varphi = f_{\varphi_{\text{even}}} + f_{\varphi_{\text{odd}}}$, and in the bias-free case ($\sigma_b^2 = 0$), $f_{\varphi_{\text{even}}}$ is even and $f_{\varphi_{\text{odd}}}$ is odd. Hence, $(f_{\text{ReLU}})_{\text{odd}} = f_{\text{ReLU}_{\text{odd}}}$ is a linear function, which explains the results of Bietti and Mairal (2019) where the odd eigenvalues satisfy $\mu_1 > 0, \mu_3 = \mu_5 = \dots = 0$. A much more general version of this calculation is performed in Proposition D.4.

By $\mathcal{P}_m$ we denote the polynomials of degree at most $m$. We say that two Hilbert spaces $H_1, H_2$ are equivalent, or $H_1 \cong H_2$, if they are equal as sets and their norms are equivalent.

**Theorem 9** (Main result, summary below). Let the activation $\varphi$ fulfill Assumption 5.

(NNGP) i) If $\sigma_b^2 \sigma_i^2 > 0$ or ($L \ge 3$ and $\varphi$ is neither even nor odd), let $s := \text{smoothness}(\varphi)$.
a) If $s = 0$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong H^{d/2+2^{1-L}}(\mathbb{S}^d)$.

b) If $1 \le s < \infty$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong H^{d/2+s+1/2}(\mathbb{S}^d)$.

c) If $s = \infty$ and $\varphi$ is not a polynomial, then $\mathcal{H}_{k_L^{\text{NNGP}}} \subset H^t(\mathbb{S}^d)$ for all $t \in \mathbb{R}$ and $H_{k_L^{\text{NNGP}}}$ contains all polynomials.

d) If $s = \infty$ and $\varphi$ is a polynomial of degree $m$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong \mathcal{P}_{mL-1}$.

ii) If $\sigma_b^2 \sigma_i^2 = 0$ and $\varphi$ is even/odd, let $s := \text{smoothness}(\varphi)$.

a) If $s = 0$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong H^{d/2+2^{1-L}}(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$.

b) If $1 \le s < \infty$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong H^{d/2+s+1/2}(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$.

c) If $s = \infty$ and $\varphi$ is not a polynomial, then $\mathcal{H}_{k_L^{\text{NNGP}}} \subset H^t(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$ for all $t \in \mathbb{R}$ and $H_{k_L^{\text{NNGP}}}$ contains all even/odd polynomials.

d) If $s = \infty$ and $\varphi$ is a polynomial of degree $m$, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong \mathcal{P}_{mL-1} \cap \mathcal{F}_{\text{even/odd}}$.

iii) If $\sigma_b^2 \sigma_i^2 = 0$, $L = 2$, and $\varphi$ is not even or odd, then $\mathcal{H}_{k_L^{\text{NNGP}}} \cong H_{\varphi_{\text{even},L}}^{\text{NNGP}} \oplus H_{\varphi_{\text{odd},L}}^{\text{NNGP}}$ where the RKHSs $\mathcal{H}_{k_L^{\text{NNGP}}, \varphi_{\text{even},L}}$, $\mathcal{H}_{k_L^{\text{NNGP}}, \varphi_{\text{odd},L}}$ are corresponding to the even/odd activation functions $\varphi_{\text{even}}$ or respectively $\varphi_{\text{odd}}$ and are given in case ii).

(NTK) i) If $\sigma_b^2 > 0$ or ($L \ge 3$ and $\varphi$ is neither even nor odd), let $s := \text{smoothness}(\varphi)$.

a) If $1 \le s < \infty$, then $\mathcal{H}_{k_L^{\text{NTK}}} \cong H^{d/2+s-1/2}(\mathbb{S}^d)$.

b) If $s = \infty$ and $\varphi$ is not a polynomial, then $\mathcal{H}_{k_L^{\text{NTK}}} \subset H^t(\mathbb{S}^d)$ for all $t \in \mathbb{R}$ and $H_{k_L^{\text{NTK}}}$ contains all polynomials.

c) If $s = \infty$ and $\varphi$ is a polynomial of degree $m$, then $\mathcal{H}_{k_L^{\text{NTK}}} \cong \mathcal{P}_{mL-1}$.

ii) If $\sigma_b^2 = 0$ and $\varphi$ is even/odd, let $s := \text{smoothness}(\varphi)$.

a) If $1 \le s < \infty$, then $\mathcal{H}_{k_L^{\text{NTK}}} \cong H^{d/2+s-1/2}(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$.

b) If $s = \infty$ and $\varphi$ is not a polynomial, then $\mathcal{H}_{k_L^{\text{NTK}}} \subset H^t(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$ for all $t \in \mathbb{R}$ and $H_{k_L^{\text{NTK}}}$ contains all even/odd polynomials.

c) If $s = \infty$ and $\varphi$ is a polynomial of degree $m$, then $\mathcal{H}_{k_L^{\text{NTK}}} = \mathcal{P}_{mL-1} \cap \mathcal{F}_{\text{even/odd}}$.

iii) If $\sigma_b^2 = 0$, $L = 2$, and $\varphi$ is not even or odd, then $\mathcal{H}_{k_L^{\text{NTK}}} \cong H_{k_L^{\text{NTK}}, \varphi_{\text{even},L}} \oplus H_{k_L^{\text{NTK}}, \varphi_{\text{odd},L}}$ where the RKHSs $\mathcal{H}_{k_L^{\text{NTK}}, \varphi_{\text{even},L}}$, $\mathcal{H}_{k_L^{\text{NTK}}, \varphi_{\text{odd},L}}$ are corresponding to the even/odd activation functions $\varphi_{\text{even}}$ or respectively $\varphi_{\text{odd}}$ and are given in case ii).

Note that $H^t(\mathbb{S}^d) \cap \mathcal{F}_{\text{even/odd}}$ is a closed subspace of $H^t(\mathbb{S}^d)$ for any $t$. We equip it with the restricted norm.

*Proof.* Lemma D.2 yields the convergence of the NNGP- and NTK-kernels. Theorem D.10 combined with Lemma 3 and a careful case distinction then yields the claim. $\square$

Note that the results of Theorem 9 can be expressed in terms of the eigenvalues $\mu_l$ of the integral operator instead of the RKHSs as discussed in Section 2; the eigenvalue-based formulation can be found in Theorem D.10.

**Remark 10 (General takeaway).** Theorem 9 is most easy to understand for the case where

* the NN has $L \ge 2$ layers, contains biases ($\sigma_b^2 > 0$), and in the NNGP case the biases are initialized with nonzero variance ($\sigma_i^2 > 0$), or

* the NN has $L \ge 3$ layers and the activation function $\varphi$ is neither even nor odd.

In this case, the decay of the eigenvalues $\mu_l$ and the corresponding RKHS depend on the smoothness $s$ of the activation function $\varphi$ as follows:

* If the activation function is a polynomial, for NTK and NNGP only finitely many eigenvalues $\mu_l$ are nonzero and the RKHSs only contain polynomials, whose maximum degree grows exponentially with the depth of the network.

* If the activation function is infinitely smooth ($s = \infty$) but not a polynomial, all $\mu_l$ are nonzero but decay faster than any inverse polynomial. Hence, the RKHSs are contained in all Sobolev spaces, but the kernels are universal. This applies for example to the GELU, SiLU/Swish, Mish, softplus, sigmoid, and tanh activation functions (though tanh is an odd function, so the conclusion only holds with biases).

* If the activation function has finite smoothness $1 \le s < \infty$, the RKHSs are equivalent to Sobolev spaces of order $d/2+s-1/2$ for the NTK and $d/2+s+1/2$ for the NNGP. This applies to ReLU, LeakyReLU, ELU ($\alpha \ne 1$) and SELU with $s = 1$ and to CELU with $s = 2$. Rectified power unit activations $\max\{0, x\}^s$ have
smoothness $s \in \mathbb{N}$, yielding a family of Sobolev spaces. Our results show that this is due to the behavior at zero and that the superlinear growth of these functions for $s \ge 2$ is not necessary.

* If the activation function is discontinuous ($s = 0$), the RKHS of the NNGP kernel is equivalent to a Sobolev space of smoothness $d/2 + 2^{1-L}$, while the NTK is not defined. ▲

**Example 11 (Special cases).** As mentioned above, two-layer networks without bias ($\sigma_b^2 = 0$ for NTK or $\sigma_b^2\sigma_i^2 = 0$ for NNGP) can have special behavior. Following Table 1, we obtain for CELU the NTK RKHS $(H^{d/2+5/2} \cap \mathcal{F}_{\text{even}}) \oplus (H^{d/2+3/2} \cap \mathcal{F}_{\text{odd}})$, so the even parts of functions are smoother than the odd parts of functions. For ELU ($\alpha \ne 1$) and SELU, we obtain $(H^{d/2+1/2} \cap \mathcal{F}_{\text{even}}) \oplus (H^{d/2+3/2} \cap \mathcal{F}_{\text{odd}})$. For ReLU and LeakyReLU, the odd parts are just linear functions. For RePU, one of the two parts are polynomials up to a certain degree.

The other special case is for even or odd activations and arbitrarily deep networks without bias. For example, tanh is an odd activation, hence the corresponding NNGP and NTK RKHSs will only contain odd functions. ▲

**Remark 12 (NTKs and NNGPs of non-smooth activations are equivalent to Matérn kernels).** Chen and Xu (2021) and Bietti and Bach (2021) showed that on $\mathbb{S}^d$, the RKHS of the deep ReLU NTK is equivalent to that of the Laplace kernel. Theorem 1 in Vakili et al. (2021) further shows that the RKHSs of deep RePU NTKs and NNGPs are equivalent to the ones of Matérn kernels, which are equivalent to Sobolev spaces. Specifically, for the Matérn kernel $k_\nu$ of order $\nu > 0$ on the sphere $\mathbb{S}^d$, it holds that $H_{k_\nu} \cong H^{d/2+\nu}(\mathbb{S}^d)$ (see also Proposition 5.2 (c) in Hubbert et al. 2023). For other not-infinitely-smooth activations, our Theorem 9 shows that their RKHSs are generally also equivalent to Sobolev spaces, and hence the corresponding kernels are equivalent to Matérn kernels. ▲

# 4 Smoothness of NNGP sample paths

**Theorem 13.** Let $k$ be a dot-product kernel on $\mathbb{S}^d$ whose RKHS is equivalent to a Sobolev space $H^{d+\alpha}(\mathbb{S}^d)$, $\alpha > 0$. Let $X$ be a Gaussian process on $\mathbb{S}^d$ with zero mean and covariance kernel $k$.

i) For any $\varepsilon > 0$ we have $P(Y \in H^{\alpha+d/2+\varepsilon}(\mathbb{S}^d)) = 0$ for any version $Y$ of $X$.

ii) For any $0 < \varepsilon < \alpha$, there exists a version $Y$ of $X$ such that $\mathbb{P}(Y \in H^{d/2+\alpha-\varepsilon}) = 1$ holds.

Theorem 13 is proven in Appendix F. Note that $H^{d/2+\alpha-\varepsilon}(\mathbb{S}^d) \hookrightarrow C(\mathbb{S}^d)$ holds for $0 < \varepsilon < \alpha$, i.e., the embedding id: $H^{d/2+\alpha-\varepsilon} \to C(\mathbb{S}^d)$, $f \mapsto f$ is continuous.

**Example 14 (Application to infinite-width networks).** Consider for simplicity the case with biases ($\sigma_b^2\sigma_i^2 > 0$) and an activation of smoothness $1 \le s < \infty$. From Theorem 9 we know $H_{\kappa\text{NTK}} \cong H^{d/2+s-1/2}(\mathbb{S}^d)$ and $H_{\kappa\text{NNGP}} \cong H^{d/2+s+1/2}(\mathbb{S}^d)$. If $s + 1/2 > d/2$, by Theorem 13, the NNGP sample paths are in $H^{s+1/2-\varepsilon}(\mathbb{S}^d)$ but not $H^{s+1/2+\varepsilon}(\mathbb{S}^d)$ for any $\varepsilon > 0$. We conjecture that randomly initialized finite-width networks are only in $H^s(\mathbb{S}^d)$ but not $H^{s+\varepsilon}(\mathbb{S}^d)$, so the infinite-width limit would gain an extra half-order of smoothness. ▲

# 5 Implications

Our results can be used to obtain a more complete understanding of neural networks in the kernel regime. Below, we list some non-exhaustive ways in which our results could complement existing theory.

By showing that the RKHSs of deep and shallow ReLU NTKs are equivalent, Bietti and Bach (2021) concluded that the NTK regime is not sufficient to model the benefits of depth for practical neural networks. Our results show that this holds for a much larger class of not infinitely smooth activation functions. On the other hand, our results show that the RKHSs are not equivalent for polynomial activations, while the situation for other infinitely smooth activations remains unclear, as more precise results are only known in the two-layer case (Murray et al., 2023).

Regarding theoretical modeling, the dynamics of gradient flow are difficult to study for ReLU activations since their derivative is discontinuous at zero. The use of smoother but not infinitely smooth activations such as CELU can enable theoretical studies of gradient flow in a setting where the RKHS is well-known, without limiting results to powers of ReLU as in previous work (Vakili et al., 2023). For example, the gradient flow analysis of Bowman and Montufar (2022) yields finite-time bounds for the deviation from the infinite-width training trajectory but requires activation functions smoother than ReLU. The use of semi-smooth activation functions might also be interesting to the NTK analysis of physics-informed neural networks (Wang et al., 2022), since their training involves taking higher-order derivatives of neural networks.

Our main result allows the application of generalization results to more activation functions. For example, the inconsistency results of Haas et al. (2023) for overfitting with NTKs and NNGPs apply whenever the RKHS is a Sobolev
space, and hence by our results, they apply to a wide range of finitely smooth activation functions. Our results also suggest that the spectral bias of neural networks (Rahaman et al., 2019) can be influenced by changing the smoothness of the activation function. Finally, while Simon et al. (2022) show that a large class of kernels can be represented as NTKs and NNGPs, our results help to elucidate the properties of activation functions that realize these kernels.

Compared to the pure kernel case, gradient flow on infinite-width NNs in the kernel regime also has to learn to remove the random initial function (Lee et al., 2019). Thanks to Theorem 13, we now know the smoothness of this function, which makes it possible to derive convergence rates for the regression of this function with the NTK. To this end, we refer to the proof of Theorem G.5 in Haas et al. (2023) for convergence rates with regularization in the noisy case, and to Theorem 3.3 in Le Gia et al. (2006) for an approach towards convergence rates of interpolation.

# 6 Related work

**Neural kernels** Multiple results have shown that in typical settings, infinite-width neural networks behave like Gaussian processes at initialization (Neal, 1996; Daniely et al., 2016; Lee et al., 2018; Matthews et al., 2018). The associated covariance function is known as the neural network Gaussian process (NNGP) kernel or random features kernel. Jacot et al. (2018) discovered that the training dynamics of such infinite-width neural networks can be described by a different kernel, the so-called neural tangent kernel (NTK). The same observation has been made by Lee et al. (2019). Follow-up work has generalized the scope of these results (e.g., Arora et al., 2019; Yang, 2019b, 2020). These works do not yet give deep insights into the nature of the NTK and NNGP kernels.

**Smoothness of neural kernels** Belkin et al. (2018) noted similarities in generalization behavior between the Laplace kernel and neural networks. A follow-up line of work managed to characterize the RKHS of different NTKs on $S^d$. Bietti and Mairal (2019) derived the structure of the RKHS of the two-layer ReLU NTK without bias ($\sigma_b^2 = 0$). Basri et al. (2019) analyzed two-layer ReLU NTKs with and without bias. Geifman et al. (2020) showed that the two-layer ReLU NTK with bias is equivalent to the Laplace kernel, and showed that the RKHS of the deep ReLU NTK is at least as large as the RKHS of the shallow one. The equivalence of these RKHSs was then shown by Chen and Xu (2021) for $\sigma_b^2 > 0$, $\sigma_i = 0$, $L \ge 2$. Simultaneously, Bietti and Bach (2021) characterized the structure of the deep ReLU NTK and NNGP without bias ($\sigma_b^2 = 0$, $L \ge 3$) and also showed some results for step activations as well as infinitely differentiable activations. Vakili et al. (2023) extended this analysis to RePU (integer powers of the ReLU activation). Finally, Haas et al. (2023) formally established the connection of these RKHSs to Sobolev spaces on the sphere. Murray et al. (2023) prove further spectral properties of the NTK, including a characterization of the RKHS of two-layer NNs for tanh and RBF activation functions that is more precise than our result for these activations. Scetbon and Harchaoui (2021) prove some results about spectral properties of dot-product kernels, with weak results concerning the NTK. However, to the best of our knowledge, we are the first to provide a characterization of the RKHS for many non-smooth activation functions like LeakyReLU, SELU, or ELU.

The NTK has also been analyzed for other architectures such as residual networks (Belfer et al., 2024) and convolutional residual networks (Barzilai et al., 2023). Dandi and Jacot (2021) investigate the contributions of individual layers towards the properties of the NTK. Simon et al. (2022) show that a large class of dot-product kernels on the sphere can be realized as NTK or NNGP kernels through suitable activation functions. While most results analyze the NTK on $S^d$, Lai et al. (2023) analyze the eigenvalue decay on $\mathbb{R}$ and Li et al. (2024) leverage known results on $S^d$ to obtain decay rates for $\mathbb{R}^{d+1}$. In terms of activation functions, these results do not go beyond the known results for $S^d$.

**Spectral properties** Panigrahi et al. (2019) investigate minimum eigenvalues of NTK matrices and find a dependence on the smoothness of the activation function. Nguyen et al. (2021) and Karhadkar et al. (2024) provide more bounds for minimum eigenvalues. These results are related to the structure of the RKHS through kernel matrix concentration inequalities but do not reveal the full structure of the RKHS.

# 7 Conclusion

We have shown general results for the structure of the RKHS for fully connected neural networks with different activation functions, as well as results for the sample path smoothness of randomly initialized wide neural networks. We refer the reader to Appendix A for a short overview of the central techniques and objects of our main proof.

**Possible extensions** Our work offers several possibilities for extensions. First, following Li et al. (2024), an extension of our results from $x \in S^d$ to $x \in \mathbb{R}^{d+1}$ could be possible. Second, one could study different activation functions in different layers; we strongly conjecture that the behavior of the NTK and NNGP will be dominated by the least smooth activation function. Third, while we study non-smoothness in zero, it might be possible to apply our proof
technique to functions that are non-smooth in other points—the main obstacle is to find reference functions $s_k$ with the given non-smoothness that are similarly amenable to analysis as the functions $s_k$ we use in the appendix. Fourth, a more precise characterization of the RKHS for infinitely smooth non-polynomial activations is still open but might require stronger tools (e.g., Minh et al., 2006; Azevedo and Menegatto, 2014; Murray et al., 2023). Finally, our results might be extensible to analyses for residual networks (e.g., Belfer et al., 2024; Tirer et al., 2022), convolutional networks (e.g., Geifman et al., 2022; Barzilai et al., 2023), or more general architectures including transformers (Yang, 2020).

## Acknowledgments

The authors thank Ingo Steinwart, Tizian Wenzel, Jens Wirth, and Alberto Bietti for valuable discussions.

This work is funded by the Deutsche Forschungsgemeinschaft (DFG) in the project STE 1074/5-1, within the DFG priority programme SPP 2298 “Theoretical Foundations of Deep Learning” (M.S.). M.S. thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support.

## Contribution statement

D.H. conceived the project and the high-level proof strategy. D.H. and M.S. contributed to proof writing and paper writing.

## References

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Russ R. Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In *Advances in Neural Information Processing Systems*, 2019.

D. Azevedo and V.A. Menegatto. Sharp estimates for eigenvalues of integral operators generated by dot product kernels on the sphere. *Journal of Approximation Theory*, 177:57–68, 2014. ISSN 0021-9045.

Daniel Barzilai, Amnon Geifman, Meirav Galun, and Ronen Basri. A kernel perspective of skip connections in convolutional networks. In *International Conference on Learning Representations*, 2023.

Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In *Advances in Neural Information Processing Systems*, 2019.

Yuval Belfer, Amnon Geifman, Meirav Galun, and Ronen Basri. Spectral analysis of the neural tangent kernel for deep residual networks. *Journal of Machine Learning Research*, 25(184):1–49, 2024.

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In *International Conference on Machine Learning*, 2018.

Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. In *International Conference on Learning Representations*, 2021.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In *Advances in Neural Information Processing Systems*, 2019.

M. S. Birman and M. Z. Solomjak. *Spectral Theory of Self-Adjoint Operators in Hilbert Space*. Springer, Dordrecht, 1987.

Benjamin Bowman and Guido F Montufar. Spectral bias outside the training set for deep networks in the kernel regime. *Advances in Neural Information Processing Systems*, 2022.

Johann S. Brauchart and Josef Dick. A characterization of Sobolev spaces on the sphere and an extension of Stolarsky's invariance principle to arbitrary smoothness. *Constructive approximation*, 38(3):397–445, 2013.

Lin Chen and Sheng Xu. Deep neural tangent kernel and laplace kernel have the same RKHS. In *International Conference on Learning Representations*, 2021.

Yatin Dandi and Arthur Jacot. Understanding layer-wise contributions in deep neural networks through spectral analysis. *arXiv:2111.03972*, 2021.

Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. *Advances in Neural Information Processing Systems*, 2016.

Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Activation functions in deep learning: A comprehensive survey and benchmark. *Neurocomputing*, 503:92–108, 2022.
Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data, 2025.

Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Ronen Basri. On the similarity between the Laplace and neural tangent kernels. *Advances in Neural Information Processing Systems*, 2020.

Amnon Geifman, Meirav Galun, David Jacobs, and Basri Ronen. On the spectral bias of convolutional neural tangent and Gaussian process kernels. *Advances in Neural Information Processing Systems*, 2022.

Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In *International Conference on Learning Representations*, 2025.

Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. In *Advances in Neural Information Processing Systems*, 2023.

David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data. *Advances in Neural Information Processing Systems*, 37:26577–26658, 2024.

Simon Hubbert, Quôc Thông Lê Gia, and Tanya M. Morton. *Spherical Radial Basis Functions, Theory and Applications*. Springer Briefs in Mathematics, 2015.

Simon Hubbert, Emilio Porcu, Chris J Oates, and Mark Girolami. Sobolev spaces, kernels and discrepancies over hyperspheres. *Transactions on Machine Learning Research*, 2023.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In *Advances in Neural Information Processing Systems*, 2018.

Kedar Karhadkar, Michael Murray, and Guido F Montufar. Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension. *Advances in Neural Information Processing Systems*, 37:138197–138249, 2024.

Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on $\mathbb{R}$. *arXiv:2302.05933*, 2023.

Quoc Thong Le Gia, Francis J Narcowich, Joseph D Ward, and Holger Wendland. Continuous and discrete least-squares approximation by radial basis functions on spheres. *Journal of Approximation Theory*, 143(1):124–133, 2006.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In *International Conference on Learning Representations*, 2018.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In *Advances in Neural Information Processing Systems*, 2019.

Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains. *Journal of Machine Learning Research*, 25(82):1–47, 2024.

Milan N. Lukić and Jay H. Beder. Stochastic processes with sample paths in reproducing kernel Hilbert spaces. *Transactions of the American Mathematical Society*, 353:3945–3969, 2001.

Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In *International Conference on Learning Representations*, 2018.

Ha Quang Minh, Partha Niyogi, and Yuan Yao. Mercer's theorem, feature maps, and smoothing. In *International Conference on Computational Learning Theory*. Springer, 2006.

Michael Murray, Hui Jin, Benjamin Bowman, and Guido Montufar. Characterizing the spectrum of the NTK via a power series expansion. In *The Eleventh International Conference on Learning Representations*, 2023.

Radford M Neal. Priors for infinite networks. *Bayesian learning for neural networks*, pages 29–53, 1996.

Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. In *International Conference on Machine Learning*, 2021.

Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? *Advances in Neural Information Processing Systems*, 2021.

Abhishek Panigrahi, Abhishek Shetty, and Navin Goyal. Effect of activation functions on the training of over-parametrized neural nets. *arXiv:1908.05660*, 2019.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In *International Conference on Machine Learning*, 2019.
Meyer Scetbon and Zaid Harchaoui. A spectral analysis of dot-product kernels. In *International Conference on Artificial Intelligence and Statistics*, 2021.

James Benjamin Simon, Sajant Anand, and Mike Deweese. Reverse engineering the neural tangent kernel. In *International Conference on Machine Learning*, 2022.

Ingo Steinwart. When does a Gaussian process have its paths in a reproducing kernel Hilbert space? *arXiv:2407.11898*, 2024.

Tom Tirer, Joan Bruna, and Raja Giryes. Kernel-based smoothness analysis of residual networks. In *Mathematical and Scientific Machine Learning*, 2022.

Sattar Vakili, Michael Bromberg, Jezabel Garcia, Da-shan Shiu, and Alberto Bernacchia. Uniform generalization bounds for overparameterized neural networks. *arXiv:2109.06099*, 2021.

Sattar Vakili, Michael Bromberg, Jezabel Garcia, Da-shan Shiu, and Alberto Bernacchia. Information gain and uniform generalization bounds for neural kernel models. In *IEEE International Symposium on Information Theory*, 2023.

Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the NTK for understanding generalization in deep learning. *arXiv:2206.10012*, 2022.

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. *Journal of Computational Physics*, 449:110768, 2022.

Jonathan Wenger, Felix Dangel, and Agustinus Kristiadi. On the disconnect between theory and practice of neural networks: Limits of the NTK perspective. *arXiv:2310.00137*, 2023.

Andreas Winkelbauer. Moments and absolute moments of the normal distribution. *arXiv:1209.4340*, 2012.

Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes. *arXiv:1910.12478*, 2019a.

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. *arXiv:1902.04760*, 2019b.

Greg Yang. Tensor programs II: Neural tangent kernel for any architecture. *arXiv:2006.14548*, 2020.

Greg Yang and Edward J Hu. Tensor programs IV: Feature learning in infinite-width neural networks. In *International Conference on Machine Learning*, 2021.

Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data. *arXiv:2407.00956*, 2024.

Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. *Advances in Neural Information Processing Systems*, 2019.
# Appendix

## Appendix Contents.

<table><tr><td>A Overview and notation</td><td>12</td></tr><tr><td>B Boundary behavior</td><td>14</td></tr><tr><td>C Proofs for dual activations</td><td>17</td></tr><tr><td>C.1 General properties</td><td>17</td></tr><tr><td>C.2 Dominating terms</td><td>19</td></tr><tr><td>C.3 Mix terms</td><td>22</td></tr><tr><td>C.4 Results for smooth terms</td><td>24</td></tr><tr><td>C.5 General activation functions</td><td>24</td></tr><tr><td>D Neural kernel proofs</td><td>27</td></tr><tr><td>D.1 Neural kernels</td><td>27</td></tr><tr><td>D.2 Even and odd parts of neural kernels</td><td>30</td></tr><tr><td>D.3 Adapting the main theorem from Bietti and Bach (2021)</td><td>31</td></tr><tr><td>D.4 Eigenvalue decay</td><td>32</td></tr><tr><td>D.5 A close look at the even/odd degree of G<sub>l</sub> and H<sub>l</sub></td><td>41</td></tr><tr><td>E Polynomial boundedness of derivatives</td><td>45</td></tr><tr><td>F Smoothness of GP sample paths</td><td>47</td></tr></table>
## A Overview and notation

Appendix A – D are dedicated to the proof of the main theorem, Theorem 9. They build on each other and should be read in order. Appendix E and Appendix F contain smaller proofs that are independent of the other appendices.

**Analysis through boundary behavior.** As mentioned before, the NTK and NNGP kernels on the sphere are dot-product kernels, i.e., they are of the form $k(x, x') = \kappa(\langle x, x' \rangle)$. To obtain eigenvalue decays for $k$, we build on the results of Bietti and Bach (2021), which requires to study the behavior of $\kappa(t)$ for $t \to 1$ and $t \to -1$. To unify these analyses, we let $\tau \in \{-1, 1\}$ and study the behavior of $\kappa_\tau : (0, 2) \to \mathbb{R}$, $t \mapsto \kappa(\tau(1-t))$ for $t \searrow 0$. We first introduce function classes with certain boundary behavior that will be central in simplifying computations later on:

**Definition A.1 (Function classes with controlled boundary behavior).** Let $\alpha, \beta, \gamma \in \mathbb{R}$. We define sets of functions $\mathcal{P}_{\alpha,\beta}$, $\mathcal{R}_{\gamma}$, and $\mathcal{Q}_{\alpha,\beta}$ as follows:

(a) **Sums of high-enough powers:** We define $\mathcal{P}_{\alpha,\beta}$ as the set of functions $f : (0, 2) \to \mathbb{R}$ of the form

$$f(t) = \sum_{i=1}^{n} a_i t^{\alpha_i} \quad \text{with} \quad n \in \mathbb{N}_0, a_i, \alpha_i \in \mathbb{R}, \begin{cases} \alpha_i > \alpha & \text{for all } i \text{ with } \alpha_i \in \mathbb{Z} \\ \alpha_i > \beta & \text{for all } i \text{ with } \alpha_i \notin \mathbb{Z} \end{cases}$$

Note the use of $>$, which will be convenient later.

(b) **Negligible remainder functions:** We define $\mathcal{R}_{\gamma}$ as the set of $C^{\infty}$-functions $f : (0, 2) \to \mathbb{R}$ such that for all $m \in \mathbb{N}_0$, the $m$-th derivative $f^{(m)}$ satisfies

$$|f^{(m)}(t)| \le O(t^{\gamma-m}) \quad \text{for } t \searrow 0.$$

Intuitively, $\mathcal{R}_{\gamma}$ contains functions that are at least as benign as $t \mapsto t^{\gamma}$, in terms of the behavior of all derivatives for $t \to 0$.

(c) **Sums of powers plus negligible remainder:** We define $\mathcal{Q}_{\alpha,\beta} := \bigcap_{\gamma \in \mathbb{R}} (\mathcal{P}_{\alpha,\beta} + \mathcal{R}_{\gamma})$. In other words, $\mathcal{Q}_{\alpha,\beta}$ contains those functions that are sums of powers plus an arbitrarily “high-order” remainder. Note for $f \in \mathcal{Q}_{\alpha,\beta}$ that in general, the number $n$ of terms in the sum of powers when writing $f$ as $f \in \mathcal{P}_{\alpha,\beta} + \mathcal{R}_{\gamma}$ will depend on the imposed order $\gamma$ of the remainder.

We will mostly use $\mathcal{Q}_{\alpha,\beta}$ in the paper. Instead of $f \in \mathcal{Q}_{\alpha,\beta}$, we also write $f(t) = \mathcal{Q}_{\alpha,\beta}(t)$ analogous to common $O$-notation. We often track coefficients of leading terms separately, e.g., writing $g(t) = at^{\alpha} + bt^{\beta} + \mathcal{Q}_{\alpha,\beta}(t)$. $\ll$

We build on the following result, formulated using the previously defined boundary function classes:

**Theorem A.2 (Adaptation of Theorem 7 in arXiv v4 of Bietti and Bach 2021).** Let $\kappa : [-1, 1] \to \mathbb{R}$ be a function that is smooth on $(-1, 1)$ such that $k_{\kappa,d}(\mathbf{x}, \mathbf{x}') = \kappa(\langle \mathbf{x}, \mathbf{x}' \rangle)$ is a positive semi-definite kernel on all spheres $\mathbb{S}^d$, $d \in \mathbb{N}_{\ge 1}$. Suppose that there exists $0 < \beta \in \mathbb{R} \setminus \mathbb{Z}$ and $b_{-1}, b_1 \in \mathbb{R}$ such that for $\tau \in \{-1, 1\}$,

$$\kappa(\tau(1-t)) = b_{\tau} t^{\beta} + \mathcal{Q}_{-1,\beta}(t).$$

Then, for a given dimension $d \in \mathbb{N}_{\ge 1}$, the eigenvalues $\mu_l = \mu_l(\kappa, d)$ as defined in Section 2 satisfy:

(a) For $l \in \mathbb{N}_0$ even, if $b_{-1} \neq -b_1$, then $\mu_l = \Theta_{\forall l}((l+1)^{-d-2\beta})$.

(b) For $l \in \mathbb{N}_0$ even, if $b_{-1} = -b_1$, then $\mu_l = o_{\forall l}((l+1)^{-d-2\beta})$.

(c) For $l \in \mathbb{N}_0$ odd, if $b_{-1} \neq b_1$, then $\mu_l = \Theta_{\forall l}((l+1)^{-d-2\beta})$.

(d) For $l \in \mathbb{N}_0$ odd, if $b_{-1} = b_1$, then $\mu_l = o_{\forall l}((l+1)^{-d-2\beta})$.

This leads to the following strategy, detailed below, for analyzing the eigenvalues of neural kernels on the sphere:

(1) Write neural kernels on the sphere as a composition of dual activation functions.

(2) Analyze the boundary behavior of dual activation functions.

(3) Analyze the boundary behavior for sums, products, and compositions of functions, which will be done in Proposition B.2.

(4) Analyze the behavior of even and odd functions to obtain stronger results for the special cases (b) and (d) in Theorem A.2.

(5) Assemble everything.

**Dual activations and neural kernels.** To apply the theorem above, similar to Bietti and Bach (2021), we derive a convenient expression for $\kappa$ using dual activation functions introduced by Daniely et al. (2016).
**Definition A.3 (Dual and rescaled activation functions).** For a function $\varphi \in L_2(\mathcal{N}(0,1))$, we follow Daniely et al. (2016) and define the dual activation

$$ \hat{\varphi} : [-1, 1] \to \mathbb{R}, t \mapsto \mathbb{E}_{(u,v) \sim \mathcal{N}(0,\Sigma_t)}[\varphi(u)\varphi(v)], \quad \Sigma_t := \begin{pmatrix} 1 & t \\ t & 1 \end{pmatrix}. $$

Moreover, for a function $\varphi : \mathbb{R} \to \mathbb{R}$ and $a \in \mathbb{R}$, we define the rescaled function

$$ \varphi_{\cdot a} : \mathbb{R} \to \mathbb{R}, \varphi_{\cdot a}(x) := \varphi(ax). \quad \blacksquare $$

Our following result, derived in Appendix D.1, shows that the NTK and NNGP kernels restricted to the sphere can be expressed using dual rescaled activations:

**Lemma A.4 (Neural kernels on the unit sphere).** Let the activation function $\varphi : \mathbb{R} \to \mathbb{R}$ fulfill Assumption 5. Consider a neural network $f_\theta : \mathbb{R}^{d+1} \to \mathbb{R}$ initialized as in Definition 1. For $l \ge 2$ and $t \in [-1, 1]$ we recursively define

$$ \begin{aligned} \alpha_1 &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 \\ \alpha_l &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 \widehat{\varphi \cdot \sqrt{\alpha_{l-1}}} (1) \\ \kappa_1^{\text{NNGP}}(t) &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 t \\ \kappa_l^{\text{NNGP}}(t) &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 \widehat{\varphi \cdot \sqrt{\alpha_{l-1}}} (\kappa_{l-1}^{\text{NNGP}}(t) / \alpha_{l-1}) \\ \kappa_1^{\text{NTK}}(t) &:= \sigma_b^2 (1 - \sigma_i^2) + \kappa_1^{\text{NNGP}}(t) \\ \kappa_l^{\text{NTK}}(t) &:= \sigma_b^2 (1 - \sigma_i^2) + \kappa_l^{\text{NNGP}}(t) + \sigma_w^2 \widehat{\kappa_{l-1}^{\text{NTK}}(t) (\varphi') \cdot \sqrt{\alpha_{l-1}}} (\kappa_{l-1}^{\text{NNGP}}(t) / \alpha_{l-1}). \end{aligned} $$

Then for all $l \ge 1$, $\mathbf{x}, \mathbf{x}' \in \mathbb{S}^d$ we have $\alpha_l > 0$ and

$$ \begin{aligned} k_l^{\text{NNGP}}(\mathbf{x}, \mathbf{x}') &= \kappa_l^{\text{NNGP}}(\langle \mathbf{x}, \mathbf{x}' \rangle) \\ k_l^{\text{NTK}}(\mathbf{x}, \mathbf{x}') &= \kappa_l^{\text{NTK}}(\langle \mathbf{x}, \mathbf{x}' \rangle). \end{aligned} $$

**Boundary behavior of dual activations.** To study the behavior of dual activations at $\pm 1$, we first need a definition: **Definition A.5 (Reference activations).** For $k \in \mathbb{N}_0$, we define the activation functions $s_k : \mathbb{R} \to \mathbb{R}$ by

$$ s_k(x) := \frac{1}{2k!} \operatorname{sgn}(x)x^k, \quad \operatorname{sgn}(x) := \begin{cases} -1 & , x < 0 \\ 0 & , x = 0 \\ 1 & , x > 0. \end{cases} $$

Let $\varphi$ be an activation function as in Assumption 5. For $k \in \mathbb{N}_0$, we define the coefficients

$$ \Delta_k(\varphi) := \varphi^{(k)}(0+) - \varphi^{(k)}(0-), $$

where $\varphi^{(k)}(0+)$ is the right-sided limit of the $k$-th derivative of $\varphi$ at zero, and $\varphi^{(k)}(0-)$ is the left-sided limit.

The motivation behind Definition A.5 is given in Lemma C.15, which says that we can decompose

$$ \varphi = \sum_{k=0}^{m-1} \Delta_k(\varphi) s_k + \varphi_m, \quad (A.1) $$

where $\varphi_m$ is $m$ times pseudo-differentiable (see Definition C.1). This decomposition is central to our proof.

As in Daniely et al. (2016), we use $h_n$ to denote the $n$-th normalized probabilist's Hermite polynomial, which means that $h_n$ is a polynomial of degree $n$, is even for even $n$ and odd for odd $n$, and $(h_n)_{n \in \mathbb{N}_0}$ forms an orthonormal basis of the Hilbert space $L_2(\mathcal{N}(0,1))$. For a function $f \in L_2(\mathcal{N}(0,1))$ and $n \in \mathbb{N}_0$, we define its $n$-th Hermite coefficient

$$ a_n(f) := \langle f, h_n \rangle_{L_2(\mathcal{N}(0,1))}, $$

such that we have the following Hermite expansion in $L_2(\mathcal{N}(0,1))$:

$$ f = \sum_{n=0}^{\infty} a_n(f)h_n. $$

(Daniely et al., 2016) show that the Hermite series of $\varphi$ is related to the Maclaurin series of $\hat{\varphi}$ via

$$ \hat{\varphi}(t) = \sum_{n=0}^{\infty} a_n(\varphi)^2 t^n. \quad (A.2) $$
This also demonstrates that the “dualization” of activation functions should be thought of as a (function-valued) quadratic form and not dualization. Using our decomposition in Eq. (A.1), we therefore obtain

$$ \hat{\varphi}(t) = \sum_{i=0}^{m} \sum_{j=0}^{m} \sum_{n=0}^{\infty} \Delta_i(\varphi) \Delta_j(\varphi) a_n(s_i) a_n(s_j) t^n \\ + \sum_{i=0}^{m} \sum_{n=0}^{\infty} 2 \Delta_i(\varphi) a_n(s_i) a_n(\varphi_m) t^n + \sum_{n=0}^{\infty} a_n(\varphi_m)^2 t^n. \qquad (A.3) $$

We will treat the first term involving $a_n(s_i)a_n(s_j)$ analytically, while using the smoothness of $\varphi_m$ to obtain fast decay rates for $a_n(\varphi_m)$, which will imply that the behavior at the boundary is dominated by the first term. In total, we obtain the following result:

**Theorem A.6 (Boundary behavior of dual activations).** Let $\varphi$ be an activation function as in Assumption 5 and let $m \in \mathbb{N}_0$ such that $\Delta_k(\varphi) = 0$ for all $k < m$ (see Definition A.5). Then, there exists $b_m > 0$ such that for $\tau \in \{\pm 1\}$,

$$ \hat{\varphi}(\tau(1-t)) = \Delta_m(\varphi)^2(-\tau)^{m+1} b_m t^{m+1/2} + Q_{-1,m+1/2}(t) $$

holds.

To deal with the derivative of the activation in the NTK, we can use the convenient formula $\hat{\varphi}' = \hat{\varphi}'$ from Daniely et al. (2016), of which a generalized formulation (for pseudo-derivatives, see Definition C.1) is proved in Lemma C.2.

**Even and odd functions.** Let $I \subseteq \mathbb{R}$ be symmetric around zero and let $f: I \to \mathbb{R}$ be a function. We say that $f$ is even if $f(x) = f(-x)$ for all $x \in I$, and that $f$ is odd if $f(x) = -f(-x)$ for all $x \in I$. Every function $f: I \to \mathbb{R}$ can be decomposed in its even and odd part

$$ f_{\text{even}}(x) = \frac{f(x) + f(-x)}{2}, \qquad f_{\text{odd}}(x) = \frac{f(x) - f(-x)}{2}. $$

We then have $f = f_{\text{even}} + f_{\text{odd}}$, $f_{\text{even}}$ is even, and $f_{\text{odd}}$ is odd.

For $n \in \mathbb{Z}$, we define

$$ \text{even}(n) := \begin{cases} 1, & n \text{ is even} \\ 0, & n \text{ is odd,} \end{cases} \qquad \text{odd}(n) := \begin{cases} 0, & n \text{ is even} \\ 1, & n \text{ is odd,} \end{cases} $$

The Hermite polynomials $h_n$ are even for even $n$ and odd for odd $n$.

We show in Proposition D.4 that two-layer bias-free neural kernels satisfy an even-odd decomposition, and deeper bias-free neural kernels can be even/odd whenever the activation function is even/odd.

# B Boundary behavior

In the following, we will prove some rules for algebraic manipulations with the function classes $\mathcal{P}_{\alpha,\beta}$, $\mathcal{R}_{\gamma}$, $\mathcal{Q}_{\alpha,\beta}$ from Definition A.1. While we mostly care about $\mathcal{Q}_{\alpha,\beta}$ later, we will first prove rules for $\mathcal{P}_{\alpha,\beta}$ and $\mathcal{R}_{\gamma}$ as a simpler intermediate step, before providing rules for $\mathcal{Q}_{\alpha,\beta}$ in Proposition B.2, which are summarized in Table B.1:

**Lemma B.1 (Rules for $\mathcal{P}_{\alpha,\beta}$ and $\mathcal{R}_{\gamma}$).**

(a) For $\alpha \in \mathbb{R}$, we have $\mathcal{P}_{\alpha,\alpha} \subseteq \mathcal{R}_{\alpha}$.

(b) Let $\gamma_1, \gamma_2 \in \mathbb{R}$. Let $f_1, f_2: (0,2) \to \mathbb{R}$ with $f_i \in \mathcal{R}_{\gamma_i}$. Then, $f_1 \cdot f_2 \in \mathcal{R}_{\gamma_1+\gamma_2}(t)$.

(c) If $f_i \in \mathcal{P}_{\alpha_i, \beta_i}$, $\alpha_i, \beta_i \in \mathbb{R}$, $\alpha_i \le \beta_i$, then

$$ f_1 \cdot f_2 \in \mathcal{P}_{\alpha_1+\alpha_2, \min\{\alpha_1+\beta_2, \beta_1+\alpha_2\}}. $$

(d) Let $A, B > 0$, $g_1: (0,2) \to (0,2)$, $g_2: (0,2) \to \mathbb{R}$. Suppose that $g_2 \in \mathcal{R}_{\gamma_2}, \gamma_2 \in \mathbb{R}$ and that $g_1(t) = at^\alpha + \mathcal{R}_{\gamma_1}(t)$, $\gamma_1 > \alpha > 0$ and $a > 0$. Then,

$$ g_2 \circ g_1 \in \mathcal{R}_{\alpha\gamma_2}. $$

(e) Let $J \subseteq \mathbb{R}$ be an interval, let $g_1: (0,2) \to J$, and let $g_2 \in C^\infty(J)$ with $0 \in J$. Moreover, suppose that $g_1(t) = at^\alpha + bt^\beta + \mathcal{P}_{\alpha,\beta}(t) + \mathcal{R}_{\gamma}(t)$ with $\lim_{t \searrow 0} g_1(t) = 0, 0 \le \alpha \le \beta \le \gamma, \alpha \in \mathbb{N}_0, \beta, \gamma \in \mathbb{R}$ and $a, b \in \mathbb{R}$. Then,

$$ g_2(g_1(t)) = g_2(0) + g'_2(0)\alpha at^\alpha + g'_2(0)\beta bt^\beta + \mathcal{P}_{\alpha,\beta}(t) + \mathcal{R}_{\gamma}(t). $$
(f) Let $g_1: (0, 2) \to (0, \infty)$ such that $g_1(t) = at^\alpha + P_{\alpha,\alpha}(t) + R_\gamma(t)$ for $a, \alpha > 0$ and $\gamma > \alpha$. Then, for $\delta > 0$,
$$g_1(t)^\delta = a^\delta t^{\alpha\delta} + P_{\alpha\delta,\alpha\delta}(t) + R_{\gamma-\alpha+\alpha\delta}(t).$$

Proof.

(a) This is straightforward.

(b) For $n \in \mathbb{N}_0$, we have

$$|(f_1 \cdot f_2)^{(n)}(t)| = \left| \sum_{k=0}^{n} \binom{n}{k} f_1^{(k)}(t) f_2^{(n-k)}(t) \right| \\ \le \sum_{k=0}^{n} O(t^{\gamma_1-k}) O(t^{\gamma_2-(n-k)}) = O(t^{\gamma_1+\gamma_2-n}).$$

(c) Consider integers $\alpha'_i > \alpha_i$ and non-integers $\beta'_i > \beta_i$ such that $t^{\alpha'_i} = P_{\alpha_i, \beta_i}(t)$ and $t^{\beta'_i} = P_{\alpha_i, \beta_i}(t)$. We now investigate all possible products of such terms:

* The term $t^{\alpha'_1 + \alpha'_2}$ has integer power $\alpha'_1 + \alpha'_2 > \alpha_1 + \alpha_2$.

* The term $t^{\alpha'_1 + \beta'_2}$ has non-integer power $\alpha'_1 + \beta'_2 > \alpha_1 + \beta_2$.

* The term $t^{\beta'_1 + \alpha'_2}$ has non-integer power $\beta'_1 + \alpha'_2 > \beta_1 + \alpha_2$.

* The term $t^{\beta'_1 + \beta'_2}$ may have integer or non-integer power. In any case, due to the assumption $\alpha_i \le \beta_i$, we have

$$\beta'_1 + \beta'_2 > \beta_1 + \beta_2 \ge \min\{\alpha_1 + \beta_2, \beta_1 + \alpha_2\} \ge \alpha_1 + \alpha_2.$$

(d) Let $g_2 \in R_{\gamma_2}$. We show

$$|(g_2 \circ g_1)^{(n)}(t)| = O(t^{\gamma_2 \alpha - n})$$

by induction on $n \in \mathbb{N}_0$. Since $0 < \alpha < \gamma_1$, we have $g_1(t) = \Theta(t^\alpha)$ for $t \to 0$ and $\lim_{t \to 0} g_1(t) = 0$. Therefore,

$$g_2(g_1(t)) = O(g_1(t)^{\gamma_2}) = O(t^{\gamma_2 \alpha}).$$

Here, we used $g_1(t) = O(t^\alpha)$ in the case $\gamma_2 > 0$ and $g_1(t) = \Omega(t^\alpha)$ in the case $\gamma_2 < 0$. For the induction step $n \to n + 1$, we use $g'_2 \in R_{\gamma_2-1}$ to obtain

$$|(g_2 \circ g_1)^{(n+1)}(t)| = \left| \frac{d^n}{dt^n} (g'_2 \circ g_1)(t) \cdot g'_1(t) \right| \\ = \left| \sum_{k=0}^{n} \binom{n}{k} (g'_2 \circ g_1)^{(k)}(t) g_1^{(1+(n-k))}(t) \right| \\ \le \sum_{k=0}^{n} O(t^{\gamma_2-1}\alpha-k) O(t^{\alpha-1-(n-k)}) \\ = O(t^{\gamma_2\alpha-(n+1)}).$$

(e) We decompose $g_2 = p_2 + r_2$, where $p_2$ is the degree-$(N-1)$ Taylor polynomial of $g_2$ around 0 for some $N \ge 2$ to be defined later. Then,

$$p_2(g_1(t)) = g_2(0) + g'_2(0)g_1(t) + \sum_{k=2}^{N-1} \frac{g_2^{(k)}(0)}{k!} g_1(t)^k.$$

Since we required $\lim_{t \to 0} g_1(t) = 0$, $g_1$ contains no degree-zero polynomial terms. Therefore, it follows from (a)-(c) that the powers $g_1(t)^k$ for $k \ge 2$ satisfy $g_1(t)^k = P_{\alpha,\beta}(t) + R_\gamma(t)$, which shows that

$$p_2(g_1(t)) = g_2(0) + g'_2(0)at^\alpha + g'_2(0)bt^\beta + P_{\alpha,\beta}(t) + R_\gamma(t).$$

Since $\lim_{t \to 0} g_1(t) = 0$, we know that $g_1 \in R_{\tilde{\beta}}$ for some $\tilde{\beta} > 0$. For the remainder function $r_2$, we could show that $r_2 \in R_N$ but cannot directly apply (d) since it requires a lower bound of the form $g_1(t) = \Omega(t^{\tilde{\beta}})$, which does not necessarily hold if $a = b = 0$. However, here, the lower bound is not needed: Since we know that $r_2 \in C^\infty$, all derivatives of $r_2$ are bounded around zero, which is not the case for all $R_N$ functions. Hence, here is a variant of the proof of (d) for the current setting: In order to prove that $r_2 \circ g_1 \in R_{N\tilde{\beta}}$, we want to prove the following statement by induction on $n$:
<table><caption>Table B.1: Summary of the boundary computation rules in Proposition B.2.</caption><thead><tr><th>Rule</th><th>Assumptions</th></tr></thead><tbody><tr><td>Q<sub>α<sub>2</sub>,β<sub>2</sub></sub>(t) = Q<sub>α<sub>1</sub>,β<sub>1</sub></sub>(t)</td><td>α<sub>1</sub> ≤ α<sub>2</sub>, β<sub>1</sub> ≤ β<sub>2</sub></td></tr><tr><td>Q<sub>α<sub>1</sub>,β<sub>1</sub></sub>(t) + Q<sub>α<sub>2</sub>,β<sub>2</sub></sub>(t) = Q<sub>min{α<sub>1</sub>,α<sub>2</sub>},min{β<sub>1</sub>,β<sub>2</sub></sub></sub>(t)</td><td>—</td></tr><tr><td>Q<sub>α<sub>1</sub>,β<sub>1</sub></sub>(t) · Q<sub>α<sub>2</sub>,β<sub>2</sub></sub>(t) = Q<sub>α<sub>1</sub>+α<sub>2</sub>,min{α<sub>1</sub>+β<sub>2</sub>,α<sub>2</sub>+β<sub>1</sub></sub></sub>(t)</td><td>α<sub>i</sub> ≤ β<sub>i</sub></td></tr><tr><td>g<sub>2</sub>(at<sup>α</sup> + bt<sup>β</sup> + Q<sub>α,β</sub>(t)) = g<sub>2</sub>(0) + g'<sub>2</sub>(0)at<sup>α</sup> + g'<sub>2</sub>(0)bt<sup>β</sup> + Q<sub>α,β</sub>(t)</td><td>g<sub>2</sub> ∈ C<sup>∞</sup> (also at 0),<br>α ∈ N<sub>0</sub>, β ≥ α,<br>a = 0 if α = 0, b = 0 if β = 0</td></tr><tr><td>(at<sup>α</sup> + Q<sub>α,α</sub>(t))<sup>δ</sup> = a<sup>δ</sup>t<sup>αδ</sup> + Q<sub>αδ,αδ</sub>(t)</td><td>a, α, δ > 0, at<sup>α</sup> + Q<sub>α,α</sub>(t) > 0</td></tr><tr><td>Q<sub>δ,δ</sub>(at<sup>α</sup> + Q<sub>α,α</sub>(t)) = Q<sub>αδ,αδ</sub>(t)</td><td>a, α, δ > 0, at<sup>α</sup> + Q<sub>α,α</sub>(t) ∈ (0, 2)</td></tr></tbody></table>

For all $n \in \mathbb{N}_0$, if $h_2 \in C^\infty(J)$ and $M \in \mathbb{N}_0$ such that $h_2(0) = h'_2(0) = \dots = h_2^{(M-1)}(0) = 0$, then
$$|(h_2 \circ g_1)^{(n)}(t)| = O(t^{M\tilde{\beta}-n}).$$

By choosing $M := N$ large enough such that $M\tilde{\beta} \ge \gamma$ and setting $h_2 := r_2$, we can then conclude that $r_2 \circ g_1 \in \mathcal{R}_\gamma$.

**Base case:** Let $n = 0$. By applying Taylor's theorem, we find that

$$ \begin{aligned} |h_2(g_1(t))| &= \left| \sum_{k=0}^{M} \frac{h_2^{(k)}(0)}{k!} g_1(t)^k + o(|g_1(t)|^M) \right| \le \left| \frac{h_2^{(N)}(0)}{N!} g_1(t)^N \right| + o(|g_1(t)|^N) \\ &= O(|g_1(t)|^N) = O(t^{N\tilde{\beta}}). \end{aligned} $$

**Induction step** $n \to n + 1$: We apply the induction hypothesis to $\tilde{h}_2 := h'_2 \in C^\infty(J)$, which satisfies the derivative condition for $\tilde{M} := \max\{0, M-1\}$, and obtain

$$|(h'_2 \circ g_1)^{(k)}(t)| = O(t^{N\tilde{\beta}-k}) \le O(t^{(N-1)\tilde{\beta}-k})$$

for all $0 \le k \le n$. Hence,

$$ \begin{aligned} |(h_2 \circ g_1)^{(n+1)}(t)| &= \left| \frac{d^n}{dt^n} (h'_2 \circ g_1)(t) \cdot g'_1(t) \right| \\ &= \left| \sum_{k=0}^{n} \binom{n}{k} (h'_2 \circ g_1)^{(k)}(t) \cdot g_1^{(1+n-k)}(t) \right| \\ &\le \sum_{k=0}^{n} O(t^{(M-1)\tilde{\beta}-k}) O(t^{\tilde{\beta}-1-n+k}) \\ &= O(t^{M\tilde{\beta}-(n+1)}), \end{aligned} $$

which completes the induction step.

(f) We obtain

$$ \begin{aligned} g_1(t)^{\delta} &= (at^{\alpha} + \mathcal{P}_{\alpha,\alpha}(t) + \mathcal{R}_{\gamma}(t))^{\delta} \\ &= (at^{\alpha} \cdot (1 + t^{-\alpha} \mathcal{P}_{\alpha,\alpha}(t) + t^{-\alpha} \mathcal{R}_{\gamma}(t)))^{\delta} \\ &\stackrel{(b), (c)}{=} a^{\delta} t^{\alpha\delta} (1 + \mathcal{P}_{0,0}(t) + \mathcal{R}_{\gamma-\alpha}(t))^{\delta} \\ &\stackrel{(e)}{=} a^{\delta} t^{\alpha\delta} (1 + \mathcal{P}_{0,0}(t) + \mathcal{R}_{\gamma-\alpha}(t)) \\ &= a^{\delta} t^{\alpha\delta} + \mathcal{P}_{\alpha\delta,\alpha\delta}(t) + \mathcal{R}_{\gamma-\alpha+\alpha\delta}(t). \end{aligned} \quad \square $$

We now obtain the following rules for $Q_{\alpha,\beta}$, summarized in Table B.1:

**Proposition B.2 (Rules for $Q_{\alpha,\beta}$).** Let $J \subseteq \mathbb{R}$ be an interval. Let $f_1, f_2 : [0, 2] \to \mathbb{R}$, $g_1 : [0, 2] \to J$ and $g_2 : J \to \mathbb{R}$ with

$$ \begin{aligned} f_1(t) &= Q_{\alpha_1, \beta_1}(t), \\ f_2(t) &= Q_{\alpha_2, \beta_2}(t) \end{aligned} $$

for some $\alpha_i, \beta_i \in \mathbb{R}$.
(a) **Inclusion:** If $\alpha_1 \le \alpha_2$ and $\beta_1 \le \beta_2$, then $Q_{\alpha_2, \beta_2} \subseteq Q_{\alpha_1, \beta_1}$.

(b) **Sum:** We have

$$f_1(t) + f_2(t) = Q_{\min\{\alpha_1, \alpha_2\}, \min\{\beta_1, \beta_2\}}(t).$$

(c) **Product:** If $\alpha_i \le \beta_i$, we have

$$f_1(t) \cdot f_2(t) = Q_{\alpha_1+\alpha_2, \min\{\alpha_1+\beta_2, \alpha_2+\beta_1\}}(t).$$

(d) **Composition with $C^\infty$:** Suppose that $g_1(t) = at^\alpha + bt^\beta + Q_{\alpha, \beta}(t)$ with $g_1(0) = 0$ for some $\alpha \in \mathbb{N}_0$ and $\beta \ge \alpha$. Additionally, suppose that $g_2 \in C^\infty(J)$ with $0 \in J$. Then,

$$g_2(g_1(t)) = g_2(0) + g'_2(0)at^\alpha + g'_2(0)bt^\beta + Q_{\alpha, \beta}(t).$$

(e) **Composition with power:** Suppose that $J \subseteq (0, \infty)$ with $g_1(t) = at^\alpha + Q_{\alpha, \alpha}(t)$ for $a, \alpha > 0$. Then, for $\delta > 0$,

$$g_1(t)^\delta = a^\delta t^{\alpha\delta} + Q_{\alpha\delta, \alpha\delta}(t).$$

(f) **Composition with $Q_{\delta, \delta}$:** Suppose that $J = (0, 2)$, $g_1(t) = at^\alpha + Q_{\alpha, \alpha}(t)$ with $a, \alpha > 0$, and $g_2(t) = Q_{\delta, \delta}(t)$ with $\delta \ge 0$. Then,

$$g_2(g_1(t)) = Q_{\alpha\delta, \alpha\delta}(t).$$

*Proof.*

(a) Follows from the definition.

(b) For given $\gamma \in \mathbb{R}$, write $f_i = p_i + r_i$ with $p_i \in P_{\alpha_i, \beta_i}$ and $r_i \in \mathcal{R}_\gamma$. Then, obviously $p_1 + p_2 \in P_{\min\{\alpha_1, \alpha_2\}, \min\{\beta_1, \beta_2\}}$ and $r_1 + r_2 \in \mathcal{R}_\gamma$, which shows the claim.

(c) Using a decomposition as in the proof of (b), we write $f_1f_2 = p_1p_2 + (p_1r_2 + r_1p_2 + r_1r_2)$. Here, it follows from Lemma B.1 (c) that $p_1p_2 \in P_{\alpha_1+\alpha_2, \min\{\alpha_1+\beta_2, \alpha_2+\beta_1\}}$. Moreover, using Lemma B.1 (a) we have $p_i \in \mathcal{R}_{\alpha_i}$. Using Lemma B.1 (b), we obtain

$$p_1r_2 + r_1p_2 + r_1r_2 \in \mathcal{R}_{\gamma+\min\{\alpha_1, \alpha_2\}},$$

where $\gamma \in \mathbb{R}$ was arbitrary. This shows the claim.

(d) Follows from Lemma B.1 (e).

(e) Follows from Lemma B.1 (f).

(f) Consider $\gamma \in \mathbb{R}$ to be chosen later. We can then write $g_2(t) = p_2(t) + r_2(t)$ with $p_2 \in P_{\delta, \delta}$ and $r_2 \in \mathcal{R}_\gamma$. Since $p_2$ is a sum of powers, (a) and (e) show that $p_2(g_1(t)) = Q_{\alpha\delta, \alpha\delta}(t)$. For $r_2$, we use Lemma B.1 (a), (d) to conclude that for some $\varepsilon > 0$,

$$r_2(g_1(t)) = r_2(at^\alpha + \mathcal{R}_{\alpha+\varepsilon}(t)) = \mathcal{R}_{\alpha\gamma}(t).$$

Since $\gamma$ was arbitrary, it can be chosen such that $\alpha\gamma$ can be arbitrarily large (since $\alpha > 0$). This completes the proof. □

# C Proofs for dual activations

## C.1 General properties

**Definition C.1 (Pseudo-derivative).** Let $f, g: \mathbb{R} \to \mathbb{R}$. We call $g$ a pseudo-derivative of $f$ if $g$ is Lebesgue integrable on compact intervals and

$$f(x) = f(0) + \int_{0}^{x} g(t) \mathrm{d}t$$

for all $x \in \mathbb{R}$.

<figure><img src="image_3.png" /></figure>

If $f: \mathbb{R} \to \mathbb{R}$ is continuously differentiable, then $f'$ is a pseudo-derivative of $f$. As a non-differentiable example, the Heaviside theta function $\mathbb{1}_{(0,\infty)}(x)$ is a pseudo-derivative of the ReLU function. From basic Lebesgue integration theory, it follows that pseudo-derivatives are unique up to null sets. Note that any *continuous* activation function $\varphi: \mathbb{R} \to \mathbb{R}$ fulfilling Assumption 5 is pseudo-differentiable.

The following lemma generalizes the differentiation part of Lemma 11 by Daniely et al. (2016) to pseudo-differentiable functions.
**Lemma C.2 (Properties of pseudo-derivatives).** Let $g \in L_2(\mathcal{N}(0, 1))$ be a pseudo-derivative of $f : \mathbb{R} \to \mathbb{R}$. Then,

(a) $f \in L_2(\mathcal{N}(0, 1))$,

(b) for $n \ge 1$, $a_n(f) = n^{-1/2}a_{n-1}(g)$,

(c) the Hermite expansion $f = \sum_{n=0}^{\infty} a_n(f)h_n$ converges pointwise,

(d) $\hat{f}$ is differentiable on $[-1, 1]$ with $\hat{f}' = \hat{g}$ (cf. Definition A.3).

*Proof.* Let $\phi$ be the p.d.f. of the normal distribution $\mathcal{N}(0, 1)$. Without loss of generality, let $x \ge 0$. Then,

$$ 
\begin{aligned}
f(x) &= f(0) + \int_{0}^{x} g(t) \, dt = f(0) + \left\langle \frac{\mathbb{1}_{[0,x]}}{\phi}, g \right\rangle_{L_2(\mathcal{N}(0,1))} \\
&= f(0) + \sum_{n=0}^{\infty} a_n(g) \left\langle \frac{\mathbb{1}_{[0,x]}}{\phi}, h_n \right\rangle_{L_2(\mathcal{N}(0,1))} \\
&= f(0) + \sum_{n=0}^{\infty} a_n(g) \int_{0}^{x} h_n(t) \, dt \\
&= f(0) + \sum_{n=0}^{\infty} a_n(g) \int_{0}^{x} (n+1)^{-1/2} h'_{n+1}(t) \, dt \\
&= f(0) + \sum_{n=0}^{\infty} (n+1)^{-1/2} a_n(g) [h_{n+1}(x) - h_{n+1}(0)] \\
&= f(0) + \sum_{n=1}^{\infty} n^{-1/2} a_{n-1}(g) [h_n(x) - h_n(0)] \\
&= \left[ f(0) - \sum_{n=1}^{\infty} n^{-1/2} a_{n-1}(g) h_n(0) \right] h_0(x) + \sum_{n=1}^{\infty} n^{-1/2} a_{n-1}(g) h_n(x) .
\end{aligned} 
\quad (C.1) $$

In the last step, we used $h_0(x) = 1$ for all $x$. Moreover, splitting the series in the last step is valid since the extracted series is absolutely summable:

$$ 
\begin{aligned}
\sum_{n=1}^{\infty} |n^{-1/2} a_{n-1}(g) h_n(0)| &= \sum_{n=1}^{\infty} n^{-1/2} |a_{n-1}(g)| \text{even}(n) \frac{(n-1)!!}{\sqrt{n!}} \\
&\stackrel{\text{Lemma C.8}}{\le} \sum_{n=1}^{\infty} |a_{n-1}(g)| O(n^{-3/4}) \\
&= \left\langle (|a_{n-1}(g)|)_{n \ge 1}, O(n^{-3/4}) \right\rangle_{\ell_2(\mathbb{N}_{\ge 1})} \\
&\stackrel{\text{Cauchy-Schwarz}}{<} \infty.
\end{aligned} 
$$

Here, we have used that $(a_n(g))_{n \ge 1} \in \ell_2(\mathbb{N}_{\ge 1})$ since $g \in L_2(\mathcal{N}(0, 1))$. An analogous argument shows that (C.1) also equals $f(x)$ for $x < 0$. Since $|n^{-1/2} a_{n-1}(g)| \le |a_{n-1}(g)|$, we know that the series representation (C.1) not only converges pointwise but also in $L_2(\mathcal{N}(0, 1))$, and these limits must be identical. This shows (a), (b) and (c). In order to show (d), we compute

$$ \hat{f}(t) = \sum_{n=0}^{\infty} a_n(f)^2 t^n \stackrel{(b)}{=} a_0(f)^2 + \sum_{n=1}^{\infty} n^{-1} a_{n-1}(g)^2 t^n $$

for $t \in [-1, 1]$, which implies

$$ \hat{f}'(t) = \sum_{n=1}^{\infty} n^{-1} a_{n-1}(g)^2 n t^{n-1} = \sum_{n=0}^{\infty} a_n(g)^2 t^n = \hat{g}(t). $$

This also holds at the boundary $t \in \{-1, 1\}$: Thanks to Abel's theorem, $\hat{f}$ and $\hat{g}$ are continuous on $[-1, 1]$. Without loss of generality, let $t = 1$. The mean value theorem of integration yields $\hat{f}(1) - \hat{f}(1-h) = \int_{1-h}^{1} \hat{f}'(t) \, dt = h f'(\xi_h)$ for some $\xi_h \in (1-h, 1)$, thus

$$ \hat{f}'(1) = \lim_{h \searrow 0} \hat{f}'(\xi_h) = \lim_{u \searrow 1} \hat{f}'(u) = \lim_{u \searrow 1} \hat{g}(u) = \hat{g}(1). \quad \square $$
While the dualization $\varphi \mapsto \hat{\varphi}$ from Definition A.3 is a quadratic and not a linear mapping, it still interchanges nicely with the even-odd decomposition (since even and odd functions are orthogonal w.r.t. the corresponding “inner product”):

**Lemma C.3 (General properties of dual activations).** Let $\varphi \in \mathcal{L}_2(\mathcal{N}(0, 1))$. Then,

(a) $\varphi_{\text{even}}, \varphi_{\text{odd}} \in \mathcal{L}_2(\mathcal{N}(0, 1))$ and

$$ (\hat{\varphi})_{\text{even}} = \widehat{\varphi}_{\text{even}}, \quad (\hat{\varphi})_{\text{odd}} = \widehat{\varphi}_{\text{odd}}. $$

(b) $\widehat{\varphi}, \widehat{\varphi}_{\text{even}}$ and $\widehat{\varphi}_{\text{odd}}$ are nonnegative and increasing on $[0, 1]$. Moreover, if $\varphi$ is not almost surely constant, then $\widehat{\varphi}$ is strictly increasing on $[0, 1]$.

(c) We have

$$ |\widehat{\varphi}(t)| \le \widehat{\varphi}(|t|) \le \widehat{\varphi}(1), \qquad (C.2) $$

for all $t \in [-1, 1]$. If $\varphi$ is not almost surely constant, then $|\widehat{\varphi}(t)| < \widehat{\varphi}(1)$ for $t \in (-1, 1)$. Moreover, $|\widehat{\varphi}(-1)| = \widehat{\varphi}(1)$ if and only if $\varphi$ is almost surely even or almost surely odd.

(d) We have $\widehat{\varphi}|_{(-1,1)} \in C^{\infty}((-1, 1))$.

*Proof.*

(a) It is easy to see that $\varphi_{\text{even}}, \varphi_{\text{odd}} \in \mathcal{L}_2(\mathcal{N}(0, 1))$. As mentioned in Appendix A, the Hermite polynomials are even for even $n$ and odd for odd $n$. Hence,

$$ \begin{aligned} \varphi_{\text{even}} &= \sum_{n=0}^{\infty} \text{even}(n) a_n(\varphi) h_n \\ \varphi_{\text{odd}} &= \sum_{n=0}^{\infty} \text{odd}(n) a_n(\varphi) h_n. \end{aligned} $$

We also know that the dual activation satisfies $\hat{\varphi}(t) = \sum_{n=0}^{\infty} a_n(\varphi)^2 t^n$. Since the monomials $t^n$ are also even for even $n$ and odd for odd $n$, we obtain

$$ \begin{aligned} (\hat{\varphi})_{\text{even}}(t) &= \sum_{n=0}^{\infty} \text{even}(n) a_n(\varphi)^2 t^n = \widehat{\varphi}_{\text{even}}(t) \\ (\hat{\varphi})_{\text{odd}}(t) &= \sum_{n=0}^{\infty} \text{odd}(n) a_n(\varphi)^2 t^n = \widehat{\varphi}_{\text{odd}}(t). \end{aligned} \qquad (C.3) $$

Alternatively, this statement can also be proven directly using the definition of the dual activation.

(b) It follows from the Hermite expansions in Item (a) that $\widehat{\varphi}_{\text{even}}$ and $\widehat{\varphi}_{\text{odd}}$ are nonnegative and increasing on $[0, 1]$, and therefore this also holds for $\widehat{\varphi}$. If $\varphi$ is not almost surely constant, we have $a_n(\varphi) \neq 0$ for some $n \ge 1$. Therefore, the Hermite expansion $\widehat{\varphi}(t) = \sum_{n=0}^{\infty} a_n(\varphi)^2 t^n$ implies that $\widehat{\varphi}$ is strictly increasing on $[0, 1]$.

(c) For $t \in [0, 1]$, the statement follows from (b). For $t \in [-1, 0)$, (b) implies

$$ \begin{aligned} |\widehat{\varphi}(t)| &= |\widehat{\varphi}_{\text{even}}(-t) - \widehat{\varphi}_{\text{odd}}(-t)| \\ &\le |\widehat{\varphi}_{\text{even}}(-t)| + |\widehat{\varphi}_{\text{odd}}(-t)| = \widehat{\varphi}_{\text{even}}(-t) + \widehat{\varphi}_{\text{odd}}(-t) = \widehat{\varphi}(-t) = \widehat{\varphi}(|t|) \le \widehat{\varphi}(1). \end{aligned} $$

If $\varphi$ is not almost surely constant, then $\widehat{\varphi}$ is strictly increasing on $[0, 1]$ by (b) and the previous inequality implies $|\widehat{\varphi}(t)| < \widehat{\varphi}(1)$ for $t \in (-1, 1)$. Moreover, the inequality $|\widehat{\varphi}_{\text{even}}(-t) - \widehat{\varphi}_{\text{odd}}(-t)| \le |\widehat{\varphi}_{\text{even}}(-t)| + |\widehat{\varphi}_{\text{odd}}(-t)|$ is sharp iff $\widehat{\varphi}_{\text{even}}(-1) = 0$ or $\widehat{\varphi}_{\text{odd}}(-1) = 0$, which is the case iff $\varphi$ is almost surely even or almost surely odd.

(d) By Eq. (A.2) the dual activation $\widehat{\varphi}$ is a convergent power series on $[-1, 1]$. $\square$

<figure><img src="image_4.png" /></figure>

## C.2 Dominating terms

**Definition C.4 (Double factorial).** Following Daniely et al. (2016), we define the double factorial for $n \in \mathbb{Z}$ as

$$ n!! := \begin{cases} 1, & n \le 0 \\ n \cdot (n-2) \cdots 4 \cdot 2, & n > 0 \text{ even} \\ n \cdot (n-2) \cdots 3 \cdot 1, & n > 0 \text{ odd.} \end{cases} $$

<figure><img src="image_5.png" /></figure>
**Proposition C.5.** For all $k \in \mathbb{N}_0$, the reference activation $s_k$ from Definition A.5 satisfies $s_k \in L_2(\mathcal{N}(0, 1))$ and $s_k$ is a pseudo-derivative of $s_{k+1}$. Moreover, for all $n \in \mathbb{N}_0$, the Hermite coefficients satisfy

$$a_n(s_k) = \text{odd}(n-k) \frac{(-1)^{\frac{\max(1,n-k)-1}{2}} (n-k-2)!!}{(k-n)!! \sqrt{2\pi n!}}$$

*Proof.* The first two statements are straightforward to show, hence we only show the formula for the Hermite coefficients. We first note that for $X \sim \mathcal{N}(0, 1)$, we have for odd $k$ the central absolute moments following (see e.g. Winkelbauer, 2012):

$$\begin{aligned}
\mathbb{E}[|X|^k] &= 2^{k/2} \Gamma\left(\frac{k+1}{2}\right) \pi^{-1/2} = 2^{k/2} ((k-1)/2)! \pi^{-1/2} = 2^{1/2} (k-1)!! \pi^{-1/2} \\
&= \sqrt{\frac{2}{\pi}} (k-1)!!.
\end{aligned}$$

This allows us to show the statement for $n = 0$:

$$\begin{aligned}
a_0(s_k) &= \langle h_0, s_k \rangle_{\mathcal{L}_2(\mathcal{N}(0,1))} = \int_{\mathbb{R}} s_k(x) \frac{1}{\sqrt{2\pi}} e^{-x^2/2} dx = \text{odd}(k) \frac{1}{2k!} \int_{\mathbb{R}} |x^k| \frac{1}{\sqrt{2\pi}} e^{-x^2/2} dx \\
&= \text{odd}(k) \frac{(k-1)!!}{k!\sqrt{2\pi}} = \text{odd}(k) \frac{1}{k!!\sqrt{2\pi}} = \text{odd}(0-k) \frac{(-1)^{\frac{\max(1,0-k)-1}{2}} (0-k-2)!!}{(k-0)!!\sqrt{2\pi 0!}},
\end{aligned}$$

Finally, we show the formula for all $n, k$ via induction on $k$. (We do not use induction on $n$.) For the base case $k = 0$, we note that $h_0 \equiv 1$ and $s_0 = 2^{-1/2} \sqrt{2} \mathbf{1}_{(0,\infty)} - \frac{1}{2} h_0$ almost everywhere. Daniely et al. (2016) show in Section 8 that

$$a_n(\sqrt{2}\mathbf{1}_{(0,\infty)}) = \begin{cases} 2^{-1/2} & , n = 0 \\ \frac{(-1)^{\frac{n-1}{2}} (n-2)!!}{\sqrt{\pi n!}} & , n \text{ odd} \\ 0 & , 2 \le n \text{ even.} \end{cases}$$

Therefore, we obtain for $n \ge 1$:

$$\begin{aligned}
a_n(s_0) &= 2^{-1/2} a_n(\sqrt{2}\mathbf{1}_{(0,\infty)}) = \text{odd}(n) \frac{(-1)^{\frac{n-1}{2}} (n-2)!!}{\sqrt{2\pi n!}} \\
&= \text{odd}(n-0) \frac{(-1)^{\frac{\max(1,n-0)-1}{2}} (n-0-2)!!}{(0-n)!! \sqrt{2\pi n!}},
\end{aligned}$$

which completes the base case $k = 0$ since the case $n = 0$ has already been treated above.

For the induction $k \to k + 1$, we note that $s_k$ is a pseudo-derivative of $s_{k+1}$ and use Lemma C.2 to obtain for all $n \ge 0$:

$$\begin{aligned}
a_{n+1}(s_{k+1}) &= (n+1)^{-1/2} a_n(s_k) = \text{odd}(n-k) \frac{(-1)^{\frac{\max(1,n-k)-1}{2}} (n-k-2)!!}{(k-n)!! \sqrt{2\pi(n+1)!}} \\
&= \text{odd}((n+1)-(k+1)) \frac{(-1)^{\frac{\max(1,(n+1)-(k+1))-1}{2}} ((n+1)-(k+1)-2)!!}{((k+1)-(n+1))!! \sqrt{2\pi(n+1)!}},
\end{aligned}$$

which completes the induction.

□

**Lemma C.6.** We have $\hat{s}_0(t) = \frac{1}{4} - \frac{1}{2\pi} \arccos(t)$.

*Proof.* By Section 8 in Daniely et al. (2016), the function $f := \sqrt{2}\mathbf{1}_{(0,\infty)}$ has the dual activation $\hat{f}(t) = 1 - \frac{1}{\pi} \arccos(t)$. Hence, $g := \mathbf{1}_{(0,\infty)}$ has the dual activation

$$\hat{g}(t) = \frac{1}{2} - \frac{1}{2\pi} \arccos(t).$$

Now, since $s_0 = g - 1/2$ almost everywhere, we have $a_n(s_0) = a_n(g)$ for all $n \ne 0$, and hence

$$\hat{s}_0(t) = \sum_{n=0}^{\infty} a_n(s_0)^2 t^n = C + \sum_{n=0}^{\infty} a_n(g)^2 t^n = C + \frac{1}{2} - \frac{1}{2\pi} \arccos(t)$$

for some constant $C \in \mathbb{R}$. Since $s_0$ is odd, $\hat{s}_0$ must be odd by Lemma C.3, which yields $C = -1/4$. □

□
**Lemma C.7.** For all $k \in \mathbb{N}_0$, there exists $b_k > 0$ such that for $\tau \in \{\pm 1\}$,

$$ \widehat{s}_k(\tau(1-t)) = (-\tau)^{k+1} b_k t^{k+1/2} + \mathcal{Q}_{-1,k+1/2}(t). $$

*Proof. Step 1: Analysis of $s_0$. First, consider the case $\tau = 1$. We have*

$$ \frac{d}{dt} \arccos(1-t) = \frac{1}{\sqrt{1-(1-t)^2}} = t^{-1/2}(2-t)^{-1/2}. $$

It is easily seen using induction that

$$ \frac{d^n}{dt^n} (2-t)^{-1/2} = \frac{(2n-1)!!}{2^n} (2-t)^{-\frac{2n+1}{2}}. $$

Since $t \mapsto (2-t)^{-1/2}$ is analytic in a neighborhood of $t = 0$, it is equal to its Taylor expansion:

$$ (2-t)^{-1/2} = \sum_{n=0}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{1}{2}} n!} t^n. $$

Since all coefficients are positive, we can apply the monotone convergence theorem to obtain

$$ \begin{aligned} \arccos(1-t) &= \arccos(1-0) + \int_0^t \frac{d}{du} \arccos(1-u) \, du \\ &= \int_0^t \sum_{n=0}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{1}{2}} n!} u^{n-1/2} \, du \\ &= \sum_{n=0}^{\infty} \int_0^t \frac{(2n-1)!!}{2^{2n+\frac{1}{2}} n!} u^{n-1/2} \, du \\ &= \sum_{n=0}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{1}{2}} n!} \left[ \frac{1}{n+1/2} u^{n+1/2} \right]_0^t \\ &= \sum_{n=0}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{1}{2}} n! (n+1/2)} t^{n+1/2}. \end{aligned} $$

For arbitrary $1 \le N \in \mathbb{N}_0$, this yields using Lemma C.6, the identity

$$ \begin{aligned} \widehat{s}_0(1-t) &= \frac{1}{4} - \sum_{n=0}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{3}{2}} n! (n+1/2)\pi} t^{n+1/2} \\ &= \frac{1}{4} - \frac{1}{\pi\sqrt{2}} t^{1/2} + \underbrace{\sum_{n=1}^{N-1} \frac{(2n-1)!!}{2^{2n+\frac{3}{2}} n! (n+1/2)\pi} t^{n+1/2}}_{=\mathcal{P}_{0,1/2}(t)} \\ &\quad + \underbrace{t^{N+1/2} \sum_{n=N}^{\infty} \frac{(2n-1)!!}{2^{2n+\frac{3}{2}} n! (n+1/2)\pi} t^{n-N}}_{=\mathcal{R}_0(t)} \end{aligned} $$

**Step 2: Induction on $k$.** We now show the lemma via induction on $k \in \mathbb{N}_0$. For arbitrary $\gamma \in \mathbb{R}$, by choosing $N$ with $N+1/2 \ge \gamma$, we obtain using Lemma B.1:

$$ \widehat{s}_0(1-t) = (-1)^1 b_0 t^{0+1/2} + \mathcal{P}_{-1,1/2}(t) + \mathcal{R}_{\gamma}(t). $$

For the induction step $k \to k+1$, we note that since $s_k$ is a pseudo-derivative of $s_{k+1}$ (see Definition C.1), we have $\widehat{s}_{k+1}' = \widehat{s}_{k+1}' = \widehat{s}_k$ by Lemma C.2, which yields

$$ \begin{aligned} \widehat{s}_{k+1}(1-t) &= \widehat{s}_{k+1}(1) - \int_0^t \widehat{s}_{k+1}'(1-u) \, du \\ &= \widehat{s}_{k+1}(1) - \int_0^t \widehat{s}_k(1-u) \, du \end{aligned} $$
$$ 
\begin{aligned}
&= \widehat{s_{k+1}}(1) - \int_{0}^{t} (c_k + (-1)^{k+1} b_k u^{k+1/2} + \mathcal{P}_{0,k+1/2}(u) + \mathcal{R}_{\gamma}(u)) du \\
&= c_{k+1} + (-1)^{k+2} b_{k+1} t^{(k+1)+1/2} + \mathcal{P}_{0,(k+1)+1/2}(t) + \mathcal{R}_{\gamma+1}(t)
\end{aligned}
$$

for suitable constants $c_{k+1}$, $b_{k+1} > 0$. This shows the claim for $\tau = 1$. For $\tau = -1$, we note that $s_k$ is odd for even $k$ and even for odd $k$. By Lemma C.3, the same holds for $\widehat{s}_k$. This shows

$$ 
\begin{aligned}
\widehat{s}_k(-1 + t) &= (-1)^{k+1} \widehat{s}_k(1 - t) \\
&= b_k t^{k+1/2} + \mathcal{P}_{-1,k+1/2}(t) + \mathcal{R}_{\gamma}(t).
\end{aligned}
$$

□

We will also need the asymptotic decay of $a_n(s_k)$, which will be studied in the following two lemmas.

**Lemma C.8.** For $p \in \mathbb{N}_0$ and odd $m \in \mathbb{N}_0$, we have

$$ \frac{m!!}{\sqrt{(m+p)!}} = \Theta_{\forall m}(m^{1/4-p/2}). $$

*Proof.* For even $n = 2k \ge 2$, we have $n!! = n(n-2)\cdots 2 = (2 \cdot k)(2 \cdot (k-1))\cdots (2 \cdot 1) = 2^k k! = 2^{n/2}(n/2)!$. Using Stirling's formula, we obtain

$$ \frac{(n-1)!!}{\sqrt{n!}} = \frac{n!}{n!!\sqrt{n!}} = \frac{\sqrt{n!}}{2^{n/2}(n/2)!} \sim \frac{(2\pi n)^{1/4} n^{n/2} e^{-n/2}}{2^{n/2}(\pi n)^{1/2}(n/2)^{n/2} e^{-n/2}} = \Theta_{\forall n}(n^{-1/4}). $$

By setting $n := m + 1$, we obtain

$$ 
\begin{aligned}
\frac{m!!}{\sqrt{(m+p)!}} &= \Theta_{\forall m}(m^{1/2-p/2}) \frac{m!!}{\sqrt{(m+1)!}} = \Theta_{\forall m}(m^{1/2-p/2}) \Theta_{\forall m}(m^{-1/4}) \\
&= \Theta_{\forall m}(m^{1/4-p/2}).
\end{aligned}
$$

□

**Lemma C.9.** We have $|a_n(s_k)| = \Theta_{\forall n}(\text{odd}(n-k)(n+1)^{-3/4-k/2})$.

*Proof.* Since $(k-n)!! = 1$ for $n \ge k$ by our definition of the double factorial in Definition C.4, we obtain

$$ 
\begin{aligned}
|a_n(s_k)| &\stackrel{\text{Proposition C.5}}{=} \Theta_{\forall n} \left( \text{odd}(n-k) \frac{(n-k-2)!!}{\sqrt{2\pi n!}} \right) \\
&\stackrel{\text{Lemma C.8}}{=} \Theta_{\forall n} (\text{odd}(n-k)(n+1)^{-3/4-k/2}).
\end{aligned}
$$

□

### C.3 Mix terms

Now, we want to investigate some of the mix-terms arising in the decomposition in Eq. (A.3). For convenience, we will give them a new name:

**Definition C.10.** For $i, j \in \mathbb{N}_0$, define $f_{i,j} : [-1, 1] \to \mathbb{R}$ by

$$ f_{i,j}(t) := \sum_{n=0}^{\infty} a_n(s_i) a_n(s_j) t^n. $$

Note that $f_{i,j} \equiv 0$ for odd $i-j$ and $f_{i,j} = f_{j,i}$. Moreover, $f_{i,i} = \widehat{s}_i$.

◀

**Lemma C.11** (Recursive characterization of mix-terms). Let $k, l \in \mathbb{N}_0$. Then, there exists a polynomial $p_{k,k+2l}$ such that for $t \in [-1, 1]$,

$$ 
\begin{aligned}
f_{k,k+2l}(t) &= (k+2)f_{k+2,(k+2)+2(l-1)}(t) - t f_{k+1,(k+1)+2(l-1)}(t) + p_{k,k+2l}(t) \\
f_{k,k+2l+1}(t) &= 0.
\end{aligned}
$$

*Proof.* Since $a_n(s_k) = 0$ for even $n-k$ and $a_n(s_{k+2l+1}) = 0$ for even $n - (k + 2l + 1)$, we have

$$ f_{k,k+2l+1}(t) = \sum_{n=0}^{\infty} a_n(s_k) a_n(s_{k+2l+1}) t^n = 0. $$

For the other formula, we use Proposition C.5 and obtain

$$ a_n(s_k) a_n(s_{k+2l}) = \text{odd}(n-k) \text{odd}(n-(k+2l)) (-1)^{\frac{\max(1,n-k)-1}{2} + \frac{\max(1,n-k-2l)-1}{2}} $$
$$ \frac{(n-k-2)!!(n-k-2l-2)!!}{(k-n)!!(k+2l-n)!!2\pi n!} $$

Here, odd($n-k$) odd($n-(k+2l)$) = odd($n-k$). Moreover, for odd $n-k$ and $n \ge k+2l$, we have

$$ (-1)^{\frac{\max(1, n-k)-1}{2} + \frac{\max(1, n-k-2l)-1}{2}} = (-1)^{n-k-l-1} = (-1)^{-l} = (-1)^l. $$

In the following, we write

$$ f \approx_{\text{pol}} g $$

if $f-g$ is a polynomial.

The terms $(k-n)!!$ and $(k+2l-n)!!$ are equal to one for $n \ge k+2l$. Therefore, we obtain for all but finitely many $n$

$$ a_n(s_k)a_n(s_{k+2l}) = \text{odd}(n-k)(-1)^l \frac{(n-k-2)!!(n-k-2l-2)!!}{2\pi n!}. $$

By writing $(n-k-2)!! = -(k+2)(n-k-4)!! + n(n-k-4)!!$, we obtain:

$$ \begin{aligned} & \sum_{n=0}^{\infty} a_n(s_k) a_n(s_{k+2l}) t^n \\ & \approx_{\text{pol}} -(k+2) \sum_{n=0}^{\infty} \text{odd}(n-k)(-1)^l \frac{(n-k-4)!!(n-k-2l-2)!!}{2\pi n!} t^n \\ & \qquad + \sum_{n=0}^{\infty} n \text{odd}(n-k)(-1)^l \frac{(n-k-4)!!(n-k-2l-2)!!}{2\pi n!} t^n \\ & \approx_{\text{pol}} (k+2) \sum_{n=0}^{\infty} \text{odd}(n-(k+2))(-1)^{l-1} \frac{(n-(k+2)-2)!!(n-(k+2)-2(l-1)-2)!!}{2\pi n!} t^{n-1} \\ & \qquad - t \cdot \sum_{n=1}^{\infty} \text{odd}((n-1)-(k-1))(-1)^{l-1} \frac{((n-1)-k-3)!!((n-1)-k-2l-1)!!}{2\pi(n-1)!} t^{n-1} \\ & = (k+2) \sum_{n=0}^{\infty} \text{odd}(n-(k+2))(-1)^{l-1} \frac{(n-(k+2)-2)!!(n-(k+2)-2(l-1)-2)!!}{2\pi n!} t^n \\ & \qquad - t \cdot \sum_{n=0}^{\infty} \text{odd}(n-(k+1))(-1)^{l-1} \frac{(n-(k+1)-2)!!(n-(k+1)-2(l-1)-2)!!}{2\pi n!} t^n \\ & \approx_{\text{pol}} (k+2) \sum_{n=0}^{\infty} a_n(s_{k+2}) a_n(s_{(k+2)+2(l-1)}) t^n - t \cdot \sum_{n=0}^{\infty} a_n(s_{k+1}) a_n(s_{(k+1)+2(l-1)}) t^n, \end{aligned} $$

which completes the proof.

<figure><img src="image_6.png" /></figure>

**Proposition C.12.** Let $i,j \in \mathbb{N}_0$. Then, for any $\tau \in \{\pm 1\}$ and $t \in (0, 2)$,

$$ f_{i,j}(\tau(1-t)) = Q_{-1,(i+j)/2}(t). $$

*Proof.* Recall Lemma C.11. Since $f_{i,j} = f_{j,i}$, we can assume $i \le j$ without loss of generality. Moreover, for odd $j-i$, the statement is trivial since $f_{i,j} \equiv 0$. For the remaining cases where $j-i \ge 0$ is even, it suffices to prove the following statement by induction on $l$:

For all $l \in \mathbb{N}_0$: For all $k \in \mathbb{N}_0$:

$$ f_{k,k+2l} = Q_{-1,k+l}(t). $$

For $l=0$, this follows from Lemma C.7 since $f_{k,k} = \hat{s}_k$. For the induction step, we use Lemma C.11 to obtain for all $k \in \mathbb{N}_0$:

$$ \begin{aligned} & f_{k,k+2l}(\tau(1-t)) \\ &= Q_{-1,\infty}(t) + (k+2)f_{k+2,k+2+2(l-1)}(\tau(1-t)) - \tau(1-t)f_{k+1,k+1+2(l-1)}(\tau(1-t)) \\ &= Q_{-1,\infty}(t) + Q_{-1,k+l+1}(t) - \tau(1-t)(Q_{-1,k+l}(t)) \\ &= Q_{-1,k+l}(t). \end{aligned} $$

Here, we used that for a polynomial $p$, the map $t \mapsto p(\tau(1-t))$ is also a polynomial and thus in $Q_{-1,\infty}(t)$. $\square$

<figure><img src="image_7.png" /></figure>
## C.4 Results for smooth terms

In the following, we analyze the decay of Hermite coefficients for smooth functions, which can then be used to establish the smoothness of certain components of dual activations. For smooth $f$, we could show the smoothness of $\hat{f}$ by using $\hat{f}' = \hat{f}'$, but this approach does not work directly for mix-terms, and hence the intermediate step via coefficient decay is helpful.

**Lemma C.13.** If $f: \mathbb{R} \to \mathbb{R}$ has a $m$-fold pseudo-derivative $f^{(m)}$ with $f^{(m)} \in \mathcal{L}_2(\mathcal{N}(0, 1))$, then $|a_n(f)| < o_m((n+1)^{-m/2})$.

*Proof.* By Lemma C.2, we obtain for $n \ge m$:

$$a_n(f) = [n(n-1)\cdots(n-m+1)]^{-1/2} a_{n-m}(f^{(m)}).$$

Since $a_{n-m}(f^{(m)}) < o_{\forall n}(1)$, we obtain $|a_n(f)| < o_{\forall n}((n+1)^{-m/2})$.

<figure><img src="image_8.png" /></figure>

**Lemma C.14 (Differentiability of power series).** Let $f: [-1, 1] \to \mathbb{R}$, $x \mapsto \sum_{n=0}^{\infty} b_n x^n$ with $|b_n| = O_{\forall n}((n+1)^{-(k+1+\epsilon)})$ for some $k \in \mathbb{N}_0$ and $\epsilon > 0$. Then, $f \in C^k([-1, 1])$.

*Proof.* We prove by induction on $k$ that $f \in C^k([-1, 1])$ and

$$f^{(k)}(t) = \sum_{n=k}^{\infty} n \cdots (n-k+1) b_n t^{n-k}$$

for $t \in [-1, 1]$. For $k=0$, we have $\sum_{n=0}^{\infty} |b_n| < \infty$ and the result follows from Abel's theorem on power series. Now, let the statement hold for $k-1 \ge 0$. We know from the case $k=0$ that

$$g: [-1, 1] \to \mathbb{R}, t \mapsto \sum_{n=1}^{\infty} nb_n t^{n-1}$$

as well as $f$ are continuous. By elementary analysis, $f' = g$ on $(-1, 1)$. Moreover,

$$f(1) - f(1-h) = \lim_{h' \searrow 0} f(1-h') - f(1-h) = \lim_{h' \searrow 0} \int_{1-h}^{1-h'} g(x) dx = \int_{1-h}^{1} g(x) dx = hg(\xi_h)$$

for a suitable $\xi_h \in [1-h, 1]$ by the mean value theorem of integration. Since $g$ is continuous, it follows that $f$ is differentiable in $1$ with $f'(1) = g(1)$. An analogous calculation can be applied for $t = -1$. By applying the induction hypothesis to $g = f'$, the proof is completed. $\square$

<figure><img src="image_9.png" /></figure>

## C.5 General activation functions

Now, we want to obtain the asymptotic boundary behavior in the sense of Appendix B for general activation functions $\varphi$ as in Assumption 5. To this end, we use a decomposition into reference functions and smooth remainders:

**Lemma C.15.** Let $\varphi$ be an activation function as in Assumption 5 and let $m \in \mathbb{N}_0$. Then, using $\Delta_k(\varphi)$ as defined in Definition A.5,

$$\varphi_m := \varphi - \sum_{k=0}^{m-1} \Delta_k(\varphi) s_k$$

is $m$ times pseudo-differentiable and the $m$-fold pseudo-derivative $\varphi_m^{(m)}$ is in $L_2(\mathcal{N}(0, 1))$.

*Proof.* Note that $\Delta_m$ is a linear operator with $\Delta_m(s_k) = \delta_{mk}$. Hence, for $l \le m-1$, we have

$$\Delta_l(\varphi_m) = \Delta_l(\varphi) - \sum_{k=0}^{m-1} \Delta_k(\varphi) \Delta_l(s_k) = \Delta_l(\varphi) - \Delta_l(\varphi) = 0.$$

We now prove by induction on $l$ with $0 \le l \le m$ that $g_l: \mathbb{R} \to \mathbb{R}$ defined by

$$g_0 := \varphi_m, g_l(x) := \begin{cases} \varphi_m^{(l)}(x) & , x \ne 0 \\ \varphi_m^{(l)}(0-) & , x = 0 \end{cases} \quad (l \ge 1)$$
is a $l$-fold pseudo-derivative of $\varphi_m$. For $l = 0$, we have $g_0 = \varphi_m$ by definition, hence $g_0$ is a 0-fold pseudo-derivative of $\varphi_m$. For the induction step $l \to l+1$, we observe that $l \le m-1$, hence $\Delta_l(\varphi_m) = 0$, which implies $\varphi_m^{(l)}(0-) = \varphi_m^{(l)}(0+)$ and therefore, $g_l$ is continuous. Moreover, $g_l$ is continuously differentiable on $(0, x)$ and $(-x, 0)$, and $g'_l$ is bounded on these intervals. Thus,

$$g_l(x) = g_l(0) + \int_0^x g'_l(t) dt = g_l(0) + \int_0^x g_{l+1}(t) dt$$
$$g_l(-x) = g_l(0) + \int_0^{-x} g'_l(t) dt = g_l(0) + \int_0^{-x} g_{l+1}(t) dt,$$

which shows that $g_{l+1}$ is a pseudo-derivative of $g_l$.

It remains to show that $g_m \in L_2(\mathcal{N}(0, 1))$. For $x \ne 0$, we have $s_k^{(m)}(x) = 0$ for $k < m$ and therefore $g_m(x) = \varphi^{(m)}(x)$. By Assumption 5, $\varphi|_{(0,\infty)} \in S^\infty((0,\infty))$ and $\varphi|_{(-\infty,0)} \in S^\infty((-\infty,0))$, which implies the desired integrability for $\varphi^{(m)}$, hence $g_m \in L_2(\mathcal{N}(0, 1))$. $\square$

**Theorem A.6 (Boundary behavior of dual activations).** Let $\varphi$ be an activation function as in Assumption 5 and let $m \in \mathbb{N}_0$ such that $\Delta_k(\varphi) = 0$ for all $k < m$ (see Definition A.5). Then, there exists $b_m > 0$ such that for $\tau \in \{\pm 1\}$,

$$\hat{\varphi}(\tau(1-t)) = \Delta_m(\varphi)^2(-\tau)^{m+1}b_m t^{m+1/2} + Q_{-1,m+1/2}(t)$$

holds.

**Proof. Step 0: Proof strategy** Fix $\tau \in \{-1, 1\}$. For an integer $M_1 > m$ to be defined later, we decompose $\varphi$ using Lemma C.15 as

$$\varphi = g_1 + g_2, \quad g_1 := \sum_{k=m}^{M_1-1} \Delta_k(\varphi)s_k, \quad g_2 := \varphi_{M_1}.$$

Here, Lemma C.15 tells us that $g_2$ is $M_1$ times pseudo-differentiable and $g_2^{(M_1)} \in L_2(\mathcal{N}(0, 1))$. The basic strategy is as follows: To verify $f(t) = Q_{-1,m+1/2}(t)$ for a suitable function, we need to show that for all $\gamma \in \mathbb{R}$, there exists a decomposition $f = q + r$ with $q \in \mathcal{P}_{-1,m+1/2}$ and $r \in \mathcal{R}_\gamma$. We will select $M_1$ depending on $\gamma$ to construct such a decomposition. To show $r \in \mathcal{R}_\gamma$, we need to show $|r^{(N)}(t)| = O(t^{\gamma-N})$ for all $N \in \mathbb{N}_0$. To this end, we will decompose $\varphi$ again with an order $M_2 > M_1$ that depends on $N$ (but does not change $r$).

**Step 1: Defining the decomposition of $\hat{\varphi}$.** Using Definition C.10, we obtain

$$\begin{aligned}
\hat{g}_1(t) &= \sum_{n=0}^{\infty} \left( \sum_{k=m}^{M_1-1} \Delta_k(\varphi) a_n(s_k) \right)^2 t^n \\
&= \sum_{k_1=m}^{M_1-1} \sum_{k_2=m}^{M_1-1} \Delta_{k_1}(\varphi) \Delta_{k_2}(\varphi) \sum_{n=0}^{\infty} a_n(s_{k_1}) a_n(s_{k_2}) t^n \\
&= \sum_{k_1=m}^{M_1-1} \sum_{k_2=m}^{M_1-1} \Delta_{k_1}(\varphi) \Delta_{k_2}(\varphi) f_{k_1,k_2}(t).
\end{aligned}$$

For $k_1 = k_2 = m$, we have

$$f_{m,m}(\tau(1-t)) = \widehat{s_m}(\tau(1-t)) \stackrel{\text{Lemma C.7}}{=} (-1)^{m+1} \Delta_m(\varphi)^2 \tau^{m+1} b_m t^{m+1/2} + Q_{-1,m+1/2}(t).$$

For $k_1 > m$ or $k_2 > m$, Proposition C.12 yields

$$f_{k_1,k_2}(\tau(1-t)) = Q_{-1,(k_1+k_2)/2}(t) = Q_{-1,m+1/2}(t).$$

Therefore,

$$\hat{g}_1(\tau(1-t)) = \Delta_m(\varphi)^2(-1)^{m+1}\tau^{m+1}b_m t^{m+1/2} + Q_{-1,m+1/2}(t)$$

for some constant $b_m > 0$. Moreover, for $M_1 \ge 2+m$, we have $M_1/2 \ge 3/4 + m/2$. Using Lemma C.9 and Lemma C.13, we obtain

$$\hat{\varphi}(t) - \hat{g}_1(t) = \sum_{n=0}^{\infty} \left( a_n(g_2) + \sum_{k=m}^{M_1-1} \Delta_k(\varphi) a_n(s_k) \right)^2 t^n - \sum_{n=0}^{\infty} \left( \sum_{k=m}^{M_1-1} \Delta_k(\varphi) a_n(s_k) \right)^2 t^n$$
$$ 
\begin{aligned}
&= \sum_{n=0}^{\infty} a_n(g_2) \left( a_n(g_2) + 2 \sum_{k=m}^{M_1-1} \Delta_k(\varphi) a_n(s_k) \right) t^n \\
&= \sum_{n=0}^{\infty} o((n+1)^{-M_1/2}) O((n+1)^{-3/4-m/2}) t^n \\
&= \sum_{n=0}^{\infty} o((n+1)^{-3/4-m/2-M_1/2}) t^n.
\end{aligned}
$$

Let $\gamma \in \mathbb{R}$ be arbitrary. We choose $M_1$ large enough such that

$$ 3/4 + m/2 + M_1/2 > [\gamma] + 1 \qquad (C.4) $$

Then, Lemma C.14 yields

$$ h := \hat{\varphi} - \hat{g}_1 \in C^{[\gamma]}([-1, 1]). $$

We now define the Taylor polynomials

$$ p_{\tau}(t) := \sum_{k=0}^{[\gamma]-1} \frac{\frac{d^k}{du^k} h(\tau(1-u))|_{u=0}}{k!} t^k = \mathcal{Q}_{-1,\infty}(t) $$

and the rest terms

$$ r_{\tau}(t) := h(\tau(1-t)) - p_{\tau}(t). \qquad (C.5) $$

It remains to show that $r_{\tau}(t) = \mathcal{R}_{\gamma}(t)$, which will then yield

$$ 
\begin{aligned}
\hat{\varphi}(\tau(1-t)) &= \hat{g}_1(\tau(1-t)) + p_{\tau}(t) + r_{\tau}(t) \\
&= -\Delta_m(\varphi)^2 \tau^{m+1} (-1)^{m+1} b_m t^{m+1/2} + \mathcal{P}_{-1,m+1/2}(t) + \mathcal{R}_{\gamma}(t)
\end{aligned}
$$

for arbitrary $\gamma \in \mathbb{R}$.

**Step 2: Analyzing the rest term.** Let $\tau \in \{-1, 1\}$ and $N \in \mathbb{N}_0$ be arbitrary. We need to show that $|r_{\tau}^{(N)}(t)| = O(t^{\gamma-N})$, where $r_{\tau}$ is defined in Eq. (C.5). For an integer $M_2 > M_1$ yet to be specified, we use Lemma C.15 again to decompose

$$ g_2 = \varphi_{M_2} + \sum_{k=M_1}^{M_2-1} \Delta_k(\varphi)s_k. $$

With the index set

$$ \mathcal{I} := \{m, m+1, \dots, M_2-1\}^2 \setminus \{m, m+1, \dots, M_1-1\}^2, $$

we then obtain similar to the calculation above

$$ 
\begin{aligned}
h(t) &= h_1(t) + h_2(t) \\
h_1(t) &:= \sum_{(k_1, k_2) \in \mathcal{I}} \Delta_{k_1}(\varphi) \Delta_{k_2}(\varphi) f_{k_1, k_2}(t) \\
h_2(t) &:= \sum_{n=0}^{\infty} a_n(\varphi_{M_2}) \left( a_n(\varphi_{M_2}) + 2 \sum_{k=m}^{M_2-1} \Delta_k(\varphi) a_n(s_k) \right) t^n \\
&= \sum_{n=0}^{\infty} o((n+1)^{-M_2/2}) O((n+1)^{-3/4-m/2}) t^n = \sum_{n=0}^{\infty} o((n+1)^{-3/4-m/2-M_2/2}) t^n.
\end{aligned}
$$

We now choose $M_2$ sufficiently large such that $3/4+m/2+M_2/2 > \tilde{N}+1$, where $\tilde{N} := \max\{N, [\gamma]\}$. Lemma C.14 yields

$$ h_2 \in C^{\tilde{N}}([-1, 1]). $$

Since (C.4) implies $(m+M_1)/2 \ge \gamma$, we obtain from Proposition C.12

$$ h_1(\tau(1-t)) = \mathcal{P}_{-1,(m+M_1)/2}(t) + \mathcal{R}_{\gamma}(t) = \mathcal{P}_{-1,\infty}(t) + \mathcal{R}_{\gamma}(t). $$
In other words, we can find a polynomial $p_{1,τ}$ of degree $≤ [γ] - 1$ and a function $r_{1,τ} ∈ R_γ$ such that

$$h_1(\tau(1-t)) = p_{1,τ}(t) + r_{1,τ}(t).$$

We therefore investigate

$$r_{2,τ}(t) := r_{τ}(t) - r_{1,τ}(t) = (h(τ(1-t)) - p_{τ}(t)) - (h_1(τ(1-t)) - p_{1,τ}(t)) \\ = h_2(τ(1-t)) - p_{τ}(t) + p_{1,τ}(t),$$

which satisfies $r_{2,τ} ∈ C^{\tilde{N}}([-1, 1])$. We now distinguish two cases:

* If $N ≥ γ$, we simply use the continuity of $r_{2,τ}^{(N)}$ in 0 to obtain

$$|r_{2,τ}^{(N)}(t)| = O(1) = O(t^{\gamma-N}).$$

* If $N < γ$, we proceed differently. For $0 ≤ n ≤ ⌊γ⌋ - 1$, we have

$$r_{2,τ}^{(n)}(0) = r_{2,τ}^{(n)}(0+) = r_{τ}^{(n)}(0+) - r_{1,τ}^{(n)}(0+) = 0 - 0 = 0,$$

where we used that $r_{τ}^{(n)}(0+) = 0$ by construction and that $r_{1,τ}^{(n)}(0+) = 0$ thanks to $r_{1,τ} ∈ R_γ$ and $n < γ$. Thus, Taylor's theorem with Peano's form of the remainder yields

$$|r_{2,τ}^{(N)}(t)| = \left| \sum_{k=0}^{\lceil \gamma \rceil - N} \frac{r_{2,τ}^{(N+k)}(0)}{k!} t^k + o(t^{\lceil \gamma \rceil - N}) \right| \\ = O(t^{\lceil \gamma \rceil - N}) = O(t^{\gamma - N}).$$

Since $r_{1,τ} ∈ R_γ$, we obtain

$$|r_{τ}^{(N)}(t)| ≤ |r_{1,τ}^{(N)}(t)| + |r_{2,τ}^{(N)}(t)| ≤ O(t^{\gamma-N}) + O(t^{\gamma-N}) = O(t^{\gamma-N}),$$

which completes the proof. □

# D Neural kernel proofs

## D.1 Neural kernels

Recall the network's architecture in Definition 1.

For the subsequent consideration concerning kernels, we compare the network behavior for two inputs $x, \bar{x}$. All terms
* correspond to the inputs $x$, all terms * to the input $\bar{x}$.

**Definition D.1 (Neural kernels).** Consider a network of depth $L ≥ 2$ and output dimension $d_L = 1$. Let $d = d_1 = \dots = d_{L-1}$. Define the neural network Gaussian process-kernel $k_L^{\text{NNGP}} : \mathbb{R}^{d_0} \times \mathbb{R}^{d_0} \to \mathbb{R}$ as

$$k_L^{\text{NNGP}}(x, \bar{x}) := \lim_{d \to \infty} \text{Cov}(z^{(L)}, \bar{z}^{(L)})$$

and the neural tangent kernel $k_L^{\text{NTK}} : \mathbb{R}^{d_0} \times \mathbb{R}^{d_0} \to \mathbb{R}$ as

$$k_L^{\text{NTK}}(x, \bar{x}) := \lim_{d \to \infty} \langle \nabla_{\theta} z^{(L)}, \nabla_{\theta} \bar{z}^{(L)} \rangle,$$

where $\nabla_{\theta} z^{(L)}$ denotes derivation of the output $z^{(L)}$ by all parameters $\theta$. ▲

**Lemma D.2.** Let the activation function $\varphi : \mathbb{R} \to \mathbb{R}$ fulfill Assumption 5. Then, the NNGP and NTK kernels introduced above converge almost surely and they are given by

$$k_1^{\text{NNGP}}(x, \bar{x}) := \sigma_b^2 \sigma_i^2 + \sigma_w^2 \langle x, \bar{x} \rangle \\ k_L^{\text{NNGP}}(x, \bar{x}) = \sigma_b^2 \sigma_i^2 + \sigma_w^2 \mathbb{E}_{(u,v)\sim\Sigma_{L-1}(x,\bar{x})} \varphi(u)\varphi(v) \\ k_L^{\text{NTK}}(x, \bar{x}) = \sigma_b^2(1-\sigma_i^2) + k_L^{\text{NNGP}}(x, \bar{x}) + \sigma_w^2 \mathbb{E}_{(u,v)\sim\mathcal{N}(0,\Sigma_{L-1}(x,\bar{x}))} [\varphi'(u)\varphi'(v)] k_{L-1}^{\text{NTK}}(x, \bar{x}) \\ \Sigma_L(x, x') = \begin{pmatrix} k_L^{\text{NNGP}}(x, x) & k_L^{\text{NNGP}}(x, x') \\ k_L^{\text{NNGP}}(x', x) & k_L^{\text{NNGP}}(x', x') \end{pmatrix}$$

where we define $k_1^{\text{NTK}}(x, \bar{x}) := \sigma_b^2(1 - \sigma_i^2) + k_1^{\text{NNGP}}(x, \bar{x})$.
Proof. Denote $\tilde{d}_l := d$ if $l \in \{1, \dots, L-1\}$ and $\tilde{d}_l = 1$ otherwise. Consider the inputs $\mathbf{x}, \bar{\mathbf{x}} \in \mathbb{R}^{d_0}$. For a fixed hidden layer width $d \in \mathbb{N}$, the finite version of the NTK kernel is given by

$$ \langle \nabla_{\theta} \mathbf{z}^{(L)}, \nabla_{\theta} \bar{\mathbf{z}}^{(L)} \rangle = \sum_{l=1}^{L} \langle \nabla_{\mathbf{W}^{(l)}} \mathbf{z}^{(L)}, \nabla_{\mathbf{W}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle + \sum_{l=1}^{L} \langle \nabla_{\mathbf{b}^{(l)}} \mathbf{z}^{(L)}, \nabla_{\mathbf{b}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle, \quad (D.1) $$

$$ \langle \nabla_{\mathbf{W}^{(l)}} \mathbf{z}^{(L)}, \nabla_{\mathbf{W}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle = \frac{\sigma_w^2}{\tilde{d}_{l-1}} \langle \nabla_{\mathbf{z}^{(l)}} \mathbf{z}^{(L)} \mathbf{x}^{(l-1)\top}, \nabla_{\bar{\mathbf{z}}^{(l)}} \bar{\mathbf{z}}^{(L)} \bar{\mathbf{x}}^{(l-1)\top} \rangle, $$

where inner products involving matrices are Frobenius inner products. Using the outer product structure we can rewrite the weight-matrix gradient as

$$ \begin{aligned} & \langle \nabla_{\mathbf{W}^{(l)}} \mathbf{z}^{(L)}, \nabla_{\mathbf{W}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle && (D.2) \\ &= \sigma_w^2 \langle \nabla_{\mathbf{z}^{(l)}} \mathbf{z}^{(L)}, \nabla_{\bar{\mathbf{z}}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle \frac{1}{\tilde{d}_{l-1}} \langle \mathbf{x}^{(l-1)}, \bar{\mathbf{x}}^{(l-1)} \rangle \\ &= \frac{\sigma_w^2}{d_l} \langle \sqrt{d_l} \nabla_{\mathbf{z}^{(l)}} \mathbf{z}^{(L)}, \sqrt{d_l} \nabla_{\bar{\mathbf{z}}^{(l)}} \bar{\mathbf{z}}^{(L)} \rangle \frac{1}{\tilde{d}_{l-1}} \langle \mathbf{x}^{(l-1)}, \bar{\mathbf{x}}^{(l-1)} \rangle. \end{aligned} $$

The occurring backpropagation terms can be recursively unrolled as

$$ \begin{aligned} \nabla_{\mathbf{z}^{(L)}} \mathbf{z}^{(l)} &= \frac{\sigma_w}{\sqrt{d_l}} \mathbf{W}^{(l+1)\top} \nabla_{\mathbf{z}^{(l+1)}} \mathbf{z}^{(L)} \odot \varphi'(\mathbf{z}^{(l)}) && 1 \le l \le L-1, \\ \nabla_{\mathbf{z}^{(L)}} \mathbf{z}^{(L)} &= 1, \end{aligned} $$

where $\odot$ denotes component-wise multiplication.

In order to rigorously calculate the NNGP and the NTK, we use the simplified NETSOR$^\top$ program from Section 7 in Yang (2020). To this end, define the set of $d$-dimensional initial vectors $\mathcal{V} := \{\sigma_w \mathbf{W}^{(1)}\mathbf{x}, \sigma_w \mathbf{W}^{(1)}\bar{\mathbf{x}}, \mathbf{W}^{(L)\top}, \sigma_i \sigma_b \mathbf{b}^{(1)}, \sigma_i \sigma_b \mathbf{b}^{(2)}, \dots, \sigma_i \sigma_b \mathbf{b}^{(L-1)}\}$. For $1 \le i \le d$, the process $(v_i)_{v \in \mathcal{V}}$ is a centered Gaussian process distributed as $(Z^v)_{v \in \mathcal{V}}$, where<fnref n="3" /> $\text{Cov}(Z\mathbf{W}^{(1)}\mathbf{y}, Z\mathbf{W}^{(1)}\bar{\mathbf{y}}) = \langle \mathbf{y}, \bar{\mathbf{y}} \rangle$ for $\mathbf{y}, \bar{\mathbf{y}} \in \{\mathbf{x}, \bar{\mathbf{x}}\}$, $\text{Cov}(Z\mathbf{b}^{(l)}, Z^v) = 1_{\{\mathbf{b}^{(l)}\}}(v)$ and $\text{Cov}(Z\mathbf{W}^{(L)\top}, v) = 1_{\{\mathbf{W}^{(L)\top}\}}(v)$. Define the set of $\mathbb{R}^{d \times d}$ random matrices $\mathcal{W} := \{\tilde{\mathbf{W}}^{(2)}, \dots, \tilde{\mathbf{W}}^{(L-1)}\}$ by $\tilde{\mathbf{W}}_{jk}^{(i)} \sim \mathcal{N}(0, \sigma_w^2/d)$. Now we recursively define the vectors

$$ \begin{aligned} \mathbf{h}^{(l)} &:= \sigma_w \mathbf{W}^{(1)} \mathbf{x}, && \mathbf{h}^{(l)} := \tilde{\mathbf{W}}^{(l)} \mathbf{x}^{(l-1)} && (D.3) \\ \mathbf{z}^{(l)} &:= \mathbf{h}^{(l)} + \sigma_b \sigma_i \mathbf{b}^{(l)}, && \mathbf{x}^{(l)} &:= \varphi(\mathbf{z}^{(l)}), \\ \mathrm{d}\mathbf{z}^{(L-1)} &:= \mathbf{W}^{(L)\top} \odot \varphi'(\mathbf{z}^{(L-1)}), && \mathrm{d}\mathbf{h}^{(l)} := \tilde{\mathbf{W}}^{(l+1)\top} \mathrm{d}\mathbf{z}^{(l+1)}, \\ \mathrm{d}\mathbf{z}^{(l)} &:= \mathrm{d}\mathbf{h}^{(l)} \odot \varphi'(\mathbf{z}^{(l)}), \end{aligned} $$

where the index $l$ has the range $1 \le l \le L-1$ for $\mathbf{x}^{(l)}, 1 \le l \le L-2$ for $\mathrm{d}\mathbf{h}^{(l)}, \mathrm{d}\mathbf{z}^{(l)}$ and $2 \le l \le L-1$ for $\mathbf{h}^{(l)}, \mathbf{z}^{(l)}$. Also note $\mathrm{d}\mathbf{z}^{(l)} = \frac{1}{\sqrt{d}} \nabla_{\mathbf{z}^{(l)}} \mathbf{z}^{(L)}$ for $1 \le l \le L-1$ due to the ,,missing normalization" in the definition of $\mathrm{d}\mathbf{z}^{(L-1)}$. The individual vectors in Eq. (D.3) are given by a non-linear operation or a matrix operation as in (Yang, 2020, Box 1 p. 7). This enables applying Theorem 7.2 and Box 1 from Yang (2020), which states that limits of the kernels

$$ \begin{aligned} B^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) &:= \lim_{d \to \infty} \frac{1}{d} \langle \mathbf{z}^{(l)}, \bar{\mathbf{z}}^{(l)} \rangle, \\ C^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) &:= \lim_{d \to \infty} \frac{1}{d} \langle \mathbf{x}^{(l)}, \bar{\mathbf{x}}^{(l)} \rangle, \\ D^{(l,L)}(\mathbf{x}, \bar{\mathbf{x}}) &:= \lim_{d \to \infty} \frac{1}{d} \langle \mathrm{d}\mathbf{z}^{(l)}, \mathrm{d}\bar{\mathbf{z}}^{(l)} \rangle, \\ E^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) &:= \lim_{d \to \infty} \frac{1}{d} \langle \varphi'(\mathbf{z}^{(l)}), \varphi'(\bar{\mathbf{z}}^{(l)}) \rangle. \end{aligned} $$

<footnote n="3">³Note that $Z^v$ is a real-valued random variable for $v \in \mathcal{V}$.</footnote>
exist almost surely and furthermore yields recursive formulas for those limits. Here, we emphasize the network’s depth $L$ as an extra upper index for the backpropagation terms $D^{(l,L)}$. Note that by the independence of the parameters in the network we have

$$ 
\begin{aligned}
k_L^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}}) &= \lim_{d \to \infty} \text{Cov}(z^{(L)}, \bar{z}^{(L)}) \\
&= \lim_{d \to \infty} \frac{\sigma_w^2}{d} \mathbb{E}[\mathbf{x}^{(L-1)\top} \mathbf{W}^{(L)\top} \mathbf{W}^{(L)} \bar{\mathbf{x}}^{(L-1)}] + \sigma_i^2 \sigma_b^2 \mathbb{E}[(\mathbf{b}^{(L)})^2] \\
&= \lim_{d \to \infty} \sigma_w^2 \mathbb{E} \frac{1}{d} \langle \mathbf{x}^{(L-1)}, \mathbf{x}^{(L-1)} \rangle + \sigma_i^2 \sigma_b^2 \\
&= \sigma_w^2 C^{(L-1)}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_i^2 \sigma_b^2.
\end{aligned}
$$

Here, the convergence of the expectation follows from Yang (2019a, Proposition G.4). By Eqs. (D.1) and (D.2) the NTK-kernel can be expressed as

$$ k_L^{\text{NTK}}(\mathbf{x}, \bar{\mathbf{x}}) = \sigma_w^2 \sum_{l=1}^{L} D^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) C^{(l-1)}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_b^2 \sum_{l=1}^{L} D^{(l)}(\mathbf{x}, \bar{\mathbf{x}}). $$

For a kernel $k$ denote $\Sigma_k(\mathbf{y}, \bar{\mathbf{y}}) := \begin{pmatrix} k(\mathbf{y}, \mathbf{y}) & k(\mathbf{y}, \bar{\mathbf{y}}) \\ k(\mathbf{y}, \bar{\mathbf{y}}) & k(\bar{\mathbf{y}}, \bar{\mathbf{y}}) \end{pmatrix}$. Using Eq. (D.3) and Yang (2020), the above kernels are recursively given by

$$ B^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) = \sigma_w^2 C^{l-1}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_b^2 \sigma_i^2, \qquad (D.4) $$

$$ C^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) = \mathbb{E}_{(u,v)\sim\mathcal{N}(0,\Sigma_{B(l)})(\mathbf{x},\bar{\mathbf{x}})}[\varphi(u)\varphi(v)], \quad (l \ge 1) $$

$$ C^{(0)}(\mathbf{x}, \bar{\mathbf{x}}) = \langle \mathbf{x}, \bar{\mathbf{x}} \rangle, $$

$$ D^{(l,L)}(\mathbf{x}, \bar{\mathbf{x}}) = \sigma_w^2 D^{(l+1)}(\mathbf{x}, \bar{\mathbf{x}}) E^{(l)}(\mathbf{x}, \bar{\mathbf{x}}), \quad l \le L-1, $$

$$ D^{(L,L)}(\mathbf{x}, \bar{\mathbf{x}}) = 1, $$

$$ E^{(l)}(\mathbf{x}, \bar{\mathbf{x}}) = \mathbb{E}_{(u,v)\sim\mathcal{N}(0,\Sigma_{B(l)})(\mathbf{x},\bar{\mathbf{x}})}[\varphi'(u)\varphi'(v)], $$

and a recursive formula for $D^{(l,L)}$ over the depth $L$ follows as

$$ D^{(l,L+1)} = \sigma_w^2 E^{(l)} \cdots \sigma_w^2 E^{(L)} = \sigma_w^2 E^{(L)} D^{(l,L)}. $$

The recursive formula for $k_L^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}})$ can directly be derived from Eq. (D.4). We investigate $k_L^{\text{NTK}}(\mathbf{x}, \bar{\mathbf{x}})$ by induction. For $L=1$ there is nothing to show. Assume that the claimed formula holds for $L \in \mathbb{N}$. Then we have

$$ 
\begin{aligned}
k_{L+1}^{\text{NTK}}(\mathbf{x}, \bar{\mathbf{x}}) &= \sigma_w^2 \sum_{l=1}^{L+1} D^{(l,L+1)}(\mathbf{x}, \bar{\mathbf{x}}) C^{(l-1)}(\mathbf{x}, \bar{\mathbf{x}}) \mathbf{x} + \sigma_b^2 \sum_{l=1}^{L} D^{(l,L+1)}(\mathbf{x}, \bar{\mathbf{x}}) \\
&= \sigma_w^2 E^{(L)}(\mathbf{x}, \bar{\mathbf{x}}) \left( \sigma_w^2 \sum_{l=1}^{L} D^{(l,L)}(\mathbf{x}, \bar{\mathbf{x}}) C^{(l-1)}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_b^2 \sum_{l=1}^{L} D^{(l,L)}(\mathbf{x}, \bar{\mathbf{x}}) \right) \\
&\quad + \sigma_w^2 C^{(L)}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_b^2 \\
&= \sigma_b^2 (1 - \sigma_i^2) + k_{L+1}^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}}) + \sigma_w^2 \mathbb{E}_{(u,v)\sim\mathcal{N}(0,\Sigma_L)}[\varphi'(u)\varphi'(v)] k_L^{\text{NTK}}(\mathbf{x}, \bar{\mathbf{x}}). \quad \square
\end{aligned}
$$

A dot-product kernel $k$ on the sphere $S^d$ is conveniently described by a function $\kappa: [-1, 1] \to \mathbb{R}$ by defining $\kappa(\langle \mathbf{x}, \mathbf{y} \rangle) := k(\mathbf{x}, \mathbf{y})$. We translate the NNGP- and NTK-recursion of Lemma D.2 to a recursion for the corresponding functions $\kappa$ describing the restriction of these kernels to the sphere.

**Lemma A.4 (Neural kernels on the unit sphere).** Let the activation function $\varphi: \mathbb{R} \to \mathbb{R}$ fulfill Assumption 5. Consider a neural network $f_0: \mathbb{R}^{d+1} \to \mathbb{R}$ initialized as in Definition 1. For $l \ge 2$ and $t \in [-1, 1]$ we recursively define

$$ 
\begin{aligned}
\alpha_1 &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 \\
\alpha_l &:= \sigma_b^2 \sigma_i^2 + \sigma_w^2 \overline{\varphi \cdot \sqrt{\alpha_{l-1}}} (1)
\end{aligned}
$$

$$ \kappa_1^{\text{NNGP}}(t) := \sigma_b^2 \sigma_i^2 + \sigma_w^2 t $$

$$ \kappa_l^{\text{NNGP}}(t) := \sigma_b^2 \sigma_i^2 + \sigma_w^2 \overline{\varphi \cdot \sqrt{\alpha_{l-1}}} (\kappa_{l-1}^{\text{NNGP}}(t) / \alpha_{l-1}) $$

$$ \kappa_1^{\text{NTK}}(t) := \sigma_b^2 (1 - \sigma_i^2) + \kappa_1^{\text{NNGP}}(t) $$
$$ \kappa_l^{\text{NTK}}(t) := \sigma_b^2(1 - \sigma_i^2) + \kappa_l^{\text{NNGP}}(t) + \sigma_w^2 \kappa_{l-1}^{\text{NTK}}(t) \overline{(\varphi') \cdot \sqrt{\alpha_{l-1}}} (\kappa_{l-1}^{\text{NNGP}}(t) / \alpha_{l-1}). $$

Then for all $l \ge 1$, $\mathbf{x}, \mathbf{x}' \in \mathbb{S}^d$ we have $\alpha_l > 0$ and

$$ k_l^{\text{NNGP}}(\mathbf{x}, \mathbf{x}') = \kappa_l^{\text{NNGP}}(\langle \mathbf{x}, \mathbf{x}' \rangle) $$

$$ k_l^{\text{NTK}}(\mathbf{x}, \mathbf{x}') = \kappa_l^{\text{NTK}}(\langle \mathbf{x}, \mathbf{x}' \rangle). $$

*Proof.* As $\sigma_w^2 > 0$ and $\varphi$ is not almost surely equal to zero, $\alpha_l > 0$ follows for all $l$. By a simple induction we see that $\alpha_l = k_l^{\text{NNGP}}(1)$ holds for all $l \ge 1$. The identities for the NNGP can be shown by induction on $l$ as well. The induction base $k_1^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}}) = \kappa_1^{\text{NNGP}}(\langle \mathbf{x}, \bar{\mathbf{x}} \rangle)$ is straightforward. For the induction step Lemma D.2 yields

$$ k_l^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}}) = \sigma_b^2 \sigma_i^2 + \sigma_w^2 \mathbb{E}_{(u,v) \sim \mathcal{N}(0, \Sigma_{l-1}(\mathbf{x}, \bar{\mathbf{x}}))} [\varphi(u) \varphi(v)], $$

$$ \Sigma_{l-1}(\mathbf{x}, \bar{\mathbf{x}}) = \begin{pmatrix} k_{l-1}^{\text{NNGP}}(\mathbf{x}, \mathbf{x}) & k_{l-1}^{\text{NNGP}}(\mathbf{x}, \mathbf{x}') \\ k_{l-1}^{\text{NNGP}}(\mathbf{x}, \mathbf{x}') & k_{l-1}^{\text{NNGP}}(\mathbf{x}', \mathbf{x}') \end{pmatrix} = \alpha_{l-1} \cdot \begin{pmatrix} 1 & \kappa_{l-1}^{\text{NNGP}}(\langle \mathbf{x}, \bar{\mathbf{x}} \rangle) / \alpha_{l-1} \\ \kappa_{l-1}^{\text{NNGP}}(\langle \mathbf{x}, \bar{\mathbf{x}} \rangle) / \alpha_{l-1} & 1 \end{pmatrix} $$

and the identity $k_l^{\text{NNGP}}(\mathbf{x}, \bar{\mathbf{x}}) = \kappa_l^{\text{NNGP}}(\langle \mathbf{x}, \bar{\mathbf{x}} \rangle)$ follows directly from the definition of the dual activation (A.3). A similar argument shows $k_l^{\text{NTK}}(\mathbf{x}, \bar{\mathbf{x}}) = \kappa_l^{\text{NTK}}(\langle \mathbf{x}, \bar{\mathbf{x}} \rangle)$. $\square$

## D.2 Even and odd parts of neural kernels

**Lemma D.3 (Even/odd algebra).** Let $f, g: \mathbb{R} \to \mathbb{R}$. Then,

(a) $(f+g)_{\text{even}} = f_{\text{even}} + g_{\text{even}}$ and $(f+g)_{\text{odd}} = f_{\text{odd}} + g_{\text{odd}}$.

(b) $(f \cdot g)_{\text{even}} = f_{\text{even}} \cdot g_{\text{even}} + f_{\text{odd}} \cdot g_{\text{odd}}$ and $(f \cdot g)_{\text{odd}} = f_{\text{even}} \cdot g_{\text{odd}} + f_{\text{odd}} \cdot g_{\text{even}}$.

(c) If $g$ is odd, $(f \circ g)_{\text{even}} = f_{\text{even}} \circ g$ and $(f \circ g)_{\text{odd}} = f_{\text{odd}} \circ g$. If $g$ is even, $f \circ g$ is even.

(d) If $f'$ is a weak derivative of $f$, then $(f')_{\text{even}}$ is a weak derivative of $f_{\text{odd}}$ and $(f')_{\text{odd}}$ is a weak derivative of $f_{\text{even}}$.

*Proof.* Statements (a) – (c) are straightforward to prove. For (d), we obtain for $x \in \mathbb{R}$:

$$ \begin{aligned} \int_0^x (f')_{\text{even}}(t) \, dt &= \int_0^x \frac{f'(t) + f'(-t)}{2} \, dt \\ &= \frac{1}{2} \left( \int_0^x f'(t) \, dt + \int_0^x f'(-t) \, dt \right) \\ &= \frac{1}{2} \left( \int_0^x f'(t) \, dt - \int_0^{-x} f'(t) \, dt \right) \\ &= \frac{1}{2} \left( (f(x) - f(0)) - (f(-x) - f(0)) \right) \\ &= f_{\text{odd}}(x). \end{aligned} $$

This shows that $(f')_{\text{even}}$ is a weak derivative of $f_{\text{odd}}$, and a similar computation shows that $(f')_{\text{odd}}$ is a weak derivative of $f_{\text{even}}$. $\square$

**Proposition D.4 (Special cases for NNGP/NTK with even/odd functions).** Let the activation function $\varphi$ fulfill Assumption 5.

(a) Let $\sigma_b^2 \sigma_i^2 = 0$. If $\varphi$ is even/odd, then $\kappa_l^{\text{NNGP}}$ is even/odd for all $l \ge 2$.

(b) Let $\sigma_b^2 \sigma_i^2 = 0$. Then, $(\kappa_{2,\varphi}^{\text{NNGP}})_{\text{even}} = \kappa_{2,\varphi_{\text{even}}}^{\text{NNGP}}$ and $(\kappa_{2,\varphi}^{\text{NNGP}})_{\text{odd}} = \kappa_{2,\varphi_{\text{odd}}}^{\text{NNGP}}$.

(c) Let $\sigma_b^2 = 0$. If $\varphi$ is even/odd, then $\kappa_l^{\text{NTK}}$ is even/odd for all $l \ge 2$.

(d) Let $\sigma_b^2 = 0$. Then, $(\kappa_{2,\varphi}^{\text{NTK}})_{\text{even}} = \kappa_{2,\varphi_{\text{even}}}^{\text{NTK}}$ and $(\kappa_{2,\varphi}^{\text{NTK}})_{\text{odd}} = \kappa_{2,\varphi_{\text{odd}}}^{\text{NTK}}$.

Here we denote the activation function in the index to clarify the network architecture the kernels belong to.

*Proof.*

(a) Since $\sigma_b^2 \sigma_i^2 = 0$, $\kappa_l^{\text{NNGP}}$ is odd. If $\varphi$ is even/odd, then so is $\varphi \cdot \sqrt{\alpha_l}$ and by Lemma C.3 also $\varphi \cdot \sqrt{\alpha_l}$. The claim then follows by induction using Lemma D.3 (c).
(b) We have $\kappa_{2,\varphi}^{\text{NNGP}}(t) = \sigma_w^2 \widehat{\varphi \cdot \sqrt{\alpha_1}}(t)$. Note that $\alpha_1$ does not depend on $\varphi$. By Lemma C.3, we obtain

$$ (\kappa_{2,\varphi}^{\text{NNGP}})_{\text{even}}(t) = \sigma_w^2 \widehat{(\varphi \cdot \sqrt{\alpha_1})}_{\text{even}}(t) = \sigma_w^2 \widehat{(\varphi_{\text{even}}) \cdot \sqrt{\alpha_1}}(t) = \kappa_{2,\varphi_{\text{even}}}^{\text{NNGP}}(t) $$

and similar for the odd part.

(c) Since $\sigma_b^2 = 0$, $\kappa_1^{\text{NTK}}$ is odd. Moreover,

$$ \kappa_l^{\text{NTK}}(t) = \sigma_w^2 \left( \kappa_l^{\text{NNGP}}(t) + \kappa_{l-1}^{\text{NTK}}(t) \cdot \widehat{(\varphi') \cdot \sqrt{\alpha_{l-1}}} (\kappa_{l-1}^{\text{NNGP}}(t)/\alpha_{l-1}) \right) $$

holds and we obtain the claim for $l = 2$. We use Lemma D.3 and Lemma C.3 to obtain that $\widehat{(\varphi') \cdot \sqrt{\alpha_l}}$ is odd/even if $\varphi$ is even/odd, and the claim holds for $l \ge 3$ by induction since $\kappa_l^{\text{NNGP}}(t)$ is even/odd by (a).

(d) Just as in (c), this follows from Lemma C.3, Lemma D.3 and (b). □

### D.3 Adapting the main theorem from Bietti and Bach (2021)

Here, we derive Theorem A.2, an adapted version of the main theorem of Bietti and Bach (2021) that is closer to our notation. First, we restate the original theorem with minor adaptations (using $\tau$, expanding at $t = 0$ instead of $t \in \{-1, 1\}$, rewriting derivatives of powers, using $\mathbb{S}^d$ instead of $\mathbb{S}^{d-1}$). Here, the notation $a_n \sim b_n$ means $\lim_{n \to \infty} a_n/b_n = 1$.

**Theorem D.5** (Theorem 7 in Bietti and Bach (2021), arXiv version 4). Let $k : [-1, 1] \to \mathbb{R}$ be a function that is $C^\infty$ on $(-1, 1)$ and has for $\tau \in \{1, -1\}$ the following expansion for $t \searrow 0$:

$$ k(\tau(1-t)) = p_{\tau}(t) + \sum_{j=1}^{r} c_{j,\tau} t^{\nu_j} + O(t^{\nu_1+1+\epsilon}), \qquad (D.5) $$

where $p_\tau$ are polynomials and $0 < \nu_1 < \dots < \nu_r$ are not integers and $0 < \epsilon < \nu_2 - \nu_1$. Assume further that the derivatives $k^{(s)}$ have for any $s \in \mathbb{N}_0$ the following expressions for $t \searrow 0$, where $(t^{\nu_j})^{(s)}$ is the s-th derivative of $t \mapsto t^{\nu_j}$:

$$ k^{(s)}(\tau(1-t)) = p_{s,\tau}(t) + \sum_{j=1}^{r} c_{j,\tau}(t^{\nu_j})^{(s)} + O(t^{\nu_1+1+\epsilon-s}) \qquad (D.6) $$

for some polynomials $p_{s,\tau}$. Then, for $d \ge 1$, the eigenvalues $\mu_k = \mu_k(\kappa, d)$ as defined in Section 2 satisfy, for an absolute constant $C(d, \nu_1)$,

* For $k$ even, if $c_{1,1} \neq -c_{1,-1}$: $\mu_k \sim (c_{1,1} + c_{1,-1})C(d, \nu_1)k^{-d-2\nu_1}$;

* For $k$ even, if $c_{1,1} = -c_{1,-1}$: $\mu_k = o(k^{-d-2\nu_1})$;

* For $k$ odd, if $c_{1,1} \neq c_{1,-1}$: $\mu_k \sim (c_{1,1} - c_{1,-1})C(d, \nu_1)k^{-d-2\nu_1}$;

* For $k$ odd, if $c_{1,1} = c_{1,-1}$: $\mu_k = o(k^{-d-2\nu_1})$.

We reformulate this in terms of $\mathcal{Q}$, see Definition A.1. This perspective allows conveniently handling the $O(t^{\nu_1+1+\epsilon})$-term and its derivatives. For greater precision, we use the following variant of Lemma B.2 in Haas et al. (2023) to investigate the sign of eigenvalues $\mu_i$:

**Lemma D.6** (Guaranteeing strictly positive eigenvalues for kernels on spheres). Let $\kappa : [-1, 1] \to \mathbb{R}$ be continuous, let $d \ge 1$ and consider the radial kernels

$$ k_d : \mathbb{S}^d \times \mathbb{S}^d \to \mathbb{R}, k_d(x, y) := \kappa(\langle x, y \rangle) \\ k_{d+2} : \mathbb{S}^{d+2} \times \mathbb{S}^{d+2} \to \mathbb{R}, k_{d+2}(x, y) := \kappa(\langle x, y \rangle). $$

Suppose that $k_{d+2}$ is a kernel. Then, $k_d$ is a kernel. If an eigenvalue $\mu_l$ of $k_d$ fulfills $\mu_l > 0$ and $\hat{l}$ is even/odd, then for all even/odd $l \le \hat{l}$

$$ \mu_l > 0 \qquad (D.7) $$

follows. Especially, if Eq. (D.7) holds for infinitely many even/odd $l$, then it holds for all even/odd $l$.

The only difference between Lemma B.2 in Haas et al. (2023) and the above lemma is that the eigenvalues are described in greater detail here, the proof remains the same.
**Theorem A.2** (Adaptation of Theorem 7 in arXiv v4 of Bietti and Bach 2021). Let $\kappa: [-1, 1] \to \mathbb{R}$ be a function that is smooth on $(-1, 1)$ such that $k_{\kappa,d}(\mathbf{x}, \mathbf{x}') = \kappa(\langle \mathbf{x}, \mathbf{x}' \rangle)$ is a positive semi-definite kernel on all spheres $\mathbb{S}^d$, $d \in \mathbb{N}_{\ge 1}$. Suppose that there exists $0 < \beta \in \mathbb{R} \setminus \mathbb{Z}$ and $b_{-1}, b_1 \in \mathbb{R}$ such that for $\tau \in \{-1, 1\}$,

$$ \kappa(\tau(1-t)) = b_{\tau}t^{\beta} + Q_{-1,\beta}(t). $$

Then, for a given dimension $d \in \mathbb{N}_{\ge 1}$, the eigenvalues $\mu_l = \mu_l(\kappa, d)$ as defined in Section 2 satisfy:

(a) For $l \in \mathbb{N}_0$ even, if $b_{-1} \neq -b_1$, then $\mu_l = \Theta_{\nu_l}((l+1)^{-d-2\beta})$.

(b) For $l \in \mathbb{N}_0$ even, if $b_{-1} = -b_1$, then $\mu_l = o_{\nu_l}((l+1)^{-d-2\beta})$.

(c) For $l \in \mathbb{N}_0$ odd, if $b_{-1} \neq b_1$, then $\mu_l = \Theta_{\nu_l}((l+1)^{-d-2\beta})$.

(d) For $l \in \mathbb{N}_0$ odd, if $b_{-1} = b_1$, then $\mu_l = o_{\nu_l}((l+1)^{-d-2\beta})$.

*Proof.* Set $\gamma := \beta+2$. Fix $\tau \in \{-1, 1\}$. Then, we can write $\kappa(\tau(1-t)) = b_{\tau}t^{\beta} + p(t) + r(t)$ for some $p \in \mathcal{P}_{-1,\beta}$ and $r \in \mathcal{R}_{\gamma}(t)$. To apply Theorem D.5, we can rewrite $b_{\tau}t^{\beta} = p_{\tau}(t) + \sum_{j=1}^{r} c_{j,\tau}t^{\nu_j}$ with $\nu_1 = \beta$ and $c_{1,\tau} = b_{\tau}$. The rest term $r(t)$ is covered by the $O(t^{\nu_1+1+\epsilon})$ term in Theorem D.5 by setting $\epsilon := \frac{1}{2}\min\{1, \nu_2 - \nu_1\}$, and the expressions for the derivatives in Theorem D.5 are satisfied by the definition of $\mathcal{R}_{\gamma}$.

Hence, we can apply Theorem D.5 to obtain the correct asymptotic decay rates, and it remains to show that in the cases (a) and (c), all eigenvalues $\mu_l$ are strictly positive for all $l \in \mathbb{N}_0$. This claim follows from the following considerations:

* $C(d, \nu_1) \neq 0$: While this is not shown in Bietti and Bach (2021), it follows by considering $\tilde{\kappa}(t) := (1-t)^{\nu_1}$: This choice of $\kappa$ satisfies the assumptions of Theorem D.5 with $c_{1,1} = 1 \neq 0 = c_{1,-1}$, hence the cases (a) or respectively (c) apply.

Suppose $C(d, \nu_1) = 0$, then we obtain $\mu_k \sim 0$ and hence only finitely many $\mu_k$ are nonzero. Following Appendix A in Bietti and Bach (2021), $\tilde{\kappa}$ can be expressed as

$$ \tilde{\kappa}(t) = \sum_{l=0}^{\infty} \mu_l N_{d,l} P_l(t), $$

where the $P_l$ are Legendre polynomials of degree $l$ for the dimension $d+1$. Hence, $\tilde{\kappa}$ would be a polynomial, contradicting the definition $\tilde{\kappa}(t) = (1-t)^{\nu_1}$.

* Since we assumed $k_{\kappa,d}$ to be a (positive semi-definite) kernel, we cannot have negative eigenvalues, hence $(c_{1,1} + c_{1,-1})C(d, \nu_1) > 0$ or respectively $(c_{1,1} - c_{1,-1})C(d, \nu_1) > 0$ follows in the cases (a) and (c).

* The notation $\sim$ from Theorem D.5 allows a finite number of $\mu_l$ to be zero. However, as there are infinitely many both even and odd indices for which $\mu_l$ is strictly positive, Lemma D.6 shows that $\mu_l > 0$ holds for all $l \in \mathbb{N}_0$. $\square$

## D.4 Eigenvalue decay

Some preliminary constructions useful for both the NNGP and the NTK kernel are first established here, stressing the perspective of layerwise computation in the neural network.

We will study $\kappa_L^{\text{NNGP}}$, $\kappa_L^{\text{NTK}}: [-1, 1] \to \mathbb{R}$ as compositions of functions which break the kernels given by depth $L$ down to a “composition” of kernels of depth 1.

Recall the recursive form of the NNGP-kernel on the sphere given in Lemma A.4 and the dual activation function $\hat{\varphi}$ in Definition A.3.

**Notation 7** (NNGP- and NTK related terms). Recursively define the functions $g_l, G_l, H_l: [-1, 1] \to [-1, 1]$ by

$$ \alpha_l := \kappa_{l,\varphi}^{\text{NNGP}}(1), $$

$$ g_1(t) := \frac{\sigma_b^2 \sigma_i^2 + \sigma_w^2 t}{\alpha_1}, $$

$$ g_l(t) := \frac{\sigma_b^2 \sigma_i^2 + \sigma_w^2 \hat{\varphi} \cdot \sqrt{\alpha_{l-1}}(t)}{\alpha_l}, $$

$$ G_l(t) := g_l \circ \cdots \circ g_1(t), $$

$$ H_1(t) := \sigma_b^2 + \sigma_w^2 t, $$

$$ H_l(t) := \sigma_b^2(1 - \sigma_i^2) + \alpha_l G_l(t) + \sigma_w^2 \hat{\varphi}' \cdot \sqrt{\alpha_{l-1}}(G_{l-1}(t)) H_{l-1}(t). $$

<figure><img src="image_10.png" /></figure>
By the recursive formulation of $\kappa_l^{\text{NNGP}}$, $\kappa_l^{\text{NTK}}$ in Lemma A.4 we observe for all $t \in [-1, 1]$

$$ \alpha_l G_l(t) = \kappa_l^{\text{NNGP}}(t), \\ H_l(t) = \kappa_l^{\text{NTK}}(t). $$

**Lemma D.8.** Under the assumptions of Lemma A.4, we have for any $l \ge 1$

$$ \max_{t \in [-1,1]} |g_l(t)| = g_l(1) = 1. $$

Furthermore,

$$ G_l[-1, 1) \subset (-1, 1) $$

holds for all $l \ge 1$ when $\sigma_b^2 \sigma_i^2 > 0$, and for all $l \ge 2$ when $\varphi$ is neither even nor odd.

*Proof.* Follows from the definition of $g_l$ and Lemma C.3. $\square$

**Lemma D.9 (Behavior at 1).** Let the activation function $\varphi$ fulfill Assumption 5 and let $m := \inf\{k \in \mathbb{N}_0 | \varphi^{(k)}(0+) \neq \varphi^{(k)}(0-)\}$ be its smoothness, see Definition 7. Let $l \ge 2$.

1. Let $m = 0$. Then we have for $t \searrow 0$

$$ g_l(1-t) = 1 - (c_l t^{1/2} + Q_{0,1/2}(t)), \\ G_l(1-t) = 1 - (C_l t^{2^{1-l}} + Q_{0,2^{1-l}}(t)), $$

for constants $c_l, C_l > 0$.

2. Let $m \in \mathbb{N}_{\ge 1}$. Then we have for $t \searrow 0$

$$ g_l(1-t) = 1 - (b_l t + (-1)^m c_l t^{m+1/2} + Q_{1,m+1/2}(t)), \\ G_l(1-t) = 1 - (B_l t + (-1)^m C_l t^{m+1/2} + Q_{1,m+1/2}(t)), \\ H_l(1-t) = A_l - ((-1)^{m+1} D_l t^{m-1/2} + Q_{0,m-1/2}(t)) $$

for constants $b_l = g_l'(1)$, $c_l, A_l, B_l, C_l, D_l, > 0$. Furthermore, we have for $l \ge 3$ the recursive relation
$$ C_l \ge g_l'(1)C_{l-1} $$

3. Let $m = \infty$. Then we have for any $\tau \in \{\pm 1\}$, $q \in \mathbb{R}$ and $t \searrow 0$

$$ g_l(\tau(1-t)) = Q_{-1,q}(t), \\ G_l(\tau(1-t)) = Q_{-1,q}(t), \\ H_l(\tau(1-t)) = Q_{-1,q}(t). $$

*Proof.*

1. The formula for $g_l$ follows from Theorem A.6 and Lemma D.8. We obtain the formula for $G_l$ via induction, where in the base case for $l = 1$ by definition $G_1(1-t) = 1 - c_1 t$ with $c_1 > 0$ holds. For $l = 2$ we have

$$ G_2(1-t) = g_2(G_1(1-t)) = g_2(1 - (1 - G_1(1-t))) = 1 - c_2 c_1^{1/2} t^{1/2} + Q_{0,1/2}(c_1 t) $$

as claimed. For $l \ge 3$, note that the identities $Q_{0,2^{2-l}}(t) = Q_{2^{2-l},2^{2-l}}(t)$ and $Q_{0,1/2}(t) = Q_{1/2,1/2}(t)$ hold by Definition A.1. From Proposition B.2 (e,f) we obtain

$$ G_l(1-t) = g_l(G_{l-1}(1-t)) = g_l(1 - (1 - G_{l-1}(1-t))) \\ = 1 - c_l \left( C_{l-1} t^{2^{2-l}} + Q_{2^{2-l},2^{2-l}}(t) \right)^{1/2} + Q_{1/2,1/2} \left( C_{l-1} t^{2^{2-l}} + Q_{2^{2-l},2^{2-l}}(t) \right) \\ = 1 - \left( C_l t^{2^{1-l}} + Q_{2^{1-l},2^{1-l}}(t) \right) $$

and the claim follows from the identity $Q_{2^{1-l},2^{1-l}}(t) = Q_{0,2^{1-l}}(t)$.
2. Define $\beta := m + 1/2$. From Theorem A.6 we obtain

$$g_l(1-t) = (-1)^{m+1} c_l t^\beta + Q_{-1,\beta}(t)$$

with $c_l > 0$. Lemma D.8 shows $g_l(0) = 1$ and furthermore we observe

$$\widehat{\varphi' \sqrt{\alpha_{l-1}}}(1) = \mathbb{E}_{u \sim \mathcal{N}(0, \alpha_{l-1})} [\varphi'(u)^2] > 0$$

as $\varphi$ is not constant, since $1 \le m < \infty$. From Lemma D.8 and using $b_l := g'_l(1) > 0$, it follows that

$$g_l(1-t) = 1 - \left( b_l t + (-1)^m c_l t^{m+1/2} + Q_{1,m+1/2}(t) \right).$$

Note that for the NNGP term $G_l$ we have to argue more carefully to obtain the strictly positive factor $B_l > 0$ for the linear term, while this is not required for the NTK term $H_l$ where we can rely on the more precisely calculated NNGP term. We show the formula for $G_l$ via induction. For $l = 2$ it follows directly from $G_2 = g_2 \circ g_1$ using $g_1(1-t) = 1 - c_1 t$. In the induction step, let $l \ge 3$ and define

$$\bar{G}_{l-1}(t) := 1 - G_{l-1}(t) = B_{l-1}t + (-1)^m C_{l-1}t^\beta + Q_{1,\beta}(t).$$

We rewrite

$$G_l(1-t) = g_l(G_{l-1}(1-t)) = g_l(1 - \bar{G}_{l-1}(t)) \\ = 1 - \left( b_l \bar{G}_{l-1}(t) + (-1)^m c_l (\bar{G}_{l-1}(t))^\beta + Q_{1,\beta}(\bar{G}_{l-1}(t)) \right)$$

and investigate the single summands. The linear summand straightforwardly yields

$$b_l \bar{G}_{l-1}(t) = b_l B_{l-1}t + (-1)^m b_l C_{l-1}t^\beta + Q_{1,\beta}(t).$$

The power summand can be handled with Proposition B.2 (e), where we write $\bar{G}_{l-1} = b_l B_{l-1}t + Q_{1,1}(t)$ and consequently we have

$$(\bar{G}_{l-1}(t))^\beta = (b_l B_{l-1})^\beta t^\beta + Q_{\beta,\beta}(t) \\ = (b_l B_{l-1})^\beta t^\beta + Q_{1,\beta}(t).$$

In order to calculate $Q_{1,\beta}(\bar{G}_{l-1}(t))$ we decompose $Q_{1,\beta}(t) = p(t) + q(t)$, where $p(t)$ is a polynomial with integer powers greater than 1 and $q(t) \in Q_{\beta,\beta}(t)$. Using Proposition B.2 (d) we have, as $p(0) = p'(0) = 0$ holds,

$$p(\bar{G}_{L-1}(t)) = p(B_{l-1}t + (-1)^m C_{l-1}t^\beta + Q_{1,\beta}(t)) = Q_{1,\beta}(t).$$

Proposition B.2 (f) yields $q(t) = Q_{\beta,\beta}(t) = Q_{1,\beta}(t)$ and we obtain $Q_{1,\beta}(\bar{G}_{l-1}(t)) = Q_{1,\beta}(t)$. Altogether, we see that

$$G_l(1-t) = 1 - \left( b_l B_{l-1}t + (-1)^m (b_l C_{l-1} (b_l B_{l-1})^\beta) t^\beta + Q_{1,\beta}(t) \right)$$

holds as desired, and furthermore we have

$$C_l := b_l C_{l-1} (b_l B_{l-1})^\beta \ge b_l C_{l-1} = g'_l(1) C_{l-1}.$$

We move on to calculate $H_l$ for $l \ge 2$. An elementary argument yields $\kappa^{\text{NTK}}(1) > 0$ as a consequence of $\sigma_w > 0$, hence we have $A_l > 0$ for all $l \ge 2$. Again, the induction is based on the recursive construction in Notation 7, where we handle the term arising from $\widehat{\varphi' \sqrt{\alpha_l}}$ using Theorem A.6. Since $\varphi'$ has smoothness $m-1$ we obtain

$$g'_l(1-t) = \widehat{\varphi' \sqrt{\alpha_l}}(1-t) = a_l(-1)^m t^{\beta-1} + Q_{-1,\beta-1}(t)$$

for some $a_l > 0$. For $l = 2$, the claimed form of $H_l$ now follows straightforwardly. For the induction step, assume $l \ge 3$. Then we have

$$\widehat{\varphi' \sqrt{\alpha_{l-1}}}(G_{l-1}(1-t)) = \bar{c}_l + a_l(-1)^m (\bar{G}_{l-1}(t))^{\beta-1} + Q_{0,\beta-1}(\bar{G}_{l-1}(t))$$

where $\bar{c}_l \ge 0$ holds since we have $\widehat{\varphi' \sqrt{\alpha_{l-1}}}(G_{l-1}(1-0)) = \widehat{\varphi' \sqrt{\alpha_{l-1}}}(1) \ge 0$. A similar argument as above shows

$$Q_{-1,\beta-1}(\bar{G}_{l-1}(t)) = Q_{-1,\beta-1}(t),$$
$$ (\bar{G}_{l-1}(t))^{\beta-1} = B_{l-1}^{\beta-1} t^{\beta-1} + Q_{-1,\beta-1}(t). $$

We define $\bar{D}_l := a_l B_{l-1}^{\beta-1}$ and conclude

$$ \begin{aligned} & \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1-t))H_{l-1}(1-t) \\ &= (\bar{c}_l + (-1)^m \bar{D}_l t^{\beta-1} + Q_{0,\beta-1}(t))(A_{l-1} - ((-1)^{m+1} D_{l-1} t^{\beta-1} + Q_{0,\beta-1}(t))) \\ &= A_l - ((-1)^{m+1} D_l t^{\beta-1} + Q_{0,\beta-1}(t)), \end{aligned} $$

where $D_l > 0$, and the claim follows.

3. Choose a natural number $m \ge q - 1/2$ and apply Theorem A.6 to obtain

$$ g_l(\tau(1-t)) = Q_{-1,m+1/2}(t) = Q_{-1,q}(t). $$

Using Proposition B.2 c), we obtain the claim by induction. $\square$

**Theorem D.10.** Let the assumptions of Lemma A.4 be satisfied. Choose a number of layers $L \ge 2$ and a parity $r \in \{0, 1\}$ of the eigenvalues to be analyzed. Define $\varphi^{[0]} := \varphi_{\text{even}}$ and $\varphi^{[1]} := \varphi_{\text{odd}}$.

(NNGP) Define the simplified activation $\tilde{\varphi}$ by

$$ \tilde{\varphi} := \begin{cases} \varphi^{[r]} & , \text{ if } \sigma_b^2 \sigma_i^2 = 0 \text{ and } (L = 2 \text{ or } \varphi \text{ is even or odd}) \\ \varphi & , \text{ otherwise,} \end{cases} $$

and let $s := \text{smoothness}(\tilde{\varphi})$. Note that in all cases, the kernels $k_L^{\text{NNGP}}$ and $k_L^{\text{NTK}}$ refer to the original activation $\varphi$, while the smoothness parameter $s$ is determined by the simplified activation $\tilde{\varphi}$.

(1.1) If $s = 0$, then $\mu_{2t+r,d}(k_L^{\text{NNGP}}) = \Theta_{\varphi t}((2t+r+1)^{-d-2^{2-L}})$.

(1.2) If $1 \le s < \infty$, then $\mu_{2t+r,d}(k_L^{\text{NNGP}}) = \Theta_{\varphi t}((2t+r+1)^{-d-2s-1})$.

(1.3) If $s = \infty$ and $\tilde{\varphi}$ is not a polynomial, then $\mu_{2t+r,d}(k_L^{\text{NNGP}}) > 0$ for all $t$ and $\mu_{2t+r,d}(k_L^{\text{NNGP}}) = o_{\varphi t}((2t+r+1)^{-q})$ for all $q > 0$.

(1.4) If $\tilde{\varphi}$ is a polynomial, refer to the polynomial case below.

(NTK) Define the simplified activation $\tilde{\varphi}$ by

$$ \tilde{\varphi} := \begin{cases} \varphi^{[r]} & , \text{ if } \sigma_b^2 = 0 \text{ and } (L = 2 \text{ or } \varphi \text{ is even or odd}) \\ \varphi & , \text{ otherwise,} \end{cases} $$

and let $s := \text{smoothness}(\tilde{\varphi})$.

(2.1) If $1 \le s < \infty$, then $\mu_{2t+r,d}(k_L^{\text{NTK}}) = \Theta_{\varphi t}((2t+r+1)^{-d-2s+1})$.

(2.2) If $s = \infty$ and $\tilde{\varphi}$ is not a polynomial, then $\mu_{2t+r,d}(k_L^{\text{NTK}}) > 0$ for all $t$ and $\mu_{2t+r,d}(k_L^{\text{NTK}}) < o_{\varphi t}((2t+r+1)^{-q})$ for all $q > 0$.

(2.3) If $\tilde{\varphi}$ is a polynomial, refer to the polynomial case below.

**Polynomial case:**

(3.1) Let the activation $\tilde{\varphi}(t) = \sum_{i \ge 0} \lambda_i t^i$ be a nonzero polynomial. Define the even/odd degree of $\tilde{\varphi}$ as the degree of $\tilde{\varphi}(t) + \tilde{\varphi}(-t)$ or of $\tilde{\varphi}(t) - \tilde{\varphi}(-t)$. For the NNGP, define $\sigma := \sigma_b^2 \sigma_i^2$ and for the NTK define $\sigma := \sigma_b^2$ respectively.

Define $N_{\text{even}}$, $N_{\text{odd}}$ according to the following table. By $\text{deg}_{\text{ev}}^{\text{Her}}(\tilde{\varphi})$, $\text{deg}_{\text{od}}^{\text{Her}}(\tilde{\varphi})$ we denote the even/odd degree, see Definition D.11, of $\tilde{\varphi}$ displayed in the Hermite basis, which is possible as $\varphi$ fulfills Assumption 5.

<table><thead><tr><th></th><th>N<sub>even</sub></th><th>N<sub>odd</sub></th></tr></thead><tbody><tr><td>σ<sup>2</sup> > 0, deg<sub>ev</sub><sup>Her</sup>(φ̃) > deg<sub>od</sub><sup>Her</sup>(φ̃)</td><td>deg<sub>ev</sub><sup>Her</sup>(φ̃)<sup>L-1</sup></td><td>deg<sub>ev</sub><sup>Her</sup>(φ̃)<sup>L-1</sup> - 1</td></tr><tr><td>σ<sup>2</sup> > 0, deg<sub>ev</sub><sup>Her</sup>(φ̃) < deg<sub>od</sub><sup>Her</sup>(φ̃)</td><td>deg<sub>od</sub><sup>Her</sup>(φ̃)<sup>L-1</sup> - 1</td><td>deg<sub>od</sub><sup>Her</sup>(φ̃)<sup>L-1</sup></td></tr><tr><td>σ<sup>2</sup> = 0, deg<sub>ev</sub><sup>Her</sup>(φ̃) > deg<sub>od</sub><sup>Her</sup>(φ̃)</td><td>deg<sub>ev</sub><sup>Her</sup>(φ̃)<sup>L-1</sup></td><td>deg<sub>ev</sub><sup>Her</sup>(φ̃)<sup>L-1</sup> - deg<sub>ev</sub><sup>Her</sup>(φ̃) + deg<sub>od</sub><sup>Her</sup>(φ̃)</td></tr><tr><td>σ<sup>2</sup> = 0, deg<sub>ev</sub><sup>Her</sup>(φ̃) < deg<sub>od</sub><sup>Her</sup>(φ̃)</td><td>deg<sub>od</sub><sup>Her</sup>(φ̃)<sup>L-1</sup> - deg<sub>od</sub><sup>Her</sup>(φ̃) + deg<sub>ev</sub><sup>Her</sup>(φ̃)</td><td>deg<sub>od</sub><sup>Her</sup>(φ̃)<sup>L-1</sup></td></tr></tbody></table>

Then,

$$ \mu_{2t+r,d}(k_L^{\text{NNGP}}/k_L^{\text{NTK}}) > 0 \text{ if and only if } \begin{cases} 2t+r \le N_{\text{even}}, & r \text{ even,} \\ 2t+r \le N_{\text{odd}}, & r \text{ odd.} \end{cases} $$

(3.2) If $\tilde{\varphi} = 0$, then $\mu_{2t+r,d}(k_L^{\text{NNGP}}) > 0$ if and only if $2t+r = 0$ and $\sigma_b^2 \sigma_i^2 > 0$, and $\mu_{2t+r,d}(k_L^{\text{NTK}}) > 0$ if and only if $2t+r = 0$ and $\sigma_b^2 > 0$.
The definition of $\tilde{\varphi}$ may look complicated. However, in contrast to $\varphi$, it prevents that $c_{1,1} = -(-1)^T c_{1,-1}$ in Theorem A.2, which then would not provide an exact decay rate.

*Proof.* For a bounded radial kernel $k(x, y) := \kappa(\langle x, y \rangle)$ on $\mathbb{S}^d$ we define $k_{\text{even}}(x, y) := \kappa_{\text{even}}(\langle x, y \rangle)$, $k_{\text{odd}}(x, y) := \kappa_{\text{odd}}(\langle x, y \rangle)$. The spherical harmonics of degree $l$ are even if $l$ is even and odd if $l$ is odd. Hence, we obtain

$$ \mu_{2t,d}(k) = \mu_{2t,d}(k_{\text{even}}), \\ \mu_{2t+1,d}(k) = \mu_{2t+1,d}(k_{\text{odd}}) $$

for all $t \in \mathbb{N}_0$. For the NNGP- and NTK-kernel, Proposition D.4 shows that in the cases where $\tilde{\varphi} \neq \varphi$ holds we have

$$ \begin{aligned} \kappa_{L,\varphi_{\text{even}}}^{\text{NNGP}} &= (\kappa_{L,\varphi}^{\text{NNGP}})_{\text{even}}, \\ \kappa_{L,\varphi_{\text{even}}}^{\text{NTK}} &= (\kappa_{L,\varphi}^{\text{NTK}})_{\text{even}}, \\ \kappa_{L,\varphi_{\text{odd}}}^{\text{NNGP}} &= (\kappa_{L,\varphi}^{\text{NNGP}})_{\text{odd}}, \\ \kappa_{L,\varphi_{\text{odd}}}^{\text{NTK}} &= (\kappa_{L,\varphi}^{\text{NTK}})_{\text{odd}}, \end{aligned} $$

hence we conclude

$$ \mu_{2t+r}(k_{L,\tilde{\varphi}}^{\text{NNGP}}) = \mu_{2t+r}(k_{L,\varphi}^{\text{NNGP}}), \\ \mu_{2t+r}(k_{L,\tilde{\varphi}}^{\text{NTK}}) = \mu_{2t+r}(k_{L,\varphi}^{\text{NTK}}), $$

and it suffices to investigate the kernels induced by $\tilde{\varphi}$. In the following, $g_l$, $G_l$ and $H_l$ from Notation 7 refer to the functions induced by the activation function $\tilde{\varphi}$.

## NNGP:

(1.1) Lemma D.9 yields

$$ G_L(1-t) = C_{L,+}t^{2^{1-L}} + Q_{-1,2^{1-L}}(t) $$

for $t \searrow 0$, where $|C_{L,+}| > 0$. By Theorem A.2, it suffices to show

$$ \begin{gathered} G_L(-(1-t)) = C_{L,-}t^{2^{1-L}} + Q_{-1,2^{1-L}}(t), \tag{D.8} \\ C_{L,-} \neq \begin{cases} -C_{L,+} & , \text{ if } r=0 \\ C_{L,+} & , \text{ if } r=1. \end{cases} \end{gathered} $$

(a) Suppose $\sigma_i^2 \sigma_b^2 > 0$. By Lemma D.8 we have $g_1([-1, 1)) \subset (-1, 1)$ and $G_l([-1, 1)) \subset (-1, 1)$ for $l \ge 2$. As $g_1$ is a polynomial, $g_1(-(1-t)) = Q_{-1,2^{1-L}}(t)$ holds. By Lemma C.3 (d) we have $g_l|_{(-1,1)} \in C^\infty([-1, 1))$ for all $l \ge 2$, which allows leveraging Proposition B.2 (d) to obtain $G_l(-(1-t)) = g_l(G_{l-1}(-(1-t))) = Q_{-1,2^{1-L}}(t)$ and Eq. (D.8) follows.

(b) Suppose $\sigma_b^2 \sigma_i^2 = 0$, $\tilde{\varphi}$ be neither even nor odd and $L \ge 3$. Then, $g_1(t) = t$ and $G_2(t) = \hat{\varphi}(t)/\hat{\varphi}(1)$ hold and from Theorem A.6 we obtain

$$ G_2(-(1-t)) = C_{2,-}t^{1/2} + Q_{-1,1/2}(t). $$

Lemma D.8 yields $G_l([-1, 1)) \subset (-1, 1)$ for $l \ge 2$ and we furthermore have $g_l|_{(-1,1)} \in C^\infty([-1, 1))$ by Lemma C.3 (d). Recursively applying Proposition B.2 (d) now yields

$$ G_L(-(1-t)) = g_L(G_{L-1}(-(1-t))) = Ct^{1/2} + Q_{-1,1/2}(t) = Q_{-1,2^{1-L}}(t) $$

and we have Eq. (D.8).

(c) Let $\sigma_b^2 \sigma_i^2 = 0$ and let $\tilde{\varphi}$ be odd. Then, $r = 1$ follows by construction. Proposition D.4 shows that $G_l$ is odd for all $l \in \mathbb{N}$ and we obtain

$$ G_L(-(1-t)) = -G_L(1-t) = -(C_{L,+}t^{2^{1-L}} + Q_{-1,2^{1-L}}(t)) $$

follows. As desired, we have $C_{L,+} \neq C_{L,-} = -C_{L,+}$.

(d) The case $\sigma_b^2 \sigma_i^2 = 0$ and $\tilde{\varphi}$ even is not possible since even activation functions $\tilde{\varphi}$ cannot have smoothness $s = 0$.
(1.2) Define $\beta := s + 1/2$. Lemma D.9 yields

$$G_L(1-t) = C_{L,+}t^\beta + Q_{-1,\beta}(t)$$

for $t \searrow 0$, where $|C_{L,+}| > 0$. By Theorem A.2 it suffices to show

$$G_L(-(-1-t)) = C_{L,-}t^\beta + Q_{-1,\beta}(t), \qquad (D.9)$$

$$C_{L,-} \neq \begin{cases} -C_{L,+} & , \text{ if } r = 0 \\ C_{L,+} & , \text{ if } r = 1. \end{cases}$$

(a) Let $\sigma_b^2 \sigma_i^2 > 0$. By Lemma D.8 we have $g_1([-1, 1)) \subset (-1, 1)$ and $G_i([-1, 1)) \subset (-1, 1)$ for $i \ge 2$. As $g_1$ is a polynomial, $g_1(-(-1-t)) = Q_{-1,\beta}(t)$ holds. Lemma C.3 (d) shows $g_i|_{(-1,1)} \in C^\infty((-1, 1))$ for all $i \ge 2$ which allows leveraging Proposition B.2 (d) to obtain $G_L(-(-1-t)) = g_L(G_{L-1}(-(-1-t))) = Q_{-1,\beta}(t)$ by induction. Eq. (D.9) follows.

(b) Let $\sigma_b^2 \sigma_i^2 = 0$, $\tilde{\varphi}$ be neither even nor odd and $L \ge 3$, that is, $\tilde{\varphi} = \varphi$. Then, $g_1(t) = t$ and $G_2(t) = \widehat{\varphi}(t)/\widehat{\varphi}(1)$ hold and from Theorem A.6 we obtain

$$G_2(1-t) = C_{2,+}t^\beta + Q_{-1,\beta}(t),$$

$$G_2(-(-1-t)) = C_{2,-}t^\beta + Q_{-1,\beta}(t),$$

where $|C_{2,+}| = |C_{2,-}| \neq 0$. Recursively applying Lemma D.9 furthermore yields

$$|C_{L,+}| \ge g'_L(1) \cdots g'_3(1) |C_{2,+}|.$$

Lemma D.8 yields $G_l([-1, 1)) \subset (-1, 1)$ for $l \ge 2$ and we have $g_l|_{(-1,1)} \in C^\infty(-1, 1)$ by Lemma C.3. Iteratively applying Proposition B.2 (d) yields

$$G_L(-(-1-t)) = g'_L(G_{L-1}(-1)) \cdots g'_3(G_2(-1)) C_{2,-}t^\beta + Q_{-1,\beta}(t).$$

Note that we have $g'_l(t) = \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}(t)}/\alpha_l$ and by Lemma C.3 (c) we have

$$|g'_l(t)| < |g'_l(1)|$$

for all $t \in (-1, 1)$. Altogether we conclude

$$G_L(1-t) = C_{L,+}t^\beta + Q_{-1,\beta}(t),$$

$$G_L(-(-1-t)) = C_{L,-}t^\beta + Q_{-1,\beta}(t),$$

where $|C_{L,+}| > |C_{L,-}|$ holds and we obtain Eq. (D.9).

(c) Let $\sigma_b^2 \sigma_i^2 = 0$ and let $\tilde{\varphi}$ be odd, so we have $r = 1$. By Proposition D.4, $G_l$ is odd for all $l \in \mathbb{N}$ and

$$G_L(-(-1-t)) = -G_L(1-t) = -(C_{L,+}t^\beta + Q_{-1,\beta}(t))$$

follows. We obtain Eq. (D.9) as $C_{L,+} \neq C_{L,-} = -C_{L,+}$ holds.

(d) In the case $\sigma_b^2 \sigma_i^2 = 0$ and $\tilde{\varphi}$ even Proposition D.4 yields $G_l$ even and we argue as in (c).

(1.3) For $q > 0$ choose $\tilde{q} > 0$, $\tilde{q} \notin \mathbb{Z}$ such that $-d - 2\tilde{q} \le -q$. By Lemma D.9 we have for $\tau \in \{1, -1\}$:

$$G_L(\tau(1-t)) = Q_{-1,\tilde{q}}(t)$$

and Theorem A.6 yields $\mu_{2t+r,d}(k_L^{\text{NNGP}}) = o_{\forall t}((2t+r+1)^{-d-2\tilde{q}}) = o_{\forall t}((2t+r+1)^{-q})$.

We proceed to show that all Eigenvalues $\mu_{2t+1,d}(k_L^{\text{NNGP}})$ are strictly positive. As $\tilde{\varphi}$ is not a polynomial, its Hermite series representation has infinitely many nonzero coefficients; that is, we have $\text{deg}_{\text{Her}}(\tilde{\varphi}) = \infty$. See Appendix D.5 for more details. We apply Lemma D.13.

Assume even parity $r = 0$. In the case $\text{deg}_{\text{ev}}(\tilde{\varphi}) = \infty$, the claim follows directly. In the case $\text{deg}_{\text{ev}}(\tilde{\varphi}) < \text{deg}_{\text{od}}(\tilde{\varphi}) = \infty$, we note that $\tilde{\varphi} = \varphi$ follows and investigate sub-cases. If $\sigma_b^2 \sigma_i^2 > 0$, then Lemma D.13 directly yields $\text{deg}_{\text{ev}}(G_l) = \infty$ as desired. If $\sigma_b^2 \sigma_i^2 = 0$ holds, then the definition of $\tilde{\varphi}$ implies that $\text{deg}_{\text{ev}}(\tilde{\varphi}) \ge 0$ and $L \ge 3$ hold, and again the claim follows directly from Lemma D.13. The case of odd parity $r = 1$ is proven analogously.

### NTK:

The argumentation for the NTK-kernel is more technical as for the NNGP-kernel, as by Notation 7 $H_l$ follows the recursion

$$H_1(t) = \sigma_b^2 + \sigma_w^2 t, \\ H_l(t) = \sigma_b^2(1 - \sigma_i^2) + \alpha_l G_l(t) + \sigma_w^2 \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}(G_{l-1}(t))} H_{l-1}(t), \quad l \ge 2, \qquad (D.10)$$

which involves more terms in a complicated fashion.
(2.1) Define $\gamma := s - 1/2$. Lemma D.9 yields

$$H_L(1-t) = C_{L,+}t^\gamma + Q_{-1,\gamma}(t)$$

for $t \searrow 0$, where $|C_{L,+}| > 0$. By Theorem A.2 we need to show

$$H_L(-(1-t)) = C_{L,-}t^\gamma + Q_{-1,\gamma}(t), \qquad (D.11)$$

$$C_{L,-} \neq \begin{cases} -C_{L,+} & , \text{if } r=0 \\ C_{L,+} & , \text{if } r=1. \end{cases}$$

In case (1.2) we saw for the NNGP-kernel that

$$G_l(\tau(1-t)) = c_{l,\tau}t^{\gamma+1} + Q_{-1,\gamma+1}(t) = Q_{-1,\gamma}(t) \qquad (D.12)$$

holds for $\tau = \pm 1$, $l \ge 1$. Hence, it suffices to show

$$\sigma_w^2 \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-(1-t)))H_{l-1}(-(1-t)) = C_{l,-}t^\gamma + Q_{-1,\gamma}(t), \qquad (D.13)$$

$$C_{L,-} \neq \begin{cases} -C_{L,+} & , \text{if } r=0 \\ C_{L,+} & , \text{if } r=1. \end{cases}$$

in order to obtain Eq. (D.11).

(a) Let $\sigma_b^2 > 0$. This is the most difficult sub-case. Without loss of generality assume $\sigma_w^2 = 1$.

When analyzing $H_l(\tau(1-t))$, Eq. (D.10) the constants $C_{l,\tau}$ of $t^\gamma$ stem from the summand

$$\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(\tau(1-t)))H_{l-1}(\tau(1-t)) = C_{l,\tau}t^\gamma + Q_{-1,\gamma}(t) \qquad (D.14)$$

as the other summands in Eq. (D.10) are in $Q_{-1,\gamma}(t)$ by Eq. (D.12).

We show by induction in $l \ge 2$ that the following statements hold:

$$\bullet \operatorname{sgn} C_{l,+} = (-1)^{s+1},$$

$$\bullet |C_{l,+}| > |C_{l,-}|.$$

The first point is a technical tool for the induction, ensuring that the coefficients of $t^\gamma$ for $\tau = 1$ suffer no annihilation because of different signs of its summands: The coefficients of $t^\gamma$ are the sum of the product of the $t^\gamma$-coefficient of one factor with the constant coefficient of the other factor in Eq. (D.14). The second point then proves this sub-case of the theorem.

By Theorem A.6 we have

$$\widehat{\varphi'_{\sqrt{\alpha_l}}}(\tau(1-t)) = (-\tau)^{s+1} d_l t^\gamma + Q_{-1,\gamma}(t), \qquad (D.15)$$

where $d_l > 0$, and furthermore $\widehat{\varphi'_{\sqrt{\alpha_l}}}(\cdot)$ is smooth on $(-1, 1)$ by Lemma C.3 (d).

For the induction base $l = 2$ we have $G_1(\tau(1-t)) = \tau(1-t)$, $H_1(\tau(1-t)) = \sigma_b^2 + \tau - \tau t$ which yields

$$\widehat{\varphi'_{\sqrt{\alpha_1}}}(G_1(\tau(1-t)))H_1(\tau(1-t)) = ((-\tau)^{s+1} d_2 t^\gamma + Q_{-1,\gamma}(t))(\sigma_b^2 + \tau - \tau t) \\ = (-\tau)^{s+1} d_2(\sigma_b^2 + \tau)t^\gamma + Q_{-1,\gamma}(t),$$

i.e. $C_{2,+} = (-1)^{s+1}d_2(\sigma_b^2 + 1)$ and $C_{2,-} = d_2(\sigma_b^2 - 1)$ and the claim holds for $l = 2$ since $\sigma_b^2 > 0$.

For the induction step, let the claim hold for $l \ge 2$. First we show the auxiliary statement

$$H_r(1) > |H_r(t)|, \quad t \in [-1, 1), \quad r \in \mathbb{N}, \qquad (D.16)$$

which is in the spirit of Lemma D.8, by induction in $r \ge 1$. In the base case $r = 1$, we have $H_1(t) = \sigma_b^2 + t$ and the claim holds. In the induction step, let the claim hold for $r \ge 1$. We use Notation 7 and obtain

$$H_{r+1}(t) = \sigma_b^2 + \widehat{\varphi'_{\sqrt{\alpha_r}}}(G_r(t)) + \widehat{\varphi'_{\sqrt{\alpha_r}}}(G_r(t))H_r(t).$$

Now, for any $t \in [-1, 1]$ we have $G_r(1) = 1$, $G_r(t) \in [-1, 1]$ by Lemma D.8 and by Lemma C.3 we obtain

$$\begin{align} \widehat{\varphi'_{\sqrt{\alpha_r}}}(1) &\ge |\widehat{\varphi'_{\sqrt{\alpha_r}}}(t)| \\ \widehat{\varphi'_{\sqrt{\alpha_r}}}(1) &\ge |\widehat{\varphi'_{\sqrt{\alpha_r}}}(t)| \end{align} \qquad (D.17)$$
and Eq. (D.16) follows.
By a Taylor expansion we have

$$G_l(\tau(1-t)) = G_l(\tau) - \tau t G'_l(\tau) + Q_{1,\gamma}(t) \quad (D.18)$$

as $G_l(\tau(1-t)) = Q_{-1,\gamma}(t)$ by Eq. (D.12) and furthermore $G_l(t) \in [-1, 1]$, $G_l(1) = 1$ by Lemma D.8. For $\tau = 1$ we have by Eq. (D.18) and Eq. (D.15)

$$\widehat{\varphi'_{\sqrt{\alpha_l}}}(G_l(1-t)) = (-1)^{s+1} d_l G'_l(1) t^\gamma + Q_{-1,\gamma}(t).$$

$$\widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(1-t))} H_l(1-t) = \underbrace{((-1)^{s+1} d_l G'_l(1) H_l(1) + \widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(1))}_{=C_{l+1,+}} C_{l,+}) t^\gamma}_{=} + Q_{-1,\gamma}(t)$$

where furthermore

$$G'_l(1) \ge |G'_l(t)| \quad G'_l(1) > 0 \quad (D.19)$$

hold for all $t \in [-1, 1]$ by Lemma C.3 and Notation 7. We observe that sgn $C_{l,+} = (-1)^{s+1}$ holds as claimed, that is, $C_{l+1,+}$ suffers no annihilation from different signs of its summands.

For $\tau = -1$ the possible cases are $G_l(-1) \in (-1, 1)$, for which we obtain $\widehat{\varphi'_{\sqrt{\alpha_l}}}(G_l(-1-t)) = Q_{-1,\gamma}(t)$ by Proposition B.2 (d) and Eq. (D.12), and $|G_l(-1)| = 1$, for which we obtain

$$\widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(-1-t))} = \tilde{a}_{l,\tau} d_l G'_l(-1) t^\gamma + Q_{-1,\gamma}(t),$$

$\tilde{a}_{l,\tau} = \pm 1$ and $|G'_l(-1)| \le G'_l(1)$ by Eq. (D.19). Hence, we have

$$\begin{aligned} \left| \widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(-1-t))} \right| &= a_{l,-} t^\gamma + Q_{-1,\gamma}(t) \\ |a_{l,-}| &\le |d_l G'_l(1)| \end{aligned} \quad (D.20)$$

This yields

$$\widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(-1-t))} H_l(-1-t) = \underbrace{(a_{l,-} H_l(-1) + \widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(-1))}_{=C_{l+1,-}} C_{l,-})}_{=} t^\gamma + Q_{-1,\gamma}(t)$$

and comparing the coefficients $C_{l+1,+}$ and $C_{l+1,-}$ we finally obtain

$$\begin{aligned} |C_{l+1,+}| &= |d_l G'_l(1) H_l(1)| + \left| \widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(1))}_{=C_{l+1,+}} C_{l,+} \right| \\ &> |a_{l,-} H_l(-1)| + \left| \widehat{\varphi'_{\sqrt{\alpha_l}}(G_l(-1))}_{=C_{l+1,-}} C_{l,-} \right| \\ &\ge |C_{l+1,-}| \end{aligned}$$

using Eq. (D.16) and Eq. (D.20) to compare the first summand and the induction hypothesis and Eq. (D.17) for the second summand.

(b) Suppose $\sigma_b^2 = 0$, $\tilde{\varphi}$ neither even nor odd and $L \ge 3$, that is $\tilde{\varphi} = \varphi$.

Without loss of generality assume $\sigma_w^2 = 1$ for simpler notation. We show $|C_{2,+}| \ge |C_{2,-}|$, for $l \ge 3$ we show that $|C_{l-1,+}| \ge |C_{l-1,-}|$ implies $|C_{l,+}| > |C_{l,-}|$. That yields Eq. (D.11). Furthermore we show that sgn($C_{l,+}$) = (-1)$^s$ holds, which is required for the corresponding recursion.

As $G_1(t) = t$, we have for $\tau = \pm 1$ by Theorem A.6

$$\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_1(\tau(1-t))) = (-\tau)^s C_2 t^\gamma + Q_{-1,\gamma}(t), \quad t \searrow 0,$$

where $C_2 > 0$. Now, since we have $H_1(\tau(1-t)) = \tau(1-t)$ we obtain from Eq. (D.10)

$$H_2(\tau(1-t)) = \tau(-\tau)^s C_2 t^\gamma + Q_{-1,\gamma}(t)$$

and the claim holds for $l = 2$. Now, we take a close look at $C_{l,+}$ for $l \ge 3$. Recalling $G_l(1-t) = G_{l-1}(0) - tG'_{l-1}(1) + Q_{1,\gamma}(t)$ from the considerations done for the NNGP above, we see that for

$$H_l(1-t) = \sigma_b^2(1-\sigma_i^2) + \alpha_l G_l(t) + \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1-t)) H_{l-1}(1-t)$$
the terms contributing to the coefficient of $t^\gamma$ only stem from the product of $\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1-t))$ and $H_{l-1}(1-t)$ and by Theorem A.6 we have

$$ \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1-t)) = (-1)^s G'_{l-1}(1)c_l t^\gamma + \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(1) + Q_{0,\gamma}(t) $$

for a constant $c_l > 0$. As $G'_{l-1}(1) > 0$ by Lemma D.8, the sign of the coefficient is $(-1)^s$. Together with

$$ H_{l-1}(1-t) = H_{l-1}(1) + C_{l-1,+}t^\gamma + Q_{0,\gamma}(t) $$

we obtain

$$ \begin{aligned} & \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1-t))H_{l-1}(1-t) \\ &= ((-1)^s G'_{l-1}(1)c_l t^\gamma + \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(1) + Q_{0,\gamma}(t)) \cdot (H_{l-1}(1) + C_{l-1,+}t^\gamma + Q_{0,\gamma}(t)) \\ &= \underbrace{((-1)^s G'_{l-1}(1)c_l H_{l-1}(1) + \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(1)C_{l-1,+})}_{=C_{l,+}} t^\gamma + Q_{-1,\gamma}(t). \end{aligned} $$

We observe that as $G'_{l-1}(1), c_l, H_{l-1}(1), \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(1) > 0$ holds we indeed have $\operatorname{sgn}(C_{l,+}) = (-1)^s$ and observe

$$ |C_{l,+}| > \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(1)|C_{l-1,+}|. \qquad (\text{D.21}) $$

Investigating $C_{l,-}$ we note that as $G_{l-1}([-1, 1)) \subseteq (-1, 1)$ holds by Lemma D.8, Lemma C.3 (d) yields $\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}|_{(-1,1)} \in C^\infty((-1, 1))$ and by Proposition B.2 (e) we have

$$ \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1-t)) = \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1)) + Q_{0,\gamma}(t), $$

since $G_{l-1}(-1-t) = Q_{-1,\gamma}(t)$ holds. As before, coefficients of $t^\gamma$ can only stem from the product of $\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1-t))$ and $H_{l-1}(-1-t)$ and where we have

$$ \begin{aligned} & \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1-t))H_{l-1}(-1-t) \\ &= (\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1)) + Q_{0,\gamma}(t)) \cdot (H_{l-1}(1) + C_{l-1,-}t^\gamma + Q_{0,\gamma}(t)) \\ &= \underbrace{\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1))C_{l-1,-}}_{=C_{l,-}} t^\gamma + Q_{-1,\gamma}(t). \end{aligned} $$

Lemma C.3 (c) yields

$$ \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1)) < \widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1)) $$

as $G_{l-1}(-1) \in (-1, 1)$ and we obtain from Eq. (D.21) as desired

$$ |C_{l,-}| = |\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(-1))C_{l-1,-}| < |\widehat{\varphi'_{\sqrt{\alpha_{l-1}}}}(G_{l-1}(1))C_{l-1,+}| \\ < |C_{l,+}|. $$

(c) Let $\sigma_b^2 = 0$ and let $\tilde{\varphi}$ be odd, so we have $r = 1$. By Proposition D.4 $H_l$ is odd for all $l \in \mathbb{N}$ and

$$ H_L(-1-t) = -H_L(1-t) = -(C_{L,+}t^\gamma + Q_{-1,\gamma}(t)) $$

follows. We obtain Eq. (D.11) as $C_{L,+} \neq C_{L,-} = -C_{L,+}$ holds.

(d) In the case $\sigma_b^2 = 0$ and $\tilde{\varphi}$ even Proposition D.4 yields $H_L$ even and we argue as above.

(2.2), (2.3) The arguments work just as in the NNGP cases (1.3) or respectively (1.4).

Polynomial case:

(3.1) This is the statement of Lemma D.13.

(3.2) Elementary.

□
## D.5 A close look at the even/odd degree of $G_l$ and $H_l$

In this subsection we develop the methods required to deal with smooth activations in Theorem 9, where the only difficulty not covered by Theorem A.2 is investigating which eigenvalues of the NNGP-kernel and NTK-kernel are strictly positive.

[MS→DH: should convergence of power series be discussed more?]

**Definition D.11.** Let $p(t) = \sum_{n \ge 0} \lambda_n t^n$, $q(t) = \sum_{n \ge 0} \eta_n t^n$ be power series which converge on an interval $I \subset \mathbb{R}$. We define the degree of $p$ as $\deg(p) := \sup(\{n \in \mathbb{N}_0 | \lambda_n \ne 0\} \cup \{-\infty\}) \in \mathbb{N}_0 \cup \{-\infty, \infty\}$ and the even/odd degree of $p$ as

$$ \begin{aligned} \deg_{ev}(p) &:= \deg(t \mapsto p(t) + p(-t)), \\ \deg_{od}(p) &:= \deg(t \mapsto p(t) - p(-t)). \end{aligned} $$

and say

$$ p =_{deg} q $$

if and only if $\deg_{ev}(p) = \deg_{ev}(q)$ and $\deg_{od}(p) = \deg_{od}(q)$. That is to say, we compare only the highest even/odd degrees for this notion of equality.

Let $f \in L_2(\mathcal{N}(0, 1))$ be a function. Then $f$ equals its convergent Hermite representation $t \mapsto \sum_{n \ge 0} a_n(f)h_n(t)$ almost everywhere, and we define $\deg_{Her}(f)$, $\deg_{ev}^{Her}(f)$, $\deg_{od}^{Her}(f)$ and $f =_{deg}^{Her} g$ based on the Hermite coefficients $(a_n)_{n \ge 0}$ analogously to the above definitions.

Furthermore we also use the notation $f =_{deg}^{Pow,Her} g$ to denote that the even/odd degree of $f$ and $g$ coincide when $f$ is a power series and $g$ is displayed in the Hermite basis. $\llap{\blacksquare}$

Note that $\deg_{ev}(p) = -\infty$ if $p$ is odd and respectively $\deg_{od}(p) = -\infty$ if $p$ is even, resembling the common convention for the degree of polynomials.

**Lemma D.12 (Odd/even degree of power series with non-negative coefficients).** Let $f, g, \tilde{f}, \tilde{g}$ be power series with non-negative coefficients converging on an interval $I$, such that the products and compositions below are absolutely convergent on $\bar{I}$. We use the conventions $\infty - \infty := -\infty$, $0 \cdot \infty := 0$, $a \vee b := \max\{a, b\}$.

1. (a) The sum $g + f$ fulfills

$$ \begin{aligned} \deg_{ev}(g + f) &= \deg_{ev}(g) \vee \deg_{ev}(f), \\ \deg_{od}(g + f) &= \deg_{od}(g) \vee \deg_{od}(f). \end{aligned} $$

(b) The product $g \cdot f$ fulfills

$$ \begin{aligned} \deg_{ev}(g \cdot f) &= (\deg_{ev}(g) + \deg_{ev}(f)) \vee (\deg_{od}(g) + \deg_{od}(f)), \\ \deg_{od}(g \cdot f) &= (\deg_{ev}(g) + \deg_{od}(f)) \vee (\deg_{od}(g) + \deg_{ev}(f)). \end{aligned} $$

(c) Let $f \ne 0$. The composition $g \circ f$ fulfills

$$ \begin{aligned} \deg_{ev}(g \circ f) &= (\deg_{ev}(g) \deg_{ev}(f)) \vee (\deg_{ev}(g) \deg_{od}(f)) \\ &\quad \vee (\deg_{od}(g) \deg_{ev}(f)) \vee ((\deg_{od}(g) - 1) \deg_{od}(f) + \deg_{ev}(f)) \\ \deg_{od}(g \circ f) &= (\deg_{od}(g) \deg_{od}(f)) \vee ((\deg_{od}(g) - 1) \deg_{ev}(f) + \deg_{od}(f)) \\ &\quad \vee ((\deg_{ev}(g) - 1) \deg_{od}(f) + \deg_{ev}(f)) \\ &\quad \vee ((\deg_{ev}(g) - 1) \deg_{ev}(f) + \deg_{od}(f)). \end{aligned} $$

2. Let $\tilde{f} =_{deg} f, \tilde{g} =_{deg} g$. Then we have

$$ \begin{aligned} g + f &=_{deg} \tilde{g} + \tilde{f}, \\ g \cdot f &=_{deg} \tilde{g} \cdot \tilde{f}, \\ g \circ f &=_{deg} \tilde{g} \circ \tilde{f}. \end{aligned} $$

*Proof.* 1. The requirement that $f, g$ have nonzero coefficients is crucial as it prevents the elimination of coefficients.

We only prove the even-statement in iii) as this is the most difficult statement. The other claims are similar but easier to prove.
As a consequence of the absolute convergence we observe that $g \circ f = \lim_{n \to \infty} g_n \circ f_n$ and $\deg_{\text{ev}}(g \circ f) = \lim_{n \to \infty} \deg_{\text{ev}}(g_n \circ f_n)$, $\deg_{\text{od}}(g \circ f) = \lim_{n \to \infty} \deg_{\text{od}}(g_n \circ f_n)$ holds, where $f_n(t) = \sum_{i=0}^{n} \lambda_i t^i$, $g_n(t) = \sum_{j=0}^{n} \mu_j t^j$. Hence, we can assume without loss of generality that $\deg f, \deg g < \infty$ holds and obtain

$$ g \circ f(t) = \sum_{i \ge 0} \mu_i \left( \sum_{j \ge 0} \lambda_j t^j \right)^i = \sum_{i \ge 0} \mu_i \left( \sum_{j_1, \dots, j_i \ge 0} \lambda_{j_1} \cdots \lambda_{j_i} t^{j_1 + \cdots + j_i} \right) \\ = \sum_{i \ge 0} \sum_{j_1, \dots, j_i \ge 0} \mu_i \lambda_{j_1} \cdots \lambda_{j_i} t^{j_1 + \cdots + j_i}. $$

We note that all occurring coefficients are non-negative, as they are products of the non-negative coefficients $\lambda_j, \mu_i$. In the above expressions, there are multiple summands contributing to a power $t^r$, however, the coefficient of $t^r$ in the power series $g \circ f$ is nonzero if and only if any of the summands $\mu_i \lambda_{j_1} \cdots \lambda_{j_i}$ with $j_1 + \cdots + j_i = r$ is nonzero. Define

$$ I(f) := \{i \in \mathbb{N}_0 | \lambda_i \neq 0\} \cup \{-\infty\}, \\ I(g) := \{i \in \mathbb{N}_0 | \mu_i \neq 0\} \cup \{-\infty\} $$

and it is immediately clear that the coefficient $\mu_i \lambda_{j_1} \cdots \lambda_{j_i}$ is nonzero if and only if $i \in I(g) \setminus \{-\infty\}, j_1, \dots, j_i \in I(f) \setminus \{-\infty\}$. The reason we add $\{-\infty\}$ to $I(f), I(g)$ is that it allows dealing with cases such as $f = 0$ or $f$ odd in a simple manner. In order to investigate $\deg_{\text{ev}}(g \circ f)$ we stepwise simplify:

$$ \begin{aligned} \deg_{\text{ev}}(g \circ f) &= \max_{i \in I(g)} \max_{\substack{j_1, \dots, j_i \in I(f) \\ j_1 + \dots + j_i \text{ even}}} j_1 + \dots + j_i \\ &= \max_{i \in I(g)} \max_{\substack{i \le i \\ i \text{ even}}} (i - \tilde{i}) \deg_{\text{ev}}(f) + \tilde{i} \deg_{\text{od}}(f) \end{aligned} \tag{D.22} $$

$$ = \max \left\{ \max_{\substack{\tilde{i} \le \deg_{\text{ev}}(g) \\ \tilde{i} \text{ even}}} (\deg_{\text{ev}}(g) - \tilde{i}) \deg_{\text{ev}}(f) + \tilde{i} \deg_{\text{od}}(f), \right. \tag{D.23} $$

$$ \max_{\substack{\tilde{i} \le \deg_{\text{od}}(g) \\ \tilde{i} \text{ even}}} (\deg_{\text{od}}(g) - \tilde{i}) \deg_{\text{ev}}(f) + \tilde{i} \deg_{\text{od}}(f) \} \tag{D.24} $$

$$ = \max \left\{ \begin{aligned} & \max\{ \deg_{\text{ev}}(g) \deg_{\text{ev}}(f), \deg_{\text{ev}}(g) \deg_{\text{od}}(f) \}, \\ & \max\{ \deg_{\text{od}}(g) \deg_{\text{ev}}(f), \deg_{\text{ev}}(f) + (\deg_{\text{od}}(g) - 1) \deg_{\text{od}}(f) \} \end{aligned} \right\}. $$

Here, Eq. (D.22) is obtained by observing

$$ j_1 + \dots + j_i \le \deg_{\text{ev}}(f) + j_2 + \dots + j_i, \text{ if } j_1 \text{ is even or respectively} $$

$$ j_1 + \dots + j_i \le \deg_{\text{od}}(f) + j_2 + \dots + j_i, \text{ if } j_1 \text{ is odd,} $$

where the sum on the right hand side remains even. Similarly, Eq. (D.23) follows by observing for $m \in \mathbb{N}_0$ the inequality

$$ (i - \tilde{i}) \deg_{\text{ev}}(f) + \tilde{i} \deg_{\text{od}}(f) \le \max \left\{ (2m + i - (2m + \tilde{i})) \deg_{\text{ev}}(f) + (2m + \tilde{i}) \deg_{\text{od}}(f), \right. \\ \left. (2m + i - \tilde{i}) \deg_{\text{ev}}(f) + \tilde{i} \deg_{\text{od}}(f) \right\} $$

as by assumption $\deg_{\text{ev}}(f) \ge 0$ or $\deg_{\text{od}}(f) \ge 1$ hold. Finally, Eq. (D.24) follows by observing that extremal behavior in Eq. (D.23) is obtained by the highest or the lowest valid $\tilde{i}$. The equation for $\deg_{\text{od}}(g \circ f)$ follows analogously.

2. Follows from 1. $\square$

In the following lemma, we investigate the even/odd degree of $\kappa_L^{\text{NNGP}}$ and $\kappa_L^{\text{NTK}}$ and then deploy Lemma D.6 to characterize which eigenvalues of $k_L^{\text{NNGP}}, k_L^{\text{NTK}}$ are strictly positive.

**Lemma D.13.** Let the activation function $\phi$ fulfill Assumption 5 and let $L \ge 2$. We use the conventions $\infty^0 := 1$ and $\infty - \infty := -\infty$. Then, the nonzero eigenvalues of the NNGP-kernel $k_L^{\text{NNGP}}$ and the NTK-kernel $k_L^{\text{NTK}}$ defined in Notation 7 fulfill

$$ \mu_{2k+r,d}(k_L^{\text{NNGP}}) > 0 \text{ if and only if } \begin{cases} 2k+r \le \deg_{\text{ev}}(G_L) & , r=0, \\ 2k+r \le \deg_{\text{od}}(G_L) & , r=1, \end{cases} $$
$$ \mu_{2k+r,d}(k_L^{\text{NTK}}) > 0 \text{ if and only if } \begin{cases} 2k+r \le \text{deg}_{ev}(H_L) & , r=0, \\ 2k+r \le \text{deg}_{od}(H_L) & , r=1. \end{cases} $$

For the NNGP-term $G_L$ define $\sigma^2 := \sigma_i^2 \sigma_b^2$, and for the NTK-term $H_L$ define $\sigma^2 := \sigma_b^2$. That is, the relevant row of the table is different for $G_L$ or respectively for $H_L$ in the case $\sigma_i^2 = 0$, $\sigma_b^2 > 0$.

The even/odd degree of $G_L$ or respectively $H_L$ is given by

<table><thead><tr><th></th><th>deg<sub>ev</sub>(G<sub>L</sub>), deg<sub>ev</sub>(H<sub>L</sub>)</th><th>deg<sub>od</sub>(G<sub>L</sub>), deg<sub>od</sub>(H<sub>L</sub>)</th></tr></thead><tbody><tr><td>σ<sup>2</sup> > 0, deg<sup>Her</sup><sub>ev</sub>(φ) > deg<sup>Her</sup><sub>od</sub>(φ)</td><td>deg<sup>Her</sup><sub>ev</sub>(φ)<sup>L-1</sup></td><td>deg<sup>Her</sup><sub>ev</sub>(φ)<sup>L-1</sup> - 1</td></tr><tr><td>σ<sup>2</sup> > 0, deg<sup>Her</sup><sub>ev</sub>(φ) < deg<sup>Her</sup><sub>od</sub>(φ)</td><td>deg<sup>Her</sup><sub>od</sub>(φ)<sup>L-1</sup> - 1</td><td>deg<sup>Her</sup><sub>od</sub>(φ)<sup>L-1</sup></td></tr><tr><td>σ<sup>2</sup> = 0, deg<sup>Her</sup><sub>ev</sub>(φ) > deg<sup>Her</sup><sub>od</sub>(φ)</td><td>deg<sup>Her</sup><sub>ev</sub>(φ)<sup>L-1</sup></td><td>deg<sup>Her</sup><sub>ev</sub>(φ) (deg<sup>Her</sup><sub>ev</sub>(φ)<sup>L-2</sup> - 1) + deg<sup>Her</sup><sub>od</sub>(φ)</td></tr><tr><td>σ<sup>2</sup> = 0, deg<sup>Her</sup><sub>ev</sub>(φ) < deg<sup>Her</sup><sub>od</sub>(φ)</td><td>deg<sup>Her</sup><sub>od</sub>(φ) (deg<sup>Her</sup><sub>od</sub>(φ)<sup>L-2</sup> - 1) + deg<sup>Her</sup><sub>ev</sub>(φ)</td><td>deg<sup>Her</sup><sub>od</sub>(φ)<sup>L-1</sup></td></tr></tbody></table>

*Proof.* Let $\kappa: [-1, 1] \to \mathbb{R}$, $\kappa(t) = \sum_{j \ge 0} \lambda_j t^j$ be a power series. Let $k$ be the corresponding radial kernel on $\mathbb{S}^d$ given by $k(x, y) := \kappa(\langle x, y \rangle)$ and let $(\mu_i)_{i \ge 0}$ be the decreasing series of eigenvalues of $k$ counted by algebraic multiplicity.

**Reduction to polynomial degrees of $\kappa$:** We show that the largest even/odd index $i$ corresponding to a nonzero eigenvalue $\mu_i \ne 0$ of $k$ is the same as the largest even/odd index $j$ corresponding to a nonzero coefficient $\lambda_j$ of the power series representation of $\kappa$. By convention, if there are infinitely many such indices, we say that the largest one equals $\infty$ and we say that the largest even/odd nonzero index equals $-\infty$ if all corresponding eigenvalues or respectively coefficients are equal to zero.

* Hubbert et al. (2015) show that the eigenvalues $\mu_i$ have the same sign as the Legendre coefficients $b_k(\kappa)$ obtained by displaying $\kappa$ by Legendre polynomials, which form an orthonormal basis of $L_2([-1, 1])$.

* The bases of the Legendre polynomials $(p_i)_{i \ge 0}$, the Hermite polynomials $(h_i)_{i \ge 0}$ and the monomials $(t \mapsto t^i)_{i \ge 0}$ share the properties that the $i$-th element is a polynomial of degree $i$ and is it even/odd if $i$ is even/odd. Hence, when we can display $\kappa$ as a sum in those bases, the largest even/odd index $i$ of nonzero coefficients coincide.

* Hence, we see that indeed the largest even/odd index of a nonzero eigenvalue of $k$ corresponds to the largest even/odd index of a nonzero coefficient of the power series representation of $\kappa$.

Hence, in order to obtain the largest even/odd index of a nonzero eigenvalue of $k_L^{\text{NNGP}}$ or respectively $k_L^{\text{NTK}}$ it suffices to determine the largest even/odd index of the nonzero coefficients of the power series representation of $G_L$ or respectively $H_L$. Given a nonzero eigenvalue with even/odd index, Lemma D.6 shows that all eigenvalues with smaller even/odd index are strictly positive and we can conclude the claim. Hence, all that remains is to show that $G_L$ and $H_L$ can be represented as a power series and that the even/odd degrees of $G_L$ and $H_L$ are as the table claims.

**Simplifying Notation:** By Assumption 5 we have $\varphi \in \mathcal{L}_2(\mathcal{N}(0, 1))$, enabling us to display $\varphi$ in the Hermite basis as $\varphi(t) = \sum_{i \ge 0} \eta_i h_i(t)$. Eq. (A.2) shows

$$ \hat{\varphi}(t) = \sum_{i \ge 0} \eta_i^2 t^i. \qquad (\text{D.25}) $$

Essentially, we obtain for any $\alpha > 0$

$$ \hat{\varphi}_{\cdot \sqrt{\alpha}} = \text{deg} \hat{\varphi} = \text{Pow,Her}_{\text{deg}} \varphi, \quad \hat{\varphi}'_{\cdot \sqrt{\alpha}} = \text{deg} \hat{\varphi}' = \text{Pow,Her}_{\text{deg}} \varphi'. $$

This observation allows us to work with $\hat{\varphi}$ or respectively $\hat{\varphi}'$ instead of the rescaled versions occurring in $G_l$ and $H_l$, simplifying the notation. By Eq. (D.25) all coefficients of the power series of $\hat{\varphi}_{\cdot \sqrt{\alpha}}, \hat{\varphi}'_{\cdot \sqrt{\alpha}}$ are non-negative and hence the coefficients of the power series $G_l, H_l$ are non-negative as well, enabling the use of Lemma D.12 in the following.

**Calculating $G_l$:** We directly obtain

$$ \begin{aligned} \deg(G_l) &= \deg(g_l \circ \dots \circ g_1) = \deg(\hat{\varphi})^{l-1}, \quad l \ge 2, \\ g_1(t) &= \text{deg} t + \sigma_b^2 \sigma_i^2, \\ g_l(t) &= \text{deg} \hat{\varphi}(t) + \sigma_b^2 \sigma_i^2, \quad l \ge 2. \end{aligned} $$

If $\text{deg}_{ev}(\hat{\varphi}) = \text{deg}_{od}(\hat{\varphi})$ holds, then $\text{deg}_{ev}(\hat{\varphi}) = \text{deg}_{od}(\hat{\varphi}) = \infty$ follows and the claim is trivial.
Assume deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) > deg<sub>od</sub>(⟨$\hat{\varphi}$⟩). We show the claim for deg<sub>od</sub>(G<sub>l</sub>) by induction in l ≥ 2 and directly obtain the induction base

$$
\deg_{\text{od}}(G_2) = \begin{cases} \deg_{\text{ev}}(\hat{\varphi}) - 1 & , \text{ if } \sigma_b^2 \sigma_i^2 > 0, \\ \deg_{\text{od}}(\hat{\varphi}) & , \text{ if } \sigma_b^2 \sigma_i^2 > 0. \end{cases}
$$

In the induction step, we have by Lemma D.12

$$
\begin{align*}
\deg_{\text{od}}(G_{l+1}) &= ((\deg_{\text{ev}}(\hat{\varphi}) - 1) \cdot \deg_{\text{ev}}(G_l) + \deg_{\text{od}}(G_l)) \lor ((\deg_{\text{ev}}(\hat{\varphi}) - 1) \cdot \deg_{\text{od}}(G_l) + \deg_{\text{ev}}(G_l)) \\
&\quad \lor (\deg_{\text{od}}(\hat{\varphi}) \cdot \deg_{\text{od}}(G_l)) \lor ((\deg_{\text{od}}(\hat{\varphi}) - 1) \cdot \deg_{\text{ev}}(G_l) + \deg_{\text{od}}(G_l)) \\
&= (\deg_{\text{ev}}(\hat{\varphi}) - 1) \cdot \deg_{\text{ev}}(G_l) + \deg_{\text{od}}(G_l)
\end{align*}
$$

and the claim follows. The case deg<sub>od</sub>(⟨$\hat{\varphi}$⟩) > deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) is handled analogously.

Calculating $H_l$:

If deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) = deg<sub>od</sub>(⟨$\hat{\varphi}$⟩) holds, then deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) = deg<sub>od</sub>(⟨$\hat{\varphi}$⟩) = ∞ follows and the claim is simple.

By definition we have $H_1(t) =_{\deg} t + \sigma_b^2$. For $l \ge 2$ we simplify $H_l(t)$ as

$$
\begin{align*}
H_l(t) &:= \sigma_b^2(1 - \sigma_i^2) + \alpha_l G_l(t) + \sigma_w^2 (\widehat{\varphi' \cdot \sqrt{\alpha_{l-1}}}}(G_{l-1}(t)) H_{l-1}(t) \\
&=_{\deg} \sigma_b^2(1 - \sigma_i^2) + \sigma_b^2 \sigma_i^2 + \sigma_w^2 (\widehat{\varphi \circ G_{l-1}(t) + \varphi' \cdot \sqrt{\alpha_{l-1}}}}(G_{l-1}(t)) H_{l-1}(t) \\
&=_{\deg} G_l(t) + \widehat{\varphi'}(G_{l-1}(t)) H_{l-1}(t) + \sigma_b^2.
\end{align*}
$$

A simple induction yields

$$
\deg(H_l) = \deg(G_l) = \deg(\hat{\varphi})^{l-1}
$$

and it remains to calculate deg<sub>ev</sub>(H<sub>l</sub>) for deg<sub>od</sub>(⟨$\hat{\varphi}$⟩) > deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) and deg<sub>od</sub>(H<sub>l</sub>) for deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) > deg<sub>od</sub>(⟨$\hat{\varphi}$⟩).

While we usually have $|\deg_{\text{ev}}(\hat{\varphi}') - \deg_{\text{od}}(\hat{\varphi}')| = 1$, we need some case work to cover the case where one of the degrees is $-\infty$.

**Case 1:** $\sigma_b^2 > 0$. If $\deg_{\text{ev}}(\hat{\varphi}) = \deg_{\text{od}}(\hat{\varphi})$ holds, then $\deg_{\text{ev}}(\hat{\varphi}) = \deg_{\text{od}}(\hat{\varphi}) = \infty$ follows and the claim is trivial.

Assume deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) > deg<sub>od</sub>(⟨$\hat{\varphi}$⟩). We show that deg<sub>od</sub>(H<sub>l</sub>) = deg<sub>ev</sub>(⟨$\hat{\varphi}$⟩) - 1 holds for all l ≥ 2. For l = 2 we obtain straightforwardly

$$
\deg_{\text{od}}(H_2) = \deg_{\text{ev}}(\hat{\varphi}) - 1.
$$

By induction we obtain

$$
\begin{align*}
\deg_{\text{od}}(H_{l+1}) &\ge \deg_{\text{od}}((\hat{\varphi}' \circ G_l) \cdot H_l) \\
&\ge \deg_{\text{ev}}(\hat{\varphi}' \circ G_l) + \deg_{\text{od}}(H_l) = (\deg_{\text{ev}}(\hat{\varphi}) - 1) \deg_{\text{ev}}(\hat{\varphi})^{l-1} + \deg_{\text{ev}}(\hat{\varphi})^{l-1} - 1 \\
&= \deg_{\text{ev}}(\hat{\varphi})^l - 1,
\end{align*}
$$

and as we have $\deg(H_{l+1}) = \deg_{\text{ev}}(\hat{\varphi})^l$, the claim follows. The case $\deg_{\text{od}}(\hat{\varphi}) > \deg_{\text{ev}}(\hat{\varphi})$ works analogously.

**Case 2:** $\sigma_b^2 = 0$. Assume

$$
\deg_{\text{od}}(\hat{\varphi}) > \deg_{\text{ev}}(\hat{\varphi}).
$$

**Case 2.1:** $\deg_{\text{od}}(\hat{\varphi}) = \infty$. We want to show

$$
\deg_{\text{ev}}(H_L) = \begin{cases} \infty & , \text{ if } \deg_{\text{ev}}(\hat{\varphi}) > -\infty, \\ -\infty & , \text{ if } \deg_{\text{ev}}(\hat{\varphi}) = -\infty. \end{cases}
$$

In the case $\deg_{\text{ev}}(\hat{\varphi}) > -\infty$, we have since $\sigma_b^2 = 0$

$$
\deg_{\text{ev}}(H_L) = \deg_{\text{ev}}(G_L) \lor (\deg_{\text{ev}}((\hat{\varphi}' \circ G_{L-1}) \cdot H_L)) \geq \deg_{\text{ev}}(G_{L+1}) = \infty
$$
by the previous case. In the case $\deg_{ev}(\hat{\varphi}) = -\infty$, $H_L$ is odd by Proposition D.4 and hence we have $\deg_{ev}(H_L) = -\infty$ as desired.

**Case 2.2:** $\deg_{od}(\hat{\varphi}) < \infty$. We show the claim by induction in $l$. For $l = 2$ we straightforwardly obtain

$$ \deg_{ev}(H_2) = \deg_{ev}(\hat{\varphi}). $$

In the induction step, let $l \ge 3$ and let $\deg_{ev}(H_l) = \deg_{od}(\hat{\varphi}) (\deg_{od}(\hat{\varphi})^{l-2} - 1) + \deg_{ev}(\hat{\varphi})$ hold as claimed. Since $\sigma_b^2 = 0$, we have

$$ \begin{aligned} \deg_{ev}(H_{l+1}) &= \deg_{ev}(G_{l+1}) \lor (\deg_{ev}((\hat{\varphi}' \circ G_l) \cdot H_l)) \\ &= (\deg_{od}(\hat{\varphi}) (\deg_{od}(\hat{\varphi})^{l-1} - 1) + \deg_{ev}(\hat{\varphi})) \lor (\deg_{ev}(\hat{\varphi}' \circ G_l) + \deg_{ev}(H_l)) \\ &\quad \lor (\deg_{od}(\hat{\varphi}' \circ G_l) + \deg_{od}(H_l)). \end{aligned} $$

Now we show that the first term is the largest one in order to obtain the claim. Firstly, we bound the second term as

$$ \begin{aligned} \deg_{ev}(\hat{\varphi}' \circ G_l) + \deg_{ev}(H_l) &\le \deg(\hat{\varphi}' \circ G_l) + \deg_{ev}(H_l) \\ &= (\deg(\hat{\varphi}) - 1) \cdot \deg(G_l) + \deg_{ev}(H_l) \\ &= (\deg(\hat{\varphi}) - 1) \cdot \deg(\hat{\varphi})^{l-1} + \deg(\hat{\varphi}) (\deg(\hat{\varphi})^{l-2} - 1) + \deg_{ev}(\hat{\varphi}) \\ &= \deg(\hat{\varphi}) (\deg(\hat{\varphi})^{l-1} - 1) + \deg_{ev}(\hat{\varphi}). \end{aligned} $$

In order to bound the third term, we recall $\deg_{od}(H^l) = \deg_{od}(\hat{\varphi})^{l-1}$, hence we need to show $\deg_{od}(\hat{\varphi}' \circ G_l) \le \deg_{od}(\hat{\varphi})^l - \deg_{od}(\hat{\varphi})^{l-1} + \deg_{ev}(\hat{\varphi}) - \deg_{od}(\hat{\varphi})$. If $\deg_{od}(\hat{\varphi}) = 1$ we can see $\hat{\varphi}' \equiv c, c \in \mathbb{R}$ and obtain $\deg_{od}(\hat{\varphi} \circ G_l) = -\infty$. Assume $\deg_{od}(\hat{\varphi}) \ge 3$. In the following we use

$$ \deg_{od}(\hat{\varphi}') \le \deg_{od}(\hat{\varphi}) - 2 < \deg_{od}(\hat{\varphi}) - 1 = \deg_{ev}(\hat{\varphi}). $$

Lemma D.12 (c) allows unrolling $\deg_{od}(\hat{\varphi}' \circ G_l)$ as

$$ \begin{aligned} \deg_{od}(\hat{\varphi}' \circ G_l) &= (\deg_{od}(\hat{\varphi}') \cdot \deg_{od}(G_l)) \\ &\quad \lor ((\deg_{od}(\hat{\varphi}') - 1) \cdot \deg_{ev}(G_l) + \deg_{od}(G_l)) \\ &\quad \lor ((\deg_{ev}(\hat{\varphi}') - 1) \cdot \deg_{od}(G_l) + \deg_{ev}(G_l)) \\ &\quad \lor ((\deg_{ev}(\hat{\varphi}') - 1) \cdot \deg_{ev}(G_l) + \deg_{od}(G_l)) \\ &\le (\deg_{od}(\hat{\varphi}) - 2) \deg_{od}(\hat{\varphi})^{l-1} \\ &\quad \lor (\deg_{od}(\hat{\varphi}) - 3) (\deg_{od}(\hat{\varphi})^{l-1} + \deg_{ev}(\hat{\varphi}) - \deg_{od}(\hat{\varphi})) + \deg_{od}(\hat{\varphi})^{l-1} \\ &\quad \lor (\deg_{od}(\hat{\varphi}) - 2) \deg_{od}(\hat{\varphi})^{l-1} + \deg_{od}(\hat{\varphi})^{l-1} + \deg_{ev}(\hat{\varphi}) - \deg_{od}(\hat{\varphi}) \\ &\quad \lor (\deg_{od}(\hat{\varphi}) - 2) (\deg_{od}(\hat{\varphi})^{l-1} + \deg_{ev}(\hat{\varphi}) - \deg_{od}(\hat{\varphi})) + \deg_{od}(\hat{\varphi})^{l-1} \\ &\le \deg_{od}(\hat{\varphi})^l - \deg_{od}(\hat{\varphi})^{l-1} + \deg_{ev}(\hat{\varphi}) - \deg_{od}(\hat{\varphi}). \end{aligned} $$

The case $\deg(\hat{\varphi}) = \deg_{ev}(\hat{\varphi}) > \deg_{od}(\hat{\varphi})$ is handled analogously. $\square$

## E Polynomial boundedness of derivatives

Here, we prove Proposition 6. Specifically, we want to find a sufficient criterion for $C^\infty$-functions $f$ such that all of their derivatives $f^{(k)}$ are polynomially bounded. To this end, we leverage that typical activation functions use symbolic expressions whose derivatives can be expressed using the same set of symbols, e.g. $\tanh'(x) = 1 - \tanh(x)^2$. Our
goal is to find a set of “base functions” whose derivatives use the same base functions and which are polynomially bounded. We then exploit that the class of polynomially bounded functions is closed under addition, multiplication and composition and therefore also contains the derivatives of the “base functions”. Higher-order derivatives are then easily treated using induction. We first formally define the relevant function classes:

**Definition E.1.** Let $I \subseteq \mathbb{R}$ be an interval. For $m \in \mathbb{N}_0$, let

$$S^{(m)}(I) := \{f \in C^m(I) \mid \forall 0 \le k \le m \exists a, b, q > 0 \forall x \in I : |f^{(k)}(x)| \le a|x|^q + b\}.$$

We note that the class $S^{(\infty)}(I)$ from Definition 4 satisfies $S^{(\infty)}(I) = \bigcap_{m=0}^{\infty} S^{(m)}(I)$.

Now, we formally define some “base functions”:

**Definition E.2.** Define sigmoid, softplus, RBF : $\mathbb{R} \to \mathbb{R}$ by

$$\text{sigmoid}(x) := (1 + e^{-x})^{-1}$$

$$\text{softplus}(x) := \log(1 + e^{-x})$$

$$\text{RBF}(x) := e^{-x^2}.$$

Moreover, let $\Phi$ be the CDF of the normal distribution $N(0, 1)$.

**Lemma E.3.** Let $I \subseteq \mathbb{R}$ be an interval and let $m \in \mathbb{N}_0 \cup \{\infty\}$. Then,

(a) If $m \ge 1$, then $S^{(m)} = \{f \in C^1(I) \mid f \in S^{(0)}(I) \text{ and } f' \in S^{(m-1)}(I)\}$.

(b) $S^{(m)}(I)$ is closed under addition, multiplication and composition of functions.

(c) $S^{(m)}(I)$ contains all polynomials.

(d) $S^{(m)}(I)$ contains sigmoid, tanh, softplus, sin, cos, RBF and $\Phi$.

*Proof.* We prove the statements for $m \in \mathbb{N}_0$, the case $m = \infty$ easily follows.

(a) This statement is straightforward.

(b) For addition, the statement is trivial. Now, let $f, g \in S^{(m)}$. Choose constants $C_1, C_2, q$ such that $|f(x)|, |g(x)| \le C_1 + C_2|x|^q$. Then, $|f(x)g(x)| = O(|x|^{2q})$ and $|f(g(x))| \le C_1 + C_2|g(x)|^q \le C_1 + C_2|C_1 + C_2|x|^q| = O(|x|^{q^2})$. This shows $f \cdot g, f \circ g \in S^{(0)}$. If $m = 0$, we are done. Otherwise, we obtain

$$(f + g)' = f' + g' \in S^{(m-1)}t$$

$$(f \cdot g)' = f' \cdot g + f \cdot g' \in S^{(m-1)}$$

$$(f \circ g)' = (f' \circ g) \cdot g' \in S^{(m-1)}$$

since $S^{(m-1)}$ is closed under addition, multiplication and composition by the induction hypothesis. By (b), we have shown that $f \cdot g, f \circ g \in S^{(m)}$.

(c) This is trivial.

(d) It is not hard to see that all of the mentioned functions are in $S^{(0)}$. For the induction step, we assume that all of the mentioned functions are in $S^{(m-1)}$. Since all of these functions are $C^\infty$ functions, their derivatives are in $C^{m-1}(\mathbb{R})$. Moreover, by inspecting the derivatives and using the previously proven facts as well as the induction hypothesis, it is easy to see that the derivatives are in $S^{(m-1)}(\mathbb{R})$:

$$\text{sigmoid}'(x) = \text{sigmoid}(x) \cdot (1 - \text{sigmoid}(x)) = \text{sigmoid}(x) \cdot (x^0 + (-x^0) \cdot \text{sigmoid}(x))$$

$$\text{tanh}'(x) = 1 - \tanh(x) \cdot \tanh(x)$$

$$\text{softplus}'(x) = \text{sigmoid}(x)$$

$$\sin'(x) = \cos(x)$$

$$\cos'(x) = -\sin(x)$$

$$\text{RBF}'(x) = 2x \cdot \text{RBF}(x)$$

$$\Phi'(x) = \frac{1}{\sqrt{2\pi}} e^{-x^2/2} = \frac{1}{\sqrt{2\pi}} \text{RBF}(2^{-1/2} \cdot x).$$

Hence, by (b), we have shown that the functions are in $S^{(m)}$.

**Proposition 6** (All common activation function satisfy Assumption 5).

(a) If $f$ is constructed by addition, multiplication, and composition of polynomials, sigmoid, tanh, softplus, sin, cos, RBF and $\Phi$, then $f \in S^{(\infty)}(\mathbb{R})$. Moreover, the functions $g(x) = \exp(ax)$ are in $S^{(\infty)}((-\infty, 0])$ for every $a \ge 0$.
(b) Assumption 5 is satisfied for all activation functions from Table 1. It is also satisfied for all $\phi \in S^{(\infty)}$.

Proof.

(a) It follows from elementary calculations that the functions $f(x) = \exp(ax)$ are in $S^{(\infty)}((-\infty, 0])$ for all $a \ge 0$. For the other functions, this follows from Lemma E.3.

(b) This follows directly from (a).

<figure><img src="image_11.png" /></figure>

## F Smoothness of GP sample paths

To prove Theorem 13, we need the following result, which is a variant of Thm. 7.4 of Lukić and Beder (2001):

**Theorem F.1** (Theorem 1.2 in Steinwart 2024). Let $H_1, H_2$ be Hilbert spaces on a set $T$ and let $X$ be a centered Gaussian process with covariance kernel $k$ such that $H_1$ is the RKHS of $k$. Then the following statements hold true:

a) If $H_1 \hookrightarrow H_2$ and furthermore this embedding is a Hilbert-Schmidt operator, then there exists a version $Y$ of $X$ with $\mathbb{P}(Y \in H_2 = 1)$.

b) Otherwise, for all versions $Y$ of $X$ we have $\mathbb{P}(Y \in H_2 = 0)$.

Now, we can prove the desired theorem:

**Theorem 13.** Let $k$ be a dot-product kernel on $\mathbb{S}^d$ whose RKHS is equivalent to a Sobolev space $H^{d+\alpha}(\mathbb{S}^d)$, $\alpha > 0$. Let $X$ be a Gaussian process on $\mathbb{S}^d$ with zero mean and covariance kernel $k$.

i) For any $\varepsilon > 0$ we have $P(Y \in H^{\alpha+d/2+\varepsilon}(\mathbb{S}^d)) = 0$ for any version $Y$ of $X$.

ii) For any $0 < \varepsilon < \alpha$, there exists a version $Y$ of $X$ such that $\mathbb{P}(Y \in H^{d/2+\alpha-\varepsilon}) = 1$ holds.

*Proof.* In case ii), the requirement $\varepsilon < \alpha$ stems from the subsequently used theory, which requires the investigated spaces to be RKHSs, which is for Sobolev spaces $H^s(\mathbb{S}^d)$ equivalent to $s > d/2$.

Considering the Sobolev space $H^{\alpha+d/2+s}(\mathbb{S}^d)$ for $s \in \mathbb{R}$ we observe that for $s > d/2$ we have $H^{d+\alpha}(\mathbb{S}^d) \not\hookrightarrow H^{d/2+\alpha+s}(\mathbb{S}^d)$ and hence we can apply Theorem F.1 b). For $s < d/2$, the embedding $H^{d+\alpha}(\mathbb{S}^d) \hookrightarrow H^{d/2+\alpha+s}(\mathbb{S}^d)$ exists and it remains to investigate when this embedding is a Hilbert-Schmidt operator. For $r \ge 0$, Brauchart and Dick (2013) shows that the Sobolev space $H^r(\mathbb{S}^d)$ is given by

$$H^r(\mathbb{S}^d) := \{f \in L^2(\mathbb{S}^d) \mid \|f\|_{H^r(\mathbb{S}^d)} < \infty\},$$

$$\|f\|_{H^r(\mathbb{S}^d)} := \sum_{l=0}^{\infty} \sum_{i=1}^{N_{l,d}} \lambda_{l,r} \hat{f}_{l,i}$$

for a sequence $\lambda_{l,r} = \Theta_{\forall l}(1+l)^{2r}$, where $\hat{f}_{l,i}$ denotes the Laplace-Fourier coefficients given by

$$\hat{f}_{l,i} := \int_{\mathbb{S}^d} f(x) Y_{l,i}(x) \, d\mu_{\mathbb{S}^d}(x),$$

where furthermore $N_{l,d} = \Theta_{\forall l}((l+1)^{d-1})$ holds. Hence, $(\lambda_{l,r}^{-1/2} Y_{l,i})_{l \ge 0, 1 \le i \le N_{l,d}}$ forms an ONB of $H^r(\mathbb{S}^d)$ which diagonalizes the embedding operator. The Hilbert-Schmidt norm of the embedding $H^{d+\alpha}(\mathbb{S}^d) \hookrightarrow H^{d/2+\alpha+r}(\mathbb{S}^d)$ is now straightforwardly given, cf. Birman and Solomjak (1987)[Chapter 11.2], by

$$\| H^{d+\alpha}(\mathbb{S}^d) \hookrightarrow H^{d/2+\alpha+r}(\mathbb{S}^d) \|_{HS} = \sum_{l=0}^{\infty} \sum_{i=1}^{N_{l,d}} \frac{\lambda_{l,d/2+\alpha+r}}{\lambda_{l,d+\alpha}}. \qquad (F.1)$$

The asymptotics of $\lambda_{l,s}$ and $N_{l,d}$ yield

$$\sum_{i=1}^{N_{l,d}} \frac{\lambda_{l,d/2+\alpha+r}}{\lambda_{l,d+\alpha}} = \Theta_{\forall l}(l+1)^{d-1} \frac{\Theta_{\forall l}(l+1)^{d+2\alpha+r}}{\Theta_{\forall l}(l+1)^{2d+2\alpha}} = \Theta_{\forall l}(l+1)^{r-1}$$

and see that (F.1) is finite if and only if $r < 0$, and the claim follows. □

<figure><img src="image_12.png" /></figure>